When is an Embedding Model  More Promising than Another?
Authors: Maxime DARRIN
Philippe Formont
Ismail Ben Ayed
Jackie CK Cheung
Pablo Piantanida
Venue: NeurIPS 2024 poster (2024)
Keywords: embedders
emeddings
molecules
nlp
llm
foundation models
evaluation
unsupervised
representation learning
information theory
Primary area: interpretability_and_explainability
TL;DR: We provide theoretical foundations for comparing embedding models and propose a tractable approach to compare them, and show that this   approach  strongly correlates with downstream tasks performances.

Abstract: Embedders play a central role in machine learning, projecting any object into numerical representations that can, in turn, be leveraged to perform various downstream tasks. The evaluation of embedding models typically depends on domain-specific empirical approaches utilizing downstream tasks, primarily because of the lack of a standardized framework for comparison. However, acquiring adequately large and representative datasets for conducting these assessments is not always viable and can prove to be prohibitively expensive and time-consuming.  In this paper, we present a unified approach to evaluate embedders. First, we establish theoretical foundations for comparing embedding models, drawing upon the concepts of sufficiency and informativeness. We then leverage these concepts to devise a tractable comparison criterion (information sufficiency), leading to a task-agnostic and self-supervised ranking procedure. We demonstrate experimentally that our approach aligns closely with the capability of embedding models to facilitate various downstream tasks in both natural language processing and molecular biology. This effectively offers practitioners a valuable tool for prioritizing model trials.

When is an Embedder More Promising than Another?
'Maxime DARRIN 1,2,3,4 iPhilippe FORMONT Ismail BEN AYED!:5
Jackie Chi Kit CHEUNG 2.3
Pablo PIANTANIDA 1,2,1,6
International Laboratory on Learning Systems, Mila Quebec Al Institute, McGill University
4 UUniversité Paris-Saclay, "ÉTS Montréal, "CNRS, CentraleSupéélec fequal contribution
maxime. darrinneiiii quebbe.
quebec whwlippe.formontteeeepppeeeeeeee www.onttmimtmllmmmmooo quebec
Abstract
Embedders play central role in machine learning, projecting any object into numer- ical representations that can, in turn, be leveraged to perform various downstream tasks. The evaluation of embedding models typically depends domain-speciff empirical approaches utilizing downstream tasks, primarily because of the lack standardized framework for comparison. However, acquiring adequately large and representative datasets for conducting these assessments is not always viable and can prove be prohibitively expensive and time-consuming. In this paper, we present unified approach to evaluate embedders. First, we establish theoret- ical foundations for comparing embedding models, drawing upon the concepts of sufficiency and informativeness. We then leverage these concepts to devise a tractable comparison criterion (information sufficiency), leading task-agnostic and celf-supervised ranking procedure. We demonstrate experimentally that our ap- proach aligns closely with the capability of embedding models facilitate various downstream tasks in both natural language processing and molecular biology. This effectively offers practitioners a valuable tool for prioritizing model trials
1 Introduction
Embeddings are prominent tool in machine learning and are used in multiple fields, such as natural
language processing 664 (33, computer vision [93 59 12 (53) or bioinformatics [67] 23 [12].
These models embed objects such as images, texts, or molecules into numerical representations that can be used to perform numerous downstream tasks by preserving key features the object [76
Depending on the data modalities, intended purpose, and available resources, embedders showcase wide variety of architectures, training settings (unsupervised, supervised, self-supervised, etc.), objectives (masked language modeling, contrastive learning, etc.) [20] [00 78 [2 33), and datasets
[65] 86 31 38 [17]. And more recently, foundation models have become a natural starting point
to create embedders 21 106 52 73
This diversity and variety of options makes selecting the most promising embedders for data distribution challenging [75]. Most work evaluates embedders focusing on the performance they enable finite set of downstream tasks [85
81 24] Nevertheless, this evaluation process encounters two primary limitations. Firstly, is not scalable concerning the number of embedders and tasks, it requires fitting downstream model for each task. Hence, prioritizing the evaluation the most promising models becomes essential to mitigate computational costs. Secondly, acquiring high-quality labels can be a time-consuming and notably expensive endeavor in various applications. To overcome these limitations, in this paper, we explore task-agnostic evaluation metrics
17 90 91
'The code used to perform all experiments is available at httttps://gtthub.oo http:///...thttooooomosossooootr
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

for embedders relying solely on pairwise comparisons between embedders, i.e., without the need for labeled data in downstream tasks.
More specifically, our contributions can he summarized as follows:
1. An innovative theoretical framework for comparing embedding models: We cast the problem of ranking embedders into the noisy communication channels ordering (Sec. 2.2) and statistical experiments comparison settings (Sec. 2.3 We exploit the notions of suffi- ciency and informativeness and relax them, leveraging the concept of deficiency introduced by Le Cam 663 [Sec. 2.4) which is reframed to account for concepts and features. These concepts provide us with tools to establish an embedder ranking.
2. A practical relaxation: Estimating deficiency presents significant challenges. We propose the concept of information sufficiency (IS), which quantifies the information required to simulate one embedder from another (Sec. 3). We estimate the information efficiency get a task-agnostic and label-free comparison tool for embedders evaluation.
3. Extensive experimental validation: The expected IS correlates with the ability of embed- ders to enable a wide range of downstream tasks. In NLP (Sec. 5) and molecular modeling (Sec. 6), our method respectively achieves Spearman ranking correlations of 0.90 (56 tasks) and 0.94 (31 tasks); providing an efficient model trial prioritization tool for practitioners.
1.1 Related works
Embedding evaluation. Embedding evaluation is mainly performed based on a limited set of downstream tasks (22 81],, for which the embeddings are used as inputs to smaller models. Therefore, embedders evaluation is field- and task-specific. In NLP. 4T 851 they rely on limited set of tasks; more recently, the Massive Text Embedding Benchmark (MTEB) (75) followed this taik-oriented trend and offered standardized test bed for embedders encompassing various down- stream tasks in NP. Devising statistical tests to compare models and learning algorithms has a long history (30) However, most works propose statistical tests relying on the performance of the down- stream tasks of interests (60 [ Other works study the expressiveness of embedders and connect it to performance on downstream tasks [107 25], but mostly focus on geometrical properties of the
high dimensional representation in self-supervised learning settings 2 42 45).
Probing. While probing methods do not aim at comparing embedders, they evaluate their repre- sentations to discover what these models have learned. They train small models on the internal representations of large models perform specific downstream tasks. These procedures allow re- searchers to assess what information is present and recoverable from these embeddings [1O] T 88 84). Other work proposed measuring mutual information (MI) between internal representations and labels. It has been used to evaluate the difficulty of a dataset as the predictiveness of the labels using the features 35 For instance, (97] evaluates the utility of representations astrophysies to predict physical properties. Following this trend, [54] leverages the point-wise MI between Gaussian dis- tributions to evaluate text-to-images and image-to-text generative models. However, none of these methods have focused on comparing embedders in the general case to the best of our knowledge.
2 Theoretical Foundations for Comparing Embedding Models
2.1 Background and notation
We assume that all considered spaces standard Borel 28 Each such space U is equipped with its Borel olaleebra B(U). The set of all probability measures on is denoted by P(U) The total variation distance between P and Q is denoted by IP-QQIITy Given joint probability measure Pxy induced by two random variables X E X and U E U. the Mutual Information [27] is denoted by U() A Markov (or transition probability) kernel between X and U is a mapping Puix B(U) X - [0, 1]. The space of all such Pujx is denoted by KCUU)) and (Mo Puix)(V|pz indicates the composition of Markov kernels M E KC(VU) and Pujx E KC(U))). For further details, refer to lo/ppeddixx A
2.2 Sufficiency and informativeness ordering of embedding models
We aim to compare embedding models without relying on labeled data for downstream tasks. Let us consider two embedding models represented by their Markov kernels (or transition probabilities)

---

Pojx E K(U))) and Pvix E K(VIX), any target set Y of (discrete or continuous) concepts and feature space X with joint probability measure Pyx E P(Y x X) induced by random variables (X, E Y x as illustrated in Figure First, we study the question:
What sufficient conditions must be met by the embedding model U relative V to guarantee that T(Y;U)= > I(Y:V) for all distributions Pyx
iefformation-thhoreee perspec- tive [271. the quality of an embedding model can be likened to the capacity of a noisy communication channel with an uncoded input (e.g., text, molecule...). where a downstream task of interest is per- formed at the output (the embedding) of the channel. Let Y E Y represent the mes- sage (the source) to be communicated over both channels; X represents the transmit- ted signal; and Pujx and Pvix the com- munication channels with outputs U and V. respectively. This process is illustrated in Figure 1 It naturally satisfies the Markov chain Y + X +> (U.V). A desirable property is that embedding models U and retain as much pertinent information as feasible to predict Y.
From
(embedder U)
(embedder simulator)
PEIx
Pvio
(concepts)
(features)
Po P
Px
Pvix
(v( ss (V
(data distribution)
(embedider V)
Figure Communicating concept y Y over two embedding models with prediction pV (V).
We shall be interested in the underlying information relationships between those embedding models that can be interpreted as channel U being "more informative" for communicating Y than channel V. The first attempt to introduce an ordering between communication channels appears in Shannon [94). Körner and Marton later introduced [57] the concepts of "less noisy" (or more informative) and "degraded" (or sufficiency) orderings between channels.
Definition 1 (Sufficiency and informativeness orderings (77) Let Peix and Pvix be two Markov kernels (embedding models).
Sufficiency U Ss V. The embedding model Poix is said to be "sufficient" for the embedding model Puix (or to be degraded w.r.t. U) if and only if there exists another Markov kernel M TV[UU) such that - from U using M without information loss).
0, ie. V be simulated
More informative U >> V. The embedding model Ptrjx is said to be "more informative" (or less noisy) than Pvix if and only if the embedding models always satisfy the inequality
I(Y:U) I(Y;V), VPYx P(Y X).
Proposition 1 (Relationships of sufficiency and information). The following relationships hold:
(i) Sufficiency + informativeness. If the embedding model Pujx is sufficient for the embedding model Pvix, i.e. U >g then U '; V. However, Informativeness + sufficiency. (ii) Informativeness higher capacity to distinguish concepts. the embedding model Pujx is more informative than embedding model Pvix, i.e. U 71 V, then
(LPopy(-luo)) Popy--lyx)) > KL. (ll for any pair of concepts (yo. y1) Y Y and all probability distributions Pyx-
An immediate consequence of claim (i) is that the sufficient condition between embedding models implies that the embedding model U more informative than the embedding model V relative to all target concepts in Y over all possible data distributions: IYY > I(Y; for all probability distributions Pyx.
Remark 1.
Although being more informative than V does not necessarily imply U >s V that being more informative ensures higher statistical discrimination capacity between any pairs of target concepts (for further discussion, see Sec. B.2)
57 66 : (ii) states
Motivated by the concepts of sufficiency and informativeness between embedding models, we can inquire about their statistical consequences for a learner conducting embeddings. More precisely, given finite set of concepts Y, U >s V, is the Bayes risk expected to be smaller when the inference is based on U than when it is based on V/?
inference task on these

---

2.3 Comparing statistical experiments with embedding Models
The pursuit of comparing statistical experiments originated from the seminal paper by Bohnenblust, Shapley, and Sherman [16]. followed by subsequent contributions by Blackwell [13] [4]. They formally established the relationships between sufficiency (Def. I) and inference procedures.
In our framework, a statistical experiment 13 consists of a mathematical abstraction (see Appendix A for further details) intended to represent downstream task where learner aims at inferring concept y E Y from the embeddings U or V. Deciding what embedder should be used to perform given task too general. In this work, we do not take into account the computational cost or the size of an embedder and solely focus on the following question:
What are the necessary and sufficient conditions that ensure that employing the embedding U for any task Pyx leads to lower risk compared to using the embedding V?
Drawing parallels with the theoretical framework established for comparing statistical experiments, relationship can be derived between the concept of sufficiency and the expected risk for a specific task (see Sec. B.5 for further discussion).
We concentrate on the scenario where consists of finite number of concepts (e.g., classification tasks), as it is significant case in its own right [004] and provide fundamental insights for the present work. The next Proposition states an important relation between the concept of sufficiency and the expected Bayes risk on any classification task.
Proposition (Comparison of embedding models through Bayes risks). Given two embedding models Pujx K(UUX) and Pvix K(V(X),, the following statements equivalent:
(i) The embedding model Puix is sufficient relative to Pvix, Le. Es
(ii) For all conditional probability measures Prix on finite alphabet Y, the Bayes risks satisfy
inf Pr (Yu *Y) &
inf Pv:V-+P(Y)
Pr (fv +Y).
Po:U-P(Y)
where YU and Yv are distributed according to Pu and pv (V), respectively.
Remark In other words, if we can fully simulate an embedder from another embedder U, the expected risk across all potential classification tasks cannot be greater when using U compared to V. The proof of this Proposition is given in Sec. B.3 is worth mentioning that various versions of this result are available in the literature [144]. However, our extension here, in a simpler setting, incorporates concepts and features into the experiment comparison framework.
2.4 Challenges in ranking embedding models and their deficiency
According to the notion of "sufficiency". we can distinguish the three following possibilities:
Equivalence: U >ss V and V >g U denoted U x V: U and V can simulate each other. Comparability: ss but V Y 5 only can be simulated from U. Non-comparability: U * V and V Ys U, neither U nor V can simulate each other.
Our results up to now only account for the two first possibilities. However, two embedders are generally not comparable (Sec. B.4). This issue was addressed by Le Cam 63] who introduced the notion of "deficiency".
Definition 2. The deficiency 8(Pojx + Pvix) of Pvix relative to Peix is defined as 63
8(Pujx + Pvix) A inf MEK((VU)
ElIMoPujx Pvix
where the infimum is taken over all Markov kernels (or transition probabilities) M E IC(V)U), mapping stochastically U and V, and measures error between the simulated and true embedders.
ό indicates how well one model can be reconstructed from the other, it induces a natural relaxation of the sufficiency where the reconstruction does not have to be perfec f for us to obtain guarantees
If U ss V. then 8(Pvix + Pojx) 0, while if U As V. then 8(Pvix - Pejx) > 0.

---

the downstream tasks performance (See SScelaroly 1). It avoids the non non-comparability problem by evaluating "how much information" we lose when passing from one model to the other one.
Le Cam 63 showed that, for a given task Y, the deficiency 8( 8(Pujy + Pvjy) is directly related to the expected Bayes risks the task (see B.6). We extend this result to the comparison of two embedding models Puix and Pvix in a task-agnostic manner and build the relation to the expected Bayes risks for any classification task Y.
Corollary 1. Given two embedding models Puix and Pvix satisfying:
(i) The deficiency 5 6(Peix - Pv x) &
(ii) For any conditional distribution Py|x on finte alphabets Y,
inf PU:U--((Y)
Pr (Yu +Y) - -
inf
Pr (Yv + *Y).
pv:V-P(()
Statement (ii) implies (i) provided that 7 > 2| and conversely, (i) implies (it) provided that
The proof of this Corollary is relegated to Sec. B.3
Remark 3. In particular, embedding model U, denoted by Rv, is upper bounded by the expected Bayes risk of the embedding model V, denoted by Ry:
infer that for any classification task Y, the expected Bayes risk of the
RV- - Rv & 8(PUj - Pvix), for all conditional distributions Pylx-
8(xx)x + )ojx)), for all conditional distributions Prxx. If both deficiencies are small, the resulting expected Bayes risks of the embedding models U and will will be close to each other for any larget task Y.
and similarly, |Ru Rvl max (8(Pu
3 Quantifying Information Sufficiency Between Embedding Models
We want to compare embedding models using the concept of deficiency, leveraging Prop. 2 and Corollary These propositions suggest that the performance any classification task of an embedding model U relative to the model V is bounded by 8(Peix -> Pvix). However, estimating the deficiency from data samples is notably challenging (95), and while upper bounds derivation exists, they do not necessarily make it tractable.
3.1 Estimating Information Sufficiency
The deficiency 8(Prjx + Pvix) between two embedding models Peix and Pvix, measures how well U can be used to simulate V using a Markov kernel M E KC(VJU). This section aims to build a tractable proxy for this reconstruction cost. To this end, we estimate how much we can reduce the uncertainty about Z by observing U by learning an appropriate Markov kernel. This corresponds to and can be interpreted the information-theorete counterpart of the deficiency. The information deficiency between U and V is then defined as:
the information sufficiency 29 4
Definition 3 (Information sufficiency). The information sufficiency [s + V), relative to para- metric classes of distributions (V) and (V|U) (multivariate Gaussian mixtures 82] is defined:
Is(U + +V inf FEFe(V)
E[- log f(V)
inf E log (((U)) MEKu(V)U)
(I)
Uncertainty V
Uncertainty when simulating from U with
Remark 4. When the information sufficiency [U + V) is large, it signifies that U offers a substantial amount of information to simulate V. proxy for small deficiency. Conversely, when [U + V) is lower, implics that the channel Pviy is subject to considerable noise or randomness, leading to greater loss of statistical information.
We hence attempt simulate V from U by learning Markov kernel M Ke(V|U), via mixture of multivariate Gaussians, and measure the uncertainty reduction it induces.
5

---

Pairwise embedder evaluation. For set of embbeders (Z represented by their Markov kernels 1 h., we compute the pairwise information sufficiency Is(Z, Z.). The pairwise information sufficiency matrix defines the adjacency matrix of a directed graph of embedders (Figure 2). Corollary shows that embedders sharing high information sufficiency are expected to perform similarly on any downstream tasks, motivating the identification of communities the graph. While the graph construction in O(N²); where N is the number of embedders, it is in practice tractable for reasonable number of embedders (refer Sec. E.6) for more details).
Mode
Figure Pairwise Is for text embedders.
Practical embedding evaluation.
We construct the set of all information sufficiency using Z ST (k) (Zs(Zx + Zi) We build our information sufficiency score (Zs score) by taking the median of ST. (k). Details on the Ig score's estimation can be found in Sec. E.1
4 Experimental Setup
We aim evaluate the practical utility of the Iş score rank and select the best embedders for given data distribution. We compare this ranking to those obtained on various downstream tasks. Our experimental protocol is divided into three main steps:
1. We evaluate the Is score of the models by identifying large and diverse dataset that is supposed to be representative of the data distribution of interest.
2. We train small feedforward neural network (pz,) ) per embedder Pz to perform each downstream task and record its performances (R² score for regression, AUROC/accuracy for binary/multiclaas classification).
3. We compare the models' performances on the downstream tasks and the Is score by measuring three types of correlations: the Pearson correlation, the Spearman correlation, and the Kendall-Tau coefficient See
Sec. E.5 for additional baselines).
Op
g+
Retrieval (15 datasets)
0.89
0.89
0.69
Absorption datasets)
Classification (12 datasets)
0.92
0.88
Distribution datasets)
0.89
0.70
Chustering (1l datasets)
0.86
0.85
0.66
Metabolism datasett)
0.94
STS (10 datasets)
0.92
0.83
Excretion (3 datasets)
0.77
0.60
Reranking (4 datasets)
0.84
0.78
0.64
Toxicity (9 datasets)
0.92
0.75
Average (56 datasets)
0.94
0.90
0.73
ADMET (31 datasets)
0.94
0.80
Additional Classif (8 datasets)
0.89
0.84
(1496 cooks) we Nec.
(a) NLP
(b) Molecular Modelling
Average (56 datasets)
ADMET
allenai-spocter
CenteictPred
udever bioom 500m
25
20
Not- Not-trained
20
G (rrank
15
GraphMivp
MAE Laro
--
FRAD_OMP
0.94
A:: 0.90
0.94 t: 0.80
t:
(d) Molecular Modeling Malli
0
5
10
15
20
25
30
Downstream tasks (rank
Downstream tasks trank 1
(c) NLP
Figure 3: Correlation between Is scores and downstream task performances in (a) NLP and (b) Molecular Modelling. @p is the Pearson correlation, ds the spearman correlation, and 7 is the Kendall- Tau coefficient. See Sec. C.3.1 for unaggregated results in NLP and Sec. D.3 in molecular modeling-
Forr the experiments in molecular modeling, in each subset regression and classification tasks are mixed, Hence, we do not compute the Pearson correlation avoid mixing scores obtained for different metries.

---

Text Embeddings Evaluation
5.1 Experimental setting
Embedders & Datasets. We compared 34 models with different training objectives, training
datasets, and architectures. We included embedders derived from modern LLM such as LLaMA [106], Mistral [52], Gemma [102]. Croissant [37 and T5 encoders [77]; common embedders derived from
BERT architectures [31
38
85 or RobERTa [1] and embedders trained on specific embeddings
objectives such Angle (64] Stella E5 models [[13), LaBSE 38
comprehensive list of the
Tab. Twith their main characteristics and links to the Huggingface Hub for reproducibility. We used them extract embeddings for many different datasets from the MTEB benchmark such as Banking77 [19], Sickr [122], Amazon polarity [72), SNLI 120 and IMDB 70
models can be found in Sec.
We provide the datasets statistics in Sec. C.I Tab. 2
Downstream tasks evaluation. We rely the results released the MTEB leaderboard and compare our rankings to the rankings and scores obtained by the different models on the different tasks. We evaluate additional tasks that are not included in the MTEB benchmark, such tweet_eval g 74 7 109, 9]. DAIR Emotion [92], agnews topic classification 123), Clinc intent detection 62
PAWS-X 118 and Rotten Tomatoes [79].
5.2 Model's Information Sufficiency analysis
gemma-7b-it
CroissanttLLBase
auprage
gemma-2b-it
CroissantCool
Unina-2- esismall
11ama-2-7b-ht
I frank
gemma 2brt
udever-bloom-560n
9amma:
average word
gtokdral
gi-t--large
gemma-2b
UAF-I
0.83
mistral
Stharguu2
LaBSE
20 Downstream tasks (rank 1
e5-large-v2
SFi-Embedding gtr-t5-base
(b)
sf modelee
Information Sufficiency
Classification accuracy
UAE-Large-VV
gtr-t5: t5algrasss-
0.55
0.7
0.45
0.6
stella-base-enay
uemma-22 26
gemma--20 -b-lt
Remmmaa 7b
prmmaa- Tb-t
genma- 28
grareus 2b-lt
gemma-1 7B
gemmas Thok
55-small
(a)
Figure 4: Figure 4a presents the information sufficiency directed graph and the induced communities. Figure 4b displays the performance on additional downstream tasks and models not evaluated in the MTEB leaderboard. FFigure sclss that instruction finetuning positively impacts the models' performance on the downstream tasks and that this improvement is captured by Is.
Correlation with downstream tasks performance. The MTEB Benchmark offers a natural starting point to compare models' ranking according to their performance downstream tasks and their Is score. In Figure 3c we show that the Ig score of an embedder correlates positively with its performance on a wide range of downstream tasks, from classification and similarity tasks to retrieval and clustering tasks. Overall, our Ig score correlates strongly with MTEB's average score (Spearman correlation of 0.90 and Pearson correlation of 0.94, see Figure 3ch and with the subtask

---

performance Figure 3ap. We extended our experiments to a more extensive set of models not included the MTEB benchmark and observed a similar trend (Figure 4b). Per-datasets results are reported in Sec. C.3 and ablations in Seec. C.3.2 All our results show that our estimation of the information sufficiency between models a good proxy for the performance of the models on a wide range of tasks.
Embedder communities. The pair- waie information sufficiency evaluation between the models can be used to cluster them into communities [15] (Fguue 4a| Figure 2 We observe that the extracted clusters group together models that are similar in their training objectives and architectures. LLM-based models such as LLaMA, Mistral, Gemma, and Croissant
clustered together, while BERT-based models share another cluster. Similarly, models trained specifically for embedding purposes, such as UAE-Large-VI and ember-vl, are grouped together. This suggests that the ordering induced by information sufficiency is meaningful and can be used to identify models with similar properties and behaviors. Consistently with Corollary I we observe that the performance of the models on the downstream tasks is similar within the same cluster (Figure C.3.5). In addition, we found that it captures improvements by both steps of pretraining and instruction fine-tuning (Figure 4c Sec. C.3.2)
6 Molecular Modeling
6.1 Experimental setting
Embedders. To process molecular data, embedders can leverage different representations of the molecules, providing an interesting benchmark evaluate the Ig score. We evaluated models derived from the molecular representation learning literature, summed up in [Sec. D.I We considered various input modalities such as string representations (SMILES [144, SELFIES (58) ), 2D-graphs by using graph neural networks (GNNs), and 3D-representations (using the TorchMD-net architecture [80).
We added randomly initialized baseline GNN model that was trained any dataset.
Datasets. To evaluate the information sufficiency between embedders, we compared the models on the ZINC 250k dataset(50), designed to gather compounds that could be relevant to wide range of therapeutic projects. This dataset contains 250k commercially available compounds meant to be used in diverse therapeutic projects.
Downstream tasks. We evaluated the embedders on downstream tasks extracted from the Therapeutic Data Commons [49] platform. This section focuses on ADMET tasks (Absorption, Distribution, Metabolism, Excretion, and Toxicity). Results on Drug-Targe interaction tasks can be found in Sec. D.4] Datasets collected are split into training, validation, and test set, following the scaffold-sphe splltt strategy, further described in see Sec. D.3
6.2 Model's Information Sufficiency analysis
Global results. The Is score ranking is consistent with the results of the embedders on the ADMET downstream tasks, achieving Spearman correlation of 0.95 and a Kendall-tau coefficient of 0.80, as reported in Figure 3d Detailed results for each of the 31 tasks are available in /Sec. D.3 in Tab. Table 3b shows the correlation between the Is score rankings and the performances obtained on the ADMET tasks within each category. High correlations are achieved within most task categories, especially when large tasks are available (containing an important number of molecules). On excretion tasks, the correlation is lower (below 0.8), which can be explained by the fact that these tasks are the most challenging regression tasks available, where the fine-tuned models reach the lowest R² scores between 0 and 0.2 (see Sec. D.3)
Most / Least promising models. We observe in Figure 5b that the most promising models are the Bett models 3 and MoIR [T2), the former trained on SMILES representations to predict variety of computationally available molecular properties, and the latter trained on 2D graphs to preserve equivalence of molecules w.n. chemical reactions. Surprisingly, these models share high predictive mutual information (being assigned to the same Louvain community in iiFuure 5ap.
'We rely on the Louvain community detection implementation from networkx 443 BBrttMTR- X stands for com.Berrt-MK model trained on X molecules.
8

---

MoleDOD SAGE
MoleOgD GIN
ContextPred
GPT-1.28 EMLM-10M
GPT-GNN
AttributeMask
GraphLog
MoleOOD GIN
MoleOOD_SAl -AGE
ContextPred
MolBert
BertMLM-SM
BertMLM-77M
MoleOOD GCN
GPT-4.7M
Not-trained
InfoGraph
AttributeMask
MolBert
BertMLM-10M
GROVER
GPT-19M
GraphCL
GraphMVP
GraphLog
3D-denoising
GraphMVP
3D-Infomax
FRAD QM9
MolR_gen
GPT-1.2B
MolR gat
BertMTR-77M
BertMTR-SM
BertMTR-10M
BertMTR-77M
MolRitag
MolR_ tag
DD-infomax
10
20
ADMET rank
a
(b)
Figure 5: (a) Pairwise information sufficiency graph between the embedders. The center color represents the ability to simulate other models, while the surrounding colors represent the ability to be simulated by other models. Red indicates high ability to simulate or be simulated, while blue indicates low ability. (b) Mean rank of the models (ordered by Is score) downstream tasks.
suggesting that they capture similar information despite significant differences their training methods. These models also appear to be the most competitive on the ADMET tasks. On the other hand, and consistently with Sun er al. [99]s s observation, training methods for 2D-GNNs such as following attribute masking and context prediction objective are deemed as the least informative according to the Is score. This explained by the simplicity of these pretraining objectives for this data modality. These methods are also among the least competitive methods the ADMET downstream tasks.
NLP-inspired models. ChhmBBert-MLA. 3), MolBert 36 and GPT 40 leverage masked lan- guage model objective applied to string representations (SMILES and SELFIES). Unsurprisingly, as seen in Figure 5a these models clustered, suggesting they capture similar information. However, they fail to simulate other models in the pool, resulting in low Is scores, a result consistent with the known limitations these pretraining objectives 23 [05). A noticeable exception is (Chm,GPT--1.2B (the biggest model the pool by far), which displays a sigmificantly higher Iş score.
"Not-trained" GNN, Figure 5h helps visualize the performances of the different models relative our baseline "Not-trained" GNN. Surprisingly, some models are ranked less promising than this baseline by the Is score. However, all of these less promising models obtain poorer performances on the downstream tasks. Similarly, except for InfoGraph [98] every model ranked more promising than the M GNN bascline and obtained better results on ADMET tasks. This surprising result validates evaluation of the Is score w.r.t this baseline.
Limitations and Conclusions
We proposed a principled approach to embedding model evaluation by framing model ranking as variation of comparing statistical experiments. Utilizing concepts of sufficiency, informativeness, and deficiency, we developed mathematically grounded metries for pairwise comparisons between embedders without relying on labeled data in downstream tasks. Our tractable relaxation, termed information sufficiency, demonstrated strong correlations with rankings based on downstream task performance in extensive experiments. Although successful, method still has least two primary

---

limitations. First, its effectiveness depends on the number and diversity of available embedders (eeeSec. E.4p. Future work could explore using randomly initialized embedders (random projections) instead of pre-trained ones. Second, we can enhance our proxy for predicting the deficiency between models by exploring better methods (e.g., estimating the f-divergence) to directly learn the Markov kernel that minimizes the total variation distance, which we leave for future research.
Acknowledgments
This work was granted access to the HPC resources of IDRIS under the allocation 2023- AD011013290RS made by GENCI, and enabled by support provided by Calcul Quebec and the Digital Research Alliance of Canada. We warmly thank Heitor Rapela, Banafsheh Karimian, and Eric Aubinais for their advice and comments about our work. We also owe special highlight to Loic Fosse for the many discussions and hindsights he provided and for the subsequent follow-up projects.
10

---

References
[1] Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In International Conference on Learning Representations. International Conference on Learning Representations, ICLR, 2017.
[2] Kumar K Agrawal, Arnab Kumar Mondal, Arna Ghosh, and Blake Richards. \alpha-req Assessing representation quality in self-supervised learning by measuring eigenspectrum decay. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 17626- 17638. Curran Associates, Inc., 2022.
[3] Walid Ahmad, Elana Simon, Seyone Chithrananda, Gabriel Grand, and Bharath Ramsundar. Chemberta-2: Towards chemical foundation models, 2022.
[4] Suguru Arimoto. Information- considerations on estimation problems. Information and control, 19()):81-114, 1971.
[5] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with joint-embedding predictive architecture, 2023.
[6] Simon Axelrod and Rafael Gómez-Bombarelli Geom, energy-annotated molecular conforma- tions for property prediction and molecular generation. Scientific Data, 91)::185, 2022.
[7] Francesco Barbieri, Jose Gamacho-Collados, Luis Espinosa-Anke and Leonardo Neves. UnietEval:Unified Benchmark and Comparative Evaluation for Tweet Classification. In Proceedings of Findings of EMNLP, 2020.
[8] Francesco Barbieri, Jose Camacho-Colladee Francesco Ronzano, Luis Espinosa-Anke Miguel Ballesteros, Valerio Basile, Viviana Patti, and Horacio Saggion. Semeval 2018 task 2: Mul- tilingual emoji prediction. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 24-33, 2018.
[9] Valerio Basile, Cristina Bosco. Elisabetta Fersini, Debora Nozza, Viviana Patti, Fran- cisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women in Twitter. In Proceed- ings of the 13th International Workshop on Semantic Evaluation, pages 63, Minneapolis, Minnesota, USA, 2019. Association for Computational Linguistics.
[10] Yonatan Belinkov. Probing classifiers: Promises, shortcomings. and advances. Computational Linguistics, 48(1):207-219, 2022.
[11] Alessio Benavoli, Giorgio Corani, Janez Demsar, and Marco Zaffalon. Time for change: tutorial for comparing multiple classifiers through bayesian analysis, 2017.
[12] Usha Bhalla, Alex Oesterling, Suraj Srinivas, Flavio P. Calmon, and Himabindu Lakkaraju. Interpreting clip with sparse linear concept embeddings (splice), 2024.
[13] David Blackwell. Comparison of experiments. In Proceedings of the second Berkeley symposium on mathematical statisties and probability, volume pages 93-103. University of California Press, 1951.
[14] David Blackwell. Equivalent comparisons of experiments. The Annals of Mathematical Statistics, 24(2):265-272, 1953.
[15] Vincent D Blondel, Lean Guillaume, Renaud Lambiotte. and Etienne Lefebvre. Fast unfolding of communities in large networks. Journal of Statistical Mechanics: Theory and Experiment, 2008 (10):0 oct 2008.
[16] Frederic Bohnenblust, Lloyd S. Shapley, and Seymour Sherman. Reconnaissance game theory. 1949.

---

[17] Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean- -an-itttstt Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Pagannii, Geoffrey Irving. Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. Improving language models by retrieving from trillions tokens, 2022.
[18] Nathan Brown, Marco Fiscato, Marwin H.S. Segler, and Alain C. Vaucher. Guacamol: Benchmarking models for de novo molecular design. Journal of Chemical Information and Modeling, 596)):096-1108,, March 2019.
[19] lñigo Casanueva, Tadas Temeinas, Daniela Gerz, Matthew Henderson, and Ivan Vulic. Efficient intent detection with dual sentence encoders. In Proceedings of the 2nd Workshop on NLP for ComAAl - ACL 2020, mar 2020. Data available at http://ittuub.oooolo/./aa.e specific-datasets.
[20] Wei-Cheng Chang, Felix X. Yu, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar. Pre-training tasks for embedding- large-scale retrieval, 2020.
[21] Chang Che, Qunwei Lin, Xinyu Zhao, Jiaxin Huang, and Liqiang Yu. Enhancing multimodal understanding with clip-based image-to-text transformation, 2024.
[22] Yanqing Chen, Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. The expressive power of word embeddings, 2013.
[23] Seyone Chithrananda, Gabriel Grand, and Bharath Ramsundar. Chemberta: Large-scale self-supervised pretraining for molecular property prediction, 2020.
[24] Hyunjin Choi, Judong Kim, Seongho Joe, and Youngjune Gwon. Evaluation of bert and albert sentence embedding performance on downstream nlp tasks, 2021.
[25] Ching- Yao Chuang, Antonio Torralba, and Stefanie Jegelka. The role of embedding complexity in domain-invariant representations, 2019.
[26] Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veličković. Principal neighbourhood aggregation for graph nets. In Advances in Neural Information Processing Systems, 2020.
[27] T. M. Cover and J. A. Thomas. Elements of Information Theory, Wiley, New York, NY, 2nd edition, 2006.
[28] H. Crauel. Random Probability Measures on Polish Spaces. Taylor & Francis, 2002.
[29] Morris H DeGroot. Uncertainty, information, and sequential experiments. The Annals of Muthematical Statistics, 33(2):404-419, 1962.
[30] Janez Demšar. Statistical comparisons of classifiers over multiple data sets. The Journal of Machine learning research, 7:1-30, 2006.
[31] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
[32] Jian Du, Shanghang Zhang, Guanhang Wu, Jose M. F. Moura, and Soummya Kar. Topology adaptive graph convolutional networks, 2018
[33] David Duvenaud, Dougal Maclaurin, Jorge Timothy Hirzel, Alán Aspuru-Guzik, and Ryan P. Adams. Convolutional networks graphs for learning molecular fingerprints, 2015.
Rafael Gómez-Bombareld
[34] Peter Ertl and Ansgar Schuffenhauer. Estimation of synthetic accessibility score of drug-like molecules based on molecular complexity and fragment contributions. J. Cheminformatics, 1:8, 2009.
[35] Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information, 2022.
12

---

[36] Benedek Fabian, Thomas Edlich, Héléna Gaspar, Marwin Segler, Joshua Meyers, Marco Fiscato, and Mohamed Ahmed. Molecular representation learning with language models and domain-relevant auxiliary tasks, 2020.
[37] Manuel Faysse, Patrick Fernandes, Nuno M. Guerreiro, António Loison, Duarte M. Alves, Caio Corro, Nicolas Boizard, João Alves, Ricardo Rei, Pedro H. Martins, Antoni Bigata Casademunt, François Yvon, André F. Martins, Gautier Viaud, Céline Hudelot, and Pierre Colombo. Croissantlmm: A truly bilingual french-english language model, 2024.
[38] Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. Language- agnostic bert sentence embedding, 2022.
[39] Shikun Feng, Yuyan Ni, Yanyan Lan, Zhi-Ming Ma, and Wei Wei-Ying Ma. Fractional denoising for 3D molecular pre-training. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 9938-9961. PMLR, 23-29 Jul 2023.
[40] Nathan Frey, Ryan Soklaski, Simon Axelrod, Siddharth Samsi, Rafael KomezzBombarell Connor W. Coley, and Vijay Gadepally. Neural scaling of deep chemical models. Nature Machine Intelligence, 5(11)::297-1333 November 2023. Publisher: Nature Publishing Group.
[41] Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings, 2022.
[42] Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann Lecun. Rankme: Assessing the downstream performance of pretrained self-super representations by their rank, 2023.
[43] Aric Hagherg, Pieter Swart, and Daniel S Chult. Exploring network structure, dynamics, and function using networkx. Technical report, Los Alamos National Lab.(L. Los Alamos, NM (United States), 2008
[44] William L. Hamilton, Rex Ying, and Jure Leskovee. Inductive representation learning on large graphs, 2018
[45] Bobby He and Mete Ozay. Exploring the gap between collapsed; whitened features in self- supervised learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 8613-8634. PMLR, 17-23 Jul 2022.
[46] Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovec. Ogb-Ise: A large-scale challenge for machine learning on graphs, 2021.
[47] Weihua Hu, Matthias Fey. Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXiv:2005.0068) 2020.
[48] Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai- Kei-Wei Chang, and Yizhou Sun. Gpt-gnn: Generative pre-training of graph neural networks. In Proceedings of the 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2020.
[49] Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovec, Connor W Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik. Therapeutics data commons: Machine learning datasets and tasks for drug discovery and development. Proceedings of Neural Information Processing Systems, NeurIPS Datasets and Benchmarks, 2021.
[50] John J. Irwin and Brian K. Shoichet. ZINC - A Free Database of Commercially Available Compounds for Virtual Screening. Journal of chemical information and modeling, 45(1):177- 182, 2005.
[51] Clemens Isert, Kenneth Atz, José Jiménez-Luna, and Gisbert Schneider. Qmugs: Quantum mechanical properties of drug-like molecules, 2021.
13

---

[52] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7h, 2023.
[53] Apoorv Khandelwal, Luca Weihs, Roozbeh Mottaghi, and Aniruddha Kembhavi. Simple but effective: Clip embeddings for embodied ai, 2022.
[54] Jin-Hwa Kim, Yunji Kim, Jiyoung Lee, Kang Min Yoo, and Sang- Woo Lee. Mutual information divergence: A unified metric for multimodal generative models, 2022.
[55] Sunghwan Kim, Jie Chen, Tiejun Cheng, Asta Gindulyte, Jia He, Siqian He, Qingliang Li, Benjamin Shoemaker, Paul A Thiessen, Bo Yu, Leonid Zaslavsky, Jian Zhang, and Evan Bolton. PubChem 2023 update. Nucleic Acids Research, 51(D1):D1333-DIS
10 2022.
[56] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
[57] J. Korner and K. Marton. Comparison of two noisy channels, 1977.
[58] Mario Krenn, Florian Häse, AkshatKumar Nigam, Pascal Friederich, and Alan Aspuru- Guzik. Self-referencing embedded strings (selfies): A 100Machine Learning: Science and Technology. 144):045024, October 2020.
[59] Yugo Kubota, Daichi Haraguchi, and Seiichi Uchida. Impression-clip: Contrastive shape- impression embedding for fonts, 2024.
[60] Alexandre Lacoste, François Laviolette, and Mario Marchand. Bayesian comparison of machine learning algorithms on single and multiple datasets. In Artificial Intelligence and Statistics, pages 665-673 PMLR, 2012.
[61] Greg Landrum, Paolo Tosco, Brian Kelley, sriniker, gedeck, NadineSchneider, Riccardo Vianello, Ric, Andrew Dalke, Brian Cole, AlexanderSavelyev. Matt Swain, Samo Turk, Dan N, Alain Vaucher, Eisuke Kawashima, Maciej Wójcikowski, Daniel Probst, guillaume godin, David Cosgrove, Axel Pahl, JP, Francois Berenger, strets123, JLVarjo, Noel O'Boyle, Patrick Fuller, Jan Holst Jensen, Gianluca Sforna, and DoliathGavid. rdkit/rdkit: 2020 10__0_1___ (QI 2020) Release, March 2020.
[62] Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. An evaluation dataset for intent classification and out-of-scope prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLL) 2019., 2019.
[63] L Le. Sufficiency and approximate sufficiency. The Annals of Mathematical Statistics, pages 1419-1455, 1964.
[64] Xianming Li and Jing Li. Angle-optimized text embeddings, 2023.
[65] Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.088 032888 2023.
[66] Dennis V Lindley. On measure of the information provided by an experiment. The Annals of Muthematical Statistics, 27(4):98-1005, 1956.
[67] Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. Pre-training molecular graph representation with 3d geometry. In International Conference on Learning Representations, 2022.
[68] Tiqing Liu, Yuhmei Lin, Xin Wen, Robert N. Jorissen, and Michael K. Gilson. Bindingdb: a web-accessible database of experimentally determined protein-ligand binding affinities. Nucleic Acids Research, 15:D198-D200 12 2006.
14

---

[69] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy. Mike Lewis, Luke Zeytlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019.
[70] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christo- pher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguisties: Human Language Technologies, pages 142-150, Portland, Oregon, USA, June 201 Association for Computational Linguistics.
[71] Hadrien Mary, Emmanuel Noutahi, DomInvivo, Lu Zhu, Michel Moreau, Steven Pak, Desmond Gilmour, Shawn Whitfield, t, Valence-JonnyHHsu Honoré Hounwanou, Ishan Kumar, Saurav Maheshkar, Shuya Nakata, Kyle M. Kovary, Cas Wognum, Michael Craig, and DeepSource Bot. datamol-io/datamol: 0.12.3, January 2024.
[72] Julian McAuley and Jure Leskovec. Hidden factors and hidden topics: understanding rating dimensions with review text. Proceedings of the 7th ACM Conference on Recommender Systems, RecSys "13, page 165-172, New York, NY, USA, 2013, Association for Computing Machinery.
[73] Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. mistraleenhance Shretteddingmmmtttechhhaaee text retrieval with transfer learning. Salesforce Al Research Blog, 2024.
[74] Saif Mohammad, Felipe Bravo-Marquess Mohammad Salameh, and Svetlana Kiritchenko. Semeval-2011 task 1: Affect in tweets. In Proceedings of the 12th international workshop on semantic evaluation, pages 1-17, 2018.
[75] Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text embedding benchmark, 2023.
[76] Kevin P. Murphy. Machine learning a probabilistic perspective. MIT Press, Cambridge, Mass. [u.a.]. 2013.
[77] Jianmo Ni, Gustavo Hernández Ábrego. Noah Constant, Ji Ma, Keith B. Hall, Daniel Cer, and Yinfei Yang. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models, 2021.
[78] Zhihong Pan, Xin Zhou, and Hao Tian. Extreme generative image compression by learning text embedding from diffusion models. arXiv preprint arXi::2211.0779,, 2022.
[79] Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categoriza- tion with respect to rating scales. In Proceedings of the ACL, 2005.
[80] Raul P. Pelaez, Guillem Simeon, Raimondas Galvelis, Antonio Mirarchi, Peter Eastman, Stefan Doerr, Philipp Thölke, Thomas E. Markland, and Gianni De Fabritiis. Torchmd-net 2.0: Fast neural network potentials for molecular simulations, 2024.
[81] Christian S. Perone, Roberto Silveira, and Thomas S. Paula. Evaluation of sentence embeddings in downstream and linguistic probing tasks, 2018.
[82] Georg Pichler, Pierre Colombo, Malik Boudiaf, Günther Koliander, and Pablo Piantanida. A differential entropy estimator for training neural networks, 2022.
[83] Tiago Pimentel, Clara Meister, and Ryan Cotterell. On the usefulness of embeddings, clusters and strings for text generator evaluation, 2023.
[84] Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. Information-theored probing for linguistic structure. In Proceedings the 58th Annual Meeting of the Association for Computational Linguistics, pages 4609-4622 2020.
[85] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks, 2019.
15

---

[86] Nils Reimers and Iryna Gurevych. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2020.
[87] R. Tyrrell Rockafellar. Convex analysis. Princeton Mathematical Series. Princeton University Press, Princeton, N. J., 1970.
[88] Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in bertology: What we know about how bert works. Transactions of the Association for Computational Linguistics, 8:842-
866, 2021.
[89] Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. Advances in Neural Information Processing Systems, 33, 2020.
[90] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Scyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to- image diffusion models with deep language understanding, 2022.
[91] Joaquim Santos, Bernardo Consoli, and Renata Vieira. Word embedding evaluation in down- stream tasks and semantic analogies. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4828 4834, Marseille, France, May 2020. European Language Resources Association.
[92] Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. CARER: Contextualized affect representations for emotion recognition. In Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing, pages 3687-3697, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.
[93] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 2021.
[94] Claude Shannon. note partial ordering for communication channels. Information and Control, I 1(4):390-397, 1958.
[95] A.N. Shiri?aev and V.G. Spokoiny. Statistical Experiments and Decisions: Asymptotic Theory. Advanced series on statistical science & applied probability. World Scientific, 2000.
[96] Hannes Stärk, Dominique Beaini, Gabriele Corso, Prudencio Tossou, Christian Dallago, Stephan Günnemann, and Pietro Liò. 3d infomax improves gnns for molecular property prediction. arXiv preprint arXiv:2110.04122 2021.
[97] Ce Sui, Xiaosheng Zhao, Tao Jing, and Yi Mao. Evaluating summary statistics with mutual information for cosmological inference, 2023.
[98] Fan- Yun Sun, Jordan Hoffman, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. In International Conference on Learning Representations, 2019.
[99] Ruoxi Sun, Hanjun Dai, and Adams Wei Yu. Does gnn pretraining help molecular representa- tion? In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[100] Duyu Tang, Furu Wei, Bing Qin, Nan Yang, Ting Liu, and Ming Zhou Sentiment embed- dings with applications to sentiment analysis. IEEE Transactions on Knowledge and Data Engineering, 28(2)496-509, 2016.
[101] Jing Tang, Agnieszka Szwajda, Sushil Shakyawar, Tao Xu, Petteri Hintsanen, Krister Wen- nerberg, and Tero Aittokallio. Making sense of large-scale kinase inhibitor bioactivity data sets: A comparative and integrative analysis. Journal of Chemical Information and Modeling, 54(3):735-743, 2014.
16

---

[102]
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love. Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette- Choo, Clément Crepy, Daniel Cer, Daphne Ippolito. David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christian Muraru, Grigory Rozhdestvenskly, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baga -aptistt Lespiau, Jeff Stamway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikula, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross Mcllroy, Ruibo Liu, Ryan Mullins, Samuel L. Smith, Sehastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. Gemma: Open models based on gemini research and technology. 2024.
[103] Erik Torgersen. Comparison of Statistical Experiments. Encyclopedia of Mathematics and its Applications. Cambridge University Press, 1991.
[104] Erik Nikolai Torgersen. Comparison of experiments when the parameter space is finite. Zeitschrift fr Wahnschenntichkkiisshheeoe und Verwandte Gebiete, 16(3):219-249, 1970.
[105] Mirko Torrisi, Saeid Asadollahi, Antonio De la Vega de Leon, Kai Wang, and Wilbert Copeland. Do chemical language models provide a better compound representation? In NeurIPS 2023 Workshop on New Frontiers of Al for Drug Discovery and Development, 2023.
[106] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao. Vedanuj Goswami. Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Maric-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mi- haylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Ilama 2: Open foundation and fine-tuned chat models, 2023.
[107] Anton Tsitsulin, Marina Munkhoeva, and Bryan Perozzi. Unsupervised embedding quality evaluation, 2023.
[108] Alexandre B Tsybakov. Introduction to Nonparametric Estimation. Springer series in statistics. Springer, Dordrecht, 2009.
[109] Cynthia Van Hee, Els Lefever, and Véronique Hoste. Semeval-2018 task 3: Irony detection in english tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 39-50, 2018
[110] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Represen- tations, 2018
[111] Luke Vilnis and Andrew McCallum. Word representations via gaussian embedding. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
17

---

[112] Hongwei Wang, Weijiang Li, Xiaomeng Jin, Kyunghyun Cho, Heng Ji, Jiawei Han, and Martin D. Burke. Shemical-reaction-aweees aware molecule representation learning, International Conference on Learning Representations, 2022.
[113] Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-superviie contrastive pre- pre-training. arXiv preprint arXiv:2212.082:
2022.
[114] David Weininger. SMILES, chemical language and information system. 1. Introduction to methodology and encoding rules. Journal of Chemical Information and Computer Sciences, 28(1):31-36, February 1988. Publisher: American Chemical Society.
[115] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2019.
[116] Minghao Xu, Hang Wang, Bingbing Ni, Hongyu Guo, and Jian Tang. supervuue graph- level representation learning with local and global structure. arXiv preprint vrXiv:2106.044 13,, 2021.
[117] Nianzu Yang. Kaipeng Zeng, Qitian Wu, Xiaosong Jia, and Junchi Yan. Learning substructure invariance for out-of-distribution molecular representations. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[118] Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. PAWS-X: Cross-lingual Adver- sarial Dataset for Paraphrase Identification. In Proc. of EMNLP, 2019.
[119] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 5812-5823, Curran Associates, Inc., 2020.
[120] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metries for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
[121] Sheheryar Zaidi, Michael Schaarsehmidt, James Martens, Hyunjik Kim, Yee Whye Teh, Alvaro Sanzhez-Gonzalezz, Peter Battaglia, Razvan Pascanu, and Jonathan Godwin. Pre- training via denoising for molecular property prediction. In International Conference on Learning Representations, 2023.
[122] Li Zhang, Steven Wilson, and Rada Mihalcea. Multi-label transfer learning for multi-relational semantic similarity. In Proceedings of the Eighth Joint Conference on Lexical and Computa- tional Semanties SEMM 2019). Association for Computational Linguistics, 2019.
[123] Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NIPS, 2015.
18

---

Appendix
Table of Contents
A Background and Notation
20
B Proofs Theoretical Results
20
B.1
Proof Proposition!!
21
Comments about capacily to distinguish concepts
Sruicie.........................
21
B.3
Proof Proposition 2 and Comolaar
22
B.4
Example of comparisons of statistical experients
23
B.5
Sufficiency and Inference Procedures with Embedding Models [ 63
24
B.6
Deficiency and Expected Risk [63
24
NLP Experiment Details
24
C.I
Models and Datasets statistics
24
Downstream tasks training details.
24
NLP Comprehensive Results
24
C33.1
Full MTEB Benchmark Results
24
C.3.2
Ablation studies)
26
C,3.3
Impact of training steps
27
C.3.4
Importance of embedding size normalization
27
C33.5
Community and cluster performance
29
C.3.6
Evaluating information sufficiency on different dalasets
D
Molecular Experiment Details
31
D.
Embedders considered
31
D.2
Details the information sufficiency estimation
32
D.3
Complementary results on ADMET tasks
33
D.4
Drug target Interaction prediction
35
E
Information Sufficiency Estimation
36
E.1
Estimation method]
................................ ................................
36
E.I.1
KNIFE Estimator
................................
36
E.1.2
Embedding dimension normalization.
37
E.1.3
Median instead of mean.
38
E.2
Hyperparameter selection
39
Impact of the task size
39
Impact of the number of models
41
Comparison with other metrics
42
E.6
Computational ressources
43
19

---

A Background and Notation
We consider all alphabets be standard Borel (8 (i.e., isomorphic to a Borel subspace of a Polish space), encompassing virtually all practical scenarios. Each such space is equipped with its Borel o-algebra B( The set of all probability measures on is denoted by P(Y). The total variation distance between P and Q in P(Y) is defined as
|P
QV
sup AEB(Y)
[P(A) - Q(A)),
(2)
and the Kullback Leibler divergence is defined by
(L((1)Q) - -1 /
dP log dP dd dQ to0
if
(3)
otherwise.
Given joint probability measure Pxy in P(XxY) induced by two random variables X E X and Y E Y with product measures Px Py, the Mutual Information is defined as - Px If Px E P(X) is probability measure induced by X E X. the Differential Entropy is defined by
((X)= -- [f log
dPx dPx,
(4)
du
where jl denotes the Lebesgue measure. Similarly, it is possible to define the conditional entropy of Y given X which is denoted by XX) The mutual information satisfies the identities [[X;
h(Y) /(Y = h(X) hx (Y) and [X) & h(Y) (see [27] for further details).
A Markov (or transition probability) kernel between X and Y is mapping T B(Y) X - [0, 1]. satisfying T(| P(YY) for all E X and T(B)-) being a measurable function on for any
B E B(Y). The space of all such T' is denoted by K(Y[X). In cases where both Y and X are
finite, any XC(Y(X)
represented as stochastic matrix with elements T(y[z), (x, y) E x Y.
Every 7 E XC(YX) induces a mapping P(X) - P(Y), denoted by T. mapping any P E P(X) to
Q = ToP E P(Y). where
Q(B) (ToP)(B) S T(Bl))P(dz), WB E EY)
(5)
We denote the composition of Markov kernels by juxtaposition: for M E KC(ZY) and T E KC(Y)X). their composition MoT T(([X) is defined by
/ M(Z2y)T(dyya))
(Z &
Vz E X, Z E B(Z).
(6)
We define the average of the total variation distance between two Markov kernels T,T" KC((X) as follows:
E 1| A EIIT(-X)) - ('(X))]
statistical model a triple My (U,) (Pojy()l)) y E Y)). where (U)) is a sample space: Y concept space, and Pujy B(U) x Y [0, 1] is Markov kernel (or transition probability).
B Proofs Theoretical Results
Proposition (Relationships of sufficiency and information). The following relationships hold:
(i) Sufficiency + informativeness. If the embedding model Puçx is sufficient for the embedding model Pvix. i.e. U >s V, then U >1 V. However. Informativeness क> sufficiency.
(ii) Informativeness higher capacity to distinguish concepts. the embedding model Puix is more informative than embedding model Pz|x, i.e. U FI V, then
KL. (Popy Popy (-ln)) > (lvo))lllvl)))
for any pair of concepts (yo, V1) E Y x Y and all probability distributions Pyx-
20

---

B.1 Proof Proposition 1
Proof. It is immediate to check that the data-proud pata--soosssing inequality and the Markov chain Y ++ X +> U' + V implies the relation in claim (i). On the other hand, the non-equivalence is proved by means of an explicit counterexample [57]. Given any < P < 1/2 with p = 1 - p. For some e, 6 > consider two discrete embedding models defined by the following matrices:
(1 1 1-6 ) PUIx with PUix (ptop"8)
1 +
Pvix = 1/2
p-8
(8)
By taking 6, > 0 small enough, both Poix and Pvix are stochastic matrices. It follows that Pvix not a degraded version of Puix but provided that e, å are sufficient small, the embedding model Peix is more informative than Pvix, which proves the claim.
In order show (ii), let Pejy and Pviy be the corresponding probability measures induced by Pxju (-lu) via the embedding models:
(Ul)( = / Poxx(Ula)Pxxy((rtl))) VyE U B(U),
(9)
and
Purr = / Pvix (V)) Vy YEY, VE
(10)
for any y E Y (un, )- For a 0 & X & let (+1r)) E P(X) and Pxir E P(X) be two arbitrary probability measures on X. Let Px (Xly) be defined by
Px (X = 1ly = yo] (8o) + 1lv 2
By replacing it into equations (9) and 10
obtain
PUiY (Ulg) 1 1y 3ol Pujy (1))) + 1 -: (Uly1)
(11)
Pvir (Vly) 1 yol Pviy (V voo) 1 Pryy
(12)
and let (v6) = A and (y1) -1 A. The above probability measures correspond to quadruple of random variables: (Y XA, U2, Vx) P(Y x xx V). Consider the function f()) defined by
f(A) = I(A; - V3)
It is not difficult to check that f(X) > O for all 0 < À < 1, and f(0) 0 which requires that ('()) O. By taking the differentiation, we obtain
('))) KL (-180) (+lu1) (1
0,
which implies the claim (ii). This concludes the proof.
B.2 Comments about capacity to distinguish concepts
Notice that the KL divergence between the induced distributions of the resulting embedding is not less for the embedding model U than Z. Indeed, consider the case of binary classification Y (yo, y1 } with uniformly distributed concepts. Pinsker's inequality [108] together with claim (iii) imply
KL (Prly (-180))l Ptriy (+181) > P| - Pvix (-lx))
From which, it is easy verify that the accuracy of the expected Bayes accuracy of the optimal classifier based on V is upper bounded by [188] Lemma 2.1]
sup Pr(w(V) = < exp V
where the exponent in the upper bound is subject to the discriminating capacity through the KI. divergence of the embedding model U on U.
21

---

B.3 Proof of Proposition and Corollary
We begin with the proof of Proposition 2
Proof. Clearly, the assumption (i) implies the statement (ii) by Pata-Processing. Conversely, let us assume point (ii) holds. This means that, for every probbaility distribution Px and all conditional probability distributions Py|x, there exists PU U P(Y) such that
Σ Pvx(U.2) x) 1. (ollul))exx((ulla) (v.x)xYxX
J (
sup
Σ Prx(u,.)
x(d()))
(13)
(y.))YXXX
where PV V + P(Y) is (possibly randomized) inference procedure is transition probabilities, which the learner can optimize to maximize the guessing probability.
Let the decision rule (vlv) = 1/ E Ay] for any partition (Ay)vet of V with A, E BV(). Then, for any PY|x, expression 13 implies the existence there exists po (y/u) such that
Uv Pv (dul=)) A,) - po (o()))))
Σ Prx(u,x)
(14)
(v.))EYEX
[/,, Prix(dolx)) - / po (c(vl)))
Σ P)x(U,
6.
(15)
(vx=)EYxX
However, we can rewrite the last expression as:
/ Pvix(dulz)) - J. po (o < 0.
sup inf
Σ PV)(v,x) (y.
(16)
Prix
By applying the minimax theorem is possible to exchange the order of the inf and the sup, which yields:
inf sup E sup Σ ((. < {A,}
(17)
Prix yEY
V/1 Puix(dula)) - 1 pu (ull))
r T(v,)).P))
(18)
We observe that
Σ (((,,))) =0,
(19)
WEY
for each X and thus,
max I' ((x, x), > 0,
(20)
yEY
where the equality holds if and only if T ((v, pG) 0 for all y Y, for each X, since by contradiction otherwise
Σ (((,,)))pu) < 0.
(21)
WEY
Therefore,
(1A, Prix(do)x) -000-1100 / Poj
inf sup E sup E Prx((uxx) (A,)
Prix DEY
= inf sup E max I' ((v, (A,)
(22)
JEYY
22

---

which means the maximum is achieved by degenerate random variables Y f(X) achieving the maximum for each X. Consequently, we have that
inf El - -PooPUpxlIy = inf sup E max {Ay
pu) 0,
(23)
LOEY
hence inf InJPPrjj Pu Pujx 11 0, and so the existence of the transition probability pu such PU that P x is sufficient for Pvix. This concludes the proof of the Proposition 2
We now show the proof of Corollary
1
Proof. Continuing from the proof of Proposition 2 which remains unchanged, until one demonstrates that for E > O,
inf sup E PG (A,) LEEY
max ((y, E
(24)
To proceed from this point, let us now examine the following quantity:
Σ
(uy,
VEY
The above quantity is the induced f--norm distance fu Po Pu)x Since, for all x E X.
between SA, P and
Σ
((((,,)),U) = 0, DEY
we have that
[((( =2
Σ
for all X
HEY
which implies that, for the strategy pu achieving the left-hand side of Eq. 24
sup E
Σ
[I,,,
< 2| sup {A,}
max I' ((v, X), yEY
< 2 Y forall x E X
{Ay} VEY
Hence,
Pvix(du|X) J. pa (o((o()))))x(((x <
sup Σ (Ag) DEY A,
This concludes the proof the Corollary
B.4 Example of comparisons of statistical experiments
Example (Statistical experiments with Gaussian embedding models). Let U| and Vlæ be inde- pendently normally distributed as N(x, d²) and (2 Nxx,2'' e²o²), respectively, with
Case of o² oā known. Here U 7s V since V + w|æ has the same distribution U |x That is strictly more informative than U. However, U is
when v ~ strictly more informative than V.
Case of known. One can observe that V since the variables Vje have the same distribution as the U, and the variables eU have the same distribution as the V.
Case of and unknown. Surprisingly, in this case U and V are not comparable.
23

---

B.5 Sufficiency and Inference Procedures with Embedding Models [3 63
Proposition 3 (Sufficiency and risks of given task on embedding models 13
An embedding model Puty E K(UUY) is deemed to be sufficient for another one Pviy E K(VY) if and only if. for
any bounded loss function where ||6] < 1, and for any inference procedure pv + Y, there exists a inference procedure (possibly randomized) pu
P(Y) such that the resulting statistical
risks satisfy
(P PU. & (V, E)2 for all y E Y. Here we denote by Ry PULY PY, ,Y, .)) and (Pyyy pV, ,) the statistical risks for the corresponding inference frameworks, respectively.
(25)
Remark 5. The restriction |e < 1 irrelevant here. However, we opt for simplicity and limit our focus to situations where encounters dominated statistical models with Polish sample spaces. In essence, various extensions do not significantly alter the conceptual aspects of the underlying statistical problem (see [033] for further details). Rather, they primarily reflect the complexity of its measure-thcoretie formulation.
B.6 Deficiency and Expected Risk 63
In 1964, Le Cam [63 clarified the relationship between the sufficiency of an embedding model on given task and its expected risk this task. The following theorem provides formal statement of this relationship.
(63] Let > 0 be fixed. Then, 8(Puly - Pvy)) if and only if. for any bounded loss Sanction É where [1E] & 1, and for any inference procedure pv using the embedding model Pviy, there exists a inference procedure (possibly randomized) py bused embedding model Puly such that the risks satisfy R.y(Pur PoPPuuyPU, pU, E & R PV, for all y E Y.
Theorem (Le Cam
C NLP Experiment Details
In this section, we provide all the necessary experimental details to reproduce the experiments in NL.P For the Is score estimation, please see Appendix E. First, we detail the models and datasets used in the experiments. We provide the training details of the downstream tasks, and finally, we present the comprehensive results of the NLP experiments.
C.1 Models and Datasets statistics
In Tab. I we provide the metadata of the models used the NLP experiments and their scores on the MTEB benchmark when they exist. We provide in Tab. 2 the statistics of the datasets used to evaluate the Is score.
C.2 Downstream tasks training details.
All the downstream tasks trained in the exact way. We use dense classifier with two hidden layers of dimension 256 and train for two epochs using ADAM [56] with learning rate of 10 33 on the official training set and evaluated on either the validation test set when they are available (with respect to the Huggingface datasets). We do not perform early stopping or selection using the validation set.
C.3 NLP Comprehensive Results
We provide in this section the unaggregated results for the main NLP experiments presented in the main text, then we provide numerous ablation studies and additional results to address different aspects of the Is score of the embeddings NLP.
C.3.1 Full MTEB Benchmark Results
The strength of the MTEB benchmark is that it evaluates embedders on a very large and diverse set of downstream tasks. We provide an Tab. 4 and Figure 6 the full results of the MTEB benchmark (English) for the models used in the NLP experiments.
24

---

Table 1: Metadata of the evaluated models and their information sufficiency.
Dim.
Max Tokens
Is
Model
SFR-Em FERR-Embeddinng-Mee
4096
32768
0.59
4096
32768
0.58
stella-base-en-V
768
512
0.57
e3-large-v2
1024
512
0.57
GritLM-7B
4096
32768
0.56
ember-vl
1024
512
0.56
sie-large
1024
512
0.56
UAE-Large- V1
1024
512
0.55
gte-base
768
512
0.55
S1 model c5
1024
512
0.55
GIST-Embedding-vvl
768
512
0.55
gir-t5-large
768
512
0.55
gtr-(3-xl
768
512
0.55
Deee-aaseenn-vle
768
512
0.54
sentence-t5-larg
768
512
0.54
gemma- /b-1t
N/A
N/A
0.54
gle-tiny
384
512
0.53
gtr-t5-base
768
512
0.53
Llama-2-7bhhi
N/A
N/A
0.53
Sentence-t3-N
768
512
0.53
gemma-2b-1t
N/A
N/A
0.52
e5-small
384
512
0.52
bge-micro-v2
384
512
0.51
all-distilroberta-vl
N/A
N/A
0.51
multilngual-e5-smal
384
512
0.51
nssnaco-berr-o--codeene
768
512
0.51
sup-simose-bowe bert base-uncased
768
512
0.50
all-MiniLM-LL-v22
384
512
0.50
all all-mpnet-base-v2
768
514
0.49
udever-bloom-S6000
1024
2048
0.49
LaBSE
768
512
0.47
average word embeddings komninos
300
N/A
0.42
average word embeddings 300ve.6B..00d
300
N/A
0.41
allenan-specte
768
512
0.38
Classification
Clustering
Retrieval
STS
Reranking
Classification
frank
0.88
0:20.77 0.85
9.88 89
0.84 0:82 0.62
0.78 077
0.85
0.00
X BIY
0.84
0:0.5
4)
Figure 6: Correlations between rankings on different subtasks and their Is score ranking.
25

---

Table 2: Statistics of the datasets used as umbrella datasets for Zg informativeness evaluation.
Size
Dataset
Split
test 7600
ag_ news
train
120000
amazon polarity
test
100000
banking77
test
3080
biosses-sts
test
182
test
2000
paws-x;e
train
49401
validation
2000
test
1066
rotten_tomatoos
train
8530
validation
1066
sickr-sts
test
6077
test 13132
snli
validation
13134
test
1821
sst2
train
67349
validation
872
sts 1tsstssts
test
4946
sts 3-sts
test
2638
14-sts
test
6351
sts
test
5170
test
2552
stsbenchmark-sts
validation
2910
test
50000
tweet_eval:emoji
train
45000
validation
5000
test
1421
tweet_evast srcce__evllemmttonn
train
3257
validation
374
test
12284
tweet_eval tweet_eval:sentiment
train
45615
validation
2000
wiki-paragraphs
validation
100000
We obtain significant positive correlations in all categories of downstream tasks. We noticed that we obtained significantly poorer results on STS, Clustering, and Reranking tasks than on classification tasks. We believe this behavior due to the nature these tasks. Indeed, they do not rely on training an additional model on top of the embeddings but rather directly use the embeddings as is in dot products or similarity measures. An embedder could produce very informative embeddings, i.e., is possible to extract the usefuul information using small model, and at the same time. these embeddings not be adequate for dot- cou-product-baae based similarity measures. We believe further investigation needed to understand the behavior of the models these tasks. Especially see if training small model on top of the embeddings can improve the performance of these tasks.
C.3.2 Ablation studies
Many factors impact the estimation of the Ig score of the models, such the dimensions of the different embeddings and the number of available embedders to evaluate. the Iş score capture many different aspects of the embeddings, such as the quality of the embeddings. We provide this section comprehensive set of ablation studies to evaluate the impact of these different factors on the Is score of the embeddings.
26

---

Table 3: Summary of the evaluated embedders with their performance on the MTEB benchmark.
Average Classification
Clustering
Reranking
Retrieval
STS
Model
Eir---emhhnd-iisiiee -istral
67.56
78.33
51.67
59.00
85.05
ocho-mistral- ceesstral--bmmstruucee rb-mmstruct-lasttoke...
77.43
46.32
58.14
55.52
82.56
Millla-base-nn-V
62.61
75.28
44.90
58.78
50.10
83.02
-2
62.25
75.24
44.49
$6.61
50.56
82.05
M-7B
66.76
79,46
50.61
60.49
57.41
83.35
ember-v
63.54
75.99
45.58
60.04
$1.92
83.34
2 Ele-large
63.13
73.33
46.84
59.13
52.22
83.35
UAFF-Large-V
64.64
75.58
46.73
59.88
54.66
84.54
Meebbaee
62.39
73.01
46.20
58.61
51.14
82.30
modcl
63.34
73.96
46.61
$9.86
51.80
83.85
GIS (15S--Embeidding-00
63.71
76.03
46.21
59.37
52.31
83.51
gtr-t3-large
58.28
67.14
41.60
55.36
47.42
78.19
20-133X1
58.42
67.11
41.51
55.96
47.96
77.80
bge-buxe-ene 1.5
63.55
75.53
45.77
58.86
53.25
82.40
sentence-13 large
72.31
41.65
54.00
36.71
81.83
gemma
N/A
N/A
NIA
N/A
N/A
N/A
58.69 69
70.35
42.09
55.77
44,92
80.46
gti-t3-base
56.19
65.25
38.63
54.23
44.67
77.07
Llama 2- 7b-hf
N/A
N/A
N/A
N/A
N/A
N/A
scntence-15-
57.87
72,84
42.34
54.71
38.47
81.66
gemmma--2--
NIA
N/A
NIA
N/A
N/A
N/A
e3-small
71.67
39,51
54.45
46.01
80.87
byc-micro-v2
56.57
68.04
39.18
42.56
78.65
all-Usttrrobrtaa-y1
N/A
N/A
NIA
NIA
N/A
N/A
multilinuual
57.87
70.74
37,08
53.87
46.64
79.10
mnsmarco--hrtee hettecooo-------oooo-o-- nomtceeeon
$2.35
64.71
37.64
$1.84
32.96
76.47
sup-sincsc-bbet--aae saa--bbac---aceee
48.87
67.32
33.43
47.54
21.82
79.12
all-Minil.M-LbbV/
56.26
63.05
42.35
58.04
41.95
78.90
all-mpnct- 11-mpnct-busc--22
57.78
65.07
43.69
59,36
43.81
80.28
udever-bloom-300
55.81
68.04
36.89
$2.60
41.19
79.93
LaBSE
45.21
62.71
29.55
48.42
18.99
70.80
average word entueddmeddd.s..... komninos
42.06
57.65
26.57
44.75
21.22
62.46
average word wwweeddneennnnnn....eeeee hlw.slllovvvv.oo.e 300d
41.96
57.29
27.73
43.29
21.62
61.85
allena--pectte
40.28
52,37
34.06
48.10
15.88
61.02
Impact of instruction finetuning. Instruction finetuning is now common practice to improve the alignment of the base of models and expand the models' reasoning capabilities. In Figiuree 4c show that instruction fine-tuning positively impacts the models' performance on the downstream tasks and that the Is score captures this improvement. In addition to studying the impact of instruction finetuning, we evaluated models at different checkpoints during their initial pretraining in Sec. C.3.3 using the CroissantLLM checkpoints 37
C.3.3 Impact of training steps
Surprisingly, we found that the number of training steps does not significantly impact the models' performance the downstream tasks
the Ig score of the embeddings. The Is score correctly captures this behavior as shown in Figure 7 We hypothesize that the Is score in terms of embeddings is, in this case, determined by a few numbers of training steps (the first 5000) and the overall architecture of the model. Training the model further even leads to decrease in performance on the downstream tasks, which is not captured by the Is score of the embeddings; this could be due the very small variation in the performance.
C.3.4 Importance of embedding size normalization
We found that considering the amount information packed by an embedding per coordinate is crucial obtain good ranking of the models. In Figure 16b show the correlation between the performance of the models on the MTEB benchmark and their Is score, not normalized by embedding size. While positive significative correlation is still present, the correlation is much weaker than when the dimension of the embeddings normalizes the information sufficiency.
27

---

Qp
Task category
Average
0.90
0.83
0.68
CutrroCCoonterfaauuulCCsiiiiiie (en)
0.75
0.67
0.50
amizoonPllartyCClsssfiittttoooot
0.78
0.77
0.58
MmazoonneiiessClasfficee (en)
0.83
0.82
0.65
Bunking77Classiication
0.92
0.84
0.65
EmotionClaasifiiattinn
0.87
0.76
0.58
Classification
ImdDClassification
0.77
0.76
0.57
lieiiveeeneettlssffccccaa (en)
0.93
0.84
0.68
MassiveSrenarioClassffee (en)
0.92
0.84
0.67
MTOOPDomainCIaasifccaoooo (en)
0.94
0.89
0.72
MTOOIntennClaasificen (en)
0.80
0.76
0.59
ToniiCoovveraiiinn@asssiioaooo
0.68
0.51
0.36
0.70
0.50
0.35
Average
0.88
0.82
0.63
ArxivClusteringg22
0.59
0.64
0.46
ArxivClusteringS25
0.66
0.73
0.54
BiorxivChusteringg2P
0.43
0.39
0.30
BiorsisChusteringS25
0.58
0.64
0.42
MedrxivClusterrnggP2P
0.41
0.34
0.26
Clustering
Medrxiv
ClustriinCssss
0.58
0.48
0.34
RedditClustering
0.92
0.84
0.67
Reddii([lusterngg221
0.86
0.88
0.72
StackExchangeeluseee
0.93
0.88
0.71
0.66
0.63
0.46
WwentyyeessrroossCluueeinnngg
0.91
0.86
0.68
Average
0.91
0,83
0.67
SprinnDuplicaueeuussttn
0.69
0.44
PairClassification
TwiiterSemEval2015
0.92
0.76
0.57
TwitterU
URttrrru .Corpus
0.85
0.81
0.64
Average
0.85
0.74
0.58
ssUUbuntuDupQuusstnt
0.85
0.65
0.53
Reranking
MinuSmmalleernkingg
0.86
0.84
0.65
SeiDoesRR
0.63
0.50
0.35
SteckOverflow wwwerfoooowDuuuustioooo
0.89
0.58
Average
0.89
0.87
0.67
ArguAna
0.80
0.77
0.59
ClimateFEVER
0.75
0.78
0.59
CQADupstackRerrieval
0.85
0.74
0.59
DBPedia
0.94
0.90
0.74
FEVER
0.87
0.86
0.67
FIQA2018
0.85
0.57
HotpotQA
0.88
0.87
0.69
Retrieval
MSMARCO
0.84
0.78
0.60
NFCorpus
0.91
0.86
0.68
NQ
0.88
0.78
0.61
QuoraRetrieval
0.90
0.80
0.63
SCIDOCS
0.79
0.64
0.49
SciFaet
0.79
0.78
0.57
Touche2020
0.71
0.58
0.41
TRECCOVID
0.73
0.65
0.51
Average
0.90
0.75
0.57
STSBenchmark
0.85
0.70
0.53
BIOSSES
0.80
0.72
0.52
SICK-R
0.83
0.64
0.46
STS12
0.80
0.63
0.44
STS
STS13
0.86
0.66
0.46
STS 14
0.89
0.68
0.47
STSIS
0.91
0.79
0.62
STS16
0.91
0.76
0.59
STS17 (en-en)
0.81
0.45
0.32
STS22 (en)
0.84 0.61
0.45
Summarization
SummFFal
0.46
0.31
0.21
Table 4: Detailed correlations between the Iş score of the models and their performance on the
MTEB benchmark.
28

---

1.0
0.510
Classification
Accuracy
0.505
0.8
Is
0.500
0.6
Informativeness
0.495
base 5k
base 50k
base_ base_100 100k
base base_150k 150k
CroissantLLMBase
CroissantCool
Figure Impact of the number of training steps on the performance of the models on the downstream tasks and their Is score.
inforatmation sufficiency
C.3.5 Community and cluster performance
gemma-7b-it
CroissantLLMBase
gemma-2b-it
CroissantCool
Llama-2-7b-hf
GritLM-78
udever-bloom 560m
average word
e5-large-v2
gemma: Beter gemma-2b
UAE-Large-V1
bge-base-en--11.5
echo-mistra
ember-v1
sf_model_es
udaver-bloom-560m
all-mpnet-base-22
LaBSE
e5-large-v2
sentence-ts-ud
EFR-Embedding serr-t5-base
multilingual-es
e5-small
gtr-t5-xl
model
LaBSE
allenai-specter
UAE-Large-VI
gtr-t5:larga lara
average
Average
Classification
Clustering
Retrieval
STS
deranking
Classification
stella-base-en-v2
e5 e5-sllll
(6)
(a)
Figure 8: We present different interesting properties of Is. In Figure 4a we show that it can be used to cluster models Figure 8hj reports the performance of the models on the different task categories. They are grouped by similar behaviors on these tasks (dendrograms) and colored by the communities discovered in the information sufficiency graph. (Only models evaluated as part of the MTEB benchmark are shown).
29

---

rotten_ tomatoes
paws-x/en
0.5
0.0
dair-ai/emotion
0.0
part--aae-uncsee
Mistral
gtr-t5-base
gtr-t5-large
gtr-t5-xl
sentence- t5-large
sentence-t5-xx
alenai-specter
average word embeddings glove. 28. 68.30dd average ____word_eeee
embedddnng komninos
all-MiniLM-L6-
UAE-Large-VV
S modeee e5
ember-v
gte-large
e5-smal!
multilingual-e--5mmae -5-llll
e5-large-v2
GritLM-7B
Llama-2-7b-ht 2-7bhf
gemma-2b-it
gemma-7b- ab-it
Soovver-bloom-5600
LaBSE
bge-base-en-V11.
UST-Embedding-vo
stella-ba base--en--n2
gte-base
age-micro-v2
gte-tiny
womppntt-baser
SFR-Embedding-
all-distirrbberte
echo-mistral- udever-
msmarco-bertee
sup-simcse-berr
Figure 9: Performance the models on the downstream tasks grouped by clusters discovered by the directed Is.
We postulate that models clustered together by information sufficiency are likely to behave similarly on the downstream tasks. We evaluate this hypothesis by grouping the models by clusters discovered using the information sufficiency and reporting their performance the downstream tasks. In Figure 85 Figure we observe that models within the same cluster tend to have similar behaviors on the downstream tasks.
C.3.6 Evaluating information sufficiency on different datasets
The Is score is evaluated with fixed dataset supposed to represent the data distribution of interest (either very diverse set or a subset following a distribution specific to subfield like medical or legal texts). We coos-eevllated the Is the models on different datasets and the performance of the models on the downstream tasks in Figure 10] We find that closer datasets in terms of the data distribution lead to higher correlation between the Iş score of the models and their performance on the downstream tasks. It is especially highlighted when comparing the correlations we get when evaluating Is on the AG News and Amazon polarity datasets. The first corresponds news articles, and the task to guess the topic, whereas Amazon Polarity corresponds to product reviews, which a sentiment analysis task. We find that the I.s score evaluated on Amazon Polarity tends to yield way better correlation with the performance the sentiment analysis downstream tasks
30

---

eg_news Common
mteb/amazon_ polarity
Pearson
Spearman
Kendall-Tau
Kendall-T Kendall-Tau
--Tau
Pearson
Spearaal
0.4
0.2
8q je_eews
dalr-a/eemoson
Imob
paws-xenn
rotten tomatoes
evalbemoji
tweet jimotion
tweet saaisentmment
clinc oos:puus
eq se_hews
dalr-a palr-a//emozzon
Imdb
paws-xien
rotten, tomatoes
tweet, evalremojj
tweet jemotioo
tweet evaissentmemnt
clinc nos:plus
ag, he_hews
sairal/emozion
Imdb
paws-xien
retten, tomatoes
tweet, ovelyemoji
tweet, jomotion
tweet evaisenntmmnn
clinc nosiplus
Task
Task
Task
Figure 10: Correlation between Zg scores computed on different datasets and the cross-performance on different tasks.
such evaeeeseaa//nnnmnnn, tweet_evalln ercet_eval/eootion, IMDB Rotten Tomatoes lesser extent dair/emotion. Interestingly, the difference is less significant on the wweet_eval/emoji subtask.
D
Molecular Experiment Details
D.1
Embedders considered
Model
SMILES
2D-GNN
3D-GNN
Architecture
Out
Dataset (size)
Not-trained
300
AttributeMask 99
GIN
300
GEOM
(50k)
ContextPred
99
GIN
300
GEOM
(50k)
GPT-GNN 48
GIN
300
GEOM
(50k)
InfoGraph
300
GEOM
(50k)
GraphCC1 119
GIN
300
GEOM
(50k)
GR
GIN
300
GEOM
(50k)
GruphLog [16
300
GEOM
(50K)
GraphMVP(67
GIN
300
GEOM
(5KK)
3D-infomax 96
PNA
800
QMugs
(620k)
11
GCN, GAT, TAG
USPTO
(1.5M)
MoleOOD
GIN SAGE, GCN
256
BACE
(400k)
ChemBHRRT MI.M
RoBERTa
600
PubChem 55
(SM, 10M, 77M)
ChemBERT MTR
RoBERTa
384
PubChem 55
(5M.
77M,
MolBert
BERT
GuacaMol
(1.6M)
ChemGiPT 40
GPT
128, 256, 2048
PuhChem
55
(10M)
3D denosing 121
TorchMD-no net
256
PCQM4Mv 46 (3.7M)
3D-fractional
39
TorchMD-net
256
PCQM4M 2 (3.7M)
Table 5: Models evaluated on the ZINC dataset.
Some models were used in different versions (architectures, number of parameters, pretraining dataset's size), such as the ChemBert models, followed by the size their datasets, or ChemGPT, followed by their number of parameters.
considered 28 models for the molecular experiments, summed up in Tab. 3
Most 2D-GNNs were trained on the GEOM 6 dataset and were gathered from repository of GraphMVP 67 model. Note that the MoleOOD 117 model was trained on the BACE [47 dataset, with a supervised task specific to the 3-secretase enzyme. result, this model can seen as "already specialized", explaining its poor performance in evaluation.
31

---

BertMLM-77M
GPT-1.28
ContextPred
ContextPred
ContextPred
BertMLM-5M
GPT-4.7M
Matk
GPT-GNN
GPT-19M
GPT-GNN
BertMLM-77H
GPT-4.7M
BertMLM-10M
MoleOOD GIN
MoleOOD
GPT-GNN
MoleOOD SAGE
AttributeMask
B5
BM5tMLM:SM
MoleOOD SAGE
MolBert
MaleOOD GCN
BerMMLM-10M
BertMLM-77M
GPT-4.7M
MolBert
BertMTR-77M
Not-trained
MoleOOD
BertMTR-10M
InfoGraph
GPT-19M
GROVER
MoleOOD GIN
BertMTR-5M
Models Z
BertMLM-10M LM-10M
GROVER
MolBert
InfoGraph
MoleOOD_SAte
GPT-19M
Not-trained
Not-trained
GraphCL
GraphMVP
InfoGraph
GraphMVP
GraphCL
MoleOOD GCN
Lraph
3D-Infomax
MolR, tag
3D-denoising
GraphLog
MolR gat
FRAD QM9
GPT-1.28
MolR gcn
3D-Infomax
BertMTR-5M
GROVER
GPT-1.2B
BertMTR-10M
3D-Infomax
MolR gcn
MolR gat
BertMTR-77M
MttributeMask
BertMTR-5M
MoiR_gen
GraphCL
BertMTR-10M
MolR gat
GraphLog
BertMTR-777
MolR tag
GraphMVP
MolR
Ability to predict denoising 3D models
Ability to be predicted by denoising 3D models
0
1
7:12 15
T5(U +
(b)
Figure 11: (a) Information sufficiency of embedders over 3D-Denoising models (left) and of 3D- Denoising models over the other embedders (right). (b) Information sufficiency of embedders in both directions. We see the 3D denoising models are among the least predicted models.
We used the RD-Kit and Datamol tool-kits 61 7 to pre- pre-process the molecules and to generate three-dimensional conformers for 3D-models. To the models using 31) views of the molecules, we generated five conformers (possible 3D configuration of the molecule) for each SMILE MIILES and kept the conformer with the lowest energy. Note that this methodology imperfect, the 3D coordinates might be noisy; however, we followed the same procedure to pre-process the ZINC dataset, to evaluate the information sufficiency, and the datasets corresponding each downstream task.
Finally we considered a variety of models architecture for 2D-GNNs notably graph isomorphism network (GIN) [115] principal neighbor aggregation networks (PNA) (26), graph convolutional
network (GCN)
33 graph attention network (GAT) [110], topology adaptive graph convolutional
networks (TAG) and GraphSAGE 44 For SMILES-base models, backbones inspired by
BERT BIJ RRBE
[69]. and GPT
Finally, both our 3D models use TorchMD [80] as
backbone.
D.2 Details on the information sufficiency estimation
3D models. The two 3D models considered (FRAD 33 and Denoising (21]) obtain high Is scores while being among the least predictable (Figure 1Tb TTbland and Figure 12p. This suggests that these models capture 3D-specific features inaccessible from other modalities while maintaining sufficient overlap to predict them.
2D-3D models Some 2D-GNNs we considered (GraphMVP and 3D-infomax) are trained to maximize the mutual information between their embeddings and 3D representations of the molecule. Hence, we expect these models to be related to the 3D-denoising models we considered. However, we observe in Figure 1lalthat these models do not achieve particularly high information sufficiency scores over 3D-denoising models. On the other hand, the 3D models achieve high information sufficiency scores over them, which might suggest that these 2D models and 3D-denoising models share information that is easier to access from the 3D models. However, we want to point out that GraphMVP and 3D-infomax are both among the most predicted models; that is to say, among the other models in our pool, they achieve the highest information sufficiency scores.
32

---

GPT-1.2B
GPT-19M
GPT-4.7M
BertMLM-5M
BertMLM-77M
BertMLM-10M
1.2
MMoBBert
Not-trained
InfoGraph
GROVER
11.0
GraphLog
GraphCL
GraphMVP
Models Z
(0+))))
Attribute Mask
00.8
MoleOOD GIN
MoleOOD GCN
MaleOOD SAGE
3D-denolsing
0.6
FRAD QM9
3D-Infomax
BertMTR-5M
BertMTR-10M
BertMTR-77M
MMoR tag
MooR MotR_gat
MolR gen
ContextPred GPT-GNN
Models U
Figure 12: Pairwise information sufficiency between molecular embedders.
D.3
Complementary results on ADMET tasks
The datasets chosen for the molecular experiments are extracted from the Therapeutics Data Commons (TDC) 49 platform. We focused our experiments on ADMET tasks, crucial for drug discovery and development, which results in total of 31 tasks, described in Tab. 6
Each dataset is split into training set, a validation set, and a test set fol- lowing scaffold split. This splitting strategy ensures that molecules shar- ing similar scaffold will be part of the same split in the task. This cor- responds a more realistic scenario, where practitioners only have access molecules belonging to the same chemical series. Classifiers then trained on the training set of each task, where the best hyperparameters and checkpoints are selected on the validation set. The final performance is finally measured on the test set, and we run each experiment 10 times with different random seeds.
Drupout rate
biddee dimension
netwurk depth
n-epochs
0. 0.2
4.8, 16, 32, 64, 128
1,
min 200
5000
Table 7: Hyperparameters tuned for the evaluation of embed- ders on ADMET downstream tasks
A grid search is performed each dataset individually to maximize the average AUROC or R score across all models for binary classification and regression. We chose maximum number of epochs depending on the task size ensure all models have time converge, limiting this amount to grow to at most 200 epochs.
Tab. also displays the variation of the correlation coefficient between the ranking obtained on the Is score and the performances obtained on the downstream tasks regarding Spearman and Kendall correlations. We can see that the Is score correlates well with the performance of downstream tasks when the amount of data available is large.
Finally, we can see in Figure 13 that by grouping models based on their performances on these tasks, we obtain a similar clustering to the one obtained on the Is score in Sec. 6.2] with NI.P-inspired models grouped. Similarly, the tinCCheBert-MTRR and MolR models also grouped.
33

---

Category
Model name
Task
cls
reg
Correlation
Avg. metric in the grid
size
Qp
Q.
P-glycoprotein Inhibition
1212
0.92
0.93
0.76
0.88 = 0.03
AqSolDB
9982
0.91
0.91
0.75
0.52 + 0.09
Lipophilicity
4200
0.88
0.89
0.71
0.29 = 0.07
Caco-2 Permeability
906
0.77
0.80
0.61
0.32 + 0.08
Absorption
Human Intestinal Absorption
578
0.79
0.77
0.54
0.67 = 0.02
FreeSolv
642
0.65
0.73
0.53
0.36 = 0.12
PAMPA Permcability
2035
0.61
0.63
0.44
0.67 = 0.02
Oral Bioavailability
640
V
0.50
0.45
0.33
0.68 + 0.02
Plasma-Protein BDR
1614
0.85
0.84
0.68
0.18 - 0.04
Distribution
Blood- Brain barrier
1975
0.79
0.81
0.60
0.32 = 0.08
VDss
1130
0.71
0.73
0.53
0.16 0.05
CYPP450 3A4 Inhib.
12328
0.96
0.96
0.85
0.80 = 0.04
CYPP450 1A2 Inhib.
12579
0.94
0.95
0.81
0.87 = 0.03
CYPP450 2C19 Inhib.
12665
0.94
0.95
0.75
0.82 = 0.03
CYPP450 2C9 Inhib.
12092
0.92
0.92
0.79
0.83 + 0.03
Metabolism
CYPP450 2D) Inhib.
13130
0.94
0.91
0.74
0.78 + 0.04
CYPP450 2D6 Substrate
664
0.75
0.74
0.57
0.75 + 0.03
CYPP450 3A4 Substrate
667
0.50
0.53
0.35
0.63 = 0.02
CYPP450 2C9 Substrate
666
0.20
0.13
0.09
0.65 = 0.02
Clearance hepatocyte
1020
0.78
0.79
0.57
0.11 + 0.03
Excretion
Half Life Clearance microsome
667
0.76
0.78
0.58
-0.06 + 0.11
1102
0.72
0.72
0.54
0.08 0.03
Tox21
7831
0.93
0.93
0.78
0.75 + 0.03
13445
0.91
0.90
0.75
0.76 + 0.04
hERG
648
0.81
0.84
0.63
0.75 + 0.03
Acute Toxicity LD50
7385
0.82
0.82
0.63
0.16 0.04
Toxicity
Ames Mutagenicity
7255
0.79
0.78
0.60
0.79 - 0.03
ClinTox
1484
0.69
0.69
0.49
0.00
Carcinogens
278
0.47
0.49
0.35
0.76 = 0.08
Drug Induced Liver Injury
475
0.39
0.36
0.25
0.83 0.03
Skin Reaction
404
0.01
0.07
0.06
0.64 = 0.03
Table 6: ADMET tasks extracted from the Therapeutic Data Commons platform 49 considered in our experiments. We report the correlation between the informativeness score and the performances of the embedders on the downstream tasks in terms of Pearson correlation @p. Spearman correlation 0. and Kendall-Tau т. We also report the average metric of the models on each task across the grid search runs, terms of I22 for regression tasks and AUROC for classification tasks. The tasks ordered within each category by the correlation with the informativness score (in terms of Spearman correlation).
34

---

MoleOOD OGB GIN
MoleOOD GCN
ChemBertMLM-
ContextPred
AttributeMaskk
ChemBertMLM-SH
gat
MolR_ gen
ChamBertMTR-100
ChamBertMTRR5MM
ChemBertMTR-77MM
DendkiingPnataannngggOCWWW
GROVER
GraphMVP
FRAD QMS
GraphCL
GraphLog
InfoGraph
ChemBertMLM--10
ChamGPT:1.28
ChemGPT-4.7M
MolBert
Clearance Micmssee Lombig dard SERG
Leaphiittty. Astm Frril A2
Warg
AMES
dhe and God dl
aeraane Host
CYP266 Subetrate Carbonte 200
Becavilabiiigg
Lie,
Carbonte
Luestr
Figure 13: Heatmap representation of the performances of the different models the downstream tasks, where embedders behaving similarly on the various tasks are clustered, and the embedders are colored based on their community computed in Sec. 6.2 based on the Ig score.
D.4 Drug target Interaction prediction
propose further evaluating the embedders on yet another type of downstream task: Drug-Target Interaction. This task aims to predict the binding affinity between a given pair (drug, target). Since none of our models can process protein sequences, we decompose each dataset into multiple regression tasks single target by querying all molecules associated with label for this target. Each task is then formulated as a set of molecules:
and their labels y {y: fie (o......
However, such tasks can be small, making it hard to build proper models from the embeddings. In contrast, the number of tasks very large, making it computationally expensive proceed to
Overall
BindingD8
Binding
Notstrained,
InfoGraph
1 (rank 11
RG 3.55
GraphCL
MoiBert
BindingDB
3D-Infomax
GPT-1:26
0s: 0.88
0.70
0:8777
8:088
10
15
Down stream tasks trank
Figure 14: Correlations of the Is score with the performances on the DTT tasks, terms of Spearman and Kendall coefficients.
35

---

an adequate hyperparameter selection. To bypass these limitations, we propose to estimate the embedded space's clustering quality for each model by measuring how close the labels of molecule are compared to its nearest neighhors for each task, In other words, we measure:
Emaua (2,
Smaass (i,2,)),
N
A=
(26)
with Saaug (1,
1lys 13 y3
Reuiigh
je (2)
and Ni (X)) is the set containing the Nneigh closest neighbors of X, which would be the performances of a K-nearest neighbors regressor on the task when using one data sample.
This quantity can be interpreted as proxy of the embedding's s capability to perform a similarity search, a classic chemo-informatic method using the similarity between different molecular projections to perform predictions. This training-free and computationally inexpensive approach allows us to evaluate the models on many tasks/targets.
We focused on 4 DTI datasets: KIBA [101). BindingDB-Kai BindingDB-KK Ki, and BindingDB- IC50 (68), with total of 1496 tasks. We removed all tasks containing less than 128 molecules to ensure minimum data for the clustering evaluation.
KIBA (175 targets)
BindingDB--K Kd (22 targets)
BindingDB-Ki (372 targets)
Binding (927 targets)
Bhicight
Bp
0.85
0.84
0.84
0.83
0.90
0.90
0.87
0.84
0.88
0.88
0.88
0.87
0.87
0.86
0.86
0.86
0.88
0.88
0.87
0.86
0.87
0.86
0.85
0.82
0.87
0.87
0.87
0.88
0.86
0.87
0.87
0.86
0.72
0.71
0.69
0.67
0.68
0.66
0.65
0.62
0.68
0.67
0.69
0.69
0.66
0.68
0.68
Table Correlation between Zş's informativness score and our clustering evaluation score En on the four DTI datasets considered.
We obtain similar results as in Sec. 6.1 our metric correlating with the performances on the different tasks considered. FFigure 14 sums up all results by establishing ranking across models and the number of neighbors, where we can see that MolR and ChemBerta-MTR appear as both the most promising models according their Is score, and the best models evaluated. Furthermore, the outliers observed iriSec. 6. show different behaviors in this setting. For instance, while Infomax seemed under-estimated and InfoGraph over-estimated by the Zg score after seeing the results on the ADMET tasks, Infograph appears under-estimated in this setup, and 3D-infomax over-estimated.
E
Information Sufficiency Estimation
E.1
Estimation method
As stated Sec. 2. the deficiency 8(Prix > PzJx) is an intractable object measuring the cost of the reconstruction of Z from U. Due to this intractability, we propose to estimate the information sufficiency (U - Z), which tractable proxy for the deficiency.
E.1.1 KNIFE Estimator
We recall the definition of the information sufficiency:
Is(U + 2) A inf fEFe(Z)
E log [(2) - EE
inf (EKu(Z
E(- log M M(Z
(27)
Uncertainty Z
Uncertainty when simulating from with M
We denote C the number modes chosen for the Gaussian mixture distributions and GMp, the Gaussian mixture distribution with C components, parametrized by μ,, E, w, with 1 T such that
36

---

(, = E (elee (2 E.) the density of multivariate Gaussian distribution with mean He and covariance Σ. the c-th mean and covariance matrix.
(e: Ee), where we is the weight of the e-th component, and
To estimate the information sufficiency between the two embedders U and Z, we follow the procedure described in KNIFE 82
Fe is hence the class of multivariate Gaussian mixtures with C' components, it is parametrized by ,4, 2,, w. These learnable parameters are optimized to maximize the log-likelihood Z
The class of Markov kernels Ke (Z)U) is also composed of multivariate Gaussian mixtures whose parameters are estimated using small feedforward neural network. Such that for each u U, the parameters of the Gaussian mixture are (u(u), (n), (n)).
In practice considered the covariance matrix be diagonal avoid the number of parameters of the Gaussian mixtures to grow too large with the dimension of the embeddings d. The number of parameters to be estimated for the Gaussian mixtures are hence: (2d + 1), with the number of parameters of the feedforward networks for the Markov kernels.
Procedure 1 Estimation of Is(U + Z), GM denotes the Gaussian Mixture model with means 14, covariances Σ and weights w.
Input: Pairs of corresponding embeddings (zi, U:)N Output: Information sufficiency (U Z)
Pzz, Σz, Wz + arg min,,, E.w Di=1 log (.,... Ei= log GMp(u,) ((c)))
Pzju, U WzJu + arg min,, E.w
Uncertainty of Z: H(2) + 1 ΣN log GM Ez.wz (zz) Uncertainty of Z given U: (Z + + Ei log
Return [U + Z) + H(Z) H(Z(U)
Both the parameters of the marginal distribution of Z and the parameters of the conditional distribution of Z given U are estimated through likelihood maximization. The uncertainty of Z is estimated by the negative log-likelihood of the data under the marginal distribution of Z. The uncertainty of Z given U is estimated by the negative log-likelihood of the data under the conditional distribution of Z given U.
E.1.2 Embedding dimension normalization.
However, this approach faces major drawback: it favors models that generate embeddings of high dimensionality. To evaluate the information sufficiency between the models, we estimate the uncertainty of Z and the uncertainty of Z given U. As seen in FFigure 15 the estimated information sufficiency is highly correlated to the dimension of the latent space of Z, favoring models with high-dimensional latent spaces. This can be explained by the fact that these embedders yield larger marginal uncertainties. The resulting difference in the uncertainties (U + Z) is hence larger in absolute values.
We can see in Figure 15 that the dimension of the latent space of Z and the uncertainty of Z are evolving linearly. We thereby divide the information sufficiency by dim(Z) (Z), which can be approximation of the normalization by the uncertainty of Z. We report in Figure 16b]results without this normalization. It still correlates significantly with the downstream tasks performance, but the correlation is stronger when the normalization is applied. Hence, we focus on the relative variation of the uncertainty of Z explained by U. Note that the uncertainty of Z can be negative, it can be assimilated to differential entropy. As a result, the "true" relative variation of the uncertainty would not be suitable for comparing different models (as it would not guarantee any ordering). While there is a general trend where larger models do have larger embeddings and perform better, well-trained smaller embeddings are competitive with larger embeddings. and the Is score captures this behavior (Figure 16a).
'In practice since this operation does not depend on U, we store the weights of this distribution, and use it for all pairs Is( 2).
37

---

uncertainty
Z!u uncertainty
2000
Z uncertainty
Z unceriainty
2000
1000
1000
0
Zs(U~Z)
Is(U - -Z)/dim(Z)
750
Is(U- ZZddm(Z)
0.75
1=(0~=))
500
0.50
250
0.25
0.00
500
1000
1500
2000
500
1000
1500
2000
Dimension of Z
Figure 15: Relationship between the dimension of Z's latent space and the quantities estimated to compute the information sufficiency in molecular modeling.
build our proxy by measuring the median values of the set SIs (k) = (Zs - Zi) for an embedder Z in our pool of models.
Average datasets)
classification datasets)
Accuracy
Cart
5-mmml
Clstering (11 datasets)
Information Sufficiency
0.55
Retrieval (15 datasets)
STS datanets)
Beranking (4 detanste)
Con
Gmaa
Embeedding dimenssi
(a) Impact of the embedding size on the quality of the embeddings for the models measured actual performance on downstream tasks and Ig score.
209
350
450
500
390
24
419
450
50
380
258
420
430
59
(b) Correlations on the MTEB benchmark when embed-
dings are not normalized by their size.
Figure 16
E.1.3 Median instead of mean.
We use the median instead of the mean to compute the Is score. The median is more robust to outliers and the distribution of available embedders. For example, if many models are very similar, the mean would be biased by these models, while the median would not. Thus, we chose the median. While this change has minor impact when there is enough diversity in the models, it can have significant impact when the models are very similar, for example, when including different checkpoints of the same model Figure 17
38

---

Average (56 datasets)
allensi-spectee
30
udever-bboam--soom
e5-small
Qp
20
(10 datasets)
0.90
0.85
Ts (rank
Retrieval (15 datasets)
0.89
0.85
0.62
10
UAE-LaReeV
Classification (12 datasets)
0.92
0.84
0.65
Clustering (11 datasets)
0.86
0.83
0.61
xa+
B+
@s:
Reranking (4 datasets)
0.86
0.79
0.65
T: 0.66
Average (56 datasets)
0.93
0.86
0.66
0
5
10
15
20
25
30
Additional Classif (8 datasets)
0.84
0.81
0.61
Downstream tasks (rank 4
(a) NLP
(b) NLP
Figure 17: Correlation between Ig score computed as the mean information sufficiency and the downstream task performances in NLP. 2p is the Pearson correlation, 2s is the spearman correlation
E.2 Hyperparameter selection
We use parametric classes composed of multivariate Gaussian mixture distributions for Fe and (Z() in the definition of the information sufficiency Fq.. 27 the number of components in the mixture crucial parameter that needs be selected concerning the data distribution of interest. We ran ablation studies to evaluate the impact of the number of components in the mixture on the information sufficiency estimation and the correlation between the Ig score and the downstream tasks performance (Figure 20). We found that the ideal number of components in NLP is 8, and in molecular modeling. it 4. Figure 18 and Figure 19 show the embeddings the models in the first two principal components.
Spearen cameltton
Impact of number of
corrostion
Impact the number mades
Impact number modes
0.77
0.76
0.5
0.56
0.00
0.50
20
Number of modes
Number modes
Number modes
(a) Spearman
(b) Kendall-Tau.
(c) Pearson.
Figure 20: Impact of the number of modes used estimate the Is score and its correlation with the downstream tasks performance in NLP. We chose to use 8 modes in practice.
E.3 Impact of the task size
Our study focused on finding the most promising model to be competitive on any downstream tasks. However, if the downstream task is not learnable, the most promising model could appear not competitive on this specific task. In particular, if the amount of data available in the downstream task is insufficient, the differences between different embedder's representations might not be easily leveraged. This phenomenon can be seen in Figure 21] highlighting how when fewer than 1000 data points are available, the correlation between the Iş score and the downstream performance becomes weaker.
39

---

Ind Vambbr
Matyelmnenii
130
haskraarraddig Lenit
40
Figure 18: 2D Projection of the embeddings of the models in the first two principal components colored by datasets in NLP
40

---

OPHONN
Grepanyy
GINRROL
2
2
31
Figure 19: 2D Projection of the embeddings of the models in the first two principal components colored by datasets in molecular modeling. Hue corresponds to the synthetic accessibility score (34].
1.0
1.0
x
2x
x
0.75
0.5
0.5
0.50
0.25
0.0
0.0
0.00
103
104
103
104
103
104
task size
task size
task size
Figure 21: Impact of the task size on the Is score's ranking's 's correlation with the downstream tasks performances molecular modeling in terms of Pearson correlation 2p. Spearman correlation @s and Kendall-Tau r coefficient.
E.4
Impact of the number of models
Typo Pearson
Type Spearman
Typc Kondall-Tau
Comelation
0.6
Average (56
Metric
Classriiation Average (12 datasets)
0.6
Clustering (11
Retrieval Average (15 dataset
0.4
STS Average (10 datas
Reranking Average
10
20
30
10 20
30
10
20
N
Figure 22: Impact of the number of models used to compute the Is score in NLP.
We evaluate the strength or Is score an embedder with respect to all the others by relaxing the "for all" conditions with the median Ig score. Thus, the number of available embedders might impact the performance of our method. Indeed, if too few embedders are available, it is likely that our evaluation would be biased by favoring models similar the few available ones. We evaluate the impact of the number of available models by sampling subsets of our global model pool to compute the Is score. In Figure 22 found that when fewer models are available, the rankings obtained using the Is
41

---

score correlate less with the downstream tasks' performance. However, as the number of models increases, the Zg score becomes a good proxy for the performance the models on the downstream tasks.
However, the evolution of this correlation is dif- ferent in the two studied domains. Even with very few models, the Is score molecular models already achieves Spearman correla- tion close to 0.8 with the downstream tasks' per- formance. The correlation is much lower in NLP and only reaches 0.8 when about 15 mod- els are available. This result be expected, seeing how sparse the pairwise comparison ma- trix in NLP compared molecular modeling. This matrix is less sparse on molecular data, the graph induced by this adjacency matrix is more connected, and having access to few nodes is enough obtain measurements on the whole graph. On the contrary, by being much sparser, having access to few nodes in this graph in NL.P might only give information about the local structure of the graph in the community of the few nodes available. This might explain why the Zg score's s correlation with the downstream tasks performance stabilizes in NLP at about ten models, which is equal to the number of communities identified the graph (in Sec. 5.2).
Type Kendall-Tau
Type
Spearman
Dataset
ADMET
Absorption
0.8
Metabolism
Tox
Distribution
0.7
0.5
Excretion
20
20
N
N
Figure 23: Impact of the number of models used to compute the Ig score in molecular modeling.
E.5
Comparison with other metrics
We evaluate other ways to measure the informativeness (Tb.. 9 and Tab. 10) of the embedders. We considered the size of the model, their embeding dimensions and the simple reconstruction when we train a cross-encoder to go from one embedding to another. Overall we found that our information sufficiency score significantly outperformed these naive metrics in both modalities.
In particular, training a cross-encoder with the la reconstruction error proves to correlate with the ability to enable downstream performance, however, this correlation remains weaker compared to using the information sufficiency.
Table 9: Comparison with Basclines: Size of the Embedder, Dimension of the embedding output (d) and the la reconstruction error of the embeddings for NLP datasets.
Size
is
2>
2.
Cr
2n
0.
Qy
0.
Classification (12 datascts)
0.46
0.42
0.32
0.52
0.66
0.55
0.92
0.88
0.73
-0.79
-0.85
-0.66
Retrieval (15 datascts)
0.40
0.39
0.29
0.46
0.63
0.52
0.89
0.89
0.70
-0.71
-0.84
-0.65
Clustering (11 datasets)
0.45
0.38
0.26
0.54
0.67
0.55
0.86
0.85
0.67
-0.80
-0.84
-0.66
STS (10 datascts)
0.27
0.35
0.25
0.34
0.52
0.82
0.62
-0.70
-0.83
-0.64
Reranking datascts)
0.33
0.33
0.26
0.41
0.61
0.50
0.84
0.79
0.64
-0.71
-0.78
-0.59
Average (56 datasets)
0.41
0.41
0.31
0.48
0.62
0.50
0.94
0.90
0.74
-0.77
-0.84
-0.65
Additional Classif (8 datasets)
0.41
0.62
0.47
0.43
0.64
0.55
0.89
0.84
0.66
-0.65
-0.72
-0.55
Table 10: Comparison with Baselines: Size of the Embedder, Dimension of the embedding output (d) and the &a reconstruction error of the embeddings for Molecular Modeling datasets.
Cp
2-
@p
e+
Pp
e+
Dp
Absorption (8 datasets)
-0.21
-0.16
-0.43
-0.29
0.89
0.70
-0.89
-0.70
Distribution (3 datasets)
-0.07
-0.03
-0.46
-0.31
0.89
0.70
-0.66
-0.66
Metabolism (8 datasets)
0.06
0.03
-0.46
40.34
0.94
0.79
-0.90
-0.71
Excretion (3 datasets)
-0.17
-0.11
-0.24
40.18
0.77
0.60
-0.77
-0.56
Toxicity (9 datasets)
0.09
0.06
-0.49
-0.35
0.92
0.75
-0.86
-0.67
ADMET (31 datasets)
-0.01
0.01
-0.47
-0.32
0.94
0.80
-0.90
-0.72
42

---

E.6
Computational ressources
Evaluating the Ig score of the models is computationally inexpensive. Evaluating the Ig score requires only single (small) GPU. All our experiments were conducted on NVIDIA V100 and NVIDIA A6000 GPUs.
Our method's main (computational) shortcoming stems from the need to compute the information sufficiency between all pairs of models. This quadratic operation in the number of models. However, in practice, optimizing and estimating the information sufficiency presented in Sec.. 3 is cheap. The complete evaluation of the 45 NLP models be done less than hours on a single GPU.
43

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We setup the problem of evaluating embedders as a communication problem and leverage information theoretic tools derive a principled evaluation method.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We provide separate section "Conclusion and Limitations" (Sec. 7) in the main paper and different ablations addressing possible shortcomings of our methods in appendices.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
44

---

Answer: [Yes]
Justification: We provide a complete description of our setting and assumptions used to derive theoretical proofs well, as the relaxation of the problem make it tractable (Sec. 2) Appendix B)
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide a description of our global experimental protocol in the main paper Sec. 4) and detailed descriptions for the different field in their respective sections (Appendix Cl Appendix D). All the datasets and models used are publicly available and the code to reproduce our results is attached to the submission.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
45

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We attached the code to the submission and provide instructions reproduce the results in the main paper and supplemental material. All datasets and models are publicly available and chosen to be easily accessible for reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
7 //nips //nips.cc/pubbe. wwtpppppccp.guude..ppspppppppppe http://dddee/oeeSSubiiieeeea (ww.oeSumiiseiooool......o
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide in Appendix Appendix C and Appendix D the details of the training and test settings for the different fields. We also provide a global description of the experimental protocol in the main paper (Sec. 4)
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: We provide error bars in the figures and tables of the main paper. In addition provide robustness checks and ablations in the appendices.
Guidelines:
46

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the details of the compute resources used in the main paper and appendices in Sec. E.6
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: Etthes/attps::/nnurrp http:ppp.....pulllcccccaciiaaaa.
Answer: [Yes]
Justification: This paper does not involve major ethical concerns and the research conducted in the paper conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
47

---

Justification: We only propose model selection method for embedders. The main concern, that is addressed in the paper, is that while our method theoretically supported and empirically validated, it might not be applicable to all downstream tasks and should be used with caution.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
We provide the citations and direct references of all the datasets and models used in our work. To ensure reproducibility, all the datasets and models used are publicly available and under open licenses.
Justification:
Guidelines:
48

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paaassttssppersseaaes paperswi thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: We only created the library that implements our methods (it will be released publicly github) and the code reproduce experiments. Both are documented as part of the code submission supplementary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
49

---

Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
50


Reviews:
[{"review_id": "ruwRuk7Dcc", "reviewer": "Reviewer_id3c", "summary": "This paper proposes a new metric to compare embedding models without relying on labeled data. The approach involves embedding data using two separate neural networks Z and U, and then trying to use embedding model U to simulate/match the output of embedding Z. They then use this to calculate an information sufficiency criterion. They motivate this with theory, and show that their information sufficiency metric aligns nicely with established embedding benchmarks across two separate domains, NLP and molecular biology.", "questions": "1. How exactly do you calculate the information sufficiency metric I_s for a given dataset in MTEB and two models? Can you expand on this in more detail (as mentioned above)?\n\n2. In the discussion section, you mention that there are other potentially promising methods for learning the Markov kernel. What do you think some other promising approaches might be?\n\n3. In Section C.3.1, the authors admit that their results are better for MTEB classification tasks when compared to STS, clustering, and reranking tasks. What numbers/tables are they referring to here?", "limitations": "It would be helpful for the authors to share code upon publication so that others can use this metric/approach.\n\n### Minor comments:\n\nTypo Figure 3(b) “Molecular Modelling”\nTypo line 213 “embbeders”", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "This is an innovative, statistics-based approach that can potentially add a lot to the rapidly evolving subfield of embedding models.\n\nThis paper validates their approach across completely different domains: NLP and molecular biology. The authors have run extensive experiments on tens of embedding models and many different datasets within the two domains.\n\nThis paper is very well written with an attention to detail (and an extensive appendix). The paper also does a valiant job of combining theory and experiment.", "weaknesses": "1. This paper puts a heavy emphasis on the theoretical motivation without giving enough detail about the actual algorithm for calculating the information sufficiency metric I_s. Line 211 states “We hence attempt to simulate Z from U by learning a Markov kernel M via a mixture of multivariate Gaussians, and measure the uncertainty reduction it induces.” It would be helpful to expand on this in detail. The reader is left only guessing how exactly this was done, and in my opinion it does not seem quite replicable. The details in the Estimation method section E.1 and Hyperparameter selection section E.2 are somewhat minimal.\n\n2. This method is touted as a principled way to compare embedding models without using data labels (but still using data in the desired domain). However it is unclear what the other tradeoffs are. Aside from the question of data labels, Is this approach more computationally efficient?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We warmly thank Reviewer id3c for their reviews. \n\n**Weaknesses:**\n\n1. We refer the reviewer to Questions below.\n2. **Computational costs.** Our method is cheap computationally. We quickly discussed the computational cost of evaluating our method in Sec. E.5. Computing the informativeness of a model requires evaluating $N$ information sufficiency, where $N$ is the number of reference models, each requires fitting two mixtures of Gaussians (the marginal and the conditional). With precomputed embeddings (~ $150$k samples per embedder), it took us less than $6$hours on a single GPU to compute the $45 \\times 45$ (all the pairs) information sufficiency to evaluate all the text embedders. As a result, in our examples, computing the informativeness score of a model takes less than 15 minutes.\n\n**Questions:**\n\n\n\n1. **Estimation of $I_S(U \\longrightarrow Z)$ (See PDF in general comment).** For a given dataset $D$, we generate the embeddings $(u_i, z_i).$ We then fit a Gaussian mixture on the embeddings $(z_i)_i$ and a parametric Gaussian mixture on the embeddings $(z_i)_i$, parametrized by $(u_i)_i$, i.e. the means and covariance matrices are estimated by a small feed-forward network taking $u_i$ as input, minimizing the negative log-likelihood of the samples $(z_i)_i$. We then compute the corresponding empirical entropies using those trained distributions (See the attached PDF file in general comment for the detailed algorithm; we will add it alongside the detailed description to the core of the paper using the additional page allowed to address the reviewers’ comments). In addition, the code source to perform this estimation is available as part of the supplementary material submitted alongside the paper.\n2. **Better estimators of the deficiency.** The information sufficiency is admittedly a proxy for the actual deficiency we wanted to evaluate. Our decision was guided by tractability and numerical stability arguments (we expand more on this in our answer to Reviewer o2cb). There is still work to be done to find a proven estimator. For example, we were able to show a more direct connection between the deficiency and the critic loss of a GAN. However, we could not make it work in practice: this more grounded approach is too numerically unstable and leads to very poor results (e.g., the information sufficiency yields correlations of $0.90$, while this more grounded approach yielded at most $0.70$). This opens an interesting line of research for the near future.  \n3. **Performance for classification tasks and others.** In Figure 3, table (a), the Kendall-Tau correlation between our metric and the performance on downstream tasks is significantly higher ($0.73$ ) for the classification tasks than for the other type of tasks ($0.69$ at best and closer to $0.65$ globally). Our interpretation of these results is that classification tasks rely on the training of an additional classifier on top of the embeddings, whereas the other tasks only rely on the dot product between embeddings to make decisions (retrieval, similarity and clustering). Our theoretical results give insight into what is doable when learning the best classifier possible, but they do not guarantee that the dot product / l2 distance in the embedding space captures any useful semantics.\n\n**Source code.** We released the source code for this project: it is available as supplementary material attached to the submission and at [[https://anonymous.4open.science/r/emir-B8D3/Readme.md](https://anonymous.4open.science/r/emir-B8D3/Readme.md)]. A public repository will be shared upon acceptance. Our implementation only relies on pre-computed embeddings and thus can directly be used for any domain as long as they have been dumped. We will make sure to expand the Estimation Method section to provide more details on how we estimate information sufficiency in practice, and we will publicly release the library and code as a practical library to estimate our metric.\n\nWe hope we were able to alleviate your concerns and that our answers will allow you to improve your already positive assessment of our work, and we remain available for any additional information.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ip9MxmLqGD", "reviewer": "Reviewer_o2cb", "summary": "Evaluating embedding models is challenging because it typically relies on various downstream task data, despite the embedding models being trained for general purposes. This paper introduces an information-theoretic metric for comparing embedding models, eliminating the need for labeled datasets in their evaluation. The core idea is that if one embedding model can simulate another in most cases, it indicates a higher capacity for distinguishing concepts by its embeddings. Empirical results in both text and molecule embedding models show that the proposed metric closely aligns with downstream task rankings. Additionally, the community analysis facilitated by this metric reveals clusters of embedding models, illustrating their relationships.", "questions": "- Does transitivity (or a similar weaker concept) hold for information sufficiency? For example, if $I_S(U → Z) > I_S(W → Z)$ and $I_S(W → Z) > I_S(X → Z),$ does $I_S(U → Z) > I_S(X → Z)$?\n- Are there specific reasons for using multivariate Gaussians in Markov kernel learning? Are there any alternatives?\n- What are the requirements for data in estimating information sufficiency? How do the quantity, quality, and diversity of datasets affect the effectiveness of the proposed metric?\n- Line 114: Should $P_{U|X}$ be $P_{Z|X}$?\n- Lines 188-189: Isn't this bound practically vacuous due to the size of X?", "limitations": "The authors adequately address the limitations.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The target problem is clear, and the proposed metric is well-motivated from information theory perspective.\n- The experiments are extensive, demonstrating the effectiveness of the proposed metric.\n- The method allows for community analysis of embedding models, which is quite interesting.\n- The paper is well-written and easy to follow.", "weaknesses": "- The connection between the concept of deficiency and information sufficiency is somewhat unclear.\n- The computation of information sufficiency in practice is not well-explained. Adding more details would be beneficial.\n- (Minor) The arrangement of tables and figures could be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer o2cb for their review and the interesting questions they raised. We do our best to answer them below.\n\n**Weaknesses**\n\n1. **Information sufficiency and deficiency. (See Official comment below for the proof)** We can establish the following connection between deficiency and mutual information, where information sufficiency serves as an estimate:\n$$\n \\\\delta(P_{U|X} \\rightarrow P_{Z|X} ) \\ge\\inf_M \\\\mathbf{E}[\\ell(X,M)] \\\\ge R_{X,\\ell}^{-1}(I(Z,X;U))\n$$\nWhere $R_{X,\\ell}$ is the rate-distortion function for $X$ and a loss function $\\ell(\\cdot)$. **This is a non-increasing function in mutual information: when mutual information is low, the deficiency is high. This suggests that the mutual information $I(Z;U)$ should be as large as possible as a necessary condition to achieve lower deficiency.**\n__Due to character limits, the proof is provided in the following comment. We will include it in the final version of the paper.__\n\n2. **Estimation of $I_S(U \\longrightarrow Z)$.** For a given dataset $D$, we generate the embeddings $(u_i, z_i)$. We then fit a Gaussian mixture on the embeddings $(z_i)_i$ and a parametric Gaussian mixture on the embeddings $(z_i)_i$, parametrized by $(u_i)_i$, i.e. the means and covariance matrices are estimated by a small feed-forward network taking $u_i$ as input, minimizing the negative log-likelihood of the samples $(z_i)_i$. We then compute the corresponding entropies using those trained distributions. (See additional PDF, for the detailed algorithm, we will add this algorithm alongside the detailed description to the core of the paper using the additional page allowed to address the reviewers’ comments). In addition, the code source to perform this estimation is available as part of the supplementary material submitted alongside the paper.\n3. We will further improve the pagination and organization of the tables and figures.\n\n**Questions**\n\n1. **Transitivity.** The example provided in the review: If $I_s (U→Z)>=I_s (W→Z)$ and $I_s (W→Z)>= I_s (X→Z)$, then $I_s (U→Z)>=I_s (X→Z)$ is always true by the transitivity of the “>=” relationship as boils down to $(a \\geq b \\text{ and } b \\geq c) \\implies a\\geq c$. \n**A less straightforward transitivity property exists for the deficiency**, indeed we can show that\n$\\delta(P_{A|X}\\rightarrow P_{C|X}) \\leq \\delta(P_{A|X}\\rightarrow P_{B|X}) + \\delta(P_{B|X}\\rightarrow P_{C|X})$.  This relation simply states that the reconstruction error of an embedder C by an embedder A is lower than the reconstruction of any embedder B by A, and the embedder C by B. __We include the proof at the end of this rebuttal and will add it to the final version of the paper__\n\n1. **Choice of Gaussian Mixtures.** Gaussian mixtures are known to be universal estimators of densities, and to our knowledge, the KNIFE paper corresponds to the state of the art for this kind of information-theoretic estimations. Another option is to directly fit cross-encoders (feedforward networks) using the MSE loss and use this reconstruction loss as score of informativeness. Our method consistently outperforms this reconstruction loss strategy, reaching higher correlation scores (a Spearman correlation of 0.84 compared 0.9 in NLP and 0.9 compared to 0.94 in molecular modeling on all benchmarks) on every benchmark we considered (See attached PDF and general comment for full results and additional baselines).\n2. **Reference set requirements**. The requirement is that the dataset used is sufficiently large to correctly represent the distribution of data that will be presented in practice (the ZINC dataset for molecules for example). In Section C.3.6, Figure 10, we evaluate the impact of evaluating the I_S on different subsets of data on the correlation with the downstream tasks performance. Evaluating broader data tends to help, but the performance is even stronger if the data used to evaluate the I_S are close to the data used for the downstream tasks. For example, evaluating $I_S$ on the Amazon polarity gives good insights on the performance on the imdb dataset whose task is related: the goal is to evaluate the sentiment about a movie. \n3. Yes, this is correct. We will correct this typo. \n4. **Tightness of the bound.** It might be the case, but we are not interested in actually estimating the bound but rather in the relationships between the terms of said bound since our goal is to compare embedding models. Our goal was to highlight the connection between the deficiency and the different risks: controlling and comparing the deficiency is still a good idea even if the bound is not tight. \n\nWe extend our thanks to Reviewer o2cb for their thorough review of our work, and we hope our answers address all their concerns so they can reassess our work’s quality.\n\n---\n\n__Proof: Transitivity of the deficiency__\nIt is easy to check that\nEq 1:\n$$\n \\\\delta(P_{A|X}\\\\rightarrow P_{C|X})  =:  \\\\inf_{M\\\\in \\\\mathcal{K}(C|A)} \\\\| M\\\\!\\\\circ\\\\!P_{A|X} -  P_{C|X} \\\\|\\_{\\\\text{TV}}  \\\\leq \\\\inf_{M\\\\in \\\\mathcal{K}(C|B)} \\\\inf_{ M'\\\\in \\\\mathcal{K}(B|A)} \\\\| M\\\\!\\\\circ (M'\\\\!\\\\circ\\\\!P_{A|X}) - P_{C|X} \\\\|_{\\\\text{TV}}.\n$$\n\nOn the other hand, for any Markov kernels $M\\in \\mathcal{K}(C|B), M'\\in \\mathcal{K}(B|A)$:\n\nEq 2:\n$$\n\\\\| M\\\\!\\\\circ M'\\\\!\\\\circ\\\\!P_{A|X} - P_{C|X} \\\\|\\_{\\\\text{TV}}= \\\\| M\\\\!\\\\circ M'\\\\!\\\\circ\\\\!P_{A|X} -  M\\\\!\\\\circ\\\\!P_{B|X} +  M\\\\!\\\\circ\\\\!P_{B|X} - P_{C|X} \\\\|\\_{\\\\text{TV}} \\\\leq \\\\| M\\\\!\\\\circ M'\\\\!\\\\circ\\\\!P\\_{A|X} -  M\\\\!\\\\circ\\\\!P\\_{B|X} \\\\|_{\\text{TV}}+\\\\|  M\\\\!\\\\circ\\\\!P\\_{B|X} - P\\_{C|X} \\\\|\\_{\\\\text{TV}},\n$$\n\nand by data processing inequality on the TV norm: \nEq 3:\n$$\n\\\\| M\\\\!\\\\circ M'\\\\!\\\\circ\\\\!P_{A|X} -  M\\\\!\\\\circ\\\\!P_{B|X} \\\\|\\_{\\text{TV}} \\\\leq \\\\| M'\\\\!\\\\circ\\\\!P_{A|X} -  P_{B|X} \\\\|\\_{\\text{TV}}.\n$$\n\nThe desired inequality follows by applying inequality Eq.3 to Eq.2, and then taking the infimum at both sides over all Markov kernels $M\\in \\mathcal{K}(C|B)$, $M'\\in \\mathcal{K}(B|A)$, and using inequality Eq.1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KRBXkZ4fKT", "reviewer": "Reviewer_UcaH", "summary": "This article introduces a novel framework, grounded in information theory, for assessing the relevance of embedding models (*embedders*). The authors begin by introducing the notion of *sufficiency* of a model A relatively to a model B, which can be used to rank embedders. They prove:\n1- \tsufficiency implies (i) a higher capacity to distinguish concepts (Prop. 1)\n2- \tsufficiency is equivalent to (ii) having a lower expected Bayes risk (Prop. 2)\nThey reasonably assume that (i) and (ii) is equivalent to “being a good embedder”.\nThe authors then introduce the *deficiency* between two models, a relaxed version of sufficiency. Theoretical bounds on the Bayes Risk, as a function of the deficiency, are given (Corr. 1). As deficiency is hard to compute, they develop a surrogate estimator termed *information efficiency* (Definition 3), which can be empirically estimated.\n\nThe authors perform experiments to validate their theoretical findings using datasets from NLP and molecular modelling. In the NLP domain, they employ datasets from the MTEB benchmark to demonstrate the correlation between their new ranking scores and model performance. For molecular modelling, they use the ADMET dataset. The authors also propose a graph visualization of the models build from the pairwise information efficiency values.", "questions": "- You mention (L211-212) that “We hence attempt to simulate $Z$ from $U$ by learning a Markov kernel”: do you need to learn a Markov Kernel for each model ? Is that computationally expensive ?\n- While framing the comparison between embedding models this way is new, it would be interesting to have . However, it would be interesting to have a baseline, even the naivest one (e.g. the number of parameters maybe?) to be able to compare your results.", "limitations": "Limitations have been adequately addressed.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- In this paper, the authors propose an original task, and the approach developed is new.\n- The paper is well organized; theory and experiments are well motivated. The progression of the authors' reasoning is very clear.\n- The proposed method allows for a graph representation of embedders according to their relative expected performance on the considered dataset.", "weaknesses": "- Several key points of the method are not explicit:\n    - While the main metric is well described, there is little information on how it is estimated in practice. Besides the fact the KNIFE estimator from [82] is used to quantify mutual information, no procedure is described and the reader is left to try to rebuild it. \n    - How to choose reference data is not elaborated on, besides L229-230.  \n    - The choice the median, discussion about number of models, or embedding dimensions are not discussed within the core of the article while it would help a lot to grasp intuitions about how the proposed metric behaves.\n- On motivation and applicability: \n     - It would be interesting for the authors to expand on the practical utility of such a method. While it seems to be an interesting alternative to benchmarks which does not require any labelled data, I am not sure of how interesting it could be for applications, especially considering the paper does not expand on how difficult and computationally costly the estimation procedure is.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank Reviewer UcaH for their detailed account of our work and the efforts they put into their review.\n\n**Weaknesses.**\n\n1. **Estimation of $I_S(U \\longrightarrow Z)$ (See PDF in general comment).** For a given dataset $D$, we generate the embeddings $(u_i, z_i)$. We then fit a Gaussian mixture on the embeddings $(z_i)_i$ and a parametric Gaussian mixture on the embeddings $(z_i)_i$, parametrized by $(u_i)_i$, i.e. the means and covariance matrices are estimated by a small feed-forward network taking $u_i$ as input, minimizing the negative log-likelihood of the samples $(z_i)_i$. We then compute the corresponding entropies using those trained distributions (see additional PDF, for the detailed algorithm, we will add this algorithm alongside the detailed description to the core of the paper using the additional page allowed to address the reviewers’ comments). In addition, the code source to perform this estimation is available as part of the supplementary material submitted alongside the paper.\n2. **Choice of Reference Data**. We focused on global evaluation with a large and diverse set of reference data that would be representative of the data distribution of most of the evaluated downstream tasks (ZINC dataset in drug discovery). In Section C.3.6, Figure 10, we used different reference datasets to evaluate our metric. We found that evaluating on data that are close to the ones of the downstream tasks produces better correlation. The most striking example is the performance on the IMBD classification task (evaluating the sentiment of a film review). When our metric is evaluated on the AG News dataset (a dataset of news articles) the correlation with that task is significantly lower than when evaluated with the overall common set or with the amazon polarity dataset, which consists of amazon reviews that are arguably close to film reviews. We plan to extend our work to practical settings to evaluate models on different reference sets to find the best embedder for a given modality or subdistribution.\n3. **Implementation details in the paper body.** We will use the additional page allowed to include reviewers’ comments to include details on the $I_S$ estimation and discuss our different choices  (number of models, choice of the median, and embedding dimensions) to give more intuitions to the reader. \n4. **Practical utility.** The main use case for our method is to compare different foundation models as it focuses on how well data are separated in the embedding space. In NLP, foundation models are trained on vast amounts of very diverse textual data, but it is unclear if it embeds well certain types of data (longer texts, simple reviews or question/answer passages etc…). In molecular modeling, since tasks are often extracted from wet-lab experiments, obtaining these labels is expensive, time-consuming, and they are often noisy. In molecular embeddings, the Information sufficiency graph and the identification of communities is helpful to see how the information encoded by a 3d model is inaccessible to the 2D models so far, even though these 2d models were trained to incorporate this information (see Sec D.2). We are currently working on applications of this work as follow up work to evaluate foundational models in medical computer vision: the goal is to evaluate the quality of the embedding models for the different modalities and data distribution they are supposed to handle.\n\n5. **Computational costs.** We discussed the computational cost of our method in Sec. E.5. Our method is actually surprisingly cheap computationally. Computing the informativeness of a model requires computing $N$ information sufficiency, where $N$ is the number of reference models, each requires fitting two mixtures of Gaussians (the marginal and the conditional). In practice, if the embeddings are precomputed (~ $150$k samples per embedder), it took us less than $6$hours on a single GPU to compute the $45 \\times 45$ (all the pairs) information sufficiency to evaluate all the text embedders. For reference, evaluating the whole MTEB benchmark for a model takes hours.\n\n**Questions.**\n\n1. **Learning kernels.** Yes, we need to fit a kernel per pair of embedders. It grows quadratically in terms of the number of embedders to build the whole graph. However, the kernels are small (3 layers feedforward networks to evaluate the conditional distributions), and the overall evaluation is quick, as discussed in our computational costs analysis (Sec E.5).\n2. **Baselines.** We included 3 additional baselines: the model sizes (as you suggested), the embedding dimensions, and a less naive one: the reconstruction error of a cross-encoder, a simple feedforward network trained to transform an embedding into another directly, and we use the average l2 loss as a score. Our method consistently outperforms all baselines, reaching higher correlation scores on all considered benchmarks (using the l2 score, the Spearman correlation achieves -0.84 and -0.9 in NLP and molecular modeling, respectively, compared to 0.9 and 0.94 with our method; more details are provided in the general comment).\n\nWe renew our thanks to Reviewer UcaH for their thorough review of our work, and we hope our answers addressed all their concerns so they can re-assess our work’s quality.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FykIbsYWN1", "reviewer": "Reviewer_4AaU", "summary": "The authors propose a task-agnostic method for the evaluation of embedding models called \"information sufficiency\". The general notion is to generate a pairwise matrix that effectively measures how well each embedding model can be used to generate the information content of the other. They compute an overall \"information sufficiency score\" for each model by computing the median of this pairwise score along one axis. They demonstrate that their metric correlates well with overall downstream task performance.", "questions": "The metric proposed here is a pairwise metric where the embedding model that can most effectively simulate other embedding models is ranked as the most effective. I wonder if it is possible that if a particular embedding model contains unique information that is not represented by any other embedding model in the chosen set, then is it possible that said embedding model would be ranked poorly? The thrust of my concern is that the adoption of this metric would favor embeddings that are more \"central\" even in edge cases where more niche information might be more useful. \n\nAs a follow up to the last question, I am wondering in what sense the question of whether one embedding model is better than another is generally useful. For most applications and downstream tasks, one does not want an embedding model that is better on average for a host of applications, they want one that is better for their proscribed use case, whatever that may be. Of course, there is the challenge that it is impractical to \"try every model\", given the breadth of available models. Is there some middle ground here, where a similar method can be used to narrow the number of models that are tested rather than to recommend a single one?  \n\nThe authors acknowledge that the usefulness of their methods depends on sampling a large number of diverse embedders with which to compare. To what degree is this circular - does this constraint require that you evaluate the set of embedders using a more traditional downstream benchmarking procedure?\n\nThe underlying method has some real similarity to the task affinity matrix from the \"Vision Taskonomy\" work by Zamir et al. (which is currently not cited), although there the idea is not so much to better compare embedding models as it is to understand the informatic structure of tasks in vision literature. Is there perhaps a way of using this pairwise similarity matrix to embed the differences in informatic content across these embedding models using this related approach?\n\n\nMinor aesthetic/typographical comments: \n\nThere is the floating phrase \"inforatmation sufficiency\" on page 26 in the supplement. \n\nHoly Boldface Batman. That's a lot of boldface. Especially the sentences that start as normal and lead into boldface.", "limitations": "Please see my questions. No obvious negative societal impacts of the work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is very well-put-together and cleanly written. The intuition behind the method is well-motivated and makes sense. Most of my outstanding questions were addressed in the supplement. The topic of comparing embedding models is of substantial relevance and interest. There is a need for a better understanding of the quality of embedding models and for a task-agnostic evaluation metric.", "weaknesses": "My primary concern centers on the potential pitfalls from using this method, described in more detail in the questions below. I am very open to raising my current score of 6 if those concerns can be addressed sufficiently.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We warmly thank Reviewer 4AaU for their detailed review and the effort they put into evaluating our work. \n\n**Questions.**\n\n1. **Orthogonal models and unique information.** Yes, a model could contain unique information and still be competitive while not predicting the others.  Our method allows the discovery of such situations by studying the communities in the predictiveness graph: a model containing only unique information would appear as a disconnected component, which should prompt further investigation. A larger set and a more diverse set of models naturally reduce this risk. (See Sec E.4 for a study of the robustness of the method for different sampling of reference models Figure 22 and Figure 23). In molecular modeling, both GPT-GNN and ContextPred appear unique and disconnected, but both of these models also end up being the worst-performing models. In our experiments, we found that models containing only niche information did not perform well, suggesting that good models have to contain, to some extent, a minimal amount of common knowledge.\n\n\n2. **Evaluation of embedding models.** Our method aims to evaluate generalistic embedders that could be used for a host of possible tasks for a data distribution of interest, thus evaluating foundational embedding models. As you pointed out, it is useful to identify interesting candidates to be used as starting points to evaluate some downstream tasks, but also, more generally, one might not want to rely on different embedders for different tasks because of the potential computational overhead and be interested in the more “generally good” embedder for their data distribution of interest. Pursuing these goals, our method provides the ranking of embedders and community analysis, uncovering similar embedders and orthogonal communities that might bear different types of information. An interesting direction would be to explore if the different clusters discovered by our method perform differently on different tasks, supporting the hypothesis that they capture different types of information (we provided a short analysis of this question in sec C.3.5 and sec D.3).\n\n3. **Diversity of reference models.** We considered the diversity of architectures, training distribution, and training objectives, and we suppose that these differences lead to different information being captured. We do not rely on performances on downstream tasks (which would indeed be circular), in Sec. E.4, Figures 22 and 23, we study the robustness of our method to the sampling of reference models and the number of sampled models.  \\\nIn NLP, 10 models suffice to reliably achieve strong correlations (~ 0.90) between the informativeness score and the performances of the model, and in molecular modeling, while with very few models, the correlations are still high (above 0.8), with 10 models we reliably achieve a correlation above 0.9. We computed the correlation scores for different subsets of reference models and reported the average and standard deviation of the correlation score. We will include this discussion in the body of the paper.\n4. **Vision Taskonomy.** It is indeed very interesting work that we shall include in our related work, and it also appeared extremely relevant to the follow-up work we were conducting! \\\nHowever, it remains unclear to us how to directly connect it to this current work. We identify two possible directions.\n    1. Frame the objectives of comparing embedders using their proposed methodology by adapting the notion of tasks in the paper of Zamir et al. \n    2. Apply the ordinal normalization and the taxonomy discovery procedure to the predictiveness matrix we get using the information sufficiency.\n\n    \tBoth approaches are non-trivial to execute as it is unclear how to reproduce the transfer modeling for each models/tasks (especially considering the different modalities used) and what would be the optimization constraints for the taxonomy discovery (they are looking for the taxonomy that minimizes the overall amount of training data required to train all models to produce a “scheduling of training”). \n\n\n    Pertaining to your question, we ran eigenvalues analysis and spectral embedding of the embedders based on the similarity matrix to extract communities and evaluate embedders' proximities and found similar results as those reported in the community analysis presented in our work (See Figure 1, in the PDF page attached to the general comment). We will include those in the appendices. \n\n\nWe extend our thanks to Reviewer 4AaU for their thorough review of our work, and we hope our answers address all their concerns so they can reassess our work’s quality.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate that all the reviewers have recognized our work's novelty, significance, and clarity, as well as its comprehensive empirical analysis.

The reviewers raised 3 main concerns: a lack of details about the estimation procedure, its practical usage, and computational cost, and requested additional baselines.

**Estimation of $I_S(U \longrightarrow Z)$.** For a given dataset $D$, we generate the embeddings $(u_i, z_i)$. We then fit a Gaussian mixture on the embeddings $(z_i)_i$ and a parametric Gaussian mixture on the embeddings $(z_i)_i$, parametrized by $(u_i)_i$, i.e. the means and covariance matrices are estimated by a small feed-forward network taking $u_i$ as input, minimizing the negative log-likelihood of the samples $(z_i)_i$. We then compute the corresponding empirical entropies using these learned distributions **(See algorithm in attached PDF for further details).** We will add this algorithm alongside the detailed description to the core of the paper using the additional page allowed to address the reviewers’ comments. 

**Source code.** In addition, the source code is available as part of the supplementary material submitted alongside the paper as well as an anonymous github repo: [[https://anonymous.4open.science/r/emir-B8D3/Readme.md](https://anonymous.4open.science/r/emir-B8D3/Readme.md)] and it will be publicly released upon acceptance as an easy-to-use library to evaluate our metric.

**Computational cost and applicability (Section E.5).** Computing the informativeness of a model requires computing $N$ information sufficiency, where $N$ is the number of reference models, each requires fitting two mixtures of Gaussians (the marginal and the conditional). In practice, if the embeddings are precomputed (~ $150$k samples per embedder), it took us less than $6$hours on a single GPU to compute the $45 \times 45$ (all the pairs) information sufficiency to evaluate all the text embedders. As a result, in our examples, computing the informativeness score of a new model takes less than 15 minutes. For reference, evaluating the whole MTEB benchmark takes hours for a single model.

**Additional baselines.**  As suggested by Reviewer UcaH, we included $3$ baselines: the number of parameters of the models, the dimension of the embeddings, and a reconstruction loss score. We fitted simple feed-forward “cross encoders” from one embedder to another. We use their reconstruction loss as a score. **We found that our method consistently outperforms these $3$ baselines.** We report the full results in the PDF page attached to the rebuttals.


|                             	| | Size | | | d | | |$\bar{I}_{\mathbf{S}}$| | |$\bar{\ell}_{2}$| |
|---------------------------------|----------|----------|--------|----------|----------|--------|----------------|----------------|----------------|----------|----------|--------|
|                             	|$\rho_p$|$\rho_s$|$\tau$|$\rho_p$|$\rho_s$|$\tau$|$\rho_p$   	|$\rho_s$   	|$\tau$     	|$\rho_p$|$\rho_s$|$\tau$|
| Classification (12 datasets)  	| 0.46 	| 0.42 	| 0.32   | 0.52 	| 0.66 	| 0.55   |**0.92**|**0.88**|**0.73**| -0.79	| -0.85	| -0.66  |
| Retrieval (15 datasets)      	| 0.40 	| 0.39 	| 0.29   | 0.46 	| 0.63 	| 0.52   |**0.89**|**0.89**|**0.70**| -0.71	| -0.84	| -0.65  |
| Clustering (11 datasets)     	| 0.45 	| 0.38 	| 0.26   | 0.54 	| 0.67 	| 0.55   |**0.86**|**0.85**|**0.67**| -0.80	| -0.84	| -0.66  |
| STS (10 datasets)            	| 0.27 	| 0.35 	| 0.25   | 0.34 	| 0.66 	| 0.52   |**0.92**|**0.82**|**0.62**| -0.70	| -0.83	| -0.64  |
| Reranking (4 datasets)       	| 0.33 	| 0.33 	| 0.26   | 0.41 	| 0.61 	| 0.50   |**0.84**|**0.79**|**0.64**| -0.71	| -0.78	| -0.59  |
| Average (56 datasets)         	| 0.41 	| 0.41 	| 0.31   | 0.48 	| 0.62 	| 0.50   |**0.94**|**0.90**|**0.74**| -0.77	| -0.84	| -0.65  |
| Additional Classif (8 datasets) | 0.41 	| 0.62 	| 0.47   | 0.43 	| 0.64 	| 0.55   |**0.89**|**0.84**|**0.66**| -0.65	| -0.72	| -0.55  |


*Comparison with Baselines: Size of the Embedder, Dimension of the embedding output ($d$) and the $\ell_2$ reconstruction error of the embeddings for NLP datasets.*




|                             	| | Size | | | d | | |$\bar{I}_{\mathbf{S}}$| | |$\bar{\ell}_{2}$| |
|---------------------------------|----------|----------|--------|----------|----------|--------|----------------|----------------|----------------|----------|----------|--------|
|                             	|$\rho_p$|$\rho_s$|$\tau$|$\rho_p$|$\rho_s$|$\tau$|$\rho_p$   	|$\rho_s$   	|$\tau$     	|$\rho_p$|$\rho_s$|$\tau$|
| Absorption (8 datasets)   | -       	| -0.21   	| -0.16  | -       	| -0.43   	| -0.29  | -       	| **0.89** | **0.70** | -       	| **-0.89** | **-0.70** |
|Distribution (3 datasets) | -       	| -0.07   	| -0.03  | -       	| -0.46   	| -0.31  | -       	| **0.89** | **0.70** | -       	| -0.86      	| -0.66      	|
|Metabolism (8 datasets)   | -       	| 0.06    	| 0.03   | -       	| -0.46   	| -0.34  | -       	| **0.94** | **0.79** | -       	| -0.90      	| -0.71      	|
|Excretion (3 datasets)	| -       	| -0.17   	| -0.11  | -       	| -0.24   	| -0.18  | -       	| **0.77** | **0.60** | -       	| **-0.77** | -0.56      	|
|Toxicity (9 datasets) 	| -       	| 0.09    	| 0.06   | -       	| -0.49   	| -0.35  | -       	| **0.92** | **0.75** | -       	| -0.86      	| -0.67      	|
|ADMET (31 datasets)   	| -       	| -0.01   	| 0.01   | -       	| -0.47   	| -0.32  | -       	| **0.94** | **0.80** | -       	| -0.90      	| -0.72      	|


*Comparison with Baselines: Size of the Embedder, Dimension of the embedding output ($d$) and the $\ell_2$ reconstruction error of the embeddings for Molecular Modelling datasets.*

We hope that these clarifications and additional comparisons address the reviewers' concerns and positively influence their evaluation of our work.

Meta-review: This paper introduces a novel, information-theoretic framework for assessing the relevance of embedding models. The authors introduce the notion of sufficiency of a model A relatively to a model B (used to rank embedders) and tie this to a related notion of informativeness (information efficiency). 

Overall, the proposed approach is quite interesting & novel and a refreshing (theoretical) way to quantify the effectiveness of different embedding models. The authors supplement the theory with good analysis & combine it with experimental results to validate the efficacy of the proposed metric. They also did a very nice job of addressing comments/concerns during the rebuttal phase. All the reviewers agree that it’s a solid paper, no concerns.
Decision: Accept (poster)