Title: When is an Embedding Model  More Promising than Another?
Venue: NeurIPS 2024
Accepted: yes
Keywords: embedders

emeddings

molecules

nlp

llm

foundation models

evaluation

unsupervised

representation learning

information theory
Abstract: Embedders play a central role in machine learning, projecting any object into numerical representations that can, in turn, be leveraged to perform various downstream tasks. The evaluation of embedding models typically depends on domain-specific empirical approaches utilizing downstream tasks, primarily because of the lack of a standardized framework for comparison. However, acquiring adequately large and representative datasets for conducting these assessments is not always viable and can prove to be prohibitively expensive and time-consuming.  In this paper, we present a unified approach to evaluate embedders. First, we establish theoretical foundations for comparing embedding models, drawing upon the concepts of sufficiency and informativeness. We then leverage these concepts to devise a tractable comparison criterion (information sufficiency), leading to a task-agnostic and self-supervised ranking procedure. We demonstrate experimentally that our approach aligns closely with the capability of embedding models to facilitate various downstream tasks in both natural language processing and molecular biology. This effectively offers practitioners a valuable tool for prioritizing model trials.

When is an Embedder More Promising than Another?

Abstract

Embedders play a central role in machine learning, projecting any object into numerical representations that can, in turn, be leveraged to perform various downstream tasks. The evaluation of embedding models typically depends on domain-specific empirical approaches utilizing downstream tasks, primarily because of the lack of a standardized framework for comparison. However, acquiring adequately large and representative datasets for conducting these assessments is not always viable and can prove to be prohibitively expensive and time-consuming. In this paper, we present a unified approach to evaluate embedders. First, we establish theoretical foundations for comparing embedding models, drawing upon the concepts of sufficiency and informativeness. We then leverage these concepts to devise a tractable comparison criterion (information sufficiency), leading to a task-agnostic and self-supervised ranking procedure. We demonstrate experimentally that our approach aligns closely with the capability of embedding models to facilitate various downstream tasks in both natural language processing and molecular biology. This effectively offers practitioners a valuable tool for prioritizing model trials. 1

1 Introduction

Embeddings are a prominent tool in machine learning and are used in multiple fields, such as natural language processing [64, 83], computer vision [93, 59, 12, 53] or bioinformatics [67, 3, 23, 112]. These models embed objects such as images, texts, or molecules into numerical representations that can be used to perform numerous downstream tasks by preserving key features of the object [76, 111].

Depending on the data modalities, intended purpose, and available resources, embedders showcase a wide variety of architectures, training settings (unsupervised, supervised, self-supervised, etc.), objectives (masked language modeling, contrastive learning, etc.) [20, 100, 78, 112, 39], and datasets [65, 86, 31, 38, 5, 117]. And more recently, foundation models have become a natural starting point to create embedders [21, 106, 52, 73].

This diversity and variety of options makes selecting the most promising embedders for a data distribution challenging [75]. Most work evaluates embedders focusing on the performance they enable on a finite set of downstream tasks [85, 17, 90, 91, 81, 24]. Nevertheless, this evaluation process encounters two primary limitations. Firstly, it is not scalable concerning the number of embedders and tasks, as it requires fitting a downstream model for each task. Hence, prioritizing the evaluation of the most promising models becomes essential to mitigate computational costs. Secondly, acquiring high-quality labels can be a time-consuming and notably expensive endeavor in various applications. To overcome these limitations, in this paper, we explore task-agnostic evaluation metrics

for embedders relying solely on pairwise comparisons between embedders, i.e., without the need for labeled data in downstream tasks.

More specifically, our contributions can be summarized as follows:

An innovative theoretical framework for comparing embedding models: We cast the problem of ranking embedders into the noisy communication channels ordering (Sec. 2.2) and statistical experiments comparison settings (Sec. 2.3). We exploit the notions of sufficiency and informativeness and relax them, leveraging the concept of deficiency introduced by Le Cam [63] (Sec. 2.4), which is reframed to account for concepts and features. These concepts provide us with tools to establish an embedder ranking.

A practical relaxation: Estimating deficiency presents significant challenges. We propose the concept of information sufficiency (IS), which quantifies the information required to simulate one embedder from another (Sec. 3). We estimate the information efficiency to get a task-agnostic and label-free comparison tool for embedders evaluation.

Extensive experimental validation: The expected IS correlates with the ability of embedders to enable a wide range of downstream tasks. In NLP (Sec. 5) and molecular modeling (Sec. 6), our method respectively achieves Spearman ranking correlations of 0 . 90 ( 56 tasks) and 0 . 94 ( 31 tasks); providing an efficient model trial prioritization tool for practitioners.

1.1 Related works

Embedding evaluation. Embedding evaluation is mainly performed based on a limited set of downstream tasks [22, 91, 81, 24], for which the embeddings are used as inputs to smaller models. Therefore, embedders evaluation is field- and task-specific. In NLP, [41, 85] they rely on a limited set of tasks; more recently, the Massive Text Embedding Benchmark (MTEB) [75] followed this task-oriented trend and offered standardized test bed for embedders encompassing various downstream tasks in NP. Devising statistical tests to compare models and learning algorithms has a long history [30]. However, most works propose statistical tests relying on the performance of the downstream tasks of interests [60, 11]. Other works study the expressiveness of embedders and connect it to performance on downstream tasks [107, 25], but mostly focus on geometrical properties of the high dimensional representation in self-supervised learning settings [2, 42, 45].

Probing. While probing methods do not aim at comparing embedders, they evaluate their representations to discover what these models have learned. They train small models on the internal representations of large models to perform specific downstream tasks. These procedures allow researchers to assess what information is present and recoverable from these embeddings [10, 1, 88, 84]. Other work proposed measuring mutual information (MI) between internal representations and labels. It has been used to evaluate the difficulty of a dataset as the predictiveness of the labels using the features [35]. For instance, [97] evaluates the utility of representations in astrophysics to predict physical properties. Following this trend, [54] leverages the point-wise MI between Gaussian distributions to evaluate text-to-images and image-to-text generative models. However, none of these methods have focused on comparing embedders in the general case to the best of our knowledge.

2 Theoretical Foundations for Comparing Embedding Models

2.1 Background and notation

We assume that all considered spaces are standard Borel [28] Each such space U is equipped with its Borel σ -algebra B ( U ) . The set of all probability measures on U is denoted by P ( U ) The total variation distance between P and Q is denoted by ∥ P -Q ∥ TV. Given a joint probability measure P XY induced by two random variables X ∈ X and U ∈ U , the Mutual Information [27] is denoted by I ( X ; U ) . A Markov (or transition probability) kernel between X and U is a mapping P U | X : B ( U ) × X → [0 , 1] . The space of all such P U | X is denoted by K ( U | X ) and ( M ◦ P U | X )( V | x ) indicates the composition of Markov kernels M ∈ K ( V | U ) and P U | X ∈ K ( U | X ) . For further details, refer to Appendix A.

2.2 Sufficiency and informativeness ordering of embedding models

We aim to compare embedding models without relying on labeled data for downstream tasks. Let us consider two embedding models represented by their Markov kernels (or transition probabilities)

P U | X ∈ K ( U | X ) and P V | X ∈ K ( V | X ) , any target set Y of (discrete or continuous) concepts and feature space X with joint probability measure P Y X ∈ P ( Y × X ) induced by random variables ( Y, X ) ∈ Y × X , as illustrated in Figure 1. First, we study the question:

What sufficient conditions must be met by the embedding model U relative to V to guarantee that I ( Y ; U ) ⩾ I ( Y ; V ) for all distributions P Y X ?

V

(concepts)

(features)

ρ

V

(

V

)

≈

ρ

̂

V

(

̂

V

)

P

U

|

X

P

X

U

P

V

|

X

(embedder U)

(embedder V)

P

V

|

U

(embedder simulator)

(data distribution)

̂

V

P

Y

|

X

X

From an information-theoretic perspective [27], the quality of an embedding model can be likened to the capacity of a noisy communication channel with an uncoded input (e.g., a text, a molecule...), where a downstream task of interest is performed at the output (the embedding) of the channel. Let Y ∈ Y represent the message (the source) to be communicated over both channels; X represents the transmitted signal; and P U | X and P V | X the communication channels with outputs U and V , respectively. This process is illustrated in Figure 1. It naturally satisfies the Markov chain Y ↔ X ↔ ( U, V ) . A desirable property is that the embedding models U and V retain as much pertinent information as feasible to predict Y .

We shall be interested in the underlying information relationships between those embedding models that can be interpreted as channel U being "more informative" for communicating Y than channel V . The first attempt to introduce an ordering between communication channels appears in Shannon [94]. Körner and Marton later introduced [57] the concepts of "less noisy" (or more informative) and "degraded" (or sufficiency) orderings between channels.

Definition 1 (Sufficiency and informativeness orderings [57]) . Let P U | X and P V | X be two Markov kernels (embedding models).

Sufficiency U ≽ S V . The embedding model P U | X is said to be "sufficient" for the embedding model P U | X (or V to be degraded w.r.t. U ) if and only if there exists another Markov kernel M ∈ T ( V | U ) such that E ∥ M ◦ P U | X -P V | X ∥ TV = 0 , i.e. V can be simulated from U using M without information loss).

More informative U ≽ I V . The embedding model P U | X is said to be "more informative" (or less noisy) than P V | X if and only if the embedding models always satisfy the inequality

Proposition 1 (Relationships of sufficiency and information) . The following relationships hold:

(i) Sufficiency ⇒ informativeness. If the embedding model P U | X is sufficient for the embedding model P V | X , i.e. U ≽ S V , then U ≽ I V . However, Informativeness ⇏ sufficiency.

(ii) Informativeness ⇒ higher capacity to distinguish concepts. If the embedding model P U | X is more informative than embedding model P V | X , i.e. U ≽ I V , then

for any pair of concepts ( y 0 , y 1 ) ∈ Y × Y and all probability distributions P Y X .

Remark 1 . An immediate consequence of claim (i) is that the sufficient condition between embedding models implies that the embedding model U is more informative than the embedding model V relative to all target concepts in Y over all possible data distributions: I ( Y ; U ) ⩾ I ( Y ; V ) , for all probability distributions P Y X .

Although U being more informative than V does not necessarily imply U ≽ S V [57, 66]; (ii) states that being more informative ensures a higher statistical discrimination capacity between any pairs of target concepts (for further discussion, see Sec. B.2).

Motivated by the concepts of sufficiency and informativeness between embedding models, we can inquire about their statistical consequences for a learner conducting an inference task on these embeddings. More precisely, given a finite set of concepts Y , if U ≽ S V , is the Bayes risk expected to be smaller when the inference is based on U than when it is based on V ?

<latexi sh

1_b

64="nc

T

uR

JmKB

+

>A

CzH

V

L

F

3U

I

d

E

k

Y

o7

S

y

w

p2

rW

X

/0

q

Of

Pv

g

9

5

N

G

Q

<latexi sh

1_b

64="or pQv7qNz

5d2

G

8

X/y

c

>A

C

n

jVHLS

FD

f

Z

ug

3

K

UE

J0mk

T

B

W

w

9

Y

R

M

I

+

O

P

<latexi sh

1_b

64="

MmS8

L7G

ZCE

cD9

V

5Y

>A

z3

j

H

T

J

F

U

+I

d

k

u

R

g

Q

0NBX2q

f

/p

v

O

P

n

r

K

o

w

W

y

<latexi sh

1_b

64="

PYuBf

K

3I5

dJo

M

Z

vW

>A

CzH

cjV

LT

FD

U

+

m

E

k

pX

R

G

S

8

/

2

7

y9

N

q

r

0

g

O

Q

n

w

<latexi sh

1_b

64="EXU9fz5

w

WNRZ8

>A

H

cjV

LT

J

FD3

+I

d

m

Y

7

Su

Q

M

O

0

q

p

/

G

2

y

r

K

B

P

<latexi sh

1_b

64="M

KvUO+k3

Dr

E/m2

w

J

>A

Cz cjVH

T

F

I

d

u

R

S

g

Q

0NBX

q

G

f8

p

P

n

7

5

Y

Z

9

W

y

<latexi sh

1_b

64="yY3A

uT

7

Bv dUDScG

/

50o

>

Cz jVHL

J

F

+I

m

E

kZ

n

K

MN

R

f

q

P2

r

W

w

O8X

p

g

9

Q

<latexi sh

1_b

64="NIQB

C2HG

S

+g7y

/Pf

5M

>A

D

cjV

L

F

W

0r

k

m

q

Z

J

T

u

z

Y

3E

v

R

Kp

w

U

9

8n

X

d

O

o

<latexi sh

1_b

64="YyK

DH

2+

dUTNnLvS

W

8o

>A

Cz cjV

J

F

3

I

m

E

k

7

u

Q

M

O

0

p

f

/

R

G

5

9

r

B

g

P

w

<latexi sh

1_b

64="r

N0Wk2g

Z3G

H

5njQ

UI

>A

C

c

V

LS

FD

7/q

m

B

v

J

K

p

8

E

d+

yP

f

Y

M

O

R

z

X

T

o

9

2.3 Comparing statistical experiments with embedding Models

The pursuit of comparing statistical experiments originated from the seminal paper by Bohnenblust, Shapley, and Sherman [16], followed by subsequent contributions by Blackwell [13, 14]. They formally established the relationships between sufficiency (Def. 1) and inference procedures.

In our framework, a statistical experiment [13] consists of a mathematical abstraction (see Appendix A for further details) intended to represent a downstream task where a learner aims at inferring a concept y ∈ Y from the embeddings U or V . Deciding what embedder should be used to perform a given task is too general. In this work, we do not take into account the computational cost or the size of an embedder and solely focus on the following question:

What are the necessary and sufficient conditions that ensure that employing the embedding U for any task P Y X leads to lower risk compared to using the embedding V ?

Drawing parallels with the theoretical framework established for comparing statistical experiments, a relationship can be derived between the concept of sufficiency and the expected risk for a specific task (see Sec. B.5 for further discussion).

We concentrate on the scenario where Y consists of a finite number of concepts (e.g., classification tasks), as it is a significant case in its own right [104] and provide fundamental insights for the present work. The next Proposition states an important relation between the concept of sufficiency and the expected Bayes risk on any classification task.

Proposition 2 (Comparison of embedding models through Bayes risks) . Given two embedding models P U | X ∈ K ( U | X ) and P V | X ∈ K ( V | X ) , the following statements are equivalent:

(i) The embedding model P U | X is sufficient relative to P V | X , i.e. U ≽ S V .

(ii) For all conditional probability measures P Y | X on finite alphabet Y , the Bayes risks satisfy

̸

̸

where ˆ Y U and ˆ Y V are distributed according to ρ U ( U ) and ρ V ( V ) , respectively.

Remark 2 . In other words, if we can fully simulate an embedder V from another embedder U , the expected risk across all potential classification tasks cannot be greater when using U compared to V . The proof of this Proposition is given in Sec. B.3. It is worth mentioning that various versions of this result are available in the literature [104]. However, our extension here, in a simpler setting, incorporates concepts and features into the experiment comparison framework.

2.4 Challenges in ranking embedding models and their deficiency

According to the notion of "sufficiency", we can distinguish the three following possibilities:

Equivalence: U ≽ S V and V ≽ S U denoted U ≈ V ; U and V can simulate each other.

Comparability: U ≽ S V but V ≽ S U only V can be simulated from U .

Non-comparability: U ≽ S V and V ≽ S U , neither U nor V can simulate each other.

̸

̸

̸

Our results up to now only account for the two first possibilities. However, two embedders are generally not comparable (Sec. B.4). This issue was addressed by Le Cam [63], who introduced the notion of "deficiency".

Definition 2. The deficiency δ ( P U | X → P V | X ) of P V | X relative to P U | X is defined as [63]

where the infimum is taken over all Markov kernels (or transition probabilities) M ∈ K ( V | U ) , mapping stochastically U and V , and δ measures error between the simulated and true embedders.

δ indicates how well one model can be reconstructed from the other , it induces a natural relaxation of the sufficiency where the reconstruction does not have to be perfect 2 for us to obtain guarantees on

̸

the downstream tasks performance (See Corollary 1). It avoids the non-comparability problem by evaluating "how much information" we lose when passing from one model to the other one .

Le Cam [63] showed that, for a given task Y , the deficiency δ ( P U | Y → P V | Y ) is directly related to the expected Bayes risks on the task (see Sec. B.6). We extend this result to the comparison of two embedding models P U | X and P V | X in a task-agnostic manner and build the relation to the expected Bayes risks for any classification task Y .

Corollary 1. Given two embedding models P U | X and P V | X satisfying:

(i) The deficiency δ ( P U | X → P V | X ) ⩽ γ .

(ii) For any conditional distribution P Y | X on finite alphabets Y ,

̸

̸

Statement (ii) implies (i) provided that γ ⩾ 2 | Y | ε and conversely, (i) implies (ii) provided that γ ⩽ ε .

The proof of this Corollary is relegated to Sec. B.3.

Remark 3 . In particular, we can infer that for any classification task Y , the expected Bayes risk of the embedding model U , denoted by R U , is upper bounded by the expected Bayes risk of the embedding model V , denoted by R V :

and similarly, |R U - R V | ⩽ max { δ ( P U | X → P V | X ) , δ ( P V | X → P U | X ) } , for all conditional distributions P Y | X . If both deficiencies are small, the resulting expected Bayes risks of the embedding models U and V will be close to each other for any target task Y .

3 Quantifying Information Sufficiency Between Embedding Models

We want to compare embedding models using the concept of deficiency, leveraging Prop. 2 and Corollary 1. These propositions suggest that the performance on any classification task of an embedding model U relative to the model V is bounded by δ ( P U | X → P V | X ) . However, estimating the deficiency from data samples is notably challenging [95], and while upper bounds derivation exists, they do not necessarily make it tractable.

3.1 Estimating Information Sufficiency

The deficiency δ ( P U | X → P V | X ) between two embedding models P U | X and P V | X , measures how well U can be used to simulate V using a Markov kernel M ∈ K ( V | U ) . This section aims to build a tractable proxy for this reconstruction cost. To this end, we estimate how much we can reduce the uncertainty about Z by observing U by learning an appropriate Markov kernel. This corresponds to the information sufficiency [29, 4] and can be interpreted as the information-theoretic counterpart of the deficiency. The information deficiency between U and V is then defined as:

Definition 3 (Information sufficiency) . The information sufficiency I S ( U → V ) , relative to parametric classes of distributions F Θ ( V ) and K Θ ( V | U ) (multivariate Gaussian mixtures [82]) is defined:

Remark 4 . When the information sufficiency I S ( U → V ) is large, it signifies that U offers a substantial amount of information to simulate V , a proxy for a small deficiency. Conversely, when I S ( U → V ) is lower, it implies that the channel P V | Y is subject to considerable noise or randomness, leading to a greater loss of statistical information.

We hence attempt to simulate V from U by learning a Markov kernel M ∈ K Θ ( V | U ) , via a mixture of multivariate Gaussians, and measure the uncertainty reduction it induces.

Model Z

Pairwise embedder evaluation. For set of embbeders ( Z k ) k represented by their Markov kernels { P Z k | X } k , we compute the pairwise information sufficiency I S ( Z k → Z l ) . The pairwise information sufficiency matrix defines the adjacency matrix of a directed graph of embedders (Figure 2). Corollary 1 shows that embedders sharing high information sufficiency are expected to perform similarly on any downstream tasks, motivating the identification of communities in the graph. While the graph construction is in O ( N 2 ) ; where N is the number of embedders, it is in practice tractable for a reasonable number of embedders (refer to Sec. E.6) for more details).

̸

Practical embedding evaluation. We construct the set of all information sufficiency using Z k : S I S ( k ) = {I S ( Z k → Z l ) } l = k . We build our information sufficiency score ( I S score) by taking the median of S I S ( k ) . Details on the I S score's estimation can be found in Sec. E.1.

4 Experimental Setup

We aim to evaluate the practical utility of the I S score to rank and select the best embedders for a given data distribution. We compare this ranking to those obtained on various downstream tasks. Our experimental protocol is divided into three main steps:

We evaluate the I S score of the models by identifying a large and diverse dataset that is supposed to be representative of the data distribution of interest.

We train a small feedforward neural network ( ρ Z k ) per embedder P Z k | X to perform each downstream task and record its performances ( R 2 score for regression, AUROC/accuracy for binary/multiclass classification).

We compare the models' performances on the downstream tasks and the I S score by measuring three types of correlations: the Pearson correlation, the Spearman correlation, and the Kendall-Tau coefficient. 3 (See Sec. E.5 for additional baselines).

Average (56 datasets)

0

5

10

15

20

25

30

Downstream tasks (rank

)

0

10

20

30

IS

(rank

)

UAE-Large-V1

e5-small

udever-bloom-560m

allenai-specter

p

: 0.94

s

: 0.90

: 0.74

(c) NLP

(b) Molecular Modelling

6

9

12

15

18

21

Downstream tasks  (

rank

)

0

5

10

15

20

25

S

(

rank

)

ContextPred

FRAD_QM9

GraphMVP

MolR_tag

Not-trained

s

: 0.94

: 0.80

ADMET

(d) Molecular Modeling

5 Text Embeddings Evaluation

5.1 Experimental setting

Embedders & Datasets. We compared 34 models with different training objectives, training datasets, and architectures. We included embedders derived from modern LLM such as LLaMA [106], Mistral [52], Gemma [102], Croissant [37] and T5 encoders [77]; common embedders derived from BERT architectures [31, 38, 85] or RobERTa [41] and embedders trained on specific embeddings objectives such Angle [64], Stella 4 , E5 models [113], LaBSE [38]. A comprehensive list of the models can be found in Sec. C.1, Tab. 1 with their main characteristics and links to the Huggingface Hub for reproducibility. We used them to extract embeddings for many different datasets from the MTEB benchmark such as Banking77 [19], Sickr [122], Amazon polarity [72], SNLI [120] and IMDB [70]. We provide the datasets statistics in Sec. C.1, Tab. 2.

Downstream tasks evaluation. We rely on the results released on the MTEB leaderboard 5 and compare our rankings to the rankings and scores obtained by the different models on the different tasks. We evaluate additional tasks that are not included in the MTEB benchmark, such as tweet_eval [8, 74, 7, 109, 9], DAIR Emotion [92], agnews topic classification [123], Clinc intent detection [62] PAWS-X [118] and Rotten Tomatoes [79].

5.2 Model's Information Sufficiency analysis

(a)

5

10

15

20

25

30

35

Downstream tasks (rank

)

0

5

10

15

20

25

30

35

IS

(rank

)

p

: 0.83

s

: 0.77

: 0.57

Llama-2-7b-hf

SFR-Embedding

UAE-Large-V1

allenai-specter

average_word

e5-large-v2

e5-small

gemma-7b-it

gtr-t5-large

gtr-t5-xl

sf_model_e5

(b)

gemma-2b

gemma-2b-it

gemma-7b

gemma-7b-it

0.45

0.50

0.55

Information Sufficiency

gemma-2b

gemma-2b-it

gemma-7b

gemma-7b-it

0.6

0.7

Classification accuracy

(c)

Correlation with downstream tasks performance. The MTEB Benchmark offers a natural starting point to compare models' ranking according to their performance on downstream tasks and their I S score. In Figure 3c, we show that the I S score of an embedder correlates positively with its performance on a wide range of downstream tasks, from classification and similarity tasks to retrieval and clustering tasks. Overall, our I S score correlates strongly with MTEB's average score (Spearman correlation of 0 . 90 and a Pearson correlation of 0 . 94 , see Figure 3c) and with the subtask

performance Figure 3a). We extended our experiments to a more extensive set of models not included in the MTEB benchmark and observed a similar trend (Figure 4b). Per-datasets results are reported in Sec. C.3.1 and ablations in Sec. C.3.2. All our results show that our estimation of the information sufficiency between models is a good proxy for the performance of the models on a wide range of tasks.

Embedder communities. The pair-wise information sufficiency evaluation between the models can be used to cluster them into communities [15](Figure 4a, Figure 2) 6 . We observe that the extracted clusters group together models that are similar in their training objectives and architectures. LLM-based models such as LLaMA, Mistral, Gemma, and Croissant are clustered together, while BERT-based models share another cluster. Similarly, models trained specifically for embedding purposes, such as UAE-Large-V1 and ember-v1, are grouped together. This suggests that the ordering induced by information sufficiency is meaningful and can be used to identify models with similar properties and behaviors. Consistently with Corollary 1, we observe that the performance of the models on the downstream tasks is similar within the same cluster (Figure C.3.5). In addition, we found that it captures improvements by both steps of pretraining and instruction fine-tuning (Figure 4c, Sec. C.3.2)

6 Molecular Modeling

6.1 Experimental setting

Embedders. To process molecular data, embedders can leverage different representations of the molecules, providing an interesting benchmark to evaluate the I S score. We evaluated models derived from the molecular representation learning literature, summed up in Sec. D.1. We considered various input modalities such as string representations (SMILES [114], SELFIES [58]), 2D-graphs by using graph neural networks (GNNs), and 3D-representations (using the TorchMD-net architecture [80]). We added a randomly initialized baseline GNN model that was not trained on any dataset.

Datasets. To evaluate the information sufficiency between embedders, we compared the models on the ZINC 250k dataset[50], designed to gather compounds that could be relevant to a wide range of therapeutic projects. This dataset contains 250k commercially available compounds meant to be used in diverse therapeutic projects.

Downstream tasks. We evaluated the embedders on 31 downstream tasks extracted from the Therapeutic Data Commons [49] platform. This section focuses on ADMET tasks (Absorption, Distribution, Metabolism, Excretion, and Toxicity). Results on Drug-Target interaction tasks can be found in Sec. D.4. Datasets collected are split into a training, validation, and test set, following the scaffold-split strategy, further described in see Sec. D.3.

6.2 Model's Information Sufficiency analysis

Global results. The I S score ranking is consistent with the results of the embedders on the ADMET downstream tasks, achieving a Spearman correlation of 0.95 and a Kendall-tau coefficient of 0.80, as reported in Figure 3d. Detailed results for each of the 31 tasks are available in Sec. D.3 in Tab. 6. Table 3b shows the correlation between the I S score rankings and the performances obtained on the ADMET tasks within each category. High correlations are achieved within most task categories, especially when large tasks are available (containing an important number of molecules). On excretion tasks, the correlation is lower (below 0 . 8 ), which can be explained by the fact that these tasks are the most challenging regression tasks available, where the fine-tuned models reach the lowest R 2 scores between 0 and 0 . 2 (see Sec. D.3).

Most / Least promising models. We observe in Figure 5b that the most promising models are the (Chem) Bert-MTR models[3] 7 and MolR[112], the former trained on SMILES representations to predict a variety of computationally available molecular properties, and the latter trained on 2D graphs to preserve equivalence of molecules w.r.t chemical reactions. Surprisingly, these models share high predictive mutual information (being assigned to the same Louvain community in Figure 5a),

(a)

10

20

ADMET rank

ContextPred

GPT-GNN

MoleOOD_SAGE

BertMLM-5M

InfoGraph

GraphMVP

3D-denoising

3D-Infomax

GPT-1.2B

BertMTR-5M

BertMTR-77M

AttributeMask

MoleOOD_GIN

BertMLM-77M

MoleOOD_GCN

Not-trained

BertMLM-10M

GROVER

GraphCL

GraphLog

FRAD_QM9

MolR_gcn

MolR_gat

BertMTR-10M

MolR_tag

(b)

suggesting that they capture similar information despite significant differences in their training methods. These models also appear to be the most competitive on the ADMET tasks. On the other hand, and consistently with Sun et al. [99]'s observation, training methods for 2D-GNNs such as following attribute masking and context prediction objective are deemed as the least informative according to the I S score. This is explained by the simplicity of these pretraining objectives for this data modality. These methods are also among the least competitive methods on the ADMET downstream tasks.

NLP-inspired models. (Chem) Bert-MLM [3], MolBert [36] and (Chem) GPT[40] leverage masked language model objective applied to string representations (SMILES and SELFIES). Unsurprisingly, as seen in Figure 5a, these models are clustered, suggesting they capture similar information. However, they fail to simulate other models in the pool, resulting in low I S scores, a result consistent with the known limitations of these pretraining objectives [23, 105]. A noticeable exception is (Chem) GPT-1.2B (the biggest model of the pool by far), which displays a significantly higher I S score.

"Not-trained" GNN. Figure 5b helps visualize the performances of the different models relative to our baseline "Not-trained" GNN. Surprisingly, some models are ranked less promising than this baseline by the I S score. However, all of these less promising models obtain poorer performances on the downstream tasks. Similarly, except for InfoGraph [98], every model ranked more promising than the "Not-trained" GNN baseline and obtained better results on ADMET tasks. This surprising result validates evaluation of the I S score w.r.t this baseline.

7 Limitations and Conclusions

We proposed a principled approach to embedding model evaluation by framing model ranking as a variation of comparing statistical experiments. Utilizing concepts of sufficiency, informativeness, and deficiency, we developed mathematically grounded metrics for pairwise comparisons between embedders without relying on labeled data in downstream tasks. Our tractable relaxation, termed information sufficiency, demonstrated strong correlations with rankings based on downstream task performance in extensive experiments. Although successful, our method still has at least two primary

limitations. First, its effectiveness depends on the number and diversity of available embedders (see Sec. E.4). Future work could explore using randomly initialized embedders (random projections) instead of pre-trained ones. Second, we can enhance our proxy for predicting the deficiency between models by exploring better methods (e.g., estimating the f -divergence) to directly learn the Markov kernel that minimizes the total variation distance, which we leave for future research.

Acknowledgments

This work was granted access to the HPC resources of IDRIS under the allocation 2023AD011013290R2 made by GENCI, and enabled by support provided by Calcul Quebec and the Digital Research Alliance of Canada. We warmly thank Heitor Rapela, Banafsheh Karimian, and Eric Aubinais for their advice and comments about our work. We also owe a special highlight to Loïc Fosse for the many discussions and hindsights he provided and for the subsequent follow-up projects.

References

Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In International Conference on Learning Representations . International Conference on Learning Representations, ICLR, 2017.

Kumar K Agrawal, Arnab Kumar Mondal, Arna Ghosh, and Blake Richards. \alpha-req : Assessing representation quality in self-supervised learning by measuring eigenspectrum decay. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 17626-17638. Curran Associates, Inc., 2022.

Walid Ahmad, Elana Simon, Seyone Chithrananda, Gabriel Grand, and Bharath Ramsundar. Chemberta-2: Towards chemical foundation models, 2022.

Suguru Arimoto. Information-theoretical considerations on estimation problems. Information and control , 19(3):181-194, 1971.

Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture, 2023.

Simon Axelrod and Rafael Gómez-Bombarelli. Geom, energy-annotated molecular conformations for property prediction and molecular generation. Scientific Data , 9(1):185, 2022.

Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa-Anke, and Leonardo Neves. TweetEval:Unified Benchmark and Comparative Evaluation for Tweet Classification. In Proceedings of Findings of EMNLP , 2020.

Francesco Barbieri, Jose Camacho-Collados, Francesco Ronzano, Luis Espinosa-Anke, Miguel Ballesteros, Valerio Basile, Viviana Patti, and Horacio Saggion. Semeval 2018 task 2: Multilingual emoji prediction. In Proceedings of The 12th International Workshop on Semantic Evaluation , pages 24-33, 2018.

Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women in Twitter. In Proceedings of the 13th International Workshop on Semantic Evaluation , pages 54-63, Minneapolis, Minnesota, USA, 2019. Association for Computational Linguistics.

Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics , 48(1):207-219, 2022.

Alessio Benavoli, Giorgio Corani, Janez Demsar, and Marco Zaffalon. Time for a change: a tutorial for comparing multiple classifiers through bayesian analysis, 2017.

Usha Bhalla, Alex Oesterling, Suraj Srinivas, Flavio P. Calmon, and Himabindu Lakkaraju. Interpreting clip with sparse linear concept embeddings (splice), 2024.

David Blackwell. Comparison of experiments. In Proceedings of the second Berkeley symposium on mathematical statistics and probability , volume 2, pages 93-103. University of California Press, 1951.

David Blackwell. Equivalent comparisons of experiments. The Annals of Mathematical Statistics , 24(2):265-272, 1953.

Vincent D Blondel, Jean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre. Fast unfolding of communities in large networks. Journal of Statistical Mechanics: Theory and Experiment , 2008(10):P10008, oct 2008.

H. Frederic Bohnenblust, Lloyd S. Shapley, and Seymour Sherman. Reconnaissance in game theory. 1949.

Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. Improving language models by retrieving from trillions of tokens, 2022.

Nathan Brown, Marco Fiscato, Marwin H.S. Segler, and Alain C. Vaucher. Guacamol: Benchmarking models for de novo molecular design. Journal of Chemical Information and Modeling , 59(3):1096-1108, March 2019.

Iñigo Casanueva, Tadas Temcinas, Daniela Gerz, Matthew Henderson, and Ivan Vulic. Efficient intent detection with dual sentence encoders. In Proceedings of the 2nd Workshop on NLP for ConvAI - ACL 2020 , mar 2020. Data available at https://github.com/PolyAI-LDN/taskspecific-datasets.

Wei-Cheng Chang, Felix X. Yu, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar. Pre-training tasks for embedding-based large-scale retrieval, 2020.

Chang Che, Qunwei Lin, Xinyu Zhao, Jiaxin Huang, and Liqiang Yu. Enhancing multimodal understanding with clip-based image-to-text transformation, 2024.

Yanqing Chen, Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. The expressive power of word embeddings, 2013.

Seyone Chithrananda, Gabriel Grand, and Bharath Ramsundar. Chemberta: Large-scale self-supervised pretraining for molecular property prediction, 2020.

Hyunjin Choi, Judong Kim, Seongho Joe, and Youngjune Gwon. Evaluation of bert and albert sentence embedding performance on downstream nlp tasks, 2021.

Ching-Yao Chuang, Antonio Torralba, and Stefanie Jegelka. The role of embedding complexity in domain-invariant representations, 2019.

Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veliˇ ckovi´ c. Principal neighbourhood aggregation for graph nets. In Advances in Neural Information Processing Systems , 2020.

T. M. Cover and J. A. Thomas. Elements of Information Theory . Wiley, New York, NY, 2nd edition, 2006.

H. Crauel. Random Probability Measures on Polish Spaces . Taylor & Francis, 2002.

Morris H DeGroot. Uncertainty, information, and sequential experiments. The Annals of Mathematical Statistics , 33(2):404-419, 1962.

Janez Demšar. Statistical comparisons of classifiers over multiple data sets. The Journal of Machine learning research , 7:1-30, 2006.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.

Jian Du, Shanghang Zhang, Guanhang Wu, Jose M. F. Moura, and Soummya Kar. Topology adaptive graph convolutional networks, 2018.

David Duvenaud, Dougal Maclaurin, Jorge Aguilera-Iparraguirre, Rafael Gómez-Bombarelli, Timothy Hirzel, Alán Aspuru-Guzik, and Ryan P. Adams. Convolutional networks on graphs for learning molecular fingerprints, 2015.

Peter Ertl and Ansgar Schuffenhauer. Estimation of synthetic accessibility score of drug-like molecules based on molecular complexity and fragment contributions. J. Cheminformatics , 1:8, 2009.

Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V -usable information, 2022.

Benedek Fabian, Thomas Edlich, Héléna Gaspar, Marwin Segler, Joshua Meyers, Marco Fiscato, and Mohamed Ahmed. Molecular representation learning with language models and domain-relevant auxiliary tasks, 2020.

Manuel Faysse, Patrick Fernandes, Nuno M. Guerreiro, António Loison, Duarte M. Alves, Caio Corro, Nicolas Boizard, João Alves, Ricardo Rei, Pedro H. Martins, Antoni Bigata Casademunt, François Yvon, André F. T. Martins, Gautier Viaud, Céline Hudelot, and Pierre Colombo. Croissantllm: A truly bilingual french-english language model, 2024.

Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. Languageagnostic bert sentence embedding, 2022.

Shikun Feng, Yuyan Ni, Yanyan Lan, Zhi-Ming Ma, and Wei-Ying Ma. Fractional denoising for 3D molecular pre-training. In Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 9938-9961. PMLR, 23-29 Jul 2023.

Nathan C. Frey, Ryan Soklaski, Simon Axelrod, Siddharth Samsi, Rafael Gómez-Bombarelli, Connor W. Coley, and Vijay Gadepally. Neural scaling of deep chemical models. Nature Machine Intelligence , 5(11):1297-1305, November 2023. Publisher: Nature Publishing Group.

Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings, 2022.

Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann Lecun. Rankme: Assessing the downstream performance of pretrained self-supervised representations by their rank, 2023.

Aric Hagberg, Pieter Swart, and Daniel S Chult. Exploring network structure, dynamics, and function using networkx. Technical report, Los Alamos National Lab.(LANL), Los Alamos, NM(United States), 2008.

William L. Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs, 2018.

Bobby He and Mete Ozay. Exploring the gap between collapsed; whitened features in selfsupervised learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 8613-8634. PMLR, 17-23 Jul 2022.

Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovec. Ogb-lsc: A large-scale challenge for machine learning on graphs, 2021.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXiv:2005.00687 , 2020.

Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. Gpt-gnn: Generative pre-training of graph neural networks. In Proceedings of the 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 2020.

Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovec, Connor W Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik. Therapeutics data commons: Machine learning datasets and tasks for drug discovery and development. Proceedings of Neural Information Processing Systems, NeurIPS Datasets and Benchmarks , 2021.

John J. Irwin and Brian K. Shoichet. ZINC - A Free Database of Commercially Available Compounds for Virtual Screening. Journal of chemical information and modeling , 45(1):177182, 2005.

Clemens Isert, Kenneth Atz, José Jiménez-Luna, and Gisbert Schneider. Qmugs: Quantum mechanical properties of drug-like molecules, 2021.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.

Apoorv Khandelwal, Luca Weihs, Roozbeh Mottaghi, and Aniruddha Kembhavi. Simple but effective: Clip embeddings for embodied ai, 2022.

Jin-Hwa Kim, Yunji Kim, Jiyoung Lee, Kang Min Yoo, and Sang-Woo Lee. Mutual information divergence: A unified metric for multimodal generative models, 2022.

Sunghwan Kim, Jie Chen, Tiejun Cheng, Asta Gindulyte, Jia He, Siqian He, Qingliang Li, Benjamin A Shoemaker, Paul A Thiessen, Bo Yu, Leonid Zaslavsky, Jian Zhang, and Evan E Bolton. PubChem 2023 update. Nucleic Acids Research , 51(D1):D1373-D1380, 10 2022.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.

J. Korner and K. Marton. Comparison of two noisy channels, 1977.

Mario Krenn, Florian Häse, AkshatKumar Nigam, Pascal Friederich, and Alan Aspuru-Guzik. Self-referencing embedded strings (selfies): A 100 Machine Learning: Science and Technology , 1(4):045024, October 2020.

Yugo Kubota, Daichi Haraguchi, and Seiichi Uchida. Impression-clip: Contrastive shapeimpression embedding for fonts, 2024.

Alexandre Lacoste, François Laviolette, and Mario Marchand. Bayesian comparison of machine learning algorithms on single and multiple datasets. In Artificial Intelligence and Statistics , pages 665-675. PMLR, 2012.

Greg Landrum, Paolo Tosco, Brian Kelley, sriniker, gedeck, NadineSchneider, Riccardo Vianello, Ric, Andrew Dalke, Brian Cole, AlexanderSavelyev, Matt Swain, Samo Turk, Dan N, Alain Vaucher, Eisuke Kawashima, Maciej Wójcikowski, Daniel Probst, guillaume godin, David Cosgrove, Axel Pahl, JP, Francois Berenger, strets123, JLVarjo, Noel O'Boyle, Patrick Fuller, Jan Holst Jensen, Gianluca Sforna, and DoliathGavid. rdkit/rdkit: 2020_03_1 (Q1 2020) Release, March 2020.

Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. An evaluation dataset for intent classification and out-of-scope prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , 2019.

L Le. Sufficiency and approximate sufficiency. The Annals of Mathematical Statistics , pages 1419-1455, 1964.

Xianming Li and Jing Li. Angle-optimized text embeddings, 2023.

Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281 , 2023.

Dennis V Lindley. On a measure of the information provided by an experiment. The Annals of Mathematical Statistics , 27(4):986-1005, 1956.

Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. Pre-training molecular graph representation with 3d geometry. In International Conference on Learning Representations , 2022.

Tiqing Liu, Yuhmei Lin, Xin Wen, Robert N. Jorissen, and Michael K. Gilson. Bindingdb: a web-accessible database of experimentally determined protein-ligand binding affinities. Nucleic Acids Research , 35:D198-D201, 12 2006.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019.

Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , pages 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics.

Hadrien Mary, Emmanuel Noutahi, DomInvivo, Lu Zhu, Michel Moreau, Steven Pak, Desmond Gilmour, Shawn Whitfield, t, Valence-JonnyHsu, Honoré Hounwanou, Ishan Kumar, Saurav Maheshkar, Shuya Nakata, Kyle M. Kovary, Cas Wognum, Michael Craig, and DeepSource Bot. datamol-io/datamol: 0.12.3, January 2024.

Julian McAuley and Jure Leskovec. Hidden factors and hidden topics: understanding rating dimensions with review text. In Proceedings of the 7th ACM Conference on Recommender Systems , RecSys '13, page 165-172, New York, NY, USA, 2013. Association for Computing Machinery.

Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. Sfr-embedding-mistral:enhance text retrieval with transfer learning. Salesforce AI Research Blog, 2024.

Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. Semeval-2018 task 1: Affect in tweets. In Proceedings of the 12th international workshop on semantic evaluation , pages 1-17, 2018.

Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. Mteb: Massive text embedding benchmark, 2023.

Kevin P. Murphy. Machine learning : a probabilistic perspective . MIT Press, Cambridge, Mass. [u.a.], 2013.

Jianmo Ni, Gustavo Hernández Ábrego, Noah Constant, Ji Ma, Keith B. Hall, Daniel Cer, and Yinfei Yang. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models, 2021.

Zhihong Pan, Xin Zhou, and Hao Tian. Extreme generative image compression by learning text embedding from diffusion models. arXiv preprint arXiv:2211.07793 , 2022.

Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the ACL , 2005.

Raul P. Pelaez, Guillem Simeon, Raimondas Galvelis, Antonio Mirarchi, Peter Eastman, Stefan Doerr, Philipp Thölke, Thomas E. Markland, and Gianni De Fabritiis. Torchmd-net 2.0: Fast neural network potentials for molecular simulations, 2024.

Christian S. Perone, Roberto Silveira, and Thomas S. Paula. Evaluation of sentence embeddings in downstream and linguistic probing tasks, 2018.

Georg Pichler, Pierre Colombo, Malik Boudiaf, Günther Koliander, and Pablo Piantanida. A differential entropy estimator for training neural networks, 2022.

Tiago Pimentel, Clara Meister, and Ryan Cotterell. On the usefulness of embeddings, clusters and strings for text generator evaluation, 2023.

Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. Information-theoretic probing for linguistic structure. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 4609-4622, 2020.

Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bertnetworks, 2019.

Nils Reimers and Iryna Gurevych. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, 11 2020.

R. Tyrrell Rockafellar. Convex analysis . Princeton Mathematical Series. Princeton University Press, Princeton, N. J., 1970.

Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in bertology: What we know about how bert works. Transactions of the Association for Computational Linguistics , 8:842866, 2021.

Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. Advances in Neural Information Processing Systems , 33, 2020.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-toimage diffusion models with deep language understanding, 2022.

Joaquim Santos, Bernardo Consoli, and Renata Vieira. Word embedding evaluation in downstream tasks and semantic analogies. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Twelfth Language Resources and Evaluation Conference , pages 4828-4834, Marseille, France, May 2020. European Language Resources Association.

Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. CARER: Contextualized affect representations for emotion recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 3687-3697, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.

Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 , 2021.

Claude E. Shannon. A note on a partial ordering for communication channels. Information and Control , 1(4):390-397, 1958.

A.N. Shiri?aev and V.G. Spokoiny. Statistical Experiments and Decisions: Asymptotic Theory . Advanced series on statistical science & applied probability. World Scientific, 2000.

Hannes Stärk, Dominique Beaini, Gabriele Corso, Prudencio Tossou, Christian Dallago, Stephan Günnemann, and Pietro Liò. 3d infomax improves gnns for molecular property prediction. arXiv preprint arXiv:2110.04126 , 2021.

Ce Sui, Xiaosheng Zhao, Tao Jing, and Yi Mao. Evaluating summary statistics with mutual information for cosmological inference, 2023.

Fan-Yun Sun, Jordan Hoffman, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. In International Conference on Learning Representations , 2019.

Ruoxi Sun, Hanjun Dai, and Adams Wei Yu. Does gnn pretraining help molecular representation? In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Duyu Tang, Furu Wei, Bing Qin, Nan Yang, Ting Liu, and Ming Zhou. Sentiment embeddings with applications to sentiment analysis. IEEE Transactions on Knowledge and Data Engineering , 28(2):496-509, 2016.

Jing Tang, Agnieszka Szwajda, Sushil Shakyawar, Tao Xu, Petteri Hintsanen, Krister Wennerberg, and Tero Aittokallio. Making sense of large-scale kinase inhibitor bioactivity data sets: A comparative and integrative analysis. Journal of Chemical Information and Modeling , 54(3):735-743, 2014.

Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. ChoquetteChoo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christian Muraru, Grigory Rozhdestvenskiy, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikuła, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross McIlroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. Gemma: Open models based on gemini research and technology, 2024.

Erik Torgersen. Comparison of Statistical Experiments . Encyclopedia of Mathematics and its Applications. Cambridge University Press, 1991.

Erik Nikolai Torgersen. Comparison of experiments when the parameter space is finite. Zeitschrift f r Wahrscheinlichkeitstheorie und Verwandte Gebiete , 16(3):219-249, 1970.

Mirko Torrisi, Saeid Asadollahi, Antonio De la Vega de Leon, Kai Wang, and Wilbert Copeland. Do chemical language models provide a better compound representation? In NeurIPS 2023 Workshop on New Frontiers of AI for Drug Discovery and Development , 2023.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.

Anton Tsitsulin, Marina Munkhoeva, and Bryan Perozzi. Unsupervised embedding quality evaluation, 2023.

Alexandre B Tsybakov. Introduction to Nonparametric Estimation . Springer series in statistics. Springer, Dordrecht, 2009.

Cynthia Van Hee, Els Lefever, and Véronique Hoste. Semeval-2018 task 3: Irony detection in english tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation , pages 39-50, 2018.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations , 2018.

Luke Vilnis and Andrew McCallum. Word representations via gaussian embedding. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015.

Hongwei Wang, Weijiang Li, Xiaomeng Jin, Kyunghyun Cho, Heng Ji, Jiawei Han, and Martin D. Burke. Chemical-reaction-aware molecule representation learning. In International Conference on Learning Representations , 2022.

Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533 , 2022.

David Weininger. SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules. Journal of Chemical Information and Computer Sciences , 28(1):31-36, February 1988. Publisher: American Chemical Society.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations , 2019.

Minghao Xu, Hang Wang, Bingbing Ni, Hongyu Guo, and Jian Tang. Self-supervised graphlevel representation learning with local and global structure. arXiv preprint arXiv:2106.04113 , 2021.

Nianzu Yang, Kaipeng Zeng, Qitian Wu, Xiaosong Jia, and Junchi Yan. Learning substructure invariance for out-of-distribution molecular representations. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification. In Proc. of EMNLP , 2019.

Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 5812-5823. Curran Associates, Inc., 2020.

Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics , 2:67-78, 2014.

Sheheryar Zaidi, Michael Schaarschmidt, James Martens, Hyunjik Kim, Yee Whye Teh, Alvaro Sanchez-Gonzalez, Peter Battaglia, Razvan Pascanu, and Jonathan Godwin. Pretraining via denoising for molecular property prediction. In International Conference on Learning Representations , 2023.

Li Zhang, Steven Wilson, and Rada Mihalcea. Multi-label transfer learning for multi-relational semantic similarity. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019) . Association for Computational Linguistics, 2019.

Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NIPS , 2015.

Appendix

Table of Contents

A Background and Notation

We consider all alphabets to be standard Borel [28] (i.e., isomorphic to a Borel subspace of a Polish space), encompassing virtually all practical scenarios. Each such space Y is equipped with its Borel σ -algebra B ( Y ) . The set of all probability measures on Y is denoted by P ( Y ) . The total variation distance between P and Q in P ( Y ) is defined as

and the Kullback-Leibler divergence is defined by

Given a joint probability measure P XY in P ( X × Y ) induced by two random variables X ∈ X and Y ∈ Y with product measures P X P Y , the Mutual Information is defined as I ( X ; Y ) = KL ( P XY ∥ P X P Y ) . If P X ∈ P ( X ) is a probability measure induced by X ∈ X , the Differential Entropy is defined by

A Markov (or transition probability) kernel between X and Y is a mapping T : B ( Y ) × X → [0 , 1] , satisfying T ( ·| x ) ∈ P ( Y ) for all x ∈ X and T ( B|· ) being a measurable function on X for any B ∈ B ( Y ) . The space of all such T is denoted by K ( Y | X ) . In cases where both Y and X are finite, any K ( Y | X ) is represented as a stochastic matrix with elements T ( y | x ) , ( x, y ) ∈ X × Y . Every T ∈ K ( Y | X ) induces a mapping P ( X ) -→ P ( Y ) , denoted by T , mapping any P ∈ P ( X ) to Q = T ◦ P ∈ P ( Y ) , where where µ denotes the Lebesgue measure. Similarly, it is possible to define the conditional entropy of Y given X which is denoted by h ( Y | X ) . The mutual information satisfies the identities I ( X ; Y ) = h ( Y ) -h ( Y | X ) = h ( X ) -h ( X | Y ) and h ( Y | X ) ⩽ h ( Y ) (see [27] for further details).

We denote the composition of Markov kernels by juxtaposition: for M ∈ K ( Z | Y ) and T ∈ K ( Y | X ) , their composition M ◦ T ∈ T ( Z | X ) is defined by

We define the average of the total variation distance between two Markov kernels T, T ′ ∈ K ( Y | X ) as follows:

A statistical model is a triple M U ≡ ( U , B ( U ) , ( P U | Y ( ·| y ) : y ∈ Y ) ) , where ( U , B ( U )) is a sample space; Y is a concept space, and P U | Y : B ( U ) × Y → [0 , 1] is a Markov kernel (or transition probability).

B Proofs Theoretical Results

Proposition (Relationships of sufficiency and information) . The following relationships hold:

(i) Sufficiency ⇒ informativeness. If the embedding model P U | X is sufficient for the embedding model P V | X , i.e. U ≽ S V , then U ≽ I V . However, Informativeness ⇏ sufficiency.

(ii) Informativeness ⇒ higher capacity to distinguish concepts. If the embedding model P U | X is more informative than embedding model P Z | X , i.e. U ≽ I V , then

for any pair of concepts ( y 0 , y 1 ) ∈ Y × Y and all probability distributions P Y X .

B.1 Proof Proposition 1

Proof. It is immediate to check that the data-processing inequality and the Markov chain Y ↔ X ↔ U ↔ V implies the relation in claim (i). On the other hand, the non-equivalence is proved by means of an explicit counterexample [57]. Given any 0 < p < 1 / 2 with ¯ p = 1 -p . For some ϵ, δ > 0 , consider two discrete embedding models defined by the following matrices:

By taking ϵ, δ > 0 small enough, both P U | X and P V | X are stochastic matrices. It follows that P V | X is not a degraded version of P U | X but provided that ϵ, δ are sufficient small, the embedding model P U | X is more informative than P V | X , which proves the claim.

In order to show (ii), let P U | Y and P V | Y be the corresponding probability measures induced by P X | U ( ·| u ) via the embedding models:

and

for any y ∈ Y = { y 0 , y 1 } . For a 0 ⩽ λ ⩽ 1 , let P X | Y ( ·| y 0 ) ∈ P ( X ) and P X | Y ( ·| y 1 ) ∈ P ( X ) be two arbitrary probability measures on X . Let P X | Y ( X | y ) be defined by

By replacing it into equations (9) and (10), we obtain

✶ ✶ and let P Y ( y 0 ) = λ and P Y ( y 1 ) = 1 -λ . The above probability measures correspond to a quadruple of random variables: ( Y λ , X λ , U λ , V λ ) ∈ P ( Y × X × U × V ) . Consider the function f ( λ ) defined by

It is not difficult to check that f ( λ ) ⩾ 0 for all 0 ⩽ λ ⩽ 1 , and f (0) = 0 which requires that f ′ (0) ⩾ 0 . By taking the differentiation, we obtain

which implies the claim (ii). This concludes the proof.

B.2 Comments about capacity to distinguish concepts

Notice that the KL divergence between the induced distributions of the resulting embedding is not less for the embedding model U than Z . Indeed, consider the case of binary classification Y = { y 0 , y 1 } with uniformly distributed concepts. Pinsker's inequality [108] together with claim (iii) imply

From which, it is easy to verify that the accuracy of the expected Bayes accuracy of the optimal classifier based on V is upper bounded by [108, Lemma 2.1]:

where the exponent in the upper bound is subject to the discriminating capacity through the KL divergence of the embedding model U on U .

B.3 Proof of Proposition 2 and Corollary 1

We begin with the proof of Proposition 2.

Proof. Clearly, the assumption (i) implies the statement (ii) by Data-Processing. Conversely, let us assume point (ii) holds. This means that, for every probbaility distribution P X and all conditional probability distributions P Y | X , there exists ρ U : U →P ( Y ) such that

where ρ V : V → P ( Y ) is a (possibly randomized) inference procedure is transition probabilities, which the learner can optimize to maximize the guessing probability.

Let the decision rule ρ V ( y | v ) = 1 [ v ∈ A y ] for any partition { A y } y ∈ Y of V with A y ∈ B ( V ) . Then, for any P Y | X , expression (13) implies the existence there exists ρ U ( y | u ) such that

However, we can rewrite the last expression as:

By applying the minimax theorem [87], it is possible to exchange the order of the inf and the sup, which yields:

We observe that

for each x ∈ X and thus,

where the equality holds if and only if Γ ( ( y, x ) , ρ U ) = 0 for all y ∈ Y , for each x ∈ X , since by contradiction otherwise

Therefore,

which means the maximum is achieved by degenerate random variables Y = f ( X ) achieving the maximum for each x ∈ X . Consequently, we have that hence inf ρ U E ∥ P V | X -ρ U ◦ P U | X ∥ TV = 0 , and so the existence of the transition probability ρ U such that P U | X is sufficient for P V | X . This concludes the proof of the Proposition 2.

We now show the proof of Corollary 1.

Proof. Continuing from the proof of Proposition 2, which remains unchanged, until one demonstrates that for ε > 0 ,

To proceed from this point, let us now examine the following quantity:

The above quantity is the induced ℓ 1 -norm distance between ∫ A y P V | X ( d v | x ) and ∫ U ρ U ( y | u ) P U | X ( d u | x ) . Since, for all x ∈ X , we have that

which implies that, for the strategy ρ U achieving the left-hand side of Eq. (24),

Hence,

This concludes the proof of the Corollary 1.

B.4 Example of comparisons of statistical experiments

Example 1 (Statistical experiments with Gaussian embedding models) . Let U | x and V | x be independently normally distributed as N ( x, σ 2 ) and N ( x, ϵ 2 σ 2 ) , respectively, with 0 < ϵ < 1 .

Case of σ 2 = σ 2 0 known. Here U ≽ S V since V + ν | x has the same distribution as U | x when ν ∼ N (0 , (1 -ϵ 2 ) σ 2 0 ) . That is V is strictly more informative than U . However, U is strictly more informative than V .

Case of x = 0 known. One can observe that U ≈ V since the variables V/ϵ have the same distribution as the U , and the variables ϵU have the same distribution as the V .

Case of x and σ unknown. Surprisingly, in this case U and V are not comparable.

B.5 Sufficiency and Inference Procedures with Embedding Models [13, 63]

Proposition 3 (Sufficiency and risks of a given task on embedding models [13, 63]) . An embedding model P U | Y ∈ K ( U | Y ) is deemed to be sufficient for another one P V | Y ∈ K ( V | Y ) if and only if, for any bounded loss function ℓ where ∥ ℓ ∥ ∞ ⩽ 1 , and for any inference procedure ρ V : V → Y , there exists a inference procedure (possibly randomized) ρ U : U →P ( Y ) such that the resulting statistical risks satisfy

Here we denote by R y ( P U | Y , ρ Y , ℓ ) and R y ( P V | Y , ρ V , ℓ ) the statistical risks for the corresponding inference frameworks, respectively.

Remark 5 . The restriction ∥ ℓ ∥ ∞ ⩽ 1 is irrelevant here. However, we opt for simplicity and limit our focus to situations where one encounters dominated statistical models with Polish sample spaces. In essence, various extensions do not significantly alter the conceptual aspects of the underlying statistical problem (see [103] for further details). Rather, they primarily reflect the complexity of its measure-theoretic formulation.

B.6 Deficiency and Expected Risk [63]

In 1964, Le Cam [63] clarified the relationship between the sufficiency of an embedding model on a given task and its expected risk on this task. The following theorem provides a formal statement of this relationship.

Theorem 1 (Le Cam [63]) . Let ε > 0 be fixed. Then, δ ( P U | Y → P V | Y ) < ε if and only if, for any bounded loss function ℓ where ∥ ℓ ∥ ∞ ⩽ 1 , and for any inference procedure ρ V using the embedding model P V | Y , there exists a inference procedure (possibly randomized) ρ U based the embedding model P U | Y such that the risks satisfy R y ( P U | Y , ρ U , ℓ ) -ε ⩽ R y ( P V | Y , ρ V , ℓ ) , for all y ∈ Y .

C NLP Experiment Details

In this section, we provide all the necessary experimental details to reproduce the experiments in NLP. For the I S score estimation, please see Appendix E. First, we detail the models and datasets used in the experiments. We provide the training details of the downstream tasks, and finally, we present the comprehensive results of the NLP experiments.

C.1 Models and Datasets statistics

In Tab. 1, we provide the metadata of the models used in the NLP experiments and their scores on the MTEB benchmark when they exist. We provide in Tab. 2 the statistics of the datasets used to evaluate the I S score.

C.2 Downstream tasks training details.

All the downstream tasks are trained in the exact same way. We use a dense classifier with two hidden layers of dimension 256 and train for two epochs using ADAM [56] with a learning rate of 10 -3 , on the official training set and evaluated on either the validation or test set when they are available (with respect to the Huggingface datasets). We do not perform early stopping or selection using the validation set.

C.3 NLP Comprehensive Results

We provide in this section the unaggregated results for the main NLP experiments presented in the main text, then we provide numerous ablation studies and additional results to address different aspects of the I S score of the embeddings in NLP.

C.3.1 Full MTEB Benchmark Results

The strength of the MTEB benchmark is that it evaluates embedders on a very large and diverse set of downstream tasks. We provide an Tab. 4 and Figure 6 the full results of the MTEB benchmark (English) for the models used in the NLP experiments.

IS

(rank

)

p

: 0.88

s

: 0.88

: 0.73

Classification

p

: 0.87

s

: 0.85

: 0.67

Clustering

p

: 0.88

s

: 0.89

: 0.70

Retrieval

p

: 0.84

s

: 0.82

: 0.62

STS

p

: 0.78

s

: 0.79

: 0.64

Reranking

p

: 0.89

s

: 0.84

: 0.65

Classification 2

Downstream tasks (rank

)

We obtain significant positive correlations in all categories of downstream tasks. We noticed that we obtained significantly poorer results on STS, Clustering, and Reranking tasks than on classification tasks. We believe this behavior is due to the nature of these tasks. Indeed, they do not rely on training an additional model on top of the embeddings but rather directly use the embeddings as is in dot products or similarity measures. An embedder could produce very informative embeddings, i.e. , it is possible to extract the useful information using a small model, and at the same time, these embeddings not be adequate for dot-product-based similarity measures. We believe further investigation is needed to understand the behavior of the models on these tasks. Especially to see if training a small model on top of the embeddings can improve the performance of these tasks.

C.3.2 Ablation studies

Many factors can impact the estimation of the I S score of the models, such as the dimensions of the different embeddings and the number of available embedders to evaluate. the I S score can capture many different aspects of the embeddings, such as the quality of the embeddings. We provide in this section a comprehensive set of ablation studies to evaluate the impact of these different factors on the I S score of the embeddings.

Impact of instruction finetuning. Instruction finetuning is now a common practice to improve the alignment of the base of models and expand the models' reasoning capabilities. In Figure 4c, show that instruction fine-tuning positively impacts the models' performance on the downstream tasks and that the I S score captures this improvement. In addition to studying the impact of instruction finetuning, we evaluated models at different checkpoints during their initial pretraining in Sec. C.3.3 using the CroissantLLM checkpoints [37].

C.3.3 Impact of training steps

Surprisingly, we found that the number of training steps does not significantly impact the models' performance on the downstream tasks nor on the I S score of the embeddings. The I S score correctly captures this behavior as shown in Figure 7. We hypothesize that the I S score in terms of embeddings is, in this case, determined by a few numbers of training steps (the first 5000 ) and the overall architecture of the model. Training the model further even leads to a decrease in performance on the downstream tasks, which is not captured by the I S score of the embeddings; this could be due to the very small variation in the performance.

C.3.4 Importance of embedding size normalization

We found that considering the amount of information packed by an embedding per coordinate is crucial to obtain a good ranking of the models. In Figure 16b, we show the correlation between the performance of the models on the MTEB benchmark and their I S score, not normalized by embedding size. While positive significative correlation is still present, the correlation is much weaker than when the dimension of the embeddings normalizes the information sufficiency.

base_5k

base_50k

base_150k

CroissantLLMBase

CroissantCool

0.495

0.500

0.505

0.510

S

Informativeness

0.6

0.8

1.0

Classification

inforatmation sufficiency

C.3.5 Community and cluster performance

(a)

Average

Classification

Clustering

Retrieval

STS

Reranking

Classification 2

GritLM-7B

e5-large-v2

UAE-Large-V1

bge-base-en-v1.5

ember-v1

sf_model_e5

udever-bloom-560m

all-mpnet-base-v2

sentence-t5-xl

multilingual-e5

e5-small

gtr-t5-xl

LaBSE

allenai-specter

average_word

(b)

0.0

0.5

rotten_tomatoes

0.0

0.5

paws-x/en

sup-simcse-bert-base-uncased

msmarco-bert-co-condensor

SFR-Embedding-Mistral

gtr-t5-base

gtr-t5-large

gtr-t5-xl

sentence-t5-large

sentence-t5-xl

allenai-specter

average_word_embeddings_glove.6B.300d

all-MiniLM-L6-v2

all-distilroberta-v1

all-mpnet-base-v2

UAE-Large-V1

sf_model_e5

ember-v1

e5-small

multilingual-e5-small

e5-large-v2

GritLM-7B

Llama-2-7b-hf

gemma-2b-it

gemma-7b-it

udever-bloom-560m

echo-mistral-7b-instruct-lasttoken

LaBSE

bge-base-en-v1.5

GIST-Embedding-v0

stella-base-en-v2

bge-micro-v2

0.0

0.5

dair-ai/emotion

We postulate that models clustered together by information sufficiency are likely to behave similarly on the downstream tasks. We evaluate this hypothesis by grouping the models by clusters discovered using the information sufficiency and reporting their performance on the downstream tasks. In Figure 8b and Figure 9, we observe that models within the same cluster tend to have similar behaviors on the downstream tasks.

C.3.6 Evaluating information sufficiency on different datasets

The I S score is evaluated with a fixed dataset supposed to represent the data distribution of interest (either a very diverse set or a subset following a distribution specific to a subfield like medical or legal texts). We cross-evaluated the I S of the models on different datasets and the performance of the models on the downstream tasks in Figure 10. We find that closer datasets in terms of the data distribution lead to a higher correlation between the I S score of the models and their performance on the downstream tasks. It is especially highlighted when comparing the correlations we get when evaluating I S on the AG News and Amazon polarity datasets. The first one corresponds to news articles, and the task is to guess the topic, whereas Amazon Polarity corresponds to product reviews, which is a sentiment analysis task. We find that the I S score evaluated on Amazon Polarity tends to yield way better correlation with the performance on the sentiment analysis downstream tasks

ag_news

dair-ai/emotion

imdb

rotten_tomatoes

tweet_eval;emoji

tweet_eval;emotion

tweet_eval;sentiment

clinc_oos;plus

Task

0.0

0.2

0.4

0.6

0.8

Pearson

Pearson

ag_news

dair-ai/emotion

imdb

rotten_tomatoes

tweet_eval;emoji

tweet_eval;emotion

tweet_eval;sentiment

clinc_oos;plus

Task

0.0

0.2

0.4

0.6

0.8

Spearman

Spearman

ag_news

dair-ai/emotion

imdb

rotten_tomatoes

tweet_eval;emoji

tweet_eval;emotion

tweet_eval;sentiment

clinc_oos;plus

Task

0.0

0.2

0.4

0.6

Kendall-Tau

Kendall-Tau

ag_news

Common

mteb/amazon_polarity

such as tweet_eval/sentiment, tweet_eval/emotion, IMDB or Rotten Tomatoes or to a lesser extent dair/emotion. Interestingly, the difference is less significant on the tweet_eval/emoji subtask.

D Molecular Experiment Details

D.1 Embedders considered

We considered 28 models for the molecular experiments, summed up in Tab. 5. Some models were used in different versions (architectures, number of parameters, pretraining dataset's size), such as the ChemBert models, followed by the size of their datasets, or ChemGPT, followed by their number of parameters.

Most 2D-GNNs were trained on the GEOM [6] dataset and were gathered from the repository of GraphMVP [67] model. Note that the MoleOOD [117] model was trained on the BACE [47] dataset, with a supervised task specific to the β -secretase enzyme. As a result, this model can be seen as "already specialized", explaining its poor performance in our evaluation.

Ability to predict denoising

3D models

BertMLM-77M

ContextPred

BertMLM-5M

GPT-GNN

MoleOOD_GIN

AttributeMask

MoleOOD_SAGE

BertMLM-10M

MoleOOD_GCN

GROVER

InfoGraph

Not-trained

GraphMVP

GraphCL

3D-Infomax

GraphLog

GPT-1.2B

BertMTR-5M

BertMTR-10M

BertMTR-77M

MolR_gcn

MolR_gat

MolR_tag

Ability to be predicted

by denoising 3D models

GPT-1.2B

ContextPred

BertMLM-10M

GPT-GNN

BertMLM-5M

BertMLM-77M

BertMTR-77M

BertMTR-10M

MoleOOD_GIN

BertMTR-5M

MoleOOD_SAGE

Not-trained

InfoGraph

MoleOOD_GCN

MolR_tag

MolR_gat

MolR_gcn

GROVER

3D-Infomax

AttributeMask

GraphCL

GraphLog

GraphMVP

(a)

0

1

S

(

Z

U

)

ContextPred

AttributeMask

GPT-GNN

BertMLM-77M

MoleOOD_GIN

MoleOOD_SAGE

BertMLM-5M

MoleOOD_GCN

Not-trained

InfoGraph

BertMLM-10M

GROVER

GraphCL

GraphMVP

GraphLog

3D-denoising

FRAD_QM9

3D-Infomax

GPT-1.2B

MolR_gcn

MolR_gat

BertMTR-5M

BertMTR-10M

BertMTR-77M

MolR_tag

Models Z

0

1

S

(

U

Z

)

(b)

We used the RD-Kit and Datamol tool-kits[61, 71] to pre-process the molecules and to generate three-dimensional conformers for 3D-models. To run the models using 3D views of the molecules, we generated five conformers (possible 3D configuration of the molecule) for each SMILES and kept the conformer with the lowest energy. Note that this methodology is imperfect, as the 3D coordinates might be noisy; however, we followed the same procedure to pre-process the ZINC dataset, to evaluate the information sufficiency, and on the datasets corresponding to each downstream task.

Finally we considered a variety of models architecture for 2D-GNNs notably graph isomorphism network (GIN) [115], principal neighbor aggregation networks (PNA) [26], graph convolutional network (GCN) [33], graph attention network (GAT) [110], topology adaptive graph convolutional networks (TAG) [32] and GraphSAGE [44]. For SMILES-based models, backbones are inspired by BERT [31], RoBERTa [69], and GPT [40]. Finally, both our 3D models use TorchMD-net[80] as a backbone.

D.2 Details on the information sufficiency estimation

3D models. The two 3D models considered (FRAD [39] and Denoising [121]) obtain high I S scores while being among the least predictable (Figure 11b and Figure 12). This suggests that these models capture 3D-specific features inaccessible from other modalities while maintaining sufficient overlap to predict them.

2D-3D models Some 2D-GNNs we considered (GraphMVP and 3D-infomax) are trained to maximize the mutual information between their embeddings and 3D representations of the molecule. Hence, we expect these models to be related to the 3D-denoising models we considered. However, we observe in Figure 11a that these models do not achieve particularly high information sufficiency scores over 3D-denoising models. On the other hand, the 3D models achieve high information sufficiency scores over them, which might suggest that these 2D models and 3D-denoising models share information that is easier to access from the 3D models. However, we want to point out that GraphMVP and 3D-infomax are both among the most predicted models; that is to say, among the other models in our pool, they achieve the highest information sufficiency scores.

GPT-1.2B

BertMLM-5M

BertMLM-77M

BertMLM-10M

Not-trained

InfoGraph

GROVER

GraphLog

GraphCL

GraphMVP

AttributeMask

MoleOOD_GIN

MoleOOD_GCN

MoleOOD_SAGE

3D-denoising

FRAD_QM9

3D-Infomax

BertMTR-5M

BertMTR-10M

BertMTR-77M

MolR_tag

MolR_gat

MolR_gcn

ContextPred

GPT-GNN

Models Z

0.4

0.6

0.8

1.0

1.2

S

(

Z

U

)

D.3 Complementary results on ADMET tasks

The datasets chosen for the molecular experiments are extracted from the Therapeutics Data Commons (TDC) [49] platform. We focused our experiments on ADMET tasks, crucial for drug discovery and development, which results in a total of 31 tasks, described in Tab. 6.

Each dataset is split into a training set, a validation set, and a test set following a scaffold split. This splitting strategy ensures that molecules sharing a similar scaffold will be part of the same split in the task. This corresponds to a more realistic scenario, where practitioners only have access

to molecules belonging to the same chemical series. Classifiers are then trained on the training set of each task, where the best hyperparameters and checkpoints are selected on the validation set. The final performance is finally measured on the test set, and we run each experiment 10 times with different random seeds.

A grid search is performed on each dataset individually to maximize the average AUROC or R 2 score across all models for binary classification and regression. We chose a maximum number of epochs depending on the task size to ensure all models have time to converge, limiting this amount to grow to at most 200 epochs.

Tab. 6 also displays the variation of the correlation coefficient between the ranking obtained on the I S score and the performances obtained on the downstream tasks regarding Spearman and Kendall correlations. We can see that the I S score correlates well with the performance of downstream tasks when the amount of data available is large.

Finally, we can see in Figure 13 that by grouping models based on their performances on these tasks, we obtain a similar clustering to the one obtained on the I S score in Sec. 6.2, with NLP-inspired models grouped. Similarly, the tinyChem Bert-MTR and MolR models are also grouped.

±

Clearance_Microsome_AZ

VDss_Lombardo

hERG

Half_Life_Obach

AMES

Lipophilicity_AstraZeneca

PPBR_AZ

CYP2D6_Substrate_CarbonMangels

CYP2D6_Veith

CYP2C9_Veith

CYP3A4_Veith

CYP1A2_Veith

BBB_Martins

Tox21

hERG_Karim

LD50_Zhu

Pgp_Broccatelli

Solubility_AqSolDB

Skin__Reaction

CYP3A4_Substrate_CarbonMangels

HydrationFreeEnergy_FreeSolv

Clearance_Hepatocyte_AZ

HIA_Hou

CYP2C9_Substrate_CarbonMangels

Carcinogens_Lagunin

PAMPA_NCATS

DILI

Bioavailability_Ma

ClinT ox

MoleOOD_OGB_GIN

MoleOOD_OGB_GCN

MoleOOD_OGB_SAGE

ContextPred

GPT-GNN

AttributeMask

ChemBertMLM-5M

MolR_tag

MolR_gat

MolR_gcn

ChemBertMTR-10M

ThreeDInfomax

ChemBertMTR-5M

ChemBertMTR-77M

DenoisingPretrainingPQCMv4

GROVER

GraphMVP

FRAD_QM9

GraphCL

GraphLog

InfoGraph

Not-trained

ChemBertMLM-10M

ChemGPT-1.2B

ChemGPT-19M

ChemGPT-4.7M

D.4 Drug target Interaction prediction

We propose further evaluating the embedders on yet another type of downstream task: Drug-Target Interaction. This task aims to predict the binding affinity between a given pair (drug, target). Since none of our models can process protein sequences, we decompose each dataset into multiple regression tasks on a single target by querying all molecules associated with a label for this target. Each task is then formulated as a set of molecules: X = { x i } i ∈{ 0 ,...,N } , and their labels Y = { y i } i ∈{ 0 ,...,N }

However, such tasks can be small, making it hard to build proper models from the embeddings. In contrast, the number of tasks is very large, making it computationally expensive to proceed to

5

10

15

20

25

Downstream tasks  (

rank

)

0

5

10

15

20

25

S

(

rank

)

Not-trained

GPT-1.2B

3D-Infomax

GraphCL

InfoGraph

BertMTR-5M

s

: 0.88

: 0.70

Overall

s

: 0.85

: 0.65

BindingDB_Kd

s

: 0.88

: 0.68

BindingDB_Ki

s

: 0.87

: 0.69

s

: 0.88

: 0.70

KIBA

an adequate hyperparameter selection. To bypass these limitations, we propose to estimate the embedded space's clustering quality for each model by measuring how close the labels of a molecule are compared to its nearest neighbors for each task. In other words, we measure:

and N i,n neigh ( X ) is the set containing the n neigh closest neighbors of x i ∈ X , which would be the performances of a K-nearest neighbors regressor on the task when using one data sample.

This quantity can be interpreted as a proxy of the embedding's capability to perform a similarity search, a classic chemo-informatic method using the similarity between different molecular projections to perform predictions. This training-free and computationally inexpensive approach allows us to evaluate the models on many tasks/targets.

We focused on 4 DTI datasets: KIBA [101], BindingDB-Kd, BindingDB-Ki, and BindingDBIC50 [68], with a total of 1496 tasks. We removed all tasks containing less than 128 molecules to ensure minimum data for the clustering evaluation.

We obtain similar results as in Sec. 6.1, our metric correlating with the performances on the different tasks considered. Figure 14 sums up all results by establishing a ranking across models and the number of neighbors, where we can see that MolR and ChemBerta-MTR appear as both the most promising models according to their I S score, and the best models evaluated. Furthermore, the outliers observed inSec. 6.1 show different behaviors in this setting. For instance, while 3D-Infomax seemed under-estimated and InfoGraph over-estimated by the I S score after seeing the results on the ADMET tasks, Infograph appears under-estimated in this setup, and 3D-infomax over-estimated.

E Information Sufficiency Estimation

E.1 Estimation method

As stated in Sec. 2, the deficiency δ ( P U | X → P Z | X ) is an intractable object measuring the cost of the reconstruction of Z from U . Due to this intractability, we propose to estimate the information sufficiency I S ( U → Z ) , which is a tractable proxy for the deficiency.

E.1.1 KNIFE Estimator

We recall the definition of the information sufficiency:

We denote C the number of modes chosen for the Gaussian mixture distributions and GM µ, Σ ,w the Gaussian mixture distribution with C components, parametrized by µ, Σ , w , with 1 T w = 1 , such that

GM µ, Σ ,w ( z ) = ∑ C c =1 w c N ( z | µ c , Σ c ) , where w c is the weight of the c -th component, and N ( z | µ c , Σ c ) is the density of a multivariate Gaussian distribution with mean µ c and covariance Σ c the c-th mean and covariance matrix.

To estimate the information sufficiency between the two embedders U and Z , we follow the procedure described in KNIFE [82].

F Θ is hence the class of multivariate Gaussian mixtures with C components, it is parametrized by µ, Σ , w . These learnable parameters are optimized to maximize the log-likelihood of Z . 8

The class of Markov kernels K Θ ( Z | U ) is also composed of multivariate Gaussian mixtures whose parameters are estimated using a small feedforward neural network. Such that for each u ∈ U , the parameters of the Gaussian mixture are µ ( u ) , Σ ( u ) , w ( u ) .

In practice we considered the covariance matrix to be diagonal to avoid the number of parameters of the Gaussian mixtures to grow too large with the dimension of the embeddings d . The number of parameters to be estimated for the Gaussian mixtures are hence: C × (2 d +1) , with the number of parameters of the feedforward networks for the Markov kernels.

Procedure 1 Estimation of I S ( U → Z ) , GM µ, Σ , w denotes the Gaussian Mixture model with means µ , covariances Σ and weights w .

Both the parameters of the marginal distribution of Z and the parameters of the conditional distribution of Z given U are estimated through likelihood maximization. The uncertainty of Z is estimated by the negative log-likelihood of the data under the marginal distribution of Z . The uncertainty of Z given U is estimated by the negative log-likelihood of the data under the conditional distribution of Z given U .

E.1.2 Embedding dimension normalization.

However, this approach faces one major drawback: it favors models that generate embeddings of high dimensionality. To evaluate the information sufficiency between the models, we estimate the uncertainty of Z and the uncertainty of Z given U . As seen in Figure 15, the estimated information sufficiency is highly correlated to the dimension of the latent space of Z , favoring models with high-dimensional latent spaces. This can be explained by the fact that these embedders yield larger marginal uncertainties. The resulting difference in the uncertainties I S ( U → Z ) is hence larger in absolute values.

We can see in Figure 15 that the dimension of the latent space of Z and the uncertainty of Z are evolving linearly. We thereby divide the information sufficiency by dim ( Z ) , which can be seen as an approximation of the normalization by the uncertainty of Z . We report in Figure 16b results without this normalization. It still correlates significantly with the downstream tasks performance, but the correlation is stronger when the normalization is applied. Hence, we focus on the relative variation of the uncertainty of Z explained by U . Note that the uncertainty of Z can be negative, as it can be assimilated to a differential entropy. As a result, the 'true' relative variation of the uncertainty would not be suitable for comparing different models (as it would not guarantee any ordering). While there is a general trend where larger models do have larger embeddings and perform better, well-trained smaller embeddings are competitive with larger embeddings, and the I S score captures this behavior (Figure 16a).

0

1000

2000

Z

uncertainty

Z

uncertainty

0

1000

2000

Z

|

U

uncertainty

Z

|

U

uncertainty

500

1000

1500

2000

0

250

500

750

S

(

U

Z

)

S

(

U

Z

)

500

1000

1500

2000

0.00

0.25

0.50

0.75

S

(

U

Z

)/

dim

(

Z

)

S

(

U

Z

)/

dim

(

Z

)

Dimension of Z

̸

We build our proxy by measuring the median values of the set S I S ( k ) = {I S ( Z k → Z l ) } l = k for an embedder Z k in our pool of models.

0.550

0.575

0.600

0.625

0.650

0.675

0.700

0.725

Classification Accuracy

500

1000

1500

2000

2500

3000

3500

4000

Embedding dimension

0.40

0.45

0.50

0.55

Information Sufficiency

300

325

350

375

400

425

450

475

500

40

45

50

55

60

65

70

UAE-Large-V1

e5-small

udever-bloom-560m

allenai-specter

Corr: 0.79

Average (56 datasets)

300

350

400

450

500

55

60

65

70

75

80

Classification  (12 datasets)

300

350

400

450

500

25

30

35

40

45

50

Clustering  (11 datasets)

300

350

400

450

500

10

20

30

40

50

60

Corr: 0.79

Retrieval  (15 datasets)

300

350

400

450

500

60

65

70

75

80

85

90

STS  (10 datasets)

300

350

400

450

500

45

50

55

60

Reranking  (4 datasets)

(b) Correlations on the MTEB benchmark when the embed- dings are not normalized by their size.

(a) Impact of the embedding size on the quality of the embeddings for the models measured as actual performance on downstream tasks and I S score.

E.1.3 Median instead of mean.

We use the median instead of the mean to compute the I S score. The median is more robust to outliers and the distribution of available embedders. For example, if many models are very similar, the mean would be biased by these models, while the median would not. Thus, we chose the median. While this change has a minor impact when there is enough diversity in the models, it can have a significant impact when the models are very similar, for example, when including different checkpoints of the same model Figure 17.

ϱ

p

ϱ

s

τ

STS (10 datasets)

0.90

0.85

0.66

Retrieval (15 datasets)

0.89

0.85

0.62

Classification (12 datasets)

0.92

0.84

0.65

Clustering (11 datasets)

0.86

0.83

0.61

Reranking (4 datasets)

0.86

0.79

0.65

Average (56 datasets)

0.93

0.86

0.66

Additional Classif (8 datasets)

0.84

0.81

0.61

(a) NLP

0

5

10

15

20

25

30

Downstream tasks (rank

)

0

10

20

30

S

(rank

)

UAE-Large-V1

e5-small

udever-bloom-560m

allenai-specter

p

: 0.93

s

: 0.86

: 0.66

Average (56 datasets)

(b) NLP

E.2 Hyperparameter selection

We use parametric classes composed of multivariate Gaussian mixture distributions for F Θ and K Θ ( Z | U ) in the definition of the information sufficiency (Eq. 27), the number of components in the mixture is a crucial parameter that needs to be selected concerning the data distribution of interest. We ran ablation studies to evaluate the impact of the number of components in the mixture on the information sufficiency estimation and the correlation between the I S score and the downstream tasks performance (Figure 20). We found that the ideal number of components in NLP is 8 , and in molecular modeling, it is 4 . Figure 18 and Figure 19 show the embeddings of the models in the first two principal components.

2

4

6

8

10

12

Number of modes

0.73

0.74

0.75

0.76

0.77

Spearman correlation

Impact of the number of modes

(a) Spearman

2

4

6

8

10

12

Number of modes

0.54

0.56

0.58

Kendall correlation

Impact of the number of modes

2

4

6

8

10

12

Number of modes

0.79

0.80

0.81

Pearson correlation

Impact of the number of modes

(c) Pearson.

E.3 Impact of the task size

Our study focused on finding the most promising model to be competitive on any downstream tasks. However, if the downstream task is not learnable, the most promising model could appear as not competitive on this specific task. In particular, if the amount of data available in the downstream task is insufficient, the differences between different embedder's representations might not be easily leveraged. This phenomenon can be seen in Figure 21, highlighting how when fewer than 1000 data points are available, the correlation between the I S score and the downstream performance becomes weaker.

10

itence-transformers/average_word_embeddings_glove.6B.300d

Ilmrails/ember-v1

sentence-transformers/msmarco-bert-co-condensor

google/gemma-7b

sentence-transformers/sentence-t5-xl

2

3

200

100

1

1

100

200

Z-

-1

0

7.5 5.0 2.5 0.0 2.55.0 7.5 10.0

E-

-2-101

2

400

200

0

200

400

-0.3 0.2 0.1 0.0

0.10.20.30.4

sentence-transformers/gtr-t5-large

BAAI/bge-base-en-v1.5

izhx/udever-bloom-560m

NousResearch/Llama-2-7b-hf

google/gemma-7bit

0.3

0.2

150

100

40

0.1

20

0.0

-.1

50

-0.2

-100

20

0.3

0.3

150 L

0.2

0.0

0.2

0.4

0.4

0.2

0.0

0.2

0.4

250 0250 500 7501000 12501500

20

100

0

100

200

300

croissantllm/CroissantCool

avsolatorio/GIST-Embedding-v0

jamesgpt1/sf_model_e5

Salesforce/SFR-Embedding-Mistral

infgrad/stella-base-en-v2

40

30

0.3

0.2

0.4

100

0.6

20

0.1

0.2

50

0.2

0.0

0.2

.0

10

0.2

0.2

50

0.4

20

-20

10

0

10

20

30

0.3 ,

0.3

-0.2-0.10.0

0.1

0.3

0.4

0.4

0.2

0.0

0.2

0.4

100

50

50

100

150

0.6

0.6

-0.4-0.20.0

0.6

sentence-transformers/LaBSE

SmartComponents/bge-micro-v2

0.4

1.0

0.2

0.5

0.0

0.0

0.0

0.5

0.5

0.2

1.0

1.0

0.4

2

2

1.5

1.00.50.00.5101.5

2.0

1.5

1.0

0.5

0.5

1.0

1.5

0.4

0.2

0.0

0.2

0.4

0.6

-2

1

0

sentence-transformers/all-mpnet-base-v2

sentence-transformers/all-istilroberta-v1

jspringer/echo-mistral-7b-instruct-lasttoken

thenlper/gte-large

intfloat/e5-small

0.6

100

0.4

75

0.2

0.2

0.2

0.2

0.0

0.2

0.0

25

25

50

0

0.1

0.1

0.0

0.1

0.1

0.4

0.4

0.2

0.0

0.2

0.4

0.6

0.2

0.4

0.2

0.0

0.2

0.4

50

75

75

-50

25

。

2550

75100

0.2

0.2

0.1

0.0

0.1

0.2

0.2

0.20.1

0.0

0.1

0.2

croissantllm/base_150k

croissantllm/CroissantLLMBase

thenlper/gte-base

croissantllm/base_100k

google/gemma-2b

20

0.2

60

0.1

0.0

5

101

0.1

15

10

0.2

10

0

20

10

0

10

20

20

20

-10

OE

-0.20.10.0

0.10.2

-15-10-5051015

20

20

20

20

40

60

sentence-transformers/gtr-t5-base

google/gemma-2b-it

sentence-transformers/gtr-t5-xl

sentence-transformers/allenai-specter

0.3

322

0.2

0.4 -

0.1

0.2

0.1

00

0.0

-.2

10

0.2

0.2

0.3

15

0.4

7.5

0.2

0.0

0.2

0.4

-15-10-5051015

20

0.2

0.0

0.2

0.4

0.6

0.2

0.0

0.2

0.4

7.5 5.0 2.5  0.0 2.55.0 7.510.0

0.6

0.8

jinai/jina-embedding-s-en-v1

.21

intfloat/multilingual-e5-small

WherelsAI/UAE-Large-V1

15

croissantllm/base_50k

0.3

sentence-transformers/sentence-t5-large

0.4

0.1

0.2

001

-0.4

0.2

0.1

2

0.6

0.2 -

0.2

0.50 0.25 0.000.250.50

0.75

0.2

0.1

0.0

0.1

0.2

0.3

-6

-10

10

-5

10

0.3

0.3

0.2

0.1

0.0

0.1

0.2

GritLM/GritLM-7B

croissantllm/base_5k

intloat/e5-large-v2

TaylorAI/gte-tiny

1.0

150

0.2

0.8

100

5

0.1

0.6

50

0.0

0.1

.2

50

0.2

-100

50

50

100

10

10

0.2

0.1

0.0

0.1

0.2

0.0

0.2

0.4

0.6

0.8

1.0

ContextPred

GPT-GNN

GraphMVP

GROVER

AttributeMask

GraphLog

GraphCL

10

100

50

20

10

20

40

20

40

20

20

10

0

0

20

10

20

20

40

20

20

10

PC1

100

200

PC1

200

400

PC1

20

20

0

PC1

20

50

PC1

0

50

25

0

PC1

25

50

0

PC1

20

InfoGraph

Not-trained

ChemBertMLM-5M

ChemBertMLM-10M

ChemBertMTR-5M

20

40

20

10

5

0

20

10

10

5

10

10

20

20

-10

5

10

10

5

10

10

20

20

25

0

25

50

0

10

20

50

50

50

0

50

50

0

50

10

0

10

PC1

PC1

PC1

PC1

PC1

PC1

PC1

ChemBertMTR-10M

ChemBertMTR-77M

ChemGPT-1.2B

ChemGPT-19M

ChemGPT-4.7M

DenoisingPretrainingPQCMv4

FRAD_QM9

10

10

20

20

10

20

10

20

10

10

10

10

20

10

ot-

10

10

10

0

10

10

0

10

=40

50

0

50

100

20

20

20

40

20

0

20

0

20

0

20

PC1

PC1

PC1

PC1

PC1

PC1

PC1

MolR_gat

MolR_gcn

MolR_tag

MoleOOD_OGB_GIN

MoleOOD_OGB_GCN

MoleOOD_OGB_SAGE

ThreeDInfomax

20

40

20

40

40

-20

20

40

0

20

20

40

0

10

20

10

30

20

20

40

20

10

20

10

OE

0

20

20

PC1

50

0

PC1

50

25

PC1

25

50

20

0

PC1

20

40

20

0

PC1

20

40

20

PC1

0

20

50

PC1

0

50

10 3

10 4

task_size

0.0

0.5

1.0

p

10 3

10 4

task_size

0.0

0.5

1.0

s

10 3

10 4

task_size

0.00

0.25

0.50

0.75

E.4 Impact of the number of models

10

20

30

N

0.4

0.6

0.8

1.0

Correlation

Type = Pearson

10

20

30

N

Type = Spearman

10

20

30

N

Type = Kendall-Tau

Metric

Average (56 datasets)

Classification Average (12 datasets)

Clustering Average (11 datasets)

Retrieval Average (15 datasets)

STS Average (10 datasets)

Reranking Average (4 datasets)

We evaluate the strength or I S score of an embedder with respect to all the others by relaxing the "for all" conditions with the median I S score. Thus, the number of available embedders might impact the performance of our method. Indeed, if too few embedders are available, it is likely that our evaluation would be biased by favoring models similar to the few available ones. We evaluate the impact of the number of available models by sampling subsets of our global model pool to compute the I S score. In Figure 22, we found that when fewer models are available, the rankings obtained using the I S

score correlate less with the downstream tasks' performance. However, as the number of models increases, the I S score becomes a good proxy for the performance of the models on the downstream tasks.

However, the evolution of this correlation is different in the two studied domains. Even with very few models, the I S score of molecular models already achieves a Spearman correlation close to 0 . 8 with the downstream tasks' performance. The correlation is much lower in NLP and only reaches 0 . 8 when about 15 models are available. This result can be expected, seeing how sparse the pairwise comparison matrix is in NLP compared to molecular modeling. This matrix is less sparse on molecular data, the graph induced by this adjacency matrix is more connected, and having access to a few nodes is enough to obtain measurements on the whole graph. On the contrary, by being much sparser,

10

20

N

0.5

0.6

0.7

0.8

Type = Kendall-Tau

10

20

N

0.7

0.8

0.9

Type = Spearman

Dataset

ADMET

Absorption

Metabolism

Tox

Distribution

Excretion

having access to a few nodes in this graph in NLP might only give information about the local structure of the graph in the community of the few nodes available. This might explain why the I S score's correlation with the downstream tasks performance stabilizes in NLP at about ten models, which is equal to the number of communities identified in the graph (in Sec. 5.2).

E.5 Comparison with other metrics

We evaluate other ways to measure the informativeness (Tab. 9 and Tab. 10) of the embedders. We considered the size of the model, their embeding dimensions and the simple reconstruction error when we train a cross-encoder to go from one embedding to another. Overall we found that our information sufficiency score significantly outperformed these naive metrics in both modalities.

In particular, training a cross-encoder with the l 2 reconstruction error proves to correlate with the ability to enable downstream performance, however, this correlation remains weaker compared to using the information sufficiency.

E.6 Computational ressources

Evaluating the I S score of the models is computationally inexpensive. Evaluating the I S score requires only a single (small) GPU. All our experiments were conducted on NVIDIA V100 and NVIDIA A6000 GPUs.

Our method's main (computational) shortcoming stems from the need to compute the information sufficiency between all pairs of models. This is a quadratic operation in the number of models. However, in practice, optimizing and estimating the information sufficiency presented in Sec. 3 is cheap. The complete evaluation of the 45 NLP models can be done in less than 6 hours on a single GPU.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We setup the problem of evaluating embedders as a communication problem and leverage information theoretic tools to derive a principled evaluation method.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We provide a separate section "Conclusion and Limitations" (Sec. 7) in the main paper and different ablations addressing possible shortcomings of our methods in appendices.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide a complete description of our setting and assumptions used to derive theoretical proofs as well, as the relaxation of the problem to make it tractable (Sec. 2, Appendix B).

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide a description of our global experimental protocol in the main paper (Sec. 4) and detailed descriptions for the different field in their respective sections (Appendix C, Appendix D). All the datasets and models used are publicly available and the code to reproduce our results is attached to the submission.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in

some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We attached the code to the submission and provide instructions to reproduce the results in the main paper and supplemental material. All datasets and models are publicly available and chosen to be easily accessible for reproducibility.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide in Appendix Appendix C and Appendix D the details of the training and test settings for the different fields. We also provide a global description of the experimental protocol in the main paper (Sec. 4).

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide error bars in the figures and tables of the main paper. In addition we provide robustness checks and ablations in the appendices.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the details of the compute resources used in the main paper and appendices in Sec. E.6.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This paper does not involve major ethical concerns and the research conducted in the paper conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: We only propose model selection method for embedders. The main concern, that is addressed in the paper, is that while our method is theoretically supported and empirically validated, it might not be applicable to all downstream tasks and should be used with caution.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: N/A

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We provide the citations and direct references of all the datasets and models used in our work. To ensure reproducibility, all the datasets and models used are publicly available and under open licenses.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We only created the library that implements our methods (it will be released publicly on github) and the code to reproduce our experiments. Both are documented as part of the code submission as supplementary material.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: N/A

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: N/A

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a task-agnostic method for the evaluation of embedding models called \"information sufficiency\". The general notion is to generate a pairwise matrix that effectively measures how well each embedding model can be used to generate the information content of the other. They compute an overall \"information sufficiency score\" for each model by computing the median of this pairwise score along one axis. They demonstrate that their metric correlates well with overall downstream task performance.\n\nSTRENGTHS:\nThe paper is very well-put-together and cleanly written. The intuition behind the method is well-motivated and makes sense. Most of my outstanding questions were addressed in the supplement. The topic of comparing embedding models is of substantial relevance and interest. There is a need for a better understanding of the quality of embedding models and for a task-agnostic evaluation metric.\n\nWEAKNESSES:\nMy primary concern centers on the potential pitfalls from using this method, described in more detail in the questions below. I am very open to raising my current score of 6 if those concerns can be addressed sufficiently.\n\nQUESTIONS:\nThe metric proposed here is a pairwise metric where the embedding model that can most effectively simulate other embedding models is ranked as the most effective. I wonder if it is possible that if a particular embedding model contains unique information that is not represented by any other embedding model in the chosen set, then is it possible that said embedding model would be ranked poorly? The thrust of my concern is that the adoption of this metric would favor embeddings that are more \"central\" even in edge cases where more niche information might be more useful. \n\nAs a follow up to the last question, I am wondering in what sense the question of whether one embedding model is better than another is generally useful. For most applications and downstream tasks, one does not want an embedding model that is better on average for a host of applications, they want one that is better for their proscribed use case, whatever that may be. Of course, there is the challenge that it is impractical to \"try every model\", given the breadth of available models. Is there some middle ground here, where a similar method can be used to narrow the number of models that are tested rather than to recommend a single one?  \n\nThe authors acknowledge that the usefulness of their methods depends on sampling a large number of diverse embedders with which to compare. To what degree is this circular - does this constraint require that you evaluate the set of embedders using a more traditional downstream benchmarking procedure?\n\nThe underlying method has some real similarity to the task affinity matrix from the \"Vision Taskonomy\" work by Zamir et al. (which is currently not cited), although there the idea is not so much to better compare embedding models as it is to understand the informatic structure of tasks in vision literature. Is there perhaps a way of using this pairwise similarity matrix to embed the differences in informatic content across these embedding models using this related approach?\n\n\nMinor aesthetic/typographical comments: \n\nThere is the floating phrase \"inforatmation sufficiency\" on page 26 in the supplement. \n\nHoly Boldface Batman. That's a lot of boldface. Especially the sentences that start as normal and lead into boldface.\n\nLIMITATIONS:\nPlease see my questions. No obvious negative societal impacts of the work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article introduces a novel framework, grounded in information theory, for assessing the relevance of embedding models (*embedders*). The authors begin by introducing the notion of *sufficiency* of a model A relatively to a model B, which can be used to rank embedders. They prove:\n1- \tsufficiency implies (i) a higher capacity to distinguish concepts (Prop. 1)\n2- \tsufficiency is equivalent to (ii) having a lower expected Bayes risk (Prop. 2)\nThey reasonably assume that (i) and (ii) is equivalent to “being a good embedder”.\nThe authors then introduce the *deficiency* between two models, a relaxed version of sufficiency. Theoretical bounds on the Bayes Risk, as a function of the deficiency, are given (Corr. 1). As deficiency is hard to compute, they develop a surrogate estimator termed *information efficiency* (Definition 3), which can be empirically estimated.\n\nThe authors perform experiments to validate their theoretical findings using datasets from NLP and molecular modelling. In the NLP domain, they employ datasets from the MTEB benchmark to demonstrate the correlation between their new ranking scores and model performance. For molecular modelling, they use the ADMET dataset. The authors also propose a graph visualization of the models build from the pairwise information efficiency values.\n\nSTRENGTHS:\n- In this paper, the authors propose an original task, and the approach developed is new.\n- The paper is well organized; theory and experiments are well motivated. The progression of the authors' reasoning is very clear.\n- The proposed method allows for a graph representation of embedders according to their relative expected performance on the considered dataset.\n\nWEAKNESSES:\n- Several key points of the method are not explicit:\n    - While the main metric is well described, there is little information on how it is estimated in practice. Besides the fact the KNIFE estimator from [82] is used to quantify mutual information, no procedure is described and the reader is left to try to rebuild it. \n    - How to choose reference data is not elaborated on, besides L229-230.  \n    - The choice the median, discussion about number of models, or embedding dimensions are not discussed within the core of the article while it would help a lot to grasp intuitions about how the proposed metric behaves.\n- On motivation and applicability: \n     - It would be interesting for the authors to expand on the practical utility of such a method. While it seems to be an interesting alternative to benchmarks which does not require any labelled data, I am not sure of how interesting it could be for applications, especially considering the paper does not expand on how difficult and computationally costly the estimation procedure is.\n\nQUESTIONS:\n- You mention (L211-212) that “We hence attempt to simulate $Z$ from $U$ by learning a Markov kernel”: do you need to learn a Markov Kernel for each model ? Is that computationally expensive ?\n- While framing the comparison between embedding models this way is new, it would be interesting to have . However, it would be interesting to have a baseline, even the naivest one (e.g. the number of parameters maybe?) to be able to compare your results.\n\nLIMITATIONS:\nLimitations have been adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nEvaluating embedding models is challenging because it typically relies on various downstream task data, despite the embedding models being trained for general purposes. This paper introduces an information-theoretic metric for comparing embedding models, eliminating the need for labeled datasets in their evaluation. The core idea is that if one embedding model can simulate another in most cases, it indicates a higher capacity for distinguishing concepts by its embeddings. Empirical results in both text and molecule embedding models show that the proposed metric closely aligns with downstream task rankings. Additionally, the community analysis facilitated by this metric reveals clusters of embedding models, illustrating their relationships.\n\nSTRENGTHS:\n- The target problem is clear, and the proposed metric is well-motivated from information theory perspective.\n- The experiments are extensive, demonstrating the effectiveness of the proposed metric.\n- The method allows for community analysis of embedding models, which is quite interesting.\n- The paper is well-written and easy to follow.\n\nWEAKNESSES:\n- The connection between the concept of deficiency and information sufficiency is somewhat unclear.\n- The computation of information sufficiency in practice is not well-explained. Adding more details would be beneficial.\n- (Minor) The arrangement of tables and figures could be improved.\n\nQUESTIONS:\n- Does transitivity (or a similar weaker concept) hold for information sufficiency? For example, if $I_S(U → Z) > I_S(W → Z)$ and $I_S(W → Z) > I_S(X → Z),$ does $I_S(U → Z) > I_S(X → Z)$?\n- Are there specific reasons for using multivariate Gaussians in Markov kernel learning? Are there any alternatives?\n- What are the requirements for data in estimating information sufficiency? How do the quantity, quality, and diversity of datasets affect the effectiveness of the proposed metric?\n- Line 114: Should $P_{U|X}$ be $P_{Z|X}$?\n- Lines 188-189: Isn't this bound practically vacuous due to the size of X?\n\nLIMITATIONS:\nThe authors adequately address the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new metric to compare embedding models without relying on labeled data. The approach involves embedding data using two separate neural networks Z and U, and then trying to use embedding model U to simulate/match the output of embedding Z. They then use this to calculate an information sufficiency criterion. They motivate this with theory, and show that their information sufficiency metric aligns nicely with established embedding benchmarks across two separate domains, NLP and molecular biology.\n\nSTRENGTHS:\nThis is an innovative, statistics-based approach that can potentially add a lot to the rapidly evolving subfield of embedding models.\n\nThis paper validates their approach across completely different domains: NLP and molecular biology. The authors have run extensive experiments on tens of embedding models and many different datasets within the two domains.\n\nThis paper is very well written with an attention to detail (and an extensive appendix). The paper also does a valiant job of combining theory and experiment.\n\nWEAKNESSES:\n1. This paper puts a heavy emphasis on the theoretical motivation without giving enough detail about the actual algorithm for calculating the information sufficiency metric I_s. Line 211 states “We hence attempt to simulate Z from U by learning a Markov kernel M via a mixture of multivariate Gaussians, and measure the uncertainty reduction it induces.” It would be helpful to expand on this in detail. The reader is left only guessing how exactly this was done, and in my opinion it does not seem quite replicable. The details in the Estimation method section E.1 and Hyperparameter selection section E.2 are somewhat minimal.\n\n2. This method is touted as a principled way to compare embedding models without using data labels (but still using data in the desired domain). However it is unclear what the other tradeoffs are. Aside from the question of data labels, Is this approach more computationally efficient?\n\nQUESTIONS:\n1. How exactly do you calculate the information sufficiency metric I_s for a given dataset in MTEB and two models? Can you expand on this in more detail (as mentioned above)?\n\n2. In the discussion section, you mention that there are other potentially promising methods for learning the Markov kernel. What do you think some other promising approaches might be?\n\n3. In Section C.3.1, the authors admit that their results are better for MTEB classification tasks when compared to STS, clustering, and reranking tasks. What numbers/tables are they referring to here?\n\nLIMITATIONS:\nIt would be helpful for the authors to share code upon publication so that others can use this metric/approach.\n\n### Minor comments:\n\nTypo Figure 3(b) “Molecular Modelling”\nTypo line 213 “embbeders”", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-27"}