How Molecules Impact Cells: Unlocking Contrastive PhenoMolecular Retrieval
Authors: Philip Fradkin
Puria Azadi Moghadam
Karush Suri
Frederik Wenkel
Ali Bashashati
Maciej Sypetkowski
Dominique Beaini
Venue: NeurIPS 2024 poster (2024)
Keywords: Multi-Modality
Contrastive Learning
CLIP
Cell Morphology
Molecules
Molecular Retrieval
Zero-Shot Learning
Cell-Painting
Primary area: machine_learning_for_other_sciences_and_fields
TL;DR: We address the challenge of contrastive phenomic molecular retrieval. We demonstrate pre-trained uni-modal representation methods can be used in a variety of  ways to significantly improve zero-shot molecular retrieval rates.

Abstract: Predicting molecular impact on cellular function is a core challenge in therapeutic design. Phenomic experiments, designed to capture cellular morphology, utilize microscopy based techniques and demonstrate a high throughput solution for uncovering molecular impact on the cell. In this work, we learn a joint latent space between molecular structures and microscopy phenomic experiments, aligning paired samples with contrastive learning. Specifically, we study the problem of Contrastive PhenoMolecular Retrieval, which consists of zero-shot molecular structure identification conditioned on phenomic experiments. We assess challenges in multi-modal learning of phenomics and molecular modalities such as experimental batch effect, inactive molecule perturbations, and encoding perturbation concentration. We demonstrate improved multi-modal learner retrieval through (1) a uni-modal pre-trained phenomics model, (2) a novel inter sample similarity aware loss, and (3) models conditioned on a representation of molecular concentration. Following this recipe, we propose MolPhenix, a molecular phenomics model. MolPhenix leverages a pre-trained phenomics model to demonstrate significant performance gains across perturbation concentrations, molecular scaffolds, and activity thresholds. In particular, we demonstrate an 8.1 times improvement in zero shot molecular retrieval of active molecules over the previous state-of-the-art, reaching 77.33% in top-1% accuracy. These results open the door for machine learning to be applied in virtual phenomics screening, which can significantly benefit drug discovery applications.

How Molecules Impact Cells:
Unlocking Contrastive PhenoMolecular Retrieval
Philip Fradkin 1.2*, Puria Azadi
Karush Suri', Frederik Wenkel', Ali Bashashati", Maciej Sypetkowski", Dominique Beaini
Valence Labs, University of Toronto, Vector Institute, University of British Columbia, Université de Montréal, Mila- Quebec Al Institute dominiqqu@@valeecolabb... com
Abstract
Predicting molecular impact cellular function is core challenge in therapeutic design. Phenomic experiments, designed to capture cellular morphology. utilize microscopy based techniques and demonstrate high throughput solution for un- covering molecular impact on the cell. In this work, we learn a joint latent space between molecular structures and microscopy phenomic experiments, aligning paired samples with contrastive learning. Specifically, we study the problem of Contrastive PhenoMolecular Retrieval, which consists zero-shot molecular struc- ture identification conditioned on phenomic experiments. We assess challenges in multi-modal learning of phenomics and molecular modalities such as experi- mental batch effect, inactive molecule perturbations, and encoding perturbation concentration. We demonstrate improved multi-modal learner retrieval through (I) uni-modal pre-trained phenomics model, (2) novel inter sample similarity aware loss, and (3) models conditioned on a representation molecular concentra- tion. Following this recipe, we propose MolPhenix, molecular phenomics model. MolPhenix leverages pre-trained phenomics model to demonstrate significant performance gains across perturbation concentrations, molecular scaffolds, and activity thresholds. In particular, we demonstrate an 8. improvement zero shot molecular retrieval of active molecules over the previous ffate-of-the-ate reaching 77.33% in top-1% accuracy. These results open the door for machine learning to be applied in virtual phenomics screening, which significantly benefit drug discovery applications.
Introduction
Quantifying cellular responses elicited by genetic and molecular perturbations represents a core challenge medicinal research [4] [57] Out of an approximate 1080 druglike molecule designs, a small number are able to alter cellular properties to reverse the course of diseases [5] [2]] In recent years, microscopy- based cell morphology screening techniques, demonstrated potential for quantitative understanding of molecule's biological effects. Experimental techniques such as cell-painting are used capture cellular morphology, which correspond to physical and structural properties of the cell 66 7. Cells treated with molecular perturbations can change morphology, which is captured by staining and high throughput microscopy techniques. Perturbations with similar cellular impact induce analogous morphological changes, allowing to capture underlying biological
"These two authors contributed equally (ordered alphahetically) and reserve the right to swap their order.
I Equal advising.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

effects in phenomic experiments, Identifying such perturbations with similar morphological changes
can aid in discovery of novel therapeutic drug candidates [50] 29
22
Determining molecular impact on the cell can be formulated as multi-modal learning problem, build
Similar text-image models, paired data is collected from phenomic experiments along with molecules used to perturb the cells. Contrastive objectives have been used an effective approach in aligning paired samples from different modalities 43 32.. A model that has learned cross-modal joint latent space must be able to retrieve molecular perturbant conditioned on the phenomic experiment. We identify this problem as contrastive phenomolecular retrieval (see Figure 2 Addressing this problem can allow for identification of molecular impact on cellular function, however, this comes with own set of challenges. [8, 2 65
allowing
rich family of methods 43
62
53
(1) Firstly, multi-modal paired phenomics molecular data suffers from lower overall dataset sizes and subject to batch effects. Challenges with uniform processing and prohibitive costs associated with acquisition of paired data, leads an order of magnitude fewer data points compared text-image datasets [49 [TI. Furthermore, data is subject to random batch effects that capture non-biologically
meaningful variation 33 55 (2) Paired phenomic-moleculee data contains inactive perturbations that do not have a biological effect or do not perturb cellular morphology. It is difficult to infer priori whether a molecule has cellular effect, leading to the collection of paired molecular structures with unperturbed cells. These data aata-points are challenging to filter out without an effective phenomic embedding, as morphological effects are rarely discernible. These samples can be interpreted as misannotated, under the assumption of all collected pairs having biologically meaningful interactions. (3) Finally, a complete solution for capturing molecular effects on cells must capture molecular concentration. The same molecule can have drastically different effects along its dose response curve, thus making concentration an essential component for learning molecular impact.
In this work, we explore the problem of contrastive phenomolecular retrieval by addressing the above challenges circumvented in prior works. Our key contributions are as follows:
We demonstrate significantly higher phenomolecular retrieval rates by utilizing a pretrained uni- modal phenomic encoder. Thus alleviating the data availability challenge, reducing the impact of batch effects, and identifying molecular activity levels.
We propose a novel soft- weighted sigmoid locked loss (S2L) that addresses the effects of inactive molecules. This is done by leveraging distances computed in the phenomic embedding space to learn sater- similarities.
We explore explicit and implicit methods to encode molecular concentration, assessing the model's ability to perform retrieval in an inter-concentrattio setting and generalize unseen concentrations.
Molecular Structures
MolGPS
Concentration
Explicit
Cuideliiee Expliik concar Bation encodingg
CNN
MolPhenix
Guidcline impliih concer baaion oncocing with $22.
0H
uni-modal prersined Phar omics and
Encader
Guideline2 armiir $21
S2L
Guidsline 2 .inderampliing
Phenomics Experiments
Phenoml
Simil arity Logits
MAE
Figure Illustration of proposed guidelines when incorporated in our MolPhenix contrastive phenomolecular retrieval framework. We address challenges by utilizing uni-modal pretrained MAE & MPNN models, inter-sample weighting with dosage aware S2L loss, undersampling inactive molecules, and encoding molecular concentration.

---

Following these principles, we build MolPhenix, auulti-modll molecular phenomics model address- ing contrastive phenomolecular retrieval (Figure [. MolPhenix demonstrates large and consistent improvements in the presence of batch effects, generalizing across different concentrations, molecules, and activity thresholds. Additionally, MolPhenix outperforms baseline methods in zero-shot setting, achieving 77.33% top- % retrieval accuracies on active molecules, which corresponds to a 8.1 improvement over the previous state-of-the-ar (SOTA) [48].
2 Related Work
Uni-modality Pretraining: Self-supervised methods have demonstrated success across a variety of domains such as computer vision, natural language processing and molecular representations
In vision, contrastive methods have been used minimize distance the model's latent space of two views of the same sample [12 51 19 21]. Reconstruction objectives have also permeated computer vision, such as masked autoencoders (MAE). MAEs typically utilize vision transformers to partition the image into learnable tokens and reconstruct masked patches [4]. These methods have been extended to microscopy experimental data designed to capture cell morphology 160 28 Phenom1 utilizes masked autoencoder with ViT-L/8+
44
61.
20
architecture and a custom Fourier domain reconstruction loss, yielding informative representations of phenomic experiments [28 13). From representational perspective, Graph Neural Networks (GNN) have been used to predict molecular properties by reasoning over graph structures. A combination of reconstruction and supervised objectives have led to models generalizing to diverse range of prediction tasks [36 66 56 47). Our work leverages uni-modal foundation models, which are used to generate embeddings of phenomic images and molecular graphs.
Multi-Modal Objectives: Multi-modal models combine samples from two or more domains. to learn rich representations and demonstrate flexible ways to predict sample properties [43 1 23
Contrastive methods minimize distances between paired samples, traditionally in text-image domains. However, training these models is computationally expensive, requiring large datasets. Multiple contributions have allowed for reduction in compute and data budgets by an order magnitude. In LiT, the authors demonstrate that utilizing uni-modal pretrained models for one or both modalities matches zero-shot performance with an order of magnitude fewer paired examples seen [63] Zhai et al. (2023) demonstrate that by replacing the softmax operation over cosine similarities with an element wise sigmoid loss, allows contrastive learners to improve performance under label noise regime [62]. By using uni-modal pre-trained modal calculate similarities between samples from one of the modalities, Srinivasa et al. (2023) have demonstrated improved performance on zero-shot evaluation [53) In our work, we build along these directions in molecular phenomic multi-modal training.
Molecular-Phenonee Contrastive Learning: Prior works in contrastive phenomic retrieval have utilized the InfoNCE objective as a pre-training technique to construct uni-modal representations Nguyen et al. (2023) propose multi-modal objective trained on hand-engineered visual features and a GNN molecular encoder. The work demonstrates improved molecular property prediction with image encoder pre-training 37 54 Recent methods have attempted to improve retrieval by using the InfoLOOB objective [41]. Specifically, CLOOME utilizes the InfoLOOB loss with hopfield networks for pero-shot retrieval on unseen data samples 45 48
38 64
InfoCORE aims to
mitigate batch effects in multimodal molecular representations, improving retrieval capabilities and property prediction by adaptively reweighting samples minimize confounding from non- bioloiilal associations [59]. Our work is parallel to the above directions, demonstrating a significant increase in paadecullr-phenoonne retrieval by building on algorithmic improvements from the multi-modality literature.
3 Methodology
In this section, explain key challenges facing phenomolecular retrieval and provide guidelines that are key methodological improvements behind the success of MolPhenix [
Preliminaries: Our setting studies the problem of learning multi- representations of molecules and phenomic experiments of treated cells [48] The aim of this work is to learn a joint latent space which maps phenomic experiments of treated cells and the corresponding molecular perturbations into the same latent space. We consider a set of lab experiments E defined as the tuple (X, M, C, V) Each experiment E E consists of data samples x, X (such as images) and perturbations m, E M
3

---

(such as molecules) which are obtained at specific dosage concentration E C, while &i E y denotes molecular activity threshold.
Figure 2 describes the problem of contrastive phenomolecular retrieval, where for single image x, the challenge consists of identify- ing the matching perturbation, m,, and con- centration, C, used to induce morphologi- cal effects. This can be accomplished in zero-shot way by generating embeddings for (m1, C1), (m. Cj) and x, using functions for. (m, c), foz (x) which map samples into Rd. Then, by defining similarity metric be- tween generated embeddings Zy; and Zm,, saim. we can rank (m), ci)...(m, Cÿ) based on com- puted similarities. contrastive phenomolecular retrieval problem would learn fo.. (m, c) and fo_ (x) that results in consistently high retrieval rates of (m, c;) used perturb x;-
{kr E 1..j)(m.. o.) (M,C)
Contrastive PhenoMolecular Retrieval
fau (m,, cz)
faatten Rea) 6 top R%
foctor ke (1..5 Semn
effective solution to the
Illustration of the contrastive phe- nomolecular retrieval challenge. Image x; and a set of molecules and corresponding concentrations (ma; C&) get mapped into 12d latent space. Their similarities get computed with feim and ranked to evaluate whether the paired perturbation appears in the top K%.
Figure 2:
In practice, the image embeddings are generated using phenomics microscopy foundation MAE model (28) 20)). We use phenomic embeddings marginalize batch effects, infer inter-sample similarities, and undersample inactive molecules. Activity is determined using consistency of replicate measurements for a given perturbation. For each sample, a p value cutoff y E is used to quantify molecular activity. Only molecules below the value cutoff &b are considered active.
Prior methods in multi-modal contrastive learning utilize the InfoNCE loss, and variants thereof 38 to maximize the joint likelihood of x; and m,, Given a set of N x N random samples (x1, m1; c1), (xN, m containing N positive samples at 1thh index and (N - x N negative samples, optimizing Equation I maximizes the likelihood of positive pairs while minimizing the likelihood of negative pairs:
log expl km.)/ + log k= exp( (Z Zmn) /T) Σk=n exp( 2ord /T)
exp((zn; b/m)//T) )
CIsfeNCE
(1)
Where Zu, Zm correspond to phenomics and molecular embeddings respectively, T softmax temperature, and corresponds to cosine similarity.
Challenge Phenomic Pretraining and Generalization
We find that using a phenomics foundation model to embed microscopy images allows for mitigation of batch effects, reduces the required number of paired data points, and improves generalization the process. While CLIP, hallmark model in the field of text-image multi-modality, was trained on 400 million curated paired data points, there an order of magnitude fewer paired molecular- phennooi-p molecule samples 43 Cost and systematic pre-processing of data make large scale data generation efforts challenging, and resulting data is affected by experimental batch effects. Batch effects induce noise in the latent space result of random perturbations in the experimental process, while biologically meaningful variation remains unchanged [39] (2). Limited dataset sizes and batch effects make it challenging for contrastive learners to capture molecular features affecting cell morphology. yielding low retrieval rates [88
We address data availability and generalization challenges by utilizing representations from large uni- modal pre-trained phenomic model, Opn. trained to capture representations cellular morphology. Ops, is pretrained on microscopy images using a Fourier modified MAE objective, utilizing the
ViT-L/8 architecture with methodology similar to Kraus al. (2024) 20 14 28 For simplicity in future sections, refer to this model Phenoml. This pretrained model allows drastic

---

reduction in the required number of paired modti samples [63] In addition, using phenomic representations alleviates the challenge of batch effects by averaging samples. 2,, generated with the same perturbation m; over multiple lab experiments Averaging model representations I S tEN"., allows marginalizing batch effect induced by individual experiments.
N
Guideline Utilizing pre-trained uni-modal encoder, Oyb, can be used to reduce the num- ber of paired data-points compared to training without prior optimization. In addition, averaging phenomic embeddings Z, from matched perturbations can alleviate batch effects.
To reason over molecular structures, we make use of features learned from GNNs trained on molecular property prediction 34. We utilize pretrained MPNN foundational model up to the order of parameters for extracting molecular representations following a similar procedure to Sypetkowski et al. (2024) [56]. We refer to this model as MolGPS.
Challenge 2: Inactive Molecular Perturbations
mi
C
The thenomics-moleculas data collection process can result in pairing of molecular structures with unper- turbed cells in cases where the molecule has no effect on cell morphology (Figure
X
Since the morphological effects observed in cell x; conditioned on the perturbation, in the absence of
P(xi m, c,)a)
a molecular effect |xx//x.. C₁ m;) ~ P(x]xx). In
Figure 3: Data generation process of phe- nomic experiment on cells Xj with molecular perturbations m; and concentrations Cj.
these samples, phenomic data will be independent, from paired molecular data, which results misanno- tation under the assumption of data-pairs having an underlying biological relationship. We demonstrate how utilizing Phenom1 to undersample inactive molecules and learn continuous similarities between samples can alleviate this challenge.
To undersample inactive molecules, we extract the embeddings from Phenom] and calculate the relative activity of each perturbation (m; c,) (M, C). This is done using the rank of cosine simi- larities between technical replicates produced for molecular perturbation against null distribution. The null distribution is established by calculating cosine similarities from random pairs of Phenom! embeddings generated with perturbation (m,. C2), (m,, C₁). Hence, we can compute a p- p-value and filter out samples likely to belong to the null distribution with an arbitrary threshold eb.
In addition, by utilizing an inter-sample aware S2L training objective, the model can learn similari- ties between inactive molecules. S2L is grounded in previous work which demonstrates improved robustness to label noise (SigLip) and learnable inter-sample associations (CWCL) [62 53). Continu- ous Weighted Contrastive Loss (CWCL) provides better multi-modal alignment using uni-modal pretrained model to suggest sample distances, relaxing the negative equidistant assumption present in InfoNCE [53]:
exp ((Zz, Zmy Zmy) //T/
LCWCL, M-X
Σ why log
Σ
Sk=1 exp ((Zzg. 2mme)/T)
CWCL weights logits with continuous measure of similarity x resulting in better alignment of embeddings Zx; and Zm, across modalities. In equation 2 w is computed using a within modality similarity function such as w. = (Zx;; /x,)/2 + 0.5. Note, the above formula used only for mapping samples from modality M to A' for which pre-trained model Opyis available.
Another work, SIgLIP, demonstrates robustness to label noise and reduces computational requirements during contrastive training [62]. It does by avoiding computation of softmax over the entire set of in-batch samples, instead relying on element-wise sigmoid operation:
LSiLU N
exp (yegl log exp
(3)
(2x,, km,) + b))
5

---

Algorithm S2L loss pseuud-immllemennation.
1: # mol_emb
molecule model embedding [n, dim]
2: phn_ emb
phenomics model embedding [n, dim]
3: # t-prime, b
learnable temperature and bias mini-baton size
4: n
5: # ()
custom similarity function
6: 7.5
similarity dampening parameters
7:
8: exp(t_prime)
9: zmol = 12_ normallzeemool_en
10: zphn 12_ 12__ormalize (phn_emb)
11: logits
dot (zmol, zphn.T) t + b
12: sim_ _am_matrix
(zphn, zphn.T) [n, nl sample similarity matrix 13: pos log- sigmoid
14: neg _og_ piggiimi logits)
15: 1 sim_matrix * pos (y sim_matrix) * neg
16: 1 sum(1) /
In equation and b are learned, calibrating the model confidence conditioned the ratio of positive to negative pairs. Vi, is set to 1 if = and -1 otherwise.
Inspired by prior works, we introduce S2L for molecular representation learning, which leverages inter-sample similarities and robustness to label noise to mitigate weak or inactive perturbations.
w exp (a(zx, kmy) + 8))
Sa.
log
+
cxp -a((zx cxp
Amy) b))
1=1
1=
In the equation above, 2x, and vm; correspond to latent representations of images and molecules, respectively. a and correspond learnable temperature and bias parameters for the calibrated sigmoid function. wi is an inter-sample similarity function computed from images using the
pretrained model Opn- To compute w:X, similarity, as was the case for Equation (more details in Appendix (D3).. Intuitively, S2L can be thought of as shifting from multi-class classification soft multi-labe problem. In our problem setting, the labels are continuous and determined by sample similarity in the phenomics space.
the arctangent of L2 distance instead of cosine
When training molecular-phenont model, mitigating the effect of inac- tive molecules in training data distribution can be carried out by undersampling inactive molecules and using inter-sample similarity aware, S2L loss (equation 4
Guideline
Challenge Variable Concentrations
Perturbation effect a cell is determined by both molecular structure and corresponding concentra- tion [58]. model capturing molecular impact on cell morphology must be able to generalize across different doses, since variable concentrations can correspond to different data distributions.
We note that providing concentrations c; as input to the model would benefit performance, this would indicate the magnitude of molecular impact. However, we find that simply concatenating concentrations does result in effective training due its compressed dynamic range. To that end, we add concentration information in two separate ways: implicit and explicit formulations.
We add implicit concentration molecular perturbation classes by using the S2L loss (Equation 4 to treat perturbation m; with concentrations c; and cj as distinct classes. This pushes samples apart in the latent space proportionally to similarities between phenomic experiments.
We add explicit concentration 6; by passing it to the molecular encoder. We explore different formulation for dosage concentrations, f' (c;), where f' maps e₁ - E. Encoded representations f' f'(c:)

---

are concatenated at the initial layer of the model. We find simple functional encodings f' (such as one-hot and logarithml to work well in practice.
When training molecular- phenommi model, conditioning (implicit and explicit) representation of concentration f' (ccc) ) aids in capturing molecular impacts on cell morphology and improves generalization to previously unseen molecules and concentrations.
Guideline 3
4 Experimental Setup
In this section, we describe evaluation datasets used, and descriptions of the underlying data modalities. To assess phenomolecular retrieval, we use 1% recall metric unless stated otherwise, as it allows direct comparison between datasets with different number of samples. Additional implementation and evaluation details can be found in Appendix D
Datasets: Our training dataset consists of fluorescent microscopy images paired with molecular structures and concentrations, which are used as perturbants. We assess models' phenomolecular retrieval capabilities on three datasets of escalating generalization complexity. First dataset, consisting of unseen microscopy images and molecules present in the training dataset. Second, a dataset consist- ing of previously unseen phenomics experiments and molecules split by the corresponding molecular scaffold. Finally, we evaluate on an open source dataset with a different data generating distribution [16]. In the case of the latter two datasets, the model is required to perform zero-shot classification, as it has no access those molecules in the training data. This requires the model to reason over molecular graphs to identify structures inducing corresponding cellular morphology changes. Using methodology described in guideline [2) we report retrieval results for all molecules as well as on an active subset. Finally, all datasets are comprised of molecular structures at multiple concentrations (001,.1,100 1.0, 10, etc.) Additional details regarding the datasets can be found in Appendix C
Modality Representations: In our evaluations, we consider different representations for molecular perturbations and phenomic experiments and quantitatively evaluate their impact.
Images: Image encoders utilize 6-channel fluorescent microscopy images of cells representing phenomic experiments. Images are 2048 x 2048 pixels, capturing cellular morphology changes post molecular perturbation. We downscale each image to 256 256 using block mean downsampling.
Phenom1: We characterize phenomic experiments by embedding high resolution microscopy images in the latent space of a phenomics model 0p as described in guideline !
Fingerprints:
Molecular fingerprints utilize RDKIT (31, MACCS [30] and MORGAN3 [46) bit
coding, which represent binary presence of molecular substructures. Additional information such atomic identity, atomic radius and torsional angles are included in the fingerprint representations.
MolGPS: We generate molecular representations from large pretrained GNN. Specifically, we obtain molecular embeddings from a IB parameter MPNN 34
Table 1: Impact of pre-trained Phenoml and MolGPS on CLOOME and MolPhenix for matched number of seen samples (Top), where we observe an 8.1 x improvement of MolPhenix over the CLOOME baseline for active unseen molecules. SOTA results trained with higher number of steps by utilizing the best hyperparameters (Bottom *). We note that MolPhenix's main components such as S2L and embedding averaging relies on having a pre-trained uni-modal phenomics model.
Active Molocules
All Molecules
Method
Medality
Unseen
Unseen Im. Mol.
Unseen Dataset
Unseen Im.
Im.
Unseen Mol.
Unseen Dutaset
CLOOME
Images& & Muli-ePS
.U766 00042
0087 + 0003
00228 + 0U57
.1447 +: .0028
0661 +: 0020
.0225 +: .014
CLOOME
Pacnoml Multi-FPS
.1699
.0012
.305
1 0004
.0665 2 01:18
.009
.0033
2171 1: 0013
.1337 .0645
Mollbanii
Picnom Mulu-FFS
7807
.0044
3655
0017
.213
MolPbenix
Phenoml MolGPS
.7466
.0014
.6387
à. .0056
4160
0016
.5012
.0002
3511
1: .004
2508
.0026
MolPbenix
Phenoml MalGPS
9689
.0017
.7733
à: .0336
.8800
0082
.5883
.0007
.324 1: 0016
2809
0060

---

Table Top-1% recall accuracy with use of the proposed MolPhenix guidelines, such as Phenom! and embedding averaging. We omit explicit concentration from this experiment.
Active Molecules
All Molecules
Unseen Images
Unseen Im. + Mol.
Unseen
Unscen Images
Unseen Mol.
Unseen
Loss
Dutaset
Im.
Dataset
CLIP
.3333
.0013
.128- = .0088
.1514 = .0388
.1761 + .0013
.1867
.0022
.0331 .0022
Hopfield-CLa CLIP
.2788
0042
3559
0042
.1256 + .0022
.1531 + 0046
.1709
.0029
.U73 0020
InfoLOOB
3351
00011
.4206
0031
.1563 + 00088
.1746 + 0003
.1860
.0009
.0755 1: .0099
CLOOME
3572
0026
0039
.1658 0063
.1668
.026
00022
DCL
.6333
0025
.6177
.0047
.3884 + 0007
.377 + 0047
.2562
00008
.1364 + .0067
CWCL
.7091 = 0015
.5299
= 0020
3356 - 0094
3635 + 0064
.2696
0019
.1526 + 0058
SigLip
0045
0065
.396
.3729
0014
.0038
S2L (ours)
9097 0020
6759
0012
4181
= 0012
.488 + 0009
2852 + 0001
.1838 + .0007
Table 3: Top- 1% recall accuracy across different concentration encoding choices with use of the proposed MolPhenix guidelines, such as Phenoml and embedding averaging.
Active Molecules
All Molecules
Implicit Concentration
Explicit
Unseen Im.
Unscon Im. Mol.
Unscon Dutaset
Unseen Im.
Unseon Mol.
Unseen Dataset
Concentration
Im.
.7350
.0071
.6509 .0104
.3333
.0004
.3610
.0025
.2688
.0044
.1532
.0017
.9097
.0020
.6759 d: 00122
.1811
.0012
.4688
.0009
.852
.0011
.1838 0007
sigmoid
.9423
0011
.7155
+ 00166
.453
.0022
.5071
.0024
.341
00026
.1444 0002
logarithm
.7451
.3700
one -ot
94130
0000
.7490
00552
4850
=00000
.5433
.0030
.3819
.0002
.384
.0049
Results and Discussion
To evaluate the effecttivneess of Guidelines I and 3 we carry out evaluation in two different settings: (1) cumulative concentrations, and (2) held-out concentrations, testing the models' ability to generalize new molecular doses. Finally, we perform comprehensive ablations testing model performance with varying data, model, and optimization parameters. The comprehensive set of results
can be found in Tables 10
I2 and 13
5.1 Evaluation on cumulative concentrations:
We demonstrate improvements in phenomolec- ular recall due to usage of a phenomics pre-trained foundation model, identify that MolPhenix set of design choices results in higher final performance, and more data effi- cient learning. Figure demonstrates recall ac- curacy on all molecules and an active subset for CLOOME and MolPhenix models, function of training samples seen.
Molecular Retrieval
0.7
Molphenis Phenoml
Compounds Activee Compounds
0.6
CLOOME-Amee -Eaggsss
Top 1% Retrieval
0.5
0.4
0.3
We observe large performance gap between models trained on Phenoml embeddings as op- posed to images, emphasizing the utility of us- ing pre-trained encoder for microscopy im- ages (Table [T. We note that provision of Phenom! (CLOOME-Phenone
0.2
0.1
Vs CLOOME- Images) significantly improves both active and all molecule retrieval by 5.69x and, 4. 4.75x re- spectively (Table I..
0.0
106
107
Number of Samples Seen
Figure 4: Comparison of training phenomic en- coder from scratch and utilizing pre-trained Phe- noml unseen dataset. X-axis plotted on logarith- mic scale.
Furthermore, we identify that while all molecules retrieval stagnates throughout train- ing, the performance on an active subset keeps improving, underscoring the importance of iden- tification of the active subset. Finally, we com- pare CLOOME and MolPhenix trained using Phenoml embeddings and find there is consistent

---

retrieval performance gap, throughout training, with 1.26 x final improvement (Figure 4 Table D Compared to CLOOME 48 trained directly on images, MolPhenix achieves an average improve- ment of 8.78x active molecules on the unseen dataset. These results verify the effectiveness of Guideline Tin accelerating training, and the importance of Guidelines and recall improvements over CLOOME.
We evaluate the impact of different loss objectives on the proposed MolPhenix training frame- work. Table 2 presents top-1% %% retrieval accuracy across different contrastive losses utilized to train molecular- molecular-phenomiis phenomies encoders on cumulative concentrations. Compared to prior methods, the proposed S2L loss demonstrates improved retrieval rates in cumulative concentration setting. Label noise and inter-sample similarity aware losses such as CWCL and SigLip also demonstrate improved performance. The effectiveness of S2L can be attributed to smoothed inter-sample similarities and implicit concentration information.
we observe recall improvements when considering both molecular structures and concentration. We note the importance of the addition of implicit concentration, further confirming the importance of considering molecular effects at different concentrations as different classes. Explicitly encoding molecular concentration with one-hot, logarithh thm and si sigmoid yields improved recall performance, where one-hot performs the best in cumulative concentration setting. These findings verify the efficacy of implicit and explicit concentration encoding outlined in Guideline
Finally, in Table 3
Table 4: Top-1 % recall accuracy of dif- Table 5: Top- % recall accuracy across different concentra- ferent loss objectives while using the tion encoding choices while using the proposed MolPhenix proposed MolPhenix guidelines, such as guidelines, such as Phenom] and embedding averaging. Phenoml and embedding averaging.
Loss
Unseen Im.
Unseen Im. + Mol.
Unseen
Implicit Concentration
Explicit
Unseen Im.
Unseen Im. + Mol.
Unseen
Concentration
Dataset
CLIP
.1099
2425
.1519
x
5942
.4315
31229
Hopfelld CLIP
.1581
2034
InfoL.0On
.1222
2496
.1501
x
.834
.4615
.3792
CLOOME
.1644
2461
.1479
sigmoid
8256
.4692
.3765
DCL
.4177
.4077
.2841
logarithm
.7953
.4466
.3664
CWCL
.5731
4403
.3232
one-hot
.7489
.4088
.3379
SigLip
5518
.2177
.3021
S2L (ours)
.8334
.4615
.3792
Results are averaged across experiments for each dropped concentration, and across three seeds. Recall reported for active molecules, while the results for all molecules can be found in Table [3
5.2 Evaluation on held-out concentrations:
Next, we evaluate recall on held-out concentrations obtain a measure of generalization performance. This evaluation allows us to capture the utility of our models for prediction of unseen concentrations, hence resembling in-silico testing. We omit concentrations from the training set and evaluate recall at the excluded data, where we observe drop in retrieval performance for unseen concentrations. Similar to cumulative concentration results, we find that using S2L improves recall over other losses and outperforms CLOOME by up to 126% (T While one-hot encoding exhibits significant improvements cumulative concentrations, its expressivity on unseen concentrations is limited (Table 55 and sigmoid encoding provides a sufficient representation of concentration.
5.3 Ablation Studies
We assess the importance of our design decisions by conducting an ablation study over our proposed guidelines. Figure Figure/Ipresents the variation of top- recall accuracy across key components such as cutoff value, fingerprint type, and embedding averaging. We observe that employing lower cutoff p value yields improved generalization for unseen dataset, while employing a higher cutoff appears to be optimal for unseen images + unseen molecules. For molecular structure representations, we find that using embeddings from the large pretrained MPNN graph based model (e.g., MolGPS) surpasses traditional fingerprints. Finally, utilization of embedding averaging demonstrates improved recall.

---

1.0
1.0
1.0
0.8
0.8
Embbdding Dimension Bog
Number
scalo)
0.6
0.6
0.4
0.4
0.3
0.01
MACOS
BDK
Mocean
Mult
Cut-Olf poulus Gog aeals?
Pingerprms
minn Averaging
Unscen Images
Unanen Imagns
Unsoen Molemijes
Unseon Dotaset
Figure Ablations of top-1 recall accuracy with (top-left) the size of embedding dimension, (top-center) number of parameters, (top-right) batch size, (bottom-left) cutoff value, (bottom- center) fingerprint type, and (bottom-right) random batch averaging. Compact embedding sizes from pretrained models, larger number of parameters, larger batch sizes, lower cutoff p-values, pretrained MolGPS fingerprints and presence of random batch averagin improving retrieval of our MolPhenix framework.
Conclusion
In this work, investigate the problem of contrastive phenomolecular retrieval by constructing joint multi-modal embedding phenomic experiments and molecular structures. We identify a set of challenges afflicting polecular-phenomit training and proposed a of guidelines for improving retrieval and generalization. Empirically, we observed that contrastive learners demonstrate higher retrieval rates when using representations from a high-capacity uni-modal pretrained model. Use of inter-sample similarities with a label noise resistant loss such as S2L allows us to tackle the challenge of inactive molecules. Finally, adding implicit and explicit concentrations allows models to generalize to previously unseen concentrations. MolPhenix demonstrates an 8.1 x improvement in zero shot retrieval of active molecules over the previous saate-of-the-ard reaching 77.33% in top- accuracy. In addition, we conduct preliminary investigation on MolPhenix ability to uncover biologically meaningful properties (activity prediction, zero-shot biological perturbation matching, and molecular property prediction in Appendix E F.2, and E.3 respectively.). We expect wide range applications for MolPhenix, particularly in drug discovery. While there's a remote chance of misuse for developing chemical weapons, such harm is unlikely, with our primary focus remaining on healthcare improvement.
While our study covers challenges in phenomolecular recall, we leave three research directions for future work. (1) Future investigations could consider studying additional modalities such as text, genetic perturbations and chemical multi-compound interventions. (2) While we propose and evaluate our guidelines on previously conducted phenomic experiments, we note that rigorous evaluation would evaluate model predictions in a wet-lab setting. (3) In addition, our work makes the assumption that the initial unperturbed cell state za can be marginalized by utilizing a single cell line with unperturbed genetic background. Future works can relax this assumption, aiming to capture innate intercellular variation.
Limitations and Future Work:
Acknowledgments and Disclosure of Funding
We thank the broader Valence Labs team and Recursion Pharmaceuticals for support the project. We thank Berton Earnshaw, Jason Hartford, Emmanuel Bengio, Oren Kraus, and Emmanuel Noutahi for providing valuable feedback the manuscript.
10

---

References
[1] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech. I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong, S. Samangooei, M. Monteiro, J. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. Binkowski, R. Barreira,
O. Vinyals, A. Zisserman, and K. Simonyan. Flamingo: visual language model for few-shot learning, 2022.
[2] S. Albelwi. Survey on self-supervised learning: auxiliary pretext tasks and contrastive learning methods in imaging. Entropy, 24(4):551, 2022.
[3] R. Balestriero, M. Ibrahim, V. Sobal, A. Morcos, S. Shekhar, T. Goldstein, F. Bordes, A. Bardes,
G. Mialon, Y. Tian, A. Schwarzschild, A. G. Wilson, J. Geiping, Q. Garrido, P. Fernandez,
A. Bar, H. Pirsiavash, Y. LeCun, and M. Goldblum. A cookbook of self-supervised learning, 2023.
[4] C. Bock, P. Datlinger, F. Chardon, M. A. Coelho, M. B. Dong, K. A. Lawson, T. Lu. L. Maroc, M. Norman, B. Song, G. Stanley, S. Chen, M. Garnett, W. Li, J. Moffat, I.. Qi, R. Shapiro, J. Shendure, S. Weissman, and X. Zhuang. High-content crispr screening. Nature Reviews Methods Primers, 2(1), Feb. 2022.
[5] R. Bohacek, C. McMartin, and W. C. Guida. The art and practice of structure-based drug design: A molecular modeling perspective. Medicinal Research Reviews, 16(1):3-50, Jan. 1996.
[6] M. Boutros, F. Heigwer, and C. Laufer. Microscopy-basee high-content screening. Cell, 163(6):114-1325, 2015.
[7] M.-A. Bray, S. Singh, H. Han, C. T. Davis, B. Borgeson, C. Hartland, M. Kost-Alimova, S. M. Gustafsdottir, C. C. Gibson, and A. E. Carpenter. Cell painting, high-content image- based assay for morphological profiling using multiplexed fluorescent dyes. Nature protocols, 117))1757-1774, 2016.
[8] M.-A. Bray, S. Singh, H. Han, C. T. Davis, B. Borgeson, C. Hartland, M. Kost-Alimova, S. M. Gustafsdottir, C. C. Gibson, and A. E. Carpenter. Cell painting, a high-content image- based assay for morphological profiling using multiplexed fluorescent dyes. Nature protocols, 1199):757-17744 2016.
[9] J. C. Caicedo, S. Cooper, F. Heigwer, S. Warchal, P. Qiu, C. Molnar, A. S. Vasilevich, J. D. Barry, H. S. Bansal, O. Kraus, et Data-analysis strategies for image-based cell profiling. Nature methods, (4(9):849-863 2017.
[10] S. Cao, P. Xu, and D. A. Clifton. How to understand masked autoencoders. arXiv preprint arXiv:2202.03670, 2022.
[11] S. N. Chandrasekaran, Ackerman, E. Alix, D. M. Ando, J. Arevalo, M. Bennion, N. Boisseau, A. Borowa, J. D. Boyd, L. Brino, P. J. Byrne, H. Ceulemans, C. Ch Ch'ng, B. A. Cimini, D.-A. Clevert, N. Deflaux. J. G. Doench, T. Dorval, R. Doyonnas, V. Dragone, O. Engkvist, P. W. Faloon, B. Fritchman, F. Fuchs, S. Garg, T. J. Gilbert, D. Glazer, D. Gnutt, A. Goodale,
J. Grignard, J. Guenther, Y. Han, Z. Hanifehlou, S. Hariharan, D. Hernandez, S. R. Horman, G. Hormel, M. Huntley, I. Icke, M. Tida, C. B. Jacob, S. Jaensch, Khetan, M. Kost-Alimova, T. Krawiec D. Kuhn, C.-H. Lardeau, A. Lembke, F. Lin, K. D. Little, K. R. Lofstrom, S. Lotfi, D. J. Logan, Y. Luo, F. Madoux, P. A. Marin Zapata, B. A. Marion, G. Martin, N. J. McCarthy, L. Mervin, L. Miller, H. Mohamed, T. Monteverde, E. Mouchet, B. Nicke, A. Ogier, A.-L. Ong, M. Osterland, M. Otrocka, P. Peeters, J. Pilling, S. Prechtl, C. Qian, K. Rataj, D. E. Root, S. K. Sakata, S. Scrace, H. Shimizu, D. Simon, P. Sommer, C. Spruiell, I. Sumia, S. E. Swalley, H. Terauchi. A. Thibaudeau, A. Unruh, J. Van de Waeter, M. Van Dyck, C. van Staden, M. Warchoi, E. Weisbart, A. Weiss, N. Wiest-Dacssle, G. Williams, S. Yu, B. Zapiec, M. Zyła, S. Singh, and A. E. Carpenter. Jump cell painting dataset: morphological impact of 136,000 chemical and genetic perturbations. bioRxiv, 2023.
[12] T. Chen, S. Kornblith, K. Swersky, M. Norouzi, and G. E. Hinton. are strong semi-supervised learners. Advances neural information processing systems, 33:22243-2222 2020.
Big self- self-supervised models
11

---

[13] M. Dehghann, J. Djolonga, B. Mustafa, P. Padlewski, J. Heek, J. Gilmer, A. P. Steiner, M. Caron, R. Geirhos, I. Alabdulmohsin, et al. Scaling vision transformers 22 billion parameters. In International Conference on Machine Learning, pages 7480-7512, PMLR, 2023.
[14] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbomn X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Hcigold, S. Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 19929, 2020.
[15] K. Dunn, A. Aotaki-Keen, F. Putkey, and L. Hjelmeland. Arpe-19, a human retinal pigment epithelial cell line with differentiated properties. Experimental eye research, 62(2):155-170, 1996.
[16] M. M. Fay, O. Kraus, M. Victors, L. Arumugam, K. Vuggumudi, J. Urbanik, K. Hansen, S. Celik, N. Cernek, G. Jagannathan, et al. Rxrx3: Phenomics map of biology. Biorxiv, pages 2023-02, 2023.
[17] C. Feichtenhofer, Y. Li, K. He, et al. Masked autoencoders as spatiotemporal learners. Advances in neural information processing systems, 35:35946-35958 2022.
[18] A. Fürst, E. Rumetshofer, J. Lehner, V. T. Tran, F. Tang, H. Ramsauer, D. Kreil, M. Kopp, G. Klambauer, A. Bitto, et al. Cloob: Modern hopfield networks with infoloob outperform clip. Advances in neural information processing systems, 35:20450-20466 2022.
[19] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z.. D. Guo, M. G. Azar, B. Piot, K. Kavukcuoglu, R. Munos, and M. Valko. Bootstrap your own latent: A new approach to self-supervised learning, 2020.
[20] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[21] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning, 2020.
[22] M. Hofmarcher, E. Rumetshofer, D.-A. Clevert, Hochreiter, and G. Klambauer. Accurate prediction of biological assays with high-throughput microscopy images and convolutional networks. Journal of chemical information and modeling. 59(3):1 1171, 2019.
[23] S. Huang, L. Dong, W. Wang, Y. Hao, S. Singhal, S. Ma, T. Lv, L. Cui, O. K. Mohammed, B. Patra, Q. Liu, K. Aggarwal, Z. Chi, J. Bjorek, V. Chaudhary, S. Som, X. Song, and F. Wei. Language is not all you need: Aligning perception with language models, 2023.
[24] W. E. Johnson, C. Li, and A. Rabinovic. Adjusting batch effects in microarray expression data using empirical bayes methods. Biostatistics, 8(1):1 (1):118-127, 2007.
[25] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Rad- ford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08.0
2020.
[26] P. Kemmeren, K. Sameith, L. A. Van De Pasch, J. J. Benschop. T. L. Lenstra, T. Margaritis, H. O'Duibhir, F. Apweiler, S. van Wageningen, C. W. Ko, et al. Large-scale genetic perturbations reveal regulatory networks and an abundance of gene-specific repressors. Cell, 157(3):40--552 2014.
[27] C. Knox, M. Wilson, C. M. Klinger, M. Franklin, E. Oler, A. Wilson, A. Pon, J. Cox, N. E. L. Chin, S. Strawbridge, M. Garcia-Patino
Kruger, A. Sivakumaran, S. Sanford, R. Doshi, N. Khetarpal, O. Fatokun, D. Doucet, A. Zubkowski, D. Y. Rayat, H. Jackson, K. Harford, A. Anjum, M. Zakir, Wang, Tian, B. Lee, J. Liigand, H. Peters, R. Q. R. Wang, T. Nguyen, D. So, M. Sharp, R. da Silva, C. Gabriel, J. Scantlebury, M. Jasinski, D. Ackerman, T. Jewison, T. Sajed, V. Gautam, and S. Wishart. Drugbank 6.0: the drugbank knowledgebase for 2024. Nov. 2023.
Nucleic Acids Research, 52(DI):D
12

---

[28] O. Kraus, K. Kenyon-Decs Deann, S. Saberian, M. Fallah, P. McLean, J. Leung, V. Sharma, A. Khan, J. Balakrishnan, S. Celik, al. Masked autoencoders for microscopy are scalable learners of cellular biology. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11757-11766. 2024.
[29] O. Z. Kraus, B. T. Grys, J. Ba. Y. Chong, B. J. Frey, C. Boone, and J. Andrews. Automated analysis of high-content microscopy data with deep learning. Molecular systems biology. 13(4)2224, 2017.
[30] H. Kuwahara and X. Gao. Analysis of the effects of related fingerprints molecular similarity using an eigenvalue entropy approach. Journal of Cheminformatiis, 13:1-12, 2021.
[31] G. Landrum et al. Rdkit: A software suite for cheminformatics, computational chemistry, and predictive modeling. Greg Landrum, 828,, 2013.
[32] F. Lanusse, L. Parker, S. Golkar, M. Cranmer, A. Bietti, M. Eickenberg, G. Krawezik, M. Mc- Cabe, Ohana, M. Pettee, B. R.-S. Blancard, Tesileanu, K. Cho, and S. Ho. Astroclip: Cross-modal pre-training for astronomical foundation models, 2023.
[33] T. Leek, R. B. Scharpf, H. C. Bravo, D. Simcha, B. Langmead, W. E. Johnson, D. Geman, K. Baggerly, and R. A. Irizarry. Tackling the widespread and critical impact of batch effects in high-throughput data, Nature Reviews Genetics, 11(10):333733, 2010.
[34] D. Masters, J. Dean, K. Klaser, Z. Li, S. Monddrell-Mander A. Sanders, H. Helal, D. Beker, A. Fitzgibbon, S. Huang, et al. Gps++: Reviving the art of message passing for molecular property prediction. arXiv preprint arXiv:2302.029.. 2023.
[35] D. Mendez, A. Gaulton, A. P. Bento, J. Chambers, M. De Veij, E. Félix, M. P. Magariños, J. F. Mosquera, P. Mutowo, M. Nowotka, et Chembl: towards direct deposition of bioassay data. Nucleic acids research, 47(D1))D930-D940, 2019.
[36] O. Méndez-Lucio, C. Nicolaou, and B. Earnshaw. Mole: molecular foundation model for drug discovery, 2022.
[37] C. Q. Nguyen, D. Pertusi, and K. M. Branson. Molecule morphlll contrastive pretraining for transferable molecular representation. bioRxiv, pages 2023-05, 05, 2023.
[38] A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.088: 1807.0374, 2018
[39] H. S. Parker and J. T. Leck. The practical effect of batch on genomic prediction. Statistical applications genetics and molecular biology. 11(3), 2012.
[40] Z. Pincus and J. Theriot. Comparison of quantitative methods for cell-shape analysis. Journal of microscopy, 227(2):440-156, 2007.
[41] B. Poole, S. Ozair, A. Van Den Oord, A. Alemi, and G. Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pages 5171-5180, PMLR. 2019.
[42] D. I.. Purich. Enzyme kinetics: catalysis and control: reference of theory and best-practice methods. Elsevier, 2010.
[43] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 9748-8763. PMLR, 2021.
[44] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. 2019.
[45] H. Ramsauer, B. Schäfl, J. Lehner, P. Seidl, M. Widrich, T. Adler, L. Gruber, M. Holzleit- ner, M. Pavlovié, G. K. Sandve, et al. Hopfield networks all you need, arXiv preprint arXiv:2008.02217, 2020.
13

---

[46] D. Rogers and M. Hahn. cxtended-commectivity fingerprints. Journal of chemical information and modeling. 50(5):742-75, 2010.
[47] Y. Rong, Y. Bian, T. Xu, W. Xie, Y. Wei, W. Huang, and J. Huang. Self-supervised graph transformer on large-scale molecular data, 2020.
[48] A. Sanchez-Fermander E. Rumetshofer, S. Hochreiter, and G. Klambauer. Cloome: contrastive learning unlocks bioimaging databases for queries with chemical structures. Nature, 2023.
[49] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman. M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, al. Laion-5b; An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278 25294,, 2022.
[50] Simm, G. Klambauer, A. Arany, M. Steijaert, K. Wegner, Gustin, Chupakhin, Y. Chong. J. Vialard, P. Buijnsters, et al. Repurposing high-throughput image assays enables biological activity prediction for drug discovery. Cell chemical biology. (5(5)661-661 2018,
[51] K. Sohn. Improved deep metric learning with multi-class n-pair loss objective. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, page 1857-1865, Red Hook, NY, USA, 2016. Curran Associates Inc.
[52] C. Soneson, S. Gerster, and M. Delorenzi. Batch effect confounding leads to strong bias in performance estimates obtained by cross-validation. PloS one, 96) 100335, 2014.
[53] R. S. Srinivasa, J. Cho, Yang. M. Saidutta, C.-H. Lee, Y. Shen, and Jin. Cwel: Cross- modal transfer with continuously weighted contrastive loss. Advances in Neural Information Processing Systems, 36, 2023.
[54] D. R. Stirling, M. J. man. Cellprofiler 4: improvements in speed, utility and usability. BMC bioinformatics, 22:1-11, 2021.
A. M. Lucas, A. E. Carpenter, B. A. Cimini, and A. Good-
[55] M. Sypetkowski, M. Rezanejad, S. Saberian, O. Kraus, J. Urbanik, J. Taylor, B. Mabey, M. Victors, J. Yosinski, A. R. Sereshkeh, et al. RxIxl: dataset for evaluating experimental batch correction methods. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4284 4293, 2023.
[56] M. Sypetkowski, F. Wenkel, F. Poursafaei, N. Dickson, K. Suri, Fradkin, and D. Beaini. On the scalability of foundational models for molecular graphs. arxiv, 2024.
[57] F. Vincent, A. Nueda, J. Lee, M. Schenone, M. Prunotto, and M. Mercola. Phenotypic drug discovery: recent successes, lessons learned and new directions. Nature Reviews Drug Discovery, 21(12):999-914, 2022.
[58] R. M. Walmsley and N. Billinton. How accurate is in vitro prediction of carcinogenicity? British Journal of Pharmacology, 162(6):250-1258,, Feb. 2011.
[59] C. Wang, S. Gupta, C. Uhler, and T. Jaakkola. Removing biases from molecular representations via information maximization, 2023.
[60] R. Xie, K. Pang, G. D. Bader, and B. Wang. Maester: Masked autoencoder guided segmentation at pixel resolution for accurate, self-supervised subcellular structure recognition. In 2023 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, June 2023.
[61] S. Zaidi, M. Schaarschmidt, J. Martens, H. Kim, Y. W. Teh, A. Fanchez-Gonzalez P. Battaglia, R. Pascanu, and Godwin. Pre-training via denoising for molecular property prediction, 2022.
[62] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre- training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975-11988 2023.
[63] X. Zhai, X. Wang, B. Mustafa, A. Steiner, D. Keysers, A. Kolesnikov, and L.. Beyer. Lit: Zero-shot transfer with locked-image text tuning, 2022.
14

---

[64] S. Zheng, J. Rao. J. Zhang, L. Zhou, J. Xie. E. Cohen. W. Lu, C. Li, and Y. Yang. Cross-modal graph contrastive learning with cellular images. Advanced Science, 11(32), June 2024.
[65] Y. Zhong, H. Tang, J. Chen, J. Peng, and Y.-.. Wang. Is self-supervised learning more robust than supervised learning? arXiv preprint arXiv:2206.05259) 2022.
[66] G. Zhou, Z. Gao, Q. Ding. H. Zheng, H. Xu, Z. Wei, L. Zhang, and G. Ke. Uni-mol: A universal 3d molecular representation learning framework. The Eleventh International Conference on Learning Representations, 2023.
15

---

7 NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: In the abstract, we claim that we build multi-modal molecular-phene phenomies model and demonstrate improvements over prior works. This is done by taking using uni-modal pre-trained phenomics model, tackling inactive molecules by undersampling and learning nnter-sample similarities. In addition, we take into account concentration in our model training. We demonstrate comprehensive results supporting these claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the conclusion, we have a limitations subsection discussing future research directions and assumptions in our work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
16

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
Our work does not contain proofs or theorems.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our work documents design decisions in detail and has comprehensive details about the underlying dataset. We document all our hyperparameter choices and model architectural decisions. Our evaluation is performed allowing for benchmarking of other methods. To reproduce the pre-trained phenomics model, we base our architecture the work from [28), for which they have also provided access to a snakker model, namely Phenom-Beta via a web platform hosted on the BioNeMo platform /icttss:/wweet we based our architecture on [56]. for which the authors provide all the code and data needed to reproduce it. We further note that the molecular model can be replaced by simple molecular fingerprints with only a slight drop in performance.
publicly accessible dataset RXRX3,
ai / phenom To reproduce the pre-trained molecular model,
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
17

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they (c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: As part of the submission, we unable to provide code to reproduce model training due to use of its proprictary nature. The training dataset also an asset a private institution, meaning that we unable to be made publicly accessible. The unseen dataset RXRX3 is, however, open source and can be used by the community to evaluate public phenomics models.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc)
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips. //nips.ccppuliee wtt.cccppppllccc waablic/guiies/Coodoeooo Sttp:/oooSSSmisiioonneoe
swssssiiicciiiinci for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We provide details regarding our hyperparameter choices in the Appendix In addition we document the of scaffold splitting for Unseen Molecules & Images dataset. Unseen Dataset RXRX3 is publicly accessible.
C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
18

---

7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Our reported results averaged over random seeds used to initialize the model and dictating stochasticity during model training. We report most standard deviations in the main text, and the remaining ones all present the Appendix E
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative етror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide details on compute time for each experiment in Appendix D.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:p////haaaasscccccppp//hhhht
Answer: [Yes]
Justification: The research described does not violate the NeurIPS Code of Ethics. Our experiments do not include human subjects, we follow fair use of data, privacy, and do not release model weights for mitigating impact measures.
Guidelines:
19

---

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our work discusses the potential in which MolPhenix can have positive societal impact and we touch on the extenralities in our concluding statements.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
In our work we do not release model weights the underlying code.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
20

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Assetts used are referenced and licenses checked or otherwise not released publicly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing not human subject research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
21

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing not human subject research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22

---

A Glossary
Cell morphology: The examination and characterization of cellular structure, including shape, size, and organization, which can provide insights into cellular function and health [40].
Cell line (ARPE-19): A specific immortalized human retinal pigment epithelial cell line, widely utilized in ophthalmic research due to its differentiated properties and stability [15].
Molecular concentration: The quantitative measure of specific molecule's 's abundance within
defined volume, typically expressed in molar units 422]
Cell staining: laboratory technique involving the application dyes or fluorescent markers to enhance the visibility and differentiation of cellular components under mieroscopic examination [9]
Molecular perturbations: Induced alterations cellular molecular systems, often used to study cellular responses and regulatory networks [26].
Inactive molecule perturbations:
Cellular system alterations caused by molecules lacking significant
biological activity.
Systematic non-biological variation between groups of samples in experiment, resulting from technical or experimental factors rather than true biological differences. This phc- nomenon is commonly observed in high- high-throughput molecular biology experiments, such as microar- ray studies, mass spectrometry, and single-cell RNA sequencing [24].
Batch effects:
Molecular fingerprints: Distinctive patterns of molecular features that characterize specific cellular states or responses, often used for comparative analyses and classification {
B Assumption of the Initial Cell State
There an important distinction between phenomics molecule and text - image contrastive training although there are initial similarities. In the text image domain the two modalities are directly generated by the same latent variable which is the underlying semantic class. Whereas in phenomics molecule, the observed phenomics variable is actually conditioned on molecular structure and the initial state. There are two important conclusions from this: (1) This indicates that if molecular structure has no effect on the initial cell state, there will not be positive pairing between the molecular structure and morphological patterns captured by phenomics, making it indistinguishable from control image. (2) There an underlying assumption that the initial cell state zQ constant. In accordance with this assumption we utilize experiments with fixed cell line, HUVEC-19 19, and a constant genetic background. Future works can relax this assumption by taking into account phenomics experiments of the cells prior to the perturbation. This can allow the models to generalize beyond single cell line and to diverse genetic backgrounds.
C Dataset
Models have been trained using our in house training set and we have conducted our evaluation on two novel datasets and an open-source molecule dataset (16]
Training Set: Our training dataset comprises 1,316,283 pairs of molecules and concentration
concentration combinations, complemented by fluorescent microscopy images generated through over 2,150,000 phenomic experiments.
Evaluation set Unseen Images + Seen Molecules: The first set consists of unseen images and seen molecules. Unseen microscopy images are associated with 15,058 pairs of molecules and concentrations from the training and selected randomly.
Evaluation set 2 Unseen Images + Unseen Molecules: The second set includes pre- viously unseen molecules. and images (consisting of 45,771 molecule and concentration pairs). Predicting molecular identities of previously unseen molecular perturbations corresponds to zero- shot prediction. Scaffold splitting was used split this validation dataset from training ensuring minimal information leakage.
Evaluation set Unseen Dataset: Finally, we utilize the RXRX3 dataset [66, an open- source out of distribution (OOD) dataset consisting of 6,549 novel molecule and concentration
23

---

pairs associated with phenomic experiments. The distribution of molecular structures differs from previous datasets, making this challenging zero-shot prediction task.
C.1 Concentration Details
Additional details regarding the number of molecules at significant concentrations of each evaluation set are available in Table 6
Table 6: Separated number of molecules for different concentrations various pvalue cut-offs
pvaluc=1.0
Unseen pvaluc=.1
Unseen pvalue=.01
Unseen Mol.
Coscentration
Unseen
Unseen
Unseen
Unseen
1497
1109
387
170
161
68
.25
1775
1111
1638
600
2033
237
134
121
165
1.0
2721
11392
16:19
1259
734
390
672
390
1787
4018
1636
1329
644
516
929
413
3.0
74
10454
0
12
1540
0
4
729
5.0
0
0
27
20
10.0
2712
11392
1636
2544
8117
792
2116
4815
25.0
2916
0
0
1734
0
0
950
Unique molocules
3026
14256
1639
2729
9857
823
2309
5778
642
D Implementation Details
In our experiments we report the top 1% recall metric as is agnostic to the size of the dataset used. Across different datasets, top metric can correspond to varying levels difficulty due the number of negatives evaluated. Top 1% can be used to compare models with different batch sizes, datasets, and evaluations with different number of concentrations.
D.1 Hyperparameters
Our design choices and utilized hyperparameters for is presented Table 7 We set batch size to 512 through experiments presented in top section of [an Figure since training CLOOME model on images is not efficient compared to using pretrained models. In addition, results presented at bottom section of Table I are based on the best parameters found through described ablation studies (section E.5)
Table 7: Hyperparameter values utilized in our proposed MolPhenix training framework for MolGPS version. For non-MolGPS version y 2.75 Ç is 1.0.
Parameter
Value
number
learning rite
1e-3
weight decay
3e-3
aptimizer
Adank
training hatch size
8192
validation
emhedding dim
512
model size structure
medium (38.7 M)
ResNet
Linear layer ResNet
layer
epochs
100
self similarity clip val
.75
leamahle temgerature initialization
2,302
leariable bias initialization
-1.0
Distance
arctangent
1.7
0.75
D.2 Resource Computation
We utilized an NVIDIA A100 GPU to train Molphenix using Phenoml and MolGPS embeddings, which takes approximately 44.75 hours each. For loss comparison experiments, we run each model using 3 different seeds and different losses, resulting total of hours of GPU processing time. For concentration experiments train runs, one for each concentration, with 3 seeds each

---

1.0
random mol
0.8
high pval
pval
Proportion
0.6
hight high-low
0.4
0.0
arctan distance
1.0
random mol
0.8
high pval
low pval
Proportion
0.6
hig I- low
0.0
casine distance
Figure 6: Plotted are cumulative densities of distance metrics for cosine similarity and arctangent of 12 distances between embeddings. Random mol corresponds Phenoml distances between random molecules, high pval corresponds to distances between molecules with high p-values, low pval corresponds to distances between active molecules with low p-values, finally high-low corresponds to distances between active and inactive molecules.
totaling 21 runs per set of parameters. With sets of parameters evaluated (3)) that amounts to 2,500 A100 compute hours. Moreover, we employed 8 NVIDIA A100 GPUs to train CLOOME model on phenomics images, with an average of 40 hour usage per run. Across three seeds, that amounts to 1000 hours of A100 GPU usage (8 GPUs for 40 hours 3 times).
Note that, without accounting for the time train Phenoml, MolPhenix is 8.4 faster than the CLOOME baseline.
D.3 S2L Distance function
To calculate inter sample distances, we utilize arctangent 12 distances between Phenom1 embed- dings. More specifically, we calculate distances with
4 π
arctan [xx 2x, 12//c
(5)
where is constant indicating the median 12 distance between null set of embeddings. Empirically, we've found that setting similarities below a threshold to improves model performance: [w]
Usage of arctan-12 distances is motivated by an observation that cosine similarities do not effectively separate inactive molecules from other molecular pairs (Figure 6). To alleviate inactive molecule challenge, we require significant separation of CDF curves of inactive perturbations (p value > .9) and active molecules (p < .01). We observe that in both the plots using arctangent and cosine similarities achieves this purpose. However, if we compare high p-value curves with high-low, we find that the case of cosine similarities they are almost identical. This indicates that the distribution of cosine similarities between active inactive molecules is almost identical to that of inactive inactive molecules. contrast, when using arctangent similarities, we observe that the two CDF curves are well separated.
25

---

This property of 12 distances can inform our model training to identify paactive-inactivy molecules. These results informed our decision utilize arctangent of 12 distances specify sample similarities for the S2L loss.
D.4 Intuition for S2L Loss
In this section we aim provide some additional intuition for the S2L loss and further relate it to previous works. We will first assess the conceptual similarities between InfoNCE and CWCL loss and then justify a similar extrapolation for the relationship between S2L and SigLIP losses.
InfoNCE can be considered a special case the CWCL loss where Way is set to O for all pairs of i and unless i j. Conceptually this is equivalent to stating that all the negative pairs are equally distant from the reference sample. We will consider uni directional loss CWCL, for identifying X from M:
exp (lZr; f/T E exp (iZnes Bmma bmma/T)
LCweL
Σ j=1
whi log
N Σラ=
If set Wij 0 when * and otherwise then the term EN - uy evaluates to and the above expression simplifies to:
LInFONCE 1 N log exp((Za,) Zmn \/T
NV Σ Lk=1 expl(z.n Zmn /TT)
In the case of CWCL, a non 0 way determined by within modality similarity function informed by pre-trained model, allows for an additional inductive bias. It is especially beneficial in training setting with a limited dataset-size and in the presence of inactive negative molecules.
Similarly SigLip can be considered special case of S2I. when w when i + j and wx = 1 in the case = j. This is the formulation of $21.
W a (zx, )my) +b +
Lsu.
log
N
+ exp
exp (a Zmn, + b)
can be simplified to SigLIP by setting w to when Vi thus setting the term (1-w?))
corresponding to i + j. we set w to 0 thus negating the first part of the Ls21. loss, evaluating
1+exp (22x 2my +0)
1 N
Sigh
log
+ exp
2m,) b))
Having 0 w allows to inform the training by going between discrete negative labels to continuous informed by some prior information. This information is given by a pre-trained encoder Ophi, in case but can be informed by any pre-trained model.
There are lot of domain specific choices that can be made to inform the choice for setting w which we discuss in the appendix. Briefly, we identify that modified /2 loss is most effective for identifying inactive molecules.
E
Additional Results
E.1 Predicting molecular activity
Given the significance identifying active molecules, we evaluate the ability of the chemical encoder to predict molecular activity. To do so, assessed whether embeddings generated from the chemical
26

---

encoder can be used to predict calculated p-values for unseen molecules. We fit logistic regression on molecular embeddings from the training set, classifying whether a molecular perturbation and concentration have a p-value below .01. We find that the trained logistic regression is capable of predicting molecular activity on two downstream datasets with a non-overlapping set of molecules, Figure 8 In addition, we provide a u-map of molecular embedding for the unseen dataset RXRX3, colored by p-value. We qualitatively observe a clustering of active molecules using a U-map (Figure 7. It demonstrates that predicting compounds activity is possible using MolPhenix chemical encoder as molecules representations are distinct, independent of the experimented dosage concentration.
27

---

UMAP Mol embeddings RXRX3
colored by p-values
0.01
0.1
1.0
unbedding clbme 2emiee
Concentration: 0. 0.25μM
Concentration: 1.0pM
UMAP Mot ansondii BORC
Concentration: 2.5μM
Concentration: 10.0pM
Figure 7: U-map demonstrating dimensionality reduction of the chemical embeddings of unseen dataset RXRX3. First two dimensions visualized and points are colored corresponding their activity measured phenomics experiments. Activity evaluated using p-values calculated using technical replicability of Phenoml embeddings. Top plot shows the u-map figure of all chemical embeddings, and bottom figure contains u-map figure representations at specific concentrations.
28

---

ROC curve New Dataset
ROC. curve Validation New Molecules
AUC-0.90
1.0
AUC-0.89
0.6
0.6
TPR
0.4
0.4
0.2
0.2
0.0
0.2
1.0
FPR
FPR
Figure 8: Top left: ROC AUC of logistic regression predicting molecular activity on new dataset. Top right: ROC AUC of logistic regression predicting molecular activity on validation dataset with new molecules and new images.
E.2 Zero Shot Biological Validation
We conduct preliminary investigation into whether MolPhenix can be used to identify biological relationships without the need for conducting the underlying experiments. To this end, we evaluate on subset of ChEMBL with curated pairs of gene knockouts and molecular perturbants [35). These pairs of perturbations were curated due to the similarity of their effects on cells, although these might not always be captured through phenomic experiments. Thus, there is maximum performance that can be reached through just phenomic data, which we assume to achieved by experimental data embedded using Phenom1.
To evaluate MolPhenix's 's ability to identify previously known biological associations directly from data, we embed phenomics experiments from gene knockouts using the vision encoder. To perform in-silico screening, we then embed the molecular structures associated with positive pairs using the chemical encoder. Generating molecular embeddings and the corresponding concentrations does not utilize any experimental data. We then calculate cosine similarities between embeddings of phenomics experiments evaluating gene knockouts, and representations of the chemical representations along with encoded concentrations. Using the computed cosine similarities we are then able to assess whether MolPhenix is capable of identifying known associations between gene knockouts and molecular structures. Since there is no information on molecular concentration which the cells must be treated with, we repeat the experiment across 4 concentrations. To get null distribution of cosine similaritics we take pairs genes knockouts and molecules for which there are no annotated relationships. We calculate cut-off for low and high percentiles, and then evaluate what percentage of pairs of genes and molecules with known relationships exceed the set thresholds.
Figure 9 demonstrates that in-silico screening using MolPhenix Molecular encoder is capable of recovering a significant portion of known interactions. This is performed without the use of exper- imental data on the molecular encoder. It is difficult to estimate an upper bound on the expected performance due to uncertainty the quality of curation of known pairs, presence of unknown associations between genes and molecules, and uncertainty regarding molecular concentration. There clear trend however that MolPhenix molecular encoder is capable of recovering a meaningful fraction of these interactions.
29

---

Concentraatoo: 0.0025
Concentration: 0.01
Concentration: 0.925
Concentration: 0.1
0.5
Model, p value: 0.1
Phonomi
(Eiperimental)
0.3
MaiPhontx Molocalor (m-Siica)
02
0.0
0.5999.
1.89
2.88
5-39
10.00
03599.5
1.99
2.98
5-35
10.90
0.5 8.549.5
5:55
18:00
0.5-9
1:09
2.80
5:5
1099
Coshe Similarity Percentlies Blo scale)
Figure 9: Evaluation of 0-shot ChEMBL identification of gene knockout and molecular phenomic similarities. On the X axis are percentile ranges, at which points the threshold is computed for cosine similarities. On the y axis is plotted total recall of recovered known interactions. Grey plotted for each range indicate baseline recall. Orange line indicates MolPhenix-Molccuue encoding of chemical compounds and MolPhenix- Vision for encoding gene knockout phenomics experiment. Blue line indicates Phenoml encoding of phenomics experiments for both the molecular perturbation and gene knockouts. In-silico encoding of molecular perturbation, well the corresponding concentration, recovers a significant fraction of observed interactions.
E.3 Molecular Property Prediction
We expand our evaluation with additional experiments supporting the utility of MolPhenix beyond retrieval. We conduct a KNN evaluation of the MolPhenix latent space, assessing the learned embedding on 35 molecular property prediction tasks across the Polaris and TDC datasets (Table X and 99. We find that MolPhenix trained with fingerprint embeddings consistently outperforms standalone input fingerprints. demonstrating that the MolPhenix latent space effectively clusters molecules according to their biological properties. We observed an interesting effect where prediction quality is positively correlated with implied dosage, indicating that MolPhenix learns dosage-specingg effects. In addition, utilizing
Table 8: Comparison of a KNN applied on MolPhenix molecular embedding with traditional fingerprints different tasks of TDC and Polaris datasets. Mean results for TDC, Polaris and together are available in the last three columns. Binary fingerprints use tanimoto similarity, while doating-point fingerprints use cosine similarity.
TDC Seatte Bond Mas
Potera WwwtttWooooottio Mun
syp AOHODA
K48
0443
1227
4.45
+1.9
360
4150
coscatandd Ba
4.15
30

---

Table 10: Evaluation on cumulative concentrations for active molecules: Average Top-1% % and Top-5% recall accuracies of methods utilizing different contrastive learning loss functions and concentration encoding information. We evaluate all methods on unseen images, unseen images and unseen molecules and an unseen dataset for zero-shot retrieval. Entries in bold denote best performance when the loss function is fixed while entries in highlight denote best performance across all guidelines.
sap-5%
Explet
Madality
Tmages
Arg
Union Tragee
Arg
Concoration (maaa)
Liaen Nauacalo
(0-bb20)
Unavre Metioalcs
(2-dar)
Bepteli CLIP
Piccel
.578
3559
1256
2466
5457
67711
.370
5159
CLOOME CLOOME CLOOME
Racoel
3513
A348
1658
3185
6330
1259
3018
5836
sigeed
5813
2908
3390
4380
8718
7168
Plemml
-9557
3255
8120
-230
IX3.
Pherent
A171
K101
DCL
losasilia
.034
1052
6799
058)
5708
4609
3419
DCL
unc but
.981
2002
.091
1834
5770
1873
.411
CWTT.
Phecol
(05]
5366
(WCT.
sigreld
Recoml
9138
1965
1910
1077
56-3
7070
8464
CWCL
loporiton
Pherans
7131
3367
Sut-ip
Dermml
7753
6:11
3555
M35
39015
5213
Siglay
signnid
Stplap
logonteen
Phemal
9493
7229
3926
3556
Stlapp
MScPhaus (mas)
Hazsl
0007
-181
1679
3764
3550
6436
.3247
aignsed oew-be
Milthmn (win))
Pherml
J430
7881
.0099
-756
1816
1964
.663
MiPPasis
Pheeoel
MaGGS
5053
MyPaasis
legastion
Phonoml
MarGs
9111
5471
5688
1924
MSuPazsix
Table 9: Comparison of KNN applied on MolPhenix molecular embedding with MolGPS on different tasks of TDC and Polaris datasets. Mean results for TDC, Polaris and together are available in the last three columns.
THC Saanbaatio Mass
Drante Monn
Du Maun
3aY UR shat
Ma bes
JU
vid
E.4 Addressing Challenges in Contrastive Phenomic Retrieval
Table an [2 show the complete Top 1% and 5% results of evaluation on cumulative concentrations on only active and all molecules, respectively. Similarly, Table II and 13 presents the full retrieval results of held-out concentrations experiments. In comparison to prior loss functions, S2L loss objective demonstrates consistent high retrieval rate in all tasks and molecular groups (i.e. all or active molecules), while using the same modality (Phenom]) and with or without explicit concentration information.
E.5
Ablation Studies
17 18 and 19 present top-1% recall accuracy across for the full ablation study on the variation MolPhenix key components. We note that compact embedding sizes from pretrained models stabilize training. This indicates that embeddings are expressive and accurately capture intricate aspects of molecules. Larger batch sizes result in a greater number of negative samples, hence improving performance. This is in line with prior contrastive learning methods continuing to improve by increasing the batch size [12. Increasing the number of parameters leads to more expressive models thereby enhancing retrieval performance. This result is in accordance with
Figure and Table 15 16
31

---

Table 11: Evaluation on held-out concentration for active molecules: Average Top- 1% and Top-5% recall accuracies of methods utilizing different contrastive learning loss functions and concentration encoding information. We evaluate all methods on unseen images, unseen images and unseen molecules and an unseen dataset for zero-shot retrieval. Entries in bold denote highest performance when the loss function fixed while entries in highlight denote highest performance across all guidelines.
Hpp66
Meteod
Explict Convrittion (wer))
Unieca
Unieza
Ulouee Vileoriee
Unown Melreske
(5-dwr)
CLIP Hlpptet eT
.109
.2455
.1519
.066
3991
Peoool
.1881
31588
.1186
.1644
.3783
.413
3065
.347
ho10008
Pravoni
.122
8013
3815
4330
CLOOME
Phasai
.114
2461
-1430
2015
4350
4050
3526
4158
logerttve
Phevil
9981
33.55
1865
1871
CLOOME
csc-bel
Phosoni
.3033
2540
.550
2251
3557
A345
3407
4635
IK3.
eepreed
-7282
1061
-757
IXL
Rhasal
.003
2517
1869
4156
5607
1604
DCL
.052
5009
.624
CWCT
Pheonl
-531
1003
A111
5316
6311
2019
(WTT
egreit
-7788
1641
2191
CWCL
Pazsii
.2448
4009
3508
.053
.5037
X294
2001
5327
Stlipp
Phooni
.211
3021
210
2027
Stlap
egreed
Phauni
2368
7023
2915
Splpp
esp-bcl
Pazsen
-751
4121
33445
.229
X045
1557
3357
Abuthmax
egrreed oselbut
Phamei
1636
A113
2447
MSiPaara (ens))
2439
2553
1065
3664
3379
.015
5315
6905
3644
7144
Paeasni M:4CPS
A277
7041
6706
XsiPhasa
Phoauni Mc4GPS
8213
7117
2916
Abuthmas (nen)
lopartea
Pheneni &
4237
268
2412
2011
MoCGPS
7591
7540
Table 12: Evaluation on cumulative concentrations for active and inactive perturbations Average Top-1 % and Top-5% Recall accuracy of methods utilizing different contrastive learning methods. Best performing methods are highlighted in bold.
Loss
Explicit
Modalry
Usseen Images
Uaszen Images Urness Medccalies
Unieen Dotaset (0-4b-4)
Arg.
Usseen Imnagee
Uasezn Lmages Urewss Meliculles
Unieen Dotaset (0-dbo0)
Adg.
Conceneation
Phensni
.1751
.167
.0344
.1454
.3710
.3669
.2655
.3151
HupGcla-CL.IP
Phanoni
.1511
.1700
.675
.304
.3464
.3637
.1922
.0144
Inf4 LOO
Phanom
.1746
1160
0045
1450
3697
.1755
2058
3170
CLIMOMI CLOOME CLOOME
Phenem]
1998
2005
0911
10330
39365
3521
2033
signooi
Pheus]
.3855
2627
4401
Phenom]
.145
.200
3054
.4720
DCL
Phensn)
.3277
2562
1364
2401
4856
4170
2765
.931
DCL
vignoot
Phenmm
.8511
1580
7009
5433
6722
5155
1581
4010)
DCL
Jogaiiim
Phonsm)
.4953
3615
2122
.5533
.6311
5531
3587
.1.0
one-her
Phonen
.5236
1288
.5168
.6791
3570
2068
.5443
CWCL
signoid
Phenen1
.5700
2101
.5542
.037
CWCL
Joguzifim
Pheaomi
.5146
.246
.3706
3660
CWCL
one bot
Phesom]
.5401
2336
.362
.0822
A001
.5635
SigLip
Phenso)
.329
1470
.2511
.5200
2835
.0022
sigmasi
Phanml
5021
3225
2073
3456
6360
5231
3430
500
SigLip
logarition
Pheism)
5156
3635
2233
3675
.652
5689
3653
.385
Sigl.ip
Phamoii
.3544
3745
2217
.3816
.6858
5928
3945
.5977
(mmn)
Phenen
4688
3176
3121
4554
(oun))
signood
Phenom]
.352
.300
821 (ound)
one- for
Phenem]
5433
2819
2284
.879
6954
5895
4000
.8436
Phaneni
2001
3130
4567
M-GGP
S2L
Phenem]
.953
3230
2957
3537
6345
5105
3750
5056
MJGGP
S21 (nand)
lngrrtt
Phonen)
S101
3589
2535
.742
3500
1942
5350
MaGGP
S2L (ours)
one boc
Phenom]
.3300
3720
2676
3032
.6870
.988
4326
5695
MoGGPS
32

---

Table 13: Evaluation on held-out concentrations for active and inactive perturbations Average Top- 1op-1% 1% and Top-5% Recall accuracy of methods utilizing different contrastive learning methods. Best performing methods are highlighted bold.
tope1%
tep-5%
Explicil Concermation
Mudilty
Unaeen Imagr
Ussoon Imters Molecales
Unser Ditecct
Aug.
Uraczn Imago
Ussccn Imagrs Melecties
Unsaza Derect
Aug.
stoo
sho0
CLIP
Phesemi
.1644
.111
.964
.1553
.2916
22455
.2366
.4766
HopfolaCCLIPP
Phanomi
.1200
1021
.0755
.0550
.3465
.2287
2005
2217
IALLO08
Phesom]
.1715
1250
.5540
2249
2495
CLOOME
Phenen)
.1745
2010
1248
.000
2487
2355
2439
CLOOME
signodd
Phanim]
.2913
.1388
1162
1614
.5189
3636
.211
.440
CLOOME
lepeift
Phanml
.279
1081
0993
1464
4958
3644
2334
3042
CLOOME
00e bo
Phesem]
.2366
2070
2074
1430
.5012
2224
2345
33955
Phenel
1835
vignos
Phamml
1726
.2311
3108
.204
1581
2458
3015
CWC
Phenmm
4135
1967
2511
61.76
1065
M94
CWCL
signoo
Phessm]
.5112
1856
.811
.2426
.6901
3384
3190
.314
DWCT
legeifmm
Phonel
.654
1996
1709
2690
.550
1105
7065
5148
SigLip CWCL
one bot
Pheaem]
.4651
1012
.1734
.2776
.6465
2019
3104
.020
1190
.200
2914
Jouarttm ona-lax
Phenen]
.2766
Sp
Phanml
4657
1443
1141
2517
18544
2879
2790
2847
828. (nurs)
Phanom
.3366
1842
1713
2061
6961
1892
1045
8921
821. (mur))
sigmari
Phamiml
L5489
18999
.1783
.3120
3469
3201
.3372
528. (ouna)
Japeilii
Phanmm
-0.36
1701
.727
2851
3542
3157
5575
S21, (ouns)
one bor
Phenem]
.4726
1537
1521
2895
.696
2096
2887
2959
S2L (ours)
Pheaom1
5248
1529
.910
2596
.904
.3268
3305
3251
MsGGPS
S22. (ouns)
sigmooi
Phanom
.5388
.897
2029
3088
.7088
3427
3495
362
M-GGP
(ouzs)
Jogusitim
.003
2013
.311
.411
S21.
.1762
.2511
3187
3067
McGGP
recent advances in language modelling and scaling laws across different data and compute budgets
25
Model size
Depth
Width
Unseen images
Unseen images Unseen molecules
Unseen dataset
(0-shol)
Tiny- 2.7m
ResBlocks
256
.8337
.7186
.4030
Small- 1aalll9..4m) 9.4m
6 ResBlocks
512
.9744
.7352
.4562
Medium - 38.7
8 ResBlocks
1024
.9400
.7490
.485
Table 14: Ablations across different model sizes. expressive.
Larger capacity models are found to be more
Batch size
Unseen images
Unseen images + Unseen molecules
Unseen dataset
(0-shot)
128
.8600
.7163
.4044
512
.9252
.7511
.467
2048
.9450
.7616
.440
8192
.9489
.7563
.4966
Table 15: Ablation across different batch sizes. Larger batch sizes benefit contrastive learning.
E.6 Investigating Other Pre-trained Phenomic Encoders
To investigate the impact of pre-trained encoders, we perform additional experiments evaluating a supervised phenomic image encoder (Table 20p. Instead of Phenom1, we trained Molphenix frame- work using AdaBN, a CNN-based supervised phenomic encoder, with an analogous implementation discussed in [55]. We find that the general trends between Phenom] and AdaBN are consistent with slight decrease in overall performance. These findings provide additional support to the generality of the proposed guidelines.
E.7 Integrating MolGPS Embeddings With Other Fingerprints
Molphenix architecture flexible, allowing that the proposed components replaced by other phenomic molecular pretrained models. We leveraged from MolGPS, which is a MPNN based
33

---

1.0
1.0
1.0
Recalle
0.6
0.6
0.6
0.4
0.4
0.4
256
107
128
2048
8132
Embedding Dimeasion (log
Number scale)
(log
1.0
1.0
1.0
0.9
0.8
0.6
0.6
0.4
0.5
01
MACCS
RDKI
Morgan
Mato
Malos
wD Enbelkinsy
web tratedang
Cut-OIT p-value (log scale)
Fingerpeint
Random Embodding Averaging
Unsoon Imagrss
Unseen Images
Unseen Molocales
Unsven Datasel
Figure 10: Ablations of top-1 % recall accuracy with (top-left) the size of embedding dimension, (top-center) number of parameters, (top-right) batch size, (bottom-left) cutoff p value, (bottom- center) fingerprint type, and (bottom-right) random batch averaging. Compact embedding sizes from pretrained models, larger number of parameters, larger batch sizes, lower cutoff p-values, pretrained MolGPS fingerprints and presence of random batch averagin improving retrieval of our MolPhenix framework.
Dim size
Unseen images
Unseen images Unseen molecules
Unseen dataset
(0-shot)
256
.9452
.7510
4929
512
.9430
.7490
.4850
1024
.9392
.7288
.4710
Table 16: Ablation across different embedding dimensions. Compact embedding sizes capture more
molecular information.
GNN model with IB parameters which allows maximize architecture expressivity while minimizing the risk of overfitting [34] (56) For additional investigation, we combine MolGPS molecular embeddings with RDKIT, MACCS, and Morgan fingerprints and show that they can provide Molphenix with richer molecular information and yields overall higher performance of MolPhenix both cumulative and held-out concentration scenarios. Results for active and all molecules retrieval of Molphenix trained on the discussed combinational molecular embeddings are available in table 21 and 22
34

---

off
Unseen images
Unseen images Unseen molecules
Unseen dataset
(0-shot)
p 1.0
9312
7057
.6866
p<u. 1
94.30
.7490
4850
p< 01
9284
.7192
.5005
Table 17: Ablation across different p- p-value cutoff threhsolds, p values < ..1 benefit retrieval of active molecules.
lingerprint
unseen imagges
Uniccen imigges molocule
Uniseen dalasel
MACCS
.9150
.5033
RDKit
.934]
.6693
.3925
Morgon
.9524
.7477
4613
Mult-EPPs
.9430
.7490
485
Phenoon
MolGPS
.9430
.7514
-5577
Table 18: Ablation across different fingerprint types, A combination of embeddings bootstrapped from Phenoml and MolGPS significantly benefit retrieval.
images
Unieen images motecules
.4759
Wia Rundom Enbealing
Table 19: Ablation across random embedding averaging. Utilzzing random batch averaging stabilizes training and benefits retrieval.
Table 20: Evaluation on cumulative concentrations while using AdaBN. Molphenix is trained combination of RDKIT, MACCS, and Morgan fingerprints in this experiment
Methed
Ianges
Usscca Dutsoet
Unseon
Concestration
Molecules
(0-bbet)
Unseca Molecules
(0-sho)
top-1% active moleculos
top-5% i active molscules
MolPaannix Mollfaentx
AdaBN
2568
.5336
.523
.581
.9582
.5772
-7646
sipmood loganthi one- hor
AdaBN
911
.858
.833
.911
7597
A2213
.797
Mallftentx Mailfnentx
AdaIN
9155
.8105
A242
.801
$729
8132
A303
8188
AGMIN
9187
.1255
4225
A612
9744
8102
5419
8155
Inp-1% molecules
Exp-5%
Molitsenix Mailfsenis
AdaNN
.993
.2409
1899
.867
5981
4081
2885
.305
signoid Jeganilu onc-boo
AdaBN
.104
.142
1957
3401
6496
5165
331
.009
Mallisenis
AdaBN
5379
.3003
2071
-614
6867
.561
3605
5345
MallPsenis
AdaEN
.476
.425
2082
3661
7807
.541
3603
.5477
Table 21: Evaluation on cumulative concentrations while combining MoIGPS, RDKIT, MACCS, and Morgan fingerprints.
Images
Brages Unuzze
Trrige
Ang
Concentraion
Unaon Midocalos
(Dubbar
(1-dvit)
txp-1
1ep-5% active
MalPheiic Malthamic
Plaooet MirGatS
9181
2112
/117
-016
0756
215
87g
signoke leginiton
Phensesl Moccit Tips
9/25
7416
$119
-307
0817
MV11
7651
$736
Maltfunic
Phenoral Moocors 1ips.
9815
9161
10000
lsp-1% medacudon
bxp-5% all molerals
Malthmii
sigmald
Phenons MirGiS 1pp.
5136
541
2152
ATK3
.11
1799
-12
Mallhmiic
Ieganihnn
Phensel McCIPS 1pp.
$25
976
2645
3396
K65
5543
411
5256
Mallhaiic
Phenoral MicCPS 3pp
6982
5916
4251
Table 22: Evaluation on heldout concentrations while combining MolGPS, RDKIT, MACCS, and
Morgan fingerprints.
Mebed
Beplia
Medalit
Uneen Imagee
Uneee Imagce Unaoma Medocales
Usueze Dutnact
Are.
Uraocn Imago
Usaem brages +
Uream Ustect
Ang.
Concentratios
(0-bbe))
(0-dbol
1 actine malecules
tep. octive motecuks
Malthenis
MalPhenis
Inpatilin
Pacsona & MaCIPS &
$1
$107
J3766
5565
0544
7405
6895
2039
MalPhesis
oue bol
Picasrel & ScoIPS & 1ps
7487
A409)
3830
.535
0030
5827
6520
-7566
melecales
all molecules
Malthenis MalPhenis
Pieaanl & McoIPS (ps
5339
.1980
1156
3305
6958
2003
4224
4417
simnsi one-hol
3115
MoxGPS
Maltoaic
Phenend MirGiS 1pp
4040
3700
6635
2943
4114
4445
35


Reviews:
[{"review_id": "A3hIqSAoz9", "reviewer": "Reviewer_YXwu", "summary": "This work focuses on predicting the molecular impact on cellular functions and investigates the problem of contrastive phenomolecular retrieval. It introduces MolPhenix, a model that leverages a joint latent space between molecular structures and microscopy-based phenomic experiments using contrastive learning. The main contributions include the use of a pre-trained phenomics model, a novel inter-sample similarity-aware loss (S2L), and molecular concentration conditioning, leading to a significant improvement over the previous state-of-the-art in zero-shot molecular retrieval of active molecules.", "questions": "For the pretrained GNN, are there any specific advantages to choosing the current one, given that there are many other pretrained GNNs that can be used to extract molecular representations?\n\nAre there any discussions regarding the results of different concentration encoding choices? In a cumulative concentration setting, one-hot performs the best, while in a held-out concentration setting, not using any explicit concentration is the best choice overall.", "limitations": "The authors have adequately addressed the limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The introduction of the MolPhenix model, which utilizes a pre-trained phenomics model, a novel inter-sample similarity-aware loss (S2L), and molecular concentration conditioning, is original and demonstrates improvements over existing methodologies. \n\nThe experimentation of this work is comprehensive and detailed to justify the proposed method. \n\nThe paper is well-structured and clearly written with clear explanations of the methodologies. \n\nThe problem investigated is important in the field of drug discovery and has a growing interest.", "weaknesses": "For the captions of Table 2~5, it is better to mention the experimental setting (cumulative concentration & held-out concentrations). The current captions are nearly the same (Table 2 vs. Table 4, and Table 3 vs. Table 5), which is kind of confusing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for a thorough and rigorous examination of our paper. Below we aim to address the questions and clarifications to further improve the work.\n\n**Concern #1: For the captions of Table 2~5, it is better to mention the experimental setting.**\n\nThank you for your feedback. We changed the captions to clarify and simplify the experimental setting and explanation of results in Tables 2, 3, 4 and 5. For example, following is the updated caption for Table 2 “ **Evaluation on cumulative concentrations:** Top-1\\% recall accuracy with use of the proposed MolPhenix guidelines evaluating impact of training loss on retrieval. We omit explicit concentration from this experiment.”.\n\n**Concern #2: “Specific advantages to choosing the current [GNN].”**\n\nMolphenix architecture is flexible, allowing that the proposed components be replaced by other phenomic or molecular pretrained models. However, we leveraged from Mol-1, which is a MPNN based GNN model with 1B parameters which allows us to maximize architecture expressivity while minimizing the risk of overfitting [2, 3]. We also note that combining Mol-1 molecular embeddings with ECFP, MACCS, and Morgan fingerprints can provide Molphenix with richer molecular information and yields overall higher performance of MolPhenix in both cumulative and held-out concentration scenarios. Results for active and all molecules retrieval of Molphenix trained on the discussed combinational molecular embeddings are available in table 2 and 3 of our global response (attached pdf). To evaluate the impact of GNN encoder we also perform a fingerprint ablation assessing the impact of fingerprint expressivity on retrieval indicated in Figure 5 of the paper.\n\n**Concern #3: “discussions regarding the results of different concentration encoding choices”**\n\nWe note in the last sentence of section 5.2 in the paper that one-hot encoding shows significant improvements in a cumulative concentration setting, however it is limited to unseen dosage. In evaluation on held-out concentration, the model isn’t required to discriminate between molecules with different concentrations, thus explicitly providing dosage isn’t directly useful. In this experiment, sigmoid embedding can be thought of as a continuous way of separating “high” and “low” concentrations, showing effectiveness in this setting. We believe that the best encoding choice is dependable on the type of in-silico application. For example, if the objective is to identify unseen molecules with same morphological impacts or if the objective to simulate the impact of a molecule for un-tested dosage. We thank the reviewer for providing this feedback and we will add this additional discussion to the final version of the paper.\n\nThank you for your positive feedback, and we will be happy to further discuss and answer any questions you may have.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qpb0qiytZS", "reviewer": "Reviewer_r9h5", "summary": "This paper introduces MolPhenix, a model designed to learn a joint latent space between molecular structures and microscopy phenomic experiments, addressing the challenge of contrastive phenomolecular retrieval. The authors point out three key challenges in this domain: limited paired data & batch effects, inactive molecular perturbations, and variable concentrations. To address these issues, they propose a set of guidelines: 1) leveraging a pre-trained phenomics foundation model (ph-1), 2) mitigating the impact of inactive molecules through undersampling and a novel soft-weighted sigmoid locked loss (S2L), and 3) encoding molecular concentration both implicitly (within the S2L loss) and explicitly (by parsing dosage concentration).\n\nThe primary experiments focus on the task of phenomolecular retrieval. For active molecules, MolPhenix achieves up to 77.33% top-1% retrieval accuracy, representing an 8.1-fold improvement over the baseline. The paper includes necessary ablation studies and evaluations across multiple datasets, demonstrating the effectiveness of their approach in both cumulative and held-out concentration settings.", "questions": "Regarding the equality of involved datasets:\n\n1. How would the model's performance vary if only the open-source RxRx3 data or only the private novel data were used?\n2. Given that many components of MolPhenix are publicly pretrained models, is it feasible to construct a comparable model using solely open-source resources?", "limitations": "The authors have properly adressed the limitations in their paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel approach: The paper introduces MolPhenix, a comprehensive framework that addresses several key challenges in contrastive phenomolecular retrieval by combining multiple innovative techniques. This new framework demonstrates notably strong performance.\n2. Concentration encoding: The study explores both implicit and explicit methods for encoding molecular concentration, enhancing the model's ability to capture dose-dependent effects and generalize across concentrations. The explicit concentration module and inactive molecule undersampling techniques may inspire future multi-molecule research.\n3. Detailed ablation: The paper provides an in-depth analysis of results, including the impact of various components (e.g., loss functions, concentration encoding methods) on model performance, presented in both the main body and appendix.", "weaknesses": "The following are just some minor concerns, listed in descending order of importance:\n\n1. Insufficient justification for S2L loss: While the paper introduces the S2L loss as a key contribution, it lacks a thorough theoretical analysis explaining why this loss function is suitable for the phenomolecular retrieval task. A more rigorous mathematical analysis of S2L would strengthen the paper's contribution.\n2. Inadequate ablation of pretrained components: As a phenomolecular retrieval \"framework\", the study should include ablation studies using different molecular and phenomic encoders. However, it appears that experiments were conducted only with fixed molecular GNN and phenomic models.\n3. Oversimplified treatment of batch effects: The paper claims to address batch effects through embedding averaging, but this approach may be too simplistic. A more detailed explanation of why simple averaging can alleviate batch effects, or some minor adjustments to this batch effect removal procedure, would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for providing detailed feedback on the paper. \n\n**Concern #1: Additional justification for the S2L loss**\n\nIn this section we provide some additional intuition for the S2L loss and further relate it to previous works. We first assess the conceptual similarities between InfoNCE and CWCL loss and justify a similar extrapolation for the relationship between S2L and SigLIP losses.\n\nInfoNCE can be considered a special case of the CWCL loss, where $w_{ij}$ is set to 0 for all pairs of $i$ and $j$ unless $i=j$. Conceptually, this is equivalent to stating that all the negative pairs are equally distant from the reference sample.  We will consider a uni directional loss CWCL, for identifying $\\mathcal{X}$  from $\\mathcal{M}$:\n\n$$\n\t\\mathcal{L}\\_{\\text{CWCL}, \\: \\mathcal{M} \\rightarrow \\mathcal{X}} = - \\frac{1}{N}\\sum\\_{i=1}^N\n\t\\left[ \\frac{1}{\\sum\\_{j=1}^{N} \\mathbf{w}^{\\mathcal{X}}\\_{i,j}}\n\t\\sum\\_{j=1}^{N} \\mathbf{w}^{\\mathcal{X}}\\_{i,j} \\log\n\t\\frac{\n\t\\exp{\\left(  \\langle \\mathbf{z}\\_{x\\_i},\\mathbf{z}\\_{m\\_j} \\rangle / \\tau \\right) }}\n\t{ \\sum\\_{k=1}^{N} \\exp{ \\left( \\langle \\mathbf{z}\\_{x\\_j}, \\mathbf{z}\\_{m\\_k} \\rangle / \\tau \\right) }} \\right].\n$$\n\nIf we set $w_{ij} = 0$ when $i \\neq j$ and $1$ otherwise then the term $\\Sigma_{j=1}^{N} w_{i,j}^\\mathcal{X}$ evaluates to 1 and the above expression simplifies to:\n\n$$\n\t\\mathcal{L}\\_{\\text{InfoNCE}} = - \\frac{1}{N}\\sum\\_{i=1}^N\n\t\\left[ \\log \\frac{\n\t\\exp(\\langle \\mathbf{z}\\_{x\\_i}, \\mathbf{z}\\_{m\\_i} \\rangle / \\tau)\n\t}{\n\t\\sum\\_{k = 1}^{N} \\exp(\\langle \\mathbf{z}\\_{x\\_i}, \\mathbf{z}\\_{m\\_k} \\rangle / \\tau) } \\right].\n$$\n\nIn the case of CWCL, a non 0 $w_{i,j}$ determined by a within modality similarity function informed by a pre-trained model\n\nSimilarly SigLip can be considered a special case of S2L when $w_{i,j}^{\\mathcal{X}} = 0$ when $i \\neq j$ and $w_{i,j}^{\\mathcal{X}} = 1$  in the case $i=j$. This is the formulation of S2L\n\n$$\n\t\\mathcal{L}\\_{\\text{S2L}} =  \n\t- \\frac{1}{N}\n\t\\sum\\_{i=1}^N\n\t\\sum\\_{j=1}^N\n\t\\log \\left[\n\t\\frac{\\mathbf{w}^{\\mathcal{X}}\\_{i,j}}\n\t{1 + \\exp \\left( -\\alpha \\mathbf{z}\\_{\\mathbf{x}\\_{i}}.\\mathbf{z}\\_{\\mathbf{m}\\_{j}} + b \\right) } +\n\t\\frac{(1 - \\mathbf{w}^{\\mathcal{X}}\\_{i,j})}\n\t{1 + \\exp \\left( \\alpha \\mathbf{z}\\_{\\mathbf{x}\\_{i}}.\\mathbf{z}\\_{\\mathbf{m}\\_{j}} + b \\right) }\n\t\\right].\n$$\n\nIt can be simplified to SigLIP by setting $w_{i,j}^{\\mathcal{X}}$ to $1$ when $y_{i,j} = 1$ thus setting the term $\\frac{(1 - \\mathbf{w}^{\\mathcal{X}}\\_{i,j})}{1 + \\exp \\left( \\alpha \\mathbf{z}\\_{\\mathbf{x}\\_{i}} \\cdot \\mathbf{z}\\_{\\mathbf{m}\\_{j}} + b \\right)}$, corresponding to $i \\neq j$, we set $w_{i,j}^{\\mathcal{X}}$ to $0$ thus negating the first part of the $\\mathcal{L}_{S2L}$ loss, evaluating to:\n$$\n\\mathcal{L}\\_{\\text{SigLIP}} =\n\t- \\frac{1}{N}\n\t\\sum\\_{i=1}^N\n\t\\sum\\_{j=1}^N\n\t\\left[ \\log\n\t\\frac{1}{1 + \\exp{ \\left( \\mathbf{y}\\_{i,j}(-\\alpha \\: \\langle \\mathbf{z}\\_{\\mathbf{x}\\_{i}}, \\mathbf{z}\\_{\\mathbf{m}\\_{j}} \\rangle + b) \\right) }}\n\t\\right].\n$$\n\nHaving a $0 \\leq w_{i,j}^{\\mathcal{X}} \\leq 1$ allows us to inform the training by going between discrete negative labels to continuous informed by some prior information. This information is given by a pre-trained encoder $\\theta_{Phi}$, in our case but can be informed by any pre-trained model.\n\n**Concern #2: “Inadequate ablation of pre-trained components”**\n\nTo investigate the impact of pre-trained encoders, we perform additional experiments evaluating a supervised phenomic image encoder and highlight the ablation study of molecular fingerprints described in Figure 5. \n\nInstead of Ph1, we trained Molphenix framework using AdaBN, a CNN-based supervised phenomic encoder, with an analogous implementation discussed in [1]. We find that the general trends between Ph-1 and AdaBN are consistent with a slight decrease in overall performance. These findings provide additional support to the generality of the proposed guidelines.\n\nIn addition, we leveraged from Mol1, which is a MPNN based GNN model with 1B parameters - an expressive capable molecular encoder [2, 3]. We note that combining Mol-1 embeddings with ECFP, MACCS, and Morgan fingerprints can provide Molphenix with richer information and yields overall higher MolPhenix performance (Table 2 and 3 of PDF in general response). We also performed an ablation study over publicly available fingerprint encoding methods, which are an effective baseline for molecular representations. We demonstrate that MolPhenix demonstrates strong retrieval performance with the use of fingerprints as an alternative to pre-trained GNN (Figure 5).\n\n**Concern #3: “Oversimplified treatment of batch effects”**\n\nThe reviewer is accurate in noting that batch effect area is a smaller portion of the overall paper contributions as it is a rich area of research, especially in the biological sciences. Our intention was to highlight the ability of averaging phenomic encoder embeddings which is infeasible when working with samples directly in the image space. We perform an ablation, studying the effects of taking a random number of embeddings when performing averaging and find a small improvement in retrieval performance (Figure 5).\n\n**Concern #4, Q1 & Q2:**\n\nTraining data employed is comprised of 1.3M pairs of perturbations, however, RXRX3 is composed of 1,674 known chemical entities at 8 concentrations each and is primarily used as a validation dataset. JUMP-CP [4] is a promising open source dataset which is being released in increments is a promising future phenomic data resource. With the availability of JUMP-CP it will be possible to train a fully open source analog of Molphenix. A beta version of Ph-1 is publicly available and will be linked in the updated paper. In our ablations demonstrated in Figure 5 we show that molecular fingerprinting methods are a strong baseline and are comparable with Mol-1 performance.\n\nKindly let us know if our response above addressed your concerns. We will be happy to further discuss and answer any questions you may have.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bwW2ciGGH3", "reviewer": "Reviewer_kfG5", "summary": "The paper introduces MolPhenix, a framework for contrastive phenomolecular retrieval that integrates phenomic data and molecular structures into a joint embedding space. Key contributions include combining phenomic and molecular data for improved retrieval accuracy, proposing effective training guidelines, and addressing cumulative concentrations and label noise. The framework demonstrates significant performance improvements over existing methods, particularly in zero-shot settings, and is supported by comprehensive experiments and ablation studies.", "questions": "1. For molecular representations, did you consider combining GNNs and fingerprints? In other tasks, such as property prediction, their combination often surpasses the performance of each method individually.\n2. Did you consider to report and compare metrics for top-5 and top-10 recall?", "limitations": "The authors have adequately addressed the limitations.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper is well-structured and comprehensive, showcasing rigorous experimentation through extensive ablation studies, comparisons with baseline methods, and evaluations. Its strength lies in the detailed and methodical approach to validating the MolPhenix framework, providing strong evidence for its effectiveness and potential applications. The experimental design is robust, demonstrating the framework's superiority in various scenarios and contributing valuable insights to the field. Additionally, the integration of molecular and phenomic data into a joint multi-modal embedding offers a fresh perspective, enhancing the overall impact and originality of the work.", "weaknesses": "1. The clarity of this work could be significantly improved. The task and background are not well stated, and the related work is not adequately introduced. Many biological terminologies are mentioned, creating a large gap between the introduction and the real dataset.\n2. The paper could benefit from including comparisons with a broader range of state-of-the-art methods, particularly those from adjacent fields such as multi-modal representation learning in genomics and proteomics. This would highlight the specific innovations of MolPhenix more distinctly.\n3. The significance of the work could be more explicitly articulated and demonstrated. Although terms like drug discovery are mentioned, the real-world impact is not clear for readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for providing detailed feedback on our paper. We aim to address the feedback point by point below: \n\n**Concern #1: The clarity of this work could be significantly improved**\n\nWe thank the reviewer for this constructive feedback, we believe this work is best assessed with the required context of molecular biology, so we aim to enhance the accessibility of our paper by adding a biological terms explanation in the appendix. In particular, we will define the following terms: *phenomics, cell morphology, cell line (ARPE-19), molecular concentration, phenomolecular retrieval, cell staining, molecular perturbations, inactive molecule perturbations, batch effects, molecular fingerprints, initial cell state assumption*.\n\nWhile we agree the clarity of the paper can always be further improved, we note that the presentation of our paper was a highlight for other reviewers: YXwu “The paper is well-structured and clearly written”, ELHC “The paper is very well written”, Vz3t “The paper is well-written, and the motivation is clear.” Getting high ratings in soundness (3, 3, 4, 3) and presentation (3, 3, 4, 3). Additional information on related works can be found in section 2, and we have expanded the “Related Work” section on molecular-phenomic approaches.\n\nPlease let us know if this clarifies the accessibility of the paper, or if there is any additional terminology that we can add to the glossary.\n\n**Concern #2: Comparisons with a broader range of state-of-the-art methods in … multi-modal representation learning in genomics and proteomics**\n\nWe’ve restricted the scope of this paper to studying phenomics and molecular modalities. While expanding to other biological modalities such as genomics or proteomics would undoubtedly be very interesting, it requires a significant commitment to curating the data and innovation in biological sequence model space. This is a direction we leave for future exploration.\n\nThe reviewer could be interested in a 0-shot evaluation that we perform to investigate MolPhenix’s ability to generalize to genetic knock-out perturbations (Appendix D2). We assess the model’s 0-shot generalization capabilities that are known to have similar molecular effects to learned molecular perturbations. The evaluation demonstrates that MolPhenix learns the landscape of genetic perturbations, allowing the model to recover known biological pairs.\n\nOn the methodological front, we compare to SOTA in the Image-Text multi-modal training consisting of DCL(2021), CWCL (2023), and SigLip (2023) - recent methods that have demonstrated significant success. In addition, we conduct thorough evaluations benchmarking recently published CLOOME (November 2023) model.\n\n**Concern #3: Significance of the work could be more explicitly articulated and demonstrated**\n\nAlthough the main text of our paper is mostly focused on pheno-molecular retrieval, we have some initial experiments in the appendix assessing the model’s ability to perform other biologically relevant tasks. We perform pheno-activity experiments, demonstrating that the learned embedding is predictive of molecule - concentration tuple morphology impact (Appendix E1). This opens the door to potential in-silico activity pre-sceening and in-silico dose-response curve construction. In addition, we perform 0-shot biological activity experiments mentioned earlier in Appendix D2.\n\nIn the general reviewer response, we include new experiments showcasing the effectiveness of the learned latent space by conducting a KNN evaluation of the MolPhenix latent space. We assess the learned embedding on 35 molecular property prediction tasks across the Polaris and TDC datasets (see Table 4 in the attached PDF). Our findings indicate that MolPhenix, when trained with Fingerprint embeddings, consistently outperforms standalone input fingerprints, effectively clustering molecules according to their molecular properties.\n\n**Concern #4: “Combining GNNs and fingerprints”**\n\nThis is a valuable evaluation for the completeness of our work. Please find the results for these experiments in the general reviewer response (Table 2 and 3 in the attached PDF). We find that combining pre-trained GNN and molecular fingerprints further enhances retrieval performance.\n\n**Concern #5: “Report and compare metrics for top-5 and top-10 recall”**\n\nWe also report top-5% accuracy metrics in supplementary tables 8, 9, 10 and 11. We choose to report hits in top-K% since we have variable sized test sets. Top K metrics do not control for the difficulty of this task unless artificially subsampled to a pre-determined size. This is proportionally equivalent to Top-K% evaluation metrics, but has the downside of being stochastic contingent on composition of the batch. \n\nKindly consider adjusting our overall score if our response addressed your primary concerns. We would be happy to answer any additional questions you may have in order for you to support acceptance of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vz87temeAJ", "reviewer": "Reviewer_ELHC", "summary": "The paper makes a significant step forward in the task of phenotype-molecular retrieval–the task to find the molecule applied to perturb a set of cells given a microscopy readout. This problem can be modeled using multi-modal contrastive learning. The proposed MolPhenix method leverages pre-trained foundation models to encode the microscopy images and molecular structures. Then a novel contrastive loss is used which takes into account domain specific issues like batch effect and different concentrations of the applied molecules. Through thorough evaluations and ablations the authors show increased performance over a baseline.", "questions": "- Please elaborate on generative or mechanistic opportunities of the model\n- You argue MolPhenix cannot be applied to images. Please elaborate since instead of ph-1 a simple non domain-specific image feature extractor could be used. Have you studied the results?\n- Please comment on which model weights of ph-1, Mol-1, and MolPhenix are publicly available or you will be releasing", "limitations": "- Limited reproducibility due to private datasets and the model weights likely not being released", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper tackles a task relevant for drug discovery and shows a significant step forward in predictive performance\n- The paper is very well written\n- The authors perform extensive experiments on held out molecules, phenotypes and dataset\n- The CLOOME baseline is tuned convincingly\n- The proposed guidelines for the task are supported by ablation studies\n- The paper combines and improves upon SOTA methods in multi-modal contrastive learning\n- The authors show domain knowledge by incorporating reasonable data processing and training strategies (batch effect removal + undersampling of inactive molecules)", "weaknesses": "- The authors focus on retrieval instead of evaluation of the latent space. Given the expressiveness of the leveraged foundation models, generating phenotypes or molecules should be possible. It would be great if the authors could comment on this or other explorations of the phenotypic space.\n- The authors train on a private dataset which makes the paper almost not reproducible. However, due to detailed model descriptions, the guidelines could be evaluated on other datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive, insightful comments and the detailed feedback. We believe your perspective will help shape this into a stronger work. Below, we aim to discuss each of the suggestions and discussion points individually.\n\n**Discussion point #1: “Generating phenotypes or molecules should be possible”**\n\nWe agree with the reviewer that this is a really exciting future direction for the work. As a first step in establishing pheno-molecular multi-modal learning, we decided to focus on the retrieval task due to its straight forward quantitative evaluation advantage. Our goal in this work was to establish a set of design decisions and guidelines that we can quantitatively investigate for effective latent space stratification.\n\nHigh pheno-molecular retrieval rates would indicate to us, that the model learns effective stratification of the latent space, opening up the door for future experiments with molecular and phenomics generation. This is a critical research direction with applications in drug-discovery such as identifying phenotypically similar molecular analogs with applications to generic molecule research.\n\nThere is a unique set of challenges in evaluating high quality, diverse molecule designs. The state of the art for assessing molecule quality are still biological experiments, which we are excited to explore in future works.\n\n**Discussion #2: The authors focus on retrieval instead of evaluation of the latent space.**\n\nWe certainly agree with the reviewer that MolPhenix embeddings should be able to provide additional interesting assessments as to the quality of the model latent space. To that end we have multiple experiments described in Appendixes E1 and E2 which we could not include in the main text due to space constraints. In addition, we conduct experiments assessing the quality of the learned molecular representation by experimenting on 35 downstream tasks, discussed in more detail in the general response (Table 4 attached PDF).\n\nSome brief details on appendix experiments: We were interested in assessing whether our molecular encoder captures pheno-activity throughout training. We found that by specifying only the molecular structure and the corresponding concentration, we are able to predict molecular activity levels with an ROC-AUC of 0.9. Visualization of the model latent space can be found in supplementary Figure 7. These results open up the door for in-silico activity screening that have previously been infeasible. In addition, these can be used as in-silico dose-response curve evaluations, finding an activity - dosage trade-off for a previously unscreened molecule.\n\nIn appendix E2 we evaluate MolPhenix’s ability to identify previously known concordant perturbations between small molecules and genetic knockouts. From a database of known relationships, we encode the molecular perturbation with the MolPhenix molecular encoder and assess whether it is able to match them with a corresponding embedding of a genetic knockout. To create an embedding of a genetic perturbation, we embed results from a phenomic experiment of cell lines with a corresponding gene knocked out. We find that this in-silico perturbation concordance experiment is able to provide strong results relative to a fully experimental baseline.  \n\nWe believe these findings are important initial experiments demonstrating downstream utility applications of MolPhenix learned embeddings.\n\n**Concern #3: The authors train on a private dataset which makes the paper almost not reproducible. However, due to detailed model descriptions, the guidelines could be evaluated on other datasets.**\n\nWe are unfortunately unable to release the training dataset for MolPhenix but aim to disseminate generalizable findings that can be helpful to other scientists working in this domain. To that end we provide pseudo-code and algorithmic descriptions for S2L loss. Additionally, our algorithm implementation is kept in PyTorch-like syntax for easier reproducibility. In addition, we evaluate our models on a large openly accessible dependent RxRx3 dataset which can be used for evaluation of other models and benchmarking other design choices by the community.\n\n**Concern #4: Please elaborate on generative or mechanistic opportunities of the model**\n\nThis is an important point that we hope that we’ve addressed in sufficient detail in discussion points 1 and 2.\n\n**Concern #5: You argue MolPhenix cannot be applied to images. Please elaborate since instead of ph-1 a simple non domain-specific image feature extractor could be used.**\n\nWe thank the reviewer for bringing up this important point, and would like to clarify that any sufficiently expressive image feature encoder can be used. In the general reviewer response, for example, we demonstrate that we can use an alternative encoder that is trained in a supervised fashion to predict identity of genetic perturbations (as an alternative to θPh-1). We hope that these experiments in addition to use of an ensemble of publicly accessible fingerprints provide sufficient evidence that the guidelines are generalizable across a number of public and private encoders.\n\nWe have updated Table 1 caption to be a more clear description of our training pipeline: “We note that MolPhenix’s main components such as S2L and embeddings averaging relies on having a pre-trained uni-modal phenomics model.”\n\n**Concern #6: Please comment on which model weights of ph-1, Mol-1 and MolPhenix are publicly available or you will be releasing?**\n\nWe note that the code and training data for Mol-1 are available, which we will provide references to in the final version of the paper. In addition, a public version of θph-1 will be available for inference.\n\nThank you for your positive review and your interest in our work. We will be happy to further discuss and answer any additional points of interest.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ougu39RWhm", "reviewer": "Reviewer_Vz3t", "summary": "The paper introduces MolPhenix for tackling the contrastive phenomolecular retrieval problem. MolPhenix employs a uni-modal pretrained phenomics model, an inter-sample similarity-aware loss function, and conditions on the representation of molecular concentration. This approach effectively addresses challenges such as experimental batch effects, inactive molecule perturbations, and encoding perturbation concentration. Experimental results demonstrate the model's effectiveness in the retrieval task.", "questions": "- The retrieval accuracy results in the paper are reported as top 1% accuracy. However, this metric may be less informative when the retrieval set is very large, as the top 1% can still include a substantial number of molecules. Could the authors also provide results for top N accuracy, where N = 1, 10, and 100?", "limitations": "The authors have noted several limitations in their methods and experiments. Additional limitations are listed in the **Weaknesses** and **Questions** sections.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and the motivation is clear.\n- The identified challenges are highly relevant and significant in the biological context, and the proposed methods are logical and well-conceived.\n- The experimental results indicate substantial improvements over baselines across various settings in the retrieval task.", "weaknesses": "- In Table 2 and Table 4, there is a significant performance increase for DCL, CWCL, SigLip, and S2L compared to other baselines. The source of these improvements is unclear. Conducting an ablation study for each of the three components of the method, corresponding to the three challenges, would provide more insights.\n- The paper primarily compares MolPhenix with CLOOME and a few other general domain objectives. However, numerous related studies specifically within the molecular-phenotype contrastive learning domain [1-6] are not discussed or compared.\n- The paper emphasizes the retrieval task, with experimental results showing only the top 1% recall accuracy. This limits the overall impact of the findings. Considering broader application scenarios based on the proposed method could enhance the paper's significance.\n\n[1] Cross-modal graph contrastive learning with cellular images. bioRxiv 2022.\n\n[2] Contrastive learning of image- and structure-based representations in drug discovery. ICLR MLDD 2022.\n\n[3] Molecule-Morphology Contrastive Pretraining for Transferable Molecular Representation. ICML CompBio 2023.\n\n[4] MMCL-CDR: enhancing cancer drug response prediction with multi-omics and morphology images contrastive representation learning. Bioinformatics 2023.\n\n[5] Removing Biases from Molecular Representations via Information Maximization. ICLR 2024.\n\n[6] Learning Molecular Representation in a Cell. arXiv 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for providing detailed feedback for our paper. Below, we aim to address your concerns point-by-point\n\n**Concern #1: In Table 2 and Table 4, there is a significant performance increase for DCL, CWCL, SigLip, and S2L compared to other baselines. The source of these improvements is unclear.**\n\nWe assess the effectiveness of S2L, SigLip, and DCL losses by analyzing the gradient flow of the InfoNCE and DCL formulations. In particular, we analyze the cases where inactive molecules cause the gradient to vanish, inhibiting training using InfoNCE and variants thereof.\n\nDecoupled contrastive learning (DCL) loss is an effective alternative to the InfoNCE loss due to the modification by removing the positive term from the denominator:\n$$\n\\mathcal{L}\\_{\\text{DCL}} = - \\frac{1}{N}\\sum\\_{i=1}^N\n\\left[ \\log \\frac{\n\\exp(\\langle \\mathbf{z}\\_{x\\_i}, \\mathbf{z}\\_{m\\_i} \\rangle / \\tau)\n}{\n\\sum\\_{k = 1, k \\neq i}^{N} \\exp(\\langle \\mathbf{z}\\_{x\\_i}, \\mathbf{z}\\_{m\\_k} \\rangle / \\tau) + \\cancel{\\exp(\\langle \\mathbf{z}\\_{x\\_i}, \\mathbf{z}\\_{m\\_i} \\rangle / \\tau)} } \\right].\n$$\n\nThe authors show that when computing the gradient of InfoNCE, the term $q_{B,i}$ which they name the negative-positive coupling (NPC) term modulates the gradient of each sample and in cases where the term becomes small gradient flow is inhibited:\n\n$$\nq\\_{B,i} = 1 - \\frac{\n\t\\exp(\\langle \\mathbf{z}\\_{x\\_i}, \\mathbf{z}\\_{m\\_i} \\rangle / \\tau)\n\t}{\n\t\\sum\\_{k = 1, k \\neq i}^{N} \\exp(\\langle \\mathbf{z}\\_{x\\_i}, \\mathbf{z}\\_{m\\_k} \\rangle / \\tau) + \\exp(\\langle \\mathbf{z}\\_{x\\_i}, \\mathbf{z}\\_{m\\_i} \\rangle / \\tau)\n\t}.\n$$\n\nNPC can be small when the positive samples are too close to one another, when there is a small number of negative samples, or when the negative samples are too simple to discriminate versus the positive pair. By removing $\\exp(\\langle \\mathbf{z}\\_{x\\_i}, \\mathbf{z}\\_{m\\_i} \\rangle / \\tau)$ from the denominator, DCL simplifies the training thus removing the NPC term from the gradient calculation.\n\nWe hypothesize that in our case inactive molecules tend to be simple negatives, thus inhibiting the gradient flow that would otherwise be helpful to the model training. Training the model with a loss that separates the positive gradient term (attracting samples from two modalities) from the negative (repelling negative pairs) achieves higher overall performance. Similarly to DCL,  SigLIP and S2L losses are computed for each pair of samples independently, thus separating the positive and negative loss terms. Thus the gradient calculation for informative samples is unaffected by the NPC term, resulting in higher overall performance.\n\n**Concern #2: The paper primarily compares MolPhenix with CLOOME and a few other general domain objectives… numerous related studies specifically within the molecular-phenotype contrastive learning domain.**\n\nWe thank the reviewer for an in depth exploration of relevant works. We note that the second paper linked is an early presentation of CLOOME (2023) to which we extensively compare (2nd referenced paper in reviewer suggested list). In addition we utilized a number of strong baselines from the Image-Text multi-modality literature such as CWCL (2023), SigLIP (2023), and DCL (2021).\n\nWe agree with the reviewer that we can strengthen the pheno-molecular multi-modal related works section. We focussed our assessment on works that innovate on the methodological components but agree it is important to capture a broader perspective. We note that these additional works support the importance of MolPhenix as it demonstrates the importance of learning an effective joint embedding of phenomic experiments and molecular structures. MolPhenix contributions can be used in further improvement of utilizing phenomics and cell-profiliing experiments for cancer cell line drug response prediction (4th referenced paper in reviewer suggested list) and gene knockout predictions (6th referenced paper in reviewer suggested list). As the authors of “Cross-modal graph contrastive learning with cellular images” conclude in their paper: “There are still some challenges that need to be addressed, such as inherent data noise and batch effect. These could be resolved by designing specific encoders for cellular images, optimizing cross-modal fusion mechanisms, and introducing heterogeneous cross-modal data”. Our work aims to introduce general methods for extracting additional information from phenomic data such as k-patch averaging, S2L inter-sample similarity aware loss, and concentration encoding.\n\nWe have updated our related Molecular-Phenomic Contrastive Learning section to reference and discuss the listed works.\n\n**Concern #3: Could the authors also provide results for top N accuracy, where N = 1, 10, and 100?**\n\nIn the Image-Text domain it is common to provide N=1,10,100 0-shot class retrieval results. Since the number of classes between studies for a given dataset is consistent, top-K results always correspond to a semantically meaningful metric.\n\nHowever, in our case we believe these statistics might be misleading since different works evaluate on datasets of different sizes, thus making the N=1 recall task of varying difficulty in a dataset of 100 molecules VS 100,000. Other published works resort to a similar strategy, for example “Cross-modal graph contrastive learning with cellular images” in Table 1 report Hit@1, 5, 10 which actually correspond to percentages due to the subsampling of the overall data to 100 data points. Similarly, “Molecule-Morphology Contrastive Pretraining for Transferable Molecular Representation” report top 1, 5, and 10% retrieval in Figure 2.\n\nWe note that in the external dataset - held out dose evaluation setting, the dataset consists of 1639 molecules making the top1% include just 16 molecules total. \n\nWe thank you for a detailed and thorough assessment of this work. Please let us know if there are any additional points we can discuss, to improve your assessment of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for providing detailed feedback on the paper. We are appreciative of the general support regarding the thoroughness and value of our scientific work:

- “experimental design is robust, demonstrating the framework's superiority in various scenarios and contributing valuable insights to the field” - kfG5
- “significant step forward in the task of phenotype-molecular retrieval” - ELHC
- “The experimentation of this work is comprehensive and detailed to justify the proposed method.” - YXwu
- “comprehensive framework that addresses several key challenges.. demonstrates notably strong performance” - r9h5
- “the proposed methods are logical and well-conceived” - Vz3t

We also appreciate the comments on the clarity of delivery, which is crucial for interdisciplinary works:

- “The paper is well-structured and clearly written with clear explanations of the methodologies.”  - YXwu
- “The paper is well-structured and comprehensive” - kfG5
- “The paper is very well written” - ELHC
- “The paper is well-written and the motivation is clear.” - Vz3t

The reviewer feedback has been extremely useful in improving the work in terms of clarity and additional evidence. In our rebuttal, (1) broaden the scope of our work beyond pheno-molecular retrieval by highlighting and performing additional experiments, (2) we demonstrate additional encoders to support the generalizability of our guidelines, (3) enhance the overall clarity and scientific background.

We expanded our evaluation with additional experiments supporting the utility of MolPhenix beyond retrieval. We conducted experiments evaluating the learned latent space, pheno-activity prediction, and zero-shot biological perturbation matching. In the attached PDF document, reviewers will find a KNN evaluation of the MolPhenix latent space, assessing the learned embedding on 35 molecular property prediction tasks across the Polaris and TDC datasets (Table 4 attached PDF). We find that MolPhenix trained with Fingerprint embeddings consistently outperforms standalone input fingerprints, demonstrating that the MolPhenix latent space effectively clusters molecules according to their molecular properties. We observed an interesting effect where prediction quality is positively correlated with implied dosage, indicating that MolPhenix learns dosage-specific effects. Furthermore, we aim to expand on this analysis in the appendix of the full paper. Additionally, we point reviewers to pheno-activity experiments demonstrating that MolPhenix can predict dosage-dependent molecular activity, opening opportunities for in-silico prediction of previously unseen molecular structures and dosages. Furthermore, we performed zero-shot biological perturbation predictions, by pairing known biological relationships between molecular structures and gene knockout phenotypes. This analysis provides preliminary evidence that MolPhenix learns underlying biological signals. These experiments are described in appendices E1 and E2, highlighting the utility of the learned latent space for biological challenges beyond identifying pheno-similar molecules (Supplementary Figure 7, 8).

Several reviewers (YXwu, r9h5, ELHC) recommended expanding our evaluation to include additional encoders to demonstrate a broader capability of pheno-molecular recall guidelines. To that end, we conducted an additional study evaluating a supervised pre-trained vision encoder trained to predict the identity of a perturbation. The results are available in the included one-page document. In brief, we demonstrate the proposed guidelines generalize to a supervised CNN encoder used as a phenomolecular backbone. Additionally, we highlight our ablation study (Figure 5) where we investigate the impact of publicly available molecular fingerprinting methods as molecular encoders.

Finally, we received valuable feedback on clarifying background terminology, adding pheno-molecular prior works, and improving the clarity of our table legends. We aim for this work to be accessible to scientists across disciplines, and have added a glossary of terms to the appendix and expanded the related works. We also provide additional justification and intuition for the S2L inter-sample similarity-aware loss in individual response to reviewer r9h5.

We thank the reviewers for a careful reading of our paper and their broadly positive feedback. We believe that our changes, guided by your suggestions, strengthen the paper in terms of clarity and contribution for which we are grateful. We hope this work will be of interest to the broader community.

References for all the responses:

- [1] Sypetkowski, Maciej, et al. "Rxrx1: A dataset for evaluating experimental batch correction methods." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
- [2] Masters, Dominic, et al. "Gps++: An optimised hybrid mpnn/transformer for molecular property prediction." arXiv preprint arXiv:2212.02229 (2022).
- [3] Sypetkowski, Maciej, et al. "On the Scalability of GNNs for Molecular Graphs." arXiv preprint arXiv:2404.11568 (2024).
- [4] Chandrasekaran, Srinivas Niranj, et al. "JUMP Cell Painting dataset: morphological impact of 136,000 chemical and genetic perturbations." BioRxiv (2023): 2023-03.

Meta-review: The authors present a cross-modal framework that tackles key challenges in contrastive phenomolecular retrieval. The framework shows good performance, particularly in capturing dose-dependent effects through both implicit and explicit concentration encoding methods. The paper also includes a comprehensive ablation study, detailing the contributions of various components such as loss functions and concentration encoding, with results thoroughly presented in both the main text and rebuttal.  Overall, all reviewers agree that the paper makes an interesting contribution to phenomic-based drug discovery, addressing a class of practically significant problems. However, I suggest the authors provide a more thorough discussion and refinement in the writing and related work sections, particularly concerning the comments of kfG5 and Vz3t, to better acknowledge relevant contributions in the field.
Decision: Accept (poster)