CellCLIP - Learning Perturbation Effects in Cell Painting via Text-Guided Contrastive Learning
Authors: MingYu Lu
Ethan Weinberger
Chanwoo Kim
Su-In Lee
Venue: NeurIPS 2025 poster ()
Keywords: Contrastive learning
CLIP
cell morphology
Cell Painting
cross-modal retrieval
intra-modal retrieval
zero-shot Learning
Primary area: machine_learning_for_sciences
TL;DR: We propose CellCLIP, a multimodal contrastive learning framework for Cell Painting data. It aligns image profiles and perturbation descriptions using pretrained encoders and a novel channel-wise scheme.

Abstract: High-content screening (HCS) assays based on high-throughput microscopy techniques such as Cell Painting have enabled the interrogation of cells' morphological responses to perturbations at an unprecedented scale. The collection of such data promises to facilitate a better understanding of the relationships between different perturbations and their effects on cellular state. Towards achieving this goal, recent advances in cross-modal contrastive learning could, in theory, be leveraged to learn a unified latent space that aligns perturbations with their corresponding morphological effects. However, the application of such methods to HCS data is not straightforward due to substantial differences in the semantics of Cell Painting images compared to natural images, and the difficulty of representing different classes of perturbations (e.g. small molecule vs CRISPR gene knockout) in a single latent space. In response to these challenges, here we introduce CellCLIP, a cross-modal contrastive learning framework for HCS data. CellCLIP leverages pre-trained image encoders coupled with a novel channel encoding scheme to better capture relationships between different microscopy channels in image embeddings, along with natural language encoders for representing perturbations. Our framework outperforms current open-source models, demonstrating the best performance in both cross-modal retrieval and biologically meaningful downstream tasks while also achieving significant reductions in computation time. Code for our reproducing our experiments is available at https://github.com/suinleelab/CellCLIP.

CelICLIP Learning Perturbation Effects in Cell Painting via Text-Guided Contrastive Learning
Mingyu Lu; Ethan Weinberger, Chanwoo Kim, Su-In Lee Paul G. Allen School of Computer Science & Engineering
University of Washington
{mingyulu ewein, chanwkim, (atpmusiiii,uunneeeeeeec.). washington.edu
Abstract
High-content screening (HCS) assays based on high-throughput microscopy tech- niques such as Cell Painting have enabled the interrogation of cells' morphological responses to perturbations at unprecedented scale. The collection of such data promises to facilitate a better understanding of the relationships between different perturbations and their effects on cellular state. Towards achieving this goal, recent advances in cross-modal contrastive learning could, in theory, be leveraged to learn a unified latent space that aligns perturbations with their corresponding mor- phological effects. However, the application of such methods to HCS data is not straightforward due to substantial differences in the semantics of Cell Painting im- ages compared to natural images, and the difficulty of representing different classes of perturbations (e-g. small molecule vs CRISPR gene knockout) in single latent space. In response to these challenges, here we introduce CelICLIP, cross cross-modal contrastive learning framework for HCS data. CelICLIP leverages pre-trained image encoders coupled with novel channel encoding scheme to better capture relationships between different microscopy channels in image embeddings, along with natural language encoders for representing perturbations. Our framework out- performs current open source models, demonstrating the best performance in both cross-modal retrieval and biologically meaningful downstream tasks while also achieving significant reductions in computation time. Code for our reproducing our experiments is available at https: stttps://gituut http://sinweeeea//cllllaoa/
Introduction
grand challenge in cellular biology is understanding the impacts of different perturbations, such as exposure to chemical compounds or gene knockouts, on cellular function. In pursuit of this goal, a number of high-content screening (HCS) techniques have been developed that combine image-based deep phenotyping with high-throughput perturbations (20] 30). For example, the Cell Painting protocol
4 stains cells with six fluorescent dyes highlighting distinct cellular components (e.g.
Hoescht staining for nuclear DNA), which are then imaged in five channels via automated microscopy (Figure D). Cell Painting now routinely combined with other biotechnological advances to measure cells' response to thousands of chemical and/or genetic perturbations in parallel at low cost [0] 36
Despite the promise of this data, extracting meaningful representations of cells' underlying states from their images presents formidable challenges. Traditional HCS data analyses use custom software tools extract domain-expert-craftee morphological features (e.g. nucleus size) from each image, with the
resulting collection of features referred to as a cell's profile 66 8). More recent works have found that
self-supervised deep learning methods based on DINO
33 or masked autoencoder (MAE; (21)
Equul contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(4)
(e)
Creea-dartee magg emboddiing
CrossChannelFomee
Transforer
(b)
Fammer
Pooling
Dreses
Peature
Eetrweter
quif SNIL-5:00 CO
Figure 1: Overview of CeIICLIP. (a) Cells' responses to perturbation are measured by the Cell
Painting assay, which captures a set of distinct cellular components in five imaging channels. (b) CelICLIP aligns perturbation embeddings generated from natural language descriptions with embed- dings of corresponding images produced by a new CrossChannelorrme architecture designed to account for the idiosyncracies of Cell Painting. (c) For perturbation i, CrossChannelFormer takes as
input images of all cells receiving perturbation and pools them into a single embedding ps that takes into account the relationship between information contained in the different Cell Painting channels.
architectures can capture more subtle changes in morphology compared to human-designed features, resulting in profiles that better reflect known relationships between perturbations.
In order to more explicitly capture the relationships between perturbations' effects on cellular state, recent work has applied cross modal contrastive learning (CL) techniques [35] to learn aligned repre- sentations of both perturbation labels and Cell Painting images. In particular, recent work has applied CL to this task by treating images and their corresponding perturbation labels as paired samples from different modalities [16] [38]. Post-training, such methods can be applied to systematically identify perturbations with similar effects based on relative proximity in the perturbation encoder's representation space. Moreover, novel perturbations can be input to the perturbation encoder to predict their morphological effects based on distance previously seen perturbations.
Despite their promise, previous CL methods for Cell Painting come with significant drawbacks. First, unlike natural images, where the standard RGB channels all capture semantically related phenomena, microscopy images are comprised of a larger number of channels each capturing semantically independent information. Second, Cell Painting image-perturbattoo structure where multiple images are associated with the same perturbation label. Standard CL, however, treats all image pairs from different instances negatives, even if they share the same label, leading to a substantial number of false negatives where perceptually similar images are incorrectly pushed apart in the embedding space 15] 45). Furthermore, not all Cell Painting images faithfully represent the intended perturbation effect, as some perturbations may be unsuccessful due to technical issues (e.g.. poor guide efficiency with ERISPR-mediater gene knockouts), effectively resulting in false positive pairs. Yet, existing CL methods for HCS data fail to account for these properties (38] or are not openly available (16). Finally, previous works [6] (38) have exclusively focused on chemical perturbations, relying on graph representation learning techniques applied to chemical structures to obtain perturbation representations. Thus, these methods cannot be applied to other classes of perturbations (e.g. CRISPR gene knockouts), and it is unclear how to represent perturbations from different classes within a unified input space for CL.
pairs exhibit "many-to-one""
To address these issues, here we propose CelICLIP (Figure [p). CL framework designed for the unique challenges in Cell Painting perturbation data. Our framework employs off-the-shelf pretrained vision models alongside a novel channel-aware encoding scheme (Figure (Tb) to facilitate reasoning across the different sources of information captured in Cell Painting channels; in addition. our image encoding architecture incorporates techniques from the ciciiip--iinstane-eeariii literature pairs. To represent images' corresponding perturbations, CelICLIP employs natural language encoders, enabling ap- plication to multiple classes of perturbations with a single model. We benchmarked CelICLIP on pooile-perturbation retrieval for unseen compounds A and its ability to recover known biological relationships between perturbations [10 28 We find that CelICLIP achieves strong performance compared to the previous state-of -f-the--rttttttt while taking a fraction of the time to train. Our method also shows promising results pross-perturbation class matching, thereby providing a scalable and effective solution for biological discovery.
24 to account for the many-to-one nature of Cell Painting perturbation-imal
2

---

2 Background
Representation Learning for Cell Painting Self-supervised learning (SSL) deep learning tech- niques have been successfully applied to experimental microscopy data, with recent studies demon- strating their ability to capture intricate details of cellular morphology [40 [3] better than hand-crafted features as implemented in standard software packages like CellProfiler [8. Initial applications of SSL methods to Cell Painting relied on architectures designed for natural RGB images, where information across channels exhibits strong correlations [2]. On the other hand, Cell Painting channels each capture distinet biological structures (e.g. actin via phalloidin, mitochondria via MitoProbe, etc.). To account for this, recent works [2] 27 have proposed so-called channel-agnostic vision transformers (CA-ViTs), that use separate tokens for each channel in spatial patch rather than aggregating information across all channels. However, Ci-V substantially increase computational costs due to an increased number of tokens, since each sub-patch corresponds to a token
Contrastive Learning Cross-modal contrastive learning methods such CLIP B5] learn aligned latent representations across multiple modalities (e.g.. text and image). Such methods optimize a symmetric contrastive loss to maximize the similarity between correct pairs while minimizing it for incorrect ones. Specifically, given a batch of N cross-modal pairs {uu, r)) ] where u E 4 and E V (e-g. text and images), let fo and 96 be encoders that map inputs to embeddings p. fe (u.) and = go (tj), respectively. The CLIP loss consists of cross entropy objectives encouraging alignment from + and 2/ + V, with
learnable temperature parameter:
N Σ
exp((p; q:)/T) log N Σラ= exp((p: (/)/T) Σイ= exp((p) 4i)/T)
exp((ps (T)TT)
CCLP
log
(1)
While CL excels in cross-modality alignment, several challenges remain in its application to Cell Painting perturbation data. First, while a thorough body of work exists studying encoding strategies for natural images and natural language (39 46 II, encoding strategies specifically designed for Cell Painting images and cellular perturbations remain largely underexplored. Second, each perturbation typically associated with multiple images, so naively attempting align individual Cell Painting images with perturbation labels may introduce substantial number of false negatives. Third, pertur- bations in Cell Painting experiments can span variety of classes, including chemical perturbations, gene knockouts, and open reading frames (ORFs). Yet, previous contrastive learning approaches for Cell Painting, such as CLOOME (38 and MolPhenix [16] have exclusively focused on chemical perturbations, limiting their applicability. Finally, model weights for some recent state-of-the- methods (e.g. [16) are not openly available, and retraining these models from scratch is infeasible due to the use of large dlosed-sourc datasets and significant GPU resources. Thus, there exists need for nigh-performance open-source alternatives.
Methods
Here we present CelICLIP in detail. We first present CrossChameelFormes our image encoding scheme that facilitates CL with Cell Painting data (Section 3.I); due to the idiosyncracies of Cell Painting perturbation datasets, this encoding must be done with care, and we cannot naively reuse strategies for CL with natural images. We then present our approach for encoding perturbation labels (Section) 3.2 and outline our training strategy (Section 3.3
3.1
CrossChannelFormer
Our CrossChannelFormer architecture for image encoding (Figure [e) consists of three steps that allow to effectively account for the nuances of Cell Painting perturbation data without introducing prohibitive computational costs. We describe these steps below.
Step Cell profiles from natural image foundation models.
Recent works have developed image foundation models trained on natural images, such as DINOv2 33 which have demonstrated strong
For image of size la x with channels, CA-VITs produce 2 c tokens assuming patch size p p.
3

---

capabilities in capturing global structural image features. Rather than training new models from scratch, we choose to adopt these pretrained models as image encoders in the CelICLIP framework. However, unlike natural images with the standard set of RGB channels, Cell Painting images contain a variable number of channels corresponding to the specific stains used in an experiment. To work around this difference and enable models trained on natural images to be applied to Cell Painting data, we treat each Cell Painting channel as an independent grayscale image and extract embeddings separately. Formally, for each perturbation we denote the collection of Cell Painting images corresponding to that perturbation as U;- = { where ur E RCC XX denotes an individual image. For each image ug. we may apply a feature extractor the that maps individual channels ug to an embedding = = vh ( E gd. This produces a channel-wis -wise embedding matrix
2k [ 2 ... CTT E RCxd
that we employ as an image's 's profile. In our experiments, we used a frozen, pretrained DINOv2 (giant) model for our feature extractor v. and explore the impact of different choices for ee in Appendix A.I
Step 2: Pooling profiles within perturbations. The standard CLIP model aligns pairs of natural
images and corresponding text annotations. However, the presence of both false negative (i.e., images sharing the sample perturbation label) and false positive pairs (i.e., images resembling controls) may lead to subpar results when attempting align individual Cell Painting image profiles and correspond- ing perturbation annotations. To accommodate this issue, instead of aligning pairs of perturbation labels and individual images, we instead align perturbation labels with an aggregated summary of all images that received that perturbation. Specifically, for the set of images U, corresponding to perturbation i, we define their corresponding pooled profile for channel c
m(U,)* S (1= k= 1,... N.)) E IR"
where S(-) denotes permutation- invariann transformation function applied channel-wise (e.g., mean computed separately for each channel). For our experiments we use the gated attention pooling operator of llse al. [24] for S, and we explore the impact of different choices for in Appendix AA. In addition to improving performance on downstream tasks, per-perturbation pooling also improves computational efficiency: instead of computing a contrastive loss for each of the N; individual profiles associated with perturbation i, we compute single pooled representation. This reduces the number of positive pairs per perturbation from N; to one, leading to an approximate N,-fold speedup in training time compared to instance-level supervision.
Step 3: Reasoning across information from different channels. Beyond just different number of channels, the relationships between Cell Painting image channels exhibit substantial differences compared to those between natural image channels. In particular, while natural image channels share significant amount of information, Cell Painting channels correspond to distinct biological stains, each highlighting independent cellular structures. Thus, to effectively learn meaningful embeddings of Cell Painting images, it is necessary to explicitly reason between information in different channels 2 and naively reusing existing vision encoders designed for RGB images as done by CLOOME 38 may produce subpar performance. While previously proposed CA-ViT architectures (2) do reasoning, as discussed previously this ability comes at the cost of substantially increased computational overhead due to the larger number of tokens required.
enable
Thus, to enable cross-channe reasoning while minimizing computational costs, we introduce Cross- ChannelFormer, a specialized encoder architecture for CelICLIP. Unlike the standard Vision Trans- former (VIT: M... where each input token represents a multi-channel image patch, CrossChan- nelFormer takes pooled profiles u(U.) (Equation (3) that capture the global cellular features associ- ated with specific stains. We then introduce a set of learnable channel embeddings, [chn',... chn ). where each chn e Rd encodes information unique to its respective channel. Finally, we prepend learnable classifier token cls E Red to the sequence, which aggregates global image features across all channels. The resulting input sequence to the transformer is:
[cls, (UU)) + chn', (U(,)² + chn? ...... ccn?...... (..)) +chn
where ((,)) corresponds to the e-th channel of (((.)). This design is inspired by Bao et al. [ but differs in that we reason over the global per-channel information captured in ((U)) rather than local per-channel spatial patches. This results in a substantially reduced computational burden compared to CA-viTs (2 (27 as CrossChannelFormer requires only C' + tokens for each perturbation.

---

Following the original VIT, we feed the above sequence into Transformer encoder. The Transformer encoder consists of alternating layers of multi-head self-attention and MLP blocks, with layer normalization applied before each block and residual connections established after each block. The final layer representation of the CLS token serves as the projection in the latent space.
Altogether, our proposed CrossChannelFormer image encoding strategy offers three main advantages over previous approaches. First, our approach allows pretrained on natural image data, which are far more plentiful than specialized models pretrained on Cell Painting images. Second, by aligning pooled image profiles with perturbation labels instead of individual images, we effectively resolve the many-to-on one nature of Cell Painting perturbation data. Finally, our CrossChannelFormat technique captures relationships between Cell Painting channels with significantly better computational efficiency.
reuse off-the-shelf vision encoders
3.2 Perturbation Encoding via Natural Language
Previous contrastive learning methods for Cell Painting rely on pertuubaationn-las--ppeee encoders to represent perturbation treatments. For instance, CLOOME (38) encodes chemical compounds by passing Morgan fingerprints (32] through a simple multilayer perceptron (MLP), and Molphenix relies on graphical neural networks (GNN) [66]. This setup is not ideal, as different perturbation types (e.g., chemical compound vs gene knockouts) require distinct encoder networks, making it challenging to incorporate data from multiple perturbation types into the contrastive learning process.
To address this, we adopt simple
each perturbation using text. Since most perturbations and their associated metadata can be effectively captured through textual descriptions
37 11. text serves as an efficient intermediate modality for generalizing across multiple perturba- tion types. We construct a corresponding text prompt that encodes information on cell types and perturbation--peecfic details. For example, to encode the chemical compound butyric acid, drug affecting cell growth, we use the prompt:
"A cell painting image of U2OS cells treated with butyric acid, SMILES: CCCC(O)=O."
Similarly, for a CRISPR perturbation, the prompt is structured as:
"A cell painting image of U2OS cells treated with CRISPR, targeting genes: AP2S1."
By representing perturbations as text prompts, our approach facilitates encoding arbitrary pertur- bations from different classes, simplifying training across diverse perturbation types. It can also potentially integrate relevant textual metadata, enhancing perturbation retrieval across experiments.
For our experiments, we use pretrained BERT model 112 which uses the WordPiece tokenizer [6
and supports 512 tokens. The prompt template used is provided in Appendix B
3.3 CelICLIP Training Objective
In standard contrastive learning, the objective is to align matched data pairs by minimizing the loss in Equation (I). In Cell Painting, in addition to cross-modal retrieval [6 38). many intra-modal downstream tasks, such as matching related perturbations [0. maintaining adequate performance intra-modal tasks is non-trivial; for example, in the context of Cell Painting perturbation screens, many distinct perturbations induce highly similar morphological profiles, i.e., fo- (I(U))) z so- (n(U,)) for + j. where Sa- is oracle profile encoder. In this setting, treating all non-matching pairs as hard negatives can degrade retrieval performance by encouraging too much separation between biologically similar perturbations.
are also important. However,
To address this, we adopt the Continuously Weighted Contrastive Loss (CWCL) 41 in CeLICLIP. CWCL replaces binary labels with continuous similarity- -based weights (Figure Tb), applying soft labeling scheme within the cross-entropy loss. The loss helps training cross-modal models by ensuring that similarities picked up by a strong pretrained unimodal embedding model maintained in the cross-modal representation space. In our case, we apply the CWCL loss by computing target similarities between per-perturbattin pooled profiles that morphologically similar profiles remain close in CelICLIP's embedding space. For the ponile-to-pertturaatin direction, 21 + V, our adapted CWCL objective is given by
N
N
exp((p: 9i//T)
LOWCLuY
=
N JEIN
!!!
log
Σ exp((ps (p) 4x)/T)
(5)
1=1
j=1
5

---

Here represents the output of CrossChamneFoommer applied to images corresponding to perturbation 2. 9, corresponds to our encoding of the natural language description of perturbation and w denotes the similarity between the pooled profies μ(U,) and j(U;), used reweight the alignment from modality L/ to modality V. We compute w as the average channel-wise cosine similarity between pooled profiles In Appendix A.4 we provide results illustrating the benefits of our choice over other
potential contrastive losses.
For the perturbation-tt-proffft direction, V + 24 we apply the standard CLIP loss (Equation as we found computing similarities either within the input space V (i.e., discrete tokens) or in the projected space Q (i.e., BERT embeddings) did not provide meaningful signals for soft-label supervision. Thus, the final training objective is the sum of the two directional losses:
Fiotal LCWCLM-V + LCLIP.V-U-
(6)
4 Experimental Setup
In this section, we describe the datasets and tasks used evaluate our framework.
4.1
Cross-Modality Retrieval Between Profiles and Perturbations
We first benchmarked CelICLIP's performance by assessing its ability to retrieve test set perturbations given corresponding Cell Painting image profiles treated with each perturbation as done in prior works [6 35] 38]. That is, for a given model compute perturbation embeddings Q along with corresponding pooled Cell Painting profile embeddings P in the shared latent space. Given the per-perturbation pooled Cell Painting profile embeddings,
then compute cosine similarities with all perturbations' embeddings in the test set and retrieve the top-k most similar perturbations; idcally, the image profiles' true perturbation should be contained in this nearest neighbors Our evaluation metric, Recall @ (R @ k), measures whether the correct perturbation appears in the top-k retrieved results, with k 1, 5, 10. We denote this task profile-topperee perrturbaation retrieval.
Swapping the roles of perturbations and Cell Painting profiles, we may similarly evaluate perturbation- 1o-profile retrieval, where, given perturbation embedding, we compute similarities with pooled Cell Painting profile embeddings. For this task we again use Recall @ for evaluation.
4.2 Intra-Modality Retrieval on Biologically Mcaningful Tasks
To further assess if CelICLIP's image encoder captures biologically meaningful signals, we evaluated its profile embeddings P on the following three intra-modal retrieval tasks proposed in Chan- drasekaran et [10] and Kraus et al. [28]. For each task, we assess if the embeddings corresponding to cells receiving a given perturbation are close to those for other biologically related perturbations.
Replicate detection measuring how well replicates across batches of given perturbation can be distinguished from negative controls. In other words, profiles treated with the same perturbation- but collected in separate patches-shoule be close in embedding space.
Sister perturbation matching assesses whether cells treated with "sister" perturbations targeting the same genes, which should thus induce similar morphological changes, are close in embedding space. This includes comparisons within the same perturbation class (e.g., compound-compound) and across different classes (e.g.. CRISPR-compound)
Zero-shot gene-gene relationship recovery evaluates models' generalization by assessing whether their profile embeddings for cells receiving gene knockout perturbations in previously unseen dataset reflect known relationships between genes. Methods that construct accurate relational maps may be more likely to uncover novel biological interactions.
For our evaluations on the above tasks, we compute batch-efffect-correeeed pooled embeddings ő; using matched negative controls as in Celik al. (9. For given condition i, we compute the cosine similarity between D; and embeddings for other conditions p,and then rank them to perform retrieval.
SSpecifically, wty
(a(U)) (U
0.5, ensuring wt/ [0, 1].

---

Table 1: Benchmarking CeIICLIP and baseline methods on perturbation-to-preee and profile-to- perturbation retrieval performance for unseen molecules from Bray al. 44 We report mean Recall 1, @5, and @ 10 + standard deviation across random seeds for both tasks. Higher recall corresponds to better performance. Best results are shown in bold.
Model
Perturb-to-profile (%)
Profile-to-perturb (%)
R@
R@55
R
R@1
R@5
R 10
CLOOME
0.27 + 0.20
1.25 = 0.42
2.46
+ 0.56
0.07 = 0.09
1.44 + 0.48
2.56 = 0.89
CLOOME
0.45 + 0.13
1.60
0.12
3.04
0.12
0.48 0.16
1.79 + 0.13
3.12 0.25
CLOOME (CLOOB)
0.37 + 0.10
1.56
0.10
2.75
+ 0.02
0.20 + 0.03
1.76 0.10
3.13 0.17
MolPhenix* (S2L)
0.28 1 0.11
1.56
3,00
0.26
0.42
1.54
3.01 0.20
MolPhenix* * (SigCLIP)
0.56
0.12
2.78 = 0.23
4.30
+ 0.16
0.66:...2
2.55
4.34 0.24
CelICLIP (Ours)
1.18
0.20
4.49 + 0.06
7.37 +: 0.20
1.25 + 0.10
4.82 +: 0.10
7.39 0.23
1 Indicates that CLOOME uses the same image profile encoding procedure architecture as MolPhenix*
Retrieval performance is evaluated against ground-truth labels via task-specific metries used in previous work. For replicate detection and sister perturbation matching, we report mean average precision (mAP) in Chandrasekaran al. [T]. For biological relationship recovery, we follow Kraus et al. [28) and report recall over the top and bottom 5% of the similarity distribution (i.e.. 10%
recall), Further details regarding the computation of these metrics are provided in Appendix
4.3 Datasets
For cross-modality retrieval, following Sanchez-Fernander et al. (38] we utilize curated version of Bray et al. [4, comprising approximately 284,034 five-channel Cell Painting images corresponding to 10,578 small-molecule perturbations. We partitioned the dataset by perturbation into train, validation, and test sets with a 70/10/20 split, resulting in 2,115 unseen small molecules in the test set.
For replicate detection & sister perturbation matching, we employ CPJUMP1 [0O, which features 186,925 eight-channel microscopy images. These images include three bright-field channels in addition to five Cell Painting dye channels. They are perturbed across 650 distinct perturbations, including compounds and genetic modifications such as CRISPR and ORF interventions. For each perturbation class, we applied 70/10/20 split for training, validation, and testing.
RxRx3-core (28), a curated subset of RxRx3 [15], covering 736 gene knockouts. Gene-gene relationships are obtained from set of public databases (Reactome, HuMAP, SIGNOR, StringDB, and CORUM: Table S.10) as Kraus et [27]
For zero-shot gene-gene relationship recovery,
Further details on datasets and preprocessing are provided in Appendix D
4.4 Model training
For cross-modality retrieval evaluation Bray al. 44, CelICLIP was trained with 50 epochs using a batch size of 512 and an AdamW optimizer. The learning rate was set at 2 x 10-4 with cosine annealing and restart. The temperature parameter, т, initialized to 14.3. For replicate detection and sister perturbation matching in CPJUMPI I [10], we reused our model trained on the Bray et al. dataset and fine-tuned it for another 50 epochs, using the same parameter settings as during their initial training. For gene relationship recovery using RxRx3-core, we perform this evaluation in "zero-shot" manner and evaluate models trained on Bray et al. without subsequently fine-tumi finettuning on RxRx3-core. More details about training CelICLIP and other baselines can be found in Appendix
5 Results
5.1
Cross-modality retrieval
We present our results for cross-modality retrieval tasks (i.e., perturbation-to-profff and profile-to-
perturbation) in Table [ For our benchmarking we compared CelICLIP against previous of the-art CL methods for Cell Painting perturbation screens: CLOOME [8 and MolPhenix [16]. For CLOOME we used the same architecture with optimal hyperparameters specified in Sanchez-
7

---

Table 2: Ablation studies of various vision and perturbation encoder combinations and retrieval performance in 4 We report mean Recall 1, @5, and @ 10 standard deviation across random seeds for perturbation-to-pee to-offile and Fingerprint; O indicates text prompt. Best results are shown in bold.
to-perturbation retrieval tasks. A indicates Morgan
Vision Encoder
Perturb. Encoder
Train (hr)
Perfurb-fo-prollle
Profilc-t-pperuub (%)
R@1
R@5
R
R@1
Re5
R@10
ResNect-111
A MLP
49.9
0.27 = 0.20
1.25 0:42
2.46 :: 0.56
0.07
0.09
1.44 =
2.56 :: 0.89
ResNet-101 101
A MPNN+
45.2
0.20 0.46
1.30 + +.311
1.03 + 0.21
0.15
055++-11 0:12
1.50- - 041
2.78 + 0.51
ResNet-101
O BERT
0.74 0.07
2.67
4
0.46
3,95 0.49
0.57
CA-MAB
25.6
1.57
2,00
ChanneVii
U + BERT
29.5
0.78 - 0.15
3.23 0:33
5.15 005::000
0.69 2: 0.18
2.92 0.39
5.01 0.38
CrosChannsPormmrr
D BERT
12.2
1,16 0.12
3,95 1
5.74 0.36
1.05 0.08
3.84 0.43
5,90 0.28
CrosChannelFormes
0.06
1.37
Truned without poolice pooling,
Table 3: Retrieval performance of CelICLIP trained with various perturbation prompts. R @5, and @ 10 + standard deviation across random seeds for perturbation copttttiionott-pppe to-profill and profile-o-peerurbin retrieval tasks.
Prumpt modification
Perturb-te-profile (%)
Profite-to-perturb (%)
R@
R@5
R@
R@01
R@5
R@1
Original
1.18 1.18 0.20
4.49 + 0.06
7.37 0.20
1.25 +: 0.10
4.82 = 0.10
7.39 + 0,23
Without SMILES string
0.13
0.68
0.19
1.16
0.07
0.63 0.17
1.24
Without cell type information
1.11
4.02 0.25
6.59
1.10::0.23
4.07 = 0.29
6.36 + 0.32
Without drug name
0.29
4
7.37 - 0.29
1.61 + 0.226
4.89 = 0.19
7.21 0.27
DrugBank description
0.16
4.26 0.43
6.94 0.37
1.16 0.17
4.34 0.36
6.71 0.26
Fernandez et al. [88 along with subsequent variations of the model described in Fradkin et al. [16]
Because Fradkin et al. [J6's implementation of MolPhenix relied on closed-source components trained on non-public datasets, compared against an approximation of their framework which we designate MolPhenix We refer Appendix B for further details.
Overall, we found that CelICLIP demonstrated substantially higher performance at both cross- modsl retrieval tasks compared to baseline methods. To understand the source of performance gains in CelICLIP, we conducted a series of ablations to understand the contribution of each component in CelICLIP to retrieval performance. Specifically, starting with CLOOME's proposed encoding scheme (Appendix where individual images encoded using ResNet50 are aligned with chemical perturbations encoded using Morgan fingerprints combined with an MLP, we gradually replaced each of CLOOME's components with those of CelICLIP and assessed each change's impact on model performance (Table 2). We describe our findings from these experiments in detail below.
Language can effectively represent perturbations. We began by replacing CLOOME's chemical structure encoder, which feeds chemicals' Morgan fingerprints through a multi-layer perceptrion (MLP), with the natural language encoder used in CelICLIP while holding all other model components fixed. We found that this change alone yielded significant performance gains for both retrieval tasks. Notably, we found this result continued to hold even after replacing the generic MLP used in CLOOME with an MPNN+ network designed specifically for molecular property prediction [1..
Effect of perturbation prompts. We next investigated alternative prompting strategies to assess the contribution of individual components. To do so, we re-trained CelICLIP after removing specific elements of the prompt template (Section 3.2). such as cell type, drug name, SMILES string, one at time (Table 3). We found that removing the SMILES string caused a substantial drop in performance, while removing cell type information produced a smaller degradation. In contrast, removing drug names had little effect, likely because the richer SMILES representation already captures most of the relevant information.
We also tested whether prompts could enriched with external metadata by incorporating DrugBank descriptions (26) for compounds present in the database. However, this did not yield performance improvements over the original template. likely reason the limited coverage of DrugBank, which includes only about 10% of the compounds in Bray et al. 44, many of them experimental

---

Table 4: Benchmarking results for replicate detection and sister perturbation matching on CP-JUMPI 10. and cene-gene relationship recovery with RxRx3-core (28]. Performance is evaluated using mean average precision (mAP) across perturbations for replicate detection and sister perturbation matching, and recall for relationship recovery. Higher values correspond to better performance.
Method
Replicate Detection (mAP)
Sister Perturb, Matching (mAP)
Gene-Gene Relationship Recovery (Rocall)
Within Claes
Across Class
CORUM
HuMAP
Reactome
SIGNOR
StringDB
Crozz-model
CLOOME
.575
.215
.226
.997
.679
.327
.309
.510
MolPhemix
011
CeNCLIP
.663
.413
.043
.714
.778
.427
.388
.618
Unimadal (seff aupervieed)
OpenPhenom-S116
.377
.219
031
649
723
418
386
.579
Unimodal (weakly supervised)
VIT-L/16
.133
.283
002
.681
.788
.388
.30
.587
molecules. Overall, these findings suggest that our original prompting strategy is already highly effective.
Cross-channel reasoning improves retrieval performance. We next investigated the impact of CrossChannelFormer's s ability to reason across global Cell Painting channel information. To do so, we replaced CLOOME's ResNet image encoder with our CrossChannelFormee encoder. To isolate the effects of CrosCChanelFormer'ss ability to reason across channels from the effects of per-perturbation pooling, in this experiment we removed the CrossChannelFormerr pooling function and trained the resulting model (denoted as CrossChannefformer
Table 2 on individual iiage-perturbattin pairs.
We found that this change led to an additional increase in model performance. To understand how our approach compared to previous channel- vision encoding approaches, we ran this same experiment using a channel agnostic MAE (CA-MAE) and ViT (ChannelViT) as vision encoders. We found that CrossChannelFormee consistently outperformed these baselines on both tasks. This demonstrates that image embeddings extracted from models pretrained on natural images can be effectively leveraged with Cross CrossChannelFormet Additionally, our approach substantially reduced training time, achieving 3.9 times speedup compared to CLOOME and a 2.2 times speedup compared to other channel-agnostic methods. This efficiency stems from CrossChannelFormet operating in compact feature space and requiring only C' + 1 tokens per instance.
Pooling yields improved alignment and computational efficiency. Finally, we evaluated the impact of the attention- based pooling operator [2 within CrossChannelFormer compared to instance- level training (i.e., no pooling). We found that including pooling resulted in yet another increase in model performance. In addition, by reducing the number of pairs for contrastive loss computation, pooling yielding a 6.7 times speedup in training time relative to instance-level training. We also explored the impact of varying attention-bra based pooling yielded the best results among the pooling operators we considered. Moreover, to validate that the attention pooling learns to prioritize cells that most strongly reflect the perturbation effect, we performed a top-n removal experiment: for the Bray et al. 4 retrieval tasks, we removed the n samples with the highest attention weights for each perturbation and then recomputed retrieval performance. This led to a substantial performance drop (Table S.3). demonstrating that attention pooling effectively identifies and emphasizes the subset of cells carrying the strongest perturbation signal.
choice of pooling operator (Appendix [A.2). and found that
Overall, our results demonstrate that each of our main design choices in CelICLIP contributes to improved retrieval performance and computational efficiency compared to prior work.
5.2 Intra-modality retrieval
For our intra-modal retrieval benchmarking tasks, only image profile embeddings (and not per- turbation embeddings) are required. Thus, to further contextualize our model's performance, for these tasks we also compared against OpenPhenom (27]. state-of-ther the-arh-rr unimodal self-supervised representation learning model for Cell Painting images. Specifically, for our experiments we used OpenPhenom-S/11 which was the only model openly released by Kraus et al. [27] As in Kraus et al. (2], we also provide results for unimodal weakly supervised vision transformer baseline trained to predict perturbation labels for further comparison. Finally, for these experiments compared

---

against the best performing variants of CLOOME and MolPhenix in our cross-modal retrieval results.
Replicate detection and sister perturbation matching. We report results for these tasks in T We found that cross-modal CL outperformed our unimodal baselines. Moreover, within the considered CL. approaches, we found that CelICLIP strongly outperformed all other approaches for replicate detection. For perturbation matching, when comparing perturbations targeting the same genes within the same perturbation class (e.g., compounds vs compounds), CeIICLIP again performs the best among all approaches. For cross-class perturbation matching, (e.g. CRISPR vs compounds), CeIICLIP again performs the best, though the overall performance for all machinee-earringg -aannidg-- methods remains low. This suggests that, despite targeting the same gene, morphological changes across different perturbation classes remain highly distinct, aligning with previous findings [1O).
Recovering known biological relationships. Table reports recall known biological relation- ships among genetic perturbations in RxRx3-core. Across all benchmark databases, found that CelICLIP achieved the best recovery of known gene-gene relationships compared to baseline models. Notably, found that replacing the CWCL loss used in CelICLiP with the standard CLIP loss leads to worse performance on this task (Table SS.5). illustrating the benefits of using soft labeling for alignment in the image profile space P. Altogether, these results further demonstrate that CeIICLIP can recover meaningful relationships between perturbations its image profile latent space.
Conclusions
Here we addressed the challenge representation learning for Cell Painting perturbation screens by introducing CelICLIP, a cross-modall CL framework that unifies perturbations across classes through textual descriptions. As part of our framework, we also developed CrossChannelFormet a multi- instance, transformer-basee architecture that efficiently captures channel dependencies and processes profile data while reducing computational costs. Our results demonstrate that CelICLIP improves cross-modal retrieval performance, intra-modal retrieval for downstream tasks, and generalization across perturbation types. Overall, CelICLIP offers a promising solution for analyzing high-content morphological screening data, and future work will explore the impact of various contrastive losses and the contributions of each Cell Painting channel to downstream performance.
Limitation & Future Work While CelICLIP demonstrates strong performance, current evaluations are limited to datasets collected in previous works. Incorporating further validation CelICLIP's results via additional wet lab experiments would better assess its real-world applicability. Another limitation is data uccessibility: the high cost of generating Cell Painting images has resulted in many datasets and models remaining proprictary, limiting the reproducibility and benchmarking of models in this space more generally. Future work should prioritize the development of open, representative benchmarks. Finally, by aligning Cell Painting images with textual descriptions of perturbations, CelICLIP also opens new directions for conditional generation of phenotypic representations from natural language, paving the way for text-driven synthesis and simulation in drug discovery.
7 Acknowledgement
We thank members of the Lee lab for providing feedback on this project and the reviewers for their constructive comments. This work was funded by the National Science Foundation (DBI-1552309 and 155-1759487;; and the National Institutes of Health [RO1 AG061 132, RO1 EB035934, and RF1 AG088824].
10

---

References
[1] Dosovitskiy Alexey. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv: 2010.11923 2020.
[2] Yujia Bao, Srinivasan Sivanandan, and Theofanis Karaletsos. Channel vision transformers: An image is worth с x 16x x 16 words. arXiv preprint arXiv:2309.161000 2023.
[3] Yoav Benjamini and Yosef Hochberg. powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological), 57(1):289-3066 1995.
Controlling the false discovery rate: a practical and
[4] Mark-Anthony Bray, Shantanu Singh, Han Han, Chadwick T Davis, Blake Borgeson, Cathy Hartland, Maria Kost-Alimova, Sigrun M Gustafsdottir, Christopher C Gibson, and Anne E Carpenter. Cell painting, a high-content image-based assay for morphological profiling using multiplexed fluorescent dyes. Nature protocols, 179:1777-1777, 2016.
[5] Jaeseok Byun, Dohoon Kim, and Taesup Moon. Mafa: Managing false negatives for vision- language pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27314-27324, 2024.
[6] Juan C Caicedo, Sam Cooper. Florian Heigwer, Scott Warchal, Peng Qiu, Csaba Molnar, Aliaksei S Vasilevich, Joseph D Barry, Harmanjit Singh Bansal, Oren Kraus, et al. Data-analysis strategies for image-based based cell profiling. Nature methods, 14(9):849 063:88393,6,,,, 2017.
[7] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski. and Armand Joulin. Emerging properties in self-supprvised vision transformers. In Proceedings of the IEEE/VV international conference on computer vision, pp. 9650-9660, 2021.
[8] Anne Carpenter, Thouis R Jones, Michael R Lamprecht, Colin Clarke, In Han Kang, Ola Friman, David A Guertin, Joo Han Chang, Robert A Lindquist, Jason Moffat, et al. Cellprofiler: image analysis software for identifying and quantifying cell phenotypes. Genome biology, 1-11, 2006.
[9] Safiye Celik, Jan-Christian Huetter, Sandra Melo, Nathan Lazar, Rahul Mohan, Conor Till- inghast, Tommaso Biancalani, Marta Fay, Berton Earnshaw, and Imran S Haque. Biological cartography: Building and benchmarking representations of life. In NeurIPS 2022 Workshop Learning Meaningful Representations of Life, 2022.
[10] Srinivas Niranj Chandrasekaran, Beth A Cimini, Amy Goodale, Lisa Miller, Maria Kost- Alimova, Nasim Jamali, John G Doench, Briana Fritchman, Adam Skepner, Michelle Melanson, et al. Three million images and morphological profiles of cells treated with matched chemical and genetic perturbations. Nature Methods, pp. 1-8, 2024.
[11] Yiqun Chen and James Zou. Genept: simple but effective foundation model for genes and cells built from chatgpt. bioRxiv, pp. 2023-10, 2024.
[12] Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pp. 4171-4186, 2019.
[13] Michael Doron, Théo Moutakanni, Zitong S Chen, Nikita Moshkov, Mathilde Caron, Hugo Touvron, Piotr Bojanowski, Wolfgang M Pernice, and Juan C Caicedo. Unbiased single-cell morphology with self-supervised vision transformers. bioRxiv, 2023.
[14] Kevin Drew, Chanjae Lee, Ryan Huizar, Fan Tu, Blake Borgeson. Claire D McWhite, Yun Ma, John B Wallingford, and Edward M Marcotte. Integration of over 9,000 mass spectrometry experiments builds global map of human protein complexes. Molecular systems biology, 13 (6):922 2017.
[15] Marta M Fay, Oren Kraus, Mason Victors, Lakshmanan Arumugam, Kamal Vuggumudi, John Urbanik, Kyle Hansen, Safye Celik, Nico Cernek, Ganesh Jagannathan, et Rxrx3: Phenomics map of biology. Biorxiv, pp. 2023-02, 2023.

---

[16] Philip Fradkin, Puria Azadi, Karush Suri, Frederik Wenkel, Ali Bashashati, Maciej Sypetkowski, and Dominique Beaini. How molecules impact cells: Unlocking contrastive phenomolecular retrieval. arXiv preprint (rXxv:2409.08300 2024.
[17] Andreas Fürst, Elisabeth Rumetshofer, Johannes Lehner, Viet T Tran, Fei Tang. Hubert Ram- sauer, David Kreil, Michael Kopp. Günter Klambauer, Angela Bitto, et al. Cloob: Modern hopfield networks with infoloob outperform clip. Advances in neural information processing systems, 35:20450-20468, 2022.
[18] Mare Gillespie, Bijay Jassal, Ralf Stephan, Marija Milacic, Karen Rothfels, Andrea Senff- Ribeiro. Johannes Griss, Cristoffer Sevilla, Lisa Matthews, Chuqiao Gong, et al. The reactome pathway knowledgebase 2022. Nucleic acids research, 50(D1):D687-D92, 2022.
[19] Madalina Ciurgiu, Julian Reinhard, Barbara Brauner, Irmtraud Dunger-Kaltenbar Gisela Fobo, Goar Frishman, Corinna Montrone, and Andreas Ruepp. Corum: the comprehensive resource of mammalian protein complexes-200 2019.. Nucleic acids research, (7(DD):D55--DD 2019.
[20] Jiacheng Gu, Abhishek Iyer, Ben Wesley, Angelo Taglialatela, Giuseppe Leuzzi, Sho Hangai, Aubriamna Decker, Ruoyu Gu, Naomi Klickstein, Yuanlong Shuai, et al. Mapping multimodal phenotypes to perturbations in cells and tissue with crisprmap. Nature Biotechnology, pp. 1-15, 2024.
[21] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 16000-16009, 2022.
[22] Markus Hofmarcher, Elisabeth Rumetshofer, Djork-Are Clevert, Sepp Hochreiter, and Gunter Klambauer. Accurate prediction of biological assays with high-throughput microscopy images and convolutional networks. Journal of chemical information and modeling, 59(3): 166-1111,,, 2019.
[23] Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovec. Ogb- Isc: A large-scale challenge for machine learning graphs. arXiv preprint arXiv:2103.09430, 2021.
[24] Maximilian Ilse, Jakub Tomezak, and Max Welling. Attention- based deep multiple instance learning. In International conference on machine learning, pp. 2127-2136. PMLR, 2018.
[25] Alexandr A Kalinin, John Arevalo, Loan Vulliard, Erik Serrano, Hillary Tsang, Michael Bornholdt, Bartek Rajwa, Anne E Carpenter, Gregory P Way, and Shantanu Singh. A versatile information retrieval framework for evaluating profile strength and similarity. bioRxiv, 2024.
[26] Craig Knox, Mike Wilson, Christen M Klinger, Mark Franklin, Eponine Oler, Alex Wilson, Allison Pon, Jordan Cox, Na Eun Chin, Seth A Strawbridge, et al. Drugbank 6.0: the drugbank knowledgebase for 2024. Nucleic acids research, 52(D1):D1255DD12)
2024.
[27] Oren Kraus, Kian Kenyon-Dean, Saber Saberian, Maryam Fallah, Peter McLean, Jess Leung, Vasudev Sharma, Ayla Khan, Jia Balakrishnan, Safiye Celik, et al. Masked autoencoders for microscopy are scalable learners of cellular biology. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11757-11768, 2024.
[28] Oren Kraus, Federico Comitami, John Urbanik, Kian Kenyon-Dean Lakshmanan Arumugam, Saber Saberian, Cas Wognum, Safiye Celik, and Imran S Haque. Rxrx3-core: Benchmarking drug-target interactions in high-content microscopy. arXiv preprint arXiv:2503.201 2025.
[29] T Kudo. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing, arXiv preprint 100.::1800002222 2018
[30] Takamasa Kudo, Ana M Meireles, Reuben Moncada, Yushu Chen, Ping Wu, Joshua Gould, Xiaoyu Hu, Opher Kornfeld, Rajiv Jesudason, Conrad Foo, et al. Multiplexed, image-based pooled screens primary cells and tissues with perturbview. Nature Biotechnology, pp. 1-10, 2024.
12

---

[31] Dominic Masters, Josef Dean, Kerstin Klaser, Zhiyi Li, Sam Maddrell-Mander, Adam Sanders, Hatem Helal, Deniz Beker, Ladislav Rampášek, and Dominique Beaini. Gps++: An optimised hybrid mpnn/transformer for molecular property prediction. arXiv preprint arXiv:2212.02222, 2022.
[32] Harry L Morgan. The generation of a unique machine description for chemical structures-a technique developed at chemical abstracts service. Journal of chemical documentation, 5(2): 107-113, 1965.
[33] Maxime Oquab, Timothée Dareet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
[34] Livia Perfetto, Leonardo Briganti, Alberto Calderone, Andrea Cerquone Perpetuini, Marta lannuccelli, Francesca Langone, Luana Licata, Milica Marinkovic, Anna Mattioni, Theodora Pavlidou, al. Signor: a database of causal relationships between biological entities. Nucleic acids research, 44(D1): D5445D5544 2016.
[35] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pp. $748-8763, PMLR, 2021.
[36] Meraj Ramezani, Erin Weisbart, Julia Bauman, Avtar Singh, John Yong, Maria Lozada, Gre- gory P Way, Sanam L Kavari, Celeste Diaz, Eddy Leardini, et al. A genome-wide atlas of human cell morphology. Nature Methods, pp. 1-13, 2025.
[37] Shaghayegh Sadeghi, Alan Bui, Ali Forooghi, Jianguo Lu, and Alioune Ngom. Can large language models understand molecules? BMC bioinformatics. 25(1):225, 2024.
[38] Ana Fernadez,,r,,ndzz, Elisabeth Rumetshofer, Sepp Hochreiter, and Günter Klambauer. Cloome: contrastive learning unlocks bioimaging databases for queries with chemical structures. Nature Communications, 14(1):7339, 2023.
[39] Rico Sennrich. Neural machine translation of rare words with subword units. arXiv preprint 100.v:1508.07909 2015.
[40] Srinivasan Sivanandan, Bobby Leitmann, Eric Lubeck, Mohammad Muneeb Sultan, Panagiotis Stanitsas, Navpreet Ranu, Alexis Ewer, Jordan E Mancuso, Zachary F Phillips, Albert Kim, et al. pooled cell painting crispr screening platform enables novo inference of gene function by self-supervised deep learning. bioRxiv, pp. 2023-08 00,,0,822 2023.
[41] Rakshith Sharma Srinivasa, Jaejin Cho, Chouchang Yang. Yashas Malur Saidutta, Ching- Hua Lee, Yilin Shen, and Hongxia Jin. Cwel: Cross-modal transfer with continuously weighted contrastive loss. Advances in Neural Information Processing Systems, 36, 2023.
[42] Maciej Sypetkowski, Frederik Wenkel, Farimah Poursafaei, Nia Dickson, Karush Suri, Philip Fradkin, and Dominique Beaini. On the scalability of gnns for molecular graphs. Advances in Neural Information Processing Systems, 37:19870-19906, 2024.
[43] Damian Szklarczyk, Annika L. Gable, Katerina C Nastou, David Lyon, Rebecca Kirsch, Sampo Pyysalo, Nadezhda T Doncheva, Marc Legeay, Tao Fang. Peer Bork, et al. The string database in 2021: customizable protein-protein networks, and functional characterization of user-uploaded gene/measurement sets. Nucleic acids research, 49(D1):D005-D61 2021.
[44] Iulia Turc, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint arXiv: 1oXiv:9908) 1908.08962, 2019.
[45] Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. Medclip: Contrastive learning from unpaired medical images and text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, Conference on Empirical Methods Natural Language Processing, volume 2022, pp. 3876, 2022.
13

---

[46] Yonghui Wu. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint 1rXiv:160.008/44 2016.
[47] Zheni Zeng, Yuan Yao, Zhiyuan Liu, and Maosong Sun. A deep-learning system bridging molecule structure and biomedical text with comprehension comparable to human professionals. Nature communications, 13(1):862, 2022.
[48] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference computer vision, pp. 11975-11986, 2023.
[49] Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint 3xXiv:3333.00155 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims stated the abstract and introduction accurately reflect the contributions and scope of the paper. These claims are supported by the experimental results and analyses presented throughout the work.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification: included in the Limitation & Future work Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [NA]
Justification: Our paper does not include theoretical results. Instead, we show performance improvement of method through experimental evidence.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide a detailed description of our proposed method, CeIICLIP, and its
components in Section 3 Implementation details for all baselines are included in Appendix E
All experiments are conducted using publicly available datasets, as described in Appendix D
Experiment setups and evaluation metrics are described in Section and Appendix C
We
also include our code the supplementary to support full reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
16

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: All experiments conducted using publicly available datasets, as described in Appendix Experiment setups and evaluation metries are described in Section 4 and Appendix C
We also include code in the supplementary support full reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 //mips.cc/public. www.c//giuee/e/eeee httpc///bbbbceggadee////etetbaaa Cttp///ooeeSSeieeeeeeeeeee wwwteeessuuiiiiiiiniiiiiii wwtssssssinoiiyyyo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Datasets details, including preprocessing and train/val/test splitting pro- vided in Appendix D Model and baselines training details for each datasets are provided in Appendix [
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report appropriate statistical measures throughout the paper. For cross- modal retrieval experiments, we provide confidence intervals. For zero-shot biological relationship recovery, we report recall across varying similarity thresholds. For replicate detection and sister perturbation matching, we compute mean and standard deviation across time points, perturbation types, and cell types.
17

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the eπror bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
show in tables
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Experiment compute resources are provided in Appendix B and Appendix Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttpp/://eeurppp.cc///////eeees
Answer: [Yes]
Justification:
This work complies with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
18

---

Justification: This work does not directly pose any foresecable societal impacts. While morphological profiling techniques could, theory, be repurposed for harmful applications such as chemical weapon development, this risk is highly speculative and unlikely given the nature of the data and the absence of actionable chemical synthesis or targeting mechanisms in this work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no known risks of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All baseline models, datasets, and preprocessing code used in this work are properly cited, and their licenses and terms of use are fully respected. We include citations to the original sources, specify dataset versions where applicable, and ensure compliance with license requirements for all reused assets.
19

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
This work uses publicly available datasets and does not introduce any new datasets.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This work does not involve any human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
20

---

Answer:
[NA]
Justification:
This work does not involve any human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs were used solely for auxiliary tasks such as table formatting, figure refinement, and writing assistance.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non standaadd components.
Please refer to our LLM policy for what should or should not be described.
21

---

Table S.I: Retrieval performance of CelICLIP trained with profiles generated from various pretrained imaging models on perturb-to-proofe profile and profile-to-perturi tasks. Results are reported as Recall @ (%) for k = 1, 1,5, 10.
Image Encoding Rackbone
# of Params of GELOPS
Perturb-to-profile (%)
Profite-to-perttrb (%)
R@
R@5
RG:10
R@1
R@5
DINOv1
86. 86.4M
66.87
0.75 4: 0.08
2.77 2: 0.13
0.33
0.86 4: 0.16
2.77
00:00.. 0.27
4.80 1: 0.19
DINOv2 (small)
22.06M
5.53
1.19 0.32
1,06
4: 0.227
6.57 0.33
1.32 1 0.17
1,34
2.39
6.63 1: 0.19
DINOy2 (banc)
86,55M
21.97
11991.0.12
1.03
1: 0.20
6.24
0.40
1.25
1 0.12
3.95
0.18
6.32 2:
DINOv2 (large)
77.83
4.33
0.43
1.16 0.33
6.50
DINO<2 (giant)
1136. 18M
291.13
1.18 0.20
4.49
1: 0.06
7.37 0.20
125
1 0.10
4.82
0.10
7.39 0.23
OpenPlacnomn-3 S/16
178.05M
83:42
0.94 1: 0.04
3,73
- 0.13
6.22
2
0.20
1.77
0.14
4.13 = 0.19
6.25 1: 0.08
Table S.2: Retrieval performance of CelICLIP trained across different pooling strategies on perturb- to-profile and profilett--perture tasks. Results are reported as Recall @ k (%) for k = 1,5, 10.
Pooling Strategy
Perturb-to-profile (%)
Profileto-peeruub
(%)
R@1
R@s
R 10
R@1
R@5
R 10
Attention based
1.18 0.20
4.49
0.06
7.37 1: 0.20
1.25 1 0.10
4.82 1 0.10
7.39 0.23
Median
1.8 0,20
4,03
0.39
6.26 +: 0.30
1.0 1 0.13
3,85 + 0.30
6.00
0.13
Mean
1.18 + 0.12
4,45 0.23
6.83 + 0.20
1.35 + 0.41
0.2
021
6.69
0.28
K-means
1.15 0.12
4,03 0.19
6.17 + 0.15
1.04 + 0.07
3,89
0.26
6,06 + 0.12
Hierarchical clustering
1.10
0.09
1.12
4.17
0.15
Appendix
A Additional Results
A.1 Effects of Different Image Profile Encoding
CelICLIP provides a flexible framework for integrating off-the-shelf pretrained image foundation models into Cell Painting analyses (Section 3.2). To understand the impact of different vision encoding backbones on CelICLIP's performance, we conducted an ablation study where we varied CelICLIP's image encoder while holding all other aspects of our framework constant. Specifically, for this experiment we considered DINOvl along with DINOv2 models of varying sizes. Aligning with results for natural images, we found that increases in model size broadly led to increased performance on our retrieval tasks (Table
To understand the impact of using image encoder backbones originally trained natural images versus those trained directly on Cell Painting data, we also applied OpenPhenom-S/16 an openly available masked autoencoder model pretrained on Cell Painting data (27). Interestingly, we found that using OpenPhenom-S/16 did not result in superior performance compared to DINO models trained on natural images. This suggests that, despite not being originally trained on microscopy data, foundation models trained on diverse natural image distributions combined with small tweaks account for differences in channels as in CeIICLIP can achieve competitive performance Cell Painting data.
A.2 Effect of Various Pooling Strategy
Table S.2 summarizes the retrieval performance across different pooling strategies as described in Section 3.1 Overall, aitention-bbasd pooling yields the best performance, particularly on the Recall @ 10. Among non- con-attention methods, mean pooling consistently outperforms median, K-means, and hierarchical clustering, suggesting its effectiveness as simple yet strong baseline.
A.3 Impact of top-n instances
To mitigate the influence of false positives, we applied per- per-perturbation pooling of cellular profiles using the ABMIL pooling operator (24), which adaptively attends to samples most representative of a perturbation. We further investigated the importance of accounting for false positives by removing the most highly attended samples (i.e., those most indicative of effective perturbation). cross-modal retrieval tasks. This ablation led to substantial performance degradation (see (S.) suggesting that for each perturbation, only relatively small fraction of cells exhibit strong perturbation effects.
22

---

Table S.3: Retrieval performance after removing the top-n contributing instances. Results are reported
as Recall h (%) with h 1,5, 10.
Removal of top-n instances
Perturb-to-profile
(%)
Profile-oo-peturb (%)
R@1 1
R@5
R@ 10
R@! 1
R@5
R@ 10
0
1.32
4.59
7.51
1.18
4.82
7.56
1
1.08
4.60
7.13
1.04
4.58
7.61
2
1.13
4.25
6.38
0.99
4.34
6.47
1.13
4,20
5.91
0.89
4.06
6.09
0.04
5,01
0.09
5.34
5
0.00
1.46
3.68
0.00
2.88
4.34
Table S.4: Retrieval performance of CeIICLIP trained across different loss functions on perturb-to-
profile and profile-to-perturb tasks. Results reported Recall@k (%) for k 1, 10.
Loss type
Perturb-to-proolle (%)
Profile-to-perturb (%)
R@1
R@5
R@10
R 1
R@5
R@
CLIP 35
1.26
0.10
4.24
0.06
7.01 4: 0.41
1.29 :: 0.19
4.15
0.31
6.55 4: 0.29
CLOOB 17
0.87
0.07
3.32
+ 0.36
5.39
+: 0.45
0.91 + 0.14
3.52
0.26
5,51 + 0.43
SigCLIP
1.10
0.08
3.54
6.03
0.11
3.79
0.09
6.22
0.1.1
821. [16
1.04
0.19
3,93
+ 0.45
6.83
+ 0.50
1.08
+ 0.10
4.25
+ 0.14
6.65
+ 0.21
CWCL 41
1.18
0.20
4.49
0.06
7.37
0.20
1.25
4: 0.10
4.82
0.10
7.39
0.23
Table S.5: Comparison of CelICLIP with different contrastive losses for zero-shot biological relation- ship recovery (RxRx3-core). Results report average recall across cosine similarity thresholds (5th and 95th percentiles).
Loss type
Gene-Gene Relationship Recovery (Recall)
CORUM
HuMAP
Reactome
SIGNOR
StringDB
CLIP
.673
.751
.416
.370
.595
CLOOB
.696
.772
.418
.373
.598
SigCLIP
.741
.800
.466
.444
.646
S2L
.691
.765
.416
.35
.603
CWCL.
.714
.778
.427
.388
.618
A.4 Effect of Different Contrastive Loss
Table summarizes cross- modal retrieval performance across different contrastive learning losses. CWCL consistently achieves the highest Recall @5 and Recall @ 10 in both perturbationnto-preee -o-profile and profile-to-periiee no-perturbation tasks, while remaining competitive at Recall CLIP ranks second overall, and CLOOB shows the weakest performance. Although both CWCL. and S21. use similarity- based weighting, CWCL. outperforms S2I. across all retrieval metrics. This suggests that computing similarity from embeddings extracted from pretrained imaging foundation models is more effective for alignment than using projected embeddings in the shared space, as defined in Fradkin et al. [16].
Table S.5 presents intra-modal gene-gene recovery performance. Sigmoid loss (SigCLIP) achieves the highest recall across all benchmarks, followed by CWCL. This indicates that contrastive objectives with relaxed pairwise penalties may better preserve biological structure. Overall, CWCL demonstrates strong performance across both settings, highlighting ability balance cross-min modal alignment with intra modal consistency.
23

---

Table S.6: Comparison of models for zero-shot biological relationship recovery (RxRx3-core).
Results report average recall across cosine similarity thresholds (5th and 95th percentiles).
Method
Gene-Gene Relationship Recovery (Recall)
CORUM
HuMAP
Reactome
SIGNOR
StringDB
Cross-itoaal contrastive learning
CLOOME
.327
-100
Malphenix
.399
.599
.330
.297
.476
CeIICLIP (DINOv1)
.720
.774
.45
.378
.614
CelICLIP (DINOv2 Small)
.427
CeIICLIP (DINOv2-B)) Base)
.706
.766
.46
.006
616
CelICLIP (DiNOv2-Large)
.725
.780
.400
.406
.632
CelICLIP DDNOV2--Giat
.714
.778
.427
.388
.618
HCS-pretrained Channel-agnostic MAE
OpenPhenoon- S/16
649
.723
418
.386
.599
ImageNet- pretttanddd classifiers weakly supervised models
VIT-L/16
.811
.788
.388
.380
587
A.5 Zero-shot Biological Relationship Recovery RxRx3-core
We evaluate recall over the most extreme 2% to 20% of the similarity distribution on RxRx3-core (Figure S.1). including pathway annotations from CORUM, HuMÁP, Reactome, SIGNOR, and STRING databases. As shown in the figure below, CelICLIP consistently achieves the highest recall across thresholds, outperforming existing multimodal contrastive methods such as CLOOME and MolPhenix. Among II variants, performance is comparable across different image tokenizers, indicating robustness to the choice of visual backbone (Table S.6).
0.0
Meppheeic
Obe Specrreeooo
0.0
DaRCL P eev2eeeer
0.4
0.32.99
8,108000
8.00..99
0.88..00
ReaDome
Sty Strigg
0.7
0.6
0.5
0.3
0.2
01
6014.00
8.06
eakedd
0.00..00
6520.0
6094 59
02040 up
802...9
6oz
aos
eake oe
Figure S.1: Zero-shot gene-gene relationship recovery on RxRx3-core, evaluated across varying thresholds from Recall [0.01,0.99] (top and bottom 1 1)) Recall @ 20% [0.10,0.90] (top and bottom 10%), using pathway annotations from CORUM, HuMAP, Reactome, SIGNOR, and STRING.
A.6 Effect of Text Encoder Parameters
In Table S.7 we present additional experiments using smaller BERT variants (64M and 11.3M parameters) from Ture et al. [44 as the text encoder for CelICLIP on the cross-modal retrieval task 4. For reference, the MPNN+ graph neural network encoder in our MolPhenix baseline contains approximately 10M parameters. While performance decreases slightly with smaller BERT models, even the -arammeer variant substantially outperforms the MPNN+ bascline. These results
24

---

Table S.7: Retrieval performance with different BERT parameter sizes. Results are reported as Recall (%) with k 1, 5, 10.
# BERT params
Perturb-to-proflle (%)
Profile-to-perturb (%)
R@1
R@5
R@
R@1
R@5
110M (ours)
1.18 0.20
4.49 1 0.06
7.37 1 0.20
1.25 0.10
4.82 1 0.10
7.39 - 0.23
64M
1.53
0.17
4.40
0.21
6.90
1.41 0.17
4.67
0.24
6.98 + 0.14
11.3M
1.13 0.14
4.04 0.31
6.28 + 0.19
1.28 0.17
4.11 0.24
6.20 = 0.19
Table S.8: Retrieval performance with different perturbation encoders. @ (%) with k = 1,5, 10.
Results reported as
Perturb. Encoder
Perturb-to-profie (%)
Profile-to-pertubb (%)
R@1 1
R@5
R@ 10
R@1
R@5
R@ 10
BiomedBERT
1.46 0.10
4.11
6.19 + 0.26
1.51 + 0.09
4.34 0.13
6.57 0.19
KV-PLM
1.29 = 0.19
4.41 0.31
6.92 = 0.32
1.41 0.16
4.57 0.30
6.81 + 0.21
BERT
1.18 + 0.20
1.49 + 0.06
7.37 + 0.20
1.25 + 0.10
4.82 + 0.10
7.39 + 0.23
indicate that the gains in cross-modal retrieval are not merely due to larger model capacity, but rather arise from the effectiveness of our text- bcse- encoding strategy.
A.7 Do Domain-Adapted BERT Help?
In Table S.8 we compare several pretrained BERT models, including BiomedBERT [49] trained on biomedical PubMed corpora, and KV-PLM 47) BERT-based model pretrained both biomedical texts and SMILES strings. Interestingly, while KV-PLM outperforms BiomedBERT, it still under- performs relative to our original BERT choice, suggesting that domain adaptation alone does not guarantee superior performance for cross-modal retrieval.
25

---

Table S.9: Image foundation models used to generate per-channel embeddings, along with their parameter sizes and total embedding extraction time for Bray et [ measured using a single NVIDIA A40 GPU.
Extraction times were
Model
Parameter Count (M)
Extraction Time (hr)
DINO (VIT-B/16)
86.4
5.21
DINOv2 (ViT-S/14)
22.06
1.83
DINOv2 (VIT-B/14)
86.58
2.78
DINOv2 (ViT-L/14)
304.37
5.22
DINOv2 (ViT-G/14)
1136.48
CA-MAE (OpenPhenommS/))
178.05
2.89
B CelICLIP Training & Implementation Details
Feature extractors for Cell Painting profiles To generate channel profiles (Section 3.1p, we experimented with range of image foundation models, including DINO, DINOv2 (small, base, large, giant), and CA-MAE (OpenPhenom-S/10 (Table S.9). All model checkpoints were obtained from Hugging Face Cell Painting images were treated grayscale inputs, preprocessed according the requirements of each model, and augmented using a multi-crop strategy to enhance robustness.
Perturbation encoding For generating text prompts, we adopted the following template:
"A fcell rype) treated with (perturbation).
with /detailed perturbation information, such as SMILES or target genes)".
We also experimented with alternative templates, such as:
"A cell painting image of (cell rype] treated with (perturbation), with (detailed perturbation information, such SMILES target genes)",
well other variants, but observed no significant difference in cross-modal retrieval performance.
Attention-based multi-instance learning (ABMIL). To accommodate the varying number of Cell Painting images per perturbation, we introduce muuliiinstaane learning (MIL) pooling mechanism [24] in CrossChanelformner Instead of relying on single instance, CrossChannel processes multiple input profiles associated with the perturbation and aggregates them into single representation.
Formally, given perturbation and associated set of image profiles NN we define the pooled profile u(U:) follows:
(U,)) = (u(U,)) ((U)) (U.)))) Cxdd
where y denotes the aggregated representation for channel e E (1,...,C).
Each channel-wise pooled representation is obtained via permutation-invariic aggregation function S(-):
m(U.)² = ({=f N.)) e Rd.
The aggregation function S(-) can be instantiated any permutation-invariaa transformation function applied channel-wise, such as maximum or mean operator. To better capture variability among instances, we adopt gated attention pooling 244] defined
N
((U))
of E Rdd where
exp (wT (tanh( 2o
of
N; k/- exp (wT (lanh (V=g) (U(
26

---

Here, w E R' and U, V E Ri are learnable parameters, @ denotes element- multiplication. and (-) is the sigmoid activation function. While we adopt attention- based pooling in this work to model instance-level variability, the aggregation function S(.) is flexible and can be instantiated as simpler alternatives, such as mean pooling [[6].
CelICLIP backbone Our CrossChamnlfoommer backbone consists of transformer model with 12 layers, 8 attention heads, and 5 512-dimensional embedding space, For the text encoder, we employ the pre-trained BERT 12 It supports text lengths of up to 512 and uses WordPiece [29] as its tokenizer.
Training details For retrieval evaluation on Bray et al. [ CelICLIP was trained for 50 epochs with a batch size 768 using the AdamW optimizer. The learning rate was set to 2 x 10 with cosine annealing and restarts. The temperature parameter was initialized as 14.3.
For RxRx3-core, we performed zero-shot evaluation by reusing the models trained on Bray et al. to assess their ability to recover known biological relationships.
For CP-JUMPI, reuse the model trained with Bray al. 4 and further fine-tune with CP-JUMPI.
Fine-tuning was performed for an additional 50 epochs using the same hyperparameter settings as in pretraining.
27

---

Experiment & Evaluation Metrics Details
C.1 Cross-Modal Retrieval
We evaluate cross-modality retrieval using Recall@k, standard metric that measures the proportion of queries whose correct match appears the top &o retrieved items. Given batch of N paired samples {(u, e))] where u, E u and V are Cell Painting profiles and perturbation and trained encoder fø U + P and J6 V Q, we compute the similarity matrix S E RNXN where
(pi Q+)
Sty
(10)
Ipd |
Recall@ K in the u + V direction (e.g., profile-to to molleeclles) is defined as:
N
Recall @ ku-
1 [ranky(q: pp;) k),
(1I)
where ranky(p; (4) is the rank of the correct match qe when all qj are sorted by similarity ps, and [[:] is the indicator function. The reverse direction y - U is defined analogously. In this work, we report Recall@k with & E {1,5, 10).
C.2 Biological Evaluation of Learned Representations via Intra-Modal Retrieval
Here provide details for infra-modality retrieval for Cell Painting analysis p tasks include (1) replicate detection, (2) sister perturbation matching and (3) zero-shot recovery of known biological relationships.
10 27
These
C.2.1 Replicate Detection (CP-JUMPI [00)
For replicate detection, we follow the protocol of Chandrasekaran al. [T0] and Kalinin et al. 225 In particular, during model training and evaluation we pool images across both perturbations and experimental batches (corresponding e-g. different well positions). That is, for each combination of perturbation i and batch 8, we obtain an embedding pf- Each p is then corrected for batch effects using matched negative controls from the experimental batch produce corrected embedding p.
Given these aatch-effect--ooreect pooled embeddings, compute cosine similarities between query 3 and candidate embeddings b corresponding replicates of the same perturbation i or negative controls, and then rank candidates descending order. Following Chandrasekaran et al. report average precision (AP) evaluation metric for this task:
110.
AP
[R P&
(12)
where Ps and R3 denote the precision and recall at rank k, respectively. To assess statistical significance, we perform permutation testing by shuffling rankings 100,000 times to construct a null distribution. We then apply multiple comparison corrections (3] and filter non-significant AP values. For each perturbation replicate in different batches, we compute AP scores, which are then averaged to obtain mean average precision (mAP) score representing the perturbation's phenotypic activity. Finally, we use mAP across classes, defined by specific perturbations or gene associations, evaluate the performance in both tasks.
Our evaluation pipeline wes implemented following the official benchmark protoco
C.2.2 Sister Perturbation Matching (CP-JUMPI 10)
For sister perturbation matching, we again follow the protocols of Chandrasekaran et al. 10 and Kalinin al. 25 However, after computing batch-effecct-ooreeteed pooled embeddings of as
Benchmark pipeline for CP-JUMP1
28

---

described above, we now perform a second aggregation step across experimental conditions to produce single Pe for each perturbation across batches. That is, we compute
A(
(13)
where A is some permutation-invariaa aggregation function. For these experiments we used the mean operation to aggregate across batches.
Using these aggregated results, we then similarly we compute cosine similarities between a query p. and candidate embeddings Pj. and rank candidates in descending order. As described above we again use mean average precision across classes as our evaluation metric for this task.
C.2.3 Zero-shot Recovery of Known Biological Relationship (RxRx3-core
28
This task evaluates whether the learned embeddings capture biologically meaningful structure by assessing their ability to recover known functional relationships among perturbations, such as genes within the same pathway or protein complex. Pathway annotations are obtained from HuMAP, CORUM, StringDB, Reactome, and SIGNOR (Table
For each perturbation we compute aggregated embedding P; described in Appendix C.2.2 2.2 but using the median for A as in Kraus et al. [27]. For each perturbation i we then evaluate the cosine similarity of i4s with emheddings Pj from other perturbations. All perturbation pairs then ranked by similarity to assess the recovery of known biological relationships.
Recall is then evaluated over the most extreme 10% of the similarity distribution, including the top 5% (most similar pairs) and bottom 5% (most dissimilar pairs), and is defined as:
# of known positives recovered in top & bottom 5%
Recall @ 10%
(14)
Total # of known positives
A random embedding space yields a baseline recall of 10%. Higher recall indicates stronger alignment with curated biological interaction networks. We also report recall 2% ([0.01, 0.99]) to recall @ 20% ([O.10, 0.90)) in Appendix A.5 Evaluation was performed following the official RxRx3-core benchmark protocol
C.3 Batch Effect Correction
For the batch correction operations mentioned above, we followed the procedure of Celik et al. 9 implemented in 5 for RxRx3-core and CP-JUMP1. PCA kerne was fit on all control profiles across experimental batches and used to transform all embeddings. For each batch, a separate StandardScaler was fit on the transformed control embeddings and applied normalize all embeddings. We experimented with RBF, polynomial, and linear kernels and selected the best- performing kernel for each method.
FFAAR Benchmark
Scikit-learn learn KemelICA
29

---

D Datasets & Preprocessing
Bray al. The datase L consists of 919,265 five-channel microscopy images with resolution 520 x 696 corresponding to 30,616 different molecules. These images were captured using 406 multi-well plates, with each image representing a view from sample within one well, Six adjacent views collectively form one sample.
Sanchez-Fermandez et al. 38 refined the dataset by removing images that were of focus, exhibited high fluorescence, or contained untreated control cells. The final dataset comprises 28.4k images linked 10.8k unique molecules, split into training, validation, and test sets. The processed dataset is publicly available a Next, for retrieval evaluation on unseen compounds (Section 4.1). following al. [38). removed samples from the test set that corresponded to the same molecule and plate to mitigate plate effects. The remaining samples are referred to as the final "test set", which consists of 2,1 115 unique compounds.
CP-JUMPI CP-JUMPI [10] comprises approximately 340k eight-channel (3 bright field chan- nels) microscopy images of resolution 1080 1080. We use images from the batch named 2020 1110_11__1111 11_01111110000 10_0040_PPP SL JUMP1, which contains 186,925 images. This dataset features a comprehensive collection of perturbations conducted on U2OS and A549 cell lines, including 52 replicates. For perturbations, it includes 301 small-molecule compounds (46 controls), 335 sgRNAs (CRISPR) targeting 175 genes (88 control sgRNAs), and 175 ORFs (45 controls) for the corresponding genes. [0] also provides annotations of associations between genes and compounds, enhancing its utility for exploring gene function and compound effects. We split the dataset into 70/10/20 for training, validation, and testing. Raw data, relevant metadata, and gene annotation can be found ir
a curated subset of the RxRx3 dataset 15). It includes 222,601 six-channel fluorescent microscopy images of HUVEC cells stained using modified Cell Painting protocol, covering 736 gene knockouts (non-blinded) and 1,674 compounds across eight concentra- tions, along with control wells. Gene-gene relationships are benchmarked using Reactome, HuMAP, SIGNOR, StringDB, and the CORUM collection, Table 10 Additional details regarding RxRx3- core can be found in Kraus et al. (28).
RxRx3-core RxRx3-core
Table S.10: Description datasets used for curating gene-gene interactions
Datasets
Description
Reactome
Protein-protein interactions based on curated pathways.
StringDB
43
Proteins associations based on functional interactions.
HuMAP
Gene clusters representing protein complexes.
CORUM
19
Gene clusters corresponding to protein complexes. Protein-protein interactions in signaling pathways.
SIGNOR
34
Image Preprocessing For both datasets, our preprocessing followed the protocols established by et al. [38] and Hofmarcher et al. [22], which consist of converting the original
TIF images from 16-bit to 8-bit and removing the 0.0028 % of pixels with the highest values To maintain channel consistency between datasets, we processed the CP-JUMP1 images and reordered channels to match the five-channel format of
GigaDB doi.org/ wwwpg/00552//1035511111
Curated dataset Bray et al. 4
Github repository of Chandrasekaran al.
10
RxRx3-core dataset
TIF files preprocessing
30

---

Text Prompt:
HUVEC cells treated with compound: Epothilone SMILES:
Text Prompt: HUVEC cells treated with compound: control
Prompt:
HUVEC cells treated with crispr, targeting genes RPL27A
Figure S.2: Example Cell Painting images and corresponding prompts in RxRx3-core.
31

---

E Baselines Training & Impllementatinn details
E.1 Baselines
CLOOME We follow the official CLOOME implementation 12 For retrieval evaluation on the dataset from Bray al. [4, adopt the best-performing hyperparameters reported in Sanchez- Fernandez et al. [38). using a ResNet-50 as the vision encoder and a four-layer MLP as the molecule encoder, excluding the Hopfield layer. The model is trained using the original contrastive loss, with raw Cell Painting images paired with max-pooled combination of Morgan and RDKit count-based fingerprint resulting an 8192-bit imput representation. We refer the variant that incorporates the Hopfield layer for CLOOB loss computation, as described in Sanchez-Fernandee et al. (38), as CLOOME (CLOOB).
The training setup includes a batch size of 256, the AdamW optimizer, and learning rate of x 10-3 with cosine annealing and restarts. The learnable temperature parameter is initialized to 14.3. The model is trained for 70 epochs. We use the same training configuration for all CLOOME variants. For CP-JUMPI, since CLOOME is originally designed for small molecules, we fine-tune the model using only the small-molecule subset of the CP-JUMPI training set, with the same hyperparameters as used for training on Bray et al.
Following Fradkin et al. [6]. we refer to the variant of CLOOME that uses phenomic Sonflls--seccficallyyy the peoned embeddings for each perturbation- input, as CLOOME For the profile modality, this variant replaces the vision encoder with fixed phenotypic embeddings while retaining CLOOME's original molecule encoder for chemical compounds.
CLOOME
MolPhenix * As the official codebase for MolPhenix is not publicly available, we re-implement the model based on the descriptions provided in Fradkin et al. [16]. The original feature extractors for phenomic (profile) and molecular proprietary datasets lack open-source implementations. Therefore, substitute these components with publicly available alternatives.
(7 and MolGPS [42] -are either trained on
For phenomic features, we implement a 16-layer MPNN+ model trained on the PCQM4M G25_N4 N4 dataset 23 for 40 epochs, which has demonstrated comparable performance to MolGPS 42 Using these pre-extracted features, we reconstruct MolPhenix according to the architecture reported in the original paper: six ResNet blocks followed by linear layer for the phenomic modality, and one ResNet block followed by a linear layer for the molecular modality. The embedding dimension is set to 512. The embedding dimension is set 512. The model is trained with S2I. loss which applies an arctan distance function to projected embeddings in the shared latent space, using a similarity clipping threshold of 0.75, and hyperparameters 7 1.75 and 5 = 0.75, consistent with the original work. We also experimented with the sigmoid loss introduced by Zhai et al. [48) and found it outperforms S2L in our setting. Accordingly, we adopt sigmoid loss as the default objective. Given possible differences from the original implementation, we refer to our version as MolPhenix
OpenPhenom-S/16 the feature extractor. For molecular features,
For CP-JUMP1, MolPhenix fine-tuned using only small molecules from the CP-JUMP1 training set, following the same training parameters as for Bray et al. [4.
Weakly Supervised Learning (WSL) Following Kraus al. (27). we constructed a Vision Trans- former (VIT) Large with a patch size of 16 (ViT-L/I6"4) modified to accommodate five input channels [] A classifier head was attached, and the model was trained for 10 epochs with a learning rate of 1 x 10-3 and weight decay. The batch size was set to 256. We used the output from the penultimate layer as the learned embeddings for evaluation.
Channel-Agnostie Masked Autoencoder (CA-MAE) OpenPhenom-S/16 CA-MAE 27 is a channel-agnosiee image encoder based on ViT-S/16 backbone, specifically designed for microscopy image featurization. It employs a vision transformer with channel-wist cross-attention over patch tokens to generate contextualized representations for each channel independently. For our evaluation.
"ACLOOME: GitHub repository
The official sources for the RDKit library
Py Torch Image Models
32

---

Table S.11: Hyperparameter search range for cross-moda retrieval.
Hyperparameter
Range / values
Learning rate
5.x 10 1 10 . 10 2
10 10
Warmup steps
200, 500, 800, 1500,
Cosine-annealing restart cycles
3,4
Temperature (r)
Table S.12: Hyperparameter search range for CelICLIP ablations.
Hyperparameter
Range / values
Epochs Learning rate
50, 60. 70
x 10
10 3, 5 10
Cosine-anneuling restart cycles
5, 7,
Temperature (r)
14.3, 30
Table S.13: Hyperparameter search range for sister perturbation matching and replicate detection.
Hyperparameter
Range/ / values
Epochs
5. 10, 20, 30, 40, 50, 60. 70 128, 256, 512
Batch size
Learning rate
5 x 10 x 10 ? x 10 2
10
x
10
4,
5
10
1x
10
10
Warmup steps
500, 800, 1000
we use the only publicly available version OpenPhenom--/16 Since OpenPhenom--Se is already pretrained CP-JUMP1 and RxRx3, use the pretrained encoder to extract image embeddings without performing any additional fine-tuning.
E.2 Hyperparameter Search
For CelCCLIP, CLOOME (Phenomic), and MolPhenix* results in Table we perform hyperparameter search over Table S.IT For batch size, we use the largest possible setting on RTX 6000 GPUs: CeIICLIP, MolPhenix and CLOOME (Phenomic) are trained with batch size of 512, 768, and 768, while CLOOME is trained with batch size of 256.
For the CelICLIP ablations reported in Table 2 we sweep the ranges in Table S.12 For the sis- ter perturbation matching and replicate detection experiments (Table we sweep the ranges in [S.1e For both of the unimodal CA-MAE and OpenPhenom-S// models specifically, these models' training data already included CP-JUMP1, we did not perform any additional training steps with these models
E.3 Software & Hardware Details
This study employs the Py PoTch package tutorial (version 2.2.1). All experiments are conducted on systems equipped with 64 CPU cores and the specified NVIDIA GPUs. Models were trained with the largest possible batch size on RTX 6000 GPUs.
33


Reviews:
[{"review_id": "wD6WULHMG1", "reviewer": "Reviewer_8uCR", "summary": "This paper introduces a new framework to predict the perturbation effects assessed by high-content screening. The performance is assessed through the application of their method, CellCLIP, to several data sets.", "questions": "Can the authors evaluate the stability of their perturbation embeddings? Also, is there a way to evaluate the performance across studies?", "limitations": "The limitations were discussed.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: There is a clear need to develop effective models that predict the effects of various types of perturbations using data accumulated in the literature. The authors have proposed several approaches to address the limitations of existing methods, including the one-to-many mapping of images to perturbations and the need to integrate different modes of perturbations. The performance suggests the advantages of their strategy.\n\nWeaknesses: The embedding for perturbation based on prompt engineering may introduce uncertainties and variations across users/runs. It is also challenging to assess the generalizability of the proposed methods across different datasets. Therefore, we have several questions for the authors.\n\n1. To ensure that the authors optimize choices of prompts, we believe that the other should consider different methods of text description. For example, we may use gene/molecular information from NCBI and Drugbank as descriptions, and text encoders from OpenAI or Gemini as perturbation encoders. These choices should be considered in the benchmarking analysis. \n\n2. The authors support design of their proposed multi-cell encoder. How to select the best number of images used in each encoder, and could we just use the mean avenge are the simple aggregator? \n\n3. The authors should include hyper parameter information and justify the comparison is fair and other methods are also optimized in the benchmarking set.\n\n4. Since we have large-scale cell painting data, is it possible to pre-train an image encoder rather than using the DINOv2 for normal image set? Will it be helpful for having a better performance?", "was_revised": true, "final_justification": "Keep my score", "rebuttal": "We thank the reviewer for carefully examining our work and providing their feedback. We’re happy to hear that the reviewer recognizes the “clear need to develop effective models” in this area, and we’re thrilled that the reviewer found that our results “suggests the advantages of [our] strategy”. Below, we respond to the reviewer’s concerns. Please let us know if we can further clarify these points.\n\n### **Re: Optimizing the choice of prompt**\n\nWe thank the reviewer for raising this important point. While we acknowledge that CellCLIP’s performance may vary with prompt format, it is designed as a general, flexible framework that allows users to represent perturbations using diverse metadata tailored to their needs; we note that the prompt template used in our paper is just one example. Although the current prompting strategy consistently achieves state-of-the-art performance across all evaluated datasets, we agree that prompt engineering could offer room for further improvement. In response to the reviewer’s suggestion, we explored alternative prompting strategies. These provided insights into component-level contributions, but none of them outperformed the original. Results are detailed below:\n\nFirst, we performed an ablation study where we re-trained and re-evaluated CellCLIP after removing different components of our original prompt template (e.g. cell type, drug name, SMILES string, etc.) on the cross-modal retrieval task for the Bray et al. dataset. We found that removing the SMILES string led to a large degradation in performance, and that removing cell type information led to a smaller degradation. On the other hand, removing drug names did not negatively affect the model’s performance, likely due to relevant information already being captured in the richer SMILES representation.\n\nSecond, following the reviewer’s advice, we experimented with incorporating descriptions from an external source into the prompt template for CellCLIP. Specifically, we added DrugBank descriptions for any compounds that could be found in the database (correspondsing to approximately 10% of the compounds in the Bray et al. (2017) dataset). We note that the match rate was low, as many compounds in this dataset are small experimental molecules. After incorporating available DrugBank descriptions into the prompts, we found no improvement in performance compared to our original template. While it is possible that retrieval performance could benefit from more comprehensive metadata, this was not feasible given the limited annotation coverage in DrugBank. Nonetheless, these results suggest that our original prompting strategy is already highly effective.\n\n| Ablation  | Chem-to-Image (%) R@1 |  R@5 |R@10 | Image-to-Chem (%) R@1 | R@5 |  R@10 |\n|-|-|-|-|-|-|-|\n| Original | 1.18 (0.20) | 4.49 (0.06) | 7.37 (0.20) | 1.25 (0.10) | 4.82 (0.10) | 7.39 (0.23) |\n| No cell info | 1.11 (0.13) | 4.02 (0.25) | 6.59 (0.33) | 1.10 (0.23) | 4.07 (0.29) | 6.36 (0.32) |\n| No drug name | 1.47 (0.29) | 4.77 (0.20) | 7.37 (0.29) | 1.61 (0.26) | 4.89 (0.19) | 7.21 (0.27) |\n| No SMILES | 0.13 (0.05) | 0.68 (0.19) | 1.26 (0.14) | 0.14 (0.07) | 0.63 (0.17)  | 1.24 (0.13) |\n| With DrugBank description | 1.13 (0.16) | 4.26 (0.43) | 6.94 (0.37) | 1.16 (0.17)  | 4.34 (0.36) | 6.71 (0.26) |\n\n### **Re: Experiments with different text encoders (e.g. from OpenAI) for the perturbation encoder** \n\nWe thank the reviewer for this suggestion. In response, we conducted an additional set of experiments where we used OpenAI’s `text-embeddings-3-small` model with our prompt template to generate perturbation embeddings, followed by a 4-layer MLP as the perturbation encoder for CellCLIP on the Bray et al. cross-modal retrieval tasks. We chose this embedding model specifically because embeddings from OpenAI’s decoder-only models, such as ChatGPT, are not accessible to users.  We found that this resulted in substantially worse performance compared to our original choice of BERT. This suggests that general-purpose embeddings, even from powerful embedding models, are insufficient for domain-specific tasks without fine-tuning of the base text encoder.  \n\n|Perturbation Encoder| Chem-to-Image (%) R@1 |R@5|R@10| Image-to-Chem (%) R@1 |R@5 | R@10 |\n|-|-|-|-|-|-|-|\n| BERT | 1.18 (0.20) | 4.49 (0.06) | 7.37 (0.20) | 1.25 (0.10) | 4.82 (0.10) | 7.39 (0.23) |\n| OpenAI `text-embeddings-3-small` with MLP | 0.32 (0.12) | 1.72 (0.28) | 2.80 (0.19) | 0.09 (0.11) | 0.5 (0.05) | 0.89 (0.11) |\n\n### **Re: How to select the best number of images used in each encoder?**\n\nFor all experiments presented in our manuscript, when pooling images within a perturbation we always used **all** available images for that perturbation. As shown in our submission, we found that this strategy consistently yielded strong performance compared to baseline methods.\n\n### **Re: Could we just use the mean or other simpler aggregators?**\n\nAs part of our initial submission we included results from benchmarking the impact of different choices of aggregation operator on CellCLIP’s performance for the cross-modal retrieval tasks for the Bray et al. dataset. We found (Table S.2) that attention-based pooling led to superior performance compared to simpler pooling operations like the mean or median.\n\n### **Re: Hyperparameter tuning for CellCLIP and baselines.**\n\nThe results reported in our manuscript correspond to the best‑performing models selected through extensive hyperparameter searches for both CellCLIP and baseline models. The details of these sweeps can be found in Appendix E.2, which we reproduce below for the reviewer's convenience.\n\n* Benchmarking results presented in Table 1: Here the best performing CellCLIP and baseline models were selected via a grid search over the following hyperparameter ranges:\n    * Learning rate: {7 × 10⁻⁵, 5 × 10⁻⁵, 1 × 10⁻⁵, 1 × 10⁻⁴, 2 × 10⁻⁴, 3 × 10⁻⁴, 5 × 10⁻⁴, 1 × 10⁻³, 1 × 10⁻²}\n    * Warmup steps: {200, 500, 800, 1000, 1500}\n    * Number of cosine restarts: {1, 2, 3, 4}\n    * Temperature τ: {14.3, 20, 30}\n* CellCLIP ablation results (Table 2): For each row in the table we performed a hyperparameter sweep over:\n    * Learning rate: {0.0005, 0.001, 0.005}\n    * Cosine annealing restart cycles: {5, 6, 7, 8, 9, 10}\n    * Temperature τ: {14.3, 20, 30}\n    * Epochs: {50, 60, 70}\n* Sister perturbation matching & replicate detection results (Table 3): Here we swept over the following hyperparameter ranges for all multimodal models. (For both of the unimodal CA‑MAE and OpenPhenom‑S/16 models specifically, as these models’ training data already included CP-JUMP1, we did not perform any additional training steps with these models):\n    * Learning rate: {1 × 10⁻⁵, 1 × 10⁻⁴, 2 × 10⁻⁴, 3 × 10⁻⁴, 5 × 10⁻⁴, 1 × 10⁻³, 1 × 10⁻²}\n    * Epochs: {5, 10, 20, 30, 40, 50, 60, 70}\n    * Batch size: {128, 256, 512}\n    * Warmup steps: {500, 800, 1000}\n\n* Zero-shot gene-gene relationship recovery results (Table 3): Because this task involved zero‑shot retrieval, for all multimodal models we reused the models pretrained on Bray 2017 with no additional fine tuning. For unimodal models we did not perform any additional training steps.\n\n### **Re: Is it possible to pre-train an image encoder on Cell Painting data rather than use an encoder pretrained on natural images?**\n\nIn our original submission (Appendix  A.1), we also experimented with using a specialized image encoder model pretrained on Cell Painting data as the vision encoder for CellCLIP. In particular, we used OpenPhenom [1], the current state-of-the-art openly available pretrained Cell Painting image encoder. However, we found that using OpenPhenom resulted in consistently worse performance compared to natural image encoders. Indeed, OpenPhenom (178.05 M parameters) was even outperformed by much smaller natural image encoders (e.g. DINOv2 small with 22.06 M parameters). This result can be found in Table S.1 in our submission, which we reproduce below for the reviewer’s convenience.\n\nWe believe that this result may be due to the relatively small amount of publicly available Cell Painting data available for training compared to natural images. For example, OpenPhenom was trained on ~3 M Cell Painting images collected from current publicly available datasets. This is far smaller than the typical size of datasets used to train natural image foundation models (e.g. ~142 M images for DINOV2 [2]), which likely reduces the potential benefits of pretraining/transfer learning.\n\n| Image Encoding Backbone | # of Params | Perturb-to-profile (%) R@1 |  R@5 |  R@10 | Profile-to-perturb (%) R@1 | R@5 |  R@10 |\n|-|-|-|-|-|-|-|-|\n| DINOV1 | 86.4 M | 0.75 ± 0.08 | 2.77 ± 0.13 | 4.74 ± 0.33 | 0.86 ± 0.16 | 2.77 ± 0.27 | 4.80 ± 0.19 |\n| DINOV2 (small) | 22.06 M | 1.19 ± 0.32 | 4.06 ± 0.27 | 6.57 ± 0.33 | 1.32 ± 0.17 | 4.34 ± 0.39 | 6.63 ± 0.19 |\n| DINOV2 (base) | 86.58 M | 1.19 ± 0.12 | 4.03 ± 0.20 | 6.24 ± 0.40 | 1.25 ± 0.12 | 3.95 ± 0.18 | 6.32 ± 0.38 |\n| DINOV2 (large) | 304.37 M | 0.97 ± 0.12 | 4.33 ± 0.43 | 6.76 ± 0.54 | 1.16 ± 0.33 | 4.23 ± 0.40 | 6.50 ± 0.47 |\n| DINOV2 (giant) | 1136.48 M   | 1.18 ± 0.20 | 4.49 ± 0.06 | 7.37 ± 0.20  | 1.25 ± 0.10 | 4.82 ± 0.10 | 7.39 ± 0.23 |\n| OpenPhenom-S/16 | 178.05 M | 0.94 ± 0.04 | 3.73 ± 0.13 | 6.22 ± 0.20 | 1.27 ± 0.14 | 4.13 ± 0.19 | 6.25 ± 0.08  |\n\n### **Re: Can the authors evaluate the stability of their perturbation embeddings? Also, is there a way to evaluate the performance across studies?**\n\nTo assess the stability of our method, all results in our experiments corresponded to trials with three random seeds. Moreover, to ensure the generality of our method our experiments contained results from multiple datasets (Bray et al., CPJUMP1, and RxRx3 Core). **Please let us know if you would like further information or if we have misunderstood your question.**\n\n**References:**\n\n1: https://huggingface.co/recursionpharma/OpenPhenom \n\n2: DINOv2: Learning Robust Visual Features without Supervision, https://arxiv.org/pdf/1908.08962"}, {"review_id": "DytDXOmvJO", "reviewer": "Reviewer_CsPQ", "summary": "This paper introduces CellCLIP, a cross-modal contrastive learning framework for HCS data. CellCLIP leverages pre-trained image encoders coupled with a novel channel encoding scheme to better capture relationships between different microscopy channels in image embeddings, along with natural language encoders for representing perturbations. Besides the good performance, this approach does not require to pre-train a model which comes with significant advantages over many similar approaches (e.g. Kraus et. al 24). Indeed, pre-training is data and compute intense. \n\nI think that the paper is relevant and very well executed. I recommend an acceptance. \n\nDisclaimer: Due to the very large amount of papers I got assigned to review I am forced to keep my reviews high-level and could not dive into technical details.", "questions": "None", "limitations": "None", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths \n- Careful system design with well-motivated architectural choices (e.g., pooling, channel reasoning, text encodings).\n- Comprehensive experimental validation across retrieval and biological interpretability tasks.\n- Strong quantitative improvements over baselines. This is particularly remarkable as some of the baselines have been produced using very large computational resources. \n- Efficient implementation with substantial speed-ups over previous work.\n\nWeaknesses:\nNone", "was_revised": true, "final_justification": "I keep my score and recommend the acceptance of the paper.", "rebuttal": "Dear reviewer CsPQ,\n\nThank you for the kind words and for voting for acceptance. We’re thrilled to hear that you believe our approach “comes with significant advantages over many similar approaches”, and that our paper “is relevant and very well executed”. \n\nIf there is any other information we can provide/experiments we could run during the discussion period that would allow you to further raise your score, please let us know.\n\nThanks again,\n\n**Due to space constraints, we have dedicated the remainder of the space to responding to Reviewer Hzy8's questions/minor concerns.**\n\n---\n\n## **Questions/minor concerns from Reviewer Hzy8:** \n \n### **Re: How do the different components of the prompt affect model performance?**\n\nWe thank the reviewer for raising this question. In response, we performed an ablation study where we retrained and re-evaluated CellCLIP after removing different components of our prompt (e.g. cell type, drug name, SMILES string, etc.) on the cross-modal retrieval task for the Bray et al. dataset. We found that removing the SMILES string led to a large degradation in performance, and that removing cell type information led to a smaller, but consistent, degradation in performance. On the other hand, removing drug name did not negatively affect the model’s performance, likely due to any relevant information already being captured in the richer SMILES representation.\n\n| Ablation | Chem-to-Image (%) R@1 | Chem-to-Image (%) R@5 | Chem-to-Image (%) R@10 | Image-to-Chem (%) R@1 | Image-to-Chem (%) R@5 | Image-to-Chem (%) R@10 |\n|-|-|-|-|-|-|-|\n| Original | 1.18 (0.20) | 4.49 (0.06) | 7.37 (0.20) | 1.25 (0.10) | 4.82 (0.10) | 7.39 (0.23) |\n| No cell info | 1.11 (0.13) | 4.02 (0.25) | 6.59 (0.33) | 1.10 (0.23) | 4.07 (0.29) | 6.36 (0.32) |\n| No drug name  | 1.47 (0.29) | 4.77 (0.20) | 7.37 (0.29) | 1.61 (0.26) | 4.89 (0.19) | 7.21 (0.27) |\n| No SMILES | 0.13 (0.05) | 0.68 (0.19) | 1.26 (0.14) | 0.14 (0.07) | 0.63 (0.17) | 1.24 (0.13) |\n\n### **Re: Why did we specifically use BERT for text encoding, and why not use a domain-specific model (e.g. BioMedBERT)?**\nWe decided to use BERT (e.g. as opposed to the encoder from the original CLIP model) because we required an encoder with sufficient **input context length** to enable sufficiently rich and detailed textual inputs. For example, the text encoder from the original CLIP model is limited to a 77‑token context window. This limited window is insufficient for representing SMILES strings (the average length of a SMILES string is 98 characters in the Bray et al. dataset) together with cell information and other metadata. On the other hand, BERT supports up to 512 tokens. Moreover, several works [1, 2, 3] have shown promising performance with BERT-based text encoder for vision-language alignment tasks. \nIn response to the reviewer’s second question regarding the performance of domain-specific models, **we conducted an additional set of experiments using the domain-specific BioMedBERT as our text encoder** for the Bray et al. cross-modality retrieval tasks. We found (see table below) that the performance of BioMedBERT was largely inferior to that of the standard BERT model across various choices of vision encoder. This may be because many of the compounds in Bray et al. are experimental and thus not represented in BioMedBERT's training corpus, which consists of PubMed and PubMed Central articles.   \n| Perturb. Encoder | Chem-to-Image (%) R@1 | Chem-to-Image (%) R@5| Chem-to-Image (%) R@10 | Image-to-Chem (%) R@1 | Image-to-Chem (%) R@5 | Image-to-Chem (%) R@10 |\n|-|-|-|-|-|-|-|\n| BiomedBERT | 1.46 (0.1)| 4.11 (0.13) | 6.19 (0.26) | 1.51 (0.09) | 4.34 (0.13) | 6.57 (0.19) |\n| BERT | 1.18 (0.20) | 4.49 (0.06) | 7.37 (0.20) | 1.25 (0.10) | 4.82 (0.10) | 7.39 (0.23) |\n\n### **Re: Do we need so many layers for CrossChannelFormer?**\n\nThank you for raising this question. In response, we performed an additional set of experiments investigating the impact of the number of layers in CrossChannelFormer on performance for the Bray et al. cross-modal retrieval task. Although the number of tokens (i.e., wavelength channels) in the HCS data is fewer than 10, we found that increasing the number of layers in CrossChannelFormer is necessary for achieving strong model performance.\n\n| # Layers in CrossChannelFormer | Chem-to-Image (%) R@1 | Chem-to-Image (%) R@5| Chem-to-Image (%) R@10 | Image-to-Chem (%) R@1 | Image-to-Chem (%) R@5 | Image-to-Chem (%) R@10 |\n|-|-|-|-|-|-|-|\n| 12-layer | 1.18 (0.20) | 4.49 (0.06) | 7.37 (0.20) | 1.25 (0.10) | 4.82 (0.10) | 7.39 (0.23) |\n| 6-layer | 0.79 (0.09) | 3.00 (0.27) | 5.09 (0.26) | 0.77 (0.19) | 2.94 (0.30) | 4.66 (0.34) |\n| 3-layer | 0.59 (0.08) | 2.27 (0.43) | 3.88 (0.58) | 0.58 (0.21) | 2.24 (0.6) | 3.82 (0.91) |\n\n\n### **Re: Do we see the benefits of transfer learning when pretraining on Bray et al. and fine-tuning on CP-JUMP1?**\n\nWe thank the author for raising this question. In the initial iterations of our CP-JUMP1 experiments we attempted to train CellCLIP directly on CP-JUMP1 (i.e., without a pretraining step). However, we found that doing so resulted in extremely poor performance, with CellCLIP’s mean average precision (mAP) being close to zero. We believe this poor performance was due to the relatively small number of perturbations in this dataset (811 unique perturbations, with about ~500 left for training after performing a train-val-test split).\n\nIn contrast, when we first pretrained the model on the larger Bray et al. dataset and then fine-tuned on CP-JUMP1, we observed substantial improvements in downstream performance across both CellCLIP and all baseline models. **These results show clear benefits of transfer learning in this setting.**\n\n**References:**  \n[1] \"BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation\" https://arxiv.org/abs/2201.12086  \n[2] \"BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models\" https://arxiv.org/abs/2301.12597  \n[3] \"BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs\" https://arxiv.org/abs/2303.00915"}, {"review_id": "rdQmRpCmyg", "reviewer": "Reviewer_Hzy8", "summary": "This paper introduces CellCLIP, a cross-modal contrastive learning framework to capture the connections between Cell Painting image profiles and textual descriptions of cell perturbation effects. CellCLIP develops a CrossChannelFormer architecture to combine representations from multiple channels encompassing independent information. CellCLIP addresses the false negative problem by combining different Cell Painting image profiles with the same perturbation effect with a pooling layer, and introducing a weighted term that penalizes pushing similar profiles too far. Experiment results show that CellCLIP significantly outperforms existing methods in both cross-modal retrieval and intra-modal retrieval tasks.", "questions": "My major concerns are listed in Weaknesses. Here are a few minor questions:\n- How does each component (cell type, perturbation, and detailed perturbation information) of the designed prompt contribute to the overall performance? I speculate that there could be some interesting observations and insights.\n- Why do authors choose BERT for text encoding? Will domain-adapted language models trained on scientific literature and molecular SMILES, like BiomedBERT [1] and KV-PLM [2] bring further benefits?\n- The CrossChannelFormer consists of 12 layers to encode C+1 features, where C is no greater than 10. I'm not sure if a deep model is required to encode such a short sequence.\n- The model is pretrained on [3] for 50 epochs and fine-tuned on CPJUMP1 for another 50 epochs, raising concerns on catastrophic forgetting. Do the authors notice the benefits of transfer learning?\n\nI promise to raise my scores to 4 if my concerns are addressed satisfactorily.\n\nReferences\n\n[1] BioMedBERT: A Pre-trained Biomedical Language Model for QA and IR.\n\n[2] A deep-learning system bridging molecule structure and biomedical text with comprehension comparable to human professionals.\n\n[3] Cell painting, a high-content image-based assay for morphological profiling using multiplexed fluorescent dyes.", "limitations": "Yes.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 2, "originality": 4, "strengths_and_weaknesses": "Strengths\n- The motivation of the paper, i.e., the multi-channel challenge, the false negative problem, and the false positive problem, is clear and closely related to real-world scientific applications.\n- The introduced CrossChannelFormer architecture is intuitive and efficient.\n- The experiment results show significant improvements over prior works. The analyses of the choice of encoding backbones, pooling strategies, and contrastive objectives are quite comprehensive. \n\nWeaknesses\n- I'm not fully convinced that the false negatives and false positives are real existing problems in HCS data. Quantitative and qualitative analyses on the data sources are required to justify the existence of the problem and that the proposed method mitigates the problem. \n- The authors suggest using texts instead of molecular structures to encode cell perturbations to enhance the transferability of the model. While this makes sense, I'm wondering if the improvements on cross-modal retrieval may arise simply from using a larger model, and if the model really generalizes to the sister perturbation matching task.\n- The clarity of the paper needs further improvement:\n  - In Lines 46-61 the authors introduce too many challenges (at least 5), and in lines 62-74 I don't find a clear correspondence of how the proposed approach addresses each of the problems.\n  - In Section 3.1, it took me some effort to understand why pooling profiles address both false negative and false positive problems.\n  - The CWCL objective in Equ. (5) seems problematic: $exp(<p_i\\cdot q_i>/\\tau)\\rightarrow exp(<p_i\\cdot q_j>/\\tau)$. Further explanations on how CWCL resolves false negatives are recommended.", "was_revised": true, "final_justification": "The paper is well-motivated, featuring solid technical designs and thorough experimental evaluations. My primary concerns regarding the clarity and motivation of the work—particularly with respect to the issues of false positives and false negatives—were adequately addressed during the rebuttal phase. Overall, I consider this to be a high-quality submission and recommend it for publication.\n\nThat said, I am not deeply familiar with the task of perturbation effect prediction, and therefore I am uncertain about the broader impact of the paper on the wider research community. This limits my confidence in assigning a higher score. If fractional scores (e.g., 4.5) were permitted, I would have assigned such a score.", "rebuttal": "We thank the reviewer for carefully examining our work and providing their feedback. We’re thrilled to hear that you found the motivation of our work to be “clear and closely related to real-world scientific applications” and that you recognize our results “show significant improvements over prior works”. In the space below we respond to the reviewer’s specific concerns. **Due to space constraints, our response to the reviewer's questions/minor concerns can be found in the response box for CsPQ. Please let us know if we can further clarify any of these points.**  \n\n# **Weaknesses/major concerns:**  \n\n### **Re: How do we know that high-content screening (HCS) data has false positives and false negatives?**\n\nWe thank the reviewer for raising this and for allowing us to clarify these points. First, we emphasize that: **in general, cellular perturbation screens - including high-content screening (HCS) datasets - are well-known to be substantially impacted by false positive and false negative pairs.**\n\n**False positive pairs often arise due to technical limitations with cellular perturbation technologies.** For instance, in genetic perturbation screens, CRISPR guide RNAs are used to perturb specific genes. However, due to off-target guide effects (i.e., guide has the wrong effect) [1] or low guide efficiency (i.e., guide does not have an effect) [2], guide RNAs often fail to induce the intended gene knockdown. Indeed, previous work [3] has estimated that only 5%-10% of guide RNAs have their intended effect >50% of the time. depending on the specific guide. Similarly, in chemical screens, exposure effects on cells may vary depending on cells’ positions on experimental plates, leading some cells to be weakly or not at all affected by the intended perturbation. To mitigate the impact of false positive pairs, we thus performed per-perturbation pooling of cellular profiles using the ABMIL pooling operator, which adaptively attends to the samples most representative of a perturbation.\n\nIn response to your comment, to better understand the impact of accounting for false positives by focusing on the most representative samples within a perturbation, we assessed the impact of removing the most highly attended samples by ABMIL (i.e., those most likely to reflect effective perturbation) for the Bray et al. cross-modality retrieval tasks. We found that doing so led to significant performance degradations (see table below), indicating that, for each perturbation, only a relatively small number of cells may exhibit clear perturbation effects.\n\n| Removal # of top contributing instances | Chem-to-Image (%) R@1 | Chem-to-Image (%) R@5 | Chem-to-Image (%) R@10 |Image-to-Chem (%) R@1 | Image-to-Chem (%) R@5 | Image-to-Chem (%) R@10 |\n| - | - | - | - | - | - | - |\n| 0  | 1.32 | 4.59 | 7.51 | 1.18 | 4.82 | 7.56 |\n| 1  | 1.08 | 4.60 | 7.13 | 1.04 | 4.58 | 7.61 |\n| 2  | 1.13 | 4.25 | 6.38 | 0.99 | 4.34 | 6.47 |\n| 3 | 1.13 | 4.20 | 5.91 | 0.89 | 4.06 | 6.09 |\n| 4 | 0.04 | 2.50 | 5.01 | 0.09 | 3.76 | 5.34 |\n| 5 | 0.0 | 1.46 | 3.68 | 0. | 2.88 | 4.34 |\n\n**False negative pairs arise due to the fact that many cellular images within an HCS dataset share the exact same perturbation label.** For example, in the RxRx3-core dataset considered in our manuscript, each perturbation is associated with ~92 cellular images. Analogous to findings with CLIP-like models applied in other domains [Wang et al., Zhao et al., Srinivasa et al.], this fact can introduce another source of instability during model training, as a model trained with the standard CLIP loss may exhibit pathological behavior when forced to push apart cells with highly similar morphological profiles and identical perturbation labels. Indeed, as demonstrated in Table 2 in the manuscript, accounting for false negatives by training on per-perturbation pooled profiles (as opposed to individual cells) resulted in substantially improved performance. \n\nMoreover, as shown in previous works introducing new HCS datasets [4, 5], groups of related perturbations (e.g. gene knockouts targeting genes in the same higher-level gene pathway), may have near-identical morphological effects. This fact further exacerbates the issue of false negatives, as cells with semantically similar image profiles yet distinct perturbations may be spuriously pushed apart in the embedding space. To mitigate this effect, we employed the CWCL loss [6], which assigns non‐zero weights, $w_{i,j}$ in eq (5), to semantically similar negative pairs, within CellCLIP rather than the standard contrastive loss, which treats all false negatives as unrelated (zero weight). As shown in our gene-gene relationship recovery results in Table S.4, using CWCL resulted in better preservation of known relationships between perturbations (recall = 0.595) compared to the standard contrastive loss (recall = 0.553).\n\n### **Re: Is the superior performance of natural language for perturbation encoding just due to increased model size?**\n\nWe thank the reviewer for raising this question. In response, we conducted an additional set of experiments (see table below) using BERT models with 64M and 11.3M from [7] as the text encoder for CellCLIP for the Bray et al. cross-modal retrieval task. Note that the MPNN+ graph-neural-network based encoder used in our MolPhenix baseline had ~10 M parameters. While we observed a slight decrease in cross-modal retrieval performance compared to the full BERT model, we found that even the 11.3M parameter BERT model substantially outperformed the MPNN+ model. \n\nThese results suggest that the improvements we observe in cross‑modal retrieval are not simply artifacts due to a larger model size, but rather stem from the use of our text-based encoding strategy.\n\n| # BERT params | Chem-to-Image (%) R@1 | Chem-to-Image (%) R@5 | Chem-to-Image (%) R@10 |Image-to-Chem (%) R@1 | Image-to-Chem (%) R@5 | Image-to-Chem (%) R@10 |\n| - | - | - | - | - | - | - |\n| 110M (ours) | 1.18 (0.20) | 4.49 (0.06) | 7.37 (0.20) | 1.25 (0.10) | 4.82 (0.10) | 7.39 (0.23) |\n| 64M | 1.53 (0.17) | 4.40 ( 0.21) | 6.90 (0.18) | 1.41 (0.17) | 4.67 (0.24) | 6.98 (0.14) |\n| 11.3M | 1.13 (0.14) | 4.04 (0.31) | 6.28 (0.19) | 1.28 (0.17) | 4.11 (0.24) | 6.20 (0.19) |\n\n### **Re: Improvements to clarity.**\n\nWe thank the reviewer for flagging these clarity issues. We provide our responses to clarify the specific points raised by the reviewer below. In the camera-ready final manuscript we will use the additional 10th page to clarify these points.\n\n* #### **Re: How do the components of our approach (lines 62-74) address the challenges identified in lines 46-61?**  \n    To clarify, the challenges identified in lines 46-61 are:  \n    \\\n    **a.** Limited availability of image encoding models for Cell Painting microscopy data\n\n    **b.** Microscopy image channels each capturing semantically distinct information\n\n    **c.** False negative pairs due to Cell Painting data being “many-to-one” (i.e., many cells share the same perturbation label)\n\n    **d.** False positive pairs due to variable perturbation efficacy\n\n    **e.** Difficulty encoding multiple classes of perturbations with a single perturbation encoder  \n    \\\n    Our approach addresses these challenges as follows:  \n    \\\n    **1)** CrossChannelFormer allows us to reuse openly available state-of-the-art vision embedding models trained on natural images for Cell Painting data, while accounting for the distinct semantic meaning encoded in different Cell Painting image channels (challenges **a** and **b**).  \n    **2)** Pooling cell image profiles mitigates the impact of both false negative pairs and false positive pairs (challenges **c** and **d**).  \n    **3)** Using a natural-language-based perturbation encoder allows us to encode multiple classes of perturbations with a single model (challenge **e**).  \n    \\\n    As demonstrated by the ablation experiments in our manuscript, **each of these modifications** led to improved performance on downstream tasks.\n\n* #### **Re: Why does pooling address both false negatives and false positives?**  \n    Thank you for pointing out this area of confusion. To clarify:  \n    \\\n    Pooling addresses the issue of false negatives by preventing cells with the same perturbation label from spuriously being considered as negative pairs. If these cells were incorrectly considered as negative pairs, the model may be forced to separate embeddings of cells with highly similar morphological profiles, which could lead to instability during training. By instead operating on pooled profiles, negative pairs will always correspond to distinct perturbations, potentially resulting in better performance.  \n    \\\n    At the same time, pooling addresses the issue of false positives by preventing representations of images assigned the same perturbation label, but which had divergent outcomes (e.g. one cell had a successful perturbation, while the other did not due to CRISPR guide efficiency issues) from being erroneously pushed together. By instead pooling morphological profiles, we may sidestep such issues stemming from variable perturbation efficacy, resulting in improved model performance.\n  \n* #### **Re: “The CWCL objective in Equ. (5) seems problematic”**\n\n    This was indeed a typo. Thank you for your attention to detail, and we will fix this in the final manuscript. \n\n**References:**  \n[1] https://www.nature.com/articles/nmeth.4293  \n[2] https://www.nature.com/articles/nbt.3026  \n[3] https://www.nature.com/articles/s41586-018-0686-x  \n[4] https://www.nature.com/articles/s41592-024-02241-6  \n[5] https://www.biorxiv.org/content/10.1101/2023.08.13.553051v1.full  \n[6] https://arxiv.org/abs/2309.14580  \n[7] https://arxiv.org/pdf/1908.08962\n\n---\n## **Please see the rebuttal box for reviewer CsPQ for our responses to questions/minor concerns.**"}]

Author rebuttal:

Meta-review: The paper proposes CellCLIP, a cross-modal contrastive learning framework for connecting cell painting images with textual perturbation descriptions. Reviewers praised the strong technical contributions, and comprehensive experiments demonstrating consistent gains over baselines.
Initial concerns focused on the clarity of the false positive and false negative issues, the choice of text encoders, and generalization. The authors provided detailed rebuttals with additional experiments, including ablations, comparisons with domain-specific encoders, and transfer learning analyses. These satisfactorily addressed all concerns, and reviewers updated or confirmed their positive recommendations.

With all reviewers supporting acceptance after rebuttal, this paper is recommended for acceptance.
Decision: Accept (poster)