A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomics Representations through Morphological Features
Authors: Ihab Bendidi
Yassir El Mesbahi
Alisandra Kaye Denton
Karush Suri
Kian Kenyon-Dean
Auguste Genovesio
Emmanuel Noutahi
Venue: ICML 2025 poster (2025)
Keywords: Multimodality
Distillation
Cross Modal Distillation
Transcriptomics
Data Augmentations
Microscopy Imaging
Gene Relationships
Primary area: applications->health_medicine

Abstract: Understanding cellular responses to stimuli is crucial for biological discovery and drug development. Transcriptomics provides interpretable, gene-level insights, while microscopy imaging offers rich predictive features but is harder to interpret. Weakly paired datasets, where samples share biological states, enable multimodal learning but are scarce, limiting their utility for training and multimodal inference. We propose a framework to enhance transcriptomics by distilling knowledge from microscopy images. Using weakly paired data, our method aligns and binds modalities, enriching gene expression representations with morphological information. To address data scarcity, we introduce (1) *Semi-Clipped*, an adaptation of CLIP for cross-modal distillation using pretrained foundation models, achieving state-of-the-art results, and (2) *PEA* (**P**erturbation **E**mbedding **A**ugmentation), a novel augmentation technique that enhances transcriptomics data while preserving inherent biological information. These strategies improve the predictive power and retain the interpretability of transcriptomics, enabling rich unimodal representations for complex biological tasks.

A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomics Representations through Morphological Features
Ihab Bendidi 123 Yassir El Mesbahi 12 Alisandra K. Denton 12 Karush Suri 12 Kian Kenyon-Dean
Auguste Genovesio Emmanuel Noutahi 12*
Valence Labs, Montréal, Canada RRecursion Salt Lake City, USA
Ecole Normale Supérieure PSL. Paris, France
Correspondence to: <augustcegenovesiioeems
<emmanuel catiaauul@@aaeenclaab..coom
Abstract
1. Introduction
Understanding how cells respond various stimuli is fun- damental to uncovering cellular functions and identifying novel drug targets. However, current technologies are lim- ited in capturing the full range of cellular activities under diverse conditions, especially given the immense complex- ity of biological systems (Conesa et al., 2016; Kharchenko, 2021). For instance, the interaction of over 20,000 hu- man protein- coding genes with the estimated 10 possible chemical compounds (Reymond, 2015) far exceeds manual analysis, necessitating computational methods. Advances in deep learning for biology, such as predicting protein structures (Jumper et al., 2021). modeling molecular bind- ing (Corso et al., 2023; Evans et al., 2021), and uncovering biological patterns through microscopy and gene expression data (Kraus et al., 2024; Bendidi et al., 2024b; Bourriez et al., 2024), offer powerful tools address this challenge from unimodal perspective. However, separately modelling data from various omics modalities, such morphological features, proteomics, and transcriptomics provides unique yet partial insights into cellular behavior (Miao et al., 2021; Carpenter et al., 2006; Lopez et al., 2018). By combining these perspectives through multimodal fusion, researchers can construct more comprehensive representations of biolog- ical systems (Lu et al., 2021; Rosen et al., 2023), revealing connections critical for accelerating drug discovery. How- ever, collecting multimodal data paired at the sample level remains infeasible currently due to massive experimental costs and technical challenges.
Understanding cellular responses to stimuli crucial for biological discovery and drug devel- opment. Transcriptomies provides interpretable. gene-level insights, while microscopy imaging offers rich predictive features but is harder to in- terpret. Weakly paired datasets, where samples from different modalities are not from the same biological replicate but share key metadata such as cell line and perturbation, enable multimodal learning but are scarce, limiting their utility for training and multimodal inference. We propose a framework to enhance transcriptomics by distill- ing knowledge from microscopy images. Using weakly paired data, our method aligns and binds modalities, enriching gene expression representa- tions with morphological information. To address data scarcity, we introduce (1) Semi- Clii-Clipped, an adaptation of CLIP for cross-modal distillation using pretrained foundation models, achieving foate-of-the-ar results, and (2) PEA (Perturbation Embedding Augmentation), a novel augmenta- tion technique that enhances transcriptomics data while preserving inherent biological information. These strategies improve the predictive power and retain the interpretability of transcriptomics, en- abling rich unimodal representations for complex biological tasks.
Given the challenges of collecting fully paired data across biological modalities, our focus is weakly paired datasets, where clusters of samples from modalities share mon biological state. In this setting, two samples from different modalities are considered "paired" if they belong
Proceedings of the nd International Conference on Ma- chine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomic Representations
to the same biological state or metadata (Xi et al., 2024). In our case, this means transcriptomics and microscopy imaging samples that are not from the same biological repli- cate but share the same cell line and were exposed to the same perturbation. However, even weakly paired datasets remain scarce due to the cost and complexity of aligning states across modalities. Only a few such datasets exist, limiting their utility for training or fine-tuning models and making simultaneous inference both modalities, for mul- timodal fusion for example, impossible to scale with current resources. To address these constraints, we aim to train models using the limited weakly paired data from transcrip- tomics and microscopy imaging, while enabling them to operate on a single modality, transeriptomics, during infer- ence. This approach leverages the complementary strengths of the modalities: microscopy images are rich in visual phenotypic features with strong predictive power but are challenging to interpret, while transcriptomics data suffers from weaker predictive power, but is more directly inter- pretable at the gene level, making it easier to connect to biological mechanisms (Kraus et al., 2024; Bendidi et al., 2024b). The complementarity between these modalities motivates the development of strategies transfer the rich phenotypic insights from microscopy into transcriptomics representations.
ing deeper insights into biological processes and expanding their utility across diverse applications.
To summarize, we introduce in this work a recipe for trans- ferring knowledge from morphological features to transcrip- tomics representations in weakly paired datasets, composed of the following contributions:
We present Semi-Clipped, a straightforward adapta- tion of CLIP (Radford et al., 2021) that leverages pre- trained large unimodal foundation models with train- able adapters. It achieves state-of-the the-ttt performance in cross-modal distillation under data-scarce conditions for our biological modalities.
We introduce PEA, Perturbation Embedding Augmentation, a novel biologically inspired data augmentation technique for representations of transcriptomics, that introduce significant variation in the training data while retaining meaningful biological information of each sample. PEA improves cross-modal distillation in our low data regime and widely outperforms existing augmentation techniques at uncovering novel biological relationships.
2. Related Works
To overcome pairing scarcity in training, we propose two practical solutions: cross-modal knowledge distillation and biologically inspired data augmentation. Knowledge distilla- tion facilitates the transfer of information from one modality to another to enhance its urility. For instance, the predic- tive strength of morphological features in microscopy im- ages can enrich transcriptomics representations, making them more powerful for downstream tasks like drug dis- covery (Kraus et al., 2024; Replogle et al., 2022; Ye et al., 2018; Chandrasekaran et al., 2023; Bourriez et al., 2024; Sanchez et al., 2025). However, most distillation techniques rely on supervised objectives, which require precise labels that often unavailable for most biological modalities. Alternatively, unsupervised alignment methods aim to un- cover shared structures between modalities, though this is challenging due to the distinct biological relationships each captures (Appendix Figure 6). We instead propose to lever- age alignment techniques for cross-modal distillation by binding transcriptomies frozen morphological representa- tions. We further introduce novel biologically inspired data augmentation technique tailored for transcriptomics vectors, which preserves biological information while introducing meaningful variation to the dataset. This augmentation ap- proach addresses the scarcity of paired data by improving the richness and robustness of transcriptomics representa- tions, enhancing their predictive power while retaining their inherent interpretability. By combining these strategies, our framework enriches gene expression representations, offer-
Cross-Modal Knowledge Distillation. Knowledge dis- tillation transfers knowledge from a teacher model to a student by aligning output distributions, typically using Kullback-Leibler (KL) divergence (Hinton et al., 2015). Variants introduce gradient similarity (Zhu & Wang. 2021), correlation (Huang et al., 2022), structural losses (Park et al., 2019). Cross-modal methods leverage strong modal- ities to guide weaker ones, often relying on label informa- tion (Gupta et al., 2016; Roheda et al., 2018; Xue et al., 2021; Lee et al., 2023). For instance, C2KD (Huo et al., 2024) uses an online filtering mechanism for soft label alignment, while SHAKE (Li & Zhe, 2022) employs shadow adapters for bidirectional distillation. XKD (Sarkar & Etemad) com- bines self supervude learning with cross moosl distillation but requires large paired datasets. To our knowledge, no distillation approach has effectively leveraged unsupervised cross-modal alignment in the context of limited weakly paired data.
Multimodal Learning. Multimodal learning encom- passes approaches for aligning or merging data types for robust inference. CLIP (Radford et al., 2021) aligns image and text into a shared space, while CSA (han Li et al., 2024) uses pretrained unimodal models for few-shot alignment. Methods like SigClip (Zhai et al., 2023). VICReg (Bardes et al., 2022), and DCCA (Lan et al., 2020) enhance align- ment through sulf-supervised learning or correlation maxi- mization but depend on significant shared information (Tsai
2

---

A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomic Representations
et al., 2021). Multimodal distillation approaches (Yang et al., 2024; Wu et al., 2023; Fang et al., 2021; Wang al., 2022) typically require paired data and focus on multimodal- to-multimodal distillation and do not leverage multimodal alignment for cross-modal distillation when only one modal- ity is available at inference. Relevant to our setting, (Hager et al., 2023) proposed a contrastive learning framework com- bining images and tabular data, structurally similar to our caanssoopyyranserrpttoooo
3. Proposed Approach
Problem Formulation. We consider two biological data modalities: a teacher modality T and a student modality S. each offering distinct perspectives on cellular behav- ior. Let &r and Xs represent the datasets from these modalities. The samples IT E XT and x E Xs correspond to the same biological perturbation and cell type but are not strongly paired due to biological variabil- ity. Each sample is annotated with weak labels p (per- turbation) and I (cell type). Both datasets are organized into biological batches By = (by.1. by. br, and Bs = (bg,1, bs.,, bs.j Each batch by. E BBr and bs.m E Bs consists of set of samples (17 Nr. and {a j=1 Ns.m and each batch includes, in addition to per- turbed samples, a number of control (unperturhed) samples, C...m
setting, demonstrating the via- bility of such combinations for predictive and interpretable medical tasks.
Biologically Relevant Representations. Advances in mi- croscopy imaging models have driven progress in high- content screening (Kraus et al., 2024; Yao et al., 2024; Kenyon-Dean et al., 2025; Wenkel et al., 2025), histopathol- ogy (Saillard et al., 2024; Chen et al., 2024; Vorontsov et al., 2024), and specialized architectures (Bourriez et al., 2024; Pham & Plummer, 2024). In transcriptomics, foundation models (Cui et al., 2024; Yang et al., 2022; Theodoris et al., 2023; Wen et al., 2024) show promise but often underper- form simpler models in biologically relevant tasks, with scV1 (Lopez et al., 2018) as an exception (Liu et al., 2023; Bendidi et al., 2024b). Microscopy imaging complements transcriptomies (Camunas-Sole Soler, 2024), but while unimodal datasets (Replogle et al., 2022: Chandrasekaran al., 2023; Fay et al., 2023) are growing, weakly paired multimodal datasets remain scarce. Recent methods (Xi et al., 2024; Watkinson et al., 2024;
denoted by {aT, (Tkkees J and {~) 10 for CM.K 2.
Proposed Distillation Method. Given the scarcity of weakly paired data, we adopt pretrained and frozen uni- modal encoders Er : Xr - Rdr and Es : Xs + Rds, following (Fradkin et al., 2024). These encoders produce embeddings =(i) = (x (8)) and =(()) = Eg (xxg)
) for the teacher and student modalities, respectively. Our objective is to learn a mapping function fs : IRds + IRdr that aligns student embeddings to the teacher embedding space, yield- ing transformed embeddings n = fs(zg) (1) that integrate properties from the teacher modality T. We aim to achieve the dual objective of leveraging weak biological labels for pairing while minimizing reliance on them as learning objec- tives, since such labels underperform compared to unsuper- vised objectives in microscopy imaging (Kraus et al., 2024), and preventing mutual drift between modalities with limited shared information. We propose Semi-Clipper forward adaptation of the CLIP loss (Radford et al., 2021) for cross-modal knowledge distillation. This approach uses the frozen unimodal encoders to generate embeddings 27 and zs- The teacher representation zт is fixed, while an adapter function fs is trained on the student modality by optimizing the CLIP loss between hgg and 27 to produce aligned embeddings hg. By freezing the teacher embedding space, this avoids dependence on massive amounts of paired data for encoder training and ensures one-way way knowledge transfer from the teacher to the student, mitigating mutual drift and feedback from the student to the teacher.
et al., 2023; Xie et al., 2023) address this kind of limitation for different modalities by leveraging weak pairings through pretrained models with trainable adapters (Fradkin et al., 2024).
straight-
Data Augmentations for Biology. Data augmentations are crucial in addressing data scarcity for biology, as biolog- ically meaningful augmentations can stabilize and improve performance with limited biological datasets (Moutakanni et al., 2024; Bendidi et al., 2023; 2024a). In computational biology, image augmentations have typically focused on basic transformations like rotations (Alfasly et al., 2024: Lafarge & Koelzer, 2022) or differentiable techniques us- ing adversarial learning for domain generalization (Ruppli et al.. 2022; Zhou et al., 2024). For transcriptomics, data being in representation format allows leveraging existing representationnlevel augmentation methods (DeVsies & Tay- lor, 2017; Li et al., 2022), though their efficacy for biologi- cal contexts remains uncertain. Recently, new biologically inspired techniques have emerged specifically for augment- ing transcriptomics and biological representations (Kircher et al., 2022; Li et al., 2023; Nouri, 2025).
Batch Correction for Data Augmentation. In biolog- ical datasets, batch effects, or variability caused by dif- ferences in experimental conditions, introduce noise that can obscure meaningful patterns. Traditional batch correc- tion techniques (Bendidi et al., 2024b; Celik et al., 2024; Ando et al., 2017) address this by centering embeddings on control (unperturhed) samples within each batch, reducing
3

---

A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomic Representations
noise while preserving the signal. Typically used as post-
Algorithm 1 Semi-Clipped with PEA implementation
processing step, these corrections shift the embedding distri- bution while retaining key information for downstream anal- ysis. To tackle the scarcity of paired biological modalities, we introduce PEA (Perturbation Embeddings Augmenta- tion), a novel biologically inspired augmentation technique that repurposes batch correction as data augmentation ap- plied directly to the student embeddings during training. Specifically, function A (IRds, Xg) -> Iede is randomly selected from a set A of batch correction transformations and applied to the student embeddings =g" Augmented (()
for each batch (2)) E (Xs, Xr) do
Extract using frozen encoders =s Es( and 2T
Ex((
Sample batch correction function A ~ A Drop a random subset of steps in A + A'
Sample a random subset of control samples X Apply batch correction: = = X's,, (()) Compute transformed embeddings: hiS = fs Apply TVN correction to teacher embeddings: 2s.
(2Tr) and compute CLIP loss:
embeddings =S.4 = A(=s) (8) x are then passed to the student adapter fs for cross-modal knowledge distillation. To ensure the teacher embeddings focus on relevant infor- mation, fixed batch correction B (Ando et al., 2017) is applied to the teacher modality.
B Σ log
exp(sim hs 25 )/T)
L=
exp(sim (hg) 24
(b.3)) )
To augment transcriptomics data while preserving biological relevance, we extend traditional batch correction techniques into stochastic augmentation framework. Specifically, for each sample, we randomly select one batch correction transformation A from predefined set of normalization techniques, ensuring controlled variability in perturbation embeddings. Each selected transformation A Rd + Rds falls into one of these categories: (1) centering, which shifts embeddings by subtracting batch-wise control means to re- move batch-speeific offsets: (2) scaling, which normalizes variance across features to enhance comparability; and (3) principal component-based transformations that reweight variance along principal axes, emphasizing biologically rel- evant information while reducing batch artifacts. To intro- duce further stochasticity, we drop a random subset of the steps of each correction method per sample rather than al- ways applying them sequentially. Additionally, the number of control samples used for correction randomly sampled per training sample, increasing diversity and robustness to out-of-domain experimental shifts in the learned distri- butions. Detailed implementation of our batch correction techniques is provided in Appendix Section C.
Backpropagate loss L and update adapter fs
end for
rayed bulk transeriptomies samples (HUVEC-CMPD) and 20,000 microscopy images of human umbilical vein en- dothelial cells (HUVEC), cells from cell painting, both cov- ering 1,700 chemical perturbations at three concentrations. Each transcriptomics sample can pair with multiple imaging samples based on treatment and concentration, with one pair randomly selected per epoch. These pairs are weakly paired-i.e., they do not originate from the same biological replicate but share the same cell line and perturbation meta- data, ensuring comparable biological states across modali- ties. For encoding microscopy images, we use the pretrained Phenom-1 I model (Kraus et al., 2024), a state-of-the pre- trained model trained on 93 million microscopy images. For transcriptomics, we compare three models: a simple se VI- like MLP! trained from scratch on the HUVEC- bulk dataset, VI (Lopez et al., 2018), model known for strong performance on small datasets, outperforming existing tran- scriptomics pretrained models (Bendidi et al., 2024b), and similarly trained from scratch on the HUVEC-CMPH bulk dataset, and a pretrained scGPT (Cui et al., 2024) (a pre- trained model trained on 33 million transcriptomics sam- ples). A three-layer MLP adapter fs (input size ds, output size dy) with ReLU activations is trained for the student modality, while both encoders remain frozen. Consistent with (Kenyon-Dean Dean et al., 2025), control samples are ex- cluded from paired data for knowledge distillation and only used for batch correction. The adapter is trained with a tem- perature of 0.1, 1, learning rate of 0.001, batch size of 1,024, and over 150 epochs.
This method introduces controlled and diverse distributional shifts, helping Is learn robust, biologically meaningful rep- resentations by ignoring batch-induced variability. During inference, batch correction is applied to the student embed- dings 25 align them with the training distribution, further improving robustness. Algorithm 1 details the process, en- suring the adapter captures biologically relevant features while increasing training diversity and preserving biological information in low-data settings.
4. Experimental Setup
Data & Model Training. We use microscopy imaging as the teacher modality and transcriptomics as the stu- dent modality. The training dataset includes 130,000 ar-
Vs is an ML.--aaeed VAE conditionned on a batch label.
4

---

A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomic Representations
Evaluation Setting. The evaluation focuses on assessing the quality of transcriptomic representations after knowl- edge distillation, emphasizing biological relevance and inter- pretability. We use a hierarchical benchmarking framework for transcriptomic representations (Bendidi et al., 2024b)
20-
Roccll redttoon
30
10
(Appendix Section B) with two primary tasks: (1) Retrieval of known biological relationships, this task evaluates the ability of the learned representations capture established biological relationships by retrieving known interactions between genes. Using cosine similarity of gene embed- dings, predicted relationships are validated against annota- tions from CORUM, HuMAP, StringDB, Reactome, and SIGNOR databases. Success is measured by recall scores averaged across these databases, reflecting how well the rep- resentations align with known biology. (2) Transcriptomic interpretability preservation, this task measures how well the distilled embeddings retain information necessary for reconstructing original gene expression profiles. It evaluates two complementary metrics: the Structural Integrity score, which quantifies how accurately the model preserves the relationships between control and perturbation samples, and the Spearman correlation, which assesses the rank-bac baaeed agreement between predicted and true gene expression pro- files. The average of these metries provides comprehensive measure of interpretability preservation.
teit lite sap crined from scrtth
Pretrained seji
In Adapter
Protrained scin + Adspter image Adapter
scceet finttoeed
ScOPT Adupter
SCEPT Adaeter Image Adupter
Figure I. Impact of training choices on Semi- Semi-Clipped performance for known hiological relationship recall on HUVEC-KO. Finetun- ing multimodal training from scratch underperforms due to lim- ited weakly paired data, while using adapters on pretrained models significantly improves results. The best performance is achieved with Semi-Clipped: a single transcriptomic adapter aligned to frozen image representations.
tinct distribution shifts. (1) Experimental variability: The HUVEC-KO dataset contains arrayed bulk transcrip- tomics data from 120,000 genetically perturbed sample, with around 300 CRISPR gene Knock-Out (KO) in HU- VEC cells, unlike the training set, which uses chemical perturbations. This dataset does not share any experiment with the training set, and evaluates generalization to un- seen experiments and unseen genetic perturbations. (2) Quantification method shift: The LINCS dataset (Sub- ramanian et al., 2017) includes 443,000 arrayed bulk tran- scriptomics samples across 31 cell types and 5,157 CRISPR gene KO, using the L1000 assay, transcript abundance measurement method different from the requencing-based approach in training. (3) Single-cell adaptation: The SC- RPE/ dataset (Replogle et al., 2022) consists of 247,915 1 single-cell transcriptomic samples from retinal pigmented epithelium cells with 2,393 CRISPR knockouts, testing the transition from bulk transcriptomies (training dataset) to single-cell transeriptomics. Together, these three OOD eval- uation settings introduce significant distribution shifts on different aspects, testing the model's 's robustness to new cell types, experimental conditions, and gene expression quan- tification methods.
Success is defined as improving retrieval scores while main- taining interpretability metrics comparable to unimodal transcriptomic representations. This dual focus ensures that the student representations do not collapse or lose transeriptomic-specffe information by ignoring it and re- lying solely on morphological features. Using these tasks, we compare our Semi-Clipped approach, with and with- out PEA, against standard multimodal alignment and cross- modal knowledge distillation methods. For alignment, we include CLIP (Radford et al., 2021), SigClip (Zhai et al., 2023), VICReg (Bardes et al., 2022), and DCCA (Lan al., 2020). For distillation, we evaluate KD (Hinton et al., 2015), SHAKE (Li & Zhe, 2022), and C2KD (Huo et al., 2024). All methods use the same pretrained encoders with trainable adapters. For distillation approaches, teacher and student adapters are unimodally pretrained with perturbation la- bels before fine-tuning via their respective methods. We benchmark PEA by applying to zs during training, and compare existing biological and transcriptomics data augmentation approaches: MWO (Kircher et al., 2022), VI denoising (Lopez et al., 2018), MDWGAN-GP (Li et al., 2023), ScGFT (Nouri, 2025), and their combination with and without PEA. Hyperparameters are optimized via grid search on validation split, with results averaged across multiple seeds.
5. Results
We aim to evaluate the impact of Semi-Clipped and PEA both independently and in combination. Our primary ob- jective is to improve biological relationship recall on OOD datasets compared to the corresponding unimodal transcrip- tomic baseline while preserving or enhancing interpretabil- ity in transcriptomics.
Evaluation Datasets. We assess generalization on three Out-Of-Distribuute (OOD) datasets, each introducing dis-
5

---

A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomic Representations
Distillation Benchmark
a.
Semi--
Sigcip
ViCRtg
DOGA
SHARKE
C2KD
Unimodal Baseline
incerpretabiitty Iessrvvation
Rndationsiip
pectabliity
Interpretablitty
m wooed cre
decae
drck
HUVEC-KO
UNCS
SC-RPE1
b.
Augmentation Benchmark
KM
NDW-GAN
CFF
MNO
w/o PEA
PEA
wi PEA
Wr Aug
Tx Interpreinbillyy Preservation
Krown Bo Relatonstip Recal
Ta byterpeetatiiit Preservation
Known Bis Relanlooshipp Recall
Interpretatiiity Preservation
Krown fio Reiationship Recal
17.5
54.
11.2
35.
cal
earry
Hert
mary
Pea
nu
al nee
nea
HUVEE-KO
LINCS
SC-RPE1
Figure 2. Performance comparison of the distillation and augmentation components of our approach compared to existing distillation methods (a) and biological data augmentation techniques (b) across five training seeds. Higher is better for all metrics. Semi-Clipped and PEA maintain interpretability and achieve the highest performance on all OOD datasets. (a) Z-scores of evaluation metrics (relationship recall and Tx preservability) are shown, with cool colors for label-based methods and warm colors for label-free approaches, without data augmentation. (b) Raw scores are shown for relationship recall and Tx preservability. Transcriptomics data augmentations, MWO (Kircher al., 2022), sc denoising (Lopez et al., 2018), MDWGAN GP (Li et al., 2023), scGFT (Nouri. 2025), are applied within Semi-Clipped training. We compare training results where we simultaneously use all evaluated data augmentations, both with and without PEA, to assess its additional impact in a practical setting on both evaluation tasks.
5.1. Semi-Clipped Enables Robust and Generalizable Transcriptomic Representations
SeVI pretrained encoder for transcriptomics. For clar- ity, we define label-free approaches those that do not use biological labels in the training objective, even if labels are used for modality pairing. To ensure a fair comparison of the core methods, no data augmentation is applied. Figure (a) presents the performance of Semi-Clipped against various label-based and label-free distillation ap- proaches on the Transcriptomic Interpretability Preservation and Known Biological Relationship Recall tasks across all three OOD datasets. Scores are standardized as z-scores and averaged over 5 seeds, with higher values indicating better performance. Distillation methods using label super- vision (cool colors) generally show weaker relationship re- call compared to unsupervised multimodal methods (warm colors) and even underperform the unimodal bascline in LINCS and SC-RPEI. In contrast, Semi-Clipped achieves the highest relationship recall in HUVEC-KO and SC-RPE1 while also slightly surpassing the unimodal baseline in tran- scriptomics interpretability in HUVEC-KO. This suggests successful knowledge transfer from morphology to tran- scriptomies without sacrificing interpretability. In LINCS, Semi- Cemi-Clipped performs competitively, outperforming all aabel-supervised distillation methods and the unimodal base- line in relationship recall while closely matching the best unsupervised multimodal methods. On the transcriptomics
To analyze the effect of different training choices Semi- Clipped performance, we first examine its impact on known biological relationship recall using the HUVEC-KO dataset. Without data augmentations and using the CLIP loss, we conduct two comparisons. Figure (left) compares training an Vi-lke MI.P from scratch for the distillation task against using a pretrained sc model. Additionally, it evaluates the effect of introducing an image adapter instead of relying solely on a transcriptomics adapter while keep- ing image embeddings frozen. Figure (right) compares finetuning a pretrained scGPT model for distillation versus freezing scGPT and training a transcriptomics adapter on its own or with an image adapter. We find that leverag- ing a pretrained encoder with adapters consistently outper- forms both training from scratch and finetuning for both scVI and scGPT. Furthermore, aligning transcriptomic rep- resentations to frozen image embeddings, as proposed in Semi-Clipped, yields superior performance compared to also training a microscopy imaging adapter.
We evaluate Semi-Clinn Clippelspssd's s ability to learn generalizable and biologically meaningful representations of transcrip- tomics compared existing distillation methods, using an
6

---

A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomic Representations
scanscciptoomss Adapter LR
Temperature
Batch Size
Epochs
19.75
10.7
19.55 59
13.4
19.4
19.5
19 00
score
10.50
13.4
10.3
10.2
10:27-
19.3
19.1
13.0
18.00
15.73
19.0
10:05 0.0001 000L 0.01 0.1
0.5
238 514 1004
200
Learning Rate
Temperature
Batch Size
[pochs
Figure 3. Ablation study on the known relationship recall score hyperparameters choices (Tx Adapter learning rate, CLIP loss temperature, batch size, and training epochs) for training Semi-Clipped on the HUVEC-KO dataset, including the selected optimal configuration (dotted vertical line). For each studied parameter, we set all other hyperparameters at their best performing value. While performance varies with parameter changes, the method remains largely robust, showing minimal degradation and no collapse
preservation metric for LINCS and SC-RPEI, Semi-Clipped retains strong interpretability, slightly trailing the unimodal baseline but outperforming most distillation approaches. This minor limitation likely reflects the challenge of main- taining interpretability across unseen cell types. Overall, Semi-Clippedé effectively balances generalization and inter- pretability across all OOD settings, consistently achieving the most robust performance across all metrics and demon- strating its strength as a distillation method.
conduct a broader comparison. Specifically, we compare Semi-Clipped trained with all existing biological augmenta- tions except PEA against its performance when trained with the full set of augmentations, including PEA. Figure (b) presents the results of this comparison. Across all three eval- uation datasets, PEA achieves state-of-the-a performance in Known Biological Relationship Recall, significantly out- performing all existing approaches. It also preserves tran- scriptomic interpretabilityy matching the no-augmentation baseline in SC-RPEI while surpassing it in HUVEC-KO and LJNCS. Notably, PEA alone improves performance over not using augmentations by 17% in HUVEC-KO, 55% in LINCS, and 20% in SC-RPEJ. More strikingly, PEA out- performs the combined effect of all other biological aug- mentations used together, highlighting its strong biological foundation and ability to introduce meaningful variation to the distillation process. Furthermore, integrating PEA with all other augmentations further enhances performance be- yond using PEA alone, demonstrating its complementarity to existing transcriptomies augmentation techniques. This combined approach yields the highest overall improvements, increasing performance over the no-aagmentation baseline by 25% in HUVEC-KO, 69% in LINCS, and 26% in SC- RPEI. These results confirm that PEA not only provides substantial individual benefits but also synergizes effectively with existing augmentation strategies.
To further assess Semi-Clipped'ss robustness, we conduct a detailed ablation study on individual hyperparameters when trained independently on the HUVEC-KO dataset (Figure
3). We isolate the effect of each parameter by fixing all others to their optimal values. The results reveal that while performance fluctuates with changes in configuration, Semi- Clipped remains resilient, exhibiting only minor degrada- tion without any performance collapse. Optimal learning achieved with a balanced learning rate, lower tempera- ture for the CLIP loss term, and larger batch sizes, though the method still performs competitively even with small batches. Additionally, increasing the number of training epochs yields substantial improvements. These findings reinforce the method's stability and reliability across a wide range of training conditions.
5.2. PEA Enhances Distillation Across Methods and Synergizes with Existing Augmentations
We assess whether PEA enhances performance across differ- ent distillation approaches beyond Semi-Clipped and com- pare its impact on various methods. Specifically, we apply PEA to KD, SHAKE, VICReg, and Semi-Clipped and eval- uate its effect on benchmark tasks. Each method is trained over 15 different seeds, both with and without PEA, and we use a Wilcoxon signed-rank test to determine the statisti- cal significance of improvements. Table summarizes the results: PEA consistently enhances performance across all three OOD datasets for every distillation approach, with par- ticularly strong gains in LINCS and SC-RPEI. This confirms that PEA is broadly beneficial across methods. Notably,
We further evaluate the effectiveness of our proposed PEA data augmentation in enhancing distillation performance across both evaluation tasks. Using Semi-Clipped as the base model, we compare its performance over five training seeds in three settings: (I) without any data augmentation, (2) with multiple existing biologically inspired transcrip- tomics augmentations from the literature, each used sepa- rately, and (3) with PEA as the sole augmentation. This initial evaluation isolates the specific contribution of PEA. Additionally, to reflect multiple augmentations are typically applied together, we
training conditions where

---

A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomic Representations
HUVEC-KO
LINCS
SC-RPEI
Method
Tx Preservation
Known Relationships
Tx Preservation
Known Relationships
Tx Preservation
Known Relationships
Random baseline
33.92 0.09
10.37 0.11
47.09 0.07
10.81 1 0.04
25.34 0.11
10.03
Unimodal haseline
52.23 0.34
16.51 0.85
93.35 0.07
12.21 0.11
37.75 + 0.28
25.29 0.24
KD
$2,90 0.31
16.00 1.36
92.69 1: 0.15
11.83 1. 0.27
37.43 1. 0.32
23,9 0.18
KD PEA
54.12 = 0.63
20.05 1.78
1.9.11 + 0.29
15.73 + 0.59
137.55 = 0.38
1.20.2 + 0.36
SHAKE
51.93 0.80
17.02 1.02
91.64 1: 0.23
12.09 1. 0.49
36,95 1. 0.46
25.13 0.19
SHAKE+ PEA
12.993 = 0.83
19.98 1.34
1.92.33 + 0.31
16.84 + 0.51
3.3.155 0.51
130.81 + 0.31
VICReg
51.8 0.39
17:25 4: 1.14
91.19 :: 0.19
12.96 1: 0.45
36.75 0.17
32.19 = 0.26
VICReg PEA
53.77 + 0.66
102046 0.83
1.1.22 + 0.25
18.112 + 0.19
4.3633 + 0.22
1.8.14 + 0.29
Semi- Cemi-Clipped
52.78 0.27
19.71 = 1.19
92.71 # 0.23
12.68 +: 0.33
37,54 +: 0.19
32.65 = 0.21
Semi-Clipped PEA
53.87 + 0.37
23.05 0.42
13.155 + 0.38
19.63 + 0.18
1 1.7.56 + 0.15
1.3984 + 0.23
Table 1. Performance improvement of different distillation methods with and without PEA under all OOD settings. We average the scores of 15 different seeds for each model, and the p-value of every result improvement is below 0.05 using the Wilcoxon signed-rank statistical test. Improvements from using PEA are indicated with upward arrows. For each OOD setting, the best-performing model shown bold, and the second-best is underlined. Using PEA as data augmentation for distillation approaches preserves the transcriptomics information while widely improving the zero-shot retrieval of known biological relationships for all the three OOD datasets used for evaluation.
also significantly improves Transcriptomic Interpretabil- ity, likely due its ability to preserve biological informa- tion while introducing controlled variations, enhancing the signal-to-noise ratio. All gains in Known Relationship Re- call between PEA and non-PEA settings are statistically significant (p-values < 0.05). Importantly. Semi-Clipped remains the top-performing approach in Known Biological Relationship Recall across all evaluation datasets when us- ing PEA, while also achieving the second-best performance in Transcriptomic Interpretability Preservation across all datasets.
5.3. Semi-Clipped with PEA Enables Synergistic Integration of Morphological and Transcriptomic Insights
We analyze the biological insights provided by Semi- Clipped trained with PEA, comparing the known biological relationships it retrieves to those identified independently by unimodal microscopy imaging and transcriptomics models on the HUVEC-KO OOD dataset. Specifically, we evalu- ate the quantity and overlap of relationships retrieved by KD, SHAKE, VICReg, and Semi-Clipped, all trained with PEA, to assess whether these models remain faithful to transcriptomics-speee
We analyze the contribution of each PEA component to performance improvements by conducting an ablation study on the HUVEC-KO dataset. We evaluate its impact on KD, SHAKE, VICReg, and Semi-Clipped, progressively adding PEA components the base distillation methods without augmentations. Each step in the ablation builds upon the pre- vious one: (1) Fixed biological augmentation applying predefined set of batch correction techniques. (2) Inference on TVN-coorected embeddings: applying Typical Varia- tion Normalization (TVN) (Ando et al., 2017) correction to 2y at inference before passing them to the adapter fs. (3) Augmentation stochasticity randomly dropping subset of batch correction steps to introduce variation. (4) Control sampling: randomly sampling varying amount of control samples for correction, completing the full PEA approach. Table 2 summarizes the results, averaged over 15 seeds and reporting Known Biological Relationship Recall for each distillation approach. Every component contributes incre- mental improvements, with control sampling providing the strongest boost, particularly for Semi-Clipped. Importantly, all distillation methods show consistent performance gains at each step, indicating that each PEA component plays critical role in enhancing distillation outcomes.
relationships exhibit modality drift. This is quantified by measuring the intersection be- tween relationships retrieved by each distillation method and those identified by the unimodal transcriptomics model. Figure 4 presents Venn diagrams of these intersections. Semi- Semi-Clipped shows strong alignment with transcriptomics- retrieved relationships while also capturing additional bi- ological insights typically associated with morphological features. In contrast, while the other distillation approaches retrieve many known relationships. they exhibit minimal overlap with those identified by transcriptomics alone. No- tably, KD and SHAKE, both label-based methods, demon- strate particularly weak alignment with transcriptomics re- lationships, likely due to the confounding effects of weak biological labels used during training. These findings sug- gest that Semi-Clipped effectively preserves transcriptomic insights while significantly emriching them with complemen- tary morphological information, achieving a better balance between biological faithfulness and multimodal integration.
We next investigate whether distilling morphological fea- tures into transcriptomies yields a purely additive effect or if it generates emergent synergies between modalities. To assess this, we analyze the set Distillation \ (Transcrip- tomics U Microscopy) in Figure 4, representing relationships
8

---

A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomic Representations
PEA Configuration
KD
SHAKE
VICReg
Ours
Microscopy Imaging ng
Transcriptomics
Distillation
Base Method
16.00
17.02
17.25
19.71
KD
SI IAKE
+ Fixed Bio-Aug
16.76
17.22
17.97
19.95
+ Inference on TVN
18.76
18.32
18.62
20.58
Aug, Stochasticity
19.37
18.84
19.43
21.79
Ctrl Sampling (PEA)
20.65
19.98
20.46
23.05
125
15
114
14
14
12
22
Table Ablation study of different PEA components on HUVEC- KO evaluation dataset, on retrieval of known relationships of dif- ferent distillation methods, averaged over 15 seeds. We see that combining all PEA components achieves signiticant improvements, especially when using our Semi-Clipped approach.
11
57
31
VICReg
Semi-Clipped
uniquely retrieved by the distillation model but absent in unimodal transcriptomics or microscopy imaging. For all evaluated methods, we perform Gene-Set Enrichment Anal- ysis (GSEA) (Subramanian et al., 2005) to identify enriched biological pathways within this set compared to other dis- tinct relationships retrieved by each distillation approach, filtering for gene sets with p-values 0.01. Surprisingly, KD, SHAKE, and VICReg fail to significantly enrich any bi- ological pathway, whereas Semi-Clipped uniquely enriches pathways related to the cell cycle and post-translational modifications (Appendix Table 3). This suggests that dis- tilling morphological traits into transcriptomics using our approach enhances the capture of cell cycle-related informa- tion, which may be less detectable or noisier in either modal- ity alone. This outcome likely arises from Semi- Clipp''ss s ability to integrate rich phenotypic information from mi- croscopy imaging, including morphological traits, spatial organization, and cellular process indicators, with transcrip- tomic markers such as mitochondrial RNA gene counts, often associated with cell cycle activity. This fusion enables deeper biological synergy, allowing distillation to reveal novel biological insights that neither modality could achieve independently, while still permitting unimodal inference rather than requiring multimodal fusion.
15
102
15
110
21
35
21
10
25
1/
19
25
Figure 4. Venn diagrams of retrieved biological relationships for KD, SHAKE, VICReg, and Semi-Clipped (all trained with PEA) on the HUVEC-KO OOD dataset. Semi-Clipper shows the high- est overlap with transcriptomics while integrating morphological insights, whereas KD and SHAKE exhibit the weakest alignment, possibly due to reliance on weak biological labels. Detailled mea- sures of the gains and losses of each method in each modality are available in Figure 5.
critical expression signals, PEA introduces plausible vari- ability while maintaining essential biological properties. This approach significantly improves cross-moda distilla- tion performance, increasing Known Biological Relation- ship Recall in OOD tests while preserving interpretability. Beyond aligning transcriptomic and morphological informa- tion, Semi- cemi-Clipped reveals emergent biological synergies, particularly in cell cycle regulation and post-translational modifications. Despite these advantages, challenges remain. Random pairing within treatment groups may dilute repre- sentation quality when subtle intra-group differences exist, highlighting the need for better matching strategies. Lim- ited large-scale paired data also restricts broader applica- bility. Nonetheless, Semi-Clipped is computationally effi- cient: training on scaled version of our dataset with 1.3 million weakly paired samples takes only 19 hours on a sin- gle H100 GPU, thanks to the use of frozen backbones and lightweight adapters. As multimodal datasets grow, scaling these methods could further advance biological research and cross-modal understanding.
6. Discussion
In this work, we introduced Semi-Clipped, self self-supervised framework for distilling morphological knowledge of biol- ogy into transcriptomic representations using multimodal alignment techniques. Additionally, we proposed PEA, a biologically informed augmentation strategy that repurposes batch correction to enhance representation learning. Our results demonstrate that Semi-Clipped outperforms existing distillation methods while preserving transcriptomic inter- pretability. Furthermore, we show that label-free distillation consistently surpasses label-based approaches, reinforcing that biological labels often lack the granularity needed to fully capture cellular complexity. A key contribution of this work is the reinterpretation of batch correction as a biologically meaningful data augmentation. Unlike conven- tional transcriptomic data augmentations that may disrupt
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning in its application to life sciences. There are many potential societal consequences of our work,

---

A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomic Representations
especially relating the discovery of new biological rela- tionships and potential drug treatments. Utmost care should be taken to validate safety and efficacy of model predictions in pre-clinical trials.
exploring perturbative maps of transcriptional and mor- phological data. PLOS Computational Biology, 20(10); 1-24, 10 2024. doi: 10. 10.1371/jourrmal.pbb 101100.0
Chandrasekaran, S. N., Ackerman, J., Alix, E., Ando, D. M., Arevalo, J., Bennion, M., Boisseau, N., Borowa, A., Boyd, J. D., Brino, L.., Byrne, P. J., Ceulemans, H., Ch'ng, C., Cimini, B. A., Clevert, D.-A,, Deflaux, N., Doench, J. G., Dorval, T., Doyonnas, R., Dragone, V., Engkvist, O., Faloon, P. W., Fritchman, B., Fuchs, F., Garg, S., Gilbert, T. J., Glazer, D., Gnutt, D., Goodale, A., Grignard, J., Guenther, J., Han, Y., Hanifehlou, Z., Hariharan, S., Her- nandez, D., Horman, S. R., Hormel, G. Huntley, M., Icke, I., lida, M., Jacob, C. B., Jaensch, S., Khetan, J., Kost- Alimova, M., Krawiec, T., Kuhn, D., Lardeau, C.-H., Lembke, A., Lin, F., Little, K. D., Lofstrom, K. R., Lotfi, S., Logan, D. J., Luo, Y., Madoux, F., Marin Zapata, P. A., Marion, B. A., Martin, G., McCarthy, N. J., Mervin, L., Miller, L.., Mohamed, H., Monteverde, T., Mouchet, E., Nicke, B., Ogier, A., Ong, A.-L., Osterland, M., Otrocka, M., Peeters, J., Pilling. J., Prechtl, S., Qian, C., Rataj, K., Root, D. E.,, Sakata, S. K., Scrace S., Shimizu, H., Simon, D., Sommer, P., Spruiell, C., Sumia, I,, Swalley, S. E., Terauchi, H., Thibaudeau, A., Unruh, A., Van de Waeter, J., Van Dyck, M., van Staden, C., Warchot, M., Weisbart, E., Weiss, A.. Wiest-Daesske N., Williams, G., Yu, S., Zapiec, B., Zyta, M., Singh, S., and Carpenter, A. E. JUMP cell painting dataset: morphological impact of 136,000 chemical and genetic perturbations. bioRxiv, March 2023.
References
Alfasly, S., Shafique, A.. Nejat, P., Khan, J., Alsaafin, A., Alabtah, G., and Tizhoosh, H. Rotation-agnostic image representation learning for digital pathology. In CVPR, 2024.
Ando, M., McLean, C. Y., and Berndl, M. Improving phe- notypie measurements in high-content imaging screens. bioRxiv, July 2017.
Bardes, A., Ponce, J., and LeCun, Y. Vicreg: Variance- invariance-covariance regularization for self-supervised learning. In ICLR, 2022.
Bendidi, I., Bardes, A., Cohen, E., Lamiable, A., Bollot, G., and Genovesio, A. No free lunch in self supervised representation learning, 2023.
Bendidi, I., Bardes, A., Cohen, E.,, Lamiable, A., Bollot, G., and Genovesio, A. Exploring self-supervised learning biases for microscopy image representation. Biological Imaging, 4, 2024a. doi: 10.1017/5263900324000011
Bendidi, I., Whitfield, S., Kenyon- Deayn, K., Yedder, H. B., Meshahi, Y. E.,, Noutahi, E., and Denton, A. K. Bench- marking transcriptomics foundation models for perturba- tion analysis one pea still rules them all. In NeurlPS AIDrugX Workshop, 2024b.
Chen, R. J., Ding, T., Lu, M. Y., Williamson, D. F. K., Jaume, G., Song, A. H., Chen, B., Zhang, A., Shao, D., Shaban, M., Williams, M., Oldenburg, L., Weishaupt, L. L., Wang, J. J., Vaidya, Le, L. P., Gerber, G., Sahai, S., Williams, W., and Mahmood, Towards general- purpose foundation model for computational pathology. Nature Medicine, 2024.
Bourriez, N., Bendidi, I., Cohen, E., Watkinson, G., Sanchez, M., Bollot, G., and Genovesio, A. ChAda- ViT Channel Adaptive Attention for Joint Representa- tion Learning of Heterogeneous Microscopy Image.. In CVPR, 2024.
Conesa, A., Madrigal, P., Tarazona, S., Gomez-Cabree D., Cervera, A., McPherson, A., Szezešniak, M. W., Gaffney, D. J., Elo, L. L.,, Zhang, X., and Mortazavi, A. A survey of best practices for RNA-seq data analysis. Genome Biol., 17(1), December 2016.
Monunas-Solers J. Integrating single-cell transcriptomics with cellular phenotypes: cell morphology, ca2+ imag- ing and clcctrophysiology. Biophys. Rev., 16(1):89-107, February 2024.
Corso, G., Stärk, H., Jing, B., Barzilay, R., and Jaakkola, T. Diffdock: Diffusion steps, twists, and turns for molecular docking. In ICLR, 2023.
Carpenter, E., Jones, T. R., Lamprecht, M. R., Clarke, C., Kang, I. H., Friman, O., Guertin, D. A., Chang, J. H., Lindquist, R. A., Moffat, J., Golland, P., and Sabatini, D. M. CellProfiler: image analysis software for identi- fying and quantifying cell phenotypes. Genome Biol., 7 R10),R100, October 2006.
Cui, H. Wang, C., Maan, H., Pang. K., Luo, F., Duan, N., and Wang, B. ScGPT: toward building foundation model for single-cell multi-omics using generative AI. Nat. Methods, (480)-:470-1888 August 2024.
Celik, S., Hütter, J.-C., Carlos, S. M., Lazar, N. H., Mohan, R., Tillinghast, C., Biancalani, T., Fay, M. M., Earnshaw, B. A., and Haque, I. S. Building, benchmarking, and
DeVries, T. and Taylor, G. W. Dataset augmentation in feature space, 2017.
10

---

A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomic Representations
Evans, R., O'Neill, M., Pritzel, A., Antropova, N., Senior, A., Green, T., Zidek, A., Bates, R., Blackwell, S., Yim, J., Ronneberger, O., Bodenstein, S., Zielinski, M., Bridgland, A., Potapenko, A., Cowie, A.. Tunyasuvunakool, K., Jain, R., Clancy, E., Kohli, P., Jumper, J., and Hassabis, D. Protein complex prediction with SophaFold-Multingg
K., Kohli, P., and Hassabis, D. Highly accurate pro- tein structure prediction with alphafold. Nature, 596 (7873):583-589, Aug 2021. ISSN 1476-4687. doi: 0000888611586-00--388811
Kenyon-Dean, K., Wang, Z. J., Urbanik, J., Donhauser, K., Hartford, J., Saberian, S., Sahin, N., Bendidi, I., Celik, S., Fay, M., Vera, S. R., Haque, I. S., and Kraus, O. Vitally consistent: Scaling biological representation learning for cell microscopy. ICML, 2025.
biorxiv, October 2021.
Fang, Z., Wang, J., Hu, X., Wang, L., Yang, Y., and Liu, Z. Compressing visual- linguussi model via knowledge distillation, In ICCV, 2021.
Kharchenko, P. V. The triumphs and limitations of compu- tational methods for scRNA-seq. Nat. Methods, 18(7): 723-732, July 2021.
Fay. M. M., Kraus, O.. Victors, M., Arumugam, L., Vuggu- mudi, K., Urbanik, J., Hansen, K., Celik, S., Cernek, N., Jagannathan, G., Christensen, J., Earnshaw, B. A., Haque, S., and Mabey, B. Rxrx3: Phenomics map of biology. bioRxiv, 2023. doi: 10. 10.1100/2223302.0.5223333
Kircher, M., Chludzinski, E., Krepel, J., Saremi, B., Beineke, A., and Jung, K. Augmentation of transcrip- tomic data for improved classification of patients with respiratory diseases of viral origin. Int. J. Mol. Sci., 23 (5):2481, February 2022.
Fradkin, P., Azadi, P., Suri, K., Wenkel, F., Bashashati, A., Sypetkowski, M., and Beaini, D. How molecules impact cells: Unlocking contrastive phenomolecular retrieval. In NeurlPS, 2024.
Kraus, O., Kenyon- Dean, K., Saberian, S., Fallah, M., McLean, P., Leung, J., Sharma, V., Khan, A., Balakr- ishnan, J., Celik, S., Beaini, D., Sypetkowski, M., Cheng, C. V., Morse, K., Makes, M., Mabey, B., and Earnshaw, Masked autoencoders for microscopy are scalable learners of cellular biology. In CVPR, 2024.
Gupta, S., Hoffman, J., and Malik, J. Cross modal distilla- tion for supervision transfer. CVPR, 2016.
Hager, P., Menten, M. J., and Rueckert, D. Best of Both Worlds: Multimodal Contrastive Learning with Tabular and Imaging Data. In CVPR, pp. 39924-23935, Los Alamitos, CA, USA, June 2023, IEEE Computer Society. doi: 10.1 10.1109/CVPPS55222922228 20233322777999222292 URL / http:///a...eeeeeopprrrrrrsseeee 02291.
Lafarge, M. W. and Koelzer, V. H. Rotation invariance and extensive data augmentation: A strategy for the mitosis domain generalization (midog) challenge. In Biomedical Image Registration, Domain Generalisation and Out-of- Distribution Analysis, pp. 62-67, Cham, 2022. Springer International Publishing. ISBN 778-3-030-97281-3.
han Li, P., Chinchali, S. P., and Topeu, U. Csa: Data- efficient mapping of unimodal features to multimodal features, 2024.
Lan, Y.T.,, Liu, W., and Lu, B.-L. Multimodal emotion recognition using deep generalized canonical correlation analysis with an attention mechanism. In IJCNN, 2020. doi: 10. 111////NNNNNN......... 202.09.....200222222..2
Hinton, G., Vinyals, O., and Dean, J. Distilling the knowl- edge in neural network. In NeurIPS Deep Learning Workshop, 2015.
Lee, P., Kim, T., Shim, M., Wee, D., and Byun, H. Decom- posed cross-modal distillation for rgb-based temporal action detection. In CVPR, 2023.
Huang, T., You, S., Wang, F., Qian, C., and Xu, C. Knowl- edge distillation from stronger teacher. In NeurIPS, 2022.
Li, H., Miao, C., Leung, C., Huang, Y., Huang, Y., Zhang, H., and Wang, Y. Exploring representation- augmen- tation for code search, 2022.
Huo, F. Xu, W., Guo, J., Wang, H., and Guo, S. C2kd: Bridging the modality gap for cross-modal knowledge distillation. In CVPR, 2024.
Li, L. and Zhe, J. Shadow knowledge distillation: Bridging offline and online knowledge transfer. In NeurlPS, 2022.
Jumper. J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Zidek, A., Potapenko, A., Bridgland, A., Meyer, C., Kohl, S. A. A., Ballard, A. J., Cowie, A., Romera-Paredes. B., Nikolov, S., Jain, R., Adler, J., Back, T., Petersen, S., Reiman, D., Clancy, E., Zielinski, M., Steinegger, M.. Pacholska, M., Berghammer, T., Bodenstein, S., Silver, D., Vinyals, O., Senior, A. W., Kavukeuoglu,
Li, R., Wa, J., Li, G., Liu, J., Xuan, J., and Zhu, Q. Mdwgan- gp: data augmentation for gene expression data based on multiple discriminator WGAN-GP. BMC Bioinformatics, 24(1):427, November 2023.
Liu, T., Li, K., Wang, Y., Li, H., and Zhao, H. Evaluating the utilities of foundation models in single-cell data analysis. bioRxiv, September 2023.

---

A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomic Representations
Lopez, R., Regier, J., Cole, M. B., Jordan, M. I., and Yosef, N. Deep generative modeling for single-cell transcrip- tomics. Nature Methods, 15(12):1053-1058 2018.
ISBN 9783031 doi: 10. 1007/978-3--011-6666--1 10. URL http::ttpp/.d...// doi. org/10. 2=g/10.1007/978 3- 0-1-000----0 3--01-1676007_10
Switzerland, 2022.
Lu, S., Fürth, D., and Gillis, J. Integrative analysis methods for spatial transcriptomics. Nat. Methods, 18(11):1282 1283, November 2021.
Saillard, C., Jenatton, R., Llinares-López, F,, Ma- riet, Z., Cahané, D., Durand, E., and Vert, J.-P. H-optimus-O, 2024. URL https:
/ http://....bbboo....eeeeeeeeeeee main/model wwweem/oeeoootiimmimeeeeo
Miao, Z., Humphreys, B. D., McMahon, A. P., and Kim, J. Multi-omics integration in the age of million single-cell data. Nat. Rev. Nephrol., 17(11):710-724, November 2021.
Sanchez, M., Bourriez, N., Bendidi, I., Cohen, E., Svatko, I., Del Nery, E., Tajmouati, H., Bollot, G., Calzone, L., and Genovesio, A. Large scale cell painting guided com- pound selection reveals activity cliffs and functional rela- tionships. May 2025.
Moutakanni, T., Oquab, M., Szafraniec, M., Vakalopoulou, M., and Bojanowski, P. You don't 't need domain-specinn data augmentations when scaling self-supervised learning. In NeurlPS, 2024.
Fanchez-Fernandez, A., Rumetshofer, E., Hochreiter, S., and Klambauer, G. CLOOME: contrastive learning un- locks bioimaging databases for queries with chemical structures. Nat. Commun., 14(1):7339, November 2023.
Nouri, N. Single-cell RNA-seq data augmentation using generative fourier transformer. Commun. Biol., 811):113, January 2025.
Park, W., Kim, D., Lu, Y., and Cho, M. Relational knowl- edge distillation. In CVPR, June 2019.
Sarkar, P. and Etemad, A. Xkd: Cross-modal knowledge dis- tillation with domain alignment for video representation learning. Proceedings of the AAAI Conference on Artifi- cial Intelligence 38. doi: 10. 1609/aaa... v38113.29407.
Pham, C. and Plummer, B. A. Enhancing feature diversity boosts channel-adaptive vision transformers. In NeurlPS, 2024.
Subramanian, A., Tamayo, P., Mootha, V. K., Mukherjee, S., Ebert, B. L.., Gillette, M. A., Paulovich, A., Pomeroy, S. L., Golub, T. R., Lander, E. S., and Mesirov, J. P. Gene set enrichment analysis: A knowledge-baad based approach for interpreting genome-wide expression profiles. Proc. Natl. Acad. Sci. U. S. A., 102(49): October 2005.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, Learning transferable visual models from natural language supervision. In NeurlPS, 2021.
Replogle, J. M., Saunders, R. A., Pogson, A. N., Hussmann, A., Lenail, A., Guna, A., Mascibroda, L., Wagner, G., Iremadze, N., Oberstrass, F., Lipson, D., Bonnar, J. L., Jost, M., Nor- man, T. M., and Weissman, J. S. Mapping information- rich secottpe-phenootee landscapes with genome-scale perturb-seq. Cell, 185(14):2559-255 1111992555..22 2575.e28, July 2022.
Subramanian, A., Narayan, R., Corsello, S. M., Peck, D. D., Natoli, T. E., Lu, X., Gould, J., Davis, J. F., Tubelli, A. A., Asiedu, J. K., Lahr, D. L., Hirschman, J. E., Liu, Z., Donahue, M., Julian, B., Khan, M., Wadden, D., Smith, I. C., Lam, D., Liberzon, A., Toder, C., et al. A next generation connectivity map: L1000 platform and the first 1,000,000 profiles. Cell, 171(6))1477-1452ccm
E. J., Adelman, K., Lithwick-Yannn
2017.
Reymond, J.-L. The chemical space project. Acc. Chem. Res., 48(3):722-730, March 2015.
Theodoris, C. V., Xiao, L., Chopra, A., Chaffin, M. D., Sayed, Z. R., Hill, M. C., Mantineo, H., Brydon, E. M., Zeng, Z., Liu, X. S., and Ellinor, T. Transfer learn- ing enables predictions in network biology. Nature, 618 (7965):616-6)) June 2023.
Roheda, S., Riggan, B. S., Krim, H., and Dai, L. Cross- modality distillation: A case for conditional generative adversarial networks. In ICASSP, 2018. doi: 10.1 ICASSP.2018.84633
Tsai, Y.-H. H., Wu, Y., Salakhutdinov, R., and Morency, L.-P. Self-supervised learning from a multi-view perspec- tive, 2021. URL httht::///// //arxiv./// org/ htgg/bb///at abs/2006. 05576.
Rosen, Y., Roohani, Y., Agarwal, A., Samotorčan, L., Tab- ula Sapiens Consortium, Quake, S. R., and Leskovec, J. Universal cell embeddings: A foundation model for cell biology. bioRxiv, November 2023.
Vorontsov, E., Bozkurt, A., Casson, A., Shaikovski, G., Zelechowski, M., Liu, S., Severson, K., Zimmermann, E., Hall, J., Tenenholtz, N., Fusi, N., Mathieu, P., van Eck, A., Lee, D., Viret, J., Robert, E., Wang, Y. K.,
Ruppli, C., Gori, P., Ardon, R., and Bloch, I. Optimiz- ing Transformations for Contrastive Learning a Dif- ferentiable Framework, pp. 96-105. Springer Nature
12

---

A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomic Representations
Kunz, J. D., Lee, M. C. H., Bernhard, J., Godrich,
Yang, F,, Wang, W., Wang, F., Fang, Y., Tang. D., Huang. J., Lu, H., and Yao, J. scBERT large-scale pretrained deep language model for cell type annotation of single- cell RNA-seq data. Nat. Mach. Intell., 4(10):852-866 September 2022.
R. A., Oakley, G., Millar, E., Hanna, M., Retamero, J., Moye, W. A., Yousfi, R., Kanan, C.. Klimstra, D., Rothrock, B., and Fuchs, T. J. Virchow: A million- slide digital pathology foundation model, 2024. URL. https: /tt./aaxvvvv.///// wwtg/ab////
abs/ 2wt.s/339..00 07778.
Yao, H., Hanslovsky, P., Huetter, J.-C., Hoeckendorf, B., and Richmond, D. Weakly supervised set-consistency learning improves morphological profiling of single-cell images. In CVPR Workshops, 2024.
Wang, Z., Codella, N., Chen, Y.C.,, Zhou, L., Dai, X., Xiao, B., Yang. J., You, H., Chang, K.-W., fu Chang, S., and Yuan, L. Multimodal adaptive distillation for leveraging unimodal encoders for vision- language tasks, 2022. URL https: //tpaiv///vvvv//// wwg/ abs 2204 10496.
Ye, C., Ho, D. Neri, M., Yang, C., Kulkarni, T., Rand- hawa, R., Henault, M., Mostacci, N., Farmer, P., Renner, S., Ihry, R., Mansur, L., Keller, C. G., McAllister, G., Hild, M., Jenkins, J., and Kaykas, A. DRUG-seq for miniaturized tigh-thrrughhut transcriptome profiling in drug discovery. Nat. Commun. 9(1):4307, October 2018.
Watkinson, G., Cohen, E., Bourriez, N., Bendidi, I., Bollot, G., and Genovesio, A. Weakly supervised cross modal learning in high-content screening. In ISBI, 2024. doi: 10.1 00.100//BB5567002240066655
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training, In ICCV, 2023.
Wen, H., Tang, W., Dai, X., Ding, J., Jin, W., Xie, Y., and Tang, J. CellPLM: Pre-training of cell language model beyond single cells. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: //openreview, net/ forum? 2 iddBBXxPDeedd..
Zhou, Y., Badgery, H., Read, M., Bailey, J., and Davey, C. E. Dda: Dimensionality driven augmentation search for contrastive learning in laparoscopic surgery, 2024. URL attps://arxiv. org/
Wenkel, F., Tu, W., Masschelein, C., Shirzad, H., Eastwood, C., Whitfield, S. T., Bendidi, I., Russell, C., Hodgson, L., Mesbahi, Y. E., Ding, J., Fay, M. M., Earnshaw, B., Noutahi, E., and Denton, A. K. Txpert: Leveraging bio- chemical relationships for out-o--distribttion transcrip- tomic perturbation prediction, 2025.
00907.
Zhu, Y. and Wang. Y. Student customized knowledge distil- lation: Bridging the gap between student and teacher. In ICCV, 2021. doi: /WWp//CCVV.92..221.....
Wu, K., Peng, H., Zhou, Z., Xiao, B., Liu, M., Yuan, L., Xuan, H., Valenzuela, M., Chen, X. S., Wang, X., Chao, H., and Hu, H. Tinyclip: Clip distillation via affinity mimicking and weight inheritance. In ICCV, 2023.
Xi, J., Osea, J., Xu, 7.,, and Hartford, J. Propensity score alignment unpaired multimodal data. In NeurlPS, 2024.
Xie, R., Pang, K., Chung, S. W., Perciani, C., Mac- Parland, S., WANG, B., and Bader, G. Spatially re- solved gene expression prediction from histology im- ages via bi-modal contrastive learning. Teiety-sevetth Conference on Neural Information Processing Systems, 2023. URL httph:tt//// ptttps://opeereveee n forum? tMdADo...
Xue, Z., Ren, S., Gao, Z. and Zhao, H. Multimodal knowledge expansion. In ICCV, 2021. doi: 10.1 CCCV4899222021.000083
Yang, C., An, Z., Huang, L., Bi, J., Yu, X., Yang, H., Diao, B., and Xu, Y. Clip-kd: An empirical study clip model distillation, In CVPR, 2024.
13

---

A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomic Representations
A. Detailed Experimental Setup
A.1. Encoders
We use three main models for our experiments, in addition to the MLP trained from scratch. Phenom-1 (Kraus et al., 2024) is a Vision Transformer- based model with 300 million parameters, trained using a Masked Autoencoder (MAE) framework. is pretrained on RPI-93M, dataset of 93 million microscopy images, capturing diverse cellular phenotypes across CRISPR, chemical, and soluble perturbations, making it highly effective for large-scale cellular morphology analysis. scVI (Lopez et al., 2018) is probabilistic generative model designed for single-cell RNA sequencing (scRNA-seq) data, trained using a Variational Autoencoder (VAE) framework. It encodes high-ding nigh-dimensional gene expression data into a biologically meaningful latent space, leveraging a zero-inflated negative binomial (ZINB) reconstruction objective to model overdispersion and dropout effects in transcriptomic data. seGPT (Cui et al., 2024) is a transformerrbased foundation model pretrained on 33 million scRNA-seq samples using a masked language modeling objective. It captures complex gene-gene and gene-cell interactions, with fine-tuning capabilities for tasks like cell type annotation, multi-omic integration, and perturbation response prediction. These models provide robust, biologically relevant representations tailored for microscopy and transcriptomics data.
A.2. Implementation details
The MLP adapters is and fr used in this work are fully connected feedforward networks designed to align embeddings from the transcriptomics (Tx) and microscopy imaging encoders into a shared latent space. For the transcriptomics adapter, the architecture comprises an input layer of size 256, two hidden layers with dimensions 512 and 1024 respectively, and an output layer of size 768. The image adapter follows similar design, with an input size of 768, two hidden layers of size 1024, and an output layer of size 768. ReLU activations are applied to all hidden layers, while the output layer uses a linear activation.
For VICReg, learning rates for the Tx and image adapters were 0.1 and 1 10- respectively, and training spanned 10 epochs with minimum learning rate of 10-10; the VICReg loss parameters (similarity, variance, and covariance weights) were kept at their default settings. Similarly, SigClip used Tx and Img adapters learning rates of 0.1 and 10-$, respectively, with training conducted for 10 epochs and a minimum learning rate of 10 10, the temperature and normalization parameters were kept at their defaults. For DCCA, the Tx and Img adapters were trained with a learning rate of 10-6 and 10-8 respectively over 50 epochs, a minimum learning rate of 10 10 and loss parameters including an output dimension size of 30, usage of all singular values, and an epsilon of 10-6 The SHAKE method utilized Tx and Img adapters learning rates of 0.1 and 10-8, Tx and Img classifier learning rates of 10- 4 and 10-7 respectively, and a temperature of 9, with loss balancing hyperparameters a 10 and B 0.001; training was conducted over 10 epochs with a minimum learning rate of 10 10, For KD, the Tx adapters and classifier learning rates were 0.1 and 10-4, respectively, with temperature of 9 and a = 10, trained for 10 epochs with a minimum learning rate of 10-10 10 Lastly, C2KD employed Tx and Img adapters learning rates of 0.1 and 10 10-6, Tx and Img classifier learning rates of 10-5 and 10-3 respectively, and a temperature of 2 with a Kendall Rank Correlation threshold of 0.3; training spanned 30 epochs with a minimum learning rate of 10-7, At evaluation step, we perform TVN alignment for all output embeddings for the Known Relationship Recall benchmark, and use raw embeddings for Transcriptomic Interpretability Preservation benchmark, is used in (Bendidi et al., 2024b).
B. Evaluation tasks
B.1. Known Biological Relationship Recall
The Known Relationship Recall score is a benchmarking metric introduced in (Celik et al., 2024) and designed to evaluate the extent to which a perturbative map captures established biological relationships. This score serves as a proxy for assessing the biological relevance of the map and its ability to uncover meaningful interactions between genes. By comparing predicted relationships within the map to curated annotations from biological databases, the Known Relationship Recall score provides a quantitative measure of the map's fidelity to known biology.
The computation of the Known Relationship Recall score follows these steps:
1. Pairwise Similarity Computation: For each pair of genes (gi, gj) in the map, we compute the cosine similarity between

---

A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomic Representations
their aggregated embeddings xg; and Xgy- The cosine similarity is defined
(X Xg,) = (Xgo Xay
xxg. |
where (xg. Xg5) is the dot product of the embeddings, and |xx;: is the Euclidean norm of Xg.-
2. Selection of Predicted Relationships: Relationships are classified as "predicted" if their cosine similarity scores fall into the top or bottom relationships according to a percentage threshold (usually 5%) of the distribution of all pairwise similarities. High similarity scores indicate cooperative relationships, while low scores suggest functional opposition.
3. Validation Against Biological Databases: The predicted relationships are validated using established biological annotations from databases such as CORUM, HuMAP, Reactome, SIGNOR, and StringDB. Only gene that appear in the perturbation dataset are considered for pairs in the database.
4. Recall Calculation for Each Database: For each database, the recall is computed as the fraction of annotated relationships that are successfully identified among the predicted relationships:
# (True Positive Relationships)
Recall # #Total Annotated Relationships Map)
Here, true positive relationships are those annotated in the database that also fall within the predicted set. The final Known Relationship Recall score is computed as the mean of the recall values across the five databases.
The Known Relationship Recall score provides a single aggregated metric that encapsulates the map's 's ability recapitulate established biological relationships. A high score indicates strong alignment with existing annotations, demonstrating the map's utility in representing meaningful biological interactions.
B.2. Transcriptomic Interpretability Preservation
The Transcriptomic Interpretabiliiyy Preservation, first introduced as linear interpretability evaluation in (Bendidi et al., 2024b), is an evaluation framework designed to assess how well model captures and preserves biologically meaningful patterns in transcriptomic data. This task evaluates the quality of the model's ' internal representations and their ability to reconstruct gene expression profiles accurately while maintaining the structural relationships between control and perturbation conditions. By focusing on both the accuracy of reconstructed gene expression profiles and the preservation of batch-specific control perturbaottt relationships, this metric provides a holistic view of the model's capability to retain original transcriptomic interpretability. The evaluation relies on two complementary metrics, which are averaged compute the final Transcriptomic Interpretability Preservation score:
Structural Integrity Score: This metric quantifies how well the model preserves the relationships between control and perturbation conditions within each biological batch. The Structural Integrity score is computed as:
Structural Distance
Structural Integrity =
Structural Distancemax
where the Structural Distance measures the Frobenius norm of the difference between centered predicted and actual gene expression matrices, and Structural Distancemax is the theoretical maximum distance, as derived in (Bendidi et al., 2024b). score close to indicates strong preservation of the structural relationships.
Spearman Correlation Reconstruction: This metric evaluates how accurately the model reconstructs original gene expression profiles from its internal latent representations. The Spearman correlation is calculated between the predicted and true gene expression profiles, providing robust measure of rank-based agreement.
To provide comprehensive evaluation, the Transcriptomic Interpretubility Preservation metric is computed as the average of the Structural Integrity score and the Spearman correlation of reconstruction. By evaluating both aspects, the metric ensures that a model not only produces high-quality reconstructions but also retains the underlying biological structure of the data. This crucial for downstream applications such as identifying gene interactions or studying the effects of perturbations various conditions.
2

---

A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomic Representations
C. Batch Correction Techniques
C.1. Centering
Centering involves adjusting the dataset such that each feature has a mean zero. This is achieved by subtracting the mean of each feature from the data. Given feature matrix X I where n the number of samples and is the number of features, the centered matrix X is computed as:
=
Xajz
Vi W
Vj 1, m.
72
This step shifts the data so that each feature's 's mean is zero. In batchecorrected biological datasets, centering is typically applied to remove the influence of negative control embeddings, facilitating the focus on perturbation effects.
C.2. Center Scaling/Standardizzttion
Center wcaling/Stndardizzttonn extends centering by adjusting each feature so that it has unit variance. This ensures comparability across features. For a centered matrix X, the scaled matrix X defined as:
X Jj
n 1
Xi
X2
kyj
72
Vi =
vj
m,
where oj represents the standard deviation of the j-th feature. Center scaling is important for techniques like Principal Component Analysis (PCA), which are influenced by the scale of the data.
C.3. Typical Variation Normalization (TVN)
Typical Variation Normalization (TVN) is a technique designed to enhance the representation of biological data by minimizing batch effects and accentuating subtle phenotypic differences. TVN is particularly relevant in high-content imaging screens and other scenarios with significant batch variability.
TVN begins by computing the principal components of control samples (negative control conditions) to identify the primary directions of variation. PCA is performed on the centered control data Nevntrll to obtain principal components (V1...., Vm with each component representing a variance direction in the data space. The normalization process involves the following steps:
Centering and Scaling Negative Controls: The negative control data Xcontol centered and scaled as:
X Conntrol
X xoontool
Mcontml
Control
where Mcontrol and Goontrol are the mean and standard deviation the control embeddings.
Principal Component Analysis (PCA): PCA is conducted on Xoontoo derive principal components. The matrix
W E Æmxm consists of columns that are the component vectors Vj.
3. TVN Transformation: The transformation matrix I' is constructed to normalize variance along each principal component
axis:
T = W - D 1/2 W where a diagonal matrix of the eigenvalues associated with the principal components.
Application to All Embeddings: The transformation is applied to all embeddings Xall as:
XTvN T Xan-
This step reduces unwanted variation while emphasizing important biological differences, enabling focus on subtle or rare phenotypic features without batch-related artifacts.
D. Additional Results
3

---

A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomic Representations
# of relationship counts
KD
SHAKE
VICReg
Semi-Clipped
Distillation Relationship Gains relative to Tx
59
44
50
36
Distillation Relationship Losses relative to Tx
39
28
30
Transcriptomic Relationship Preservation
22
33
32
56
Total Recalled Relationships
81
77
82
92
Figure 5. Comparison of relationship gains and losses across cross-modal distillation methods shown in Figure 4. Our approach achieves the highest overall relationship recall and best preserves transcriptomic information.
Relationships
Top 2% Bottom 2% Relationships
Relacionships
Relationships
Top 5% Bottom 5% Relationships
51
102
2779
653
123
85
172
3277
452
113
253
611
1081
259
159
350
3842
731
1236
330
441
4032
944
1098
335
355
438
743
805
533
1191
1360
11
1733
35
73
71
69
103
101
166
159
741
1104
1414
1605
2779
157
759
284
1264
3490
1604
3685
564
3607
2090
F12
144
17
29
115
316
45
434
81
908
1167
1272
Transcriptomics
Semi-Clipped
Semi-Clipped Random
Figure Eiterature-nowwe biological relationships retrieved through the LINCS dataset by the transcriptomics and microscopy imaging unimodal encoders, alongside proposed Semi- Cemi-Clipped approach, without data augmentations, and null distribution through randomization of the perturbation labels of the pretrained Semi-Clipped. Semi-Clipped remains consistent with transcriptomics while distilling relationships from microscopy imaging. displaying distinct pattern from the null distribution.

---

A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomic Representations
Top 1% & Bottom 1% Reletionships
Top 2% Bottom 2% Relationships
Top Bottom Relationships
Bottoo 5% Reletionships
3
3
104
43
110
22
10
28
12
18
30
24
20
21
22
10
16
41
40
17
15
28
97
26
102
33
18
42
25
75
17
13
5
2
0
Transcriptomics
Phenomics
Semi-Clipped
Semi-Clipped Random Labels
Figure Literature-knnn biological relationships retrieved by the transcriptomies and microscopy imaging unimodal encoders, alongside proposed Semi-Clipped approach (first row), without data augmentations, across different retrieval thresholds (columns) the HUVEC-KO dataset. Semi-Clipped remains consistent with transcriptomics while distilling new relationships from microscopy imaging. In second and third row, compare Semi-Clipper to null distribution achieved through randomization of the perturbation labels of the pretrained Semi-Clipped. Our approach displays a distinct pattern from the null distribution, and aligns better both modalities than random.

---

A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomic Representations
Enriched Pathways
Source
P-valuc
REACTOME CELL CYCLE CHECKPOINTS
Semi-Clipped \ (Tx Img)
0.0323
KEGG ANTIGEN PROCESSING AND PRESENTATION
(Semi-Cliips) Img) 0.0049
KEGG P53 SIGNALING PATHWAY
ISemi-Clippe Img)
0.0033
KEGG RIG I LIKE RECEPTOR SIGNALING PATHWAY
(Semi (Semi-Clipped Img) Tx
0.0082
REACTOME ADAPTIVE IMMUNE SYSTEM
(Semi-Clipped Img) Tx
0.0114
REACTOME ANTIGEN PRESENTATION FOI DIDING ASSEMBLY Y AND PEPTIDE LOADING OF CLASS I MHC
(Semi- Semii-Cliped ( Img) Tx
0.0049
REACTOME ANTIGEN PROCESSING CROSS PRESENTATION
(Semi- Semi-Cliiped ( Img) Tx
0.0016
REACTOME ASPARAGINE N LINKED GLYCOSYLATION
(Semi Clipped Img)
0.0049
REACTOME CALNEXIN CALRETICULIN CYCLE
(Semi-Clippe maiii-Clippedde Img)
0.0049
REACTOME CELL CYCLE
(Semi- Semmi-Cliped ( Img)
REACTOME CLASS MHC MEDIATED ANTIGEN PROCESSING PRESENTATION
SemiiClipped Img)
0.0049
REACTOME DDXS8 IFTHI MEDIATED INDUCTION OF INTERFERON ALPHA BETA
Semi Clipped Img) Tx
0.0082
REACTOME DEUBIQUITINATION
Semi- Clipped Img) Tx
0.0130
REACTOME GI S DNA DAMAGE CHECKPOINTS
(Smii Semi-Clipped Img) Tx
0.0033
REACTOME G ALPHA Q SIGNALLING EVENTS
Semi Clipped Img) Tx
0.0065
REACTOME HEMOSTASIS
Semi Clipped Img)
0.0082
REACTOME INNATE IMMUNE SYSTEM
Semii Semi-Clipped Img)
0.0227
REACTOME NEGATIVE REGULATORS OF DDX58 IFIHI SIGNALING
Semi Cemi-Clipped Img)
0.0033
REACTOME GLYCAN TRIMMING THE AND CALNEXIN CALRETICULIN CYCLE
Semi Clipped Img)
0.0049
REACTOME OVARIAN TUMOR DOMAIN PROTEASES
Semi- Semi-Clipped Img)
0.0016
REACTOME PLATELET ACTIVATION SIGNALING AND AGGREGATION
Semi- SliipClippeed
Img) Tx
0.0065
REACTOME POST TRANSLATIONAL PROTEIN MODIFICATION
(Semi- SemiiClipped
Img) Tx
0.0002
REACTOME REGULATION OF TP53 ACTIVITY
(Semi Clipped n Img) Tx
0.0179
REACTOME REGULATION OF TP53 ACTIVITY THROUGH METHYLATION
(Semi- Semi-Clipped r Img) Tx
0.0016
REACTOME REGULATION OF TP53 ACTIVITY THROUGH PHOSPHORYLATOO
(Semi- Semi-Cliipedd (1) Img) Tx
0.0179
REACTOME REGULATION OF TP53 EXPRESSION AND DEGRADATION
(Semi- Cemi-Clipped Img)
0.0016
REACTOME RNA POLYMERASH TRANSCRIPTION
(Semi Clipped Img)
REACTOME SIGNALING BY GPCR
(Semi- (Semi-Clipped
Img)
0.0082
REACTOME STABILIZATION OF PS3
(Semi- Dempi-Clipeed Img) 0.0016
REACTOME TRANSCRIPTIONAL REGULATION BY TP53
(Semi-Clipped Img) Tx
0.0195
Table Gene Set Enrichment Analysis (GSEA) results on the HUVEC-KO dataset, highlighting enriched pathways identified uniquely in the Semi-Clipped approach compared to the transcriptomics and microscopy imaging unimodal encoders. The first row represents pathways uniquely enriched Semi-Clipped after excluding the union of transcriptomics and morphological relationships, revealing enrichment in cell cycle pathways. The subsequent rows list pathways enriched in the intersection of Semi-Clipped and microscopy imaging, excluding transcriptomics relationships, which shows that addition to Semi-Clipped unique enriched pathways, our approach is also enriched by morphology specific pathways.
6


Reviews:
[{"review_id": "cvOvt8g2to", "reviewer": "Reviewer_rGLV", "summary": "This paper aims to extract representations of transcriptomics by distilling knowledge from microscopy images. The authors introduce (1) semi-clipped for cross-modal distillation from pretrained foundation models, and (2) perturbation embedding augmentation to generalize transcriptomics data.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer rGLV for their review. We address reviewer’s comments below, in order to improve the clarity of the contribution of our submission : \n\n- __*\"This paper doesn't show empirically that the learnt representation of transcriptomics is relatively comprehensive.\" :*__ We respectfully disagree. In the submitted manuscript, we proposed two different and complementary ways of evaluating empirically whether the learnt representation of transcriptomics through our approach is completely comprehensive : i) Both in Figure 2 and Table 1, we evaluate on the transcriptomics interpretability task, which is a published benchmark for evaluating whether the representation of transcriptomics is comprehensive and includes all information of the original transcriptomics data, through both reconstruction tasks and structural measures of data. ii) In figure 3, we focus on downstream tasks, and show that for discovering target relationships, our distilled transcriptomics representations uncover the same relationships the original transcriptomics data uncovers, while improving it further by uncovering imaging and other new relationships. Additionally, reviewer T8LP describes our evaluation as _“that SemiClipped, which freezes the microscopy imaging encoder, recalls the most relationships overall and from those recalled by the unimodal transcriptomic model.”_. This shows that we have evaluated the comprehensiveness of our transcriptomics representations thoroughly, and have shown that it conserves the original transcriptomics information even after distillation.\n\n- __*“t-SNE for representation distribution” :*__ We add in this anonymous link (https://imgur.com/a/GCi2FYj) a UMAP projection of our learnt features of the distillation model (bottom), compared to the transcriptomics features only (top). We compare both models at batch effect reduction (left, Mixed clusters is better) and perturbation separation (right, separated clusters is better). While both approaches are good at reducing batch effect, the distillation approach is largely better at separating different perturbations. This is also reflected in the NMI metric for clustering the different perturbations with a K-Means (0.128 for the transcriptomics only, 0.481 for the distillation). We had previously hesitated to add UMAP/t-SNE views due to their anecdotal nature. We will however add this figure to the final manuscript for enhanced clarity.\n\n- __*\"Lack of ablation study when teacher representations are not frozen.\" :*__ We respectfully disagree.  In the submitted manuscript, we show in Figure 1 that unfreezing the teacher representations through adding an Image Adapter leads to a drop in performance, likely due to modality drift. We add an additional evaluation of scGPT used with an Image Adapter in our response to reviewer rL4s for more comprehensiveness. \n\n- __*“Proposed reference” :*__ We thank the reviewer for the reference, we will include it in the final manuscript in the related works section (Biologically Relevant Representations).\n\n- __*“The methodology contribution is limited” :*__ We wish to point out that in addition to being the first to prove that distillation from Imaging to Transcriptomics is possible, we have proposed PEA, a completely novel and new approach for data augmentation on omics data that preserves biological information for multimodal learning, tackling one of the main problems of the omics field : Limited paired data and lack of biology preserving data augmentations. We also want to point out that reviewer rL4s described our claim and approach as *“PEA augmentation is interesting and generally improves the performance”*, while reviewer T8LP describes our approach as *“This is a strong paper, with high quality authorship, clear and concise results, and well thought out experiments. The novel augmentation method for transcriptomics data could have impact alone.“* and *“By leveraging these modalities together, the growing power of microscopy models can be leveraged to create strong transcriptomic prediction models.”*. Reviewer DZgQ described our work as *“For the first time, batch correction is restructured as data augmentation, addressing the scarcity of biological multimodal data while ensuring performance improvement with interpretability.”*. This being an application track submission for ICML, where submissions are judged by their real world relevance and the robustness of the claims and experiments, we hope this clarifies the reviewer’s assumptions about our work.", "claims_and_evidence": "The concept of 'semi-clipped' is not essentially different to the vanilla CLIP, but simply with fixed pre-trained microscopy image features using weakly pair samples.", "methods_and_evaluation_criteria": "One significant concern: Microscopic images and transcriptomics data, although they may have shared information, but the feature spaces may not be highly overlapping. In other words, these two modals have complementary information, which cannot be mutually replaced. Forcing the transcriptomics data to extract only the features that are similar to the image data may cause the loss of distinct information of transcriptomics. This paper doesn't show empirically that the learnt representation of transcriptomics is relatively comprehensive.", "theoretical_claims": "No Theoretical contribution is made in this work.", "experimental_designs_or_analyses": "* No t-SNE to show the distribution of learnt features. \n* Lack ablation study when teacher representations are not frozen.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "[1] Spatially Resolved Gene Expression Prediction from Histology Images via Bi-modal Contrastive Learning. This paper may be relevant which uses CLIP for spatial transcriptomics learning.", "other_strengths_and_weaknesses": "Well written and easy to follow, yet the methodology contribution is limited.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PpCb1SWNDI", "reviewer": "Reviewer_rL4s", "summary": "The paper introduces Semi-Clipped, a method that transfers morphological information from microscopy images to transcriptomic data through cross-modal knowledge distillation. The authors adapt the CLIP loss by freezing a pretrained teacher encoder (for images) and learning a trainable adapter for transcriptomics. The authors also introduce PEA (Perturbation Embedding Augmentation), a data augmentation technique based on batch correction methods to introduce biologically plausible variation into the transcriptomic profile. Experiments on multiple out-of-distribution datasets (e.g., HUVEC-KO, LINCS, and SC-RPE1) are presented. The paper claims improved performance and enhanced biological signal relative to both unimodal baselines, existing cross-modal distillation approaches and augmentation techniques.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer rL4s for their review, and for acknowledging the robustness demonstration of our claims. We will respond below to all their comments : \n\n- __*“The reviewer is concerned that Semi-Cipped outperforms Clip in a context dependent way” :*__ We thank the reviewer for pointing this out. To improve the demonstration of our claims, we add to Figure 1 of the final manuscript the performance of scGPT + Tx Adapter + Image Adapter. The revised figure can be found in this anonymized link (https://imgur.com/a/CDmKd3v). We show through this result that Semi-Clipped still outperforms Clip even in the scGPT context. To further illustrate this, we will add Clip as an additional approach in Figure 2.a in the final manuscript, to showcase with more clarity how Semi-Clipped compares to Vanilla Clip and other approaches.  \n\n- __*“There's a potential distribution shift issue since scVI was pretrained on single-cell data, while this study uses arrayed bulk sequencing data” :*__ We thank the reviewer for pointing this out, and apologize for the confusion in the submitted manuscript. We do not have the distribution shift issue, as we pretrain scVI on arrayed bulk sequencing data (HUVEC-CMPD dataset), which is also used for the distillation training. While scVI was originally developed for single cell data, recent published benchmarks (e.g [Bendidi et al. 2024](https://arxiv.org/abs/2410.13956)) have shown that scVI still outperforms all models even when trained on Bulk sequencing data, which motivated our choice. We  have updated the final manuscript to better reflect and clarify this point, given the additional page of main content available.\n\n- __*“The paper adequately covers relevant literature but contains numerous citation errors” :*__ We sincerely thank the reviewer for pointing this out. We have now fixed all errors in all the paper’s citations, this will be reflected in the final manuscript.", "claims_and_evidence": "- The claim that Semi-Clipped achieves improved cross-modal knowledge distillation is convincingly demonstrated through comprehensive experiments and comparisons with multiple competitive baseline methods (including KD, SHAKE, VICReg, and others).\n- PEA augmentation is interesting and generally improves the performance of multiple methods tested, the authors validated through statistically significant improvements.", "methods_and_evaluation_criteria": "- Although Figure 1 has motivated Semi-Clipped over vanilla clip, the paper could benefit from comparing the methods with Clip loss in distillation benchmark in Figure 2(a). The reviewer is concerned that Semi-Cipped outperforms Clip in a context dependent way, as the scGPT + Tx Adaptor + Image Adaptor is omitted in the Figure 1.\n- There's a potential distribution shift issue since scVI was pretrained on single-cell data, while this study uses arrayed bulk sequencing data.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The choice of benchmark metrics is interesting. The experimental designs and analyses appear sound.", "relation_to_broader_scientific_literature": "The paper adequately discusses relevant works in the field.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "The paper adequately covers relevant literature but contains numerous citation errors,  the reviewer did not do an exhaustive check but the following are obviously wrong:\n\n- Geneformer is incorrectly cited as \"Chen, T. K., Wang, Z., Li, X., Li, Y., and Huang, K. Gene-former: A foundation model for generalizable gene ex-pression learning. bioRxiv, pp. 2023.01.14.524028, 2023.\n  - instead of \"Theodoris, Christina V., Ling Xiao, Anant Chopra, Mark D. Chaffin, Zeina R. Al Sayed, Matthew C. Hill, Helene Mantineo et al. \"Transfer learning enables predictions in network biology.\" *Nature* 618, no. 7965 (2023): 616-624.\"\n- scGPT is wrongly cited as: \"Wang, Z., Song, B., Zhu, T., Li, B., Hu, Q., Tao, X., Chen, F., Wang, L., and Xie, P. scgpt: Transformer-based single-cell rna-seq data analysis. bioRxiv, pp.2023.02.24.529891, 2023.\"\n  - which should be \"Cui, Haotian, Chloe Wang, Hassaan Maan, Kuan Pang, Fengning Luo, Nan Duan, and Bo Wang. \"scGPT: toward building a foundation model for single-cell multi-omics using generative AI.\" Nature Methods 21, no. 8 (2024): 1470-1480.\"\n- Similar errors appear for scBERT, Drug-seq, and others", "questions_for_authors": "Can you please share your thoughts on previous sections?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bbb6ayjlfJ", "reviewer": "Reviewer_T8LP", "summary": "Understanding how cells respond to stimuli such as genetic perturbations or chemical compounds forms a crucial part of drug discovery. This work proposes a method to enrich representations of transcriptomic data with paired morphological data. Measuring paired transcriptomic and morphological features of cells is complex, and hence datasets of this type are rare, even when considering datasets in which these measurements are only weakly paired by sharing the same metadata attribute, such as the applied perturbagen. This necessitates a method that enriches transcriptomic representations with morphological features using only a limited number of paired measurements. This work proposes using a CLIP style loss, applied to the morphological representations from a frozen pertained model, and transcriptomic representations from a pretrained transcriptomic model with an additional trainable adapter. In doing so, this work demonstrably shows that this method, SemiClipped, allows morphological features to enrich the transcriptomic representations, and that this particular method outperforms other cross-modal distillation methods. Crucially, this allows this model to retrieve a greater number of known biological relationships from transcriptomic data, whilst also maintaining a high-level of interpretability at the level of genes. \n\nIn addition to proposing SemiClipped this work also proposes a novel augmentation method for transcriptomic data. Transformations which apply batch correction, in which samples are aligned to unperturbed control measurements, are applied to transcriptomic data to yield augmented samples, and simulate greater variance in the changes in experimental conditions under which transcriptomic data may be measured. This augmentation is shown to improve performance in biological retrieval tasks and maintains strong interpretability, a weakness of more common augmentation methods.\n\n## Update after rebuttal \nAfter reading all reviews and rebuttals, I have decided to leave my score unchanged. I welcome the additional figures and tables the authors have shared during rebuttal. To raise my score from a 4 to a 5, my assessment of the overall significance of the paper would have to be changed such that I felt this paper will have major impact on the community. This is more a question of the scope of the paper, and hence has not changed during rebuttal.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer T8LP for all their comments, and for their acknowledgement of the quality of the paper and the contribution. We aim to acknowledge and answer below the questions and suggestions of the reviewer : \n\n- __*“Additional reference” :*__ We thank the reviewer for pointing us toward this publication. As tabular data shares many common points with transcriptomics data, this merits an additional discussion on this paper in the final manuscript.\n\n- __*“there a few cases where figures are not well labelled” :*__ We agree with the reviewer, and while this was a limitation of the 8-pages format,, we will take advantage of the extra space in the final manuscript to ensure that all labels and captions are self contained and self explanatory.\n\n- __*“Add y-labels to Figure 1, and Figure 2.” :*__ We add y-labels to both figures in the final manuscript. \n\n- __*“Figure 3 took some time to parse, could this data be more clearly represented in a tabular format?” :*__ We thank the reviewer for their suggestion, this could clarify further the impact of our method. We have added to the final manuscript a new table that quantifies for each approach the amount of gain and loss of relationships through the distillation compared to other approaches. The added table is available in this anonymized link (https://imgur.com/a/qZB7q9l). We see that Semi-Clipped recalls the highest amount of known relationships, preserves the original transcriptomic information the best, and minimizes the loss of relationships the most.\n\n- __*“Can you foresee the impact of PEA beyond transcriptomics data used in a preclinical setting, where datasets do not typically have a well defined notion of unperturbed controls?” :*__ PEA’s augmentation principle is especially promising for biological data in batches with control, such as proteomics and metabolomics, where batch correction techniques are well established to mitigate technical variability. Although its impact may be more limited in contexts lacking a clear notion of unperturbed controls or where robust domain adaptation methods are already well established, PEA can still broaden the scope of existing solutions. PEA’s strategy of leveraging small and stochastic technical variations could extend and be adapted to specific post-processing methodologies  in areas such as signal processing (ECG, wearable devices) or astronomy. We will add a discussion on this question to the conclusion of the final manuscript.", "claims_and_evidence": "There are three key claims made in the article, namely:\n1. that SemiClipped, the proposed method of cross-modal distillation, provides SOTA performance in the data limited regime\n2.  that by freezing the encoder of the teacher modality (in this case morphological features) they prevent drift from student to teacher. \n3. that they devise a novel biologically inspired data augmentation, PEA, that is capable of improving cross-modal distillation, and outperforms existing augmentations on benchmarks related to uncovering known biological relationships, and transcriptomic interpretability. \n\nEach of these claims are well supported by evidence. Three datasets were used for OOD evaluations, reflecting generalisation to cell types, experimental conditions and gene expression quantification technologies. Each of these datasets represents a distribution-shift that is expected between training and inference in production, hence providing a realistic evaluation of performance. The claim that SemiClipped provides SOTA performance is demonstrated throughout the paper, as it is shown to perform better than a number of strong baseline models from the literature. By comparing, in Figure 6, the biological relationships that are recalled by a unimodal pretrained transcriptomic model with those recalled by a transcriptomic model trained via cross-modal distillation, the authors demonstrate clearly that i) cross-modal distillation can allow transcriptomic models to recall biological relationships typically found in microscopy representations ii) that current methods recall more relationships than unimodal models, but can lose some relationships captured before distillation and iii) that SemiClipped, which freezes the microscopy imaging encoder, recalls the most relationships overall and from those recalled by the unimodal transcriptomic model. \n\nIn combination with Figure 1, which shows that including a trainable adapter for both the imaging and transcriptomic models leads to poorer biological recall than using a frozen imaging module and trainable adapter for transcriptomics only, this supports the claim that SemiClipped provides SOTA performance and prevents drift from student to teacher. \n\nFurthermore, Figure 2 demonstrates clearly that PEA i) provides consistent superior performance in the recall of biological relationships, and ii) in the transcriptomic interpretability of the representations for a number of benchmark augmentations in isolation and combination. The evaluation metrics for the biological recall task and transcriptomic interpretability are well explained in the Appendices. Including both of these evaluations provides a clear insight into how cross-modal distillation effects performance and interpretability.", "methods_and_evaluation_criteria": "As mentioned above, the work focuses on two tasks, recall of known biological relationships and transcriptomic interpretability. The biological recall task is motivated by the problem at hand - in drug discovery a model that can predict changes in biological relationships from transcriptomic data is crucial for the automation of assessing the impact of the many compounds that exist in the space of all possible small molecules. This work focusses on cross-modal distillation with microscopy imaging, which provides rich insight into cell state, but lacks gene level interpretability. This motivates the transcriptomic interpretability task. \n\nAdditionally, by including three benchmark datasets that simulate real changes that one would expect between model training and deployment (changes in cell types, experimental conditions and gene expression quantification technologies) the author’s provide a realistic measure of model performance.", "theoretical_claims": "This is not applicable for this work.", "experimental_designs_or_analyses": "I specifically checked the experiment for which results are shown in Figure 2, since these results support the key claims of this work. By using the three OOD evaluation datasets there were no concerns of data leakage between training and inference, and made comparisons between the proposed model and baseline models fair. I therefore see no issues with the design of the experiments used to form this figure and have confidence in these results.", "relation_to_broader_scientific_literature": "It is well known that microscopy imaging can be combined with deep learning to extract useful features that describe cell phenotype, for example see [1]. However, while these models can be used to infer relationships between genes via comparing embeddings of cells perturbed by different gene KO, these models to not provide a direct transcriptomic interpretability, in the same way that a model utilising transcriptomic data would. By leveraging these modalities together, the growing power of microscopy models can be leveraged to create strong transcriptomic prediction models. \n\n\n[1] Kraus, O., et al Masked autoencoders for microscopy are scalable learners of cellular biology. In CVPR, 2024.", "essential_references_not_discussed": "I think it is worth mentioning [1] for demonstrating that encoders trained in SSL fashion with multi-modal data can outperform unimodal encoders in the limited data regime, with imaging and tabular data (which is somewhat related to transcriptomic data).\n\n\n[1] Hager, P., Menten, M.J. and Rueckert, D., 2023. Best of both worlds: Multimodal contrastive learning with tabular and imaging data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 23924-23935).", "other_strengths_and_weaknesses": "This is a strong paper, with high quality authorship, clear and concise results, and well thought out experiments. The novel augmentation method for transcriptomics data could have impact alone, and it would be interesting to see how this could be adapted outside of a preclinical setting. \n\nThe main weakness of this work is the presentation of some of the results, there a few cases where figures are not well labelled. This is overcome by the clarity of the main text, but is a point that could be improved on.", "other_comments_or_suggestions": "- Add y-labels to Figure 1, and Figure 2. \n- Figure 3 took some time to parse, could this data be more clearly represented in a tabular format? I suppose the quantity of interest to highlight is the number of additional relationships gained via cross-modal distillation, and how many have been lost, and focussing on just these two may be enough to demonstrate the success of the method in a more immediate manner.", "questions_for_authors": "- Can you foresee the impact of PEA beyond transcriptomics data used in a preclinical setting, where datasets do not typically have a well defined notion of unperturbed controls?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P6XYKrt26t", "reviewer": "Reviewer_DZgQ", "summary": "This paper proposes a cross-modal knowledge distillation framework (Semi-Clipped) and a biologically inspired data augmentation method (PEA). The aim is to enhance the biological significance and predictive power of transcriptomic representations using weakly paired multimodal data (microscopy images + transcriptomics). By freezing the pre-trained morphological feature encoder and training a lightweight adapter, unidirectional knowledge transfer from morphological features to transcriptomics is achieved. Meanwhile, PEA enhances data diversity through randomized batch correction techniques, preserving biological information.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer DZgQ for their review, and for acknowledging the robustness of our experimental design, and the innovation behind our approach. We respond below to the comments of the reviewer : \n\n- __*“The visualization is not clear, and the y-axis of Figure 1 is not labeled.” :*__ We thank the reviewer for their feedback. For the final manuscript, we will ensure that the visualizations are clear and more zoomed in, and we will add the label to the y-axis of Figure 1 (Recall of known relationships).\n\n- __*“The specific definition of weakly paired data is not clearly provided.” :*__ We thank the reviewer for pointing this out. While we have provided  short definitions for weakly paired data (L32, L138, L214), we agree that this work would benefit from a clear and self contained definition of weakly paired data (pairs of samples from two modalities that are not paired at the sample level, but share the same labels and conditions, in our case the common condition being the same cell type and same perturbation, even if the cell samples are different in other factors). We will provide a clearly defined definition of weakly paired data in the final manuscript at the Introduction section, as well as in the Experimental setup section (section 4).\n\n- __*“Biological fidelity of PEA: How can it be proven that randomized batch correction does not disrupt key biological signals?.” :*__ We have already proven and established this in the submitted manuscript, as we proposed two different and complementary ways of evaluating empirically whether the learnt representation using the augmentations is biologically : i) Both in Figure 2.b and Table 1, we evaluate PEA on the transcriptomics interpretability task, which is a published benchmark for predicting gene expression counts of real unseen perturbations, as the reviewer has proposed, through both reconstruction tasks and conservation of structural integrity measures of the data. ii) In figure 3, we focus on downstream tasks, and show that for discovering target relationships, our distilled transcriptomics representations uncover the same relationships the original transcriptomics data uncovers, while improving it further by uncovering imaging and other new relationships. This shows that we have evaluated the comprehensiveness of our transcriptomics representations thoroughly, and have shown that it conserves the original transcriptomics information even after distillation. \n\n- __*“Hyperparameter experiments for adapter temperature, learning rate, batch size, etc., are not conducted.” :*__ We thank the reviewer for pointing this out. We have added in the final manuscript an overview of the hyperparameter experiments and their results (Results section), which we also show in this anonymized link : https://imgur.com/a/0Oekn3T.\n\n- __*“Computational efficiency: What is the training time for Semi-Clipped on a dataset with 1.3 million samples? This information affects the evaluation of the method's practicality.” :*__ We thank the reviewer for pointing this out. While there is no publicly available weakly paired (transcriptomics-imaging) dataset of more than 300k samples, if we stack our existing datasets on themselves to reach the size of 1.3 million samples, Semi-Clipped then requires a 19 hour training on one single H100 GPU. Our distillation method requires extremely minimal computing resources, as it makes use of trainable adapters and frozen backbones, while the additional PEA data augmentation only multiplies the distillation training time by x1.3. We will update our final manuscript with detailed benchmarking of the time and computing resources of our approach. \n\n- __*“Negative results disclosure: Have other modality combinations (e.g., proteomics + transcriptomics) been attempted?” :*__ We thank the reviewer for proposing this idea. While other modalities can potentially add interesting information, these different modalities (e.g proteomics) suffer from even lower availability of perturbational data than transcriptomics, and even less than that in the case of weakly paired data; thus, no, other combinations have not been attempted.", "claims_and_evidence": "\"PEA preserves biological information\": There is a lack of direct evidence for the biological validity of the augmented data (e.g., comparison with real perturbations).", "methods_and_evaluation_criteria": "Semi-Clipped avoids modality drift by freezing the teacher model, which is a reasonable design; PEA transforms batch correction into an augmentation strategy, demonstrating strong innovation.", "theoretical_claims": "There is no theoretical proof that the optimization objective of modality alignment maximizes cross-modal information transfer.", "experimental_designs_or_analyses": "Reasonable design:\n1. 15 random seed tests to improve statistical significance\n2. Controlled variable studies.\n\nProblems:\\\nLack of hyperparameter experiments (learning rate, batch size, temperature, etc.)", "relation_to_broader_scientific_literature": "1. Semi-Clipped Framework and Cross-Modal Distillation Research\n\n① Adaptation of CLIP: Semi-Clipped is based on the CLIP framework but solves the problem of requiring a large amount of paired data in traditional CLIP by freezing the pre-trained encoder of the teacher modality (microscopy images). This contrasts with methods like XKD and C2KD, which require online adjustment of dual-modal encoders and are prone to modality drift with weakly paired data.\n② Advantage of Unsupervised Alignment: Compared to distillation methods like KD and SHAKE that rely on label supervision, Semi-Clipped achieves unsupervised alignment through CLIP loss, improving biological relationship recall by 23% on the HUVEC-KO dataset, validating the limitations of biological labels.\n③ Breakthrough in Single-Modality Inference: Unlike multimodal fusion methods such as VICReg and DCCA, this framework allows single-modality (transcriptomics) inference, inheriting the predictive power of the microscopy modality while maintaining the interpretability of transcriptomics.\n\n2. PEA Data Augmentation and Bioinformatics Methods\n\n① Creative Transformation of Batch Correction: Traditional batch correction techniques like TVN are redefined as random augmentation operations. Compared to conventional image augmentations (rotation, scaling), PEA introduces controlled variations while preserving biological signals, improving performance on the LINCS dataset by 69%.\n② Innovation in Biological Data Augmentation: Unlike general augmentations such as scVI denoising and MDWGAN-GP, PEA achieves a Spearman correlation of 37.56 on the single-cell dataset SC-RPE1 by randomly controlling sample sampling and reweighting PCA variance, demonstrating its adaptability to complex biological noise.\n\n3. Expansion of Multimodal Learning Paradigms\n\n① Basic Model Adaptation: Adopting the pre-trained adapter approach of Fradkin et al., but avoiding the error accumulation of bidirectional distillation through unidirectional knowledge binding, improving the biological relationship recall of the scGPT adapter by 40%.\n② Balance of Interpretability: Under the transcriptomics interpretability evaluation framework proposed by Bendidi et al., this is the first to achieve dual optimization of structural integrity (93.15) and biological discovery (39.84 recall) in cross-modal distillation, overcoming the information loss dilemma of traditional distillation methods.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Advantages:\\\nFor the first time, batch correction is restructured as data augmentation, addressing the scarcity of biological multimodal data while ensuring performance improvement with interpretability.\n\nDisadvantages:\n1. The specific definition of weakly paired data is not clearly provided. \n2. Computational costs (time complexity, space complexity) are not discussed.\n3. Hyperparameter experiments for adapter temperature, learning rate, batch size, etc., are not conducted.", "other_comments_or_suggestions": "The visualization is not clear, and the y-axis of Figure 1 is not labeled.", "questions_for_authors": "1,Biological fidelity of PEA: How can it be proven that randomized batch correction does not disrupt key biological signals? Providing an analysis of TF activity changes could strengthen the conclusion.\\\n2,Computational efficiency: What is the training time for Semi-Clipped on a dataset with 1.3 million samples? This information affects the evaluation of the method's practicality.\\\n3,Negative results disclosure: Have other modality combinations (e.g., proteomics + transcriptomics) been attempted? Discussing failed cases could enhance the rigor of the method.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper discovers that transcriptomics and microscopy images are complementary modalities, and proposes a methodology to learn representations of transcriptomics by distilling knowledge from microscopy images. The reviewers had concerns regarding limited novelty of semi-clipped, missing ablation and visualization, no biological grounding for PEA and computational cost. The authors addressed most of the concerns, and the strengths of the paper outweighs the drawbacks.
Decision: Accept (poster)