PETRI: Learning Unified Cell Embeddings from Unpaired Modalities via Early-Fusion Joint Reconstruction
Authors: Ryan W Conrad
Ethan Weinberger
Saradha Venkatachalapathy
Yuwen Chen
Darshini Shah
Bay Johnson
Max R Salick
Vaishaali Natarajan
Emily Fox
Venue: ICLR 2026 Poster (2026)
Keywords: high-content screening
cell biology
single cell
transcriptomics
microscopy
multimodal
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: Integrating imaging and transcriptomics screening data holds promise for isolating true biological signals from modality-specific technical artifacts. However, existing multimodal embedding approaches either require pairing or fail to capture both shared and modality-specific information in an end-to-end manner. We present PETRI, an early-fusion transformer that learns a unified cell embedding from unpaired cellular images and gene expression profiles. PETRI groups cells by shared experimental context into multimodal “documents” and performs masked joint reconstruction with cross-modal attention, permitting information sharing while preserving modality-specific capacity. The resulting latent space supports construction of perturbation-level profiles by simple averaging across modalities. Applying sparse autoencoders to the embeddings reveals learned concepts that are biologically meaningful, multimodal, and retain perturbation-specific effects. To support further machine learning research, we release a blinded, matched optical pooled screen (OPS) and Perturb-seq dataset in HepG2 cells.

Published as a conference paper at ICLR 2026
PETRI: LEARNING UNIFIED CELL EMBEDDINGS FROM UNPAIRED MODALITIES VIA EARLY-FUSION JOINT RECONSTRUCTION
Ryan Conrad, Ethan Weinberger, Saradha Venkatachalapathy, Owen Chen Darshini Shah, Bay Johnson, Max R Salick, Vaishaali Natarajan & Emily B. Fox insitro
South San Francisco, CA 94080, USA
{rconrad, ethan. we rennrrrrr,, saradhavpathy, owen, da rshini, bay, max, vaishaali, wew.aiistr.myllliiisto..... com
ABSTRACT
Integrating imaging and transcriptomics screening data holds promise for isolating true biological signals from modality-speeciff technical artifacts. However, exist- ing multimodal embedding approaches either require pairing or fail to capture both shared and modality
information in an end-to-end manner. We present PETRI, early-fusion transformer that learns unified cell embedding from unpaired cellular images and gene expression profiles. PETRI groups cells by shared experimental context into multimodal "documents" and performs masked joint reconstruction with cross-modal attention, permitting information sharing while preserving modality-specifff capacity. The resulting latent space supports construction of perturbation-level profiles by simple averaging across modalities. Applying sparse autoencoders to embeddings reveals learned concepts that are biologically meaningful, multimodal, and retain perturbation- support further machine learning research, we release a blinded, matched optical pooled screen (OPS) and Perturb-seq dataset in HepG2 cells.
effects. To
INTRODUCTION
major goal cell biology to establish causal links between gene activity and cellular states (Rood et al. 2024). High-throughput perturbation technologies now profile complementary facets of these states at scale. Perturb-seq (Dixit et al.) 2016) combines CRISPR-
perturbations with effects, while Optical Pooled Screening (OPS) (Feldman et al.) 2019: Sivanandan et al., 2023 Ramezani et al. 2025) uses cost-effective flu- orescence microscopy capture morphological phenotypes. Together, transcriptomies and imaging provide complementary views of how perturbations reshape cellular state.
single-cell RNA sequencing read out transcriptome-wir
The growing availability of large Perturb-seq and OPS datasets motivates methods for multimodal representation learning that capture their shared and modality-ppecific information. Shared struc- ture can help disentangle true biological signal from technical confounders- such as random gene dropout in RNA-seq seq or intensity fluctuations in microscopy thatt are unlikely to be supported across modalities (Radhakrishnan et al., 2023). At the same time, modality-specific signals contain unique phenotypes and mechanistic clues that, when combined, yield more complete picture of perturba- tion effects.
However, integrating information across these modalities is challenging, First, obtaining data paired at the cell-level is not possible as state-of-the-art assays are destructive and cannot profile both morphology and gene expression from the same cells. Second, morphology and gene expression only partially overlap in the biology that they capture (Way et al. 2022). so models must remain robust when signals disjoint.
To address these challenges, we present PETRI, an carly-fusion self-supervised transformer that learss single-cell embeddings from unpaired images and transcriptomes. PETRI groups cells by

---

Published as a conference paper at ICLR 2026
Zomics
Zimage
Recon.
Recon. Loss
Loss
No mutual
High mutual information
information
Figure 1: Conceptual overview: Joint reconstruction of masked regions promotes multimodal inte- gration. Left: With disjoint modalities, loss matches unimodal decoders. Right: With high mutual information, cross-modal predictions reduce reconstruction loss.
perturbation into multimodal "documents" and employs self-attention propagate useful features across cells and between modalities.
We show that PETRI learns effective multimodal cellular representations. Perturbation-level em- beddings, formed by averaging PETRI cell embeddings across modalities, recover known gene-gene relationships. Ablation analysis and an investigation of modaaltyyyyty attention patterns shows that PETRI finds subtle multimodal features that decrease reconstruction loss. Finally, we use sparse autoencoders isolate these multimodal concepts and show that they are robust to batch effects and correspond to perturbation-specife morphological and molecular concepts.
To catalyze further research, we release blinded, matched OPS and Perturb-seq dataset in HepG2 cells spanning 569 CRISPR knockouts in four chemical backgrounds.
2 RELATED WORK
Representation learning for cellular images Image-based profiling aims to convert the rich in- formation microscopy images into quantitative feature vectors for downstream analysis in drug discovery and functional genomics (Chandrasekaran et al. [001)) Early deep learning approaches used weak supervision, training models to predict the experimental treatment (e.g.. drug or genetic perturbation) applied to a cell or group of cells (Caicedo et al.) 2018). However, this approach is limited by its core assumption that all perturbations produce morphological change, which is often not the case. More recent work has shifted to self-supervised learning. These methods, including vision transformer-bat art performance at inferring known biological relationships from images without relying experi- mental labels for training (Doron et al. 2023 Kraus et al,, 2024 Pham et al., 2025). Perturbation- aggregated embeddings also used for predicting a compound' mechanism of action, identifying
self-distillation and masked autoencoders, have demonstrated state-of-the-
disease-specific phenotypes, and functional gene annotation (Sivanandan et al. 2023
Representation learning for single cell transcriptomies Generative models have become cor- nerstone of single-cell transcriptomics analysis, with pioneering methods like scVI using variational autoencoders learn a probabilistic latent space that corrects for technical noise and batch effects (Lopez et al. 2018). This paradigm has evolved with the advent of foundation models for biology, which leverage the transformer architecture and pre-trainee pre-training on massive-scule datasets of tens of millions of cells (Cui al.) 2024 Theodoris et al. 2023; Gong et al. [023 Pearce et al. 2025 Kal- fon et al. 2025). Common tasks for these models include automated cell type annotation, integration of datasets from different experiments technologies, and the prediction of cellular responses to genetic or chemical perturbations.
Multimodal single cell embeddings Integrating the many modalities that be measured from a single cell is a key challenge modern biology, as highlighted by community widd efforts such the NeurlPS 2021 Multimodal Single-Cell Data Integration Challenge (Lance al. 2022 For
2

---

Published as a conference paper at ICLR 2026
datasets with modalities paired at the cell level, methods focus learning joint representation. For instance, deep generative models like MultiVI learn a probabilistic embedding of paired multi- omic data (Ashuach et al., 2023), while contrastive frameworks like scCLIP align paired chromatin accessibility and gene expression profiles (Xiong et al., 2023). This principle also extends to linking imaging with molecular data, where models like OmiCI.IP learn to associate histopathology images with their corresponding spatial transcriptomics profiles (Chen et al. 2025b).
However, collecting paired image and transcriptomics data is often expensive or technically infeasi- ble, making methods that can integrate unpaired data crucial. To address this, some approaches aim align individual cells across modalities; for example, propensity score alignment leverages shared perturbation labels to estimate a matching between cells in different datasets (Xi & Hartford (2024).
Yang et al. (202T) introduce cross- modal autoencoders which use a two-stage learning process, first fitting variational autoencoder (VAE) to images and then training a VAE for gene expression with regularization loss that forces the latent spaces to overlap. Other methods operate at the level of cell populations using weaker supervisory signals. CelICLIP learns shared embedding space between textual descriptions of perturbations and the sets of cell images resulting from them (Lu et al. 2025). Similarly, MultiMIL uses sample-level labels, such as patient disease status, in multiple-instane learning (MIL) framework to identify the specific cells in different modalities that are characteristic of that label (Litinetskaya al. 2024
Vision-language models for multimodal representation learning Recent advances in VLMs have established powerful architectures for multimodal learning. Some models utilize resampling and cross-attention mechanisms to fuse information from interleaved image and text data (Alayrac) al. 2022). while others, like LLaVA, project image features into the word embedding space and process a unified sequence with a standard self-attention mechanism (Liu et al. 2023). Although most often used for visual question answering. the VLM framework can be adapted for representa-
tion learning. For example, MoCa is trained to denoise and reconstruct both image and text inputs simultaneously. enabling it to learn effective bidirectional multimodal embeddings from large, unla- beled datasets (Chen et al. 2025a).
Matched D imaging and gene expression datasets The LINCS dataset (Way et al. 2022 includes a library of 1,327 chemical perturbations with Cell Painting and L1000 readouts. L1000 measures 978 mRNA transcripts from bulk samples, though the authors of the dataset report that it suffers from poor reproducibility of perturbation effects. Borturb-FISH (Binan et al. 2025) has matched Perturb- seq and OPS with MERFISH (Chen et af.] 2015) for 35 genetic perturbations. MERFISH provides single cell pairing of morphology and mRNA counts but is limited to few hundred genes. Relatedly, Perturb-Multi (Saunders et al.) 2025) is a unique spatial transcriptomics dataset that genetically perturbs mouse liver cells in vivo and records MERFISH and protein staining.
METHOD
For multimodal representation learning, CLIP (Radford et al. 2021) may seem like natural starting point. But, contrastive methods like CLIP are ill-suited to our scenario since their effectiveness hinges strong positive pairs distinguishable from a large number of negatives. Our dataset has limited unique treatments (about 2,200) and the modalities have no explicitly overlapping features.
This motivates a shift in perspective. Instead of treating each modality instance distinct item in a pair, we draw inspiration from VLMs that operate on mixed-mooaalit documents. A document, like webpage for example, contains data that is merely aligned by a common topic the con- text increases the odds of finding cross-modality associations. PETRI conceives of perturbations as topics and builds documents from sets of cells. Our central hypothesis is that cellular phenotypes visible in both modalities, and enriched under certain perturbations, will supply mutual information that improves the reconstruction of corrupted data (Fig. D. If the modalities have no mutual infor- mation or even contradictory information, we expect cross- modslityy attention to decrease and for reconstruction loss and representation quality to default to the level of unimodal models.
The document approach is appealing, but introduces significant technical hurdle: an exploding sequence length. single cell can be represented by hundreds of image patch tokens or thousands
Here matched refers to both datasets having the same set of perturbations.
3

---

Published as a conference paper at ICLR 2026
SgRNA1
sgRNA2
SgRNA3
sgRNA4
sgRNAS
TrP
ViT
Cell document
Senn Done
sgRNA1
SgRNA2
sgRNA3
sgRNA4
sgRNA5
Multimodal Set Transformer
d
Image Patches Latents
Gene Tokens
Latents
ViT
TrP
Count Decoder
Image Decoder
Mask Tokens
Counts Decoder
Image Decoder
The PETRI architecture. Overview of separate encoders, cell document creation, multimodal set transformer, and separate decoders. b, Data grouping and sampling by modality and perturbation. c, ViT-based masked autoencoder for images with token resampling. d, Perceiver- based masked autoencoder for transcriptomics with token resampling.
Figure 2:
of gene tokens. A set of cells would produce sequence far too long for standard transformers. Our solution is aggressive token resampling. By distilling the token representation of each cell into a small, fixed number of latent tokens, can flexibly scale the number of cells per document.
Fig. a outlines PETRI's architecture, which consists of four steps, each detailed Sec. Sec.3.1 3.4
1. Create batches where cells are grouped by perturbation to form sets (Fig.
2. Convert raw cell data (images or transcriptomes) into tokens, mask a large portion of them, and use poddllty-sppeciffe encoder to resample the unmasked tokens into small, fixed- size sequence of latents (Fig, -d).).
3. Concatenate latent tokens from cells same sets to form multimodal cell documents and process them with multimodal set transformer.
4. Split the document back into individual cell latent tokens and use modality-specinf de- coders reconstruct the original masked input from the latent representations.
3.1 BATCH CONSTRUCTION
For training, we stratify datasets into groups based on the experimental context; here we primarily focus on perturbation-level grouping e.g., unique genetic treatment like the guide RNA (sgRNA), or combination of perturbation and chemical background. From each group, we sample set of S cells with replacement for each modality. Multiple sets are collated into mini-batch and dispatched to the appropriate encoder. This process is depicted in Fig. bb.
3.2 TOKENIZATION AND PER-CELL RESAMPLING
To construct the multimodal documents, each cell must first be encoded into a compact latent repre- sentation. This is achieved with modality-speciff encoders designed with token resampling. During training, we randomly mask 75% of input tokens (image patches or genes) per cell and remove them from the sequence.

---

compute cosine Published as a conference paper at ICLR 2026
For images, follow the standard Vision Transformer (ViT) approach of embedding non- overlapping patches with position encodings. To achieve resampling, we concatenate a fixed number of learnable latent tokens, L, to the sequence of image patch tokens, N, where L < N. This com- bined sequence is processed by transformer blocks, and only the I. output latent tokens are retained as the cell's image representation (Fig.
For transcriptomics, the input sequence length, corresponding to thousands of genes, makes a stan- dard transformer architecture computationally infeasible. We therefore required architecture purpose-built for efficiently processing and resampling extremely long sequences. We adopt the Perceiver (Jaegle et al., 2021), which interleaves cross-attention layers for resampling with self- attention layers over the latent tokens only (Fig. 2d... This design directly serves our need for ag- gressive token resampling, making a natural architectural choice. Gene expression is tokenized by combining learned gene embedding with its measured log count via two-layer MLP. Similar to the image case, we retain I, latent tokens as a cell's transcriptomic representation.
3.3 MUL.TIMODAL SET TRANSFORMER (MST)
encoders output fixed number of latent tokens per cell, resulting in a tensor of shape (G x S, I., D), where G is the number of groups (e.g.. perturbations and/or conditions) in the batch, S is the set size (i.e., number of cells), L is the number of latent tokens, and D their dimensionality.
The modality-specte
To form the cell documents, we reshape this tensor to (G, x L, D) and concatenate the represen- tations from both modalities along the token dimension to form batch of unified sequences with shape (G,2 2 x S x L, D). These sequences are then processed through series of standard trans- former blocks, allowing for cross-modal and cross-cell attention. Afterwards, we split the sequence by modality and reconstitute the original (G x S, L, D) shapes for decoding.
3.4 DECODERS AND Loss
The final step is to reconstruct the original inputs from the processed latent tokens, enforcing that the latents capture comprehensive information about each cell.
The image decoder is adapted from Masked Autoencoders (MAEs) (He et al. 2022). Since our latent tokens are not tied to specific patch locations, concatenate them with full sequence of N learnable mask tokens. The decoder is trained to reconstruct the original masked image patches from this combined sequence. The loss is the mean squared error (MSE) between the reconstructed and original pixel values of the masked patches only.
For the transeriptomics decoder, mean-pool the latent tokens for each cell and pass them through a three-layer MLP that outputs value for each gene. When raw counts are available, we apply softmax over the gene dimension and use the negative log-likelihood of negative binomial distri- bution as the loss. If reconstructing fog-normalized counts instead, we use an MSE loss. Analogous the image modality, the loss calculated exclusively on masked-out genes.
3.5 EVALUATION
We evaluate two metrics on aggregated embeddings from genetic treatment metadata.
Guide Consistency. In CRISPR screens, multiple guide RNAs (sgRNAs) are designed to target the same gene for editing, and thus these guides should induce similar phenotypic effects. To assess whether the models' representations similarities of mean guide embeddings within each target gene and compare them to an empirical cardinality between unrelated sgRNAs. The metric
consistent with this prior knowledge,
null distribution of similarities with the we report is the fraction of target genes in the screen with guides that have a statistically significant (p 0.05) similarity after multiple testing correction.
StringDB edge classification. Introduced by Sivanandan al. 2023), this metric uses physi- cally interacting gene-gene pairs collected from the StringDB database as ground truth labels in a zero-shot classification task. Pairwise cosine similarities computed for aggregated target gene
5

---

Published as a conference paper at ICLR 2026
100
b
1.0
1,0
0.8
0.8
0.75
0.75
0.8
0.6
MAT
VIT MAE
0.6
0.5
0.4
VIT MAE
0.2
TiP MAE
0.2
0.25
0.25
0.0
0.0
FPR
0.5
1.0
0.0
0.0
0.25
TrP MAE
0.5
0.75
1.0
0.01
0.0
Figure 3: StringDB network edge metric analysis. The left panel of both sub- sub-figures corresponds to the HepG2 dataset and the right to Perturb-Mult.. a. ROC curves for PETRI and unimodal mod- els (HepG2: PETRI AUC=0.628, VIT MAE AUC=0.549, TrP MAE AUC=0.556; Perturb-Multi: PETRI AUC=0.655, ViT MAE AUC=0.537, TrP MAE AUC=0.571). b. 2D histograms showing the pseudo classifier probabilities for unimodal models against the probabilities in PETRI.
embeddings. The similarities are treated as pseudo classifier probabilities and we evaluate the true positive rate (TPR) at 5% false positive rate (FPR) from the ROC curve. We expect this metric to be challenging because SBringDB is neither cell nor phenotype specific and many single gene perturbations are expected have weak or no effect.
Before aggregation for both metrics we robust center scale embeddings relative to a per replicate control and apply PCA and whitening without dimensionality reduction. Justification for and results without this preprocessing are in A.5 To form a multimodal perturbation profile, for the given perturbation we average mrodality-specifie cell embeddings and then average across modalities.
EXPERIMENTS
We evaluate PETRI on two datasets and compare against CLIP and unimodal baselines.
HepG2 (Matched Perturbations): This dataset consists of matched OPS and Perturb-seq in HepG2 cells in four different chemical backgrounds. Cells were imaged with fluorescence and label-free microscopy. while a separate but matched population was profiled with whole transcrip- tome Flex sequencing. We included cells that received exactly one sgRNA from a CRISPR knockout library of 569, with four unique sgRNAs per target gene. Cells were grouped first by chemical back- ground and then by sgRNA as Fig. 2b: cells per modality were sampled from each group to form a document (see A.11. 8 latent tokens were used per cell such that documents contained 16 x x 8 = 256 tokens (see A.10 The total dataset has 2M cells (0.9M images, 1.IM transcrip- tomics). We publicly release cene-blinded version of this dataset together with this paper.
Perturb-Multi (Matched Cells): This dataset from Saunders et al. 2025) consists of spatial tran- scriptomics acquired from single section of mouse liver tissue, including paired single-cell MER- FISH measurements of 209 mRNAs and fluorescence images of 18 stained proteins. The dataset contains cells that received sgRNA from a CRISPR knockout library 203, with two unique sgR- NAs per target gene. Cells were grouped by unique cell ID such that all groups contained a single cell; the set size was therefore per modality and sampling was not required. Again, latent tokens were used per cell, giving a document length of 1 x 2 x 8 = 16 tokens.
To validate our hypothesis that joint reconstruction is robust even when the multimodal sets are not mutually informative, we additionally trained PETRI models on permuted data, where cells were randomly assigned to groups regardless of perturbation label.
Unless otherwise stated, PETRI cell embeddings were extracted directly from the output of the modality-specific encoders, i.c., before cell document creation and the MST. The MST's cross- modality attention encourages the upstream encoders to create tokens that are well aligned and compatible. As tokens approach the decoders, they become specialized for the specific reconstruc- tion task of that training step. The information sharing that happens in the MST is critical driver for multimodal integration, even if the layers themselves do not produce the best embeddings for downstream tasks. More generally, SSL methods commonly (Bordes al. 2022 benefit from

---

Published as a conference paper at ICLR 2026
Table 1: Evaluation of PETRI against unimodal and multimodal baselines (GC=Guide Consistency; StrinDBB-StingDB edge classification). CLIP* denotes two different models: using single cell pairs for Perturb-Mult and using mean aggregated perturbation- pairs for HepG2 (see A.3) Values are median across three seeds for HepG2 and one seed for Perturb-multi.
Modality
Method
Perturb-Multi
HepG2
GC
StringDB
GC
StringDB
PCA
0.030
0.073
0.304
0.150
ScGPT Pretrained
0.059
0.078
0.048
0.057
Transcriptomics
TrP MAE
0.059
0.109
0.195
0.144
PETRI Omics
0.209
0.167
DINOv2 Pretrained
0.015
0.068
0.008
0.062
Imaging
ViT MAE
0.000
0.094
0.067
0.116
PETRI Image
0.031
0.086
TrP VIT MAE Concat.
0.000
0.099
0.155
0.153
TrP VIT MAE Mean Cos.
0.035
0.068
0.178
0.163
Multimodal
TrP VIT MAE Max Cos.
0.000
0.094
0.169
0.219
Cross-modal AE
0.032
0.100
CLIP#
0,000
0.057
0.051
0.174
PETRI Permuted Data
0.163
0.260
0.274
0.255
PETRI
0.208
0.260
0.278
0.242
moving layers. Crucially, this means that the jointly trained image and gene expression encoders can be independently run at inference time to embed cells in unmatched screening data well.
4.1 PERTURBATION PROFILES FROM PETRI EMBEDDINGS RECAPITULATE KNOWN BIOLOGY
In this section, we benchmark PETRI for the task of aggregating multimodal data into holistic per- turbation embedding. As unimodal baselines, we tested strong pre-trained models, scGPT (Cui et al.) 2024) and DINOv2 (Oquab al. 2023), and modality-specifie MAES (TrP MAE and ViT MAE). For the HepG2 dataset only, we also experimented with variants of PETRI that operate on unimodal cell documents, which we designate PETRI Image and PETRI Omics. Table summarizes our find- ings. Compared to unimodal baselines, PETRI performs substantially better on both datasets and evaluation metrics. The only exception was PCA on gene expression for the HepG2 dataset, which showed slightly higher guide consistency but much lower StringDB score. ROC curves for PETRI against unimodal MAES show better detection of StringDB edges at all false positive rates B.)
Next, we considered simple late-fusion of unimodal perturbation profiles with two methods: (1) Mean or max aggregation of the modality-specific cosine similarity matrices and (2) concatena- tion of unimodal embeddings and computation of new similarity matrices. Max aggregation gave SBringDB scores close to PETRI for the HepG2 dataset, though guide consistency showed no in- crease over TrP MAE. Cross-modal autoencoders (AE) provide an alternative late-fusion method, designed expressly for unpaired data like the HepG2 dataset. Using default hyperparameters for images and gene expression, we found that the latent space alignment enforced by the loss function prevented accurate reconstruction of RNA-seq counts. Results from this model were on par with unimodal image baselines, suggesting that gene expression was not effectively fused into the frozen image-only latent space. Additional late-fusion strategies are evaluated in A.7 PETRI uniformly outperformed all late-fusion approaches.
CLIP, our carly-fusion baseline, performed worse than late-fusion, especially for Perturb-Muik We images and mRNA
included Perturb-Multi because we expected its cell-level pairs of counts would be more appropriate for CLIP. We theorize that the two modalities may not have close enough relationship for contrastive learning. As a test, we trained a ViT to regress the mRNA counts directly from protein images. On a held out validation set we found that 80% of mRNAs were predicted with ? < 0.20 (mean y = 0.117: see A.8 To adapt CLIP for the HepG2 dataset, used sets of perturbed cells, in PETRI, and mean aggregated their profiles before computing the contrastive loss. This is effectively the architecture CelICLIP without pre-trained and frozen encoders. Embeddings from this model performed worse than simple late-fusion baselines.

---

Published as a conference paper at ICLR 2026
2000
0.000
Croze-modal altertton
Image Mo0. withoou
1500
PETRO
0 0,005
0.004
1000
Htelc Demuna
0.003
PETRI HcpG2
500
0.002
PETRI Hepti
Permuted
0.001
0.0 0.2 0.4
Image MSE with omics
Cross modal Cosine Sim.
omics 4mage
Figure 4: PETRI cross mods-mod information usage. Left: BODIPY reconstruction loss with without access to perturbation perturbation-matther matched omics. Right: Example masked image of BODIPY chan- nel (top) and residual maps between the image-only reconstruction and with matched vs. random omics (bottom two; red positive, blue negative). b, Cross-modality cosine similarities of treatment- aggregated embeddings (colors as in a). c, Representative attention heads; black lines separate image (top-left) and omics (bottom-right) token blocks; off-diagonals indicate cross-modal atten- tion. d, Mean of standard deviations, o, over rows in image-omies (upper right) and omics-image (lower left) attention map quadrants, from all attention heads in the MST layers (colors as in a).
To establish that PETRI incorporates information from both modalities in its aggregated perturbation profiles, we plotted the pseudo classifier probabilities against unimodal models (Fig. 3p). Probabil- ities for PETRI were highest when they were also high in both modalities. They remained high, but decreased, where the modalities disagreed. The correspondence is stronger for HepG2 than Perturb- Multi, which appears to favor gene expression over images. Overall, this shows that PETRI learns associations between gene pairs even when that association only strongly visible in one modality.
A striking finding was that training PETRI on permuted variants of the datasets gave broadly similar results (worse on Perturb-Multi GC and better on HepG2 StringDB). Training on permuted data dis- courages coos-mtdality learning. but unimodal learning should not be affected; the model should just learn not to attend to the other modality. As such, the finding that PETRI is robust to permuted data is thus positive result. Notably, it seems that by virtue of existing in the same space, unimodal PETRI embeddings can more effectively be aggregated into a multimodal profile with simple aver- aging than more sophisticated late-fusion methods on embeddings from separately trained models.
Importantly, note that the metrics evaluated here only consider image and gene expression embed- ding averages, not whether they are coherently integrated at feature level. The following sections dig deeper and show that, with correctly matched data, PETRI actually learns cross-modaliy rela- tionships with biological relevance that are useful beyond aggregate perturbation profiles.
4.2 PETRI INTERNALLY MAKES CROSS-MODALITY PREDICTIONS
To validate that PETRI uses cross-modality information, we performed an ablation analysis. We focused on the reconstruction of the BODIPY channel, which stains for lipid droplets, in the HepG2 dataset. This choice was motivated by our screening library that includes control perturbations known to regulate lipid droplet size, quantity, and distribution. Using simple intensity threshold, selectively masked out image patches containing droplets and evaluated the MSE loss for those patches. We calculated loss when cell documents only had latent tokens from the masked image and when additionally given access latent tokens from set of unmasked gene expression profiles.
Providing gene expression tokens from the same control perturbation decreased the image recon- struction loss for BODIPY relative to the image only case (Fig, (b). The effect was sporadic across cells, but when present the MSE decrease was sizable-the the same was not true for the model trained on permuted data, signaling the benefit of the document approach for cross-modal alignment. In- specting an image where including gene expression had a large effect, we see an increase in predicted BODIPY intensity in the masked-our region over the image-only reconstruction. Providing gene ex- pression from a different control perturbation known to decrease lipid droplets, we see decrease in BODIPY intensity instead. In this targeted analysis, we conclusively showed that PETRI image reconstructions are influenced by transcriptomics data, and in biologically expected ways (see A.11
8

---

Published as a conference paper at ICLR 2026
a
Count
40
Test Accuracy
30
200
100
10
reslication
division
Petri
Petri Perm.
MAE
Petri
TIP MAE
Multimodal Dims.
Random Dims.
b
1.0
1.0
Omics Act Freq.
0.8
Omlcs Act. Freq.
0.8
cholestaral homenstaals carbonyoran metabolic
process cholestery blosymthete etto pr poess
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.00
0.25
0.50
0.75
1.00
0.75 Image Act. Freq.
respratioo, proteasome complex
activity,
Image Act. Freq.
Figure 5: SAE analysis on HepG2. Number of perturbations with 2 1 differentially activated SAE dimension in images (left) or omics (right). b, Fraction cells activating each SAE dimension in images vs. omics (left) and permuted data (right). Test-accuracies for classifiers trained to perform a aatch-prediction task using multimodal SAE dimensions vs. randomly sampled mensions (20 random draws per classifier). d, Representative images: top, non-activating samples; bottom, 99th-percentile activations. Arrows annotate the top three GO terms (pre-ranked enrich- ment; direction indicated). Channels: DAPI; BODIPY; Phalloidin (top); MitoProbe (bottom two).
Returning the hypothesis that embeddings before the MST would be aligned across modalities, measured the cosine similarity of unimodal perturbation profiles created from the PETRI modality- specific encoder outputs. Whereas the model trained on correctly matched data has clear and positive cosine similarity, the model trained on permuted data shows near orthogonality (Fig. 4b). This orthogonality would not hurt performance on the evaluation metrics in the previous section, but does point to bifurcation of the modalities in the latent space. Further evidence of this is suggested MST. Training on correctly matched data uniquely shows patterns of non-trivial and statistically significant (p < 0.001) cross-modality attention (Fig, (c.)).
by visualizing the attention heads
4.3 IDENTIFICATION OF MULTIMODAL CELLULAR PHENOTYPES
Having established that PETRI uses cross-modality information for reconstruction, we probed the concepts it learns encode. To disentangle shared and modality-specific structure in PETRI embed- dings, we trained BatchTopK sparse autoencoders (SAEs) with 15,360 dimensions and K 500.
We examined all SAE dimensions to test whether grouping cells by perturbation during training encouraged more salient perturbation-specific concepts than uninformed baselines. We performed differential analysis of SAE activations for perturbations versus the negative control, controlling false discovery rate with Benjamini Hochberr. PETRI had substantially more perturbations with at least one differentially activated dimension than unimodal models or the model trained on permuted data (Fig. 5)); this again supports that PETRI learns biologically relevant signal by coherently tegrating modalities. In particular, cells within document vary in orientation, intensity, and cell cycle, but a perturbation-enriihe phenotype provides context that reduces uncertainty (c.g., the appropriate BODIPY intensity to predict), nudging the model toward concepts that distinguish per- turbations. While supervised training also aligns concepts to perturbations, it implicitly assumes each perturbation has unique and appreciable effect, which is often violated in CRISPR screens.
We then searched for concepts shared across modalities, defining "multimodal" dimensions as those activated in 10-90% of cells in both imaging and transcriptomics. PETRI produced 298 such dimen-

---

Published as a conference paper at ICLR 2026
sions, compared with 0 for the permuted data model and 1 for CLIP (pig.5)); these results speak to PETRI's 's ability to align modalities in the latent space, resulting in multimodal concepts. Further- more, if these concepts truly reflect biology common to both modalities, they should be less sensitive to modality-specifee technical artifacts. We tested this by training logistic regression models to pre- diet OPS well identity using either the 298 multimodal SAE dimensions or 298 randomly selected ones. Classifiers using the multimodal dimensions were significantly less accurate (p < 0.001; Fig. 5c), consistent with reduced encoding of undesirable well-specific technical factors.
Finally, we inspected the multimodal SAE dimensions for interpretability. For each dimension, we selected and compared images and transcriptomic profiles with zero activation against those with activation above the 99th percentile and asked whether there were statistically significant differ- ences between these sets of images and transcriptomic profiles, respectively. For images, we ran differential analysis on handcrafted features of fluorescence intensity and nuclear and lipid droplet morphology. For transcriptomies, we performed differential expression analysis and computed pre- rank enrichment against Gene Ontology (GO) terms. In total, 127 of the 298 dimensions showed significant differences for at least one image feature and enrichment in at least one GO term.
To further interrogate these multimodal PETRI SAE dimensions for relevant biology, we searched by keywords for SAE dimensions that showed enrichment in terms related to phenotypes we know should be present in dataset: cell cycle, lipid metabolism, and mitochondrial activity. Corre- sponding images for those dimensions revealed interpretable biological concepts including DNA replication (correlating with nucleus shape and DAPI intensity). cholesterol homeostasis (correlat- ing with lipid droplet size and quantity), and aerobic respiration (correlating with mitochondrial fusion and network structure) 5d).
These results demonstrate that PETRI can be leveraged to link molecular states to morphological phenotypes at single-cell resolution. This enables researchers to confidently prioritize morphological changes that are validated by a corresponding molecular signature, ensuring they are both real and biologically relevant. SAEs trained on unimodal embeddings cannot be used for this purpose without first finding correspondences between separately learned concepts.
5 CONCLUSION
We presented PETRI, an early fusion approach for learning multimodal single cell embeddings from unpaired image and transcriptomic data; the approach is robust to the presence of only weak or variable modsl alignment. We demonstrated that PETRI-lcarned perturbation profiles (I) recover gene relationships better than unimodal and late-fusion baselines, and (2) encode batch- robust, biologically meaningful concepts that can be disentangled with sparse autoencoders.
context- paotet--groupe data can induce meaningful multi- modal alignment without explicit cross-modal losses. PETRI demonstrates this, opening a path to unifying historically separate screening modalities. At the same time, our results highlight limits of prevailing proxies (e.g., guide consistency, protein-protein interaction prediction): useful as bench- marks, but insufficient to capture the biologically meaningful structure revealed by our downstream analyses. Instead, there is a need for task- mask-grounded evaluation frameworks tailored to multimodal phenotypic screening and therapeutic discovery, with metrics that directly assess biological utility.
A key insight is that joint reconstruction
Several other important questions remain:
How closely matched must experimental contexts be to possible to incorporate other biological priors? For instance,
enable cross- modslityylyty learning? instead of documents that focus on a single perturbation, what if we centered them on protein com- plexes or pathways? Our framework provides a foundation for exploring these questions and sug- gests that strategic contextual grouping could serve as a mechanism for easily incorporating domain knowledge into representation learning.
Although PETRI is designed for images and gene expression, the core idea of using joint recon- struction from context-aligned cellular documents could be adapted to other modalities. As diverse omics technologiess continue to proliferate, approaches that can integrate complementary views of cellular state without requiring perfect experimental alignment will become increasingly valuable for advancing our understanding of cellular biology and accelerating therapeutic discovery.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
Code and the blinded HepG2 dataset will be made public. All results aside from StringDB metrics on the HepG2 dataset will be reproducible from the provided code and data.
ACKNOWLEDGEMENTS
We thank members of the Data Science and Machine Learning department at Insitro for providing early feedback on the results presented in this work.
REFERENCES
BaanBBaptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: A visual language model for few-shot learning. arXiv [cs. CV.. April 2022.
Tal Ashuach, Mariano I Gabitto, Rohan V Koodli, Muuseppe-Antoni Saldi, Michael I Jordan, and Nir Yosef. MultiVI: deep generative model for the integration of multimodal data. Nat. Methods, 20(8):1222-1231, August 2023.
Loc Binan, Aiping Jiang, Serwah A Danquah, Vera Valakh, Brooke Simonton, Jon Bezney, Robert Manguso, Kathleen B Yates, Ralda Nehme, Brian Cleary, and Samouil L Farhi. Simultaneous CRISPR screening and spatial transcriptomics reveal intracellular, intercellular, and functional transcriptional circuits. Cell, (880)))14 April 2025.
Florian Bordes, Randall Balestriero, Quentin Garrido, Adrien Bardes, and Pascal Vincent. Guillo- tine regularization: Why removing layers needed to improve generalization in self-supervised learning. arXiv [cs. June 2022.
Juan C Caicedo, Claire McQuin, Allen Goodman, Shantanu Singh, and Anne E Carpenter. Weakly supervised learning of single-cell feature embeddings. Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., 201 999809999911,, June 2018.
Srinivas Niranj Chandrasekaran, Hugo Ceulemans, Justin D Boyd, and Anne Carpenter. Image- based profiling for drug discovery: due for a machine-leaming upgrade? Nat. Rev. Drug Discov,, 20(2):145 159, February 2021.
Haonan Chen, Hong Liu, Yuping Luo, Liang Wang, Nan Yang, Furu Wei, and Zhicheng Dou. MoCa: Modality-aware continual pre-traing vee-training makes better bidirectional multimodal embeddings. arXiv [cs.CV), CVJ, June 2025a.
Kok Hao Chen, Alistair N Boettiger, Jeffrey R Moffitt, Siyuan Wang, and Xiaowei Zhuang. RNA imaging. spatially resolved, highly multiplexed RNA profiling in single cells. Science, 348(6333): aaa6090, April 2015.
Weiqing Chen, Pengzhi Zhang, Tu N Tran, Yiwei Xiao, Shengyu Li, Vrutant V Shah, Hao Cheng, Kristopher W Brannan, Keith Youker, Li Lai, Longhou Fang. Yu Yang. Nhat-Tu Le, Jun-Ichi Abe, Shu-Hsia Chen, Qin Ma, Ken Chen, Qianqian Song, John P Cooke, and Guangyu Wang. A visual- omics foundation model to bridge histopathology with spatial transcriptomics. Nat. Methods, 22 (7):1568 1121568-1582,, July 2025b.
Haotian Cui, Chloe Wang, Hassaan Maan, Kuan Pang, Fengning Luo, Nan Duan, and Bo Wang. scGPT: toward building a foundation model for single-cell multi-omics using generative AI. Nat. Methods, 2108) August 2024.
Andor Diera, Lukas Galke, and Ansgar Scherp. Isotropy matters: Soft-ZCA whitening of embed- dings for semantic code search. arXiv (cs. CL.J, November 2024.
11

---

Published as a conference paper at ICLR 2026
Atray Dixit, Oren Parnas, Biyu Li, Jenny Chen, Charles P Fulco. Livnat Jerby-Amon, Nemanja D Marjanovic, Danielle Dionne, Tyler Burks, Raktima Raychowdhury, et al. Perturb-seq: dissecting molecular circuits with scalable single-cell ma profiling of pooled genetic screens. cell, 167(7): 1853-1866, 2016.
Michael Doron, Théo Moutakanni, Zitong S Chen, Nikita Moshkow, Mathilde Caron, Hugo Touvron, Piotr Bojanowski, Wolfgang M Pernice, and Juan C. Caicedo. Unbiased single-cell morphology with supervipe vision transformers. bioRxivorg, June 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale. arXiv [cs.CV], October 2020.
David Feldman, Avtar Singh, Jonathan L Schmid-Burgk, Rebecca J Carlson, Anja Mezger, An- thony Garrity, Feng Zhang, and Paul C Blainey. Optical pooled screens in human cells. Cell, 17963):777799..e1, October 2019.
Jing Gong, Minsheng Hao, Xingyi Cheng, Xin Zeng, Chiming Liu, Jianzhu Ma, Xuegong Zhang, Taifeng Wang, and Le Song. xTrimoGene: An efficient and scalable representation learner for single-cell RNA-seq data. In Thirty-seventh Conference on Neural Information Processing Sys- tems, November 2023.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. P Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International Conference on Machine Learning, pp. 4651-4666 PMLR, July 2021.
Jérémie Kalfon, Jules Samaran, Gabriel Peyré, and Laura Cantini. ScPRINT: pre-training on 50 million cells allows robust gene network predictions. Nat. Commun., 16(1):3607, April 2025.
Oren Kraus, Kian Kenyon-Dean, Saber Saberian, Maryam Fallah, Peter McLean, Jess Le- ung. Vasudev Sharma, Ayla Khan, Jia Balakrishnan, Safiye Celik, Dominique Beaini, Maciej Sypetkowski, Chi Vicky Cheng, Kristen Morse, Maureen Makes, Ben Mabey, and Berton Earn- shaw. Masked autoencoders for microscopy are scalable learners of cellular biology. In 2024 JEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pp. 11757-11768. IEEE, June 2024.
Christopher Lance, Malte D Luecken, Daniel B Burkhardt, Robrecht Cannoodt, Pia Rautenstrauch, Anna Laddach, Aidyn Ubingazhibov, Zhi-Jie Cao, Kaiwen Deng, Sumeer Khan, Qiao Liu, Niko- lay Russkikh, Gleb Ryazantsev, Uwe Ohler, NeurlPS 2021 Multimodal data integration compe- tition participants, Angela Oliveira Pisco, Jonathan Bloom, Smita Krishnaswamy, and Fabian Theis. Multimodal single cell data integration challenge: Results and lessons learned. In NeurIPS 2021 Competitions and Demonstrations Track, pp. 162-176. PMLR, July 2022.
Anastasia Litinetskaya, Maiia Shulman, Soroor Hediyeh-zadeh, Amir Ali Moinfar, Fabiola Cu- rion, Artur Szatata, Alireza Omidi, Mohammad Lotfollahi, and Fabian Theis. Multimodal weakly supervised learning to identify disease-specific changes in single-cell atlases. bioRxiv. pp. 2024.07.2222 00.24.07.29.6666 29.6062... July 2024.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in Neural Information Processing Systems, 36:34892-34999 December 2023.
Romain Lopez, Jeffrey Regier, Michael Cole, Michael Jordan, and Nir Yosef. Deep generative modeling for single-cell transcriptomics. Nat. Methods, 15(12):1533-1055 December 2018.
Mingyu Lu, Ethan Weinberger, Chanwoo Kim, and Su-In Lee. CelICLIP learning perturbation effects in cell painting via text-guided contrastive learning. arXiv [cs.LG), June 2025.
12

---

Published as a conference paper at ICLR 2026
M Oquab, Timothée Darcet, Théo Moutakanni, Huy Q Vo. Mare Szafraniec, Vasil Khalidov, Pierre Fermandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby. Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russ Howes, Po-Yao (bernie) Huang, Shang- Wen Li, Ishan Misra, Michael G Rabbat, Vasu Sharma, Gabriel Synnaeve, Huijiao Xu, H Jégou, Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. ArXiv, abs/2304.07111 07199, April 2023.
Shang-Wen
James D Pearce, Sara Simmonds, Gita Mahmoudabadi, Lakshmi Krishnan, Giovanni Palla, Ana-Maria Istrate, Alexander Tarashansky, Benjamin Nelson, Omar Valenzuela, Donghui Li, Stephen R Quake, and Theofanis Karaletsos. A cross-species generative cell atlas across 1.5 bil- lion years of evolution: The TranscriptFormer single-cell model. bioRxiv, pp. 2025.04.25.650773. April 2025.
Chau Pham, Juan C Caicedo, and Bryan A Plummer. ChA-MAEViT: Unifying channel-aware masked autoencoders and multi-channel vision transformers for improved cross-channel learn- ing. arXiv [es. CVJ, October 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. arXiv (cs.CV), February 2021.
Adiyyanarayanan Radhakrishnan, Sam F Friedman, Shaan Khurshid, Kenney Ng, Puneet Batra, Steven Lubitz, Anthony A Philippakis, and Caroline Uhler. Cross-modal autoencoder frame- work learns holistic representations of cardiovascular state. Nat. Commun., 14(1):2436, April 2023.
Meraj Ramezani, Erin Weisbart, Julia Bauman, Avtar Singh, John Yong. Maria Lozada, Gregory P Way, Sanam L Kavari, Celeste Diaz, Eddy Leardini, et al. A genome-wide atlas of human cell morphology. Nature methods, 22(3):621 633, 2025.
Jennifer Rood, Anna Hupalowska, and Aviv Regev. Toward a foundation model causal cell and tissue biology with perturbation cell and tissue atlas. Cell, 187(17):450--455, August 2024.
Shota Sasaki, Benjamin Heinzerling, Jun Suzuki, and Kentaro Inui. Examining the effect of whiten- ing static and contextualized word embeddings. Inf. Process. Manag. 60(3):10327,, May 2023.
Reuben Saunders, William Allen, Xingjie Pan, Jaspreet Sandhu, Jiaqi Lu, Thomas K Lau, Karina Smolyar, Zuri A Sullivan, Catherine Dulac, Jonathan S Weissman, and Xiaowei Zhuang. Perturb-multinodaal: A platform for pooled genetic screens with imaging and sequencing in intact mammalian tissue. Cell, (88(17):4790-4800 .22, August 2025.
Srinivasan Sivanandan, Bobby Leitmann, Eric Lubeck, Mohammad Muneeb Sultan, Panagiotis Stanitsas, Navpreet Ranu, Alexis Ewer, Jordan E Mancuso, Zachary F Phillips, Albert Kim, et al. A pooled cell painting crispr screening platform enables de novo inference of gene function by self-supervised deep learning. bioRxiv, pp. 2023 08, 2023.
Christina V Theodoris, Ling Xiao, Anant Chopra, Mark D Chaffin, Zeina R Al Sayed, Matthew C Hill, Helene Mantineo, Elizabeth M Brydon, Zexian Zeng, X Shirley Liu, and Patrick T Ellinor. Transfer learning enables predictions in network biology. Nature, 61 611 6-64,, June 2023.
Gregory Way, Ted Natoli, Adeniyi Adeboye, Lev Litichevskiy, Andrew Yang. Xiaodong Lu, Juan C Caicedo, Beth A Cimini, Kyle Karhohs, David J Logan, Mohammad H Rohban, Maria Kost-Alimova, Kate Hartland, Michael Bornholdt, Srinivas Niranj Chandrasekaran, Marzieh Haghighi, Erin Weisbart, Shantanu Singh, Aravind Subramanian, and Anne E Carpenter. Mor- phology and gene expression profiling provide complementary information for mapping cell state. November 2022.
Cell Syst.,
Johnny Xi and Jason S Hartford. Propensity score alignment of unpaired multimodal data. Neural Inf Process Syst, 015 141:14 128, April 2024.
13

---

Published as a conference paper at ICLR 2026
Lei Xiong, Tianlong Chen, and Manolis Kellis. seCLIP: Multi-modal single-cell contrastive learning integration pre-training. In NeurIPS 2023 Al for Science Workshop, October 2023.
Karren Dai Yang, Anastasiya Belyaeva, Saradha Venkatachalapathy, Karthik Damodaran, Abigail
Katcoff, Adityanarayanan Radhakrishnan, G V Shivashankar, and Caroline Uhler. Multi-domain translation between single-cell imaging and sequencing data using autoencoders. Nat. Commun., ()1)311, January 2021.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 HEPG2 DATASET
Briefly, a pool of constitutive Case-expressing hepatocellular carcinoma cells (HepG2s) received approximately one sgRNA per cell from CRISPR knockout library of 569 genes, with four unique sgRNAs per target gene. Cells were expanded as a uniform pool to maximize parity between imag- ing and transcriptomic populations, then treated with one of four chemical treatments. Cells were imaged with fluorescence at 20X magnification by staining the cells with a variation of CellPaint, which included HOESCHT, PHALLÓIDIN, BODIPY, and MITOPROBE label DNA, actin, neu- tral lipids, and mitochondrial rRNA, respectively. Cells were additionally imaged via quantitative phase imaging and brightfield. Imaging was followed by sgRNA amplification and in-situ sequenc- ing. A parallel population was used for the creation of the transcriptomic dataset, which was gen- erated using Flex technology, with probes reading out both the transcription, as well as the gRNA sequences, within each cell.
1.0
0.8
Proportion
0.6
modality
0.4
OPS
Perturb-seq
0.2
0.0
200
400
Figure S1: ECDF of cell counts per perturbation group (sgRNA)
Due to fitness effects, the number of barcoded cells that received each sgRNA varies considerably (Figure S1). 19% of perturbation groups had fewer than 16 cells in one or the other modality, which motivated the choice of 16 the default set size for PETRI. With larger set sizes and sampling with replacement, the possibility of leakage increases, e.g., the same cell with different masking patterns can appear in the same set.
A.2 PETRI IMPLEMENTATION
Unimodal encoders and Multimodal Set Transformer (MST) The image encoder a standard ViT-Base model (Dosovitskiy al., 2020) with 85M parameters. The gene expression encoder is a Perceiver with four cross-attentie layers and 12 self-attention layers, where each cross-attention layer is followed by three self-attention layers. The total parameter count is 131M, of which 16M correspond to learnable gene embeddings. To match ViT-Base, the embedding dimension is 768. Af- ter the unimodal encoders, the MST is four layer transformer comprised of ViT-style self-attention blocks with a total of 14M parameters.
Decoders The image decoder architecture follows Masked Autoencoders and using a ViT with embedding dimension of 512 and eight transformer layers amounting 26M parameters. The gene expression decoder is a simple threc-layer MLP with hidden dimension of 128 and final output dimension of either 18082 for HepG2 or 209 for Perturb-Multi. It consists of 2M parameters.
The total parameter count is 259M with roughly equal numbers of parameters for each modality. However, the gene expression-related parameters are almost entirely in the encoder whereas sub- stantial fraction of the image-related parameters are in the decoder.
A.3 CLIP BASELINE IMPLEMENTATION
To adapt CLIP to setting, used the same architecture PETRI but removed the decoders and Multimodal Set Transformer. For Perturb-Multi, with cell pairs, we used a batch size of 4096 and the standard CLIP loss over positive and negative pairs. For HepG2, with perturbation-level
15

---

Published as a conference paper at ICLR 2026
matching, used sets of 16 cells sampled from the same perturbation positive pairs. Before computing the loss, we mean-aggregated the embeddings of all cells in the set. The batch size was 4096 cells per modality; however, after aggregation there were 4096 + 16 256 pairs over which to compute the CLIP loss.
A.4 MODEL TRAINING
PETRI and baseline models were all trained for 100 epochs. For the HepG2 dataset, every epoch included exactly 30 samples of cells drawn from each perturbation group. This balanced sampling was consistent for all models that we trained, regardless of whether they required sets of cells not. For Perturb-Multi, with cell-level pairs, this sampling was unnecessary and an epoch included each cell just once. total, HepG2 models trained for 250K iterations and Perturb-Mine models trained for 145K iterations. Training took 2 days on single node with 8 H100 GPUs and batch size of 1536 masked images and/or expression profiles.
Image augmentations and patching Berturb-Mull and HepG2 images were augmented with ran- dom vertical and horizontal flips, 360 degree rotations and 5% translations. To account for cell sizes, used center erops of 64 pixels for Perturb-Multi and 80 pixels for HepG2. VIT patch size pixels for both.
Loss weighting For the HepG2 dataset, used weight of 1.0 for the image MSE and 10-1 for the negative binomial loss. For Perturb-Multi, we used a weight of 1.0 for the image MSE and 0.01 for the mRNA count MSE.
A.5 EMBEDDING POSTPROCESSING AND EVALCATION
We preprocessed single cell embeddings before aggregating them into perturbation profiles and com- puting guide consistency and StringDB network edge metrics. For HepG2 image embeddings, we performed robust center scaling (RCC), i.e., we subtracted the median embedding of the intergenic control cells in each OPS well and divided by the interquartile range. Standardizing by per replicate controls is a common method for reducing batch effects introduced by natural well-to-welll vari- ability in culture media or fluorescence intensities and empirically works well for VIT embeddings (Kraus et al. 2024). For HepG2 gene expression embeddings, performed RCC using the global intergenic controls. Cells from the four chemical backgrounds were preprocessed independently and the evaluation metries for each background were averaged.
Since Perturb-Multi cells all come from a single batch, we used global statistics from the non- targeting control. PCA and whitening promote isotropy and enhance semantic search for language embeddings (Diera et al. 2024; Sasaki et al. 2023), with similar benefits for perturbation embed- dings (Kraus et 2024).
Table S1 shows that StringDB metric evaluation on the raw embeddings gives no better than random performance (0.05) on HepG2.
Table S1: Evaluation of unimodal and multimodal embeddings without preprocessing the HepG2 dataset for the StringDB edge classification metric.
Modality
Method
HepG2 StringDB
PCA
0.028
ScGPT -Pretrained
0.043
Transcriptomics
TrP MAE
0.037
PETRI Omics
0.040
DINOv2- - Pretrained
0.050
Imaging
ViT MAE
0.040
PETRI Image
0.043
CLIP#
0,040
Multimodal
PETRI Permuted Data
0.045
PETRI
0.031
16

---

Published as a conference paper at ICLR 2026
A.6 INFERENCE PROCEDURE AND INTERMEDIATE LAYER EMBEDDINGS
We chose to use PETRI embeddings from the outputs of the unimodal encoders because they can be computed without access to multimodal groups of cells. However, also considered using cell embeddings from the intermediate layers of the Multimodal Set Transformer. This required a more complex inference procedure.
Deterministic multimodal set inference. Each perturbation group was first shuffled to mitigate potential batch effects, then wrap-around padded to ensure the total number of cells in the group was divisible by the set size. The padded indices were then partitioned into non-overlapping sets of fixed size, creating a deterministic enumeration. The embeddings for duplicated cells were averaged such that the final inference result had no duplicates. When the number of available sets differed between modalities for the same perturbation, the modality with fewer sets was cyclically repeated to match the longer modality. This guarantees that every cell appears at least once in each modality while maintaining proper multimodal alignment between sets.
Table S2: Evaluation of PETRI embeddings from intermediate layers on the HepG2 dataset (GC=Guide Consistency; StringDB edge classification).
Layer
HepG2
GC
StringDB
Before MST (default)
0.260
0.242
After MST
Layer I
0.202
0.200
A After MST
Layer 2
0.211
0.197
After MST
Layer 3
0.220
0.190
After MST Layer
0,221
0.186
Table S2 summarizes the results and shows that the embeddings taken directly from the unimodal encoders give the best guide consistency and SBringDB scores on the HepG2 dataset.
A.7 LATE-FUSION METHODS
We tested three late-fusion methods, all of which make use of aggregated embeddings from VIT MAE and TrP MAE after RCC and whitening:
1. Concatenation: Imaging and transcriptomics each had 768 dimension embeddings, which we concatenated to produce 1,536 dimensions. With these we computed pairwise cosine similarities between target gene or sgRNA profiles, depending on the metric.
2. CCA: We fit CCA with 30 components to find shared space between embeddings from the two modalities. Embeddings from both modalities were projected into this space and we either concatenated or averaged them directly.
3. Cosine similarity matrix aggregation: Instead of working with the unimodal embed- dings, we directly aggregated the cosine similarity matrices from each modality by min, max, mean, or median.
Table $3 summarizes the results.
A.8 PERTURB-MULN VIT REGRESSION
We trained a ViT-Base with the same settings as those in A.4 The class token was processed with two-layer MLP to regress the 209 normalized mRNA counts in Perturb-Multi, loss was evaluated with MSE. Fig. S2 shows the distribution of ,² values for the mRNAs evaluated on a randomly chosen held out test containing 20% of total cells.
A.9 PERMUTED DATASET EVALUATION
We observed that the performance on guide consistency and StringDB network edge metrics was roughly equívalent between PETRI models trained on the correctly grouped and matched versions of
17

---

Published as a conference paper at ICLR 2026
Table S3: T ViT MAE late-fusion results (GC=Guide Consistency; StringDB edge classification).
Method
Perturb-Multi
HepG2
GC
StringDB
GC
String
TrP + VIT MAE Concat.
0.000
0.099
0.155
0.153
TrP + ViT MAE CCA Concat.
0.000
0.104
0.019
0.079
TrP + VIT MAE CCA Mean.
0.000
0.099
0.020
0.079
TrP + VIT MAE Min Cos.
0.000
0.094
0.115
0.116
TrP + VIT MAE Mean Cos.
0.035
0.068
0.178
0.163
TrP + VIT MAE Median Cos.
0.035
0.068
0.178
0.163
TrP + VIT MAE Max Cos.
0.040
0.094
0.169
0.219
1.0
0.8
Proportion
0.2
0.0
0.2
0.4
0.6
0.8
1.0
Figure S2: ECDF of y2 scores for predictions on each of the 209 mRNA counts in Perturb-Multi.
the benchmark datasets versus their permuted versions. Figure S3 shows the ROC curves and Venn diagrams for StringDB metrics. The results suggest that training on correct permuted datasets leads to identification of broadly similar gene pairs.
A.10 DOCUMENT LENGTH AND LATENT TOKEN COUNT
The resampling mechanism allows us to trade-off the number latent tokens for the number of cells in document. In the main experiments, we used 8 tokens per cell and set size of 16. At least for the available evaluation metrics, we did not observe a clear pattern of improvement or degradation from adjusting these parameters (Figure S4) and absolute differences were small. With smaller set sizes we might expect to see less cross-modality learning, especially if perturbations induce heterogeneous responses.
A.11 RECONSTRUCTION LOSS ABLATION ANALYSIS
Fig.4y shows the effect of a targeted ablation analysis on image reconstruction loss. At the aggregate level, we did not notice clear differences reconstruction loss for images gene expression the HepG2 dataset when removing access to the other modality. However, we do observe that image reconstruction loss lower overall than the model trained on permuted data Figure $5a,b.
For Perturb-Multi, on the other hand, there is a statistically significant (p < 0.05) increase in loss when access to the other modality removed. This effect disappears for the model trained on permuted data. Again, supporting the notion that the model learns to ignore irrelevant multimodal information.
18

---

Published as a conference paper at ICLR 2026
1.0
0.6
106
0.4
(AUC=0.635)
161
66
Petri (AUC-0.628)
0.0
1.0
0.8
0.5
0.4
Petri Permuteed (AUC=0.628)
10
45
5
655)
0.2
0.0
1.0
Figure S3: ROC curves and Venn diagrams for TP StringDB gene pair detections. Top: HepG2. Bottom: Perturb-Multi
Fraction Genes with Consistent Guides
0.30
StringDs RUC IPX et 8.5% 1P
0.255
0.29
0.28
0.250
0.27
0.240
0.235
32 Tokeer, 4 Colls
16 Tokers, Colls
Cells
2 Tokemss 64 Cells
32 Tokers, Cells
16 Tokers, Cells
Tokens, Colls
Takens, 64 Colls
8 16
16
Figure S4: Left: SB metric HepG2 with different combinations of latent tokens and set sizes. The listed number of cells is per modality, Right: Same for the guide consistency metric.
b 1.0
Recomstruction MSE
0.3
With With Virong Omicss Without Omics
0.0
Wrong Image
Baconsmrcction
Image
0.1
0,2
0.0
0.0
HepG2
HepG2 Perm.
Perturb- Multi
Perturb Muiti Perm.
HepG2
Hepc2
Perturb-
ParturD-
Perm.
Multi
Multi Perm.
Figure S5: Image reconstruction loss across all channels when randomly masking 75% of patches; includes no access omics, access to correctly paired omies, and access to random omics. b. Average reconstruction loss across all genes when randomly masking 75% of gene tokens; in- cludes no access to images, access to correctly paired images, and access random images.
19


Reviews:
[{"review_id": "7cqfEYxcc6", "reviewer": "Reviewer_hVW4", "summary": "Learning and aligning multiple biological modalities in a shared latent space is challenging due to the absence of cell-level pairing and overlap in signals. The paper aims to address this problem by proposing PETRI, an early-fusion transformer model that learns unified cell embeddings of mult-omic modalities in a shared latent space. Cellular images and gene expressions are unpaired and processed as cell documents and learned using a set transformer architecture. Per-cell token resampling is employed and learning is carried out using the MAE reconstruction objective. PETRI, when compared to unimodal and multimodal alternatives, presents imporved GC and StringDB on HepG2 and Perturb-Multi datasets. An interpretable analysis demonstrates that PETRI utilizes cross-modal information while capturing morphological activating features at cell-level resolution.", "questions": "Refer to weaknesses", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper is well written and organized.\n* The paper presents an intuitive perspective on multi-omic learning.", "weaknesses": "* **Motivation and Contribution:** The paper motivates multimodal learning of biological modalities using overlap of learning signals and absence of cell-level pairings. However, contributions do not study these aspects and move tangentially towards learning and outperforming prior architectural variants. It is unclear as to how PETRI mitigates conflicting biological signals using a joint representation space. Similar to prior methods, PETRI also learns cell-level embeddings using a masked variational inference objective. This would indicate similar learning signals propogating across modalities. How does the joint space learn and capture cross-modal features? Does PETRI mitigate conflicting gradients or it fits towards a particular modality? How are cell documents arranged in the embedding space given that these are mapped from two different modules (ViT and Perceiver)? In its current form, it remains unclear on how PETRI is addressing the central problem.\n\n* **Empirical Evaluation:** Authors execute experiments on HepG2 and Perturb-Multi datasets which make use of cellular images and transcriptomics profiles. However, the work proposes PETRI from a general multi-omics multimodal learning perspective. Empirical evaluations do not capture this general claim. What happens if we consider additional biological modalities such as genomics and phenomics? Furthermore, experiments do not study the effect of each modality and the contributions presented in the work. How does PETRI behave when one of the modalities is dropped, i.e- a unimodal PETRI? What happens if token resampling is ablated or ViT and Perceiver blocks are replaced by alternative design decisions? In its current form, the paper trivially applies a transformer alternative to the bimodal problem circumventing its technical details.\n\n* **Baselines:** The paper compares PETRI to unimodal as well as multimodal baselines. However, the comparison is limited in scope. Firstly, among the architectural baselines, majoriry of the models are unimodal (imaging or transcriptomics) which does not put the paper's multimodal contributions in persepctive. Secondly, multimodal baselines are limited to architecture and contrastive learning variants which do not highlight the significance of PETRI. What happens if we compare with multimodal VAEs? How does PETRI compare to a multimodal perceiver? How does PETRI compare to CLIP variants and multimodal diffusion? In its current form, baselines and their significance remain limited.\n\n* **Interpretability:** The paper conducts an interpretability-based analysis on how PETRI captures cross-modal features. However, I am having trouble understanding and drawing conclusions from these results. It is unclear as to how the recnstruction loss indicates cross-modal information. Ideally, the reconstruction objective only indicates whether the decoder can disentangle modality-specific information from the latents in order to obtain the original mapping. How does this indicate use of cross-modal features? Similarly for SAE experiments, it remains unclear as to how differences in the number of activated dimensions indicate intepretable cross-modal features. Essentially, a wider difference indicates that more latent dimensions were activated leading to diversity in the feature set. However, this diversity does not necessarilty correlate with the use of cross-modal representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c53gt2jT8A", "reviewer": "Reviewer_DpdC", "summary": "This paper presents PETRI, a early-fusion transformer framework designed to learn unified cell embeddings from unpaired single-cell imaging and transcriptomics data. The core idea is to group cells by a shared experimental context (e.g., a perturbation) into \"multimodal documents\" and train the model on a masked joint reconstruction task, enabling information sharing via cross-modal attention. To make this document-based approach computationally tractable, the authors employ an aggressive token resampling strategy. Through experiments on two datasets, the authors demonstrate that PETRI's unified embeddings outperform unimodal and late-fusion approaches on biological benchmarks like Guide Consistency and StringDB edge classification. Furthermore, they show that applying sparse autoencoders to the embeddings reveals biologically interpretable, multimodal concepts that are robust to technical confounders, and they validate that the model indeed uses cross-modal information for reconstruction. To facilitate further research, the authors also introduce and release a new large-scale dataset.", "questions": "1. In Figure S5, the reconstruction loss for one modality appears unaffected by the presence of the other modality's data. Could the authors clarify how this result aligns with the claim of successful cross-modal information integration?\n2. In Figure 4c, the identified \"Multimodal Dims.\" result in lower downstream task accuracy than \"Random Dims.\" Could the authors please explain this counter-intuitive finding?\n\nThe reviewer wrote the review. LLM was only used to improve both grammar and clarity.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes PETRI, a framework that effectively integrates unpaired imaging and transcriptomics data via a joint reconstruction task, addressing a challenge in the field.\n2. The work provides strong evidence of model interpretability through multiple analyses. It demonstrates cross-modal information transfer via targeted ablation studies and attention matrix visualization. Applying sparse autoencoders (SAEs) reveals biologically meaningful, multimodal concepts that are shown to be robust to technical confounders.\n3. The authors contribute a valuable new large-scale dataset of matched imaging and omics data.", "weaknesses": "1. The paper's best results come from embeddings taken before the data passes through the Multimodal Set Transformer (MST), the core component designed for fusion. The appendix (Table S2) confirms that performance on key benchmarks actually gets worse after the MST. This is counterintuitive and questions the practical benefit of the MST for the paper's main evaluation tasks.\n2. The analyses showing that the model learns biological concepts (Sections 4.2 and 4.3) rely heavily on specific, illustrative examples (e.g., reconstructing the BODIPY channel, showing a few representative images). The paper lacks broad, quantitative metrics to prove that these impressive findings are consistent across the entire dataset and not just in a few hand-picked cases.\n3. The comparison against the CLIP model may be unfair. CLIP is designed to work with perfectly paired data, but the main dataset in this paper is unpaired. The authors had to adapt CLIP for this different problem, which likely means it could not perform at its best. This may exaggerate the performance advantage of PETRI.\n4. The PETRI model is very computationally expensive, requiring powerful and specialized hardware to run. This is because it processes large \"documents\" of many cells at once.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UfzKo7teLk", "reviewer": "Reviewer_WChQ", "summary": "This work presents PETRI, a multimodal transformer for biological data that unifies cell-level transcriptional gene-expression readouts with morphology images. PETRI is a self-supervised regime that combines two masked autoencoders for each data modality and uses latent token resampling to deal with the difficulties of very long sequence lengths caused by gene expression data. The model is demonstrated to perform quite well over two different datasets in terms of guide perturbation consistency and stringdb relationship retrieval against a comprehensive suit of baselines (such as unimodal methods, multimodal baselines like simple embedding concatenation and CLIP). The authors furthermore apply sparse autoencoders to analyze the multimodal concepts learned by the model, with an interesting case-study on the GO-term enrichment of these dimensions and the interpretable biological concepts obtained (e.g. DNA replication).", "questions": "What does it mean that the dataset you will release will be \"blinded\"? Would others be able to reproduce these results from the public dataset release, or are perturbation labels required to do so?\n\nWhile described as *unpaired* data here, is the context here not more typically described as *weakly paired*, i.e. same perturbation but different cells? It seems that you are sampling groups of samples with matching guide perturbation labels from the two modalities -- i.e. line 154 you describe that you \"Create batches where cells are grouped by perturbation to form sets.\"", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- Releasing a new multimodal public dataset is highly valuable.\n- Valuable multimodal architecture: the PETRI architecture uses creative means to process replicates of the same perturbation across the two modalities, and the use of latent tokens addresses difficulties surrounding too many tokens that would otherwise be present.\n- Clear and high quality results: Robust benchmarking analysis with appropriate baselines is pursued by comparing their regime to unimodal and late-fusion multimodal models, and the performance improvements of PETRI hold.\n- Original and very significant: Highly relevant task is pursued and comparing results are obtained, indicating the important complementarity of multimodal biological screening assays in providing more comprehensive understandings of perturbations when unified together in the context of early-fusion, a highly motivating result that many may have held intuitively but, prior to this work, had yet to be thoroughly broached.\n- Creative additional investigation into the nature of their multimodal representation learning model: SAE is used to analyze their multimodal model demonstrates biologically relevant compelling and interesting results (see Figure 5)\n- Many avenues for relevant future work and additional research are opened up as a result of this work. I can think of a multitude; for example: (1) multidataset training: it could be interesting, e.g., to train a combined model on both Perturb-Multi and HEPG2; (2) exploring finetuning, e.g. it could make sense to pretrain a generic microscopy MAE on JUMP-CP data and then finetune that architecture here in PETRI, rather than training the image MAE from scratch; (3) varying the mask ratio, sometimes entirely masking the entire other modality, as being able to predict the other modality from one (e.g. if the transcriptomics is masked 100% and morphology only 50%) could be quite valuable if it works.", "weaknesses": "The only weakness I see in this paper is secondary, namely that it is lacking situated references to various related work that has pursued similar studies. For some examples, I think it could be fruitful to mention the following highly related papers to this work:\n- Simple methods like control or mean baselines turn out to be hard-to-beat in transcriptomics and could be worth discussing in the context of their benchmarking; a variety of papers discuss this, e.g.: Ahlmann-Eltze, Constantin, Wolfgang Huber, and Simon Anders. \"Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines.\" Nature Methods (2025): 1-5.\n- The PCA with whitening approach is quite important embedding post-processing step and discussed in more detail here for both transcriptomics and morphology: Celik, Safiye, et al. \"Building, benchmarking, and exploring perturbative maps of transcriptional and morphological data.\" PLOS Computational Biology 20.10 (2024): e1012463.\n- For the latest in high-performing microscopy MAEs: Pham, Chau, Juan C. Caicedo, and Bryan A. Plummer. \"ChA-MAEViT: Unifying Channel-Aware Masked Autoencoders and Multi-Channel Vision Transformers for Improved Cross-Channel Learning.\" arXiv preprint arXiv:2503.19331 (2025), NeurIPS 2025.\n- The Kraus et al. citation in this work points to the original workshop paper, which was actually expanded upon in the following archival paper accepted at CVPR. Furthermore, it may be worth considering incorporating the improvement applied to the MAE loss function (i.e., the fourier-transform reconstruction loss) to improve the quality of the morphology MAEs trained in this work: Kraus, Oren, et al. \"Masked autoencoders for microscopy are scalable learners of cellular biology.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024.\n- On the general topic of multimodal unification of transcriptomics data with morphological features, this paper seems particularly relevant: Bendidi, Ihab, et al. \"A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomics Representations through Morphological Features.\" arXiv preprint arXiv:2505.21317 (2025), ICML 2025.\n- Pretrained Dino-v2 has been evaluated on microscopy data in the following work, which also discusses further advancements to MAEs in the context of microscopy (e.g. the importance of curated training data and especially crucial is linear probing to identify intermediate layers of the ViT which obtain higher guide consistency and relationship prediction), which could be mentioned with the respect to e.g. the intermediate layer MST analysis in section A.6: Kenyon-Dean, Kian, et al. \"Vitally consistent: Scaling biological representation learning for cell microscopy.\" arXiv preprint arXiv:2411.02572 (2024), ICML 2025. (This work also publicly released of a CA-MAE ViT-S on HuggingFace, which is worth considering for baseline comparison: https://huggingface.co/recursionpharma/OpenPhenom)\n- On the topic of applying SAEs to biological foundation models, see: Donhauser, Konstantin, et al. \"Towards scientific discovery with dictionary learning: Extracting biological concepts from microscopy foundation models.\" arXiv preprint arXiv:2412.16247 (2024), ICML 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QqoaApYzgQ", "reviewer": "Reviewer_Gmu9", "summary": "The paper introduces PETRI, an early-fusion transformer framework designed to learn unified single-cell embeddings from unpaired cell morphology imaging and transcriptomic data. PETRI groups cells sharing the same perturbation into multimodal “documents” and performs masked joint reconstruction across modalities, enabling it to share information between imaging and gene expression while preserving modality-specific signals. Evaluated on both HepG2 (unpaired OPS and Perturb-seq) and Perturb-Multi (paired spatial transcriptomics) datasets, PETRI outperforms contrastive and unimodal baselines in recovering biologically meaningful perturbation profiles and cross-modal relationships. Analyses with sparse autoencoders further show that PETRI discovers interpretable multimodal cellular concepts linking morphological and molecular phenotypes, demonstrating a scalable approach for integrating heterogeneous biological assays without requiring cell-level pairing.", "questions": "- What specific limitations of existing contrastive or late-fusion multimodal methods (e.g., CLIP, MultiVI, CellCLIP) does PETRI aim to overcome? Can you provide conceptual or empirical evidence that early-fusion leads to superior integration under unpaired settings, rather than simply matching performance?\n\n- Beyond improving embedding quality, what concrete biological or biomedical applications does PETRI enable? Could the authors provide one clear use case where PETRI’s unified embeddings yield novel biological insight that unimodal or late-fusion methods cannot?\n\n- Why is reconstruction-based self-supervision preferable to contrastive or generative alignment (e.g., VAE-style, diffusion-based)? Did you observe any stability issues or mode collapse when using the joint reconstruction objective?\n\n- The paper claims PETRI learns meaningful cross-modal attention patterns. Can you quantify or statistically validate this, rather than relying solely on visual inspection? Could you measure how much of the reconstruction or embedding variance is attributable to cross-modal information versus unimodal content?\n\n-  Are the improvements over baselines statistically significant (e.g., via multiple random seeds or bootstrapping)? Can the authors report variance or confidence intervals to confirm the robustness of observed gains?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- PETRI focuses on integrating unpaired cellular modalities (images + transcriptomics), a setting that is much more realistic in biological experiments where cell-level pairing is infeasible.", "weaknesses": "- The paper motivates the need for integrating unpaired multimodal data but does not clearly articulate why existing late-fusion or generative alignment methods (e.g., cross-modal autoencoders, cross-domain VAEs) are insufficient.\n\n- Technical contribution is somewhat limited. The major novelty lies on the way to construct paired cell image-gene expression data from unpaired data (i.e., documents). Other technical parts — ViT/MAE for imaging, perceiver for transcriptomics, and multimodal masked modeling [1] are not novel idea in multimodal learning.\n\n- The choice of early fusion via joint reconstruction is not analytically or empirically contrasted with late-fusion or contrastive paradigms. It is unclear under what conditions early-fusion offers superior representational alignment or when it could fail (e.g., modalities with minimal mutual information).\n\n- The two core metrics—Guide Consistency and StringDB edge classification—capture perturbation-level similarity but do not fully reflect biological utility or multimodal alignment quality. There is no quantitative measure of cross-modal transfer or downstream biological inference.\n\n- Comparisons are restricted to unimodal MAEs, CLIP, and a few pre-trained models. Recent multimodal generative or diffusion-based frameworks (e.g., MoCa, scMultiVI, CellCLIP) are discussed but not empirically included, limiting fairness and completeness of the comparison.\n\n- The experiments lacks statistical rigor and scalability analysis. The authors did not provide error bars, confidence intervals, or multiple runs. Scalability to larger or more heterogeneous datasets, and sensitivity to document size, masking ratio, or latent dimension, are not thoroughly studied.\n\n- The study stops at embedding analysis; there is no demonstration that PETRI improves biological discovery tasks such as hit identification, functional gene grouping, or mechanism-of-action prediction.\n\nReferences:\n\n[1] Bachmann, Roman, et al. \"Multimae: Multi-modal multi-task masked autoencoders.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)