ViTally Consistent: Scaling Biological Representation Learning for Cell Microscopy
Authors: Kian Kenyon-Dean
Zitong Jerry Wang
John Urbanik
Konstantin Donhauser
Jason Hartford
Saber Saberian
Nil Sahin
Ihab Bendidi
Safiye Celik
Juan Sebastián Rodríguez Vera
Marta Fay
Imran S Haque
Oren Kraus
Venue: ICML 2025 poster (2025)
Keywords: MAE
drug discovery
microscopy
SSL
linear probing
biology
high-content screening
foundation models
Primary area: applications->health_medicine
TL;DR: We develop new methods to train scientific foundation models and evaluate a new 1.9B parameter MAE-G/8 (and others) along a large set of novel benchmarking tasks, demonstrating effective performance for many drug discovery usecases.

Abstract: Deriving insights from experimentally generated datasets requires methods that can account for random and systematic measurement errors and remove them in order to accurately represent the underlying effects of the conditions being tested. Here we present a framework for pretraining on large-scale microscopy datasets that includes three steps: (1) curating a set of diverse and self-consistent training samples, (2) scaling training of an appropriate foundation model architecture on this dataset, (3) evaluating intermediate layers of the trained model to identify the best representation for downstream tasks. Using this strategy, we present the largest foundation model for cell microscopy data to our knowledge, a new 1.9 billion-parameter ViT-G/8 MAE trained on over 8 billion microscopy image crops. Compared to a previous published ViT-L/8 MAE, our new model achieves a 60\% improvement in linear separability of genetic perturbations and obtains the best overall performance on whole-genome relationship recall, batch correction replicate consistency, and compound-gene activity prediction benchmarks.

ViTally Consistent: Scaling Biological Representation Learning for Cell Microscopy
Kian Kenyon-Dean
Zitong Jerry Wang', ! John Urbanik
Konstantin Donhauser" Jason Hartford 23
Saber Saberian
Nil Sahin Thab 2
Safiye Celik
Juan Sebastián Rodríguez Vera* 1 Marta Fay' Oren Kraus"
Imran Haque'
Abstract
Model
Deriving insights from experimentally generated datasets requires methods that can account for random and systematic measurement errors and remove them in order to accurately represent the underlying effects of the conditions being tested. Here we present a framework for pretraining on large-scale microscopy datasets that includes three steps: (1) curating a set of diverse and self- consistent training samples, (2) scaling training of an appropriate foundation model architecture on this dataset, (3) evaluating intermediate layers of the trained model to identify the best repre- sentation for downstream tasks. Using this strat- egy, we present the largest foundation model for cell mieroscopy data to our knowledge, new 1.9 pillion-parameter ViT-G/8 MAE trained on over 8 billion microscopy image crops. Com- pared to previous published VIT-L/8 MAE, our new model achieves 60% improvement in linear separability of genetic perturbations and obtains the best overall performance on whole-gan geleme relationship recall, batch correc- tion replicate consistency, and compound -ene activity prediction benchmarks
Procy laik
poalics
Dock searhh
307M 1.88
Canteed
Figure 1: (A) Overview of performance gain from differ- ent MAE pretraining and inference strategies. (B) Example whole-genome results for replicate consistency and bio- logical relationship recall on StringDB for models trained with different combinations of strategies, by model name and dataset (left right):
through microscopes studying how cells behave (Hooke (665). But with the advent of high throughput screening platforms that capture cells' responses to tens or hundreds of thousands of perturbations, we people study these images. Instead, need representa- tions of the cell images that capture the biologically mean- ingful information and capture the similarities and differ- ences in cells' response to perturbations.
longer rely on
State-of-the-art (SOTA) deep learning methods for learn- ing representations of microscopy leverage Vision Trans- formers (ViT) (Dosovitskiy al. 2020) trained with self- supervised learning (SSL) techniques (Balestriero et al. 2023 from large-scale screens (Doron et al.) 2023 Kim ct al. 2023 Bourriez et al. 2024). They have been shown to be very effective at representing the subtle changes in cell morphology response to perturbations, and masked autoencoders (He al.) 2022) in particular exhibit scaling laws for recovering known biological relationships (Kraus) et al. 2024). However, given that this data has to be col- lected experimentally, there is limit to how far we can scale these approaches before one runs out of data. To con- tinue to improve these representations, we need to use ex- perimental data as efficiently as possible. In this paper, we show how curated training datasets combined with care-
1. Introduction
Microscopy images capture detailed information about cell''s responses to perturbations: indeed many early biological discoveries were made by biologists looking
'Recursion VValence Labs 'University of Manchester. Cor- respondence to: Kian Kenyon-Dean <kian,kd @recursion.com.> Oren Kraus <oren.kraus @recursion.com>>
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025, Copyright 2025 by the author(s).
An earlier version of this work appeared at the NeurIPS 2024 Foundation Models for Science Workshop (Kenyon- al. 2024

---

ViTally Consistent: Scaling Biological Representation Learning for Cell Microscopy
ful probing of model representations allows us to scale to larger models that significantly improve recall and separa- tion of biological phenotypes.
PROTEASOME
NUCLEAR FORE
EXOSOME
TUBULIN
Careful curation of data has enabled better data efficiency in language models (Llama3 2024 Eldan & Lij 2023 Javaheripi et al. 2023). However, unlike language, we can- not simply rely on prior knowledge about the data source to weigh data (e.g. by up-weighting trusted sources like arXiv or wikipedia). Instead, we curate training dataset that is relatively uniform with respect to the observed phenotypes. Because many perturbations will not induce morphologi- cal change, a random sample of images across all perturba- tions will be dominated by cells that look like unperturbed cells. We show that one can use weaker models to filter im- ages perturbations that induce a distinct morphological phenotype, thereby reducing the absolute training set size but increasing the relative diversity. The resulting dataset, Phenoprints- 16M, has 5x fewer images than the dataset used to train Kraus et a masked autoencoder, but we find that it leads to better performance when combined with scaled compute through longer self-supervised training.
Figure 2:
Samples for subset of groups in Anax 40-class functional gene group classification task.
Kraus et al. (2023) extend to the multi-billion parameter model regime across a wide variety of newly examined
biologically-motivted benchmark tasks.
2. Related work
Dataset Curation for Foundation Models. Dataset ration is crucial for enhancing the efficiency of founda- tion models, especially in large-scale contexts. Usual ap- proaches to dataset construction are inspired by the image retrieval community (Weinzaepfel et al., 2022] Radenovid et al. 2018 [Berman et al., 2019). Existing methods often utilize pre- pre-trained models for filtering and pruning, such as vision-language models to discard irrelevant pairs (Schuh-] mann et al., 2021), semantic deduplication to remove dundancy (Abbas et al. 2023), and prototypicality-base approaches to retain representative data (Sorscher et al. 2022). However, these techniques are less effective for HCS, where redundancy, variability, and subtle morpho- logical differences make conventional filtering challenging. Our work addresses these limitations by building on Celik et al. (2024)'s perturbation consistency framework to rate balanced dataset of images across semantic classes, which is vital for effective learning under the masked ob- jectives (Zhang et al. 2022).
Data curation gives one way of improving model, but we can also optimize over the layer that representation. Traditionally. the finally hidden layer has been used (Doron et al. 2023 Kim et al.] 2023 Kraus eet al., 2.24), but there is some evidence from mechanis- tic interpretability [Templeton et al. (204) that earlier lay- ers may learn more abstract features, which may provide more useful features for biological tasks. We show that us- ing a linear probing proxy task, are able to cheaply find the best performing intermediate block for many models. We find that using intermediate layers leads to better per- formance on downstream whole-genome relationship recall and perturbational replicate consistency at a lower compu- tational inference cost, and that these results are consistent across a variety SSL ViTs trained on either microscopy or natural images. Our proxy task involves fitting set of biological linear probing tasks to evaluate representations learned by intermediate VITs blocks for microscopy data ($ () We find this task correlates extremely well with bio- logical recall benchmarks which allows us to reliably probe the model's representations.
to construct
Layer-wise Analysis of Deep Neural networks. Recent work suggests that intermediate layers (or, blocks) in large ViTs may achieve superior performance on certain linear probing tasks compared to the final encoder layer (Evei) et al. 2022 Dehghani et al., 2023). Alkin et al. (2024) ported that intermediate layers in large MAE-ViT ViT's (ViT-L, ViT-H) have superior ImageNet- IK k-NN accuracy, likely because later encoder layers become more optimized for the reconstruction task.
In summary, we present series of data curation and model probing techniques that allow experimental microscopy data. By combining these in- sights, we are able to train a new foundation model, MAE-G/8, a 1.9 billion parameter ViT-G/8 MAE trained on Phenoprints- 16M for 48,000 H100 GPU hours on more than billion samples drawn from the curated dataset (Fig- ure I/A.3 $ 3..2 resulting in significant improvements across range of challenging biological benchmarks. These re- sults indicate that the scaling properties first identified by
more effectively use
Evaluating Representations for Drug Discovery. Eval- uating the quality of biological representation learning methods for drug discovery remains challenging, as ground truth data is sparse, noisy, biased to well-studied dis- eases and pathways, and poorly annotated. Metrics have been proposed that use mean average precision (Kalinin) ct al. 2024) or AUC ROC (Sivanandan et al., 2023) to assesses how similar related samples are represented, in-
2

---

ViTally Consistent: Scaling Biological Representation Learning for Cell Microscopy
cluding replicates of the same perturbation or different per- turbations with similar annotated biological activities. Re-
was computed for CRISPR guides, siRNAs, and particular concentrations of small molecules across replicates of the same perturbation. P-values were computed for each gene and each (perturbation, concentration) pair. When multiple experiments existed for the same condition, we combined p-values using the Cauchy Combination test (Liu & Xie 2018).
cently, Celik et al. (2024) introduced terminology for de- scribing perturbative "maps of biology", in which repre- sentations of perturbations in HCS data can be placed in unified, relatable embedding spaces allowing for the gener- ation of genome-scale sets of pairwise comparisons. Here we leverage the biological relationship recall benchmark proposed by [Celik et al.] (2024), which assess how well known relationships between pairs of perturbations are re- called among the most similar or dissimilar embeddings. Computing reliable versions of these relationship bench- marks with HCS data is particularly expensive as they re- quire genome- wide embeddings to be inferred for hundreds of millions of image crops from the genome-wide RxRx3 microscopy screen (aay et al. 2023).
We repeated this procedure with a weakly supervised learn- ing (WSL) model trained on RxRx1 (Sypetkowski et al. 2023) and filtered to perturbations where any condition had a p-value < 0.01 in either the MAE-L/8 or WSL model. This process reduced our original dataset of 93M sam- ples to 16M, which we refer to as Phenoprints- 16M. While some redundancy remains when distinct perturbations have the same effect, the proportion of samples with that dif- fer from negative controls increased substantially with little decrease in overall diversity. We believe that iteratively re- peating this process with the best models from previous it- erations to guide data selection for subsequent models may be a viable strategy.
3. Vision Transformers for Microscopy Images
We train and evaluate various vision transformers (ViTs, Table 4) as encoders to extract feature embeddings from 256x266xt (HxWxC) microscopy image crops (Figure 2).
3.2. Models
Baselines, We compare to several non-finetuned baseline ViT image encoders: three different Dino-v2 backbones (Oquab et al. (024) (with 4 register tokens (Darcet et al. 2024)) trained on a curated non-biological natural image
3.1. Training Dataset Curation
Many academic and industry labs have adopted the Cell Painting imaging protocol (Bray et al. 2016). which mul- tiplexes fluorescent dyes to reveal eight broadly relevant cellular components. The datasets used here contain a six- channel implementation of Cell Painting (Figure 2), as well as brightfield images, spanning 100,000s of chemical and genetic perturbations applied dozens of cell types (Kraus) et al.] 2024). In these datasets, cells that look like un- perturbed cells tend to be very over-represented because many perturbations do no induce a morphological change. Some morphological changes are also far more common (c.g. many perturbations will kill cells, resulting in a rel- atively high proportion of dead cell morphological pheno- type). This results in significant imbalance in the morpho- logical phenotypes that the models learn to reconstruct.
dataset; MAE ViT-L/16 trained on Imagenet-21k (He) et al. 2022); and an untrained ViT-S/16. We found that channel- -wise selndatannrii worked best as the im- age normalization preprocessing for these baselines, and that the class token was slightly better than the global pool of the patch tokens (except for MAE). Convolutional weights in the patch embedding layer were repeated em- bed 6 channel images when using models trained on RGB datasets (Wightman 2019).
Prior work. Our primary point of comparison is with respect to the best pretrained foundation model presented by Kraus et al.] (022)), the MAE-ViT-L/S trained on RPI-93M. This MAE-L/8 was trained for approximately 40 epochs, learning from over 3.5 billion image crops, using the L2 mean squared error loss function plus an additional Fourier domain reconstruction loss term.
To address this, we constructed an aggressively curated training dataset ($ A. To learn an initial representation, we began by reproducing the MAF-L/8 model of Kraus jet al. (2024) on a dataset of similar size consisting of 93 million HCS images. Using this representation, we first fil- tered perturbations that did not induce consistent morpho- logical changes to cells. To perform this filtering, we uti- Mizee(Celik et al. (2024)'s non-parametric perturbation con- sistency test (S (A3)) after correcting for batch effects using Typical Variation Normalization (Ando et al. 2017 Kraus ct al. 2024). This test was applied within each experiment for computational efficiency, and we restricted the analysis to wells containing single perturbations. This consistency
CA-MAE-S/16 trained on RxRx3. We trained a new channel aggnosic MAE (Kraus et al. 2024) ViT-S/16 on the RxRx3 dataset (Fay et al., 2023 for 100 epochs. Channel- agnostic ViT' tokenize each image channel separately with shared patch embedding weights and leverage the dynamic sequence length of transformers with repeated positional encodings to train ViTs that can process images with vary- ing numbers of channels (Bao et al. 2024; Bourriez et al.] 2024
Kraus et al. 2024 Kraus et al. (2024) demonstrate
3

---

ViTally Consistent: Scaling Biological Representation Learning for Cell Microscopy
diate blocks within an encoder can provide better repre- sentation compared the final block (Alkin et al., 2024). Unfortunately, it is infeasible to search for the best block by simply performing whole-genome evaluation on each block of a large model because the evaluation is extremely time-consuming and resource intensive. For example, eval- uating the final block of MAE-G/8 required 4,000 L4 GPU hours just for inference ($ 5 We demonstrate that using block-wise linear probes provides insights into the qual- ity of biological features extracted by these models in their intermediate blocks, allowing us to trim the model to an earlier block to both reduce inference costs and improve representation quality.
(slarree accurssy
0:375
0.300
0.2
011
0.000
1/0
Normalied block pasition
Nomalized black position
(a) RxRx1 siRNA knockdown classification.
0.30
Balanced accurecy
Balanced accurcy
0.38
00:5-
0:22
0.34
VITLMNEE
NAE US (16H)
MAE-GAI G
0.32
Dino G/l4
Our block-wise search consists of training a logistic regres- sion model (linear probe) on the output features of each transformer block to predict either the gene that was per- turbed or the functional group that the gene belongs to, and test performance on held-out experiments ($ A.4).. We de- fine the optimal block by for a probing task as the block whose output features achieve the highest test balanced accuracy when trained on the probing task, across all N blocks of the encoder,
MAE-UB
0.0
untrained VITS
0.10
CA-
0.00
1/3
Redmalicedd block position
Normalized position
position
(b) Anax functional gene group classification.
Figure 3: Block-wise validation set linear probe results comparing ViT models pretrained on cell microscopy im- ages (left) versus natural images (right); note the differ- ence y-axes. (a) 1139-class RxRx1 I SiRNA knockdown classification (Sypetkowski al.] (023); (b) 40-class Anax functional gene group classification on HUVEC cell im- ages from RxRx3 CRISPR knockouts (Fay et al. 2023).
6" = arg max BalancedAccuracy(e" ("), (1.2.....
where (0)) are output features from block b of a VIT. Per- formance on our linear probing tasks can be viewed as a measure of linear separability of a feature space across ex- perimental batches.
that the large MAES with 8x8 patch size perform either bet- ter or the same as the 16x16 thannel-agnostir variants for consistently 6-channel data, so we opted to train standard MABs for the following two new models since they require fewer tokens at inference time.
RxRx1 1139-class siRNA genetic perturbation classifi- cation. We expect high quality representations of cell im- ages generate similar embeddings for cells with the same perturbation, hence simple linear probe should be able to predict gene perturbation from these representation reason- ably well. We train linear probes on the publicly-available RxRxI dataset (Sypetkowski et al. 2023) which consists of 125,510 high-resolution fluorescence microscopy images of human cells under 1,138 siRNA-induced gene knock- downs (plus unperturbed controls) across four cell types (HepG2, HUVEC, U2OS, RPE). These gene knockdowns produce strong phenotypes which makes the prediction task more feasible.
MAE-L/8 trained on P1enoprints-16M. Holding the model backbone constant compared to the MAE-VIT-L/8 by (Kraus et al., 2024). we assess the impact of our curated dataset in contrast to the 93M dataset by training a new VIT-L/8 MAE for 500 epochs on Phenoprints-166 16...
MAE-G/8 trained on Phenoprints-I6M. Holding the dataset constant compared MAE-L/8 above, we assess the impact of increased model scale in terms of parameters by training new ViT-Gigantic MAE with nearly 1.9 bil- lion parameters for 500 epochs on Phenoprints-16N 16M. Train- ing this model required 256 H100 GPUs running in parallel for over week. See $ A.2 for other hyperparameter set- tings we used for model training.
We found that, for MAE-G/8. the best features came from intermediate block b" 38 (out of 48) of the encoder, achieving a balanced accuracy (0.51) that is 8.5% greater compared to its final block's output features (Figure 3a left). Additionally, these features achieved 60% greater accuracy than the typically used final block of MAE- L/8+ (Kraus et al., 2024). We observed similar trends for ViT models pretrained on natural images. For example, DINO-G/14 and ViT-L/16 MAE trained on non-biological
4. Linear probing representation learning across VIT blocks
We improve the quality of our learned image representa- tions by leveraging previous findings that suggest interme-
4

---

ViTally Consistent: Scaling Biological Representation Learning for Cell Microscopy
Whole genome StringDe Recall
100
46
CA-MAE-S/16 (Rafo3),
44
Pearson 0.95
Pearson r=0.89
MAE-LU (RP1-93M). D
42
MAF-UB (RPI-93M), 61115
MAE-LB [P--18M)
MAE LAE LB (FP-16M), -00
38
G/8 (PP-16M),
MAE G/8 (PP-16M), 6"8.
36-
VIT-L/16 MAL, b:
Whole Whele genome Replicate Consistency (K)
VIT-1/16 MAE, =11
65
Dino S/14, b
660
Dino S124, b =
Dino-L/14,
55
Dino U14, 6 122
Dino-G/4,
50
Pearson 0.97
Pearson 0,92
Dino-G/14, b"
.5
-00
35
20
24
266
28
-30
.15
.5
.55
40
Vulidation accur un Anax lnsar probe (40 classes)
Vilication accuracy on Rattec linear probe (1139 classes)
Figure 4: Correlations between validation set linear probing (Figure 3 on Anax and RxRx1 for best and last blocks (Eq.1 compared to downstream shole-genome benchmarks (Table D for biological relationship recall on StringDB at 0.05-0.95 gene-gene cosine similarity threshold and replicate consistency KS statistic. Models with bold borders are trimmed, red natural image baseline models and blue trained on microscopy.
natural image data have their best features at blocks that positioned within the first half of the encoder. For ViT- L/16 MAE, the performance of the best block is 27% higher compared to its final block output features that are typi- cally used downstream tasks. The higher performance observed for intermediate blocks does not appear to be an intrinsic feature of the ViT architecture an untrained ViT did not exhibit such parabolic trend (Figure 3al right).
of cell type.
As shown in Figure 36 MAE-G/8 significantly outper- forms other models Anax group linear probe classifi- cation. The best representations once again are obtained from an intermediate block, achieving balanced accuracy (0.32) that is 5% greater compared its final block. We observed similar trends for ViT models pretrained natu- ral images and representations computed from microscopy images of other cell types/conditions ($ A.5 Figure 6
Anax 40-class functional gene group classification. Bi- ologically meaningful representation of microscopy im- ages of genetically perturbed cells should capture func- tional relationships between genes, hence a simple linear probe should be able to predict functional gene groups when trained on these representations. We curated small subset of 80,000 wells from RxRx3 (Fay al. 2023 to evaluate linear probes functional group prediction. We also evaluated similar whole genome knockout screens with ARPE-19 and an additional population of HUVEC cells with soluble TNFa added to all wells. We manu- ally curated Anax, set of 40 functionally-diverse gene groups containing 348 genes, with details provided in ($ A.8).. Examples of groups include major protein com- metabolic
In Figure 4. we observe that performance on this novel linear probing task correlates strongly with downstream whole-genome benchmarks across models (Table T). whether they are trained microscopy data or natural im- ages, achieving overall rank correlation p 0.97 with whole-gena chole-genome StringDB recall and p 0.91 with whole- genome replicate consistency. This strong correlation is crucial as it allows us to trim our model to the block with the best linear probe performance as a way to improve the quality of our representations for the whole-genome (T ble
5. Whole-genome benchmarking
plexes (e.g. proteasome, babosome-smalllarggee pathways (e.g. Krebs cycle) and signaling pathways (e.g. calcium signaling) (Figure 2p. These groups span broad biological processes that are conserved across cell types linear separability of these groups would likely indicate that representations biologically meaningful regardless
Table presents our benchmarks computed across the whole-genome. These evaluate the genomic representa- tions obtained for each model by aggregating millions of embeddings of cell images spanning 100,000 of genetic knockout perturbations (17,063 genes 6 single guide
5

---

ViTally Consistent: Scaling Biological Representation Learning for Cell Microscopy
Table I: Multivariate known biological relationship re- call and univariate replicate consistency benchmarks by model, encoding block b, aggregated recall over a total of 145,447 possible gene-gene relationships annotations ac- cumulated across five benchmark databases, and replicate consistency test statistics. The trimmed models used lin- ear probes to select an earlier block as the feature encoder (Fig. B Each benchmark is computed over TVN-aligned aaggaaggeetteed model embeddings, for each one higher is better, and the best overall result is in bold. We report the Recall % of biological relationships obtained below the 5th and above the 95th percentiles (0.05-0.95) of all gene-gene cosine similarities against the null distribution, as per (Celik et al. 2024] Kraus et al., 2024), with mean + the standard deviation computed over 3 different random seeds of sampling the null distribution in each benchmark run. For replicate consistency, we report the Komologorov- Smirnov (KS) and Cramer-von Mises (CM) test statistics.
We present the multivariate biological relationship recall benchmarks proposed by Celik et al.. (2024) and origi- nally evaluated for MAES by Kraus et al. 20233 2024).
These metrics evaluate how many annotated wiir-wise re- lationships are recalled from public databases (CORUM, Mu.MAP, Reactome-PPI Signor, StringDB) in the extrem- ities of a ranked list of cosine similarities of all pair-wise post-processee embeddings (details in $ AA66).
To ensure embeddings represent technical replicates of perturbations consistently, we also evaluate model perfor- mance on replicate consistency based on the experimental design used in the RxRx3 dataset. Specifically, we compare the similarity of the embedding for corresponding wells across different experiments via non-parametric statisti- cal test. The test statistic measures the difference between the perturbation replicates' similarity distribution and an empirical null distribution, with larger values indicating greater consistency ($ A.7]. To compare models, we sum- marize the resulting statistics over all technical replicates in RxRx3 by taking their median, reported in columns KS and CM (Kolmogorov-Smme Smirnov and Cramer-von Mises test statistics) in Table I
Model backbone
Recall %
KS
CM
Baseline ViTs VIT-S/16 Untrained
12
34.2 08
.30
4.3
ViT-L/16 ImgNet MAE
24
37.4 + .66
.34
5.1
Not suprisingly, even the smallest CA-MAE-S/16 trained on microscopy data outperforms the largest baselines ViTs trained on natural images. Training self-supervised on our novel carefully curated Phenoprints- dataset improves the performance of the MAES, as does trimming to an early layer detected by linear probes, while the most scaled model MAE-G/8 achieves the best overall performance in all respects when trimmed. Compared to the best previ- ously SOTA model at its normally used final block, MAE- L/8 RPI-93M (Kraus et al. 2023). MAE-G/8 trimmed ob- tains: (a) the highest statistically significant gain in rela- tionship recall over the previous SOTA, achieving a z-score of improvement of 5.21; (b) a 21% improvement in the KS statistic (.52-+.33:: and, (c) a 48% improvement in the replicate consistency CM statistic
[rimmed
11
37.8 + .05
.35
5.8
Baseline Dino ViTs VIT-S/14 Dino-V2
12
35.6 .99
.34
5.6
trimmed
5
37.2 +
.35
6.0
ViT-L/14 D2
24
35.7 + .5
.34
5.3
trimmed
12
38.8 1 .04
.36
5.9
VIT-G/14 Dino-V2
40
32.7
+ 100
.29
3.8
trimmed
16
37.5
+ 113
.33
5.2
Microscopy MAEs
CA-MAE-S/16 RxRx3
12
39.6 + 10
.47
10.4
WAE-L/8RPI-99M.
24
44.4 +
.52
12.3
trimmed
15
44.3
1.12
.57
15.2
MAE-L/8PP-166
24
44.4
.59
16.2
trimmed
20
44.7
+ .6
.59
16.2
MAE-G/8 PP-16M
48
45.4
.07
.60
16.4
Linear probing to select optimal ViT blocks leads to sig- nificant improvements even when applied to Dino- V2 and MAE models pretrained on natural images. Dino-V? V2 ViT-G is dramatically better in terms of recall and consis- teney with the early block embeddings at b" = 16 rather than the final embedding from b 40 (which performs worse than a random untrained ViT-S). Dino- ViT-S also observes improvements by using b* = 5 rather than b = 12 and outperforms Dino-V2 V2 VIT-G in replicate consistency, while the trimmed Dino-V2 ViT-L obtains the best recall among the baselines. This finding is important for the sci- entific community to consider when applying MI. foun- dation models to experimental data, as is common to take embeddings from the final layer default strategy even when processing potentially out- out-of-diitribution im-
trimmed
38
45.4
.15
.63
18.2
RNAs each) on HUVEC cells from RxRx3 (Fay et al.)
2023). Computing these benchmarks for HCS screens typi- cally requires inferring 140 million crops from the genome- wide RxRx3 microscopy screen (Kraus et al. 2023) (64 tiled crops per each of the 2.2 million wells), but, reduce compute costs, we discard the outer ring of crops, leaving the 36 center non-edge crops for each well. This requires 80 million forward passes to comprehensively evaluate new encoder. After inference, we use typical variation nor- malization (Ando et al.) 2017) and chromosome arm bias correction (Lazar et al. 2024 to post-process the embed- dings and aggregate them the gene-level.
6

---

ViTally Consistent: Scaling Biological Representation Learning for Cell Microscopy
Table 2:
Comparing MAES to manually extracted CellProfiler (McQuin et al. 2018 features on biological relationship
recall. Reported at 0.05-0.95 cosine threshold on the public JUMP-CP image dataset (Chnddaaseaaann et al. 2023), which generated by completely different labs and assay protocols compared to the images used for pretraining. Fach result has a standard deviation <
=.0033, spanning gene-gene relationships across nearly 8,000 gene-knockouts.
Model backbone
b
Pretraining data
CORUM
hu.MAP
Reactome
StringDB
CellProfiler
N/A
.219
.184
.131
.191
CA-MAF-S/16
12
RxRx3
.233
.199
.154
.214
MAE-L/8
24
RPI-93M
.248
.208
.160
.226
MAE-G/8 trimmed
38
Phenoprints- 16M
.264
.215
.165
.235
ages [Lastufka al. 2024). which, as we have shown, can significantly hinder results.
We also evaluate preliminary version of SSL VIT- L/16 trained with the Dino-V2 algorithm on the RxRx3 microscopy dataset; however, we found that it underper- formed compared to the CellProfiler 4 baseline ($ A. (Kamentsky et al. 2011) and, unlike the MAEs we trained, significantly overfit during pretraining ($ A.9). As such, we did not pursue additional analysis on this model. We leave further development of Dino-V2 for microscopy data to future work, as we were unable to determine an effective recipe for applying Dino SSL pretraining on large-scale mi- croscopy data.
5.1. Performance on external data (JUMP-CP)
In order validate that the MAES generalize to entirely novel data, we evaluated subset of models completely external public data generated by different assays and from a variety of different labs as produced by the JUMP-CP consortium (Chandrasekaran et al. 2023), Table 2 presents these results using the relationship recall benchmarks of (Celik et al. 2024), noting that only subset of 7,976 gene- knockouts are covered by this dataset. For post-processing embeddings, we use PCA with center-scaling for standard batch correction alignment. We observe that the MAEs per- form better than the CellProñiler manual feature extraction baseline ($ IT (Carpenter et al. 2006; Kamentsky et al.) 2011 and that the general trend is maintained with the trimmed MAE-G/8 obtaining the best recall overall. No- tably, recall on JUMP-CP considerably lower than on RxRx3 (Table T) likely due to different assay protocols and more variance in the data.
To evaluate each model, we first inference all 222,601 x 4 crops and then average the 4 embeddings to the well-level. Then, to perform standard batch correction alignment, we use the "EMPTY". unperturbed wells as our control pop- ulation. We fit PCA on those control embeddings, use it transform the rest, and then fit separate standard-scaler on each batch's 's controls to transform the rest. This simpli- fied alignment strategy empirically performed better than TVN on this dataset.
We present results for the benchmark measuring zero-shot prediction of compound-gene activity using cosine simi- larities between embeddings (Figure 5). This measures, for each compound, whether the cosine similarities from a model's ''s embeddings correctly rank the compound's 's known target genes higher than a randomly sampled set of other genes from a ground truth dataset. [Table 3] provides ex- act values along the max axis, which captures the strongest potential interaction regardless of concentration. The rela- tive ranking of model performance, holds as expected from the results in 4 and $ and trimming benefits both MAE- L/8 and MAE-G/B with MAE-G/8 offering a 42% (3.77 + 5.38) relative improvement over the method from previous work in predicting compound-gen -ene activity.
6. RxRx3-core benchmarking
RxRx3-core (Kraus et al., 2025) is a publicly available benchmarking dataset for assessing biological capabilities of computer vision models. RxRx3-core includes labeled images (compressed to JPEG-2000) of 735 genetic knock- outs and 1,674 mmall-molecule perturbations across eight concentrations drawn from 222,601 wells (512 x 512 x 6 pixel center-crops) drawn from the larger RxRx3 dataset.
We evaluate a random choice baseline, CellProfiler, the CA-MAE-S/16 model, the MAE-L/8 model from previous work (Kraus al., 2023), and the MAE-G/8. We eval- uate both the trimmed and full-lengti version of the latter two models to determine the impact of our model-trimming strategy for the most performant models in this context.
7. Discussion and Conclusions
This results in this work demonstrate that: (1) within the context of biological imaging, trimming many ViTs to an earlier block leads to stronger biological linearity and im-
httpes///uggiiggraae..o/o recurssiompharmm//rrx333
7

---

ViTally Consistent: Scaling Biological Representation Learning for Cell Microscopy
Random embeddings
OpenPhenom-5116
0.3
LT (RPI trimmed
Precision
IPhenoprint-1GMM
0.28
VIT G/B iPhenoprints 16K) trimmed
0.44
0.22
0.0025
0.025
0.25
1.0
Compound concentration
Figure 5: Mean average precision performance on RxRx3-core public benchmark in predicting compound activity against annotated gene targets, across all compound concentrations with error bars for 100 runs of the benchmark with different random seeds (able 3).
Table 3: Performance on public RxRx3-core compound- gene benehmark, measuring mean (# STD over 100 ran- dom seeds) average precision in predicting compound ac- tivity against target genes with its z-score of improvement over the random baseline.
relevant tasks, even in comparison model trained 5x larger dataset. Indeed, even after training the MAE-GA on more than eight billion crops from a 16-million-image dataset, validation reconstruction loss continued to im- prove. This suggests that increasing the model's s parameters further on this data may continue yield improvements.
Model
Avg. prec.
Z-score
More broadly, this work proposes reusable recipe for training and extracting optimal representations from fully self-supervised models trained on experimental data. The pattern we use can be applied to other domains that con- tain data from repeated experiments but without accurate ground truth labels. Specifically, we recommend: (a) cu- rating the training set by identifying diverse sets of sam- ples that represented consistently, e.g., by using pre- existing model to select such samples; (b) training a scaled transformer based model using self-supervised learning technique, such as masked autoencoding: and, (c) inspect- ing the performance of the trained transformer, and all baselines, at every block to identify the optimal layer for representing the data.
Random baseline
0,222 + 007
0.00
CellProfiler
0,274 + .19
2.55
Dino- V2 ViT-L/16, RxRx3
0,258 1 .15
2.13
CA-MAE-S/16, RxRx3
0.273 +
2.90
MAE-L/8, RPI-93M
0.290 .017
3.77
trimmed
0.299 1 1066
4.49
MAE-G/8, PP-16M
0.302 + .15
4.79
trimmed
0.309 + .155
5.38
proved performance downstream tasks in addition cheaper inference costs (Figure 3); (2) linear probing per- formance on subset of genetic perturbations correlates strongly with downstream performance on whole-genome benchmarks and can be used to optimize which block se- lected for representing the whole-genome (Figure 4); (3) the most scaled model, MAE-G/8 trained on the specially curated dataset Phenoprints-161 for 500 epochs, obtains the overall best performance across all benchmarks and lin- ear probes, providing further evidence for the scaling hy- pothesis in biological image data (Table I, Table 2 Ta- ble $ A.10). We have found that intentionally scaling training compute and parameters of MAEs for microscopy on curated data can benefit a wide variety of biologically
Limitations and reproducibility
In this work, we evaluated publicly available SSL ViTs as baselines and trained new MAE models trained on a specially curated microscopy dataset. Our preliminary at- tempts train ViTs with Dino on this microscopy data en- countered suboptimal performance ($ Consequently, allocated time and compute budget investigate scal- ing MAE to VIT-G/8 curated data. We recognize the potential for other SSL pretraining regimes and fine-tuning
8

---

ViTally Consistent: Scaling Biological Representation Learning for Cell Microscopy
2024 Hondru et al., 2024; Khan & Fang 2024 Alkin al. 2024) ori-
strategies (Singh al. 2023
Lehner et al.
Berman, M., Jégou, H., Vedaldi, A., Kokkinos, I., and Douze, M. Multigrain: unified image embedding for classes and instances, 2019. URL https:///axx
ented for microscopy data to lead to future improvements on these tasks. We publicly release the inference, recon- struction visualization and benchmarking code along with the full weights for CA-MAE ViT-S/16
orq hrc/abb/1900..00 05509
Bourriez, N., Bendidi, I., Cohen, E., Watkinson, G., Sanchez, M., Bollot, G., and Genovesio, A. Chada-vit Channel adaptive attention for joint representation learn- ing of heterogeneous microscopy images. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning in its applications to the sci- ences. There are many potential societal consequences of our work, especially relating to the discovery of new bi- ological relationships between genes, potential drug treat- ments for diseases, and overall accelerating the process of drug development. At the same time, the predictions of these models are not guaranteed to be correct, which is why utmost care must be taken to validate safety and efficacy in orthogonal assays and early-stage clinical trials
Bray, M.-A,, Singh, S., Han, H., Davis, C. T., Borgeson, B., Hartland, C., Kost-Alimova, M., Gustafsdottir, S. M., Gibson, C. C., and Carpenter, A. E. Cell Painting, high- content image-based assay for morphological profiling using multiplexed fluorescent dyes. Nature Protocols, 11 (9):1757-1774, 2016. ISSN 1754-2189. doi: 10.1038/ nprot.2016.105 105.
Carpenter, A. E., Jones, T. R., Lamprecht, M. R., Clarke, C., Kang. I. H., Friman, O., Guertin, D. A., Chang. J. H., Lindquist, R. A., Moffat, J., et al. Cellprofiler: im- age analysis software for identifying and quantifying cell phenotypes. Genome biology, 7:1-11, 2006.
References
Abbas, A. K. M., Tirumala, K., Simig, D., Ganguli, S., and Morcos, A. S. Semdedup: Data-efficient learning at web-scale through semantic deduplication. In ICLR 2023 Workshop on Multimodal Representation Learn- ing: Perks and Pitfalls, 2023.
Celik, S., Hütter, J.-C., Carlos, S. M., Lazar, N. H., Mohan, R., Tillinghast, C.,, Biancalani, T., Fay, M. M., Earn- shaw, B. A., and Haque, I. S. Building, benchmark- ing, and exploring perturbative maps of transcriptional and morphological data. PLOS Computational Biol- ogy, 20(10): 10 2024. doi: 10. 10. 1012463. URL httpp:: /tt:://doo..ooo www.o.oor//..110
Alkin, B., Miklautz, L., Hochreiter, S., and Brandstetter, J. Mim-refiner: A contrastive learning boost from in- termediate pre-trained representations. In NeurlPS 2024 Workshop: Self-Supervised Learning-Theor and Prac- tice, 2024.
Journal paarnall.cbb.. 1012463
Ando, D. M., McLean, C. Y., and Berndl, M. Improv- ing Phenotypic Measurements in High-Content Imaging Screens. bioRxiv, pp. 161422, 2017. doi: 10.1101/ 161422.
Chandrasekaran, S. N., Ackerman, J., Alix, E., Ando, D. M., Arevalo, J., Bennion, M., Boisseau, N., Borowa, A., Boyd, J. D., Brino, L., et al. Jump cell painting dataset: morphological impact of 136,000 chemical and genetic perturbations. bioRxiv, pp. 2023 03, 2023.
Balestriero, R., Ibrahim, M., Sobal, V., Morcos, A.,
Shekhar, S., Goldstein, T., Bordes, F., Bardes, A., Mi- alon, G., Tian, Y., Schwarzschild, A., Wilson, A. G., Geiping, J., Garrido, Q. Fermandez, P., Bar, A., Pir- siavash, H., LeCun, Y., and Goldblum, M. A Cook- book of Self-Supervised Learning. arXiv, 2023. doi: 10.48550//rxi..2344. 12210.
Chen, X., Liang, C., Huang, D., Real, E.,, Wang, K., Pham, H., Dong, X., Luong, T., Hsieh, Cu, Y., et al. Sym- bolic discovery of optimization algorithms. In Advances in Neural Information Processing Systems, volume 36, pp. 49205 49233, 2023.
Darcet, T. Oquab, M., Mairal, J., and Bojanowski, P. Vi- sion transformers need registers. In The Twelfth Inter- national Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-14. 2024.
Bao, Y., Sivanandan, S., and Karaletsos, T. Channel vi- sion transformers: An image is worth 1 x 16 x 16 words. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024.
Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heek, J., Gilmer, J., Steiner, A. P., Caron, M., Geirhos, R., Alabdulmohsin, I., et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning, pp. 7480-7512. PMLR, 2023.
mi nicroscopy
http://///agggggaaae.e.//////aaa OpenPhenom

---

ViTally Consistent: Scaling Biological Representation Learning for Cell Microscopy
Doron, M., Moutakanni, T., Chen, Z. S., Moshkov, N., Caron, M., Touvron, H., Bojanowski, P., Pernice, W. M., and Caicedo, J. C. Unbiased single-cell morphology with self self-supervised vision transformers. bioRxiv, 2023. doi: 10. 10.1101/2201 N0.1100/202..06 16.545559.
Hondru, V., Croitoru, F. A., Minaee, S., Ionescu, R. T., and Sebe, N. Masked image modeling: A survey. arXiv preprint arXiv:2448.06600 06687, 2024.
Hooke, R. Micrographia: Or Some Physiological Descrip- tions of Minute Bodies Made by Magnifying Glasses with Observations and Inquiries Thereupon. Jo. Martyn and Ja. Allestry, Printers to the Royal Society, London, 1665.
Dosovitskiy, A., Beyer, I.., Kolesnikov, A., Weissenbom, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Repre- sentations (ICLR), 2020.
Javaheripi, M., Bubeck, S., Abdin, M., Aneja, J., Bubeck, S., Mendes, C. C. T., Chen, W., Del Giorno, A., EI- dan, R., Gopi, S., al. Phi-2: The surprising power of small language models. Microsoft Research Blog. (3):3, 2023.
Drew, K. Lee, C., Huizar, R. L.,, Tu, F., Borgeson, B., McWhite, C. D., Ma, Y., Wallingford, J. B., and Mar- cotte, F. M. Integration of over 9,000 mass spectrometry experiments builds a global map of human protein com- plexes. Molecular Systems Biology, 13(6):932, 2017. ISSN 1744-4292. doi: 10.15252/ms.22117745)
Kalinin, A. A., Arevalo, J., Vulliard, L., Serrano, E., Tsang, H., Bornholdt, M., Rajwa, B., Carpenter, A. E., Way, G. P., and Singh, S. A versatile information retrieval framework for evaluating profile strength and similarity. bioRxiv, pp. 2024.04.01.58766 4 2024. doi: 10.1101/ 2024.04.01.587631
Eldan, R. and Li, Y. Tinystories: How small can lan- guage models be and still speak coherent english? arXiv preprint arXiv:2305. 07759, 2023.
Kamentsky, L.,, Jones, T. R., Fraser, A., Bray, M.-A., Lo- gan, D. J., Madden, K. L., Ljosa, V., Rueden, C., Eliceiri, K. W., and Carpenter, A. E. Improved structure, func- tion and compatibility for cellprofiler: modular high- throughput image analysis software. Bioinformatics, 27 (8): 19:1119-1180 2011.
Evei, U., Dumoulin, V., Larochelle, H., and Mozer, M. C. Head2toe: Utilizing intermediate representations for bet- ter transfer learning. In International Conference on Ma- chine Learning, pp. 6009-6033 PMLR, 2022.
Fay, M. M., Kraus, O., Victors, M., Arumugam, I.,, Vug- gumudi, K., Urbanik, J., Hansen, K., Celik, S., Cernek, N., Jagannathan, G., et al. RxRx3: Phenomics map of biology. bioRriv, pp. 2023-02, 2023.
Kenyon- K., Wang, Z. J., Urbanik, J., Donhauser, K., Hartford, J., Saberian, S., Sahin, N., Bendidi, I., Celik, S., Fay, M., al. Vitally consistent: Scaling biological representation learning for cell microscopy. In Neural Information Processing Systems Workshop Founda- tion Models for Science (NeurIPS FM4Science), 2024.
Gillespie, M., Jassal, B., Stephan, R., Milacic, M., Rothfels, K., Senff-Ribeiro. A., Griss, J., Sevilla, C.. Matthews, L., Gong, C., Deng, C., Varusai, T., Rague- neau, E., Haider, Y., May, B., Shamovsky, V., Weiser, J., Brunson, T., Sanati, N., Beekman, I.., Shao, X., Fabre- gat, A., Sidiropoulos, K., Murillo, J., Viteri, G., Cook, J. Shorser, S., Bader, G., Demir, E., Sander, C., Haw, R., Wu, G., Stein, I.., Hermjakob, H., and D Fustachio, P. The reactome pathway knowledgebase 2022. Nucleic Acids Research, 50(D1): 2021. ISSN 0305- 1048. doi: 10.10933nar/gkab1002..
Khan, M. O. and Fang, Y. What is the best way to fine- tune self self-superrvsed medical imaging models? In An- nual Conference on Medical Image Understanding and Analysis, pp. 267-281. Springer, 2024.
Kim, V., Adaloglou, N., Osterland, M., Morelli, F., and Za- pata, P. A. M. Self-supervision advances morphological profiling by unlocking powerful image representations. bioRxiv, pp. 2023 04, 2023.
Giurgiu, M., Reinhard, J., Brauner, B., Dunger- Langer--altenaach. I., Fobo, G., Frishman, G., Montrone, C., and Ruepp, A. CORUM: the comprehensive resource of mammalian protein complexes 2019. Nucleic Acids Research, 47 (Database D559e):D55.DD563 2019. ISSN 0305-1048. doi: 10. 1093////////yy973..
Kraus, O.,
K., Saberian, S., Fallah, M., McLean, P., Leung, J., Sharma, V., Khan, A., Balakrish- nan, J., Celik, S., et al. Masked autoencoders are scalable learners of cellular morphology. In Nearal Information Processing Systems Workshop on Generative Al and Bi- ology (NeurlPS GenBio). 2023.
He. K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Kraus, O., Kenyon-Dean, K., Saberian, S., Fallah, M., McLean, P., Leung, J., Sharma, V., Khan, A., Balakr- ishnan, J., Celik, S., et al. Masked autoencoders for microscopy are scalable learners of cellular biology. In
10

---

Sivanandan, S., Leitmann, B., Lubeck, E., Sultan, M. M., Stanitsas, P., Ranu, N., Ewer, A., Mancuso, J. E., W., Cesarek, ViTally Consistent: Scaling Biological Representation Learning for Cell Microscopy
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11757-11768, 2024.
Schuhmann, C., Kaczmarczyk, R., Komatsuzaki, A., Katta,
A., Vencu, R., Beaumont, R., Jitsev, J., Coombes, T., and Mullis, C. LAION-400M: Open dataset of clip-filtered 400 million image-text pairs. In NeurlPS Workshop Dat-
Kraus, O., Comitani, F., Kenyon-Dean, K., Urbanik, J., Arumugam, L.., Saberian, S., Wognum, C., Celik, S., and Haque, I. S. RxRx3-core: Benchmarking drug-target interactions in high-content microscopy. In Learning Meaningful Representations of Life (LMRL) Workshop at ICLR 2025, 2025.
acentric AI. Jülich Supercomputing Center, 2021.
Singh, M., Duval, Q., Alwala, K. V., Fan, H., Aggarwal, V. Adcock, A., Joulin, A., Dollár, P., Feichtenhofer, C., Girshick, R., et al. The effectiveness of mae pre- pretraining for billion-scale pretraining. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5484-5494, 2023.
Lastufka, E., Drozdova, M. Kinakh, V., and Voloshynovskiy, S. Vision foundation models: can they be applied to astrophysics data? In Neural Infor- mation Processing Systems Workshop on Foundation Models for Science (NeurIPS FM4Science), 2024.
Phillips, Z. F., Kim, A., Bisognano,
Lazar, N. H., Celik, S., Chen, L., Fay, M. M., Irish, J. C., Jensen, J., Tillinghast, C. A., Urbanik, J.. Bone, W. P., Gibson, C. C., et al. High-resolution genome-wide map- ping of chromosome-arm--cale truncations induced by crispr-cas9 editing. Nature Genetics, pp. 1-12, 2024.
J., Ruggiu, F., Feldman, D., Koller, D., Sharon, E., Kaykas, A., Salick, M. R., and Chu, C. A Pooled Cell Painting CRISPR Screening Platform Enables de novo Inference of Gene Function by Self-supervised Deep Learning. hioRxiv, pp. 2023.08. 13.553051, 2023. doi: 10.1101/2023.08 13.55350......
Lehner, J., Alkin, B., Fürst, A., Rumetshofer, E., Miklautz, L., and Hochreiter, S. Contrastive tuning: A little help to make masked autoencoders forget. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 2965-2973, 2024.
Sorscher, B., Geirhos, R., Shekhar, S., Ganguli, S., and Morcos, A. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Infor- mation Processing Systems, 35:19523-19536 2022.
Sypetkowski, M., Rezanejad, M., Saberian, S., Kraus, O., Urbanik, J.. Taylor, J., Mabey, B., Victors, M., Yosinski, J., Sereshkeh, A. R., et al. Rxrxl: A dataset for evaluat- ing experimental batch correction methods. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4284 4293, 2023.
Liu, Y. and Xie, J. Cauchy combination test: A powerful test with analytic p-value calculation under arbitrary de- pendency structures. Journal of the American Statistical Association, 115:393- 402, 2018.
Llama3, Т. The Llama 3 Herd of Models. arXiv, 2024. doi: 10.485550/amii.22402227773
Szklarczyk, D., Gable, A. I.., Nastou, K. C., Lyon, D., Kirsch, R., Pyysalo, S., Doncheva, N. T., Legeay, M., Fang, T., Bork, P., Jensen, L. J., and Mering, C. v. The STRING database in 2021: customizable pro- tein-protein networks, and functional characterization of user-uploaded gene/meesuremmett sets. Nucleic Acids Research, 49(D1):D055-D612 2020. ISSN 0305-1048. doi: 10. 10.1093/nar/gkaaa1074
McQuin, C., Goodman, A., Chernyshev V., Kamentsky, L.. Cimini, B. A., Karhohs, K. W., Doan, M., Ding, L., Rafelski, S. M., Thirstrup, D., et al. Cellprofiler 3.0: Next-generation image processing for biology. PLoS bi- ology, 16(7)ee2005970, 2018
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F,, El-Nouby, A., Assran, M. Ballas, N., Galuba, W., Howes, R., Huang, P.-Y,, Li, S.-W., Misra, I. Rab- bat, M., Sharma, V., Synnaeve, G., Xu, H., Jegou, H., Mairal, J., Labatut, P., Joulin, A., and Bojanowski, P. Dinov2: Learning robust visual features without super- vision. arXiv preprint arXiv:2304. 07193, 2024.
Templeton, A., Conerly. T., Marcus, J., Lindsey, J., Bricken, T., Chen, B., Pearce, A., Citro, C.,, Ameisen, E., Jones, A., Cunningham, H., Turner, N. L., McDougall, C., MacDiarmid, M., Freeman, C. D., Sumers, T. R., Rees, E., Batson, J., Jermyn, A., Carter, S., Olah, C., and Henighan, T. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Transformer Circuits Thread, 2024.
Radenović, F., Tolias, G., and Chum, O. Fine-tuning cnn image retrieval with no human annotation. IEFE trans- actions on pattern analysis and machine intelligence, 41 (7):1655-1668, 2018
Weinzaepfel, P., Lucas, T., Larlus, D., and Kalantidis, Y. Learning super-feature for image retrieval. In Interna- tional Conference on Learning Representations, 2022.

---

ViTally Consistent: Scaling Biological Representation Learning for Cell Microscopy
Wightman, R.
Pytorch image models.
ittps://githhbb com/ rwi ghtman/
pytorch image models 2019.
Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer,
Scaling vision transformers. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 12104-12113, 2022.
Zhang, Q. Wang. Y., and Wang. Y. How mask matters: Towards theoretical understandings of masked autoen- coders. Advances in Neural Information Processing Sys- tems, 35:27127-2771 2022.
12

---

ViTally Consistent: Scaling Biological Representation Learning for Cell Microscopy
A. Appendix
A.3. Perturbation Consistency
A.I. Training dataset curation details
In order to assess the consistency of the induced mor- phology on the cells by the perturbations, used non- parametric perturbation consistency test similar to the one introduced in (Celik et al. 2024). Let 19 be the embeddings for replicates of perturbation Vig on ex- periment (batch) e. As the test statistic for perturbation consistency, 5% is defined as the mean of the cosine simi- larities across all pairs of replicates of Tgg.
In order to produce Phenoprint-166 16M, we curated 93M us- ing the following steps:
Fgg.
1. Filtering out data that did not pass data quality fil- ters related to the focus of the image, quantity of dead cells, assay conditions, and presence of strong anoma- lous imaging artifacts.
2. Filtering out data with missing information about the perturbations applied, data with more than perturba- tions applied, and data of unusual size (in the image dimension number of channels).
(Eg.a Fost
n
نgpو |
where (.) and ||-|| denote dot product and Le norm.
3. Filtering out perturbation conditions that had been in less than 3 distinct experiments or 20 distinct wells so as to capture a variety of batch effects and have a broad sample of positives per class.
Statistical significance of sy is assessed using a permuta- tion test comparing against an empirical null distribution generated using the same statistic for a set of randomly lected perturbations in experiment e, (3,,,.. ".,,",,, p-value for sy is computed as follows
,3k J. The
4. Under-sampling perturbation conditions that were clearly over-represented in the dataset. Our experi- ment designs contain positive controls, negative con- trols, and wells without perturbation within each ex- periment. At this step, we keep 10% of positive con- trols and wells without any perturbation, 30% of neg- ative controls, and all other perturbation conditions.
# #(8 2 5 8gg,1
max
Pg
(3)
K
When multiple experiments existed for the same perturba- tion, combined p- values using the Cauchy Combination test (Liu & Xie 2018).
5. Filtering out wells where none of the perturbation con- ditions had a phenoprint ($A.3) (across different map types) in any experiment had been run in.
A.4. Training linear probes
In this section, provide details about the training pro- cess and preprocessing steps used in our logistic regression models. These models were trained on output features de- rived from various Vision Transformer (ViT) blocks.
A.2. Training hyperparameters
Table 5] provides the hyperparameters used for training the new vision transformers presented in this work. Each model was trained using a 75% mask ratio and the stan- dard decoder architecture for MAES (He et al., 2022). Each model was trained with the standard L2 MAE loss and the Fourier-space loss function implemented by (Kraus et al.) 2024 with weight of that the details presented by (Kraus et al., 2024) do not precisely correspond with the implementation provided in their Github repository; when reshaping the tokens to a shape compatible with the 2D Fourier transform, the per- mute operation resulted adjacent pixels being from dif- ferent channels of the input, resulting in the high frequency components of the loss being a function of the relationships between input channels. An initial investigation with a ViT- L/8 showed that changing the implementation to the one described in the paper did not dramatically change prob- ing results. As such, we used the implementation as-is and leave additional analysis of loss function design for MAEs to future work.
The data was split by experiments, ensuring that the test data originated from experiments distinct from those used for training. This approach helps validate the general- ization performance of our models across different experi- mental conditions.
0.01. We note, however,
For both RxRx1 gene prediction and Anax group predic- tion, we apply StandardScaler from the seikit-learn library as the only preprocessing step to standardize the fea- tures prior to training linear probes. StandardScal ler transformation was fitted on data from the train split. We trained the logistic regression models using scikit-learn's class. The following parame-
Logi i st ters and settings were used during model optimization:
Solver: lbfgs
Maximum Iterations: 2000
Class Weight: balanced
13

---

ViTally Consistent: Scaling Biological Representation Learning for Cell Microscopy
Table 4: Overview of vision transformer (ViT) encoders used and evaluated in this work.
Model Name
Parameters
Blocks
Model Dim
Pretraining Data
Baselines
Untrained ViT-S/16
25M
12
384
N/A
Dino-V2 V2 ViT-S/14
25M
12
384
Natural images
Dino-V2 V2 ViT-L/14
307M
24
1024
Natural images
Dino-V2 V2 ViT-G/14
1,100M
40
1536
Natural images
ViT-L/16 MAE
307M
24
1024
Imagenet-21K
MAES for microscopy
CA-MAB-S/16
25M
12
384
RxRx3
MAE-L/8
307M
24
1024
RPI-93M
MAE-L/8
307M
24
1024
Phenoprints- 16M
MAE-G/8
1,860M
48
1664
Phenoprints-16M
Table 5: Training hyperparameters for the new models presented in this work. Each used one-cycle cosine learning rate decay schedule with 10% warm-up using the Lion optimizer from (Chen al., 2023) with betas (0.9, 0.95) and weight decay of 0.05, with additional ViT settings such as LayerScale as proposed by (Dehghani et al. 2023). *Note that MAE-G/8 had multiple restarts during training due to challenges associated with massive model training on large-scale shared distributed compute clusters.
Hyperparameter
CA-MAE-S/16
MAE-L/8
MAE-G/8
Vision transformer backbone Pretraining Data
ViT-S
ViT-L
ViT-G Zhai et al.
2022)
RxRx3
Phenoprints-16M
Phenoprints- 16M
Training epochs
100
500
500"
Learning rate
le-4
3e-5
3e-5
Global batch size
2048
16384
8192
Stochastic depth
0.1
0.3
0.6
GPUs GPU-hours
16 A100s
128 H100s
256 H100s
400
15,360
48,000
For RxRx1 gene prediction, trained logistic regression models to predict one of 1139 possible perturbation la- bels (1138 genetic perturbation and non-perturbed control). For Anax group prediction, we trained logistic regression models to predict one of 40 possible function group labels
HUVES TAPO
0.8
0:2
3.20
0.9
(8 A.S). We report the balanced test accuracy as the main evaluation metric for all linear probing experiments.
0.15
UAGARUUF-MM) D4-M66-21
HAGAN po-pN
3.2
04
62
00
02
Nermaladd poetoo
Nomralioed block prsitton
A.5. Anax classification for other cell lines/treatment conditions: ARPE19 and HUVEC with TNFa background
Figure 6: Layerwise validation set linear probe perfor- mance on Anax functional gene group classification be- yond RxRx3: CRISPR knockouts in the ARPE-19 immor- talized epithelial cell-line (left), and in HUVEC cells with TNFa background (right).
We performed linear probing on imaging data obtained for retinal pigment epithelia (RPE) cell line, ARPE19, and HUVEC cells treated with an inflammatory cytokine. TNFo. We similarly observed that intermediate blocks of- ten have the most linearly separate features compared to the final block.
A.6. Biological Relationship Recall
A valuable use of large-scale HCS experiments is to per- form large-scale inference of biological relationships be- tween genetic perturbations. We evaluate each model's ability to recall known relationships by using the biolog-
14

---

ViTally Consistent: Scaling Biological Representation Learning for Cell Microscopy
ical relationship recall benchmark described in Celik al. 2024). First, we correct for batch effects using Typical Variation Normalization (TVN) (Ando et al. 2017), and also correct for possible chromosome arm biases known to exist in CRISPR-Cas9 HCS data (Lazar et al. 2024). To infer biological relationships, we compute the aggre- gate embedding of each perturbation by taking the spheri- cal mean over its replicate embeddings across experiments. We use the cosine similarity of pair of perturbation rep- resentations as the relationship metric, setting the origin of the space to the mean of negative controls. We compare these similarities with the relationships found in the fol- lowing public databases: CORUM (Giurgiu et al., 2019). hu.MAP (Drew et al., 2017), Reactome (Gillespie et al. 202T and StringDB (Szklarczyk et al. 2020) (with 995% combined score). Table [ reports the recall of known re- lationships amongst the top and bottom 5% of all cosine similarities between CRISPR knockout representations in RxRx3 Fay et al. 2023).
CPU hours
Figure Loss curve when training Dino-v2 ViT-L/16 on RxRx3.
models, we the median of CVM and KS* over all technical replicate experiment pairs eq.
Since the pairs are randomly selected for Po' the embed- dings would be mostly orthogonal thus the distribution would be centered around 0.Given that not all CRISPR knockouts would induce a morphological change in the cells, it's plausible for distribution gd to exhibit peak around O. As the model approaches the precision of an oracle, we would anticipate the mass situated around this peak to shift towards higher cosine similarity values.
A.7. Replicate Consistency
In order to assess the reproducibility of the perturbations across their technical replicates, we compare the distribu- tions of the similarities for same perturbations across repli- cates against an empirical null distribution. Specifically, for technical replicate experiments e¹ and ctj, we calculate the cosine similarity between the embeddings of perturba- tion 2y in them, denoted as s The query distribution qf is constructed by computing the cosine similarities for all per- turbations that have a matching well on experiments ea and ei An empirical null distribution of identical cardinality is created by computing cosine similarity, "" between random pairs from eg and ej, such that no pair corresponds the same perturbation, Po Using non-parametric statis- tical tests, namely Kolmogorov-Smima Von-Mises (CVM), we can evaluate the hypothesis that tt and po are drawn from the same distribution. Formally, let Q (xx) and p (x) be the cumulative distribution func- tions for g and Po respectively, then the KS statistic for the two-sample case of technical replicate experiments e and ej is defined as:
A.8. Anax Group Prediction Details
The Anax probing task introduced in this paper is intended to balance capturing a diverse range of biology that is broadly conserved between cell types with a reduced cost of execution. The name "Anax" is a reference to Anax- imander, the 6th century B.C. philosopher credited with making the first world map.
(KS) and Cramer
In curating these genes, we analyzed the sources listed in & A.6 as well as internal gene expression data to pro- duce "functional" groups corresponding to biological pro- cesses, cellular components, and molecular functions. Not all genes within each group are expected to have the same knockout phenotype, but are classified by humans as hav- ing related function linear separability of these genes would indicate that a model has learned similar concepts to those deemed significant by biologists.
k = sup, Q (I) Po: (a:)--
(4)
The Cramér-von Mises test statistic (CVM) for experi- ments e,, and ej, is computed as:
The gene groups we use for the 40-class Anax group clas- sification task ($ A.4 are listed in Table 7
CVM
1 2N
Σ
(rm- (rm--))²+(e mm)²
2
A.9. Dino V2 pretraining on microscopy data
12N
(5)
We attempted to train two Dino-v2 models on microscopy data. One ViT-L/16 from scratch on RxRx3, and another attempt of fine-tuning the MAE-L/8 on RPI-93M with the Dino-v2 losses. They had the following hyperparameter
where N is the cardinality of qf and po and 8m and rm are ranks of similarities hd and p in the combined dis- tribution of qt and po when ordered. order compare
15

---

ViTally Consistent:
Scaling Biological Representation Learning for Cell Microscopy
Table 6: RxRx3-core benchmarks for our initial attempts to train Dino-V2 V2 models on microscopy data. The latter was finetuned from the MAF-L/8 trained on RPI-93M. Results compare Table
extraction algorithms.
Benchmarking results reported for JUMP-CP
CellProfiler features ing ($ 5.1 and RxRx3-core ($ 66. JUMP-CP CellProfiler
features were downloaded from https:/cellpaininng
gallery.s. Setersss..aaazoaaws.com/mmm hwt.://wwww....o///n///////o//na
Dino V2 model
Avg. Prec.
Z-score
CellProfiler features for RxRx3-core were computed using version 2.2.0 (Kamentsky et al. 2011]. Single cells were segmented after applying illumination correction and shape, intensity, and texture features were extracted from each cell, resulting in 952 dimensional profiles. For RxRx3-core, we kept only cells in the center 512x512 crop of original 2048x2048 images and then mean aggregated them. 4,749 wells of the 222,601 RxRx3-core wells were missing CellProfiler features, so these were held out of the benchmark computations for all models on RxRx3-core.
ViT-L/16 RxRx3
0.258 = .015
2.13
VIT-L/8 (ft.) RPI-93M
0.255 + .018
1.76
settings which were tuned on another dataset: output-dim 65536, 2 global crops, 4 local crops, dino loss weight 1.0, koleo loss weight 0, ibot loss weight 1.0, Lion optimizer with max learning rate Ic-5, weight decay 0.05, betas 0.9 0.95, and cosine annealing. In both cases, we observed significant over-fitting of the loss from the start Figure 7p.
In Table 6 we show that both models fail to improve the RxRx3-core benchmark metrics (i.e., z-scores over the random baseline) versus the CA-MAE-S/16 RxRx3 model which had Z-score of 2.90 on average precision for pre- dicting compound activity. We have not found an effec- tive recipe for training Dino on microscopy data. However, we note that theoretical evidence exists arguing that MAE learning is in some ways equivalent contrastive learning (Hondru et al. 2024). so even if an appropriate Dino recipe is found it would remain to be seen if it differs substantially from MAES for microscopy given the same training com- pute. As described in the Limitations section, we expect that future work would have to dedicate significant train- ing ablations and creativity to determine the best possible training recipe for training Dino on microscopy data.
A.10. Correlation between model scale and benchmark results
In Figure 8 we show the correlations between training FLOps (floating point operations) and downstream results. Over all benchmarks we observe very strong consistent linear trend where scaling training FLOps improves over- all pwerformance. This work provides the next log step in scale as we enter into the billion-parameter model regime with MAE-G/8. These results therefore provide additional evidence that the trend initially discovered by (Kraus et al.) 2023) between FLOps and relationship recall actually ex- parameter models and even moreso for other biologically meaningful benchmarks pertaining to linear probes on small experiments and to replicate consis- tency on the whole-genome.
tends both to billion-parre billion-parameter
A. CellProfiler features
CellProfiler
bioimage analysis software (Carpenter)
ctt al. 2006; McQuin et al. 2018) was used to com- pute features using classical segmentation and feature
16

---

ViTally Consistent: Scaling Biological Representation Learning for Cell Microscopy
Table 7: Anax groups and their associated genes. corresponding genes.
This table presents comprehensive list of gene groups and their
Anax Group
Genes
Asyl Coa Binsynihesis
ELOV12, ELOVLS, ELOVLA HACDII JIACD2, IISDI7B12, SCD, SCDS, TECR
Adherens Junctions
ACTB, ACTGI, AFDN, CDIII, CTNNAL, CTNII,
CTNNBI, CTNNDI, NECTINI, NECTINI, NECTINI
Amino Acid Motabolism
ALDHAAI, ARGZ, CKMT2, CPSL, DAO, PYCR2, PYCR3, SATI
Apoptosis
CPLAR, DFF,, CASPS, CASP3, FASLG, UCL2, DFFA, XIAP, TNFSFIO, AKT3
Autophagy
AIG12, ATGS, ATGHB, ATGIC, AEG?, GABARAP, PIKIC3, PIK3KA, PRKAAL, ULKI
Beta Oxidation OF Fanty Acids
ACAAZ, ACADL, ACADM, ACADS, ACADVL, ECHST, ECT, HADH, HADHA, HADHE
Calciom Signaling
ADCYI, ADCY2, ADCY3, CALMI, CAMK2B, CAMKZD, PDELB, PDEIC, PRKACG, PREX
Clattrin Coated Vesicles
AP2AI, APZAZ, APIRI, AP2MI, AP2SI
COPI
ARCNI, COPA, COPBI, COPR2, COPE, COPGI, COPZI
COPI Vesicles
SEC13, SEC23A, SEC24B, SECHDD SECSIA
DNA Dumage Repuir
BLM, BRCA2, EMEI, NBN. POLD2, RADSIB, RADSIC, RADSID, RPAI. XRCC2
Dynein
DYNCIIII, DYNCUI,, DYNCILII DYNCILE, DYNLTI
ER Protein Translecation
SPCS3, SEC6IAI, SRP14, SRP12, SPCSI, SRPRA, SECIIA, SRP6E, SRPRE, SRP51
Exosome
DIS3, EXOSCIO, EXOSC3, EXOSCA, EXOSCS, EXOSCS, EXOSC7, EXOSCS, EXOSCO, MPHOSPH6
Gap Junctions
ADCYE, DRD2, HTRIC, TIPRZ, LPARI, PEGED, PUGERE, PLCB3, TUBAIC, TUBBI
Golgi
ACTRIO, ACTRIA, CAPZAS, COG4, CTSZ, PPICC RAHIB, SEC22C, SBC24L, TMEED
MAPK
DUSPA, PGPIX,
HSPRI, MAPIK2, MAPKAPKS, RACI, RAPIA, RASGRPI
Minochondiio Stracture
APOOL, APOO, TMEMIL CHCHIS ATPSME, MICOSIS, ATPSFIC, DNAKCH DMACZI, ATPSMP
Mitochondial Transport
ATPMFAA, COA4 COA6, COXI7, HSPAO, IDHN PITRMI, PMPCA, PMPCR, SEC2254
=/OO Pallway
CAR39, CAR3OL. FIFAERPI,
PRKKAA, RPS6KBI, RPTOR, STK STRADA, TSCI
Medisted Dociy
CASCA, EIF4A3, MAGOHI, MAGOHB, RBMRA
Pore
NUP107, NUP133, NUP153, NUP188, NUP205, NUPS7, NUPSS, NUP93
Nucleolus Structure
FBL, NATIO, NOLCI, NOPSE, UTP20
Nucienide Merabolism
ADSL ADSSI, ADSSZ ATIC, IMPDRI, IMPDHE. PAICS, PPAS, PIAT
P33 Stress Signaling
AIR, CONGL, CDKL, CHEKI, CHEK2, MOME, MDME, TPS3, TP73
Pentose Phougtate Pathway
GRPD, TALDOI, DERA, RPE, POME, PGD, PGLS, RPELI, PRESZ
Peroxisome Biology
ACOTS, AGPS, BAAT, HMGCL, HSD17B4, MLYCD, PAOX, PEXTZ, PEXS, PIPUX
Presplicesiome Complex
ALYREF, AQR, CRNKLI, DDXS, HNRNPK, LSME, PLRGI, PRPPA, SMNDCC, SESE4
Protessome
PSMAI, PSMA4, PSMBI PSMB2, PSMR7, PSMA6, PSMA3, PSMB4, PSMAS, PSMR3
Rihowome Lange
RPII3A, RPI.11, RPI.OO, RPI23A, RPI30, RPLJA, RPI P2, RPI.28 RPI.5, RPI.27A
Ribowome Small
RP52, RP56, RPS8, RPS16, RPSI), RPSJA, RPS19, RPST3, RPS4X, RPS9
RNA Polymerase II
POLRZA, POLR2B, POLREC, POLRIG, POURZI, POLROL
cyele
ACO2, DLST, FH, IDH2, IDH3 MDHE, OGIH SOHB, SUCLAZ, SUCLG2
Tight lunctions
CLDNI4, CLENIF, CLENIF,CLDNIA CLDNIS, CLDNID, CLDNA, CLDNS, CLDNN, MPPS, PARD6R PRKCT
Tranclation Initiation Complex
FIF3G FIFSA, FIFSD, FIFM) FIFIK, FIFIM FIFTR, RIFSH FIFSE, FIFM.
Transport Of Fanyy Acids
APOD, LCN12, LCNIS, LCN9, SLCITAL SLC27A4, SLC22A6
Tubulin
TUBA3C, TBCC, TBCD, TUBA-B. TUBAR, TUBALS, TUBALA, TUBBAB, ARL2, TUBAIB
Unfolded Protein Response
CXXCI, DIAJBII, KHISE MDTPSI, SUICI, TATDN2, TLNI, TSPYL2, YIFIA
V-ATFase
ATPEVIA, ATPSY, ATRSVID, ID, ATPSVIEI, ATPSV IE, ATPSVIII
17

---

ViTally Consistent: Scaling Biological Representation Learning for Cell Microscopy
FLOPS CORUM recall (Pearson: 0.99)
FLOPS hu hu,MAP recall (Peerson: 0.98)
0.62
0.44
0.61
0.43
0.60
0.42
0.59
CORUM
0.58
0.40
0.57
0.39
0.56
0.38
0.55
10a0
10
10a2
10a
102t
10ah
FLOPS 1Log Scale)
FLOPS (Log Scale))
FLOPS vs Reactome PPI recall (Pearson: 0.99)
FLOPS vs String00 recall (Pearson: 1.00)
0.49
0.26
0.48
0.47
Reactoma
0.46
0.24
0.45
0.44
0:2
0.43
10t0
10ah
10
10th
1021
10
(Lag
(Log Scall)
FLOPS Replicate Consistency (Pearson:
FLOPS Replicate Consistency (CVM) (Pearson:
0.625
18
Replictte Consesencyy (KS)
Depicate otencyy (CVM))
0.600
16
14
0.525
0.300
12
0.475
102t
FLOP5 1Log Scalt)
FLOPS (Log Scale)
FLOPS vs Anax linear probe accuracy (Pearson: 0.94)
FLOPS vs RxRx1 linear probe accuracy (Pearson: 1.00)
0.30
pobba
0.26
0.2
0.22
1000
10ah
1022
10tt
10at
102
Haa Scale)
(Log Scalk)
CA-MAE-S/16 (b" 12, RxRx3)
MAE-L/8 115, RPI-93M)
RPI-913)
MAE-L/8 (b 200, PP-16M)
MAE-G/8 PP-16M)
Figure 8: Relationship between FLOPs and benchmark evaluation results for the six whole-genome tasks Table 1 and the
two linear probing tasks (Figure 3).
18


Reviews:
[{"review_id": "4tslaZxh76", "reviewer": "Reviewer_pEaF", "summary": "This paper provides a framework to improve biological representation learning on large-scale microscopy datasets. Three steps are introduced: (1) Curating the training dataset to have a better distribution of samples over the phenotypic spectrum, (2) scaling training to larger models, and (3) evaluating intermediate representations to identify the best representation for downstream tasks.\n\nSuperior performance of the largest model trained on the curated dataset is shown over previous baselines in terms of biological relationship recall and zero-shot generalization. It is shown that finding the best layer for downstream tasks using linear probing improves accuracy, and linear probing accuracy on a smaller validation set correlates well with whole genome evaluation results.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and thoughtful feedback.\n\n**On the significance of data curation**: Filtering is critical even in large language models. Penedo et al [2023] highlight the value of de-duplication in RefinedWeb. While such techniques (e.g., MinHash) don't directly apply to images, our curation serves a similar role. Dataset curation had a strong impact on replicate consistency (KS): ViT-L/8 improved from .52 (RPI-93M) to .59 (PP-16M). We hypothesize that RPI-93M contains many control/inactive perturbations, leading to overfitting to subtle batch effects even after alignment. In contrast, PP-16M better captures phenotypic diversity and yields more consistent representations.\n\n**On the significance of model scaling**: We show that combining curation, scaling, and layer selection yields strong gains. In Table 1, ViT-G/8 on PP-16M-trimmed improves replicate consistency (KS: .52 → .63, +21%; CM: 12.3 → 18.2, +48%) over prior SOTA (ViT-L/8 on RPI-93M). This matters as biological recall is less meaningful without consistent perturbation representations.\nTable 2 shows improved zero-shot recall on an OOD dataset (JUMP-CP): ViT-G/8 trimmed recalls more interactions in the 0.05/0.95 cosine thresholds [CORUM (134), HuMAP (65), Reactome (42), STRING (157)] demonstrating generalization across experimental labs.\nTable 3 reinforces this with gains on a benchmark for compound-gene relationships, showing ViT-G/8 more effectively encodes biological interactions.\n\n**On the novelty of evaluating intermediate representations**: Layer selection has precedent in supervised transfer learning, but its use in SSL-trained transformers is rare. Prior work typically fine-tunes the penultimate layer, but in microscopy, supervised fine-tuning often reduces performance [Kraus et al. 2024]. We show that selecting intermediate layers in ViTs trained with SSL improves zero-shot performance without retraining. This holds across models, including those pre-trained on natural images, and offers a practical contribution to microscopy-based representation learning. The only related work we’re aware of is MIM-Refiner [Alkin et al. 2024], which appeared contemporaneously.\nWhile intermediate layers were known to sometimes yield better representations, their inference compute advantages became significant with recent model sizes. For ViT-G/8, metrics from layer 38 required ~3,000 L4 GPU hours vs. ~4,000 from the final layer.\n\n**On model/dataset ablations**: We appreciate the suggestion. ViT-G/8 training required 256 H100 GPUs for over a week, so full cross-comparisons were infeasible. However, we provide targeted ablations:\n* Layer selection: All models were evaluated at multiple layers. Intermediate layers consistently outperform penultimate. Proxy tasks like RxRx1 and Anax classification correlate with genome-wide evaluations, offering cost-effective benchmarks.\n* Dataset comparison: We compare ViT-L/8 trained on RPI-93M vs. PP-16M across all metrics, motivating our choice to train ViT-G/8 on PP-16M.\n* Model scaling: Scaling benefits in microscopy SSL have been demonstrated [Kraus et al. 2024, Fig. 5], supporting our decision to go beyond ViT-L/8.\n\n**On the novelty of contributions**: While curation, scaling, and layer selection have been explored individually in other domains, we demonstrate their combined effectiveness for representation learning in microscopy. These insights also apply to other biological settings with repeated measurements.\n* Curation strategy: Unlike prior SSL curation that removes duplicates, we identify experimental conditions with consistent phenotypes across replicates, improving signal and reducing batch effects. This is a novel form of statistical deduplication.\n* Model scaling: Scaling ViTs beyond ViT-L/8 provides measurable gains and supports the neural scaling hypothesis in microscopy. At this scale, sharing what works is crucial. Reproducing one ViT-G/8 run would cost \\~\\\\$470K (43,000 H100 GPU hours × \\$11/hour). Prior scaling papers (e.g., Touvron et al. 2023, DeepSeek-V3) have proven valuable by sharing such insights.\n* Layer selection: While intermediate layers are used in supervised CNNs [Moshkov et al. 2024], they’ve not been systematically explored in SSL ViTs. We show that layer-wise evaluation improves performance without fine-tuning, highlighting a practical and underused method for biology-specific tasks.\n\nWe appreciate your recognition that the paper provides strong baselines and a new foundation model, and hope our responses clarify the significance and generalizability of our contributions.\n\n**References**:\n* Alkin et al. 2024 – https://arxiv.org/abs/2402.10093\n* Kraus et al. 2024 – https://arxiv.org/abs/2404.10242\n* Moshkov et al. 2024 – https://www.nature.com/articles/s41467-024-45999-1\n* Touvron et al. 2023 – https://arxiv.org/abs/2302.13971\n* DeepSeek-V3 – https://arxiv.org/pdf/2412.19437\n* Penedo et al. 2023 – https://arxiv.org/abs/2306.01116", "claims_and_evidence": "I will address each claim one by one:\n\n(1)\t Curating the training dataset to have a better distribution of samples over the phenotypic spectrum leads to improved performance: I’m not sure if this is necessarily true. Imbalanced datasets do affect the model’s ability to learn about certain semantic concepts, but I doubt this is true at the scale of models and datasets we are talking about. It might happen that certain phenotypes have a very low frequency and the model will not learn about these, but in general imbalances in the number of samples per phenotype should not affect the semantic quality of the model’s embeddings. The results in Table 1 reflect this – the differences between the MAE L/8 trained on the RPI-93M and PP-16M are minor, and in fact are the same in terms of biological relationship recall for the untrimmed versions. Overall I don’t think there is enough evidence to suggest that the curation described here leads to meaningful results in performance.\n(2)\t Scaling model size leads to improved performance: This is not surprising, but the improvements shown in Tables 1, 2, and 3 are pretty minor given that the model size increase 6 times between MAE L/8 and MAE G/8. It is hard to evaluate whether these differences are meaningful, and even if they are meaningful, if they are worth the increase in model size.\n(3)\t Evaluating intermediate representations to choose the best layer for downstream tasks improves performance: Again, this is not really a novel idea. It is well known since the early days of representation learning that successive layers learn information at different layers of abstraction. While the penultimate layer is generally used for downstream tasks, it is not uncommon to use another intermediate layer. That said, mostly consistent, if somewhat minor, increases in accuracy are shown for most models, which does support the claim that trimming models is useful in the context of representation learning for cell microscopy.", "methods_and_evaluation_criteria": "The methods generally make sense for the problem of representation learning for microscopy, and the evaluation is shown on multiple datasets and for multiple models. None of the methods in the paper are necessarily novel, but experiments are performed at scale and a novel foundation model is introduced. \n\nI would prefer it if the evaluation of the model scaling and dataset curation were done independent of each other in Tables 1, 2, 3, i.e. train models of the same size on the 3 datasets (RxRx3, RP-93M, Phenoprints-16M) and also train models of different sizes on all 3 datasets. The scale of improvement is not drastic enough for me to determine whether either of these contributes significantly.\n\nAddition if out-of-distribution generalization on JUMP-CP and RxRx3 are good additions to the evaluation section. Biological validation is also mostly robust. While I will say that it is not surprising that linear probing accuracy on the manually curated Anax dataset correlates well with the whole-genome score, I think this is a good addition and could be a benchmark for future studies.", "theoretical_claims": "There are no theoretical claims made in the paper.", "experimental_designs_or_analyses": "NA", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths:\n\nI think the main strength of this paper is that it provides baseline results related to the effect of 3 components of the model building pipeline in the context of representation learning for microscopy – dataset curation, scale, and best intermediate layer to choose for downstream tasks. The trained foundation model as well as the results would provide a good benchmark for future work in this area. The paper is well written and understandable, and generally thorough in terms of evaluation and metrics.\n\nWeaknesses:\n\nI have two main issues with the paper: (1) The methodological contributions are minimal and the paper is pretty bland in terms of novel ideas. All of the methods discussed here have been widely evaluated in different settings, and I’m not sure if it has enough methodological novelty for being published at this venue, and (2) The performance improvements on the whole seem pretty minor to me, for example ~1-2% for the microscopy MAEs in Table 1. I think combined with the lack of novelty I this makes me question if the contribution is that strong.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yDC1upgO0U", "reviewer": "Reviewer_oHo7", "summary": "The authors present a framework for training large-scale computer vision models for microscopy imaging data.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and thoughtful comments.\n\n**On tables and metrics clarity**: We apologize for not clearly marking the Kraus et al. 2024 model in the tables. As noted in the “Models/prior work” section, MAE-ViT-L/8+ trained on RPI-93M is from Kraus et al. In Table 1 and 3, we report mean ± standard deviation computed from 3 and 100 random seeds, respectively. Table 2 caption notes a maximum standard deviation of ±0.0023. Perturbation consistency is evaluated using Kolmogorov-Smirnov (KS) and Cramér-von Mises (CVM) statistics, which compare replicate similarities to an empirical null. These are detailed on line 281 and Appendix 7, and were introduced in Celik et al [2024].\n\n**On evaluation and metrics**: Our relationship recall task evaluates whether embeddings can zero-shot identify known interactions between perturbations based on cosine similarity. Because we use genome-wide CRISPR KO screens, many true interactions may be missing from annotation databases. Thus, precision is not meaningful in this context, as we cannot assume non-annotated gene pairs are false positives.\nThe “recall % @ 0.05–0.95 cosine threshold” measures the fraction of annotated gene-gene relationships found in the most similar (top 5%) or dissimilar (bottom 5%) embedding pairs. We use multiple knowledge bases (CORUM, hu.MAP, Reactome, StringDB) and report results across them. These metrics are described in Appendix 6 and used in both Celik et al. and Kraus et al. 2024.\n\n**On methodological contributions and motivation**: This work extends efforts to scale ViT architectures and microscopy datasets. We found three key strategies that meaningfully improve performance:\n* Curation: Surprisingly, training ViT-L/8 on a smaller, curated dataset improved performance vs. RPI-93M. The curation focuses on selecting diverse experimental conditions with consistent replicate phenotypes, rather than removing redundancy as done in typical SSL data prep. This led us to train MAE-G/8 on the curated PP-16M set.\n* Scaling: Training larger ViTs continues to yield benefits, consistent with the neural scaling hypothesis. Our largest model, MAE-G/8, outperforms smaller ones in both replicate consistency and biological relationship recall.\n* Intermediate representations: We found intermediate layers often outperform penultimate layers for zero-shot tasks. Since fine-tuning is not feasible in our setting, we propose lightweight proxy tasks (RxRx1, Anax classification) to select the best layer. We show these proxy tasks strongly correlate with more expensive genome-wide evaluations.\nThese insights, while based on well-known techniques in general vision or language settings, are novel in their application and combination for microscopy-specific representation learning. They provide a practical and reproducible framework applicable to any experimental dataset with replicate measurements.\n\n**On prior work and references**: We appreciate the suggestion to cite earlier microscopy-specific deep learning work. Due to space constraints, we omitted a drafted section discussing such papers and instead prioritized prior work most relevant to our methods: dataset curation for SSL, layer selection, and embedding evaluation in microscopy. While we did not focus on contrastive approaches like CLOOME or MolPhenix, we agree they are valuable contributions. Our work differs in focus as we do not relate images to molecular structures, but instead aim to build general-purpose embeddings for microscopy data.\n\nWe hope this clarifies the design and motivation behind our choices, and the contributions our framework makes to the field. Thank you again for your detailed and constructive feedback.\n\n**References**:\n* Celik et al. 2024 – https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1012463\n* Kraus et al. 2024 – https://arxiv.org/abs/2404.10242\n* Fradkin et al. 2024 [MolPhenix] – https://arxiv.org/abs/2409.08302", "claims_and_evidence": "The authors claim to train a large-scale ViT model that should work better than a previous model by Kraus et al, 2024. \n\nThe evidence for that is quite unclear and scarce: The tables report performance values for a set of architecture, where it is unclear which architectures come from the authors and which are from previous works. Furthermore, almost all performance metrics are presented without error bars and confidence intervals (and also without statistical tests), such that it is unclear which method performs best. The effect size (difference) between methods is very small. For some metrics it remains even unclear that they are (e.g. \"KS\" and \"CM\" in Table 1).\n\nAlso the methodological contributions that are claimed are unclear: it is unclear what the main contributions to this framework are and how they are justified and motivated. It appears that the proposed steps (data curation, scaling, and selecting a block for features) are ad-hoc decisions.", "methods_and_evaluation_criteria": "The work does not propose a method, but a framework.\n\nThe benchmark datasets make sense, like RxR3 and JUMP-CP, because those are large imaging datasets. However, the evaluation metrics are not well motivated, or even completely unclear: e.g. Table 1 has only recall without mentioning precision (a method can always trade-off recall against precision), Table 2 has \"REactome\" and \"Stringdb\" as metrics, which is also unclear what kind of metric this should be and why it is relevant. Table 3 again only reports only precision and not recall (or AUC-PR). Across the whole paper, it remains unclear what the main evaluation criteria for foundation models for microscopy images are. \n\nFor comparison, ref [1] sets up a battery of zero- and few-shot downstream tasks, for which clear evaluation criteria and metrics exist. The authors should set up a set of zero- and few-shot downstream tasks (together with metrics and evaluation criteria), which should be solved by the foundation models. Also Kraus et al [2], should be an inspiration for the authors. \n\n\nReferences:\n[1] Sanchez-Fernandez, A., Rumetshofer, E., Hochreiter, S., & Klambauer, G. (2023). CLOOME: contrastive learning unlocks bioimaging databases for queries with chemical structures. Nature Communications, 14(1), 7339.\n[2] Kraus, O., Kenyon-Dean, K., Saberian, S., Fallah, M., McLean, P., Leung, J., ... & Earnshaw, B. (2024). Masked autoencoders for microscopy are scalable learners of cellular biology. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 11757-11768).", "theoretical_claims": "There are not theoretical claims in this work.", "experimental_designs_or_analyses": "The experimental analysis is mostly done on checking how good intermediate layers of the ViTs are w.r.t. to performance. This is a reasonable analysis. \n\nQuestion: Is this analysis done on a separate validation set (distinct from all other downstream tasks), such that selecting the best layer here does not introduce a bias for the later comparisons and benchmarks?", "relation_to_broader_scientific_literature": "The paper is well placed within the broader scientific literature of ViT, MAE, etc. However, early works on using deep neural nets for microscopy images are completely absent, and also other self-supervised learning approaches for microcopy images are missing. (see below).", "essential_references_not_discussed": "There are many earlier works on deep learning, e.g. CNNs, for microscopy images, e.g. ref [3], which are not mentioned and referred to. I only provide an exemplary reference here, but the authors should re-do the literature research to give a better view on this field:\n\n[3] Ciresan, D., Giusti, A., Gambardella, L., & Schmidhuber, J. (2012). Deep neural networks segment neuronal membranes in electron microscopy images. Advances in neural information processing systems, 25.\n\nAlso, contrastive learning approaches (e.g. SimCLR, CLIP, etc) are hardly mentioned, which have also been successfully applied to microscopy imaging data. The authors should provide at least a paragraph in related work on papers around that topic.", "other_strengths_and_weaknesses": "Strengths:\n- Effort to scale ViTs to large-scale microscopy datasets\n- Substantial computational resources used to develop the model\n\nWeaknesses:\n - Unclear what the novelty is or just a framework proposed\n - Limited relevance because of missing error bars and statistical test; unclear benchmark task and metrics\n - Main analysis steps are not well justified and motivated\n - Unclear what the machine learning aspect of this work is; maybe better suited for a bio-venue", "other_comments_or_suggestions": "Citing Micrographia by Hooke is really a great one!", "questions_for_authors": "What is the \"recall % @ 0.05-0.95 cosine threshold\" ???", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UXuTD1DDPe", "reviewer": "Reviewer_kmQy", "summary": "This paper proposes a three-stage framework for pretraining foundation models on large-scale microscopy datasets to address measurement errors and enhance biological signal extraction. The framework involves (1) curating diverse and self-consistent training samples, (2) scaling a vision transformer architecture (ViT-G/8 MAE) trained on 8 billion microscopy image crops, and (3) evaluating intermediate model layers to optimize representations for downstream tasks. The authors introduce the largest known foundation model for cell microscopy (1.9B parameters), demonstrating a 60% improvement in linear separability of genetic perturbations compared to the prior ViT-L/8 MAE. The model achieves state-of-the-art performance across four benchmarks: whole-genome relationship recall, batch effect correction consistency, compound-gene activity prediction, and perturbation analysis. Key innovations include systematic error mitigation through pretraining, intermediate layer feature optimization, and scaling laws applied to biological imaging.\n\n## update after rebuttal\nThanks for the clarification. I have no further questions and will keep my score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review and supportive comments.\n\n**On the cost of whole-genome benchmarking**: This was a key motivation for developing lightweight classification proxy tasks. Selecting a small subset of perturbations for genome-wide benchmarks is challenging due to the random distribution of gene KOs across HCS plates, which limits the number of relationships we can evaluate without embedding thousands of full plates. That said, a promising direction is RxRx3-core [Kraus et al 2025], a curated subset of RxRx3 that may provide more efficient evaluation going forward.\n\n**On interpretability**: We appreciate this suggestion. Related work explored mechanistic interpretability in these embeddings and directly motivated our use of intermediate layer analysis [Donhauser et al. 2024]. We agree that further investigation into which biological pathways or gene classes are best captured by these models is an important next step.\n\n**On hyperparameter choices and CNN baselines**: Patch size, mask ratio, CNN baselines (e.g., ResNet, EfficientNet), and SSL vs. supervised training regimes were thoroughly benchmarked in Kraus et al. [2024]. Our work builds directly on those findings by focusing instead on scaling ViTs, refining the training dataset, and evaluating intermediate representations.\n\n**On computational cost**: We agree the training cost is substantial (48K GPU hours), but we believe the investment is justified. Improved microscopy representations can accelerate early biological discovery and drug development. We are exploring ways to make training and evaluation more efficient in future work.\n\n**On distance metrics**: We considered alternatives but chose cosine distance due to its popularity and strong performance in deep embedding–based image retrieval. It is scale-invariant and more robust in high-dimensional spaces compared to Euclidean or Mahalanobis distance (Zhe et al. 2018, Deng et al. 2018).\n\n**On applicability to other biological imaging modalities**: We would have loved to explore this further but were limited by space. Recent work has trained ViT-G with DINO on histopathology data (Virchow 2), which suggests promising transferability to other imaging domains. We're excited by the possibility of adapting our approach to other modalities like H&E, organoids, or EM.\n\n**On novelty and contribution**: While our techniques build on existing methods, we identify three underexplored strategies that, when combined, significantly improve performance on microscopy-specific benchmarks:\n* Curation: Our dataset curation focuses on selecting diverse, experimentally consistent conditions across replicates. This contrasts with typical SSL curation strategies that aim to reduce redundancy by removing similar examples. We show that this approach enhances model consistency and downstream performance.\n* Scaling: We demonstrate that further scaling ViTs beyond ViT-L/8 continues to yield improvements, validating the neural scaling hypothesis in microscopy representation learning.\n* Intermediate representations: While intermediate layers have been used in CNN transfer learning (Moshkov et al. 2024), they remain under-utilized in ViTs trained with SSL. We show that selecting the right layer, determined via a fast, low-cost proxy task, leads to meaningful improvements in zero-shot biological recall.\n\nTogether, these findings offer a generalizable framework for building scalable, biologically relevant foundation models across experimentally derived datasets. We hope our study serves as a foundation for further work at the intersection of self-supervised learning and foundation model development for scientific datasets.\n\nThank you again for your constructive feedback and support.\n\n**References**\n* Kraus et al. 2024 – https://arxiv.org/abs/2404.10242  \n* Kraus et al. 2025 [RxRx3-core] – https://arxiv.org/abs/2503.20158\n* Donhauser et al. 2024 – https://arxiv.org/abs/2412.16247\n* Zhe et al. 2018 – https://arxiv.org/pdf/1802.09662 \n* Deng et al. 2018 – https://arxiv.org/abs/1801.07698\n* Zimmermann et al. 2024 [Virchow2] – https://arxiv.org/html/2408.00738v1\n* Moshkov et al. 2024 – https://www.nature.com/articles/s41467-024-45999-1", "claims_and_evidence": "All the claims are well-supported.", "methods_and_evaluation_criteria": "I am not familiar with the application and evaluation criteria, but they seem to make sense. The evaluation leverages multiple biologically relevant benchmarks, including Linear probing tasks (RxRx1 gene classification, Anax functional group classification) to assess representation quality. Whole-genome relationship recall using public databases (e.g., CORUM, StringDB) to measure biological relevance. Replicate consistency tests (Kolmogorov-Smirnov and Cramer-Von Mises) to ensure model robustness.\nThese benchmarks are well-suited for assessing biologically meaningful representations and align well with the intended application.\n\nSome potential issues are:\n\nThe computational cost of whole-genome benchmarking is extremely high (e.g., 80M forward passes). While this provides strong empirical evidence, an analysis of whether a subset of perturbations could yield similar insights with lower computational demands would be valuable.\n\nThe study provides some interpretability through linear probes and analysis of intermediate ViT layers, which is a notable contribution. While the correlation between linear probing results and whole-genome benchmarks is strong, further discussion on the biological interpretability of learned features would enhance the paper. For example, are certain gene classes more separable than others? Do embeddings capture known biological pathways?", "theoretical_claims": "There are no theoretical claims made in the paper.", "experimental_designs_or_analyses": "The use of self-supervised masked autoencoder (MAE) training aligns with state-of-the-art methods in computer vision and biological imaging. The authors explore different model sizes and configurations, providing valuable insights into scaling behavior. Some potential issues are that the justification for certain hyperparameters (e.g., patch size, mask ratio) is not explicitly discussed. A brief ablation study or sensitivity analysis on these parameters would strengthen the experimental design. Furthermore, the computational cost of training the 1.86B parameter model (48,000 GPU hours) is substantial, and an efficiency analysis comparing model performance versus computational cost would be beneficial.\n\nThe study uses a diverse set of evaluation metrics, but some alternative distance metrics (e.g., Euclidean, Mahalanobis) could be explored to ensure robustness.", "relation_to_broader_scientific_literature": "This work makes significant contributions to the field of biological representation learning by scaling self-supervised Vision Transformer (ViT) models for cell microscopy and improving the quality of learned embeddings. The work is related to the areas of self-supervised learning (SSL), biological image analysis, and model scaling in deep learning.", "essential_references_not_discussed": "The paper appears self-contained, with citations covering relevant literature.", "other_strengths_and_weaknesses": "The paper presents the largest-scale foundation model for cell microscopy to date, leveraging self-supervised ViTs and a curated dataset (Phenoprints-16M). While self-supervised learning (SSL) has been explored in biological imaging, the combination of large-scale MAE training, dataset curation, and systematic layer-wise probing represents a novel and impactful approach. \n\nMy major concern is about the technique innovation. Given that the study primarily applies existing methodologies, its contribution in terms of methodological novelty may be limited.\n\nThe study focuses on cell microscopy images, but its applicability to other biological imaging modalities (e.g., histopathology, electron microscopy, organoid imaging) is not discussed. A brief analysis of transferability would improve the paper’s broader impact.\n\nThe paper focuses on ViT-based MAE models but does not compare against CNN-based architectures (e.g., EfficientNet, ResNet, U-Net), which are still widely used in biomedical imaging.\nWhile ViTs have advantages in self-supervised learning and scaling, a baseline comparing MAE-G/8 to a strong CNN or hybrid transformer-CNN model would provide additional perspective.", "other_comments_or_suggestions": "I have no other comments or suggestions.", "questions_for_authors": "I have no more questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qagIcQMg7p", "reviewer": "Reviewer_onkd", "summary": "High content screening (HCS) involves subjecting cells to thousands of perturbations in parallel, and capturing subsequent morphological changes via fluorescent imaging. The scale of data generated my modern experimental workflows has since necessitated automated analysis. In light of the success of foundation models, which leverage generic datasets to create representations that are more useful than representations derived from task-specific data alone, the field of machine learning has developed a number of foundation models for HCS. Experimental data, however, is inherently limited, and hence methods are needed to address how to improve model performance without simply scaling to every larger amounts of data. \n\nThis work provides evidence to support three key claims i) that careful curation of data generated by HCS that balances a dataset in terms of morphological variation, while keeping diversity high, can improve the performance of HCS foundation models ii) that intermediate layers of models for HCS can provide stronger performing representations than the commonly used final layer and iii) that they train the largest HCS model to data, that significantly improves on a number of challenging biological benchmarks, and compare performance with models trained on non-biological images, and models trained on alternative HCS data.\n\n\n## Update after rebuttal \nI have reviewed the paper, the reviews, and rebuttals again. I am inclined to keep my score at a strong accept - this is a well written paper, with results which I think are noteworthy for the community working in machine learning for high throughout screening data.\n\nI  believe that curating a dataset by filtering images with respect to perturbation consistency and diversity is sufficiently novel for publication at ICML. I also think the work showing that the linear probing task is correlated with performance on biological recall is noteworthy, and adds merit for publication for an Applications-Driven submission.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful and encouraging review.\n\n**On training ViT-G/8 on RPI-93M**: We agree this would have been valuable. Due to compute constraints, and the insight gained from training ViT-L/8 on both RPI-93M and PP-16M, we prioritized training ViT-G/8 on the curated PP-16M dataset only. Based on the consistent improvements we observed with ViT-L/8 across replicate consistency metrics, we hypothesized that scaling model size on PP-16M would be the most effective use of resources.\n\n**On recall differences on JUMP-CP**: Thank you for highlighting this point. The comparison to RxRx3 is complicated by both the reduced set of gene KOs in JUMP-CP (∼8,000 vs ∼17,000 in RxRx3) and the increased variability due to assay differences and batch effects. As such, recall values across datasets are not directly comparable, but we agree it's a valuable direction to further explore model generalization across assay types.\n\n**On dataset curation and positive/negative controls**: We appreciate this suggestion and agree that the handling of positive and negative controls is important for shaping the final training distribution. We’ll clarify these decisions in the main text for the camera-ready version, particularly in the section describing dataset construction (lines 134–141).\n\n**On Figure 5 readability**: Thank you for pointing this out. We’ll include a summary table in the appendix to complement Figure 5 in the camera-ready version. This benchmark is also now described in detail in RxRx3-core [Kraus et al. 2025].\n\n**On impact of longer training for MAE-L/8 on RPI-93M**: This is a great question. While we did not extend training for MAE-L/8 on RPI-93M beyond what was reported in Kraus et al. 2024, the contrast between RPI-93M and PP-16M models trained for the same number of epochs suggests that dataset composition has a major effect on replicate consistency. We agree that further experiments exploring longer training on RPI-93M would help isolate the contributions of training length vs. data curation.\n\nThank you again for your supportive feedback and helpful suggestions.\n\n**References**:\n* Kraus et al. 2025 [RxRx3-core] – https://arxiv.org/abs/2503.20158", "claims_and_evidence": "There are four main claims in this paper. The experiment design and choice of benchmarks supports these claims with evidence that is sufficient for acceptance. \n\nClaim 1: by curating the images in the proposed manner, and training for longer, better performance can be achieved from fewer epochs. \n - The selection of baselines includes MAE-L/8 trained on either PP-16M or 93M. The performance in table 1 shows little change in the recall of known biological relationships between these two models, despite the large difference in number of images and epochs used in training. More importantly, the KS and CM statistics reflecting replicate consisitency, show large improvements when using the curated dataset. \n- These claims would be better had the authors trained a MAE-L/8 with 93M for longer. How much impact did training length have on replicate consistency? \n\n\nClaim 2: Intermediate layers can provide better representations for downstream tasks\n- This claim is well supported throughout the manuscript, where representations from earlier blocks are found in many cases to outperform final layer representations. This is not only noteworthy for performance gains, but also reducing inference costs. \n\nClaim 3:   linear probing performance on a subset of genetic perturbations correlates strongly with downstream performance on whole-genome benchmarks \n - Figure 4 shows a clear linear relationship, between linear probing and whole-genome benchmarks. \n\nClaim 4: MAE-G/8 outperforms SOTA across biologically relevant benchmarks \n- MAE-G/8 (trimmed) is consistently shown to outperform the well-selected baselines.", "methods_and_evaluation_criteria": "This work includes comparisons between a number of models. Each of these are variants of Vision Transformers (ViT), differing in number of parameters and training dataset. Including a MAE-L/8 trained on RPI-93M and comparing this to a MAE-L/8 PP-16M provides good evidence for the claim that effective models for HCS can be produced with curated datasets. It would have been interesting to include a comparison with the MAE-G/8 trained on the 93M dataset, albeit for less epochs, to show that it is the PP-16M dataset which is the decisive factor in scaling to the ‘giga’ family of ViTs for microscopy. \n\nThe use of linear probing to evaluate the performance of embeddings at earlier layers of a trained model, is well motivated in the paper, as it is computationally unfeasible to perform a whole genome evaluation. Using the 1139-class RxRx1 dataset for genetic perturbation prediction is a challenging task, and demonstrates the claim that earlier layers provide better embeddings with sufficient evidence. \n\nPerhaps most crucially, the model is evaluated on external data, namely the JUMP-CP dataset, which the authors describe as likely to be generated by different assay protocols. Recall of biological relationships using any model is lower, but their MAE-G/8 still shows best recall overall. \n\nOverall, high quality datasets and models are used to provide evidence to support the author’s claims. At a high level, this work is about developing a high-capability model for data generated by high content screening. Due to the variety of sources of batch effects that challenge machine learning in this domain, the combination of evaluation on high quality datasets and focussing on replicate consistency and recall of biological relationships, positions this paper well in the literature.", "theoretical_claims": "Theoretical proofs are not a point of discussion within this paper.", "experimental_designs_or_analyses": "I checked the validity of the experiment testing linear probing across different ViT blocks. A new dataset, different to the pretraining datasets used to train the models, was used for linear probing. Comparing each of these pretrained models on a new evaluation dataset is a fair comparison, and the set of baselines used for comparison are strong. Hence, the experiment seemed to be well designed and supports the relevant claim that representations from earlier model layers can be more useful than the output layer.", "relation_to_broader_scientific_literature": "This paper provides a new model for HCS that scales to a 1.9 billion parameter model. Previous work [1] has shown that the downstream performance of HCS models is correlated with the number of FLOPs in training. This work has shown that this scaling continues into the regime of billion parameter models. Crucially, to achieve this, they did not have to scale the size of their dataset, by scaling their HCS data generation pipeline, but instead only needed to curate their dataset and scale the number of model parameters and training time.\n\nThis model is a ViT and is not channel agnostic as in works [1] and [2]. Previous work has shown that regular ViTs can outperform channel agnostic ViTs and it remains an open question whether a channel-agnostic model can be developed with SOTA performance. \n\nPrevious work has shown that representations of data from layers prior to the output layer have better downstream performance, for example the work [3]. This work also demonstrates that earlier layers of a trained ViT can provide representations that perform better on downstream tasks. As such, this is not a novel result in and of itself, but this is the first time this has been demonstrated in models for HCS. However, using linear probing to more efficiently search for the best performing model is a novel result, as it was not clear that linear probing performance on a genetic perturbation classification task would correlate with the whole genome biological relationship recall task. \n\n- [1] Masked autoencoders for microscopy are scalable learners of cellular biology  O. Krauss et al 2024\n- [2] ChAda-ViT : Channel Adaptive Attention for Joint Representation Learning of Heterogeneous Microscopy Images,  N. Bourriez  et al 2024\n - [3] MIM-Refiner: A Contrastive Learning Boost fromntermediate Pre-Trained Masked Image Modeling Representations, B. Alkin et al 2025", "essential_references_not_discussed": "No essential references come to mind. The related work sections correspond to the core contributions of the paper, and mention the relevant literature.", "other_strengths_and_weaknesses": "The overall quality of authorship is very high, this was a good paper to read. The potential impact of this work is high, due to it addressing the issue of needing to generate HCS data at greater scales to scale ML models.", "other_comments_or_suggestions": "Figure 5 is a little difficult to read due to the large number of overlapping lines. Would a table work as an alternative to highlight its main points?", "questions_for_authors": "The claim that the curated dataset allows to train effective models with higher performance, by training with a curated set of images for more epochs would be more clearly supported had the authors trained a MAE-L/8 with 93M for more epochs. In particular, the recall performance between the MAE-L/8's trained on PP16M and 93M is similar in Table 1. How much impact did training length have on replicate consistency?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received very divergent scores, with a strong accept, an accept, a weak reject, and a reject. 

Specifically, the reviewers generally acknowledge the effort to train a foundation model on large-scale microscopy datasets, which could become a benchmark for future work in microscopy image analysis as well as the extensive evaluation. 
While the proposed framework, consisting of curating the training dataset, scaling to larger models, and evaluating intermediate representation, is appreciated by some reviewers, concerns about the novelty are raised. In the first revision, the reviewers generally agree on the fact that the single components used in the proposed framework are not novel, limiting the technical and methodological novelty of the work. Specifically, while two reviewers acknowledge the novelty and impact in the specific field of microscopy image analysis, the other two question the limited improvements in terms of performance that, together with the limited novelty, hinder the overall contributions of the paper. 

In the rebuttal, the authors provide more information on the specific significance of the three steps of the pre-training approach, pointing out the novelty with respect to the domain of application, and after discussion, all the reviewers maintain their original scores. 

After reading the paper, the reviews and the rebuttal, and considering the overall scores, despite the AC agreeing on the single components being not novel in terms of pre-training strategies, they believe that the main contributions may be impactful in the specific domain of application, including the release of the trained foundation model, that can potentially be exploited for more application works. 
On top of this, considering that this is an application paper, well-written and clear overall, the AC leans toward acceptance of this work.
Decision: Accept (poster)