Counting cells can accurately predict small-molecule bioactivity benchmarks

Domain: neuroscience
Published: 2026-02-06
Editorial outcome: major_revision

Abstract
Accuratelypredictingtheactivityofachemicalineachbioactivityassaybasedonitsalreadyknownpropertiesisextremelyusefulindrugdevelopment.Unfortunately,wediscoveredthatmanyassaysinwidelyusedassay-activitybenchmarkdatasetsdirectlyrelatetocellhealthandcytotoxicity.Manyotherassaysintendtocaptureamorespecificphenotype,buttheiractivecompoundsimpactcellcount,whileinactivesdonot.Inbothcases,countingcellsachievesunexpectedlyhighperformanceinthesebenchmarks,makingthemlessusefulfordiscerningwhetheradditionalproperties,suchasphenotypicprofiles(mRNAorCellPainting),provideadditionalusefulinformationonbioactivity.Toaccomplishthisgoal,werecommendfilteringbenchmarkstoexcludesuchassaysandincludingacell-countbaseline.Usingabenchmarkwith24protein-targetassays,weconfirmthatmodelsleveragingCellPaintingimage-basedprofilesoutperformedthebaselinecellcountmodel.WeproposeseveralotherpracticalrecommendationsforbenchmarkingmachinelearningmodelsforpredictingbioactivityandassessingtheaddedvalueofmRNA,protein,orimage-basedprofiles.

Introduction
Machine learning is now extensively used to predict the activity of different chemical compounds in cell-based assays, classifying them as active, inactive, or inconclusive (or toxic, non-toxic, inconclusive)1,2. Such models can reduce the need for expensive laboratory experiments and potentially allow for predicting efficacy and toxicity early and inexpensively when developing drugs. Predictive models most commonly rely on chemical structures and features derived from them, such as molecular fingerprints, which are easy to generate3,4. However, improving model performance for compounds outside the training data’s chemical space remains a critical challenge5. Capturing nuances such as activity cliffs, where similar structures show significantly different bioactivity, has proven difficult even for deep-learning architectures, suggesting that chemical representations alone are insufficient for predicting many interactions in complex biological systems6. Also, extrapolating in chemical space is not trivial, and here, incorporating biological descriptors has the potential to increase the applicability domain of a model.

Phenotypic profiling techniques, such as gene expression and morphological profiling, have emerged as promising additional data sources for each compound, which might augment molecular structures in improving chemical activity prediction. Cell Painting is an image-based profiling method that is high-throughput and cost-effective and, therefore, has significant potential7,8. Cell Painting datasets provide rich phenotypic profiles by applying a range of fluorescent stains, which capture the cell’s biological response to various compounds across various cellular compartments and substructures9. This assay has been reported to yield improved performance in downstream tasks compared to chemical structures alone8,9. Since the first demonstration that an image-based assay could predict seemingly unrelated assay outputs10, a trend has emerged: using biological response descriptors, such as transcriptomics data or Cell Painting readouts, independently or combined with chemical fingerprints, to enhance predictive models8,11.

For example, Hofmarcher et al. showed that convolutional neural networks trained on Cell Painting features predicted the outputs of 32% of 209 biological assays with area under the curve (AUC) values exceeding 0.912. Moshkov et al. evaluated chemical structures, Cell Painting images, and gene expression profiles (from the L1000 assay) across 270 assays, finding that each modality independently contributed valuable information13. Fusing the three data modalities improved AUC for 62% of assays (167 out of 270), compared with using chemical structures alone. Sanchez-Fernandez et al. introduced CLOOME, a contrastive learning framework enabling cross-modal querying of Cell Painting images with chemical structures14. They also assessed the transferability of learned representations using linear probing on 209 assay activity prediction tasks with pre-trained image embeddings from CLOOME, which did not use activity data during training. The model achieved an average AUC of 0.714, outperforming fully supervised methods. Ha et al. benchmarked learning models using Cell Painting profiles to predict outcomes of 201 assays, an updated version of the Hofmarcher dataset15. When predicting for assays not included in training the models, they found models performed well across the novel prediction tasks, achieving AUC values ranging from 0.70 to 0.87 as the dataset size increased, effectively generalizing to new assays.

However, when reviewing the assays most accurately predicted in these prior studies, we observed that many were linked to cytotoxicity or cell proliferation. There is a complex interplay between the endpoint one aims to measure (or model) in an assay and various confounding factors, which might lead to misleading performance of any predictive models trained on the data16,17. While cytotoxicity/viability are sometimes the desired assay endpoint, these assays are often run side-by-side with an actual assay of interest (in the same cell type, for example) to improve assay accuracy by identifying and excluding compounds exhibiting cytotoxic effects from progressing further in development. Combining results from the primary assay and the cytotoxicity viability screen enables identifying compounds with specific biological effects, minimizing false positives and off-target effects. For example, Esher et al. addressed the challenge of distinguishing specific from nonspecific (cytotoxicity-triggered) reporter gene activation in six Tox21 high-throughput screening assays16. “Cytotoxicity burst” refers to the activation of stress responses at compound concentrations near the cell death threshold, leading to nonspecific assay results. The authors proposed to compare assay activity to baseline toxicity—the nonspecific accumulation of chemicals in cell membranes that disrupt cellular function at critical concentrations. Even without a specific target, compounds can cause potent toxicity due to hydrophobicity, potentially skewing assay results. They found that 37–87% of active hits in the six Tox21 assays they analyzed were likely due to cytotoxicity burst, while only 2–14% were specific16. This is plausible given assay responses can arise from multiple mechanisms, not all representing compounds with the intended pharmacological action17. Hence, this illustrates that, either directly or indirectly, assay endpoints that one measures (or predicts) can be correlated with confounding factors (often cytotoxicity), and not considering the latter in both measurements and predictive models gives misleading results.

Our originally anecdotal finding that a large proportion of the well-predicted assays in prior benchmarks were cytotoxicity/viability related prompted us to test and discover that simple cell counts are surprisingly predictive in many of these benchmark assay tasks. If simple metrics like cell count are driving much of the predictive power, it would call into question the added value of more complex features, such as Cell Painting or gene expression profiles. After systematic evaluations and confirming the above findings, we identified three practical implications: (1) biased benchmark datasets have a high proportion of cell viability assays; a fairly easy-to-predict assay endpoint that should not be overemphasized in benchmarking studies, (2) many assays in benchmark datasets are specific to targets but the active chemicals in the dataset also impact cell viability and the actual readout is confounded by effects like cytotoxicity burst16, and (3) there is an absence of baseline models trained only on cell counts that can demonstrate the comparative advantage of a model trained on more complex phenotypic profiles.

Establishing simple baseline cell count models is essential in machine learning to assess the value of more complex solutions, akin to “Occam’s Razor” when choosing the simplest model to explain a given observation18,19. Such baselines can include basic molecular features like “atom counts”, which have shown strong performance in a virtual screening setting despite lacking structural information20, or mean or majority-class classifiers in regression and classification settings, respectively1,21,22. Previous studies have benchmarked models that make other kinds of predictions using gene expression data23. For example, Csendes et al. found that simple baselines outperformed state-of-the-art foundation models in predicting post-perturbation gene expression24. Similarly, SpaDo showed that in spatial transcriptomics, simple clustering based on spatial neighbor cell type proportions could achieve comparable or better results than complex graph neural network models25. Motivated by these findings, we focused extensively on benchmarking phenotypic Cell Painting screening using data in this study. In this study, we propose a baseline machine learning model using cell count to assign activity probabilities, as it proves surprisingly effective in many assays, particularly human tumor cell line growth inhibition assays. We recommend that future studies carefully select assay categories for benchmarks and establish baseline cell count models to realistically gauge performance improvements when using high-dimensional phenotypic data, such as images, mRNA, or protein profiles, and present five related recommendations for the machine learning community.

Results
Multiple bioactivity tasks are highly correlated with the simple cell count feature
We first evaluated baseline cell count models trained on a single image-based feature, namely cell count based on segmentation of nuclei in a DNA-stained image, across four previously published bioactivity prediction benchmarks: (a) Moshkov et al.13 (b) Hofmarcher et al.12 (c) Sanchez-Fernandez et al. (CLOOME)14, and (d) Ha et al.15 (Table 1). This allows us to systematically assess the predictive utility of cell count as a prediction score to understand its predictive power. The Moshkov dataset, containing assay readouts generated internally at the Broad Institute, is distinct from the Hofmarcher and Ha datasets, both derived from ChEMBL. Sanchez-Fernandez et al. based their benchmarking on the Hofmarcher dataset. Although Hofmarcher and Ha originate from the same source, they were compiled at different times, leading to minor differences in compound coverage. While the assays vary slightly, all three datasets were evaluated against the same Cell Painting imaging dataset. To allow direct comparison with prior studies, we evaluated each dataset separately rather than combining them into a single benchmark.

We began by assessing the relationship between bioactivity prediction tasks and changes in cell count by comparing cell count distributions for active and inactive compounds across three benchmark datasets. We compared the mean cell count values of active versus inactive compounds, aggregated per assay, using a two-sided independent-samples t-test and found significant differences in all three benchmark datasets (p < 10e-16, Fig. 1a–c) suggesting that cell count alone might contain a substantial predictive signal for many biological assays.

We further analyzed the predictivity of the cell count feature for benchmark datasets used in prior studies and found that solely applying a range of thresholds of cell counts successfully predicted 32% of assays (66 out of 209) in the Hofmarcher dataset, with mean balanced accuracy over 0.70 (Fig. 1d), with 64 of the 66 assays being human tumor cell line growth inhibition assays. Thresholds of cell counts predicted 22% of the tasks in the Ha dataset and 28% in the Moshkov dataset (Fig. 1e, f) with mean balanced accuracy over 0.70. This indicates that these datasets contain a substantial number of assays whose outcomes primarily correlate with cell count and, therefore, are not ideally suited for assessing data modality or model effectiveness across a broad spectrum of assays.

Throughout this work, “baseline cell count model” refers to models using only the cell count feature, with the model type matched to each reanalyzed study for direct comparison: logistic regression for Moshkov et al. and Sanchez-Fernandez et al., and XGBoost for Hofmarcher et al. and Ha et al. These trained models learn the direction of association from data, unlike the fixed threshold analysis above (Fig. 1d–f, Eq. 1), which assumes low cell count indicates activity and requires manual output reversal when assay label conventions differ.

To better contextualize the scope of what cell count captures, we analyzed whether the predictive signal from cell count correlates with cytotoxicity. Specifically, we combined the Moshkov and Hofmarcher datasets and stratified compounds by the number of assays in which they were labeled active. We compared the cell count values for compounds active in multiple assays and found that compounds labeled active in a greater number of these benchmark assays tend to show a stronger and more consistent decrease in cell count (Fig. 1g). While cell count is informative for strong and promiscuous cytotoxic responses, it is considerably less responsive to mechanism-specific or low-grade phenotypic changes, which may constrain its utility for detecting nuanced biological effects.

Cell count matches the performance of models trained on Cell Painting and gene expression profiles on existing benchmarks
We next investigated the information content in the cell count feature; for this, we followed the same train-test splits for each of the studies reanalyzed in this work, and compared our baseline cell count model performance to those models trained on the higher dimensionality data of Cell Painting and gene expression profiles.

In the Hofmarcher dataset12, which was also analyzed in Sanchez-Fernandez et al.14 and of which the Ha dataset15 is a variation, the baseline cell count models (mean AUC=0.68 ± 0.21) achieved similar performance compared to the supervised fully-connected neural network (FNN) model based on the full Cell Painting profiles (mean AUC=0.67 ± 0.19), while the convolutional neural network architectures of ResNet model trained on Cell Painting images directly (mean AUC=0.73 ± 0.19) performed better (Table 2, rows e, f, g). We observed a significant overlap of assays well-predicted across all three models (66 were well-predicted, AUC>0.8), with the two more sophisticated image-based models collectively predicting only 19 additional assays over the baseline cell count model (Fig. 2a). We found performance of all models, whether using cell count or Cell Painting features, was largely based on human tumor cell line growth inhibition assays (Fig. 2b). Ideally, benchmarks should not over-represent a single type of assay, especially viability assays (Supplementary Fig. S1 and Supplementary Data 1), given that it is a readout that can be well-predicted by a single feature such as cell count (albeit with differences due to assay conditions such as cell type and timepoint).

The Hofmarcher dataset contains cell-based assays as well as biochemical binding assays using purified protein targets. Out of the 66 unique protein targets in the Hofmarcher dataset, cell count predicted only one protein target at a performance level of AUC>0.8, while the convolutional neural network architectures of ResNet and DenseNet using images predict six targets and eleven targets respectively (Fig. 2b). This shows that while most methods perform similarly to baseline cell count models when evaluated on all assay types, when predicting small molecule activity on protein targets, Cell Painting images achieve high predictivity for many unique protein targets (for example, ATAD5, IDH1, ATXN2, MAPT, SMAD3, Hsf1, VDR, RAPGEF3, BRCA1, TDP1, and GMNN for convolutional neural network DenseNet model), many of which relate to DNA damage. This shows that the Cell Painting images are able to predict assay outcomes from many unique protein targets where the baseline cell count model fails.

We next investigated whether this substantial overlap in predictable assays was only feasible because of Cell Painting images’ inherent ability to count cells or whether other unbiased profiling methods, such as mRNA profiling, might capture cytotoxicity/viability equally well. Using the Moshkov dataset13, we found a substantial intersection between assays predicted by the baseline cell count model and Moshov’s multi-task neural network trained on gene expression data (Fig. 2c), a similar result as for models using Cell Painting data. 31 out of 49 (63%) assay endpoints well-predicted using gene expression profiles (AUC>0.7, threshold as used by the authors of the study, see Table 3d) were predicted equally well (AUC>0.7) by using the baseline cell count model. Only five assays were uniquely well-predicted by gene expression profiles, while in contrast, 33 assays were uniquely well-predicted by incorporating the entire Cell Painting feature set and 22 assays were uniquely predicted by the baseline cell count model. Overall, we found that the performance of the baseline cell count models using a single cell count feature is comparable to the models trained using Cell Painting and gene expression data for the assays evaluated in prior studies.

This analysis highlights the value of using cell counting models as an initial, interpretable approach before employing more complex features and to provide a baseline for later models.

Gene enrichment for cytotoxic and promiscuous compounds
We hypothesized that (1) a decrease in cell number (Cells_Number_Object_Number) observed in Cell Painting assays may reflect increased apoptotic activity, and (2) compound promiscuity may arise more from general cytotoxicity (e.g., apoptosis) than from selective activity. To investigate this, we leveraged gene expression data from the LINCS L1000 resource to examine the biological processes associated with compound perturbations. For this analysis, we selected only those compounds from LINCS that passed the “is_exemplar” (which identifies a single “exemplar” signature for each perturbagen in each cell line) quality filter and were tested in U2OS, MCF7, or PC3 cell lines. Of the 1,520 compounds that significantly reduced cell count in the Broad Cell Painting assay, 466 had corresponding LINCS mRNA signatures, yielding a total of 550 gene expression profiles.

Gene Ontology (GO) enrichment analysis of genes with a z-score threshold (|z|>2) revealed that transcriptional changes that were induced by compounds known to reduce cell count (by Cell Painting) were consistently associated with particular biologically relevant processes. The most frequently enriched pathways included the intrinsic apoptotic signaling pathway (n=112 compounds), mitotic cell cycle phase transitions (n=101), regulation of cyclin-dependent kinase activity, spindle organization, and nuclear division. Stress response pathways (e.g., reactive oxygen species, hypoxia) and metabolic functions (e.g., cholesterol/sterol biosynthesis) were also enriched.

Analyzing more carefully the degree of cell count impact, we found that compounds more strongly reducing cell count in Cell Painting assays showed the highest enrichment for apoptosis- and cell death–related GO terms (Fig. 2e). Over 70% of compounds in the lowest cell count bins ([−37; −50], normalized feature values against plate-specific DMSO controls) were linked to apoptosis-related terms, with this proportion decreasing as cell count increased. This underscores the importance of cell health and cytotoxicity as major drivers of both morphological and transcriptomic profiles, and highlights the need to account for cell count when benchmarking both phenotypic profiling methods.

We next explored relationships between gene expression and compound promiscuity (activity across many assays). From the set of 24,606 compounds with target activity labels (from Hofmarcher and Moskhov datasets), we extracted 8381 LINCS signatures for 4094 compounds that passed the cell line and exemplar filters. Again, the intrinsic apoptotic signaling pathway was the most frequently enriched category (n=193), supporting our hypothesis that compound promiscuity is often associated with general cytotoxic effects (Fig. 2f).

Beyond apoptosis, other recurrently enriched pathways associated with compound promiscuity included metabolic and biosynthetic processes (e.g., cholesterol/sterol biosynthesis [n=164–156], secondary alcohol metabolism [n=164–124], steroid biosynthesis [n=112]), protein phosphorylation and kinase regulation (e.g., serine/threonine kinase activity [n=161], protein kinase complex [n=131]), and cell adhesion–related categories (e.g., cadherin binding [n=116], focal adhesion [n=89]). Stress-related responses to ROS (n=101), xenobiotics (n=86), and hypoxia (n=84) were also commonly enriched, highlighting the broad cellular impact of promiscuous compounds.

Focusing on the 214 compounds that reduced cell count and were also promiscuous (Fig. 2g), we found that apoptotic enrichment increased markedly as cell count decreased. In contrast, the effect of promiscuity on apoptosis enrichment was more modest (Fig. 2h). Notably, compounds with more than one active hit and a normalized cell count below −40 showed over 70% enrichment for apoptosis-related GO terms.

Baseline cell count models show comparable AUC to meta-learning approaches
We next compared our baseline cell count models with the single-task, multi-task, and meta-learning models using Cell Painting data used by Ha et al.15. We found that the baseline cell count model achieved a median AUC of 0.59 on this benchmark, whereas the “protonetcp+” model, a metric-based meta-learning model using embeddings from Cell Painting features, attained an AUC of 0.64 (Supplementary Data 2). Notably, with regard to the exceptional performance on specific assays, such as CHEMBL211478426 (AUC of 0.87), a viability screen for the primary quantitative high-throughput screening (qHTS) aimed at identifying inhibitors of ATXN expression, the baseline cell count model achieved an AUC of 0.82. The similarity in AUC scores suggests that the baseline is competitive with the more complex model, likely because the assay is linked to cell viability.

Furthermore, Ha et al. posited that pre-training significantly enhanced model performance on other assays, such as CHEMBL211480715,27. On further examination, we found this qHTS assay was designed to identify small molecule activators of BRCA1 expression. The labeling for this assay was as follows: compounds with lower pChEMBL values were classified as activators and labeled “1”. In comparison, those with higher pChEMBL values were labeled “0”, representing likely inhibitors that could impair DNA repair mechanisms and induce cell death27. By reversing the output of the baseline cell count model to predict low cell counts as “0”, we achieved an AUC of 0.76, closely matching the best pre-trained models reported by the authors (AUC=0.77). Hence, we can conclude that it is key to use data in a suitable way when training models and also that cell counts are very predictive for this particular endpoint.

These results underscore a cautionary reminder regarding interpreting AUC values: models predicting assays related to cytotoxicity cannot be easily compared with each other; often, in the case of meta-learning, test assays predict the opposite outcome of the training assays and a model, being unaware of this, predicts the opposite of the assay outcomes precisely. Given that the user always knows what assays it intends to predict, this task does not need meta-learning but merely reversing the prediction outcomes, as demonstrated here. Thus, AUC values provide only part of, but not the most important part of, performance28.

Contrastively learned features do not outperform cell count features
We next evaluated whether contrastive-learned morphological features learn more than just cell count, in the context of assay prediction. We used features learned using CLOOME (a contrastive learning framework that pairs microscopy images and chemical structures to enhance molecular representation learning) to two logistic regression models: one trained with the cell count feature and another with cell count, and two readily-calculated features of chemical structure: molecular weight and logP14. On the Hofmarcher dataset and the publicly available CLOOME embeddings, we tested linear probing, which is a logistic regression task commonly used to assess the quality of representations learned through contrastive learning. It achieved an average AUC of 0.71 ± 0.19 and 57 well-predicted assays compared to using the cell count feature, which achieved a similar average AUC of 0.70 ± 0.22 and 64 well-predicted assays (Table 2, rows a and b). We found that furthermore the logistic regression model trained on cell count, molecular weight, and logP values achieved an average AUC (0.74 ± 0.21) and 64 assays with AUC > 0.9, thus even slightly superior to CLOOME (Fig. 2d). Overall, this highlights the importance of cell counting as a baseline cell count model and the choice of a suitable task/endpoint for model benchmarking, given there was no observed improvement in performance when using contrastive learning embedding.

Appropriate evaluation metrics and assay endpoints are essential when modeling large-scale benchmark datasets
The multi-task neural network models trained on complete Cell Painting profiles in Moshkov et al. achieved extremely high AUC values (1.0) for nine out of the original 270 assays13. Eight of these endpoints contained only one active compound, while one contained two actives. Each endpoint also contained fewer than 52 datapoints in total. This lack of data, combined with the scarcity of active compounds, made high AUCs achievable by chance, by solely predicting one active compound correctly. On inspection, nine of the ten unique compounds labeled as active for these assays were cytotoxic29–31, leading to significant cell death and low cell density in images. Example images of cells, treated with four different compounds (fluspirilene, hinokitiol, pyrazolanthrone, BRD-A64553394) and neutral control DMSO show significant cell death, dark areas, and low cell density, indicating cytotoxicity (the most extreme cases are bordered yellow) are shown in Fig. 3. The images (Fig. 3a–d) may not have been used in generating profiles, as a QC pipeline in CellProfiler is likely to reject them and only keep some images with some surviving cells32; nevertheless, the images show that these compounds are cytotoxic. Therefore, the high performance could also be attributed to models effectively predicting cytotoxicity (low cell counts).

This finding is supported by comparing the AUC scores of the different model approaches after separating the remaining 230 assay endpoints into two groups: 185 assays containing five or more active compounds and assays containing fewer than five active compounds (Supplementary Fig. S2). This reveals that a greater proportion of endpoints are predicted with higher AUC when there are fewer active compounds in the data. This is particularly true in the case of our baseline cell count model, where the use of one feature and a simpler model architecture likely led to less overfitting. Conversely, as the size of the training data increases, a baseline cell count model underperforms the two approaches trained using all CellProfiler features. This demonstrates that, for appropriate endpoints, more complex approaches are deriving additional value from high-dimensional feature sets beyond cell count.

To investigate whether the sole use of AUC could be misleading in terms of judging model effectiveness, for the remaining 230 assays in the Moshkov dataset, we plotted the AUC of each model against AUPRG, which, as mentioned above, provides a universal baseline for comparison between assays that is less impacted by severe class imbalance (Fig. 3f, g). Thresholds to categorize assay endpoints as “well-predicted” were set at 0.7 AUC and 0.3 AUPRG. Using the more stringent performance criteria meant that just 4 assays were well-predicted by our logistic regression model (Fig. 3f), in contrast to the 83 assays predicted with AUC > 0.7 or 13 with AUC > 0.9 (Table 3b). This reduction shows that AUC alone was insufficient to assess the number of “well-predicted” assays.

We also hypothesized that a number of the assay endpoints were highly correlated with cell death and thus would be predicted with high performance by a baseline cell count model. We therefore focused on two distinct endpoint types within the data: resazurin and Wnt/β-catenin signaling pathway assays. The resazurin assay is commonly used to assess both bacterial and eukaryotic cell viability33,34, whilst the Wnt/β-catenin signaling pathway is crucial for cell proliferation and survival35. The Cell Painting experiments used U2OS osteosarcoma cells where Wnt signaling is often active36,37; therefore, it is likely that active inhibitory compounds would result in reduced cell proliferation, as well as increased apoptosis and cell cycle disruption. These endpoint types accounted for 14 and eight assays, respectively, out of the 230 assays in the data. Generally, these two endpoint types were well-predicted by our baseline cell count model, with an average 0.806 AUC and 0.154 AURPG, outperforming the 0.70 average AUC achieved by Moshkov et al.’s late fusion of three different data modalities for those endpoints. Only two out of 22 assays (9%) were predicted worse than random by the baseline cell count model (Fig. 3i). However, the logistic regression model trained on all CellProfiler features results in worse model performance on these endpoint types (Fig. 3h), with five resazurin assay endpoints predicted with AUC < 0.5. Instead, other assays are predicted with greater AUC/AURPG scores in their place, resulting in a similar number of assays above our determined performance thresholds (see Tables 2a and 3b). This finding supports the necessity to filter these benchmark datasets, to focus on endpoints where applying advanced deep learning techniques will yield real benefits over and above the information contained in cell count.

To assess whether the cell count baseline and the full Cell Painting feature model were identifying the same compounds as active, we examined the true positive prediction agreement between the two models across all assay endpoints in the Moshkov dataset. There was a 49.54% overlap between the true positive predictions, suggesting the approaches are partially predicting different groups of compounds as active. To investigate the difference in these compound cohorts, we isolated the compounds that were uniquely predicted by each model. We then calculated the following physicochemical properties for each compound molecule: molecular weight (MW), lipophilicity (LogP), topological polar surface area (TPSA), hydrogen bond donors (HBD) and acceptors (HBA) and rotatable bonds. We also computed the statistical significance of the difference of the means of each metric between the uniquely predicted compounds for each model using scipy’s “ttest_ind” (p<0.05).

The true positive compounds predicted uniquely by the model trained only using cell count were found to have statistically significantly lower average MW and LogP, while having a higher average TPSA and more rotatable bonds when compared with the cohort of compounds uniquely predicted by the model trained using all CellProfiler features from Cell Painting (Supplementary Fig. S3). Generally, lower LogP combined with higher TPSA indicates that the compound should exhibit better aqueous solubility38,39. Lower MW compounds also tend to diffuse more easily across cell membranes, and more rotatable bonds indicate higher molecular flexibility—more promiscuous binding and greater likelihood of fitting into various binding sites40. Taken in combination, this demonstrates that the cell count model is more adept at identifying compounds with more broad, simplistic effects and higher general/nonspecific cytotoxicity as they are more likely to impact broad cellular processes. Conversely, the model trained on all features can categorize compounds that have a lower likelihood of being reliably delivered to cells or exhibit more subtle or specific phenotypes.

To further explore these findings, we examined the structural diversity of compounds that were correctly predicted as active by only one of the models. When comparing compound scaffolds, we observed substantially lower scaffold diversity among the compounds uniquely identified by the cell count model (528 unique scaffolds) compared with those uniquely identified by the full Cell Painting feature model (1175 unique scaffolds). This furthers supports the finding that the cell count model is more narrow in its predictive ability, but when applied in a setting where a high number of compounds are cytotoxic, or assays are primarily aimed at measuring cell viability, this approach can appear to perform on par with a model utilizing all CellProfiler features, demonstrating the necessity for the benchmark curation suggested within this paper.

Consistent cell count reduction across plates shows that benchmark active compounds often induce cell death
Cell count can be impacted by many technical variables, raising the potential for plate or batch effects to confound analysis. In this study, we leveraged Cell Painting data normalized against plate-specific DMSO controls and aggregated across replicates to mitigate technical variability inherent to high-throughput imaging assays. We also evaluated the cell count feature (among others) before aggregation and found that, although artifacts exist that can affect cell count, the predictive power of cell counts is not an artifact, but rather a real phenotypic effect. While the assay outputs we aim to predict may themselves be subject to technical or batch effects, it is important to note that these readouts originate from independent experiments distinct from the imaging data, with different plate layouts and batching, and without marked grouping of activities on particular plates.

Specifically, we found that of 402 plates from the Broad Cell Painting dataset, 99% contained both bioactive and inactive compounds, defined by activity annotations from the Hofmarcher, Ha, or Sanchez datasets. Notably, across these plates, bioactive compounds exhibited a consistent reduction in mean cell count relative to inactive compounds in 88% of cases, even before aggregating features at the compound level (Fig. 4a). This reproducibility supports the relevance of cell count as a sensitive phenotypic marker.

We found that variability in pre-normalized cell count ranks moderately (94th out of 1783 features) in terms of mean standard deviation across plates. Features such as nuclei area shape exhibited even higher inter-plate variation (Fig. 4b). Overall, the consistent decrease in cell count observed across nearly all plates for bioactive compounds reinforces confidence in cell count as a phenotypic marker; however, this also highlights that the benchmark assays currently used are heavily biased toward viability-related endpoints, and consequently, the identified bioactive compounds largely reflect viability effects.

Next, we aggregated the Cell Painting features as medians across replicates, and we filtered compounds to retain unique compound skeletons in the bioactive and inactive groups, minimizing structural redundancy. Still, we found that bioactive compounds tend to have a larger number of Cell Painting features exhibiting higher absolute correlation with the cell count feature compared to inactive compounds (Fig. 4c).

These findings suggest that bioactive compounds, likely those that reduce cell count, are associated with diverse phenotypic changes captured by the Cell Painting features. Overall, we found that the predictive power of cell counts is not an artifact of actives being concentrated on particular plates in the experiment, but rather a real phenotypic effect.

Comparing compounds to positive controls is not a good benchmark for ML models
Cell Painting experiments use positive controls, which are typically compounds that exhibit a strong phenotypic change, often including a reduction in cell counts. Therefore, studies that compare methods by evaluating whether models can predict if a compound is similar to positive controls are likely confounded by cell count differences. For example, Cross-Zamirski et al. demonstrated label-free prediction of Cell Painting profiles from brightfield images41. This approach enabled the clustering of promising compounds with a positive control compound using a k-NN classifier applied to features from actual Cell Painting fluorescent images and predicted ones derived from brightfield images. The authors reported a sensitivity of 62.5% and a specificity of 99.3% when predicting cytotoxicity, with correct classifications for eight toxic compounds.

Among the label-free models, the feature most strongly correlated with the ground truth was Cells_Neighbors_SecondClosestObjectNumber_5 (Pearson correlation=0.98), which is closely linked to cell count. Using this feature, we evaluated the 170 treatment wells in the test set and identified eight treatment compounds within 1.5 standard deviations of the positive control’s feature value (Fig. 5a). When k-NN models were trained on either the full Cell Painting profiles (ground truth) or the single cell neighbor feature, they identified 7 and 19 treatment wells, respectively, as similar to the positive control. In contrast, label-free Cell Painting profiles generated using cWGAN-GP and U-Net models identified 10 and 14 wells, respectively. As shown in Fig. 5b, the majority of wells identified by the label-free models were encompassed within the predictions of the baseline cell count model.

Because the dataset used by Cross-Zamirski et al. is not publicly shared, it remains unclear whether these well-level predictions align with their reported compound-level classifications. However, the strong correlation observed between the ground truth and the label-free cell count feature suggests that the predicted compounds from both datasets likely overlap. This evidence highlights the importance of including a baseline cell count model when presenting more complex models. Both Cell Painting and label-free models consistently identified compounds causing significant decreases in cell count, similar to the positive control compound mitoxantrone, a known cytotoxicant. Therefore, baseline cell count models serve as a reference point for evaluating and interpreting results from more complex models, particularly when the task is linked to separating positive controls from other perturbations.

Concentration-dependent effects enhance information content in Cell Painting data
Most studies use data from two large public Cell Painting datasets: Bray et al.42 and JUMP-CP43. These datasets primarily consist of single-dose experiments, where each compound perturbation was tested in multiple replicates but at one concentration, usually 10 μM. A single-dose setup makes it difficult to ascertain whether the compound concentration is optimal to capture unique effects of the compound, or alternatively, whether it is higher than desired and begins to capture effects associated with different mechanisms, including those that are outright cytotoxic (conditions where baseline cell count models perform well). Hence, we next explored whether the information content in Cell Painting profiles is concentration-dependent, using datasets with multiple compound concentrations.

To this end, we used data from a recent study by Comolet et al., which developed ScaleFEx, a memory-efficient and open-source pipeline for extracting interpretable cellular features from large high-content imaging datasets44. We evaluated the ScaleFEx feature space to identify phenotypic shifts in drug-treated cells. For example, the study validated ScaleFEx features on fibroblasts treated with a single drug, Y-39983-HCl, which revealed dose-dependent phenotypic changes. At 1.0 μM, the drug caused significant alterations, including reduced cell size and irregular cell shape, aligning with its known disruption of actin filaments. The authors showed that a logistic regression model trained in a five-fold cross-validation with plate held-out splits was able to differentiate the drug from the controls with high accuracy (AUC=1.0). At a lower dose of 0.2 μM, more subtle but still significant differences were observed, such as decreased nuclear eccentricity and altered cell granularity; this shows that lower doses can still induce phenotypic changes that are useful. We note that a leave-one-plate-out cross validation, where unique perturbations (compounds and dose) are spread across different plates, might leak information in most modeling approaches; however, in this case, it is suitable for accounting for what makes a compound perturbation unique to the imaging across the experiment. Therefore, we retained the same leave-one-plate-out cross validation as the original authors. We reproduced this experiment with only a single feature (“MitoCount”, the number of individual counts in the Mitochondria skeletonized mask, which is the closest related available feature to cell count) in order to compare their approach with a baseline cell count model. We found there was still a significant separation between the compound, Y-39983-HCl (1.0 μM), and controls when using the LR model with only MitoCount as a feature (AUC=0.85), as shown in Fig. 5c–f. For other compounds (including Y-39983-HCl at 0.2 μM), we observed better performance for ScaleFEx features (mean AUC=0.99) compared to baseline models using MitoCount (mean AUC=0.66). While the original paper identifies Zernike_ch1_1 as one of the features contributing to model performance and demonstrates its ability to distinguish Y-39983-HCl (1.0 μM) from the control, we additionally show that the MitCount feature achieves a comparable level of significance in this separation (Fig. 5f). Overall, this shows that a baseline cell count-related model is a better benchmark than a random model. It also demonstrates that where phenotypic activity is subtle (at low concentrations), the full morphological profiles outperform a single feature related to cell count (as observed in the case of Y-39983-HCl at 0.2 μM dose).

To explore the concentration-dependency of information in Cell Painting profiles further, we evaluated the dose-response dataset as a proof of concept (an earlier version of this dataset is described in Ewald et al.45). We chose six compounds, chosen to highlight the separation between cytotoxic and bioactivity signals along concentration-response curves. Specifically, we chose three compounds that induced significant cell death at higher concentrations (Fig. 6a–c). We also selected three compounds that exhibited distinct phenotypes without cytotoxicity across the tested concentration range (Fig. 6d–f). Notably, the three cytotoxic compounds showed early morphological changes at concentrations ten times lower than those causing detectable cell death, as indicated by nucleus counts (Fig. 6a–c). Although these findings are preliminary and specific to the six compounds, this is consistent with our larger analysis in Ewald et al., which found that across >1000 compounds, early morphology changes were observed at doses about an order of magnitude lower than cytotoxicity readouts. Further studies affirm that for most compounds that exhibit morphological changes, the information content in Cell Painting is dependent on dose45.

Benchmarking protein-target prediction with Cell Painting profiles compared cell counts
In order to facilitate the above recommendations, we are together with this work releasing a benchmark dataset for compound-activity prediction curated along the recommendations above from EvEBio50. This benchmark dataset comprises ligand binding and functional assays in biochemical systems for nuclear receptors (NRs), alongside cell-based activity measurements for seven-transmembrane receptors (7TMs), supplemented by cell viability assays to monitor cytotoxicity, collectively covering a broad panel of these receptor classes. After applying the same filtering steps described above (see Benchmarking Cell Painting for curated Protein-Target data), we isolated a high‑quality benchmark of 24 protein‑target assays (including agonist and antagonist modes) with binary activity labels (active/inactive) across 269 compounds, which also have Cell Painting data publicly available in the Broad dataset. Using this benchmark, models leveraging the full Cell Painting profiles outperformed the baseline cell count model, achieving a higher mean balanced accuracy (BA=0.67 vs. 0.52; see Fig. 7a). Compared to both a randomly shuffled baseline and the cell count baseline, the full-profile models delivered over 20% improvement in AUC-PR on 8 of the 24 tasks (Supplementary Fig. S4). Notably, gains in AUC-ROC were consistent regardless of the test set size or the proportion of active compounds, suggesting that these improvements are robust and not driven by dataset composition (Supplementary Fig. S4).

The assays most improved by including Cell Painting information were for GR agonism (mean AUC=0.79 with the complete Cell Painting profile vs. AUC=0.60 with cell count baseline model) and RARb antagonism (AUC=0.83 vs. AUC=0.64). The feature Nuclei_RadialDistribution_RadialCV_ER_4of4 was found to be the most predictive of this endpoint (Fig. 7b); this feature is the coefficient of variation of intensity in the ER channel within the outermost region of the nucleus object. For RARb antagonism, the most predictive feature was Nuclei_Texture_Entropy_Mito_10_0 (Fig. 7c), which measures the disorder or heterogeneity of the mitochondrial signal. Although mitochondria are cytoplasmic, limited image resolution and perinuclear overlap can cause mitochondrial signals to be captured within the nuclear mask. In both the GR agonist and RARβ antagonist cases, the phenotypes are subtle and not readily apparent by visual inspection (as confirmed in Supplementary Figs. S5 and S6, which show no substantial changes in cell count). However, these effects are best captured through image-derived features. The superior performance of models using the full Cell Painting profiles compared to cell count baselines further supports the idea that spatial and texture-based features carry the relevant biological signal.

Overall, the value of imaging-derived features largely hinges on the screening context. Previous studies have explored the importance of particular Cell Painting features with respect to mitochondrial toxicity (edge of the mitochondria object)49, cardiotoxicity (fine-grained smoothness of the ER staining and size, shape, number, or texture of nucleoli within the nucleus)51, and cytotoxicity prediction52, particularly also in muscle cells (spots edges and ridges profiles in the ER and mitochondria channels)53. Our results underscore the utility of the complete Cell Painting profile as high-content imaging data for capturing biologically meaningful phenotypes associated with signaling pathways, not just limited to cell count. These phenotypic responses become more detectable when cytotoxicity is not pervasive in the dataset, causing models to be focused on cell count over more subtle cellular changes.

Methods
Cell Painting data
We used the Bray Cell Painting dataset42 (cpg0012-wawer-bioactive-compoundprofiling) in this work, which contains images of cells treated with over 30,000 chemical perturbations. We used the dataset processed by Seal et al.49, where for each plate, the average feature value of the DMSO controls was subtracted from the average feature value of the perturbations and then divided by the standard deviation for the DMSO samples on the plate. Finally, the median feature value was calculated for each compound and dose combination. For replicates, the median feature value was considered only for doses within one standard deviation of the mean dose across all perturbations of the same compound. Thus, the features from the Cell Painting dataset used in this study are the deviation from DMSO controls on the same plate after normalization, and after aggregating the median across replicates. We use the Cell Painting descriptor “Cells_Number_Object_Number”, which correlates directly with cell count, as the sole feature in our cell-count baseline cell count models.

Bioactivity data description
We next gathered three annotated assay benchmarking datasets from the literature: Moshkov et al.13, Hofmarcher et al.12, and Ha et al.15 (Table 1). The Moshkov dataset contains bioactivity data from 270 assays, predominantly cell-based and toxicity-related (e.g., growth inhibition, viability) for 16,170 compounds, with 13.4% completeness and 2.7% active datapoints (defined from individual assay hit calls). The Hofmarcher dataset, sourced from ChEMBL, comprises data from 209 assays, featuring quantitative pChEMBL data from individual protein targets, as well as tumor cell line growth inhibition assays, for 10,573 compounds, with 2.5% completeness and 34.7% active datapoints. The Ha dataset, an update on the Hofmarcher dataset, comprises data for 201 assays from ChEMBL, focusing on protein target activity for 10,526 compounds, with 2.6% completeness and 34.7% active data points. These datasets have been used before to evaluate novel machine learning algorithms predicting compound activity using Cell Painting descriptors, making them suitable for comparing our baseline cell count models12–15.

Comparing active with inactive compounds
Based on binary activity annotations in each of the three datasets, we compared the cell count distributions between active and inactive compounds for each assay using an independent t-test (as implemented in scipy.stats55). To mitigate bias from unequal group sizes, we sampled both groups to match the size of the smaller group. We then compared the distribution of the Cells_Number_Object_Number feature for active versus inactive compounds with a t-test for independent samples using the Bonferroni correction.

Predictivity of cell count
We systematically assessed the predictive utility of the cell count feature across multiple assays by using the raw cell count as a prediction score to understand its predictive power by evaluating classification performance at different thresholds. For each assay across each of the three datasets, binary predictions were generated using a probabilistic scoring approach (logistic regression with fixed weight of −1) based on the Cells_Number_Object_Number feature. A sigmoid-like function (Eq. 1) was applied to transform feature values into probability scores, mapping the difference between each value and a specified threshold to a probability score between 0 and 1.

Probability Score = 1 / (1 + e^(v-t)) (Eq. 1)

Where v is the raw feature value for a specific compound, in this case, the Cells_Number_Object_Number, and t is the chosen threshold used to center the transformation. Equation 1 thus assigns higher probabilities to compounds with lower cell counts, consistent with cytotoxic effects.

We then evaluated how varying thresholds on the cell count distributions impacted the ability to predict assay outcomes. The separation of classes was assessed using balanced accuracy at each threshold (probability scores greater than or equal to 0.5 as positive and those below as negative), emphasizing the ability of the cell count feature to distinguish positive from negative classes. Since our goal was solely to separate classes and not to train and test an ML model, a low balanced accuracy score of 0.10 was interpreted as 0.90, given that the thresholding model is fixed to classify compounds that decrease cell count as active (and often makes opposite predictions). This approach emphasizes our focus on class separation rather than absolute predictions, meaning the assay-specific definition of activity (e.g., whether increased cell counts indicate activity) does not affect our interpretation.

Predicting compound activity from phenotypic profiles
The Moshkov dataset13 uses binary classification across 270 tasks, with scaffold-based splits for training and testing defined by the original authors. We excluded 40 assays, which did not contain enough data points to guarantee that at least one active and one inactive compound is included in the training and test data that Moshkov et al. used for each cross-validation fold. A logistic regression classifier, using sklearn’s default hyperparameters, was trained using “Cells_Number_Object_Number” as the sole feature to predict each binary endpoint. For comparison, a separate logistic regression model was trained using all CellProfiler features for each endpoint. L2 regularization with C=0.001 and balanced class weighting parameter ensured the model handled the high feature dimensionality and class imbalance well. Five-fold cross-validation was performed using the splits provided by the authors, and performance was measured using the Area Under the Receiver Operating Characteristic Curve (AUC) in combination with the Area Under the Precision-Recall-Gain Curve (AUPRG)56. AUPRG was selected as an additional metric due to the large class imbalances in most of the endpoints within the Moshkov dataset, where 232 of the original 270 tasks had at least five times more inactive than active compounds, and 185 had at least a 10:1 inactive-to-active ratio. AUPRG provides a universal baseline, which is comparable across assays with different class distributions (unlike Area Under the Precision-Recall Curve, AUCPR), and is less impacted than AUC by the abundance of negative class examples that are present in these classification tasks. For other tasks, we limited the analysis to AUC-ROC to enable direct comparison to prior published results.

Comparing the prediction of biological assays between Cell Painting and convolutional networks
We used the Hofmarcher dataset with the same train-validation-test datasets as in the original publication (70-10-20 splits repeated three times for 209 tasks)12. An XGBoost classifier (chosen for the sake of comparability with the original paper) was trained for each assay using only the cell count feature, with hyperparameter optimization performed via randomized search across a predefined grid in a 5-fold stratified cross-validation. The baseline cell count models were trained on the 80% training set and evaluated on the 20% test set. This process was repeated for all three train-test splits provided by the authors. Model performance was assessed using the AUC, allowing for direct comparison with results from Hofmarcher et al.

Comparison to contrastive learning (CLOOME): bioactivity prediction
We next compared the performance of a logistic regression (LR) model trained on the cell count feature to Sanchez-Fernandez et al.14, who trained an LR model on CLOOME embeddings for each of the 209 targets in the Hofmarcher dataset to compare feature spaces like-to-like, while keeping all other parameters identical. Since CLOOME embeddings have access to chemical structures, two baseline cell count models were considered: (1) using cell count as the sole feature and (2) using cell count along with molecular weight and logP (calculated using RDKit) as very simple chemical features. The first baseline excludes any chemical information, while the second baseline, although not explicitly trained on chemical features, incorporates some chemical-related information indirectly through molecular weight and logP. Molecular weight and logP were chosen as proxies for chemical information, as these are known confounding factors in the chemical assays57,58. Employing the same 70-10-20 data splits, multiple logistic regression models were trained per assay with varying regularization strengths, selecting the best based on validation AUC. Optimal models were then evaluated on the test set. This process was repeated for all three train-test splits provided by the authors, with evaluations based on the mean AUC for each assay.

Comparing enrichment in gene ontology for selected compounds
We performed two Gene Ontology (GO) enrichment analyses to characterize the biological processes associated with compound-induced transcriptional changes. This work was built on a previous study59, in which a framework was established to connect transcriptional and morphological perturbations.

Transcriptomic signatures were obtained from the LINCS L1000 dataset (level5_beta_trt_cp_n720216x12328.gctx) using the signatureSearch R package (v1.12.0)60. First, we analyzed compounds for which Cells_Number_Object_Number values are relatively low compared to controls in the Cell Painting assays. Second, we focused on promiscuous compounds, where we combined the Moshkov and Hofmarcher datasets and stratified compounds by the number of assays in which they were labeled active. Among the 1520 compounds in Cell Painting that have a value of Cells_Number_Object_Number lower than −14.35 (min=−50.59), 534 were also present in the LINCS resource and passed the is_exemplar filter (which identifies a single “exemplar” signature for each perturbagen in each cell line). Of these, 466 compounds were tested in U2OS (n=383), MCF7 (n=73), or PC3 (n=10) cell lines. Cell line selection followed the priority order U2OS > MCF7 > PC3, based on our prior demonstration that these cell lines share over 80% of genes with U2OS59. For the analysis on promiscuous compounds, starting with 24,606 labeled compounds (0–86 active labels per compound across the combined Hofmarcher and Moshkov datasets), we identified 8082 compounds present in LINCS that passed the is_exemplar filter. Of these, 4094 were tested in U2OS (n=3,630), MCF7 (n=434), or PC3 (n=33) cell lines. We restricted the analysis to the landmark gene set, which represents directly measured genes (978 genes) in the L1000 assay and provides the most reliable information content. For each compound, perturbed genes were selected using a z-score threshold (|z|>2). GO enrichment was performed with the clusterProfiler R package (v4.6.2), using the enrichGO function and the org.Hs.eg.db annotation database. Parameters were set as follows: pAdjustMethod=“bonferroni” and pvalueCutoff=0.05.

Few-shot learning for molecule activity prediction
Subsequently, we compared baseline cell count models to those developed by Ha et al., who used 161 assays for training a few-shot learning model and 18 separate assays as the test set15. Since no training data were available for the test assays, the baseline cell count model was trained by combining the outcomes of all 161 training assays into a single endpoint. This approach encourages the model to capture general trends rather than assay-specific information, aligning with our hypothesis that this generic information might primarily reflect cell death. Hyperparameter optimization for the XGBoost model was performed using a 5-fold stratified cross-validation strategy, with cell count as the sole feature. Model performance was evaluated individually on each of the 18 test tasks designated by the authors, using AUC as the performance metric.

Comparison to label-free Cell Painting profiles from brightfield images
We next used the test dataset provided in Cross-Zamirski et al.41, which included 77 negative DMSO wells, 26 positive wells, and 170 compound perturbation wells. We used 4 feature spaces, namely (a) Cell Painting profiles from the ground truth, (b) features from predicted images from the conditional Wasserstein Generative Adversarial Network with gradient penalty (cWGAN-GP) model, (c) features from predicted images from the U-Net model as released by the authors, and (d) the baseline cell count model using the Cells_Neighbors_SecondClosestObjectNumber_5 feature from the ground truth profiles. To identify compounds with phenotypic similarity to positive controls, we first used a k-Nearest Neighbors (k-NN) classifier (k=5) with Euclidean distance in each of the above feature spaces to distinguish between the positive and negative controls. The model was trained and evaluated through 100 iterations, where the training data was randomly resampled in each iteration to ensure balanced representation of both classes. Predictions for each of the 170 compound perturbation wells were aggregated using majority voting across all iterations, identifying those test compounds predicted as the positive control.

Comparison to feature extraction from high-content imaging screens
We next used the dataset released by Comolet et al.44, which consisted of fibroblasts from 20 donors treated with three compounds (CP21R7, Pemigatinib, and Y-39983-HCl) at two concentrations (0.2 and 1.0 μM) each in order to compare the ability of various Cell Painting feature space to identify compounds from controls. The authors performed confounder-correction (plates, wells, rows, columns, donor), normalized between 0 and 1, and averaged at the well level to mitigate noise and emphasize the predominant effects observed across the majority of the cell population44. We used two feature spaces to compare the ability to separate perturbations from DMSO controls – first, the ScaleFEx’s features, which included all extracted morphological and phenotypic descriptors from cell imaging assays and second, only the MitoCount feature extracted by ScaleFEx, which was the closest related feature to cell count44. We reproduced the methods described by Comolet et al. to assess the accuracy of a logistic regression (LR) model designed to distinguish each drug from the control (DMSO). For each drug and concentration, a balanced training set was created by sampling DMSO controls to match the minority class. For models using ScaleFEx features, a Recursive Feature Elimination (RFE) and logistic regression model was employed to identify the most predictive, non-redundant features for classification. For the baseline cell count model using MitoCount, no feature selection was used. A logistic regression model was trained and validated using leave-one-plate-out cross-validation (which results in information leak of compounds which was spread across all plates), that is, in each fold, training is performed on all but one plate, which is held out for testing. The models are trained iteratively to classify wells treated with each drug and DMSO controls. The values were corrected for plate, donor, column, and row effects and normalized to a range of 0–1. A Mann–Whitney U test was used to assess statistical differences between a drug and DMSO control, with Bonferroni adjustment applied to account for multiple comparisons.

Evaluating the effects of concentration-dose response in Cell Painting
We hypothesized that the signal in Cell Painting depends on compound concentration and that single concentration data available in public datasets may not always be optimal in detecting the morphological changes that ML models can learn from. As an illustrative example, we generated dose-response Cell Painting data for six compounds: aminohexylgeldanamycin (Hsp90 inhibitor), givinostat (HDAC inhibitor), repotrectinib (tyrosine kinase inhibitor), I-BET282 (pan-inhibitor of all eight BET bromodomains), bisindolylmaleimide V (protein kinase C inhibitor), and PP58 (Src inhibitor), in 2 replicates and ten doses.

Benchmarking Cell Painting for curated protein-target data
To evaluate the use of Cell Painting assay to predict protein targets of compounds, we curated a benchmark dataset using the recently released data from EvEBio (Release #4 as of May 17, 2025)61, covering both agonist and antagonist modes across nuclear receptors and 7‑transmembrane receptors. This dataset brings the total screened combinations to approximately 237,490 compound–target pairs, following a two-phase quantitative protocol: an initial screen at three high concentrations: (0.6, 2.5 and 10 μM), in duplicate, followed by a higher-resolution profiling phase for compounds meeting progression criteria. Activity assignments are determined algorithmically based on curve fitting, potency metrics (pXC50), and gating rules across receptor subfamilies; compounds flagged for promiscuity are annotated. We mapped this EvE dataset to available Cell Painting profiles, applied similar quality controls as above (removing promiscuous compounds and compounds that exhibit high viability, and likely inactive compounds described in the original work), and excluded viability assays. For Cell Painting data, compounds with low cell count, defined as mean minus two standard deviations, were filtered out, and assays lacking sufficient class diversity (<20 compounds per class) were removed. These filtering steps yielded a high‑quality benchmark of 24 protein‑target assays (including agonist and antagonist modes) with binary activity labels (active/inactive) across 269 compounds.

We evaluated the predictive performance of two feature sets to predict protein targets: the full Cell Painting profiles extracted using CellProfiler and a baseline cell count model. The baseline model used a single morphological feature representing cell count, while the Cell Painting feature set encompassed diverse cellular morphology descriptors. To address class imbalance, class weights were calculated using the balanced strategy (classes with fewer samples receive higher weights), ensuring equal contribution of positive and negative classes during model training. Models were trained using stratified 3-fold nested cross-validation to maintain balanced class distributions in both training and test sets. Logistic regression was employed for the baseline cell count model with the cell count feature, while Random Forest classifiers were used for the multi-dimensional Cell Painting features. For hyperparameter optimization, 2/3rds of the data was used in the inner cross-validation loop, applying RandomizedSearchCV to tune parameters. This included regularization strength for logistic regression and tree depth, leaf size, and the number of estimators for Random Forests. The best-performing models were selected based on AUC-ROC scores from the inner cross-validation loop. Classification thresholds were optimized using Cohen’s Kappa, evaluated across a range of probability thresholds to maximize agreement between predicted and true labels in the training data. Final model performance was assessed on the outer test set using metrics such as Balanced Accuracy, AUC-ROC, AUC-PR, Matthews Correlation Coefficient (MCC), and Cohen’s Kappa. To ensure robustness, random label scrambling was performed for each assay, and models were retrained on the scrambled labels to confirm that their performance was not driven by chance62.

Statistics and reproducibility
All statistical analyses were performed as implemented in scikit-learn63. All code is available at https://github.com/srijitseal/The_Seal_Files. All data and code are released via https://doi.org/10.5281/zenodo.1716818564,65.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "The authors investigate the role of cell count as a simple baseline feature in predicting small molecule bioactivity within the context of phenotypic profiling methods, such as Cell Painting and gene expression profiling. The authors argue that many widely used benchmark datasets in drug discovery are biased toward cell viability or cytotoxicity assays, where a basic metric like cell count can perform comparably to more complex, high-dimensional phenotypic profiles. By analyzing multiple datasets and introducing a new curated benchmark, the study proposes using cell count as a baseline to assess the added value of sophisticated profiling techniques. It provides actionable recommendations for improving machine learning model design and evaluation in this domain.\nThe paper is well-written and very extensive in its assessments. Overall, this work contributes to a more rigorous understanding of how machine learning models should be evaluated in drug discovery in the context of Cell Painting and gene expression.\nHowever, I have some concerns regarding the study:\n• While the abstract mentions gene expression alongside Cell Painting, the analysis heavily emphasizes image-based profiling. A more detailed comparison with gene expression data (e.g., in the Moshkov dataset) could strengthen the generalizability of the findings across phenotypic profiling methods.\n• It seems to me that the study primarily focuses on datasets that are known to contain a high proportion of cytotoxicity-related assays. It would be beneficial to extend the analysis to datasets that include more diverse bioactivity endpoints. Does their conclusion also apply to other datasets with less cytotoxic compounds?\n• Please discuss other potential biases beyond cytotoxicity (e.g., cell line effects, staining artifacts, variations in experimental conditions or protocols, etc.).\n• While the paper highlights the predictive power of cell count, it does not explore other potentially useful simple features, such as nuclear morphology or mitochondrial integrity. Are these also correlated with the endpoint?\n• I would be important if the authors provided a way to filter out datasets to remove associations highly correlated to cell count. Also, please subset existing datasets to remove this potential bias and provide this as a contribution to the community.\n• The paper argues that cell count explains much of the predictive power, but it does not fully dissect whether specific Cell Painting features provide additional mechanistic insights beyond viability.\n• The manuscript’s analysis of concentration-dependent effects is compelling but restricted to only six compounds. This small sample size undermines the generalizability of the findings and the claim that multi-dose experiments are critical for robust phenotypic profiling.\n• There is an unclear breakdown of assay types. The authors note that many benchmark essays are related to cell viability or cytotoxicity, but they do not quantify or categorize the assay types across datasets. Could they provide a table that breaks down the assay type in each dataset? (e.g., viability, protein-target, etc.)\n• The manuscript employs different baseline models (e.g., logistic regression, XGBoost) across analyses, which may be confusing about the primary baseline being advocated and complicate comparisons. XGBoost with a single feature is an overkill. Can the authors also use the raw cell count as a prediction score to understand its prediction power?\n• Why did the authors use only molecular weight and lopP as chemical features? What is the rationale behind choosing only these?", "reviewer_2": "This is an interesting and valuable study that challenges prevailing assumptions in phenotypic screening by showing that simple cell count features can perform comparably to more complex morphological or transcriptomic features (e.g., Cell Painting or gene expression) in predicting compound bioactivity. The manuscript is well-written and addresses a timely and relevant question in the field of machine learning for drug discovery. I appreciate the authors' efforts in systematically reevaluating benchmark datasets and proposing practical recommendations. However, I believe the manuscript would benefit from addressing the following points before publication:\n1. On the specificity of cell count as a proxy for toxicity:\nThe authors argue that many assays are primarily capturing cell health or cytotoxicity signals, which may explain why cell count alone performs well. However, cytotoxicity is not binary—it can vary in intensity and specificity. Could cell count only detect strong cytotoxic effects while missing more nuanced or phenotype-specific signals? I encourage the authors to further clarify the scope and limits of what cell count can capture, perhaps by stratifying compounds by degree or type of toxicity and evaluating the cell count model's sensitivity.\n2. On the complementarity between deep learning models and cell count models:\nWhile the authors compare cell count with several state-of-the-art models, the current manuscript does not analyze what differences exist between the compounds identified as active by different methods. A Venn diagram is not enough. I suggest the authors perform a deeper analysis of the compounds that are uniquely identified by deep learning models but missed by cell count models. What are the chemical or functional characteristics of these compounds? It is possible that these uniquely identified compounds have more subtle or complex modes of action, and understanding this may significantly strengthen the paper's conclusions.\n3. On clarity of the conclusions:\nAlthough the authors have conducted extensive experiments, the conclusions—particularly in the discussion section—are not sufficiently distilled. I suggest the authors revise the discussion to clearly articulate their key findings, implications for future benchmarking practices, and the broader impact of their work on method development in the field.\n4. On model simplicity and interpretability:\nThe authors make an important point regarding the value of simple and interpretable methods in bioactivity prediction. This aligns with recent findings in other domains of computational biology. For example, SpaDo (Genome Biology, 2024) showed that in spatial transcriptomics, simple clustering based on spatial neighbor cell type proportions could achieve comparable or better results than complex graph neural network models. I recommend that the authors cite such work to further reinforce the message that biological interpretability and simplicity should be weighted more heavily in computational model development.", "reviewer_3": "Summary\nThis manuscript presents an analysis of assay predictability from image-based phenotypic assays, specifically the Cell Painting assay, and the role of cell count as a feature that can drive performance. The problem of assay prediction is very complex due to the multiple data sources that interact and the many confounding factors that can lead to incorrect interpretations. The analysis and effort presented by the authors is a valuable step forward to improve clarity about how this problem can be approached and what corrections need to be done going forward to advance the state of the art in this field.\nWhile the questions and presented analysis are promising, the main limitation of this study is that it continues to rely primarily on the same phenotypic dataset, which may no longer be useful to make progress in assay prediction problems. The study makes the assumption that cell count is a clean readout, while in practice, the dataset used may have significant technical artifacts due to plate layouts and batch effects. The pattern reported in this paper may be a more fundamental issue with the imaging dataset instead of being a generalized phenotypic property for assay prediction. In a related note, the assay endpoints across the three benchmark sets analyzed may all be related to each other because they may activate the same (polypharmacological) compounds. These problems are part of the challenges of the assay prediction problem, and need to be addressed in this study to more convincingly support the conclusions.\nMain comments\n1. Figure 1 shows cell count distributions and Figure 2 shows how many assays can be predicted with cell counts relative to other predictors. The main question is: are cell counts batch-effect independent? Or are cell counts biased by batch effects as much as other features? It is well known that cell density is a significant source of technical (not phenotypic) variation, and the cpg0012-wawer-bioactivecompoundprofiling dataset may have significant batch biases, because bioactive compounds were placed in certain plates different from those used for non bioactive compounds. How are cell counts correlated with plates or batches? A potential answer could be obtained by looking at the distribution of cell counts in control well only, but other types of analysis may be needed to clarify this question.\n2. If artifacts that affect cell count exist, how can we make sure that the predictive power of cell counts is not an artifact, but rather a real phenotypic effect? The main concern is that all the predictors (including cell counts) may be leveraging technical variation to predict the same subset of bioactive compounds (Fig 2.), which may come from the same plates (i.e., same technical artifact).\n3. Fig. 1 shows similar results in three datasets, but in reality, it’s the same phenotypic dataset. In other words, the cell counts come from the same imaging dataset. The difference between the three datasets is the assays only. In fact, the Hofmarcher and Ha datasets appear to be almost identical except for a handful of assays, according to the text and references. Is there any relationship between the Moshkov and Hofmarcher assays? Are the compounds labeled as active in Moshkov also active in Hofmarcher? Ultimately, it would be best to unify the set of assays into a single coherent benchmark. The presentation of results in three datasets (Moshkov, Hofmarcher, and Ha) is artificial, and makes it look as if the problem is widespread, but ultimately we are talking about the same phenotypic dataset. A unified view should be presented, with similar technical choices, such as cross validation differences and so on.\n4. To confirm the claim more broadly, another phenotypic dataset should be analyzed as well. The manuscript mentions the JUMP dataset, for instance, which could be used as an independent source of phenotypic variation to investigate the role of cell counts for assay prediction. If the same compounds are present across the dataset, these can help to further investigate if the patterns observed in Fig. 1 and others are prevalent beyond the dataset that the community has used. Plate layouts will be different, and technical variation may have different biases which can lead to different results. The paper presents preliminary results with another set in Figure 8, but its scope is limited to support the broader claims.\n5. Continuing with the batch effects issue in the selected dataset, are the models labeled as “All Features – Logistic Regression” in Figures 5 and 6 batch corrected? While the architecture is simpler, the features may be contaminated by technical variation that reveals where the samples come from (bioactive plates) rather than finding real morphological signals.\n6. In a separate note, one of the recommendations for benchmarking is to keep assays that have sufficient positive and negative examples (more than 40). However, in many assay prediction problems, collecting such a high number of positive examples is unrealistic or simply defeats the purpose of predictive modeling because 40 hits may already be sufficient to move a drug discovery project forward. While having a high number of positive and negative examples is desirable for machine learning researchers, how can the authors include realistic, rare hit assays in their benchmarks? It is highly recommended to consider such examples, which may be more interesting for the community in practice.\nOther comments:\n7. Figure 1(a-c): are the points compounds or assays? If compounds, it seems too few points with respect to what’s reported in the text (10K to 16K). If assays, it seems too many points with respect to what the caption says (~200). How were these points selected for the plot?\n8. Are the phenotypes presented in Figure 4 consistent across replicates?\n9. Lines 203 to 212 assert that AUPRG has benefits over AUPRC and AUC, giving the impression that the metrics were going to be unified and reported with the suggested metric. However, the results in the main manuscript are all based on AUC-ROC only. Why is the AUC-ROC adopted in this manuscript if another metric may have better interpretation properties?"}, "author_response": "Response to review:\nSmall molecule bioactivity benchmarks are often well-\npredicted by counting cells\nWe sincerely thank the reviewers and editor for their thoughtful and detailed feedback. We are\ndelighted that the reviewers found our manuscript well-written, timely, and a valuable contribution\nto the field. We added analyses and have revised the manuscript extensively in response to the\ncomments and summarize our responses and changes below. All changes are marked in the\nrevised manuscript and supplemental files.\nREVIEWER COMMENTS\nReviewer #1 (Remarks to the Author):\nThe authors investigate the role of cell count as a simple baseline feature in predicting\nsmall-molecule bioactivity within the context of phenotypic profiling methods, such as\nCell Painting and gene expression profiling. The authors argue that many widely used\nbenchmark datasets in drug discovery are biased toward cell viability or cytotoxicity\nassays, where a basic metric like cell count can perform comparably to more complex,\nhigh-dimensional phenotypic profiles. By analyzing multiple datasets and introducing a\nnew curated benchmark, the study proposes using cell count as a baseline to assess the\nadded value of sophisticated profiling techniques. It provides actionable\nrecommendations for improving machine learning model design and evaluation in this\ndomain.\nThe paper is well-written and very extensive in its assessments. Overall, this work\ncontributes to a more rigorous understanding of how machine learning models should be\nevaluated in drug discovery in the context of Cell Painting and gene expression.\nHowever, I have some concerns regarding the study:\n• While the abstract mentions gene expression alongside Cell Painting, the analysis\nheavily emphasizes image-based profiling. A more detailed comparison with gene\nexpression data (e.g., in the Moshkov dataset) could strengthen the generalizability of the\nfindings across phenotypic profiling methods.\nWe have now added a new analysis titled “Gene Enrichment for Cytotoxic and Promiscuous\nCompounds”:\nWe hypothesized that (1) a decrease in cell number (Cells_Number_Object_Number)\nobserved in Cell Painting assays may reflect increased apoptotic activity, and (2) compound\npromiscuity may arise more from general cytotoxicity (e.g., apoptosis) than from selective\nactivity. To investigate this, we leveraged gene expression data from the LINCS L1000\nresource to examine the biological processes associated with compound perturbations. For\nthis analysis, we selected only those compounds from LINCS that passed the “is_exemplar”\n(which identifies a single ‘exemplar’ signature for each perturbagen in each cell line) quality\nfilter and were tested in U2OS, MCF7, or PC3 cell lines. Of the 1,520 compounds that\nsignificantly reduced cell count in the Broad Cell Painting assay, 466 had corresponding\nLINCS mRNA signatures, yielding a total of 550 gene expression profiles.\nGene Ontology (GO) enrichment analysis of genes with a z-score threshold (|z| > 2) revealed\nthat transcriptional changes that were induced by compounds known to reduce cell count (by\nCell Painting) were consistently associated with particular biologically relevant processes.\nThe most frequently enriched pathways included the intrinsic apoptotic signaling pathway (n\n= 112 compounds), mitotic cell cycle phase transitions (n = 101), regulation of cyclin-\ndependent kinase activity, spindle organization, and nuclear division. Stress response\npathways (e.g., reactive oxygen species, hypoxia) and metabolic functions (e.g.,\ncholesterol/sterol biosynthesis) were also enriched.\nAnalyzing more carefully the degree of cell count impact, we found that compounds more\nstrongly reducing cell count in Cell Painting assays showed the highest enrichment for\napoptosis- and cell death–related GO terms (Figure 4a). Over 70% of compounds in the\nlowest cell count bins ([-37; -50], normalized feature values against plate-specific DMSO\ncontrols) were linked to apoptosis-related terms, with this proportion decreasing as cell count\nincreased. This underscores the importance of cell health and cytotoxicity as major drivers of\nboth morphological and transcriptomic profiles, and highlights the need to account for cell\ncount when benchmarking both phenotypic profiling methods.\nWe next explored relationships between gene expression and compound promiscuity (activity\nacross many assays). From the set of 24,606 compounds with target activity labels (from\nHofmarcher and Moskhov datasets), we extracted 8,381 LINCS signatures for 4,094\ncompounds that passed the cell line and exemplar filters. Again, the intrinsic apoptotic\nsignaling pathway was the most frequently enriched category (n = 193), supporting our\nhypothesis that compound promiscuity is often associated with general cytotoxic effects\n(Figure 4b).\nFigure 4. (a) Proportion of compounds enriched for apoptosis/cell death (red), other GO terms\n(blue), or no significant GO terms (gray), stratified by normalized cell counts bins,\nstandardised against plate-specific DMSO controls. Compound counts per bin are shown\nabove bars. Apoptosis-related enrichment exceeded 70% in two of the three lowest bins and\ndeclined with increasing cell count. (b) Similar analysis of compound promiscuity (i.e., number\nof active target labels for each compound). Apoptosis enrichment generally increases among\ncompounds with an increase of active labels. (c) Euler diagram showing overlap between\ncytotoxic (n=1,520; orange) and promiscuous (n=24,606; blue) compounds, before (light) and\nafter (dark) GO enrichment filtering. 214 compounds were shared between the two enriched\nsubsets. (d) Heatmap showing apoptosis enrichment (%) across combinations of cell count\nbins and promiscuity bins. Each tile indicates the percentage and count (apoptosis / total).\nColor intensity reflects the level of apoptotic enrichment.\nBeyond apoptosis, other recurrently enriched pathways associated with compound\npromiscuity included metabolic and biosynthetic processes (e.g., cholesterol/sterol\nbiosynthesis [n = 164–156], secondary alcohol metabolism [n = 164–124], steroid\nbiosynthesis [n = 112]), protein phosphorylation and kinase regulation (e.g., serine/threonine\nkinase activity [n = 161], protein kinase complex [n = 131]), and cell adhesion–related\ncategories (e.g., cadherin binding [n = 116], focal adhesion [n = 89]). Stress-related\nresponses to ROS (n = 101), xenobiotics (n = 86), and hypoxia (n = 84) were also commonly\nenriched, highlighting the broad cellular impact of promiscuous compounds.\nFocusing on the 214 compounds that reduced cell count and were also promiscuous (Figure\n4c), we found that apoptotic enrichment increased markedly as cell count decreased. In\ncontrast, the effect of promiscuity on apoptosis enrichment was more modest (Figure 4d).\nNotably, compounds with more than one active hit and a normalized cell count below -40\nshowed over 70% enrichment for apoptosis-related GO terms.\nWe have updated our Abstract to include these findings:\n“We found that compounds that reduce cell count and exhibit promiscuous activity often\ntrigger apoptosis-related gene expression, suggesting general cytotoxicity, rather than\nselective targeting, drives their biological impact.”\nOur fundamental finding is that cell count is a simple baseline that can predict assay outcomes\nas well as Cell Painting can, given the current flawed benchmarks which are discussed in the\npaper. Other studies have already compared Cell Painting and gene expression in assay\nprediction [9, 18, 19, 20] so we now discuss these in the Introduction section (see text below).\nWe also emphasize our finding in this study that 63% (31/49) of the assays in the Moshkov\ndatasets which were well-predicted by gene expression were also well-predicted by cell count\n(Figure 2c), which can be largely attributed to the shared cytotoxicity signal. Finally, we add a\nsection to the introduction describing the more general issue in machine learning for biology, that\ncomplex models are often beat by simple baselines:\n“Previous studies have benchmarked models that make other kinds of predictions using gene\nexpression data.[18] For example, Csendes et al. found that simple baselines outperformed\nstate-of-the-art foundation models in predicting post-perturbation gene expression.[19]\nSimilarly, SpaDo showed that in spatial transcriptomics, simple clustering based on spatial\nneighbor cell type proportions could achieve comparable or better results than complex graph\nneural network models.[20] Motivated by these findings, we focused extensively on\nphenotypic screening using Cell Painting data in this study.”\n• It seems to me that the study primarily focuses on datasets that are known to contain a\nhigh proportion of cytotoxicity-related assays. It would be beneficial to extend the\nanalysis to datasets that include more diverse bioactivity endpoints. Does their\nconclusion also apply to other datasets with less cytotoxic compounds?\nWe have also now added a new curated benchmark (EvE Bio dataset)[29], which was just\nreleased and meets the goal of containing more diverse bioactivity endpoints. The EvE Bio\nbenchmark (now described within Methods) comprises 24 curated protein target assays\n(agonist/antagonist) (Figure 11). On this benchmark, Cell Painting profiles outperform cell count,\ndemonstrating the value of high-dimensional features for more specific endpoints, consistent with\nour overall findings. A detailed discussion of these results is now included within Results and\nDiscussion.\nWe do hope that our paper draws attention to the fact that popular benchmarks do contain a high\nproportion of cytotoxicity-related assays; we are not convinced that this fact is well-known! We\nhave added a discussion on this:\n“A key limitation of all previously existing benchmark datasets from publicly available\nbioactivity data are that they include assays that are off-target or secondary pharmacology\nscreens rather than solely measurements against the compounds’ intended primary targets.\nAs a result, the compounds tested often share overlapping chemical features and target\nsimilar protein families, reducing the independence between assays and potentially inflating\nmodel performance due to correlated activity profiles. Further, the functional relevance of\nassay readouts can vary substantially depending on the compound’s mode of action. For\nexample, antagonists tested in the absence of an agonist may not elicit functional responses\ndetectable by high-content imaging, while agonists can induce receptor desensitization,\ninfluencing time-dependent phenotypes. These complexities limit the interpretability of\nmorphological changes and challenge the ability to distinguish specific pharmacological\nmechanisms from indirect or off-target effects. Some compounds commonly annotated as\nGPCR actives may, in fact, be modulators of related pathways such as phosphodiesterases\n(e.g., IBMX), transporters, or metabolic enzymes.[54] Without well-controlled assay\nconditions, such compounds could be misclassified as receptor agonists or antagonists,\nfurther confounding model training and evaluation. Despite efforts to curate the dataset (such\nas removing assays with excessive cytotoxicity and cell line inhibition) these confounding\nfactors remain inherent to the current public datasets.”\n• Please discuss other potential biases beyond cytotoxicity (e.g., cell line effects, staining\nartifacts, variations in experimental conditions or protocols, etc.).\nThank you for the suggestion, this is a great idea. We have updated the Recommendations and\nadded a discussion regarding using control wells and QC steps to mitigate such artifacts.\n“Additional factors influencing model interpretation include the choice of cell lines, treatment\ndurations, and compound concentrations.”\n“Beyond cytotoxicity, additional sources of bias can influence predictive performance. These\ninclude cell line–specific responses, staining or imaging artifacts, and variability in\nexperimental conditions or protocols (e.g., incubation times, compound dosing, or reagent\npreparation). While our cross-dataset analysis mitigates some of these issues, we\nrecommend that studies explicitly evaluate such factors (e.g., following protocols like those\nof the OASIS[55]). In particular, chemicals should not be grouped onto plates by bioactivity;\ninstead, randomized plate layouts are advisable. A key pitfall to avoid is running Cell Painting\nassays with the exact same plate layout as the target assay, because this allows technical\nartifacts to confound both readouts. Importantly, none of the datasets used in this study suffer\nfrom this issue.”\nWe have added new text discussing potential batch effects and confounders (Reviewer 3 raised\nthis as well) - see the new section entitled “Consistent cell count reduction across plates shows\nbenchmark active compounds often induce cell death”. We have included a new sample analysis\nof cell count at the per-plate level before aggregation, after normalising with respect to within-\nplate controls.\n“In this study we leveraged Cell Painting data normalized against plate-specific DMSO\ncontrols and aggregated across replicates to mitigate technical variability inherent to high-\nthroughput imaging assays. We also evaluated the cell count feature (among others) prior to\naggregation and found that, although artifacts exist that can affect cell count, the predictive\npower of cell counts is not an artifact, but rather a real phenotypic effect. While the assay\noutputs we aim to predict may themselves be subject to technical or batch effects, it is\nimportant to note that these readouts originate from independent experiments distinct from\nthe imaging data, with different plate layouts and batching, and without marked grouping of\nactivities on particular plates.\nSpecifically, we found that of 402 plates from the Broad Cell Painting dataset, 99% contained\nboth bioactive and inactive compounds, defined by activity annotations from the Hofmarcher,\nHa, or Sanchez datasets. Notably, across these plates, bioactive compounds exhibited a\nconsistent reduction in mean cell count relative to inactive compounds in 88% of cases, even\nbefore aggregating features at the compound level (Figure 7a). This reproducibility supports\nthe relevance of cell count as a sensitive phenotypic marker.\nWe found that variability in pre-normalized cell count ranks moderately (94th out of 1783\nfeatures) in terms of mean standard deviation across plates. Features such as nuclei area\nshape exhibited even higher inter-plate variation (Figure 7b). Overall, the consistent decrease\nin cell count observed across nearly all plates for bioactive compounds reinforces confidence\nin cell count as a phenotypic marker; however, this also highlights that the benchmark assays\ncurrently used are heavily biased toward viability-related endpoints, and consequently, the\nidentified bioactive compounds largely reflect viability effects.”\n• While the paper highlights the predictive power of cell count, it does not explore other\npotentially useful simple features, such as nuclear morphology or mitochondrial integrity.\nAre these also correlated with the endpoint?\nIn our experience, many image-based features correlate with cell count, given that cell\nproliferation and cell crowding dramatically impact morphology, and we confirmed that for the\nBroad Cell Painting dataset - we have added a correlation plot of cell count with other features\nfor bioactive vs inactive compounds:\n“Next we aggregated the Cell Painting features as medians across replicates, and we filtered\ncompounds to retain only unique compound skeletons in the bioactive and inactive groups,\nminimizing structural redundancy. Still, we found that bioactive compounds tend to have a\nlarger number of Cell Painting features exhibiting higher absolute correlation with the cell\ncount feature compared to inactive compounds (Figure 7c).\nThese findings suggest that bioactive compounds—likely those that reduce cell count—are\nassociated with stronger phenotypic changes captured by the Cell Painting features.\nConversely, inactive compounds display fewer such correlations, supporting the specificity of\nthese features in reflecting relevant bioactivity impacting cell viability.”\nFigure 8. (c) Distribution of the number of features with high absolute correlation to the cell\ncount feature (after aggregation) for bioactive and inactive compounds (after chemical\nskeleton level deduplication).”\nWe have also explained our choice of cell count features and that the cell count feature contains\na lot of the signal claimed from more complex features. This is why we used “MitoCount” from\nScaleFEx as an example (Figure 8) and found that while it can predict toxicity at high\nconcentrations, full profiles outperform it for subtle phenotypes. We now emphasize that the cell\ncount is representative but not the only informative simple feature.\n• It would be important if the authors provided a way to filter out datasets to remove\nassociations highly correlated to cell count. Also, please subset existing datasets to\nremove this potential bias and provide this as a contribution to the community.\nWe have now released a filtered benchmark (available via Zenodo) and highlighted assays\ndominated by cell count (Figure 1, Supplementary Data). This is one of our main\nrecommendations, and the code associated with the paper (see the Github link in the paper)\ndemonstrates how the filtering is done, so that others can adapt to new datasets.\nAlthough not directly related to this query, we also realized we should more carefully document\nrecommendations for cleaning profiling datasets before using them in benchmark testing. We\nadded to the recommendations a section on cleaning the Cell Painting datasets:\n“For high-dimensional “profiling” datasets such as Cell Painting or gene expression, we\nrecommend applying quality control (QC) post-processing, followed by plate- and batch-level\ncorrections where necessary. Compounds that markedly reduce cell count—confirmed either\nvisually or via cell count metrics—should be removed. To avoid confounding effects, features\nthat are highly correlated with cell count (e.g., absolute correlation > 0.95) should also be\nexcluded. Replicate perturbations can then be aggregated using the mean or median, but it\nis advisable to first address systematic artifacts, such as plate rows with consistently lower\ncell density.”\n• The paper argues that cell count explains much of the predictive power, but it does not\nfully dissect whether specific Cell Painting features provide additional mechanistic\ninsights beyond viability.\nWe address this in two ways: anecdotally to show particular mechanistic insights, and more\ncomprehensively. First, we evaluated the influence of individual features on the benchmark\ndataset from EveBio, which followed the recommendations of removing viability assays,\npromiscuous, and cytotoxic compounds.\n“The assays most improved by including Cell Painting information were for GR agonism\n(mean AUC=0.79 with the complete Cell Painting profile vs AUC=0.60 with cell count baseline\nmodel) and RARb antagonism (AUC=0.83 vs AUC=0.64). The feature\nNuclei_RadialDistribution_RadialCV_ER_4of4 was found to be the most predictive of this\nendpoint (Figure 11b); this feature is the coefficient of variation of intensity in the ER channel\nwithin the outermost region of the nucleus object. For RARb antagonism, the most predictive\nfeature was Nuclei_Texture_Entropy_Mito_10_0 (Figure 11c), which measures the disorder\nor heterogeneity of the mitochondrial signal. Although mitochondria are cytoplasmic, limited\nimage resolution and perinuclear overlap can cause mitochondrial signals to be captured\nwithin the nuclear mask. In both the GR agonist and RARβ antagonist cases, the phenotypes\nare subtle and not readily apparent by visual inspection (as confirmed in Supplementary\nFigures S4 and S5, which show no substantial changes in cell count). However, these effects\nare best captured through image-derived features. The superior performance of models using\nthe full Cell Painting profiles compared to cell count baselines further supports the idea that\nspatial and texture-based features carry the relevant biological signal.”\nWe have also added a new section that is more comprehensive, comparing true predictions from\nthe model using all Cell Painting features with true predictions from the model using the cell count\nfeature:\n“To assess whether the cell count baseline and the full Cell Painting feature model were\nidentifying the same compounds as active, we examined the true positive prediction\nagreement between the two models across all assay endpoints in the Moshkov dataset.\nThere was a 49.54% overlap between the true positive predictions, suggesting the\napproaches are partially predicting different groups of compounds as active. To investigate\nthe difference in these compound cohorts, we isolated the compounds which were uniquely\npredicted by each model. We then calculated the following physicochemical properties for\neach compound molecule: molecular weight (MW), lipophilicity (LogP), topological polar\nsurface area (TPSA), hydrogen bond donors (HBD) and acceptors (HBA) and rotatable\nbonds. We also computed the statistical significance of the difference of the means of each\nmetric between the uniquely predicted compounds for each model using scipy’s “ttest_ind” (p\n< 0.05).\nThe true positive compounds predicted uniquely by the model trained only using cell count\nwere found to have statistically significant lower average MW and LogP, while having a higher\naverage TPSA and more rotatable bonds when compared with the cohort of compounds\nuniquely predicted by the model trained using all CellProfiler features (Supplementary Figure\nS2). Generally, lower LogP combined with higher TPSA indicates that the compound should\nexhibit better aqueous solubility.[44,45] Lower MW compounds also tend to diffuse more\neasily across cell membranes, and more rotatable bonds indicate higher molecular flexibility\n- more promiscuous binding and greater likelihood of fitting into various binding sites.[46]\nTaken in combination this demonstrates that the cell count model is more adept at identifying\ncompounds with more broad, simplistic effects and higher general/nonspecific cytotoxicity as\nthey are more likely to impact broad cellular processes. Conversely, the model trained on all\nfeatures is able to categorise compounds which have a lower likelihood of being reliably\ndelivered to cells, or exhibit more subtle or specific phenotypes.\nTo further explore these findings, we examined the structural diversity of compounds that\nwere correctly predicted as active by only one of the models. When comparing compound\nscaffolds, we observed substantially lower scaffold diversity among the compounds uniquely\nidentified by the cell count model (528 unique scaffolds) compared with those uniquely\nidentified by the full Cell Painting feature model (1,175 unique scaffolds). This further supports\nthe finding that the cell count model is more narrow in its predictive ability, but when applied\nin a setting where a high number of compounds are cytotoxic, or assays are primarily aimed\nat measuring cell viability, this approach can appear to perform on par with a model utilizing\nall CellProfiler features, demonstrating the necessity for the benchmark curation suggested\nwithin this paper.”\nFigure S2. Distribution of physicochemical properties for uniquely predicted true positive\ncompounds from each model. Histograms compare the distributions of (A) molecular weight\n(MW), (B) lipophilicity (LogP), (C) topological polar surface area (TPSA), (D) number of hydrogen\nbond donors (HBD), (E) number of hydrogen bond acceptors (HBA), and (F) number of rotatable\nbonds between compounds uniquely predicted as true positives by the Cell Count Feature Model\n(cc_feat) and the All Feature Model (all_feat). Statistical comparisons were conducted using\nindependent t-test; features with p < 0.05 are considered significantly different.\nFurther our analysis on the benchmark datasets and Figure 11 shows that Cell Painting features\ndo indeed capture assays beyond viability. We have now expanded the discussion on important\nCell Painting features on a case-by-case basis.\n“Overall, the value of imaging-derived features largely hinges on the screening context.\nPrevious studies have explored the importance of particular Cell Painting features with\nrespect to mitochondrial toxicity (edge of the mitochondria object)[22] and cytotoxicity\nprediction in muscle cells (spots edges and ridges profiles in the ER and mitochondria\nchannels)[53]. Our results underscore the utility of the complete Cell Painting profile as high-\ncontent imaging data for capturing biologically meaningful phenotypes associated with\nsignaling pathways, not just limited to cell count. These phenotypic responses become more\ndetectable when cytotoxicity is not pervasive in the dataset, causing models to be focused\non subtle cellular changes rather than cell count.”\n• The manuscript’s analysis of concentration-dependent effects is compelling but\nrestricted to only six compounds. This small sample size undermines the generalizability\nof the findings and the claim that multi-dose experiments are critical for robust phenotypic\nprofiling.\nAfter the paper was sent for review, a subset of our authors pre-printed a study on a larger set of\nappropriate data for this analysis, over 1,000 compounds - at our request, they included cell count\nanalysis in their study (Ewald et al. https://pmc.ncbi.nlm.nih.gov/articles/PMC11785178/). We\nnow refer to this work in our discussion, which supports our findings that morphology changes\nprecede cell death.\n“Although these findings are preliminary and specific to the six compounds, this is consistent\nwith our larger analysis in Ewald et al., which found that across >1000 compounds, early\nmorphology changes were observed at doses about an order of magnitude lower than\ncytotoxicity readouts. Further studies affirm that for most compounds that exhibit\nmorphological changes, the information content in Cell Painting is dependent on dose. [Ewald\net al 2025]”\n• There is an unclear breakdown of assay types. The authors note that many benchmark\nassays are related to cell viability or cytotoxicity, but they do not quantify or categorize\nthe assay types across datasets. Could they provide a table that breaks down the assay\ntype in each dataset? (e.g., viability, protein-target, etc.)\nGood idea. We now include a table (Supplementary Table S1) and figure (Supplementary Figure\nS1) categorizing assays based on ChEMBL categorization: e.g., viability (cell-line) vs. protein\ntarget (single protein). This was only possible for public datasets where such information was\navailable. Not all assays from all sources (particularly the Broad dataset) were annotated to make\nthis classification possible.\n• The manuscript employs different baseline models (e.g., logistic regression, XGBoost)\nacross analyses, which may be confusing about the primary baseline being advocated\nand complicate comparisons. XGBoost with a single feature is an overkill. Can the authors\nalso use the raw cell count as a prediction score to understand its predictive power?\nWe now clarify that the raw cell count feature was used directly in threshold-based predictions\n(Figure 1). We used LR for consistency and interpretability, and XGBoost only when required for\ndirect comparison to prior benchmarks. We added this clarification to the Results and Methods.\n“We first evaluated baseline cell count models trained on a single image-based feature,\nnamely cell count….This allows us to systematically assess the predictive utility of cell count\nas a prediction score to understand its predictive power.”\n“We systematically assessed the predictive utility of the cell count feature across multiple\nassays by using the raw cell count as a prediction score to understand its predictive power\nby evaluating classification performance at different thresholds.”\n• Why did the authors use only molecular weight and lopP as chemical features? What is\nthe rationale behind choosing only these?\nThese were chosen to match prior CLOOME benchmarks which had access to the full chemical\nstructure. Molecular weight and logP were chosen as proxies, as these are known confounding\nfactors in the chemical assays. We now clarify this and also show that adding them to the cell\ncount improves performance slightly (Table 2, row d).\n“Since CLOOME embeddings have access to chemical structures, two baseline cell count\nmodels were considered: (1) using cell count as the sole feature and (2) using cell count\nalong with molecular weight and logP (calculated using RDKit) as very simple chemical\nfeatures. The first baseline excludes any chemical information, while the second baseline,\nalthough not explicitly trained on chemical features, incorporates some chemical-related\ninformation indirectly through molecular weight and logP. Molecular weight and logP were\nchosen as proxies for chemical information, as these are known confounding factors in the\nchemical assays.[25,26]”\nReviewer #2 (Remarks to the Author):\nThis is an interesting and valuable study that challenges prevailing assumptions in\nphenotypic screening by showing that simple cell count features can perform comparably\nto more complex morphological or transcriptomic features (e.g., Cell Painting or gene\nexpression) in predicting compound bioactivity. The manuscript is well-written and\naddresses a timely and relevant question in the field of machine learning for drug\ndiscovery. I appreciate the authors' efforts in systematically reevaluating benchmark\ndatasets and proposing practical recommendations. However, I believe the manuscript\nwould benefit from addressing the following points before publication:\n1. On the specificity of cell count as a proxy for toxicity:\nThe authors argue that many assays are primarily capturing cell health or cytotoxicity\nsignals, which may explain why cell count alone performs well. However, cytotoxicity is\nnot binary—it can vary in intensity and specificity. Could cell count only detect strong\ncytotoxic effects while missing more nuanced or phenotype-specific signals? I encourage\nthe authors to further clarify the scope and limits of what cell count can capture, perhaps\nby stratifying compounds by degree or type of toxicity and evaluating the cell count\nmodel's sensitivity.\nGreat point - we now add a new analysis on this. Figure 1g now shows the cell count feature\nvalue for compounds that are active in n assays (combining Moshkov and Hofmarcher datasets).\nThis shows that cell count contains information that is increasingly more predictive of strong and\npromiscuous cytotoxic responses across different cell types, and less for subtle phenotypes.\nWe have added a new discussion to this effect:\n“To better contextualize the scope of what cell count captures, we analyzed whether the\npredictive signal from cell count correlates with cytotoxicity. Specifically, we combined the\nMoshkov and Hofmarcher datasets and stratified compounds by the number of assays in\nwhich they were labeled cytotoxic. We compared the cell count values for compounds active\nin multiple assays and found that compounds active in a greater number of these benchmark\nassays tend to show a stronger and more consistent decrease in cell count (Figure 1g).\nOverall, this indicates that the cell count feature is more predictive for compounds that are\npromiscuous.”\n2. On the complementarity between deep learning models and cell count models:\nWhile the authors compare cell count with several state-of-the-art models, the current\nmanuscript does not analyze what differences exist between the compounds identified as\nactive by different methods. A Venn diagram is not enough. I suggest the authors perform\na deeper analysis of the compounds that are uniquely identified by deep learning models\nbut missed by cell count models. What are the chemical or functional characteristics of\nthese compounds? It is possible that these uniquely identified compounds have more\nsubtle or complex modes of action, and understanding this may significantly strengthen\nthe paper's conclusions.\nThis is an interesting exploratory direction! We performed two new analyses related to this: one\nlooking for mechanistic insights at the assay level (which assays are better-predicted with full\nCell Painting profiles vs cell count) - see above (Reviewer 1 comment “The paper argues that\ncell count explains much of the predictive power, but it does not fully dissect whether specific Cell\nPainting features provide additional mechanistic insights beyond viability”)\nAt the compound level, for the Moshkov dataset only, the authors released predictions from each\ncompound in the original study, and we used these predictions to conduct a new analysis (now\nincluded within the paper in section “Appropriate evaluation metrics and assay endpoints are\nessential when modelling large-scale benchmark datasets”):\n“To assess whether the cell count baseline and the full Cell Painting feature model were\nidentifying the same compounds as active, we examined the true positive prediction\nagreement between the two models across all assay endpoints in the Moshkov dataset.\nThere was a 49.54% overlap between the true positive predictions, suggesting the\napproaches are partially predicting different groups of compounds as active. To investigate\nthe difference in these compound cohorts, we isolated the compounds which were uniquely\npredicted by each model. We then calculated the following physicochemical properties for\neach compound molecule: molecular weight (MW), lipophilicity (LogP), topological polar\nsurface area (TPSA), hydrogen bond donors (HBD) and acceptors (HBA) and rotatable\nbonds. We also computed the statistical significance of the difference of the means of each\nmetric between the uniquely predicted compounds for each model using scipy’s “ttest_ind” (p\n< 0.05).\nThe true positive compounds predicted uniquely by the model trained only using cell count\nwere found to have statistically significant lower average MW and LogP, while having a higher\naverage TPSA and more rotatable bonds when compared with the cohort of compounds\nuniquely predicted by the model trained using all CellProfiler features (Supplementary Figure\nS2). Generally, lower LogP combined with higher TPSA indicates that the compound should\nexhibit better aqueous solubility.[44,45] Lower MW compounds also tend to diffuse more\neasily across cell membranes, and more rotatable bonds indicate higher molecular flexibility\n- more promiscuous binding and greater likelihood of fitting into various binding sites.[46]\nTaken in combination this demonstrates that the cell count model is more adept at identifying\ncompounds with more broad, simplistic effects and higher general/nonspecific cytotoxicity as\nthey are more likely to impact broad cellular processes. Conversely, the model trained on all\nfeatures is able to categorise compounds which have a lower likelihood of being reliably\ndelivered to cells, or exhibit more subtle or specific phenotypes.\nTo further explore these findings, we examined the structural diversity of compounds that\nwere correctly predicted as active by only one of the models. When comparing compound\nscaffolds, we observed substantially lower scaffold diversity among the compounds uniquely\nidentified by the cell count model (528 unique scaffolds) compared with those uniquely\nidentified by the full Cell Painting feature model (1,175 unique scaffolds). This further supports\nthe finding that the cell count model is more narrow in its predictive ability, but when applied\nin a setting where a high number of compounds are cytotoxic, or assays are primarily aimed\nat measuring cell viability, this approach can appear to perform on par with a model utilizing\nall CellProfiler features, demonstrating the necessity for the benchmark curation suggested\nwithin this paper.”\nFigure S2. Distribution of physicochemical properties for uniquely predicted true positive\ncompounds from each model. Histograms compare the distributions of (A) molecular weight\n(MW), (B) lipophilicity (LogP), (C) topological polar surface area (TPSA), (D) number of\nhydrogen bond donors (HBD), (E) number of hydrogen bond acceptors (HBA), and (F)\nnumber of rotatable bonds between compounds uniquely predicted as true positives by the\nCell Count Feature Model (cc_feat) and the All Feature Model (all_feat). Statistical\ncomparisons were conducted using independent t-test; features with p < 0.05 are considered\nsignificantly different.\n3. On clarity of the conclusions:\nAlthough the authors have conducted extensive experiments, the conclusions,\nparticularly in the discussion section, are not sufficiently distilled. I suggest the authors\nrevise the discussion to clearly articulate their key findings, implications for future\nbenchmarking practices, and the broader impact of their work on method development in\nthe field.\nWe appreciated this feedback and rewrote the conclusions to clearly outline key takeaways and\nour five recommendations for future benchmarking.\n4. On model simplicity and interpretability:\nThe authors make an important point regarding the value of simple and interpretable\nmethods in bioactivity prediction. This aligns with recent findings in other domains of\ncomputational biology. For example, SpaDo (Genome Biology, 2024) showed that in\nspatial transcriptomics, simple clustering based on spatial neighbor cell type proportions\ncould achieve comparable or better results than complex graph neural network models.\nI recommend that the authors cite such work to further reinforce the message that\nbiological interpretability and simplicity should be weighted more heavily in\ncomputational model development.\nWe now cite SpaDo (Genome Biology 2024) and other examples emphasizing model simplicity\nin biology. Thank you for this valuable suggestion. We have added a discussion on the same.\n“Previous studies have benchmarked models using gene expression data[18]. For example,\nCsendes et al. found that simple baselines outperformed state-of-the-art foundation models\nin predicting post-perturbation gene expression.[19] Similarly, SpaDo showed that in spatial\ntranscriptomics, simple clustering based on spatial neighbor cell type proportions could\nachieve comparable or better results than complex graph neural network models.[20]\nMotivated by these findings, we focused extensively on phenotypic screening using Cell\nPainting data in this study.”\nReviewer #3 (Remarks to the Author):\nSummary\nThis manuscript presents an analysis of assay predictability from image-based\nphenotypic assays, specifically the Cell Painting assay, and the role of cell count as a\nfeature that can drive performance. The problem of assay prediction is very complex due\nto the multiple data sources that interact and the many confounding factors that can lead\nto incorrect interpretations. The analysis and effort presented by the authors are a\nvaluable step forward to improve clarity about how this problem can be approached and\nwhat corrections need to be made going forward to advance the state of the art in this\nfield.\nWhile the questions and presented analysis are promising, the main limitation of this\nstudy is that it continues to rely primarily on the same phenotypic dataset, which may no\nlonger be useful to make progress in assay prediction problems. The study makes the\nassumption that cell count is a clean readout, while in practice, the dataset used may have\nsignificant technical artifacts due to plate layouts and batch effects. The pattern reported\nin this paper may be a more fundamental issue with the imaging dataset instead of being\na generalized phenotypic property for assay prediction. In a related note, the assay\nendpoints across the three benchmark sets analyzed may all be related to each other\nbecause they may activate the same (polypharmacological) compounds. These problems\nare part of the challenges of the assay prediction problem and need to be addressed in\nthis study to more convincingly support the conclusions.\nMain comments\n1. and 2. Figure 1 shows cell count distributions, and Figure 2 shows how many assays\ncan be predicted with cell counts relative to other predictors. The main question is:\nare cell counts batch-effect independent? Or are cell counts biased by batch effects as much\nas other features? It is well known that cell density is a significant source of technical (not\nphenotypic) variation, and the cpg0012-wawer-bioactivecompoundprofiling dataset may\nhave significant batch biases, because bioactive compounds were placed in certain plates\ndifferent from those used for non-bioactive compounds. How are cell counts correlated\nwith plates or batches? A potential answer could be obtained by looking at the distribution\nof cell counts in the control well only, but other types of analysis may be needed to clarify\nthis question.\nIf artifacts that affect cell count exist, how can we make sure that the predictive power of\ncell counts is not an artifact, but rather a real phenotypic effect? The main concern is that\nall the predictors (including cell counts) may be leveraging technical variation to predict\nthe same subset of bioactive compounds (Fig. 2), which may come from the same plates\n(i.e., same technical artifact).\nThe reviewer raises a very important point here, which we have addressed in several ways in the\nrevised manuscript. First, we should have mentioned in the paper that the data used in modelling\nused controls to normalize features, including cell counts, on a per-plate basis; this should\nmitigate (though may not eliminate) plate-level and batch-level effects. We have now added a\nstatement to this effect in the methods:\n“Thus, the features from the Cell Painting dataset used in this study are the deviation from\nDMSO controls on the same plate after normalization, and after aggregating median across\nreplicates.”\nThat said, we began to address the concern by analyzing the placement of bioactive vs inactive\ncompounds in the Cell Painting data. Among the 402 plates analyzed, 398 contained both\nbioactive and inactive compounds (defined as being active in at least one assay in Hofmarcher,\nHa or Sanchez datasets). Of these 398 plates, bioactive compounds exhibited a lower mean cell\ncount than inactive compounds in 351 plates. The remaining 4 plates, which contained only\nbioactive compounds, included just 8 compounds in total. We added a discussion of this and\nemphasize why our data uses plate negative controls (DMSO) for normalization:\nFigure 7. (a) Distribution of the difference between mean cell count values for bioactive versus\ninactive compounds across 402 plates from the Broad Cell Painting dataset. (b) Ranking of top\n94th out of 1783 features in inter-plate variability. (c) Distribution of the number of features with\nhigh absolute correlation to the cell count feature (after aggregation) for bioactive and inactive\ncompounds (after chemical skeleton level deduplication\n“Consistent cell count reduction across plates shows benchmark active compounds often\ninduce cell death\nCell count can be impacted by many technical variables, raising the potential for plate or\nbatch effects to confound analysis. “In this study we leveraged Cell Painting data normalized\nagainst plate-specific DMSO controls and aggregated across replicates to mitigate technical\nvariability inherent to high-throughput imaging assays. We also evaluated the cell count\nfeature (among others) prior to aggregation and found that, although artifacts exist that can\naffect cell count, the predictive power of cell counts is not an artifact, but rather a real\nphenotypic effect. While the assay outputs we aim to predict may themselves be subject to\ntechnical or batch effects, it is important to note that these readouts originate from\nindependent experiments distinct from the imaging data, with different plate layouts and\nbatching, and without marked grouping of activities on particular plates.\nSpecifically, we found that of 402 plates from the Broad Cell Painting dataset, 99% contained\nboth bioactive and inactive compounds, defined by activity annotations from the Hofmarcher,\nHa, or Sanchez datasets. Notably, across these plates, bioactive compounds exhibited a\nconsistent reduction in mean cell count relative to inactive compounds in 88% of cases, even\nbefore aggregating features at the compound level (Figure 7a). This reproducibility supports\nthe relevance of cell count as a sensitive phenotypic marker.\nWe found that variability in pre-normalized cell count ranks moderately (94th out of 1783\nfeatures) in terms of mean standard deviation across plates. Features such as nuclei area\nshape exhibited even higher inter-plate variation (Figure 7b). Overall, the consistent decrease\nin cell count observed across nearly all plates for bioactive compounds reinforces confidence\nin cell count as a phenotypic marker; however, this also highlights that the benchmark assays\ncurrently used are heavily biased toward viability-related endpoints, and consequently, the\nidentified bioactive compounds largely reflect viability effects.”\nNext we aggregated the Cell Painting features as medians across replicates, and we filtered\ncompounds to retain only unique compound skeletons in the bioactive and inactive groups,\nminimizing structural redundancy. Still, we found that bioactive compounds tend to have a\nlarger number of Cell Painting features exhibiting higher absolute correlation with the cell\ncount feature compared to inactive compounds (Figure 7c).\nThese findings suggest that bioactive compounds—likely those that reduce cell count—are\nassociated with diverse phenotypic changes captured by the Cell Painting features. Overall,\nwe found that the predictive power of cell counts is not an artifact of actives being\nconcentrated on particular plates in the experiment, but rather a real phenotypic effect. ”\nWe agree that prediction of assay activity may in theory be confounded by plate layout effects;\nhowever, in this study we used three different sources of Cell Painting datasets – AstraZeneca\ndataset, Broad dataset, and the NY Stem Cell Foundation dataset – which all had differing plate\nand batch layouts for compounds making it impossible for those to be predictive for assay\nendpoints which came from still other plate and batch layouts. The assay outputs that we predict\nmay also have technical/batch effects, but they are completely separate experiments, which\nmakes it more unlikely that batches or plates or well positions ‘match’ the imaging data.\nThe assay outputs we aim to predict may also contain technical or batch effects; however,\nbecause each assay output experiment is an entirely separate experiment from the Cell Painting\nexperiment (different laboratory, typically testing different chemical libraries in different\nplate/batch layouts), it is unlikely that batch, plate, or well-position effects align with the imaging\ndata in any consistent way.\nOur results are overall the same across the datasets, that is, cell count is a good predictor of\n“activity”. Because cell death is essentially represented in cell counts, this seemed a reasonable\nhypothesis, as “active” compounds mostly have lower cell count compared to inactive\ncompounds (across all three Cell Painting datasets). We show that drop in cell count for “active”\ncompounds is consistent across plates where both inactive and bioactive compounds are\npresent.\nWe have now added a new section on the “Limitations of this Study”\n“We also acknowledge that technical artifacts, such as plate layout effects, can confound\nassay activity predictions, particularly when bioactive compounds cluster on specific plates.\nIn the case of the Broad dataset, by analyzing plate-level data, we demonstrate consistent\nreductions in cell count for bioactive compounds (as defined by current benchmarks) across\nplates, indicating that the predictive power of cell count is unlikely driven solely by technical\nbias. Furthermore, given that the Broad, AstraZeneca, and New York Stem Cell Foundation\nCell Painting datasets image sets were all independently created and not likely to follow the\nsame plate and batch layouts, this rules out artifacts as yielding falsely positive results; for\nnew datasets, the potential impact of plate and batch layout needs to be evaluated for each\ncase separately.”\n3. Fig. 1 shows similar results in three datasets, but in reality, it’s the same phenotypic\ndataset. In other words, the cell counts come from the same imaging dataset. The\ndifference between the three datasets is the assays only. In fact, the Hofmarcher and Ha\ndatasets appear to be almost identical except for a handful of assays, according to the\ntext and references. Is there any relationship between the Moshkov and Hofmarcher\nassays? Are the compounds labeled as active in Moshkov also active in Hofmarcher?\nUltimately, it would be best to unify the set of assays into a single coherent benchmark.\nThe presentation of results in three datasets (Moshkov, Hofmarcher, and Ha) is artificial,\nand makes it look as if the problem is widespread, but ultimately, we are talking about the\nsame phenotypic dataset. A unified view should be presented, with similar technical\nchoices, such as cross-validation differences, and so on.\nWe now clarify in the text that the Moshkov dataset (assays performed internally at the Broad\nInstitute) is distinct from the Hofmarcher/Ha datasets, which are indeed derivatives of each other,\nboth being collected from ChEMBL, but collected at different times, resulting in a few different\ncompounds and assays. We now emphasize that while the assays differ to some extent, the\nimaging data used to predict them is shared across the Hofmarcher/Moshkov/Ha benchmarks.\nWe also explain the decision not to group all assays into a single dataset so that we could directly\ncompare our results against prior published results (e.g., Figures 1–2). We have now added text\nto Results to this effect:\n“The Moshkov dataset, containing assay readouts generated internally at the Broad Institute,\nis distinct from the Hofmarcher and Ha datasets, both derived from ChEMBL. Sanchez-\nFernandez et al. based their benchmarking on the Hofmarcher dataset. Although Hofmarcher\nand Ha originate from the same source, they were compiled at different times, leading to\nminor differences in compound coverage. All three datasets were evaluated against the same\nCell Painting imaging dataset. To allow direct comparison with prior studies, we evaluated\neach dataset separately rather than combining them into a single benchmark.”\nWe now also provide a new benchmark dataset from EveBio that is a different source for assays\n- it measures the agonism and antagonism of compounds across various targets, with\ncompounds that induce viability removed from both the target assay space and the Cell Painting\nspace; hence models here learn activity free from large deviations in cell count. We quote part of\nthe new section below:\n“In order to facilitate the above recommendations, we are together with this work releasing a\nbenchmark dataset for compound-activity prediction curated along the recommendations\nabove from EvEBio.[52] This benchmark dataset comprises ligand binding and functional\nassays in biochemical systems for nuclear receptors (NRs), alongside cell-based activity\nmeasurements for seven-transmembrane receptors (7TMs), supplemented by cell viability\nassays to monitor cytotoxicity, collectively covering a broad panel of these receptor classes.\nAfter applying the same filtering steps described above (see Benchmarking Cell Painting for\ncurated Protein-Target data) we isolated a high-quality benchmark of 24 protein-target\nassays (including agonist and antagonist modes) with binary activity labels (active/inactive)\nacross 269 compounds which also have Cell Painting data publicly available in the Broad\ndataset. Using this benchmark, models leveraging the full Cell Painting profiles outperformed\nthe baseline cell count model, achieving a higher mean balanced accuracy (BA = 0.67 vs.\n0.52; see Figure 11a). Compared to both a randomly shuffled baseline and the cell count\nbaseline, the full-profile models delivered over 20% improvement in AUC-PR on 8 of the 24\ntasks (Supplementary Figure S3). Notably, gains in AUC-ROC were consistent regardless of\nthe test set size or the proportion of active compounds, suggesting that these improvements\nare robust and not driven by dataset composition (Supplementary Figure S3).”\n4. To confirm the claim more broadly, another phenotypic dataset should be analyzed as\nwell. The manuscript mentions the JUMP dataset, for instance, which could be used as an\nindependent source of phenotypic variation to investigate the role of cell counts for assay\nprediction. If the same compounds are present across the dataset, these can help to\nfurther investigate if the patterns observed in Fig. 1 and others are prevalent beyond the\ndataset that the community has used. Plate layouts will be different, and technical\nvariation may have different biases, which can lead to different results. The paper\npresents preliminary results with another set in Figure 8, but its scope is limited to\nsupporting the broader claims.\nThe JUMP-CP dataset is not yet fully prepared for batch-corrected analysis and was therefore\nexcluded from this study. Instead, in the revised manuscript we present results from two\nadditional Cell Painting chemical-perturbation datasets beyond the Broad dataset: the New York\nStem Cell Foundation dataset (Comolet et al.), and the AstraZeneca dataset (Cross-Zamirski et\nal.). We show that models trained to predict targets, compound dose, or positive controls (the\ntask differs depending on the dataset) perform comparably to cell count–based models.\nAcross all datasets, the conclusions remain consistent: bioactive compounds tend to show a\nreduction in cell count relative to inactive compounds in most assays in current popular\nbenchmarks. When benchmark tasks are biased toward viability-related endpoints, cell count\nalone is often sufficient to predict assay outcomes from imaging data.\nWe have now added a statement to this effect in the Limitations Section:\n“Across all datasets analyzed—the Broad, AstraZeneca, and New York Stem Cell Foundation\nCell Painting datasets—we evaluated whether cell count alone could predict various assay\noutcomes or intended tasks set in the original studies (such as separating drugs from positive\ncontrols). In each dataset, we found that cell count performed comparably to models using\nfull Cell Painting profiles. Our results consistently show that bioactive compounds, as defined\nin these benchmarks, tend to reduce cell count, especially when the assays in these\nbenchmarks are biased toward viability-related endpoints.”\n5. Continuing with the batch effects issue in the selected dataset, are the models labeled\nas “All Features – Logistic Regression” in Figures 5 and 6 batch corrected? While the\narchitecture is simpler, the features may be contaminated by technical variation that\nreveals where the samples come from (bioactive plates) rather than finding real\nmorphological signals.\nYes. We have now clarified that CellProfiler features were plate-normalized and DMSO-corrected\nas in prior work. The reviewer is correct that in the original Bray 30k compound set, the \"bioactive\"\nset of compounds was indeed a separate batch of plates, so they were quite concentrated. Here,\nhowever, we are using bioactives to mean the ones annotated as active in the assay dataset,\nwhich does not relate to what Bray et al refer to as bioactive. We also showed that, as per assay\nactivity, both bioactive and inactive compounds (by our definition) are present in each plate of\nthe Cell Painting dataset from the Broad Institute rather than concentrated on a few plates;\nfurther, to model, we used median aggregated profiles, which further reduces confounding\ntechnical factors as replicates are spread over separate plates (though typically in the same well\nposition within each plate). We have added this to the Methods and Results sections as indicated\nabove.\n6. In a separate note, one of the recommendations for benchmarking is to keep assays\nthat have sufficient positive and negative examples (more than 40). However, in many\nassay prediction problems, collecting such a high number of positive examples is\nunrealistic or simply defeats the purpose of predictive modeling because 40 hits may\nalready be sufficient to move a drug discovery project forward. While having a high\nnumber of positive and negative examples is desirable for machine learning researchers,\nhow can the authors include realistic, rare hit assays in their benchmarks? It is highly\nrecommended to consider such examples, which may be more interesting for the\ncommunity in practice.\nThis point is well-taken; we have added this point to our recommendations. We suggest using\nbalanced accuracy or qualitative analysis for assays with few positive example compounds and\nemphasize that prediction may still be useful with few actives. Apart from the ‘project’ view, we\nalso emphasize there needs to be more focus on understanding readouts and technologies\nbetter. We have now reworded the recommendations to:\n“For example, with models that achieve an AUC of 0.85, we recommend using AUC as a\nmetric only when the held-out test sets contain at least 20 active and inactive compounds.42\nWhen this is not feasible (for assays with few positive example compounds), AUC may not\nbe an appropriate metric. Instead, we suggest evaluating models using absolute predictions\ninstead of predicted probabilities, using metrics such as balanced accuracy, while carefully\naccounting for potential confounding effects from data imbalance using random prediction\nbaselines and y-scrambling.”\nOther comments:\n7. Figure 1(a-c): Are the points compounds or assays? If compounds, it seems too few\npoints with respect to what’s reported in the text (10K to 16K). If assays, it seems too many\npoints with respect to what the caption says (~200). How were these points selected for\nthe plot?\nThe points represent compounds; given there are too many to represent, the Seaborn swarmplot\noverlaps and clusters them together, especially in areas of high density. We have now addressed\nthis in the Figure.\n8. Are the phenotypes presented in Figure 4 consistent across replicates?\nYes, these were consistent across all sites and replicates; for the reader we used a single image.\nWe have now linked the visualizations to the data viewer portal that shows images from different\nsites and replicates. “Images of cells from other sites and replicates of these compounds show\nthe same trends and are viewable at https://idr.openmicroscopy.org”\n9. Lines 203 to 212 assert that AUPRG has benefits over AUPRC and AUC, giving the\nimpression that the metrics were going to be unified and reported with the suggested\nmetric. However, the results in the main manuscript are all based on AUC-ROC only. Why\nis the AUC-ROC adopted in this manuscript if another metric may have better\ninterpretation properties?\nWe were limited to AUC-ROC to enable direct comparison to prior published results, but used\nAUPRG for a comparative answer. We do compare results with AUCPR in certain cases where\nnecessary to understand the predictivity of bioactive compounds. We have clarified the language\nin the text to reflect the same.\n“AUPRG provides a universal baseline, which is comparable across assays with different\nclass distributions (unlike Area Under the Precision-Recall Curve), and is less impacted than\nAUC by the abundance of negative class examples that are present in these classification\ntasks. For other tasks, we limited the analysis AUC-ROC to enable direct comparison to prior\npublished results, but used AUPRG and AUCPR for a comparative answer.”"}, {"round": 2, "reviewer_comments": {"reviewer_1": "Thank you for carefully addressing all the comments and suggestions from my previous review. I have gone through the revised version of your manuscript, and I am pleased to see that my concerns have been resolved. The revisions have strengthened the paper and improved its overall clarity and impact.\nI have no further comments and recommend the manuscript for publication.", "reviewer_2": "The manuscript has been substantially improved, and most previous concerns have been fully addressed. The additional analyses greatly enhance the clarity and rigor of the work.\nOne minor issue remains: while Figure 1g clarifies that cell count captures strong and promiscuous cytotoxic responses, the manuscript still does not explicitly discuss its limited sensitivity to subtle or mechanism-specific effects. I suggest adding a brief statement in the Discussion to acknowledge this limitation.\nWith this minor revision, the paper will be suitable for publication.", "reviewer_4": "Authors have performed a comprehensive and timely benchmark study about cell counting and Cell Painting verified from multiple aspects. The manuscript is well-organized and the results are comprehensive and convinced. The editor contacted me and asked me to review the author's responses to Reviewer #3's concerns. Authors have completely addressed the concerns proposed by Reviewer #3. The revised manuscript is well-written and the response letter is detailed. I have no additional suggestions."}, "author_response": "Reviewer #1 (Remarks to the Author):\nThank you for carefully addressing all the comments and suggestions from my previous review. I have\ngone through the revised version of your manuscript, and I am pleased to see that my concerns have\nbeen resolved. The revisions have strengthened the paper and improved its overall clarity and impact.\nI have no further comments and recommend the manuscript for publication.\nReviewer #1 (Remarks on code availability):\nI checked the repository but did not run the code.\nReviewer #2 (Remarks to the Author):\nThe manuscript has been substantially improved, and most previous concerns have been fully addressed.\nThe additional analyses greatly enhance the clarity and rigor of the work.\nOne minor issue remains: while Figure 1g clarifies that cell count captures strong and promiscuous\ncytotoxic responses, the manuscript still does not explicitly discuss its limited sensitivity to subtle or\nmechanism-specific effects. I suggest adding a brief statement in the Discussion to acknowledge this\nlimitation.\nResponse:\nWe have modified the sentence in the “Multiple bioactivity tasks are highly correlated with the\nsimple cell count feature” section in response:\n“While cell count is informative for strong and promiscuous cytotoxic responses, it is\nconsiderably less responsive to mechanism-specific or low-grade phenotypic changes, which\nmay constrain its utility for detecting nuanced biological effects.”\nWith this minor revision, the paper will be suitable for publication.\nReviewer #4 (Remarks to the Author):\nAuthors have performed a comprehensive and timely benchmark study about cell counting and Cell\nPainting verified from multiple aspects. The manuscript is well-organized and the results are\ncomprehensive and convinced. The editor contacted me and asked me to review the author's responses\nto Reviewer #3's concerns. Authors have completely addressed the concerns proposed by Reviewer #3.\nThe revised manuscript is well-written and the response letter is detailed. I have no additional suggestions."}]

Decision letter