dataesr/scientific-paragraphs-categorization huggingface dataset:
2018; Warembourg et al. 2019). In total, earlylife exposome data was composed of 133 variables. A detailed description of all included exposures can be found in Supplementary Table  1. Regarding outcome data, 9 continuous health outcomes were investigated (Supplementary Table  2 ). Outcomes were assessed at the same time as the exposome and included parameters related to (1) obesity and cardiometabolic health, (2) respiratory health, and (3) cognition and mental health. In total, our dataset included 133 exposures (Supplementary Table  1 ) and 9 outcomes (Supplementary Table  2 ) in 881 individuals. The main research question was to identify exposures simultaneously affecting multiple health outcomes. The way exposures affect each outcome does not need to be necessarily the same, so we identified both overall risk exposures negatively affecting most of the outcomes and also ambiguous factors (being protective for some outcomes but risky for others).
Thus, we opted for the alternative of regressing out the effect of confounders on both outcomes and exposures before the data analysis (partialling-out approach). The whole matrix of estimated coefficients for all exposures and outcomes by each of the methods can be found in Supplementary Table  5. Output models were quite different in terms of sparsity degree (Fig. 1  and Supplementary Fig. 1 2 3 4 5 6 ). The sparsest model was generated by the Bayesian approach MBSP (with only 4 selected exposures affecting studied outcomes: copper, lead, DDE and HCB). On the contrary, the least sparse model was obtained by the GFLasso method (45 selected exposures). In general, all methods were quite consistent in the identification of exposures with effects on the multiple outcomes. Especially, the most overlapping findings were evidenced for the pairs (sRRR-MTL_L21, and mRRR-sRRR).
) can accommodate high dimensional datasets without suffering a loss of estimation efficiency. Likewise, some of them also allow correcting for the effect of confounders, which is a crucial step in environmental epidemiology. According to the specificities of each problem, and depending on the goal of each research, one could navigate through Table  1  to choose the most suitable approach. For example, we may not always expect that evaluated exposures affect all outcomes under study. In this situation, MTL_L21, mRRR and sRRR (which do not allow intrarow sparsity) might not be preferred options. Despite all the benefits highlighted in the manuscript for these approaches, some drawbacks have also been identified. For example, the need for priori knowledge about the grouping structure of predictors and the long computing times for the identification of hyperparameters in GroupRemMap pose a burden for applying the method.
2021 ). On the contrary, successful applications of some of the proposed methods can be found in the literature in the field of molecular and biomedical data  (Curtis et al. 2013; de Abreu e Lima et al. 2018; Li et al. 2017). For example, MTL has been successfully applied in the past for the identification of shared imaging features that simultaneously predict two subtypes of bipolar disorders  (Wang et al. 2015b) , or for the study of the shared behavioral rhythms that simultaneously predict ten symptoms of Schizophrenia  (Tseng et al. 2020). In order to discuss the applicability of each method to exposome data and give more specific details on their implementation, we decided to apply them all to a real dataset from the HELIX project. From the results, it seems that the GFLasso tends to estimate more homogeneous effects (the same exposure tends to affect most of the outcomes similarly).
Among the included techniques, only MBSP addresses the lack of uncertainty quantification in high-dimensional inferences. Therefore, this should be another key issue to be considered in future developments, as others have discussed  (Dunson 2018). Fortunately, there is growing literature seeking to address the lack of uncertainty quantification in high-dimensional inferences; for example, focused on penalized optimization methods, such as Lasso  (Basu et al. 2021; Miller and Breheny 2023). On this matter, and considering the growing popularity of machine learning approaches, exposome researchers should clearly state that highly predictive black box algorithms, along with an estimate of the important variables, are not enough. Instead, we crucially need tools to tell us how reliable our variable selection decisions are given the sample size, dimensionality, and correlation structure of the data at hand.
Acknowledgment ATHLETE  project has received funding from the  European Union 's  Horizon 2020 research and innovation programme  under grant agreement No  874583. This publication reflects only the authors' view and the European Commission is not responsible for any use that may be made of the information it contains. We acknowledge support from the grant  CEX2018-000806-S  funded by  MCIN/AEI/ 10. 13039/ 501100011033 , and support from the  Generalitat de Catalunya  through the  CERCA Program. We also thank the support from the grant  FJC2021-046952-I  funded by  MCIN/AEI/ 10. 13039/501100011033  and, by  "European Union NextGenerationEU/PRTR " and acknowledge funding from the  Ministry of Research and Universities of the Government of Catalonia  ( 2021-SGR-01563 ).
Accurately quantifying cellular morphology at scale could substantially empower existing single-cell approaches. However, measuring cell morphology remains an active field of research, which has inspired multiple computer vision algorithms over the years. Here, we show that DINO, a vision-transformer based, self-supervised algorithm, has a remarkable ability for learning rich representations of cellular morphology without manual annotations or any other type of supervision. We evaluate DINO on a wide variety of tasks across three publicly available imaging datasets of diverse specifications and biological focus. We find that DINO encodes meaningful features of cellular morphology at multiple scales, from subcellular and single-cell resolution, to multi-cellular and aggregated experimental groups.
These properties make DINO especially useful for cellular biology problems where the goal is to test hypotheses and uncover novel phenomena, and for powering various bioimage analysis applications  31, 38, 39. Our study demonstrates that DINO is a general, self-supervised approach to uncover meaningful biology from fluorescent microscopy images by analyzing three publicly available datasets with diverse technical specifications and biological goals (Figure  1 ). The results show that DINO features support a wide range of downstream tasks, from subcellular to population-level analyses, showing great potential for biological discovery.
Self-supervised vision transformers for single-cell images The DINO  30  algorithm is a self-supervised learning strategy to transform images into data representations for quantitative analysis. This self-supervised strategy only requires images to create a meaningful feature space without using annotations or prior knowledge. DINO trains a feature extraction model using different views of the same image, which are shown independently to a student and a teacher network: during training their outputs should match (Figure  1A  and Methods). By optimizing this simple rule, DINO learns features that capture semantic properties of images. In the case of images of single cells, these features are expected to encode biologically meaningful properties of cellular morphology. The feature extraction model trained with DINO is a vision transformer (ViT), based on a neural network architecture with self-attention layers (Methods) originally developed for natural language processing  35  (Figure  1B ).
ViTs process an image by first decomposing it into a sequence of local patches and then treating them as if they were tokens in a sentence. In this way, ViTs capture long range dependencies between image regions by paying attention to all tokens simultaneously. This is in contrast to convolutional networks (CNNs), which are based on local feature analysis progressively expanded to cover larger spatial areas with pooling operations. While CNNs have been successful in many supervised image analysis problems, ViTs are more effective in self-supervised settings and scale efficiently to learn from large unannotated datasets. Self-supervised transformers also exhibit a unique ability to generate emerging properties and behaviors, which has been observed in images  30, 40  and natural language  41.
DINO uncovers the biological structure of imaging experiments We used DINO to analyze three publicly available imaging datasets (Figure  1C ), and we observe that it discovers meaningful factors of variation directly from the images in all cases. The three datasets included in this study are the Allen Institute WTC-11 hiPSC (WTC11) dataset 45 , the Human Protein Atlas (HPA) dataset  1  , and a collection of Cell Painting datasets (CPC) 29,46,47. These datasets represent a diverse set of imaging techniques, cell lines, and experimental conditions, which have been used to study intracellular organization (WTC11), subcellular protein localization (HPA), and the effects of compound perturbations (CPC). All three datasets were pre-processed to obtain single-cell images using segmentation algorithms optimized for each case (Methods), and the DINO model was trained on single cells for each dataset separately to produce a morphology feature space.
This morphology feature space captures the phenotypic diversity of cells in the corresponding imaging study. Specifically, DINO identifies clusters of cellular structures in the WTC11 dataset (Figure  2A ), clusters of cell lines in the HPA dataset (Figure  2D ), and clusters of mechanisms of action (MOA) in the CPC dataset (Figure  S1 ). These results illustrate how DINO encodes meaningful information without explicitly being trained to identify or detect such phenotypes. Importantly, we find that these morphology feature spaces also encode additional aspects of cellular variation in a hierarchically organized way: e. g. , DINO locally encodes cell-cycle stages for each cellular structure cluster in WTC11 (Figure  2A ), and protein localization variations for each cell line in HPA (Figure  2D ).
The ability of DINO to encode diverse factors of variation in an unbiased way could enable various types of phenotypic analyses. Consistent with this, we transformed the DINO features using the single-cell data integration algorithm Harmony  48  to remove variation across the cellular structure clusters in WTC11 and the cell line clusters in HPA. Harmony integrates the corresponding clusters and preserves biological variation in both cases, resulting in feature spaces that respectively align cell-cycle stages across cell structures in the WTC11 dataset (Figure  2B ), and that encode the continuum of protein localization variations in the HPA dataset (Figure  2E ). Notably, the resulting integrated feature spaces qualitatively resemble the structure identified by specialized methods, using hand-engineered features (Figure  2B ), or supervised methods trained specifically for protein localization classification (Fig.
2E ). To explore the hierarchy of the factors of variation encoded in DINO features quantitatively, we used a mutual-information based metric  49  to associate individual features with known data annotations (Methods). We find that DINO features capture a greater diversity of factors of variation compared to engineered features or supervised models, which specialize on specific variation according to the labels used for training (Figures  2C  and 2F ). Importantly, DINO captures not only biologically meaningful features, but also technical variation. Supervised learning regimes, in which feature extractors are often trained end-to-end, are prone to exploiting spurious correlations. 50  However, similar to gene-expression data  51  , DINO features can be transformed to correct for unwanted variation prior to downstream analyses (Figure  2B ,E), potentially avoiding confounders.
Nevertheless, effectively mitigating confounders remains an open challenge, which we do not investigate here. 45. C) Stacked-bar plot displaying the fraction of features that are strongly associated with three factors of variation annotated in the WTC-11 hiPCS dataset (colors). Three feature representation strategies are compared (bars). D) UMAP visualization of single-cell morphology features obtained with DINO for images in the Human Protein Atlas dataset. The left-hand side plot shows points colored by cell line, and the right-hand side plot shows points colored by protein localization labels. The title and frame colors match the legend in F for factors of variation. E) UMAP visualizations for the same single-cell images in D. The left-hand side plot shows DINO features integrated with Harmony over cell lines.
DINO features can predict expert-defined cellular phenotypes A common task in bioimage analysis is the classification of phenotypes of interest according to expert-defined annotations. DINO features show highly competitive performance when used to train classifiers with manually annotated examples, even though the feature extractors are optimized independently with self-supervision and not trained to solve these tasks explicitly (Methods). We evaluated the performance of classifiers trained with DINO features on two classification tasks concerning manual annotation of single-cell phenotypes: cell-cycle stage classification in the WTC11 dataset, and protein localization in the HPA dataset. No fine-tuning of the DINO models was involved for training any of the classifiers. For the cell-cycle stage classification task, we sampled a balanced subset of cells from WTC11 that represents six stages according to available annotations (Methods).
We trained three Multi-Layer Perceptron (MLP) classifiers with the same architecture using the following features for comparison: engineered features available in the WTC11 dataset  45  , DINO features from a model trained with natural images (ImageNet DINO  52  ), and DINO features from a model trained with WTC11 cells. The two sets of DINO features outperformed engineered features (Figure  3A  and 3B), highlighting how self-supervision automatically captures information relevant for biological analyses. The DINO model trained on WTC11 achieves competitive results despite being trained with fewer images than ImageNet DINO (1. 2 vs 0. 2 million, 0. 9% improvement over engineered features). ImageNet DINO seems to encode cell cycle variation more prominently (Figure  S2 ) because the nucleus channel is processed separately, partially avoiding confounders and obtaining superior performance (18% relative improvement over engineered features).
DINO features were also highly competitive in Kaggle-hosted, public benchmarks on both HPA whole-image (containing multiple cells) and single-cell datasets: an MLP classifier trained on frozen DINO features ranks in the top 13th and 12th percentile respectively, among hundreds of highly specialized submissions (Figure  3D, E ). Using the same MLP classifier architecture, we further compared ImageNet DINO features, HPA DINO features trained on whole-images or single-cells, as well as features from the 1st and 2nd ranked solutions in the single-cell and whole-image classification challenges, respectively (Bestfitting  21, 22  and DualHead  22  ). Note that the top competitors designed ensemble models with many CNNs to improve their results  21, 22. To compare the quality of features rather than classification strategies, we evaluated a single CNN from their ensemble to compare with single DINO networks.
Three classifiers trained with the input features in C are highlighted in the plot with lines and points of the corresponding colors. The dashed line is the cumulative distribution of the number of competitors (right y-axis). E) Single-cell protein localization classification challenge 22. The histogram shows the distribution of the number of Kaggle competitors (right y-axis) according to their performance (x-axis, higher is better). Three classifiers are highlighted in the plot as in D. The dashed line is the cumulative distribution of the number of competitors (left y-axis). F) Mechanism of action prediction for compounds in the LINCS Cell Painting dataset  47. Three identical MLP classifiers are trained with input features: CellProfiler features (green), Cell Painting CNN features  29.
DINO improves the prediction of compound bioactivity Imaging is extensively used to quantify the effects of pharmacological compounds at high throughput where the phenotypes of interest cannot be annotated by hand. The CPC dataset is an example of such an application. Here, existing knowledge from chemical biology is used to create predictive models of compound bioactivity by looking at the response of cells to perturbations in images  54, 55. We find that DINO improves the ability of such models to predict the MoAs of compounds and the results of diverse bioactivity assays using Cell Painting images 56. For this evaluation, we first trained a DINO model with the same dataset as in the Cell Painting CNN-1 model 29 (Methods). Next, we computed features and created well-level representations for the LINCS dataset to predict MoAs  47  , and for the BBBC036 dataset  46  to predict the readouts of 270 assays (see below).
For MoA prediction, we used annotations from the Connectivity Map project, which were featured in a Kaggle challenge to classify molecules based on transcriptional profiling  57. The evaluation protocol was adapted for image-based profiling using the LINCS Cell Painting dataset  47  , a library of more than 1,500 compounds screened on A549 cells. We used the same profiling pipeline and classifier models (Methods) to compare three feature representations: engineered features obtained with CellProfiler  58, 59  , features obtained with weakly supervised Cell Painting CNN-1 model  29, 60  , and DINO features trained on Cell Painting images. The results show that learned features improve MoA classification compared to engineered features, and DINO features perform better than the CNN model (Figure  3F ).
Note that MoA classification is approached at the well-level instead of classifying images of single cells. The latter could potentially leverage single-cell heterogeneity to improve prediction performance even further. We also evaluated the problem of predicting compound activity in the BBBC036 Cell Painting dataset with 30,000 small molecules screened on U2OS cells  46. The goal here is to use image-based features to determine if compounds are hits in one of 270 historical assays collected at the Broad Institute, given a sparse set of assay readouts  53. We followed the same profiling pipeline and classifier models (Methods) to compare CellProfiler features, Cell Painting CNN-1 features, and DINO features, following a well-level prediction approach. DINO features improved performance with respect to both CellProfiler and the CNN model, achieving higher overall precision-recall scores (Figure  3G , 10% and 2% improvement over baselines, respectively) and predicting bioactivity for more assays than the baseline models at all thresholds of accuracy (Figure  3H , 23% and 16% improvement over baselines, respectively).
DINO enables unbiased profiling of cellular morphology Unbiased profiling of biological conditions aims to uncover the relationships among groups of samples, to discover similarities or differences across them. DINO features are effective for unbiased image-based profiling, as indicated by its ability to cluster samples in a biologically meaningful manner (Figure  2 ). Here, we aggregated single-cell DINO features according to the conditions of interest in each dataset to build morphological profiles, then computed the cosine similarity across all conditions, and finally, clustered the similarity matrices to reveal groups related by their morphological structure (Figure  4 ). The HPA dataset includes images of 35 human cell lines, which may be related to each other by e. g. anatomical origin or disease state.
Aggregation of DINO features according to the 28 HPA protein localization groups in the HPA dataset and ordering them by the ground truth protein localization hierarchy annotated by experts  27  also results in a meaningful similarity matrix (Figure  4C ), revealing protein localizations in the nucleus and in the cytoplasm as two major clusters, as well as several minor clusters within them (Methods). Importantly, these broadly recapitulate ground truth annotations, achieving a Mantel statistic between the DINO similarity matrix and the ground truth annotations of 0. 52, outperforming the supervised CNN features (0. 48). We conducted a similar profiling analysis for cell cycle stages in the WTC11 dataset and observed clustering in two major groups: interphase/prophase cells and mitotic cells (Figure  4E ).
Analysis of single-cell DINO features using the diffusion pseudotime (DST) algorithm  63  further revealed a cycle-like ordering of cells (Figure  4D ) in a sample of images balanced to represent the six stages equally (Methods). Finally, we created perturbation-level profiles for 1,249 compounds in the LINCS Cell Painting dataset by aggregating single-cell DINO features. Visual inspection of the cosine similarity matrix suggests these profiles naturally uncover similarities between perturbations that share similar MoA annotations (Figure  4F ). To quantify the ability of DINO features to group compounds with similar MoAs, we followed the benchmark evaluation presented in the Cell Painting CNN-1 study  29  , which is based on perturbation matching with nearest-neighbors search on three Cell Painting datasets.
DINO features performed better than CellProfiler features in all three datasets (3%, 5% and 22% improvement in BBBC037, BBBC022 and BBBC036, respectively), and exhibited competitive performance with respect to the weakly supervised Cell Painting CNN features (Figure  S6 , ~3% below in all datasets). Together, these results underpin the inherent biological fidelity of DINO features, and their potential to enable new and complement existing routes for unbiased biological discoveries.
A representative subset of the cell line points are annotated. C) Matrix of cosine similarities among DINO features aggregated by protein localization groups, with rows and columns ordered by the ground truth annotations. The clusters highlighted in green are protein localizations in the nucleus or the cytoplasm. The red clusters correspond to secondary groupings of the protein localizations, annotated by experts 27 (Methods). D) Pseudotime analysis of cell cycle stages in the WTC11 dataset using the Diffusion Pseudotime (DPT) algorithm 63 on the features extracted by DINO trained on ImageNet. Points in the plot are single cells colored by cell-cycle stages. E) Matrix of cosine similarities between DINO features aggregated by cell-cycle stage groups in the WTC11 dataset. F) Matrix of cosine similarities between DINO features aggregated at the treatment-level in the LINCS Cell Painting dataset.
We found that DINO features are able to rank genes according to their heterogeneity score (Figure  5 ). We ranked genes according to the standard deviation of single-cell feature vectors, and then calculated the percentage of genes annotated as spatially or intensity heterogeneous within the top-k genes (Methods). The result shows that DINO features from a model trained on the HPA single-cell dataset encodes single-cell variation useful to correctly predict the ground truth heterogeneity of genes with peak accuracy of 64% at the top k=24 results, compared to 48% at k=34, and 32% at k=79 for a supervised protein localization CNN and an ImageNet pretrained DINO model, respectively (Figure  5B ). The peak prediction improves to 70% when genes are ranked based on the variance of all single-cells that are stained for each gene (Methods).