dataesr/scientific-paragraphs-categorization huggingface dataset:
Acknowledgements NJC and JCD were supported by the  UK Medical Research Council and Department for International Development  (grant nos. MR/R015600/1  and  MR/T016434/1 ). NJC was supported by a  Sir Henry Dale Fellowship , jointly funded by  Wellcome  and the  Royal Society  (grant no. 104169/Z/14/A ). JCD also acknowledges PhD funding from the  Wellcome Trust  (grant no. 102169/Z/13/Z ). The GPS study was cofunded by the  Bill and Melinda Gates Foundation  (grant code  OPP1034556 ), the  Wellcome Sanger Institute  (core  Wellcome  grants  098051  and  206194 ), and the  US Centers for Disease Control and Prevention. WPH was supported by the  National Institutes of Health  (grant number  R01 AI106786 ). PT was supported by the  Wellcome trust  (grant number  106698 ). Alejandra Corso:  Administracion Nacional de Laboratorios e Institutos de Salud, Buenos Aires, Argentina ;  Diego Faccone :  Instituto Nacional de Enfermedades Infecciosas (INEI)-Administracion Nacional de Laboratorios e Institutos de Salud (ANLIS) "Dr.
D ue to the advent of high-throughput technologies, highdimensional "omics" data are produced at an increasing pace. In cancer biology, in particular, national and international consortia, such as The Cancer Genome Atlas (TCGA), have profiled thousands of tumor samples for multiple molecular assays, including mRNA, microRNAs, DNA methylation, and proteomics  1. Moreover, multiomics profiling approaches are currently being transposed at single-cell level, which further stresses the need for methods and tools enabling the joint analysis of such large and diverse datasets  2. While multiomics data are becoming more accessible, studies combining different omics are more common. This multiomics integration is frequently performed by sequentially combining results obtained on single omics (a. k.
Integrating these large-scale and heterogeneous sources of data allows researchers to address crucial objectives, including (i) classifying cancer samples into subtypes, (ii) predicting the survival and therapeutic outcome of these subtypes, and (iii) understanding the underlying molecular mechanisms that span through different molecular layers  4. Designing theoretical and computational approaches for the joint analysis of multiomics datasets is currently one of the most relevant and challenging questions in computational biology  4, 5. Indeed, the different types of omics have a large number of heterogeneous biological variables and a relatively low number of biological samples, thereby inducing statistical and computational challenges, in addition to the typical challenges of "Big Data".
1a ). Thus, DR combined with dedicated downstream analyses provides information on all the key objectives mentioned above, namely the classification of samples into subtypes, their association with outcome/survival, as well as the reconstruction of their underlying molecular mechanisms. As a consequence, the design of DR approaches for the joint analysis and integration of multiple omics (jDR) is currently a highly active area of research  8, 9, 11, 12, 15. Here, we report an in-depth comparison of nine representative state-of-the-art multiomics joint Dimension Reduction (jDR) approaches, in the context of cancer data analysis. We extensively benchmark these approaches, spanning the main mathematical formulations of multiomics jDR, in three different contexts (Fig. 1b ). First, we simulate multiomics datasets and evaluate the performance of the nine jDR approaches in retrieving groundtruth sample clustering.
Second, we use TCGA multiomics cancer data to assess the strengths of jDR methods in predicting survival, clinical annotations, and known pathways/biological processes. Finally, we evaluate the performance of the methods in classifying multiomics single-cell data from cancer cell lines. All these analyses allow formulating recommendations and guidelines for users, as well as indications for methodological improvements for developers. We also provide the Jupyter notebook multiomics mix (momix) and its associated Conda environment containing all the required libraries installed (ComputationalSystemsBiology/momix-notebook). Overall, momix can be used to reproduce the benchmark, but also to test jDR algorithms on other datasets, and to evaluate novel jDR methods and compare them to reference ones.
These matrices can then be used to cluster samples and identify molecular processes. b Workflow of our benchmark, subdivided in three subparts: First, we simulated multiomics datasets and evaluated the performance of the nine jDR approaches in retrieving ground-truth sample clustering. Second, we used TCGA multiomics cancer data to assess the strengths of jDR methods in predicting survival, clinical annotations, and known pathways/biological processes. Finally, we evaluated the performances of the methods in classifying multiomics single-cell data from cancer cell lines. iCluster  16  , Integrative NMF (intNMF)  17  , Joint and Individual Variation Explained (JIVE)  18  , Multiple co-inertia analysis (MCIA)  19  , Multi-Omics Factor Analysis (MOFA)  15  , Multi-Study Factor Analysis (MSFA)  20  , Regularized Generalized Canonical Correlation Analysis (RGCCA)  21  , matrix-trifactorization (scikit-fusion)  22  , and tensorial Independent Component Analysis (tICA)  23.
Seven of the nine jDR approaches are extensions of DR methods previously used for single-omics datasets: intNMF is an extension of non-Negative Matrix Factorization (NMF); tICA is an extension of Independent Component Analysis (ICA); MCIA and JIVE are different extensions of Principal Component Analysis (PCA); and MOFA, MSFA, and iCluster are extensions of Factor Analysis. As a consequence, the different jDR algorithms make different assumptions on the distribution of the factors (Methods). The different jDR approaches also make different assumptions on the across-omics constraints on the factors (Fig. 2 ). Some algorithms, such as intNMF, consider the factors to be shared across all omics datasets. In contrast, the factors of RGCCA and MCIA are different for each omics layer, i. e. , they are omics-specific factors.
These omics-specific approaches still maximize some measures of interrelation between the omicsspecific factors, such as their correlation (RGCCA), or their coinertia (MCIA). Finally, JIVE and MSFA consider mixed factors, decomposing the omics data as the sum of two factorizations, one containing a unique factor matrix shared across all omics, and the second having omics-specific factor matrices. Of note, jDR methods considering omics-specific and mixed factors take into account the complementarity of the multiomics data. Most of the jDR approaches can manage different features (e. g. , genes, miRNAs, CpGs…), but require a match between the samples of the different omics datasets (columns of the X i matrices, see Fig. 2 ). Some algorithms, such as MOFA, scikitfusion and JIVE, can also cope with omics matrices having not all samples in common.
Benchmarking joint Dimensionality Reduction approaches on simulated omics datasets. We first evaluated the jDR approaches on artificial multiomics datasets (Fig. 3a ). We simulated these datasets using the InterSIM CRAN package  24. This package generates three omics datasets with imposed reference clustering. Starting from real omics (DNA methylation, transcriptome and protein expression) extracted from TCGA ovarian cancer datasets, InterSIM generates clusters and associates features to these clusters by shifting their mean values by a fixed amount. InterSIM preserves the covariance matrix between all pairs of omics and thereby maintains realistic inter-and intraomics relationships. Importantly, we selected this approach to avoid making assumptions on the distribution of the data. Indeed, alternative simulation approaches assume specific sample distributions (e.
g. , Gaussian in ref. 14  ). Assuming a Gaussian distribution, for instance, would favor jDR methods that also make the assumption of Gaussian sample distribution. We simulated multiomics data with five, ten, and fifteen clusters. In addition, each set of clusters is simulated in two versions, either with all clusters of the same size, or with clusters of variable random sizes (Methods). We applied the nine jDR methods, requiring the decomposition of multiomics data into five, ten, and fifteen factors, depending on the simulated datasets. The performances of the nine jDR approaches are then compared based on their clustering of samples. As mentioned before, intNMF and iCluster are intrinsically designed for sample clustering, while the remaining seven algorithms detect factors without providing a direct clustering.
Benchmarking joint Dimensionality Reduction approaches on cancer datasets. In the second step, we downloaded TCGA multiomics data for ten different cancer types  11  (http://acgt. cs. tau. ac. il/ multi_omic_benchmark/download. html). These data are composed of three omics layers: gene expression, DNA methylation, and miRNA expression. The number of samples ranges from 170 for Acute Myeloid Leukemia (AML) to 621 for Breast cancer. Importantly, we do not have ground-truth cancer subtypes to evaluate the performances of the jDR methods. However, in Breast cancer, we compared the jDR clustering results with two subtypings: the ER/ PR/HER-2 subtyping based on Estrogen Receptor (ER), Progesterone Receptor (PR) and HER-2 immunohistochemistry markers  25  , and the Cluster of Cluster Assignment (COCA) integrative classification performed by the TCGA consortium  26.
Both subtypings cannot be considered as ground-truth for evaluating jDR clustering performances. The ER/PR/HER-2 overlaps with the PAM50 subtyping, which is obtained using only transcriptomics data, and composed of four subtypes: Basal, Her2, Luminal A and Luminal B  27. The COCA subtyping is integrative but has been obtained by separately clustering different omics and then performing a consensus of the obtained results. Thereby, it does not take into account the complementarity of the various omics. We decomposed the multiomics Breast cancer datasets in four factors, and used the Jaccard Index (JI) and Adjusted Rand Index (ARI) to evaluate the overlap between the clustering obtained from these four factors and the Breast cancer subtypings ER/PR/ HER-2 and COCA (Supplementary Fig. 1 ). Most of the methods display low JI ([0.
2;0. 6]) and ARI ([0. 2;0. 5]) values. JIVE shows the best performances according to both JI and ARI (JI = 0. 4, ARI = 0. 4). MCIA has the best performances according to ARI (ARI = 0. 5), and intNMF has good performances according to JI, but with high variability ([0. 2;0. 8]), which results in a low ARI value (ARI = 0. 28). In order to evaluate the methods on the full set of cancer multiomics, we then tested the associations of the factors with survival, clinical annotations and biological annotations (Fig. 4 ). It is to note that the Factor Analysis approach MSFA did not converge to any solution and was thereby not considered. We applied the eight remaining jDR approaches to each of the ten cancer multiomics datasets, jointly decomposing them in ten factors, as in the work of Bismeijer and colleagues  28.
For the three annotation sources (MsigDB Hallmarks, REACTOME and Gene Ontology), the number of metagenes identified by the different jDR methods enriched in at least a biological annotation are plotted against the selectivity of the associations between the metagene and the annotation. See Supplementary Fig. 3  for the results corresponding to the other seven cancer types. Finally, we assessed the jDR methods performances in associating factors with biological processes and pathways (see Fig. 4d  for the methodology, Fig. 6  and Supplementary Fig. 4  for the results). To achieve this goal, we needed to take into account genes (i. e. , weight matrices) and not samples (i. e. , factor matrices). We computed the number of metagenes (corresponding to the columns of the transcriptomics weight matrix) enriched in at least one biological annotation from Reactome, Gene Ontology (GO), and cancer Hallmarks annotation databases (Methods).
Overall, among all jDR methods, tICA and MCIA resulted in the most promising results for two out of three annotation databases considered in this study, and displayed the best average performances across the three annotations databases (Fig. 6 ). Benchmarking joint Dimensionality Reduction approaches on single-cell datasets. Similarly to bulk multiomics analyses, the joint analysis of single-cell multiomics is expected to provide tremendous power to untangle the cellular complexity. jDR approaches are expected to compensate for the strong intrinsic limitations of single-cell multiomics, such as small number of sequencing reads, systematic noise due to the stochasticity of gene expression at single-cell level, or dropouts  [29] [30] [31]. However, the nine jDR algorithms that we are considering (except MOFA) have been designed and applied to bulk multiomics data.
7b ). Overall, with MCIA, tICA and MSFA were the best-performing algorithms. Multiomics mix (momix) Jupyter notebook. To foster the reproducibility of all the results and figures presented in this benchmark study, we implemented the corresponding code in a Jupyter notebook available at https://github. com/ ComputationalSystemsBiology/momix-notebook, together with a Conda environment containing all the required libraries installed. Written in R, this notebook is structured in three main parts corresponding to the three test cases here considered (simulated data, bulk TCGA cancer data and single-cell data). Importantly, this notebook can be easily modified to test the nine jDR algorithms on user-provided datasets. The notebook can also be easily extended to benchmark novel jDR algorithms on our three test cases.
Discussion We benchmarked in-depth nine jDR algorithms, representative of multiomics integration approaches, in the context of cancer data analysis. In contrast to previous comparisons  [11] [12] [13] [14]  , our benchmark not only focuses on the evaluation of the clustering outputs, but also evaluates the biological, clinical, and survival annotations of the factors and metagenes. Existing comparisons also mainly use simulated data, while we here further consider large datasets of bulk cancer multiomics, as well as single-cell data. When performing clustering on simulated multiomics datasets, intNMF, intrinsically designed as a clustering algorithm, displayed the most promising results. MCIA, MOFA, and RGCCA showed the best performance among the set of methods not intrinsically designed for clustering.
In the cancer data benchmark, when we evaluated the associations of the factors with survival or clinical annotations, MCIA, JIVE, MOFA, and RGCCA were the most efficient methods. When assessing the associations of the metagenes with biological annotations, MCIA and tICA were the most efficient. Finally, in the last benchmark, when clustering single-cell multiomics data, MSFA and tICA, as well as MCIA and intNMF, outperformed other approaches. As mentioned earlier, intNMF, representative of the Nonnegative Matrix Factorization (NMF) approaches, performs well for the clustering tasks, i. e. , for detecting substantial patterns of variation across the omics datasets. This is observed for both simulated bulk data clustering and single-cell data clustering. Hence, intNMF should be prioritized by researchers focusing on clustering samples.
Please note that we followed the mathematical formulations and notations provided in each corresponding publication. Integrative non-negative matrix factorization (intNMF). intNMF 17 is one of the numerous generalizations of NMF to multiomics data. The method decomposes each omics matrix X i into a product of non-negative matrices: the factor matrix W, and an omics-specific matrix H i : X i ¼ WH i ; for i ¼ 1; :::; P with W and H i positive matrices for i ¼ 1; :::; P: The algorithm minimizes the objective function Q ¼ min WH P P i¼1 θ i X i À WH i : Once the matrices W and H i i ¼ 1; :::; P have been computed, samples are assigned to clusters based on the W matrix; Each sample is associated with the cluster in which it has the highest weight. The algorithm is implemented into the CRAN R package intNMF (https://cran.
r-project. org/web/packages/IntNMF/index. html). Joint and individual variation explained (JIVE). JIVE  18  is an extension of PCA to multiomics data. JIVE decomposes each omics matrix into three structures: a joint factor matrix (J), a omics-specific factor matrix (A) and a residual noise (E): X i ¼ J þ A i þ E i ¼ U i S þ A i þ E i ; for i ¼ 1; :::; P ð2Þ with E i , A i and U i are n i k ð Þmatrices and S is a common score matrix explaining variability across multiple data types. The algorithm minimizes E k k 2 , with E i ¼ X i À U i S À A i and E ¼ ½E 1 ::: E P T. JIVE is implemented into the R package r. jive (https://cran. r-project. org/web/ packages/r. jive/index. html). Multiple co-inertia analysis (MCIA). MCIA  19  , is an extension of co-inertia analysis (CIA) to more than two omics datasets.
MCIA factorizes each omics into omicsspecific factors X i ¼ A i F i þ E i ; for i ¼ 1; :::; P; ð3Þ by first applying a dimensionality reduction approach, such as PCA, to each omics matrix X i separately and then maximizing their co-inertia, i. e. , the sum of the squared covariance between scores of each factor: argmax q 1 1 ::: q 1 P X P k¼1 cov 2 X i k q i k ; X i q i À Á ð4Þ with var X i q i À Á ¼ 1 and q i correspond to the global PCA projections. MCIA is implemented in the R package omicade4 (https://bioconductor. org/packages/ release/bioc/html/omicade4. html). Regularized generalized canonical correlation analysis (RGCCA). RGCCA  21  is one of the most widely used generalizations of CCA to multiomics data. Similarly to MCIA, RGCCA factorizes each omics into omics-specific factors: X i ¼ A i F i þ E i ; for i ¼ 1; :::; P: ð5Þ RGCCA maximizes the correlation between the omics-specific factors by finding projection vectors u i such that the projected data have maximal correlation: argmax i;j Corr X i u i ; X j u j À Á for all possible couplesi; j ¼ 1; :::; P: Solving this optimization problem requires inversion of the covariance matrix.
MOFA is an extension of Factor Analysis to multiomics data, but it is also partially related to iCluster. However, differently from iCluster, MOFA does not assume a normal distribution for the errors but supports combinations of different omics-specific error distributions. The code to run MOFA is available at https://github. com/bioFAM/ MOFA. The MOFA package further implements an automatic downstream analysis pipeline for the interpretation of the obtained factor and weight matrices through pathways, top-contributing features or percentage of variance-explained interpretation. Tensorial independent component analysis (tICA). A natural extension of DR methods to multiomic data is based on the use of tensors, i. e. , higher-order matrices. Indeed, all the methods designed for single-omics can be naturally extended to multiomics with tensors.
Given that multiple tensorial versions of ICA exist, we considered the tensorial fourth-order blind identification (tFOBI), whose implementation in R was reported by Teschendorff et al. 23. Multi-study factor analysis (MSFA). MSFA 20 is a generalization of Factor Analysis (FA), which models the omics matrices X i as the sum of data-specific and shared factors: X i ¼ ΦF i þ Λ i L i þ E i ; for i ¼ 1; :::; P: ð10Þ where E i has a multivariate normal distribution and the marginal distributions of F i , L i , and X i are multivariate normal. MFSA is implemented in R and available at https://github. com/rdevito/MSFA.
Hence, scikit-fusion can naturally combine additional side information in the factorization of the multiomics data, such as protein-protein interactions, Gene Ontology annotations. It is implemented in Python and available at https://github. com/marinkaz/scikit-fusion. Factor selection for performance comparisons. The jDR approaches make different assumptions on the cross-omics constraints of the factors. The various jDR can be thus classified in shared factors, omics-specific factors and mixed factors approach. To use the factor matrices to compare the performances of the various jDR methods, e. g. , to cluster the samples based on the factors, we had to select which factor matrix to use for each jDR. Shared factors jDR methods compute a unique factor matrix, which is used in our benchmark.
Omics-specific jDR methods compute a factor matrix for each omics dataset. In these cases, we selected the factor matrix associated with transcriptomic data for our benchmark. However, jDRs with omics-specific factors maximize correlation or co-inertia between the various omics-specific factor matrices. The values of the transcriptomic factor matrix are then influenced by the other omics. Finally, for mixed factors jDRs methods, we considered the joint factor matrix F. As a consequence, all jDR methods with omics-specific and mixed factors contain more information in their factorization than that considered here for sake of comparison. Data simulation. The simulated multiomics datasets have been produced by the InterSIM CRAN package  24. InterSIM simulates multiple interrelated data types with realistic intra-and inter-relationships based on the DNA methylation, mRNA gene expression, and protein expression from TCGA ovarian cancer data.
We generated 100 simulated datasets, with a number of clusters set by the user. We considered five, ten, and fifteen clusters in this study. The proportion of samples belonging to each subtype is also set by the user, while we considered here two conditions with equally sized clusters and variable random sizes, respectively. Clustering of factor matrix. To identify the clusters of samples starting from the jDR factor matrix, we applied k-means clustering to the factor matrix (kmeans function in R). We chose k-means for clustering in agreement with the use of kmeans in iCluster and euclidean distance in intNMF for clustering. As k-means clustering is stochastic, we performed clustering 1000 times and computed a consensus consisting in the most frequent associations between samples and clusters Comparing jDR algorithm clusters to ground-truth clusters.
Data availability The simulated data are produced using the R package InterSIM V2. 2 and can be reproduced using our jupyter notebook (https://github. com/ComputationalSystemsBiology/momixnotebook)  37. The cancer TCGA data were dowloaded from http://acgt. cs. tau. ac. il/ multi_omic_benchmark/download. html. The single-cell data are available in the data/ folder of our github repository (https://github. com/ComputationalSystemsBiology/momixnotebook)  37. Annotations (i. e. , Gene Ontology, Reactome and MSigDB Hallmarks) were fetched from fgsea V1. 16. The remaining data are available in the article, Supplementary Information and Jupyter notebook  37.
Following evidence from refs. 21, 22, 29, 30 , we operationally defined slow waves as the occurrence of large-amplitude waves within the delta ([1-4] Hz) range. We first detected the occurrence of slow waves in each EEG electrode using an established approach developed in wakefulness and sleep (see refs. 30, 31, 36 and "Methods"). Both the temporal profile and topographical distribution of slow waves detected during the tasks smiling faces) and a SART on digits (NoGo trials: 3). Face/Digit presentation was continuous (new face/digit every 0. 75-1. 25 s). Images of faces were obtained from the Radboud Face Database (see "Methods"). b Every 30 to 70 s, participants were interrupted and instructed to report their mental state (see "Methods" and Supplementary Methods). Most importantly, they were asked to indicate whether they were focusing on the task (task-focused: ON), thinking about nothing (mind blanking: MB) or thinking about something other than the task (mind wandering: MW).