dataesr/scientific-paragraphs-categorization huggingface dataset:
861 and a relative error of 5. 84%, with 176. 934 RMSE. Adding 20% of solutions (i. e. , 50% of solutions) improved the performance of the deep learning model even further where it could achieve R 2 as high as 0. 891, RMSE of 176. 882, and RE as low as 5. 84%, respectively. Like the validation phase, the random forest model has consistently shown to be the second/third best model. Fig. 3  illustrates predictions using 30%, 50%, and 70% of learners' solutions from the top three models along with the ground truth. Fig. 5  in Appendix A provides line charts showing the top three models for sequence lengths of 40%, 60%, 80%, 90%, and 100% of the learners' actions. The high performance of the validated models on the independent game dataset provides evidence for the applicability of the trained models (deep learning, and random forest in particular) to early predict learners' performance.
e. , the final score in a game episode) in each time window. Once the time windows have been generated from a sequence of learners' actions, their score in each game episode is used to signal their final performance. To evaluate the feasibility of the GameEPM, we performed two sets of experiments using cross-validation and an independent game dataset. Regarding the first research question, our experiment results showed that the GameEPM approach could at early stages predict the learners' final score in the upcoming game episodes using the cross-validated deep learning model, with R 2 as high as 0. 816 in the validation phase and 0. 891 in the application phase. Concerning the second research question, a comparison of the results of different traditional machine learning methods with the deep learning model showed that, in the validation and application phases, up to 0.
30 and 0. 35 R 2 gain can be achieved by using the deep learning model, respectively. Regardless of the nature of the datasets (5611 solutions in the dataset used in the validation phase, and 560 solutions in the dataset used in the application phase) and the sequence of learners' actions, the deep learning model consistently outperforms all other models. With respect to the third research question, using only 30% to 50% of learners' action sequences, our proposed approach could achieve performance as competitive as those with more data (i. e. , > 50%), indicating its potential for early detection of at-risk learners. For instance, we found that using only the first three solutions of learners who completed the game with 10 solutions, our approach can predict their final with R 2 around 0.
Signaling pathways can be activated through various cascades of genes depending on cell identity and biological context. Single-cell atlases now provide the opportunity to inspect such complexity in health and disease. Yet, existing reference tools for pathway scoring resume activity of each pathway to one unique common metric across cell types. Here, we present MAYA a computational method that enables the automatic detection and scoring of the diverse modes of activation of biological pathways across cell populations. MAYA improves the granularity of pathway analysis by detecting subgroups of genes within reference pathways, each characteristic of a cell population and how it activates a pathway. Using multiple single-cell datasets, we demonstrate the biological relevance of identified modes of activation, the robustness of MAYA to noisy pathway lists and batch effect.
Introduction The identification of cell type and function is the driving force of a majority of single-cell studies. Such approaches are based on lists of canonical marker genes and pathway databases. Standard scRNA-seq analysis pipelines involve steps of dimensionality reduction and clustering before starting any marker or pathway analysis  [1] [2] [3]  , which makes the resulting conclusions highly dependent on the chosen algorithm and clustering parameters. In the case of oncogenic datasets, such clustering-based approaches appear inadequate to identify shared transcriptional programs across tumors as cancer cells tend to cluster independently per patient  [4] [5] [6] [7] [8] [9]  rather than group by biological similarities. Several approaches have emerged, bypassing dimensionality reduction and clustering, by proposing to score pathway activity directly in individual cells rather than clusters.
Such pooling of gene-based measurements into scores for gene lists has proven extremely powerful for the interpretation of sparse and noisy scRNA-seq datasets  10, 11. A recent benchmark  12  presented Pagoda2  13  and AUCell  14  as two of the top performing tools for pathway activity scoring. They are based on different scoring methods -AUCell estimates the proportion of highly expressed genes in each pathway while Pagoda2 uses the weights of the first principal component from Principal Component Analysis (PCA) -and each proposes a way to select significant scores. Nonetheless, both tools compute a unique activity score by pathway for all cells, implying that genes of a given signaling pathway should have coordinated expression across cell types. Biological evaluation of pathway activation and more recently single-cell studies have repeatedly demonstrated the heterogeneity of cell functions depending on the biological context.
Yet a majority of single-cell studies study pathway activation with single scores based on gene lists built from bulk data. Such curated gene lists represent the current reference biological knowledge and are the only available key to make biological sense of sparse and noisy scRNA-seq data. Adding more specialized curated gene lists to databases -detailing cellular functions according to cell identity -is ongoing but it will take some time to be completed. In order to already inspect existing pathway databases with single-cell resolution, we developed MAYA (Multimodes of pathwAY Activation), a tool that detects for each pathway the different modes of activation across cell types, each mode relying on different subsets of genes. We argue that MAYA could be a way for currently available biological knowledge to meet the granularity reached by single-cell data and help researchers go deeper in their understanding of complex cellular mechanisms.
Particularly, in the case of oncogenic datasets, we show that MAYA can detect cell type specific modes of pathway activation for both the microenvironment and tumor cells, identifying common transcriptional programs across patients. CC-BY-NC-ND 4. 0 International license made available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprint this version posted July 20, 2022. ; https://doi. org/10. 1101/2022. 07. 19. 500633 doi: bioRxiv preprint.
MAYA method MAYA enables comprehensive pathway study thanks to multimodal activity scoring of gene lists in individual cells (Fig. 1 ). Provided a scRNA-seq count matrix and pathway lists, MAYA detects all biologically relevant ways to activate each pathway relying subgroups of genes and summarizes their activity in each cell in a multimodal pathway activity score matrix (Fig. 1a ). This activity matrix can then be used to identify groups of cells sharing similar activation of provided pathways and as a dimensionally-reduced dataset for cell visualization (Fig. 1b ). As a comparison, reference tools that measure pathway activity, such as AUCell  14  or Pagoda2  13  , provide a unique activity score per pathway where MAYA can provide several. MAYA is built on two main functions that are applied to each provided gene list: the detection of activation modes and the selection of biologically relevant ones.
Detection of modes is performed thanks to a PCA on a normalized gene-cell expression matrix restricted to pathway genes (Fig. 1a ). The purpose of such decomposition of the matrix is to find, within the pathway, genes whose expression is coordinated and variable across cells, and to simultaneously score their activity in individual cells. Each principal component (PC) represents a possible mode of activation of the pathway, that is characterized by the genes that contribute the most to the PC, and by a score that corresponds to the cell coordinate on the PC. Each gene can contribute to several PCs and therefore to several modes. However, all detected modes might not reflect a relevant biological pattern in the data and could be driven by outliers, either cells and/or genes, and this probability increases as modes explain less and less variance in the dataset.
1d, e ). Default cutoff value was chosen to maximize the number of modes detected as informative while keeping a high average number of genes significantly contributing to each mode (Supplementary Fig. 1f ). Although MAYA's main purpose is to detect multimodal activation of pathways, it can also perform unimodal activity scoring, to detect cell identity from any cell marker gene lists. To this end, we have developed a built-in function to leverage MAYA's scoring and informativity methods to automatically annotate cells in a dataset. This approach is based on activation of the first mode of provided cell type markers lists, using PanglaoDB  16  by default (Methods). This function allows cluster-free cell type annotation in a timely fashion as it annotates a dataset of around 16,000 cells in less than 1 minute and 125,000 cells in approximately 15 minutes (Supplementary Fig.
MAYA detects biologically relevant multimodal pathway activity in kidney The main distinguishing feature of MAYA over existing pathway activity scoring tools is the multimodality of its activity score, which proves useful when studying broad pathways in complex biological systems. We first sought to demonstrate its ability to detect cell-type specific activation modes of hallmark pathways. For that, we ran MAYA on a dataset of normal kidney and immune cells from Young et al. 17  , from which we selected cells from 5 distinct subtypes for clarity (n=1,252). We used the MSigDB Hallmark pathways  18  as input gene lists, covering main biological functions. Unsupervised clustering on the multimodal activity matrix shows MAYA detects modes that distinguish different cell populations (Fig. 2a ).
MAYA detects biologically relevant multimodal pathway activity in colon We then illustrated the relevance of the biological insight gained by using multimodal pathway analysis for another tissue with a dataset of colon and immune cells from Lee et al. 22  -from which we selected cells from 10 distinct cell types (n=1,415) -and using the MSigDB KEGG and REACTOME pathways  23. Both analyses recover cell-type specific activation modes, given the clustering of cells by cell type on the heatmaps derived from the activity matrix (Supplementary Fig. 3a, c ). Focusing on KEGG cell adhesion molecules list, we observed that MAYA was able to detect several well-known types of cellcell adhesion processes starting from the mixed general reference list (Fig. 3a, b  and Supplementary Fig. 3b ): (i) mode 1 driven by the expression of HLA genes coding MHC class II molecules  24  , detected in antigen-presenting cells -monocytes and dendritic cells -and B cells, with a specificity of 0.
Finally, mode 2, mainly active in monocytes and dendritic cells (specificity of 0. 33 and 0. 16 respectively), involves genes associated with acidification of intracellular organelles through colocalization of V-type proton ATPases  35  (ATP6V1B2, ATP6AP1, ATP6V1F, ATP6V0E1, ATP6V0D2  36  ) and Chloride channels  37  (TTYH3, CLIC2), a process necessary for phagocytosis. Altogether, as for the kidney, starting from reference databases, MAYA untangles pathway activities specific to each cell type, revealing precise cell functions.
MAYA automatically assigns cell identity We then leveraged MAYA's scoring and selection ability to automatically and robustly assign cell identity. We applied MAYA to PanglaoDB cell type marker lists and the subsets of kidney and colon datasets used previously (Fig. 4a, d ). We demonstrate that MAYA enabled an automated and accurate annotation of each cell in the two datasets, using the initial cell type annotation by authors as a reference (Fig. 4b, e ). We compared the accuracy of our predictions with the ones obtained with three other algorithms: AUCell 14 , Pagoda2  13  and Cell-ID  38  , a cell type identification method based on Multiple Correspondence Analysis (MCA). MAYA presents among the highest rates of recall and precision for both datasets (Fig. 4c ,f and Supplementary Fig. 4a, b ).
We finally tested the scalability of MAYA and its ability to detect rare cell types on a dataset with 16,815 cells from ovarian tumors 6 (Supplementary Fig. 4c ). Overall, MAYA had an average precision of 51% and recall of 68%. Notably, B cells were identified with a precision and recall of 98% when they represent only 4. 9% of the dataset and endothelial cells with a precision of 100% and recall of 85% when they represent 0. 2% of cells in the dataset (Supplementary Fig. 4d ). Lower precision is achieved for some types probably due to overlap between cell type markers in PanglaoDB, such as between NK cells and T cells (28 shared markers out of 80 and 95 markers respectively), dendritic cells and macrophages (34 shared out of 121 and 128 markers respectively), and endothelial cells and fibroblasts (13 shared out of 187 and 171 markers respectively).
All three pairs of cell types share more genes than with any other type from the PanglaoDB lists. Furthermore, as batch effect is a main concern in single-cell analyses, notably for data visualization and cell annotation, we tested whether MAYA was affected by such technical biases. We worked on a dataset containing n=5,179 cells from laryngeal squamous cell carcinoma biopsies of 2 patients with a batch effect between patients  39. Using standard gene-based scRNA-seq matrix processing, cells from the same cell types -whether cells from the microenvironment or the tumors -indeed cluster by patient whereas clustering on the MAYA activity matrix groups cells by cell type, with cells from both patients within the same cluster (Fig. 4g ). To quantify the inter-patient overlap between clusters of similar cell types, we computed the Shannon Diversity Index (SDI) for both methods as well as for clusters obtained with the reference integration tool Harmony 40 (Supplementary Fig.
MAYA detects common modes of pathway activation across cancer patients Patient-specificity of cancer cells is currently a major limitation for the comprehensive study of oncogenic scRNA-seq datasets. Cells of the microenvironment coming from different patients can easily group together, showing the absence of a major batch effect between samples, while tumor cells form distinct clusters  [4] [5] [6] [7] [8] [9]. Such behavior is thought to be due in part to the genetic variations across tumor cells from different patients, notably copy-number variations. Integration methods, correcting for general batch effect in samples, such as Harmony  40  , are not suited to deal with such cell-type specific effect. We demonstrate here that MAYA can be an alternative to gene-based or integration-based methods to identify common transcriptional features between cancer cells across patients.
Using an ovarian cancer dataset, we show that MAYA identifies several modes of pathway activation shared across patients (Fig. 5a, b  and Supplementary Fig. 5a, b ) that are associated with known cancer hallmarks. Indeed, top specific modes of epithelial cancer cells reflect the expression of targets of the oncogene KRAS, genes associated with early response to estrogen or the P53 pathway (specificity of 0. 63, 0. 45 and 0. 31 respectively), that all relate to tumor growth and proliferation (Fig. 5b ). MAYA also identifies modes of pathway activation specific to tumor microenvironment populations. It notably detects a cell-type specific activation of complement genes in macrophages (specificity of 0. 24) and of angiogenesis-related genes in cancer-associated fibroblasts (CAFs) (specificity of 0.
40). MAYA multimodality allows to untangle several cell-type specific modes of activation for biological phenomena that are commonly difficult to sort out between cell populations within the tumors and their microenvironment. For example, MAYA detects different modes of epithelial-to-mesenchymal transition (EMT) (Fig. 5c ): mode 1 specific to CAFs/mesothelial cells (specificity of 0. 47 and 0. 36 respectively), mode 2 specific to tumor cells (specificity of 0. 30) and mode 3 to macrophages (specificity of 0. 19) (Fig. 5d ). MAYA identifies a combination of genes that characterizes EMT occurring in epithelial cells, with LAMA3 and LAMC2 being exclusive to this cell type (Fig. 5e ). These two genes expressed by basal epithelium code for two subunits of laminin 332, an essential component of epithelial basement membrane that promotes tumor cell motility  41, 42.
Discussion MAYA sorts out the different modes of pathway activation specific to each cell type, by automatically detecting gene subgroups within reference pathways and computing several scores of pathway activation. We show that MAYA leverages existing biological knowledge to extract cell-type specific ways of activating pathways from single-cell datasets. In addition to pathway analysis, MAYA also performs automated cell typing as a side function, making it an all-in-one tool for both cell type and cell function identification. MAYA proves particularly useful for single-cell cancer datasets, by (i) identifying common modes of pathway activations across patients in tumor cells, and also by (ii) dissecting the contribution of each population -fibroblast, immune & tumor cell -to the activation of a given pathway.
We have also challenged the robustness to noise of our scoring and informativity methods and showed MAYA can detect relevant biological signal from noisy pathway lists. It can prove very useful as we know pathway and cell markers manual curation is very time-consuming. Here, we argue that MAYA can take as input non-curated and potentially very exhaustive pathway or cell type lists and detect biological signal if they contain any. We also leveraged our methods of scoring and selection of informative scores to propose a built-in function to automatically annotate cells using PanglaoDB cell type markers lists. This method performs better with MAYA scores than Pagoda2 or AUCell scores and has performance results equivalent to or better than Cell-ID  38  , a package specialized in cell type annotation.
MAYA is scalable to large datasets (>100,000 cells, in 15 minutes) and it is able to accurately detect and annotate cell populations representing less than 5% of cells. MAYA is therefore an all-in-one tool proposing both cell type identification -like Cell-ID  38  , CellTypist  56  and scGate  57  -and multi-modal pathway analysis. MAYA also enables to identify shared identity expression patterns between cells from the same type across patients, which proves useful in case of batch effect. Indeed, as MAYA focuses on cell identity by looking only at genes considered as markers, it does not detect the variations between patients driven by other sets of genes that are not related to cell type identity and that lead to the formation of different clusters in a classical gene-based analysis. Finally, MAYA brings particular biological insights when studying single-cell datasets from cancer patients that do not suffer from batch effect on all cell types but from patient-specificity for tumor cells.
Both inter and intra-patient features of MAYA will enable the identification of shared therapeutic vulnerabilities across patients, as well as various strategies to target them within the tumor eco-system. provided for 125,139 cells, with 72,502 cells passing the author's quality control criteria. MAYA automatic annotation function was run on the dataset before and after QC filtering to evaluate its scalability to large datasets. For our detailed pathway analysis, only normal kidney cells were selected based on author's annotation (categories "Normal_mature_kidney" and "Normal_mature_kidney_immune"). Cells from 5 distinct cell types out of 28 were selected after default Seurat processing and clustering (aliases 8T, AV2, MNP1, G and M) for a total 1,252 cells. Colon dataset: Raw count matrix and cell annotations were downloaded from the NCBI Gene Expression Omnibus (GEO) database under the accession code GSE144735 for the KUL3 cohort.
Only protein-coding genes were kept for downstream analysis. MAYA automatic annotation function was run on this full dataset -including normal, tumor and border cells -to evaluate its scalability to large datasets. For our detailed pathway analysis, cells from Class "Normal" and from 10 out of the 35 cell types identifies by the authors were selected, representing a total of 1,415 cells. Ovary dataset: Count data were downloaded from the NCBI Gene Expression Omnibus (GEO) database with accession code GSE165897. Only cells labelled as treatment-naïve for the treatment phase metadata field were kept for downstream analysis, representing a total of 16,815 cells. Larynx dataset: Count data were downloaded from the NCBI Gene Expression Omnibus (GEO) database with accession code GSE150321 (2 files, one for each patient), for a total of 5,179 cells.
Reference databases: PanglaoDB was downloaded from the website (https://panglaodb. se/) and loaded in R with the provided command line. Markers lists are categorized by organs. Some can be considered as generic organs that should always be tested for a dataset (connective tissue, smooth muscle, immune system, vasculature, blood, epithelium, skeletal muscle), others are more specific such as kidney or lungs and can be loaded on demand. The full Panglao gene list can be loaded as well. Kidney related lists were loaded for the kidney dataset, GI tract related lists for the colon dataset, and finally no other list than generic types for the larynx and ovary datasets. MSigDB gene lists (Hallmark, KEGG and REACTOME) were downloaded from the Broad Institute website (http://www. gsea-msigdb. org/gsea/msigdb/collections.
jsp) in their version 7. 4. For the Reactome database, only pathways comprising between 100 and 300 genes were kept for efficiency purposes, which represents 165 pathways kept over 1615. Matrix preprocessing: All count matrices were processed with Seurat v3 to get the gene-based cell embeddings and check the consistency of author's annotations. Matrices were log-normalized using scale factor 10,000. Top 2,000 variable features were found using "vst" method. PCA and UMAP computed with default settings, using first 10 PCs for UMAP, which constitutes the "gene-based UMAP". For the larynx dataset, the two datasets were read separately and merged in a unique Seurat object of 5,179 cells. The authors did not provide their annotation, so we followed the default Seurat pipeline on each individual count matrix, performed PCA and default clustering.
As this process relies on the detection of inflection points that depends itself on the adjustment of the density curve to the distribution, we start with an adjustment meant to detect global variations of distributions and if none are detected we test a more fitted adjustment to ensure no significant local variation was missed. Then follow two additional checks to ensure the biological relevance of the detected mode. First, we filter out modes that are activated in very few cells as they could be outliers. The user can adjust this parameter based on what he expects to observe in the dataset or the number of cells from rarer cell type or set it to default 5%. The second biological check is based on the number of genes potentially contributing to the mode. However, it is hard to set a definition of what is a contributing gene to PCA; here we consider that contributing genes contribute more than they would be expected i.