dataesr/scientific-paragraphs-categorization huggingface dataset:
In that respect, the ARI of a consensus clustering in relation to its inputs should be maximized for a maximum of input clusterings, including for those coming from different omics. The highest similarity between consensus and input clusterings from different omics sources is observed for the SARC cancer dataset (see Fig. 4 ), with ARI values ranging from 0. 4 up to 0. 9 for at least one input clustering computed from each of the three omics datasets, suggesting similar associations at different molecular levels. For the other cancer types, the agreement between omics sources is less straightforward. Interestingly, for all cancer types, COCA and ClustOmics consensus clusterings resemble the same input clusterings (computed from the same set of omics sources), thus suggesting that some omics are more appropriate to explain molecular differences between individuals.
For the cancer datasets showing higher heterogeneity among input clusterings survival quality, ClustOmics found significant survival p values for AML, KIRC, LIHC and SKCM, despite some low-quality clusterings that were given as input. Clinical labels found enriched in consensus clusters are listed in Table  2. AML clusters were found enriched for both CALGB and FAB classifications, KIRC clusters for histologic grade and pathologic M and T (referring to the TNM classification of tumors  [25] ), LIHC clusters for gender, age at diagnosis and fetoprotein outcome value, while SKCM clusters showed no enriched clinical parameters. While BIC consensus clustering did not show good survival-wise results, pathologic M, N and T labels were found enriched in clusters, as well as pathologic stage, histological type, PAM50 call, and estrogen and progesterone receptor status.
Study case: BIC single-to-multi consensus clustering In this section, we focus on the consensus clustering of the 15 single-omics clusterings for the BIC dataset (five clustering methods, listed in the previous section, applied on three omics data types) and analyze these results in parallel to the PAM50 classification. As the PAM50 classification is computed from the expression of 50 specific genes, while in this work, we capitalize on three different omics, a certain heterogeneity in the clusters when compared to the PAM50 prediction is expected. Moreover, this heterogeneity is to be further explored, as it could reveal subtypes that are not distinguishable when considering only PAM50 genes but that are heterogeneous when integrating other data sources. From the 1096 patients available in the BIC dataset (of which only 621 patients are measured for the three omics), ClustOmics succeeded in primarily classify 600 multiomics patients in consensus clusters, with a number of supports threshold of 11 (see Table  3 ).
More specifically, in this work, we address the cancer subtyping problem from a personalized medicine-related perspective, which is gaining increasing attention. To treat patients according to their disease profile, one should be able to distinguish between disease subtypes. These disease subtypes can be predicted from omics data (traditionally gene expression but also methylation, miRNA, etc. ) by performing patient clustering (hierarchical clustering, density-based clustering, distribution-based clustering, etc. ). Our novel graph-based multi-integration method can fuse multiple input clustering results (obtained with existing clustering methods on diverse omics datasets) into one consensus clustering, regardless of the number of input clusters, number of objects clustered, omics and methods used to generate the input clusterings.
Though our method does not formally weight input datasets (e. g. , according to their level of confidence), one can artificially enhance the impact of one or several omics sources by providing supplementary single-omics input clusterings. In the same way, when dealing with missing data, patients measured with all omics are more likely to accumulate supports and therefore more likely to cluster together. In a context of multisource integration, favoring individuals with the least quantity of missing data makes it possible to highlight the predictions supported by several data sources, which is the desired behavior. In a context of single-source integration, the same set of objects is usually used in all input clusterings (apart from a few specificities of the input clustering tools used). TCGA real datasets from three different omics and ten cancer types were analyzed with respect to two integration scenarios: (1) fusing multi-omics clusterings obtained with existing integrative clustering tools and (2) fusing omics-specific input clusterings.
For the same reason, it is important to highlight that as long as the signals in the available omics are strong, ClustOmics is able to cluster samples that do not appear in all omics datasets, making use of available data and addressing the issue of partial data. Finally, though presented in a disease subtyping context, one should grasp that our method is not limited to this application case. ClustOmics is generic and adaptable to a wide range of biological questions, as one can use any kind of partitioning of the data, including clinical labels, groups of genes of interest, etc. , as an input clustering. A major strength of ClustOmics resides in its exploratory aspect, resulting both from a flexible intrinsic model that gives the user complete power on the integration scenario to investigate and from the use of the graph-oriented database Neo4j.
Datasets and tools used for computing input clusterings We used ClustOmics to predict cancer subtypes from gene expression, microRNA expression and DNA methylation datasets available in The Cancer Genome Atlas (TCGA)  [42]. Our case-study is based on the same datasets as in Rappoport and Shamir's review on multi-omics clustering methods  [1]. The data cover ten cancer types: leukemia (AML), breast (BIC), colon (COAD), glioblastoma (GBM), kidney (KIRC), liver (LIHC), lung (LUSC), ovarian (OV), sarcoma (SARC) and skin (SKCM). For each cancer type, from 197 and up to 1098 patients were measured for at least one of the three omics (expression, miRNA and methylation), of which 170 to 621 patients were measured for all three. More details on missing data per cancer type are given in Table  5. To generate input clusterings to be fused by ClustOmics, we used five state-of-theart integrative clustering tools, summarized in Table  6 : PINS  [6] , SNF  [7] , NEMO  [8] , rMKL  [9]  and MultiCCA  [10].
Computation of COCA consensus clusterings To assess the performance of ClustOmics with respect to a state-of-the-art integration method based on consensus clustering, we applied cluster-of-clusters analysis (COCA) wMQ(IG, C) = 1 k × max(w IG ) k � i=1   2 × W (e ii ) |C i | × (|C i | -1) - 1 k -1 � j� =i W (e ij ) |C i | × |C j |   [5] on each integration scenario and from the same set of input clusterings as for Clus-tOmics. COCA had already been applied to cancer-subtyping in a multi-omics context  [43, 44]. COCA is an integrative clustering tool based on the consensus clustering (CC) algorithm introduced by Monti et al. [45]. The CC algorithm implements a resampling-and co-occurrence-based strategy to assess the stability of clusters when analyzing a single dataset. By resampling a single dataset multiple times and applying a clustering algorithm on each perturbed dataset, and from the co-occurrences counts of samples in clusters, a consensus matrix is computed and used as a similarity matrix to compute a final consensus clustering.
Therefore, in the discussion of the results, we consider survival and clinical analysis as ways to interpret patterns captured by the various clustering results and do not favor one metric over the other. Finally, to evaluate differentially expressed genes across consensus clusters generated using the StoM scenario on the BIC study case, we applied the Kruskal-Wallis test on each gene available from the BIC expression dataset. The p values were adjusted to control the false discovery rate (FDR) [50], filtered with a 0. 001 significance threshold, and top 1000 most significant genes were retained for further Table  7  Pancancer and cancer-specific labels used for clinical label enrichment analysis. Pathological M, N and T labels refer to the TNM staging system, which describes the anatomical extent of tumor cancers  [25]  Pan.
Implementation ClustOmics is implemented based on the Neo4j graph database management system and uses APOC and Graph Data Science Neo4j libraries. Queries on the graph database are performed in Cypher, Neo4j's graph query language, and are encapsulated in Python scripts. To facilitate its use, ClustOmics can be run through the Snakemake workflow management system. ClustOmics was tested on a desktop-computer with an Intel Xeon processor (2. 70 GHz, 62 GB of RAM) running on Ubuntu 18. 04. For the TCGA real datasets that it was applied to, the ClustOmics runtimes range from a few minutes for small datasets (AML, multi-to-multi scenario) up to 2 h for the largest dataset (BIC, single-to-multi scenario), with most of the computation time being consumed for the construction of the integration graph.
With the graph stored in a Neo4j database, this step is only to be performed once for each integration scenario, and parameters for graph filtering can be further set and tuned without recomputing the graph. The ClustOmics source code, released under MIT license, and the results obtained on the ten cancer types with the two integration scenarios described in this paper are available on GitHub: https:// github. com/ galad rielb riere/ Clust Omics.
Availability of data and materials The datasets analysed in the study were obtained from Ron Shamir's lab and are available at http:// acgt. cs. tau. ac. il/ multi_ omic_ bench mark/ downl oad. html. ClustOmics' source code, released under MIT licence, as well as the results obtained on TCGA cancer data are available on Github: https:// github. com/ galad rielb riere/ Clust Omics.
Therefore, based on our previous experience of formalizing the Regulatory Circuits data-sets with Semantic Web Technologies, we decided to create a new tool for transcriptional networks inference, that could solve these issues. Results: Our tool, Regulus, provides candidate signed TF-gene relations computed from gene expressions, regulatory region activities and TF binding sites data, together with the genomic location of all entities. After creating expressions and activities patterns, data are integrated into a RDF endpoint. A dedicated SPARQL query retrieves all potential TF-region relations for a given gene expression pattern. These ternary TF-regiongene pattern relations are then filtered and signed using a logical consistency check translated from biological knowledge. Regulus compares favorably to its closest network inference method, provides signs which are consistent with public databases and, when applied to real biological data, identifies both known and potential new regulators.
Background: regulatory networks inference Table  1  presents a survey of transcriptional regulatory networks inference methods, which were analyzed for their ability to answer our goals regarding the description of normal and pathological B cell differentiation. The criteria were: (i) uses information from gene expression, TF binding, regulatory regions and their accessibility, (ii) can work with very limited number of human samples (iii) can predict inhibitions as well as activations, i. e. provides signed networks and (iv) can be adapted to new dataset, i. e. is reproducible and reusable. In terms of data, 11 out of the 14 methods reviewed in Table 1 use time series of gene expressions as the only input data and for 10 of them this is the only mandatory entry. This suggests that the regulatory regions impact is not taken into account in most of the methods.
Few methods use information about TFs binding sites or regulatory regions -among them we can find Regulatory Circuits (10) -and none of them checks whether the proposed TFs are expressed in order to play their predicted roles. The resulting networks may then contain candidate relations which are not consistent with the biological situation and with limited relevance to use in a personalized medicine clinical setting. We also observed that most methods from Table  1  produce networks with weighted edges, based on statistical or probabilistic analyses and require large datasets acquired at several time points. This required number of samples is a strong limitation to the application of these methods to human data (in particular for clinical data) with a limited number of available samples when considering disease-related and patientspecific regulatory networks.
Regulatory Circuits The only method which overcomes the main limitations of the reviewed methods in terms of use of regulatory landscape information and applicability to clinical settings in human is the Regulatory Circuits project. This work was a great effort to merge impressive amounts of data from the ENCODE (25), FANTOM5  (26)  and RoadMap Epignomic  (27)  consortia to describe human cell-type or disease specific regulatory networks  (10). The project resulted in 394 cell type-specific gene-regulatory networks, in which TF-gene relations are associated with different weights according to the considered cell type. However, as noticed above, a single network in the dataset (named "CD20+ B cells") encompassed all the B cell subtypes of our case-study. This prevented us to understand the differences between Bcell subtypes, underlining a granularity issue.
As a first step towards the adaptation of Regulatory Circuits to the analysis of our case study, we successfully integrated the data used to build TF-gene regulatory networks into a unique structured graph  (28) , therefore facilitating the re-use of this resource. More precisely, in this previous work, we used Semantic Web Technologies, a generic data and knowledge integration framework  (29, 30) , to generate a unique RDF dataset that can be queried by dedicated SPARQL queries. This allowed recomputing the relations between a TF and a gene published by the Regulatory Circuits project. Although our approach allowed to improve the Regulatory Circuits project results reusability, it also highlighted that its design makes it impossible to provide information about the activating or inhibiting role of TFs involved in regulations.
Regulus is based on Semantic Web Technologies, extending the methods introduced in  (28). Its main principles are to (1) take into account regulatory factors (TF and regions) activities, (2) propose an original discretization of the activities into patterns, (3) produce signed networks inferred by a logical consistency step and (4) be easily reusable and applicable to many datasets. Regulus has been developed to be stringent and to limit the space of the candidates TF-genes relations highlighting the candidate relations which are the most likely to occur. By applying Regulus to published or original datasets, we show that it can describe regulatory networks with validated signed relations, it is able to highlight known regulators of a specific biological process and provides a list of candidate new regulators, that can be further refined.
Main characteristics of Regulus. Regulus is available as a Conda / SnakeMake package at https:// gitlab. com/gcollet/regulus. It allows, from gene expression and regulatory regions activity tables, with their respective genomic coordinates, to infer consistent TFregion-gene relations with patterns indications for all entities. As inputs, Regulus requires: (i) a list of genes with their expression in a selected number of cell types, (ii) a list of selected regulatory regions with their activity as two text files, (iii) genomic locations of the genes and regions as bed files and (iv) TF binding sites locations (bed file). These can be provided by the user (TF data from a specific ChIP-seq analysis for example), but we provide an implementation using the genome-wide TF binding sites coordinates from Regulatory Circuits (10), containing curated binding sites for a total of 643 TF.
Each averaged expression value therefore gets an attribute from 1 to 4 corresponding to the interval it belongs to. Patterns are computed by a python script. Figure  1  shows the pattern attribution of a gene, Figure  1a  shows the expression of the gene in read count per million and the attributed pattern for each cell type. Figure  1b  illustrate the pattern creation: the higher point of expression is put to 4, the lowest to 1 and then the space is divided in four equal intervals, each resting point is put to their interval number, in this case leading to the pattern 3441. Finally, genes with low expression as defined in differential expression analysis have been granted the profile 0000 and have been removed from the implementation. Genes and regions with constant expression in all population have been granted the profile 5555.
Data graph for the integration and query in Regulus. From the previous pre-processed data transformed from tabulated to TTL format by a custom python script, Regulus generates a structured RDF graph of data that can further be queried with Semantic Web technologies. The construction of the RDF dataset requires the introduction of unique identifiers and of reified entities  (35, 36)  to describe some relations. (1) Identifiers For the regions, we used a unique identifier designed after the type of region (i. e. ATAC_; and Region_ in the text) followed by the row number at which they appear in the region localization file -this ensured that a same number was never used twice for different entities. For genes and TF, we kept their usual names (HGNC Gene Symbols) as identifiers. (2) Reified entities As RDF does not allow relations to bear a score, relations of distance between genes and regions and of TF binding into regions were both inserted by using reified relations.
This method generated new entities with devoted identifiers and   bearing either the distance value or the inclusion. (3) Additional information References to Uniprot and Ensembl were added for the genes to link our data with public databases. We also kept the localization information for all features, as a potential filtering parameter. To find the potential regulators of a given set of genes, Regulus uses the SPARQL query described in Figure  2  on the previously generated graph of data to extract all TF-Region-Genes triples and their related patterns.
Consistency table to assign roles to relations in Regulus. Regulus relies on a principle of consistency between genomic landscape, genes and TF expressions to decide if a relation is susceptible to exist. This consistency also allows predicting an inhibition or activation effect for each relation. Figure  3  bottom presents a graphical representation of 2. 6 Datasets used for validation and testing the consistency principle behind assignment tables. When the chromatin is not accessible and the TF not expressed (First line of Figure  3a ): if there is no gene expression then the TF is more likely to act as an activator and if there is a high gene expression then the TF is more likely to act as an inhibitor. When the chromatin is accessible and the TF expressed (Figure  3b ): if there is a gene expression then the TF is more likely to act as an activator and if the gene expression is low or null, then the TF is more likely to be an inhibitor.
GO and MeSH annotation. To validate the TFs inferred by Regulus, the stand-alone module ClassFactorY allows for automatic queries of the GO, Uniprot and PubMed databases. A user-defined list of GO annotations is used to verify if candidates TFs are annotated by these terms. The MeSH terms are retrieved from a user-defined list of PubMed publications about the biological context and the module counts the number of citations associating the TFs to one or several of the terms. An annotation-based score is then calculated and provided as a help-decision tool for end users. 2. 6. Datasets used for validation and testing. 2. 6. 1. "Roadmap Epigenomics" RNA-seq datasets. For validating gene inclusion in the networks computed by Regulus we used the same "Roadmap Epigenomics" RNA-seq datasets and methods that were used in  (10).
B-cells datasets. To investigate Regulus prediction abilities, we used datasets of differentiating B cells (accessible on demand), comprising 3 replicates of RNA-seq for naive B cells (NBC), IgM secreting or IgG secreting memory B cells (IgM+ or IgG+ MBC). Data were aligned on the hg19 human reference genome and gene expressions for 26,734 genes were calculated with featureCounts. Raw counts files were then used for differential expression analysis with DE-Seq2, allowing us to create a list of unexpressed (0000 pattern), invariable (5555 pattern) and variable genes and TF (used to compute relations). ATAC-seq data was obtained on the same cell types (n = 1), aligned on hg19, 35,078 regions were called with MACS2. An union of all regions was made and reads were counted for each region, normalized by sequencing depth and region size to compute read densities, using an in-house bash script.
Results and Applications 3. 1. The Regulus tool. We designed Regulus, a transcriptional regulatory networks inference tool dedicated to the analysis of few and biologically-close datasets. The tool relies on Semantic web to integrate expression and epigenetic data. Figure  4  presents the different steps of the Regulus pipeline. The preprocessing steps (blue steps in the Figure  Figure 4d , detailed in Figure  4a ) consist in (i) transforming individual gene expressions and regulatory regions activity (read densities) into patterns, (ii) finding relations between genes and regions by computing the distances between neighboring genes and regions, (iii) finding the inclusion of TF binding sites into regions. The pre-processed data are then integrated using Semantic Web technologies (yellow step of In blue: pre-processing steps based on biology (a), expliciting the relations between the entities: inclusion of the TFBS in region, distance between region and genes and in red the pre-treatment done: discretization and filtration.
Regulus generates large networks which includes low expressed genes. Networks topology is refined by the consistency step. Table 5b presents the number of relations obtained by Regulus for these four datasets. Data comprise 16,888 genes expressions, 43,012 regulatory region activities and 124,358,159 TF binding sites genomic coordinates. Just after the query and before running the consistency table, we can see that we had the same number of relations (3,005,934 TF-region-Gene or 1,869,854 TF-Gene), this is due to the fact that we have the exact same information on the TF binding sites and regions for the four sets. As seen in Table  5b  the consistency step allows to generate networks which are different in size and quality. They represent about 10% of the possible relations predicted by the query, underlining the filtering power of the consistency step.
Networks computed on dissimilar sets of tissues are slightly smaller than the ones computed with similar subsets. This lower number of relations may be explained by the increased lack of consistency in TF-gene regulations across widely different tissues. Inclusion of low-expressed genes. For validating gene inclusion of our networks we used the same "Roadmap Epigenomics" RNA-seq datasets that were used in  (10). As seen in Table  5c , for the networks computed using Regulus, we (d) Relations found in Trrust and Signor and coherence of signs. True: number of relations with the same sign as the database, False: relations with different sign than the database, Unknown: relations non signed or signed + and -in the database. For the union of Trrust and Signor: different: relations signed differently in the two databases, True: relation signed the same as both databases and False: relations signed differently than the databases.