dataesr/scientific-paragraphs-categorization huggingface dataset:
For some drugs, no feature meets with statistical significance, leading to an empty phenotype profile-which means there are no specific characteristics in the patient record that could be indicative of needing a dose change. To allow interpretation of our results, we built a phenotype profile browser, available at http://snowflake. loria. fr/p450/. It enables browsing phenotype profiles of drugs and drug sets, and displays the top features of each type. To aid interpretation, phenotype profiles also show the number of publications found in PubMed that mention both the drug and the code/condition/lab test included in the profile. Predicting drug dose reduction. We evaluated the effectiveness of phenotype profiles to predict the need for a patient to receive a dose-reduction to a single drug (or to a set of drugs) in two ways: first, by 10-fold cross-validation and second, by holding out the last year of data as a test-set.
Discussion We trained classifiers to predict dose changes using three types of features from EHR data. Our underlying assumption is that the need for dose changes is marker for an inappropriate drug response, such as a drug adverse response or lack of response. Our classifier (trained on the phenotypic profiles) successfully predicts dose reductions for 23 drugs out of 34 drugs in the study, but is ineffective in predicting dose increases. We also found that the laboratory test orders generated better predictions than other features. Finally, we find that when evaluating the prediction ability using a held out test set from a later calendar year (i. e. , totally unseen 'future data'), the performance is lower than, but consistent with, the commonly done 10-fold cross validation. For predictive models in medicine, an unseen future dataset is the most appropriate evaluation setup because it mirrors real-life usage of a model.
The opportunity is underscored by the fact that roughly 10,620,000 individuals per year in the US receive a new prescription for one of the 34 drugs we built models for (estimated using 2007-2014 Truven MarketScan Commercial Claims and Encounter database). The ability to prospectively identify a large fraction of these individuals (roughly 1,949,000 per year for the 10 drugs for which the dose reduction prediction works well) is highly significant.
While lab orders are not exactly phenotypic observations, they can be considered proxies for a suspected underlying condition. Only 2 in 10 lab orders were encoded with a terminology in our dataset, therefore we used the laboratory order codes directly rather than mappings to a reference terminology. We used the hierarchies of ontologies to generate an expanded feature set, as illustrated in Fig. 3. If a dose reduction interval is associated with the SNOMED CT concept Stomatitis, then it will automatically be associated with parent terms Inflammatory disorder of digestive tract and Disorder of digestive tract as defined in the SNOMED CT hierarchy. This expansion, on the parent-child hierarchy of the ontology, aims at capturing more information and improving classification of dose changes. Diagnostic codes and conditions are expanded using ICD-9-CM and SNOMED CT respectively, whereas lab orders were not associated with any hierarchy, thus not expanded.
Drugs and drug sets with less than 300 instances or with an empty phenotype profile (i. e. , no feature was found with sufficient p-value, RR and IC) were excluded from the evaluation; leaving 34 drugs and 23 drug sets to build classifiers for. Every classifier is evaluated with a 10-fold cross-validation and hold last year out validation. In each evaluation setting, the feature selection was achieved on the training data only, meaning that a distinct phenotype profile was computed for each fold of the 10-fold cross-validation. In the hold last year out setting, 2014 data that are left out comprises respectively 17. 5%, 18. 0% and 18. 9% of the dose reduction, increase and continuation intervals of the 2008-2014 data set. Such hold out evaluation illustrates the ability of our model to perform prediction on totally unseen data and also provides the most conservative estimate of performance in the face of data non-stationarity  34.
Interpretation of predictions. For interpreting the relationships between features in phenotype profiles and drug sensitivity patterns observed, we (MW, MC, and NJ) reviewed the results and investigated the biomedical literature. To facilitate our interpretation, we developed a web application with the JavaScript library D3JS (https://d3js. org), available at http://snowflake. loria. fr/p450/. and Information Content (IC). The elim method helps filtering results of the ontology expansion process  31. This method cannot be applied to laboratory test orders since those are not encoded with ontology codes.
Acknowledgements This work is supported by: The Snowflake/Snowball Associate Team funded by  Inria ; The  PractiKPharma  project ( ANR-15-CE23-0028 ) funded by The  French Research Agency (ANR) ;  The Widen Horizons program  of the IDEX " Lorraine Université d'Excellence " ( 15-IDEX-0004 ); The  NIGMS  grant  R01 GM101430  and  NLM  grant  R01 LM011369. We thank  Juan H. Banda ,  Alison Callahan ,  Rohit Vahisht  and  Rainer Winnenburg  for inputs in data processing, and IC calculation.
Motivation: Matrix factorization methods are widely exploited in order to reduce dimensionality of transcriptomic datasets to the action of few hidden factors (metagenes). Applying such methods to similar independent datasets should yield reproducible inter-series outputs, though it was never demonstrated yet. Results: We systematically test state-of-art methods of matrix factorization on several transcriptomic datasets of the same cancer type. Inspired by concepts of evolutionary bioinformatics, we design a new framework based on Reciprocally Best Hit (RBH) graphs in order to benchmark the method's reproducibility. We show that a particular protocol of application of Independent Component Analysis (ICA), accompanied by a stabilisation procedure, leads to a significant increase in the inter-series output reproducibility.
Moreover, we show that the signals detected through this method are systematically more interpretable than those of other state-ofart methods. We developed a user-friendly tool BIODICA for performing the Stabilized ICAbased RBH meta-analysis. We apply this methodology to the study of colorectal cancer (CRC) for which 14 independent publicly available transcriptomic datasets can be collected. The resulting RBH graph maps the landscape of interconnected factors that can be associated to biological processes or to technological artefacts. These factors can be used as clinical biomarkers or robust and tumor-type specific transcriptomic signatures of tumoral cells or tumoral microenvironment. Their intensities in different samples shed light on the mechanistic basis of CRC molecular subtyping.
Introduction Large-scale cancer genomics projects, such as The Cancer Genome Atlas (TCGA) and the International Cancer Genome Consortium (ICGC), are generating an overwhelming amount of omics data from multiple platforms and 10X Genomics released a scRNAseq dataset composed of 1. 3 million mouse brain cells  (Villani et al. , 2017). The available data offer us the unpredicted opportunity to understand cancer, its phenotypic properties, onset, progression and response to treatment. On the other hand, the massive sample size and high dimensionality of the current genomic data opens to all the computational and statistical challenges typical of "Big Data". New computational and statistical paradigms are thus essential to leverage the full power of high-throughput data, including bulk transcriptomic, epigenetic, proteomic and single-cell data.
Also because each gene in MF has a chance to contribute to several hidden factors, unlike standard clustering methods. The current state-of-art in the field of matrix factorization is represented by Principal Component Analysis (PCA), Non Negative Matrix Factorization (NMF) and Independent Component Analysis (ICA). Despite similar formulation of the approximation problem, each algorithm constrained in its own way and being applied to the same dataset, can lead to different sets of hidden signals  (Zinovyev et al. , 2013). The existing differences among the various MFs impede to perform a one-to-one mapping between the factors identified by the different methodologies. A more sophisticated solution is thus required to compare state-of-art MFs in biology and, to our knowledge, no such framework has been developed till so far.
In this manuscript we developed a new metaanalysis method, based on exploiting Reciprocal Best Hit (RBH) relations between metagenes and clustering the RBH network. The method allows extracting robustly reproducible and biologically relevant features without imposing thresholds on the correlation coefficients unlike standardly used correlation networks. We also suggest a set of relevant criteria for benchmarking the matrix factorization methods used to produce the RBH network. We took advantage of multiple publicly available transcriptomic datasets from the same cancer type to test the reproducibility of the results obtained by different approaches. The framework that we here proposed and our use of publicly available data guarantees that any future methodology can be easily compared against those considered in this paper.
Description of the considered state-of-art MF approaches The general idea behind Matrix factorization algorithms is to reduce data dimensionality through a decomposition approach. Given the natural representation of high-dimensional biological data as a matrix of measurements (expression counts, methylation levels, protein concentrations, etc) with different samples represented in the columns and different molecules of interest (genes, proteins, etc. ) represented in the rows. MFs decompose such a matrix X (n x m) into the product of an unknown mixing matrix A (n x k) and an unknown matrix of source signals S (k x m). In the following, we denote the columns of the molecule-level matrix "metagenes" and the rows of the sample-level matrix "metasamples". Metasamples and metagenes are learned based upon the assumption that the number k of biological factors occurring in the input dataset is smaller than either the number of rows or columns in the input matrix (X).
, 2016; Barillot, 2013). We here compared both approaches. We will call in particular "Stabilized ICA" the protocol previously proposed by us that maximizes kurtosis of metagenes and searches for stable components by performing a bootstrap approach and thus is able to prioritize stable components  (Biton et al. , 2014; Kairov et al. , 2017). We will instead denote with " ICA' " the application of ICA that maximizes kurtosis of metasamples, which corresponds to apply ICA to the transposed expression matrix, operation that in MATLAB is denoted with the symbol " ' ", from which the choice of the name is originated. For a typical transcriptomic dataset analysis, the advantages of Stabilized ICA are: a) having the number of objects larger than the number variables improves convergence of the algorithm and stability of the resulting components; b) resulting metasamples (unlike metagenes) might appear strongly correlated, reflecting coupling of certain biological mechanisms.
Biological context and datasets chosen for the comparison We here chose colorectal cancer (CRC) as biological context for our comparison. The choice of working in a cancer context is due to the fact that cancer is characterized by wider transcriptomic variations. In addition, a lot of transcriptomic datasets are already available and carefully annotated in cancer biology. Our framework takes advantage from having several independent transcriptomic datasets for the same cancer type. Among the various cancers, CRC is one of the most common cancer both in men and women and it is the one for which 14 independent and large transcriptomic data sets could be easily downloaded. Indeed, strong efforts have been done by multiple research groups to identify CRC subtypes, i. e. groups of CRC samples with homogeneous molecular and/or clinical characteristics.
Recently a consensus CRC subtyping (CMS) has been proposed  (Guinney et al. , 2015). In this work a collection of all the transcriptomic datasets previously employed for subtyping has been made available to the public, this is the resource that we employed for our comparison. For details concerning the datasets see Supplementary Table  1. Moreover, to test the effects of the platform on our analysis we also used 4 TCGA ovarian cancer datasets profiled with various microarrays technologies: Affymetrix U133, Agilent and Affymetrix HuEx, plus RNAseq. These 4 datasets have a total of 418 common samples. They are thus the perfect resource to test how the choice of the platform affects the different MFs.
In our previous work, we have shown that in case of ICA, overdecomposition is not detrimental for the interpretability of the resulting components  (Kairov et al. , 2017). The same is evidently true for PCA, since the resulting components are orthogonal. For NMF the number k of components in which a dataset should be decomposed is frequently decided by looking at the last local maximum of the cophenetic coefficient, summarizing the results of a consensus over different runs of the algorithm  (Brunet et al. , 2004). We thus chose to also compare our four algorithms against the version of NMF whose number of components is chosen based on the cophenetic coefficient, called in the following "cophNMF". Such comparison is reported in Supplementary Table  2. As shown in Figure  1  our new framework is composed of 4 main steps to be separately performed for each MF algorithm.
The only inputs required to perform the comparison are as many independent transcriptomic datasets as possible for the same biological context. In our case, these were the 14 CRC transcriptomic datasets from  (Guinney et al. , 2015). At step 1, each dataset is decomposed into a set of metagenes and metasamples. One of the main methodological novelties of this work is represented by step 2, where the graph of univocal correspondences between the metagenes obtained from the various independent datasets is reconstructed. Considering two transcriptomic datasets (T 1 and T 2 ), we define as best-hit of a metagene (M 2 n among all the metagenes of dataset T 1. Here and in the following we will refer to the obtained graph as Reciprocal Best Hit (RBH) graph. This name is chosen in analogy with the namesake most common working definition of orthology in comparative genomics.
Essentially, when the proteins encoded by two genes, each in a different genome, find each other as the best scoring match in the other genome the two genes are considered to be orthologous according to the RBH procedure  (Tatusov et al. , 1997; Bork et al. , 1998). The idea behind our approach is thus to identify orthologous biological factors across different transcriptomic datasets. This approach is conceptually very different from the frequently used approach of constructing a correlation graph. Indeed when using correlation a complete graph is reconstructed. To extract information from it some links need to be filtered out. Different filtering approaches exist, going from a simple thresholding to more sophisticated strategies  (Serrano et al. , 2009). Moreover, once the optimal filtering algorithm has been chosen, also the value of the threshold needs to be selected.
The obtained results are thus dependent on both the chosen filtering approach and threshold value. On the opposite, RBH is free of threshold choice and it leads to relatively sparse graphs. In Supplementary Figure  1  we compare the number of RBHs and the dimension of the largest connected component of the correlation network for various thresholds vs. the RBH network in all the MFs. Following the reconstruction of the RBH graph, we observed that the components detected by NMF were strongly biased toward the genes' average expression (see Supplementary Figure  2 ), i. e. the vector containing in each row the average expression of a gene across all the samples of the dataset. This resulted in a RBH graph where most of the metagenes are strongly correlated with each other, which makes the comparison with other MFs impossible.
Interpretation of the communities obtained in the RBH graph Once defined the best performing MF algorithm, we investigated which already known and new insights could be obtained using our newly proposed framework. With this aim we added to the analysis other four datasets: single-cell RNAseq from normal and tumoral CRC tissue  (Li et al. , 2017) , Patient-derived Xenograft (PDX) CRC Models transcriptomic data and liver metastasis (LM) transcriptomic data  (Isella et al. , 2017). Given the heterogeneity of such data in respect to the previous 14 we only employed them for the biological characterization and not in the assessment of MF algorithm performances. We characterized the communities obtained in the RBH graph with all the available biological annotations: MsigDB signatures  (Liberzon et al.
, 2011) , cell types signatures  (Aran et al. , 2017) , tissue-specific Transcription Factor (TF)-target associations  (Marbach et al. , 2016) , ToppGene  (Chen et al. , 2009) , the clinical annotations available for the various transcriptomic datasets, the CMS and CRIS subtypes  (Guinney et al. , 2015; Isella et al. , 2017)  and finally the cell types associations available from the single-cell RNAseq data  (Li et al. , 2017). We employed the metasamples of all the components contained in a community to test the association with clinical, CMS and CRIS annotations. We tested the significance of such annotation by performing a two-sided Wilcoxon or Kruskal-Wallis depending if the comparison was involving two classes (such as gender) or more than two (such as the 4 CMS subtypes), respectively.
BIODICA tool for computing and interpreting the stabilized independent components We have developed and released to public BIODICA (ICA applied to BIOlogical Data) tool, available from https://github. com/LabBandSB/BIODICA. BIODICA implements the protocol of Stabilised ICA to transcriptomic and other omics data. BIODICA provides both a command line and a user-friendly Graphical User Interface (GUI) for high-performance ICA analysis, including bootstraping and further stability analysis. BIODICA is based on previous high-performing implementations of fastICA and icasso algorithms, allowing to boost the ICA computation time by an order of magnitude compared to the existing fastICA implementation in R. BIODICA provides the possibil-ity to run ICA decomposition of several orders and it computes the Maximally Stable Transcriptome Dimensionality (MSTD) measure, which can be used to determining the optimal number of independent components  (Kairov et al.
, 2017). Moreover, BIODICA provides several tools for downstream interpretation of the resulting metagenes and metasamples. This interpretation includes standard functional analyses such as Preranked Gene Set Enrichement Analysis  (Subramanian et al. , 2005)  and hypergeometical tests using automated ToppGene web-service  (Chen et al. , 2009) , but also a built-in database of previously computed metagenes whose biological interpretation was already established. The results of this comparison are summarized in interactive html-based tables and JavaScript-based plots, which greatly facilitate exploration of the results. BIODICA also provides some original ways to analyze the metagenes using projection on top of molecular maps (such as InfoSig-Map,  (Cantini et al. , 2018) ). Finally, BIODICA implements the algorithm for constructing RBH graphs, used in the current study.
Controlling the number of false discoveries First, we want to assess the ability of the different MFs in producing false positives, or equivalently we want to evaluate how many components identified by the various MFs are prone to noise. Having multiple independent transcriptomic datasets from the same biological condition (in our case CRC), we can expect the same biological factors to be dominating the MF results in at least few datasets. As a consequence, a metagene should find a RBH in at least one other dataset. We thus considered that, if this does not happen, that metagene/node is with high probability a false positive result. To measure the number of false positive results in the outputs of the different MFs we evaluated the number of disconnected nodes/metagenes. As shown in Figure  3A , Stabilized ICA with 65 disconnected metagenes outperforms other approaches.
Reproducibility of the identified components In order to evaluate the reproducibility of the metagenes output of the different MFs we computed the number of links in their RBH graphs. Indeed, working with 14 CRC datasets, in an optimal scenario a metagene should find 13 RBHs corresponding to the metagenes that reflect the same biological factor in all the remaining 13 CRC transcriptomic datasets. In reality, this is not always the case given that a biological factor can be underrepresented in some datasets due to the choice of the samples or to their number. However, higher is the number of RBHs lower is the deviation of the performances of a MF approach in respect to the optimal scenario. As shown in Figure  3B , Stabilized ICA with 2900 RBHs, identifies approximately 1000 more RBHs in respect to the alternative approaches, strikingly outperforming them.
Presence of a cluster graph-like topology in the RBH graph Concerning then the topological structure of the RBH graph, the best MF algorithm should derive a clustergraph like graph, i. e. a disjoint union of cliques. Indeed as discussed above an optimal MF algorithm should find a component for each relevant biological factor underlying the transcriptome. Working with various transcriptomic datasets obtained from the same disease (CRC), those components associated to the same biological factor should cluster together forming a pseudo-clique. The final structure of the optimal RBH graph should be thus composed of various pseudo-cliques sparsely connected one to each other. In order to verify how the RBH graphs resulting from the different MF approaches are close to this optimal topology, we considered four well-established measures: (i) clustering coefficient; (ii) modularity; (iii) number of communities and (iv) average size of the communities.
The first two measures evaluate how evident is the presence of communities in the graph. The average size and number of the communities are instead used to evaluate how consistently each MF algorithm merges components obtained from different datasets. From the results reported in Figure  3C -F the superior performances of Stabilized ICA with respect to alternative approaches can be clearly appreciated. Especially the clustering coefficient and modularity are strikingly higher in Stabilized ICA in respect to ICA', NMF and PCA. Only for the number of communities NMF obtains a similar, even if slightly lower, value in respect to Stabilized ICA. As shown in Supplementary Table  2 , also concerning the topology of the RBH graph the performances of NMF do not improve if considering cophNMF.
Indeed a good regression would correspond to R 2 value near to 1 and low BIC and AIC values. Such scores are reported in Figure  3 G-I. The specific values obtained by the single scores are reported in Supplementary Table  3. As shown in Figure  3  G-I and Supplementary Table 2, Stabilized ICA is the approach that better approximates all three tested biological factors. In particular, NMF does not identify any component that can significantly predict the gender signal. We then investigated the selectivity of such predictions, meaning the ability of the MF approach to define a clear one-to-one association between a biological signal and a component. To test for the selectivity of the different MFs we focused on the components obtained on the GSE39582 dataset (see Supplementary Table  1 ) and considered the R 2 obtained in the previously computed regressions by all the 100 components.