netDx: interpretable patient classification using integrated patient similarity networks

Article19 March 2019Open Access Transparent process netDx: interpretable patient classification using integrated patient similarity networks Shraddha Pai Shraddha Pai orcid.org/0000-0002-1048-581X The Donnelly Centre, University of Toronto, Toronto, ON, Canada Affiliate Scientist, The Centre for Addiction and Mental Health, Toronto, ON, Canada Search for more papers by this author Shirley Hui Shirley Hui The Donnelly Centre, University of Toronto, Toronto, ON, Canada Search for more papers by this author Ruth Isserlin Ruth Isserlin The Donnelly Centre, University of Toronto, Toronto, ON, Canada Search for more papers by this author Muhammad A Shah Muhammad A Shah The Donnelly Centre, University of Toronto, Toronto, ON, Canada Search for more papers by this author Hussam Kaka Hussam Kaka The Donnelly Centre, University of Toronto, Toronto, ON, Canada Search for more papers by this author Gary D Bader Corresponding Author Gary D Bader [email protected] orcid.org/0000-0003-0185-8861 The Donnelly Centre, University of Toronto, Toronto, ON, Canada Department of Molecular Genetics, University of Toronto, Toronto, ON, Canada Department of Computer Science, University of Toronto, Toronto, ON, Canada The Lunenfeld-Tanenbaum Research Institute, Mount Sinai Hospital, Toronto, ON, Canada Search for more papers by this author Shraddha Pai Shraddha Pai orcid.org/0000-0002-1048-581X The Donnelly Centre, University of Toronto, Toronto, ON, Canada Affiliate Scientist, The Centre for Addiction and Mental Health, Toronto, ON, Canada Search for more papers by this author Shirley Hui Shirley Hui The Donnelly Centre, University of Toronto, Toronto, ON, Canada Search for more papers by this author Ruth Isserlin Ruth Isserlin The Donnelly Centre, University of Toronto, Toronto, ON, Canada Search for more papers by this author Muhammad A Shah Muhammad A Shah The Donnelly Centre, University of Toronto, Toronto, ON, Canada Search for more papers by this author Hussam Kaka Hussam Kaka The Donnelly Centre, University of Toronto, Toronto, ON, Canada Search for more papers by this author Gary D Bader Corresponding Author Gary D Bader [email protected] orcid.org/0000-0003-0185-8861 The Donnelly Centre, University of Toronto, Toronto, ON, Canada Department of Molecular Genetics, University of Toronto, Toronto, ON, Canada Department of Computer Science, University of Toronto, Toronto, ON, Canada The Lunenfeld-Tanenbaum Research Institute, Mount Sinai Hospital, Toronto, ON, Canada Search for more papers by this author Author Information Shraddha Pai1,2, Shirley Hui1, Ruth Isserlin1, Muhammad A Shah1, Hussam Kaka1 and Gary D Bader *,1,3,4,5 1The Donnelly Centre, University of Toronto, Toronto, ON, Canada 2Affiliate Scientist, The Centre for Addiction and Mental Health, Toronto, ON, Canada 3Department of Molecular Genetics, University of Toronto, Toronto, ON, Canada 4Department of Computer Science, University of Toronto, Toronto, ON, Canada 5The Lunenfeld-Tanenbaum Research Institute, Mount Sinai Hospital, Toronto, ON, Canada *Corresponding author. E-mail: [email protected] Molecular Systems Biology (2019)15:e8497https://doi.org/10.15252/msb.20188497 PDFDownload PDF of article text and main figures. Peer ReviewDownload a summary of the editorial decision process including editorial decision letters, reviewer comments and author responses to feedback. ToolsAdd to favoritesDownload CitationsTrack CitationsPermissions ShareFacebookTwitterLinked InMendeleyWechatReddit Figures & Info Abstract Patient classification has widespread biomedical and clinical applications, including diagnosis, prognosis, and treatment response prediction. A clinically useful prediction algorithm should be accurate, generalizable, be able to integrate diverse data types, and handle sparse data. A clinical predictor based on genomic data needs to be interpretable to drive hypothesis-driven research into new treatments. We describe netDx, a novel supervised patient classification framework based on patient similarity networks, which meets these criteria. In a cancer survival benchmark dataset integrating up to six data types in four cancer types, netDx significantly outperforms most other machine-learning approaches across most cancer types. Compared to traditional machine-learning-based patient classifiers, netDx results are more interpretable, visualizing the decision boundary in the context of patient similarity space. When patient similarity is defined by pathway-level gene expression, netDx identifies biological pathways important for outcome prediction, as demonstrated in breast cancer and asthma. netDx can serve as a patient classifier and as a tool for discovery of biological features characteristic of disease. We provide a free software implementation of netDx with automation workflows. Synopsis netDx is a supervised patient classification algorithm based on the paradigm of patient similarity networks. It integrates multi-omic data and uses biological pathway information to help with model interpretability. In a cancer survival prediction benchmark, netDx performs competitively or better than a diverse panel of machine-learning algorithms. When patient similarity is defined by pathway-level gene expression, netDx identifies biological pathways predictive of outcome, as demonstrated in diverse data sets (breast cancer and asthma). netDx is freely available as an R package and as a Docker image. Code, tutorials and worked examples are available at: http://netdx.org. Introduction The goal of precision medicine is to build quantitative models that guide clinical decision-making by predicting disease risk and response to treatment using data measured for an individual. Within the next 5 years, several countries will have general-purpose cohort databases with 10,000 to > 1 million patients, with linked genetics, electronic health records, metabolite status, and detailed clinical phenotyping; examples of projects underway include the UK BioBank (Sudlow et al, 2015), the US NIH Precision Medicine Initiative (https://obamawhitehouse.archives.gov/precision-medicine), and the Million Veteran Program (http://www.research.va.gov/MVP/). Additionally, human disease-specific research projects are profiling multiple data types across thousands of individuals, including genetic and genomic assays, brain imaging, behavioral testing, and clinical history from integrated electronic medical records (Hudson et al, 2010; Calkins et al, 2015; Collins & Varmus, 2015) (e.g., the Cancer Genome Atlas, http://cancergenome.nih.gov/). Computational methods to integrate these diverse patient data for analysis and prediction will aid understanding of disease architecture and promise to provide actionable clinical guidance. Statistical models that predict disease risk or outcome are in routine clinical use in fields such as cardiology, metabolic disorders, and oncology (Wilson et al, 1998; Schmidt et al, 2005; Gail et al, 2007; Lee et al, 2014). Traditional clinical risk prediction models typically use generalized linear regression or survival analysis, in which individual measures are incorporated as terms (or features) of a single equation. Standard methods of this type have limitations analyzing large data from genomic assays (e.g., whole-genome sequencing). Machine-learning methods can handle large data, but are often treated as black boxes that require substantial effort to interpret how specific features contribute to prediction. Black box methods are unlikely to be clinically successful, as physicians frequently must understand the characteristic features of a disease to make a confident diagnosis (Castelvecchi, 2016). Interpretability is particularly required in genomics because of relatively smaller sample sizes and to better understand the molecular causes of disease so that targeted therapies can be designed. Further, many existing methods do not natively handle missing data, requiring data pruning or imputation, and have difficulty integrating multiple different data types. The patient similarity network framework can overcome these challenges and excels at integrating heterogeneous data and generating interpretable models. In this framework, each feature of patient data (e.g., gene expression profile, age) is represented as a patient similarity network (PSN; Fig 1A). Each PSN node is an individual patient, and an edge between two patients corresponds to pairwise similarity for a given feature. For instance, two patients could be similar in age, mutation status, or transcriptome. PSNs can be constructed based on any available data, using a similarity measure (e.g., Pearson correlation and Jaccard index). Because all data are converted to a single type of input (similarity networks), integration across diverse data types is straightforward. Patient similarity networks (PSN) are a recently introduced concept and have been used successfully for unsupervised class discovery in cancer and type 2 diabetes (Wang et al, 2014; Li et al, 2015), but have never been developed for supervised patient classification. Figure 1. The netDx method Patient data, provided as input to netDx in the form of tables. The simple example for predicting low/high risk of disease uses clinical, genomic, metabolomic, and genetic data as input. Patient similarity networks (or PSN) are networks with patients as nodes and weighted edges representing patient similarity by some measure. netDx converts patient data into a set of patient similarity networks. netDx identifies which networks strongly relate high-risk patients (here, clinical and metabolomic data) and which relate low-risk patients (clinical and gene expression data). Feature selection is used to score networks by their ability to predict patient class (details in Fig EV1). netDx output. netDx returns several types of output. Top-scoring features are combined into a single view, or network of overall patient similarity, which can be used to classify new patients based on relative similarity to known patient classes. netDx also provides standard classifier performance metrics and scores for the predictive value of individual features. Network-based visualization of top predictive pathways. If pathway features are used, netDx identifies and visualizes the pathways most useful for classification. Download figure Download PowerPoint We describe netDx, the first PSN-based approach for supervised patient classification. In this system, patients of unknown status can be classified based on their similarity to patients with known status. It applies the idea of recommender systems, similar to those used in Amazon or Netflix (“find movies like this one”), to precision medicine (“find patients who don't respond to therapy”). This process is clinically intuitive because it is analogous to clinical diagnosis, which often involves a physician relating a patient to a mental database of similar patients they have seen. As demonstrated below, netDx has strengths in classification performance, heterogeneous data integration, and interpretability. Results Algorithm description The overall netDx workflow is shown in Fig 1, with detailed views of particular steps in Figs EV1 and EV2. The example in Fig 1 conceptually shows how patient similarity networks (PSNs) can be used to predict whether a patient is at high or low risk of developing a disease based on one or more patient-level data types. Similarity networks are computed for each patient pair and for each data type. In this example, high-risk patients are more strongly connected based on their clinical profile, which may capture age and smoking status, and metabolomic profile. Low-risk patients are more similar in their clinical and genomic profiles. The goal of netDx is to identify the input features predictive of high and low risk, and to accurately assign new patients to the correct class. Click here to expand this figure. Figure EV1. Detailed workflow of netDx predictor, indicating stochastic componentsFeature selection is shown on the left and uses only training samples (blue boxes). Top-scoring features are used to score the held-out test set (red boxes). Input networks are created and loaded into a GeneMANIA database. Feature selection is achieved by running 10 GeneMANIA queries, once per patient label or category. Note that GeneMANIA queries are run twice. They are first run on a database containing only training subjects and used to compute feature scores. The second time, they are run to assign labels to blind test samples (right). In this instance, the database contains both training and test samples, and is limited to feature-selected networks. Stochastic components of the model are indicated by a black asterisk. Download figure Download PowerPoint Click here to expand this figure. Figure EV2. Conceptual overview of the GeneMANIA algorithm, used by netDx for network integrationGeneMANIA is a network-based recommender system that ranks all nodes by similarity to an input query (or “positive” nodes). In netDx, the nodes are patients and GeneMANIA uses the set of input patient similarity networks (left). The patient ranking is achieved by a two-step process. First, input networks are integrated into a single association network via regularized regression that maximizes connectivity between nodes with the same label and reduces connectivity to other nodes (middle); this step computes network weights corresponding to predictive value for each network. Second, label propagation is applied to the integrated network starting with the query nodes (red), thereby ranking patients from most to least similar to the query (right). Download figure Download PowerPoint Input data design Each patient similarity network (PSN) is a feature, similar to a variable in a regression model. A PSN can be generated from any kind of patient data, using a pairwise patient similarity measure (Fig 1A and B). For example, gene expression profile similarity can be measured using Pearson correlation, while patient age similarity can be measured as normalized age similarity. A reasonable design is to define one similarity network per data type, such as a single network based on correlating the expression of all genes in the transcriptome, or a single network based on similarity of responses to a clinical questionnaire. If a data type is multivariate, defining a network for each individual variable will result in more interpretable output. However, this approach may lead to too many features generated (e.g., one network each for millions of SNPs), which increases computational resource requirements and risk of overfitting. Thus, as with any machine-learning task, there is a trade-off between interpretability and overfitting/scalability. To help address this problem for gene-oriented data (e.g., transcriptomics), we group gene-based measurements into biological pathways, which we assume capture relevant aspects of cellular and physiological processes underlying disease and normal phenotypes. This biological process-based design generates ~2,000 features from gene expression profiles containing over 20,000 genes, with one feature per pathway, but has the limitation that not all genes can be mapped to pathways. Selecting features informative for class prediction In machine learning, the feature selection step identifies the features (here, networks) with the highest generalizable predictive power. In the case of netDx, predictive features are identified for each class using feature selection. netDx is trained on samples from the class of interest, using cross-validation (Figs EV1 and EV2) and an established network integration algorithm (GeneMANIA; Mostafavi et al, 2008). When provided with a collection of PSNs, GeneMANIA takes a query of patients as input (“+” nodes) and solves a constrained regression problem on the network to maximize edges that connect query patients, that is, enriched for (+,+) interactions, relative to other edges in the network. The ideal network is one connecting all patients of the same class without any connections to other classes; an example would be a network connecting all treatment responders, and all non-responders, without edges between the two. The least useful network is one that connects patients from one class to patients from other classes, without connecting any patients within the same class. The query-driven regression weights each network, with higher weights reflecting a greater enrichment of (+,+) edges. netDx assesses the robustness of this feature selection by performing repeated GeneMANIA queries with different subsamples of the training samples. The netDx score for a given feature is the number of times that feature was assigned a positive score in a query across resampling rounds. This scoring process is repeated for each class. The classifier's sensitivity and specificity can be tuned by thresholding this score (for instance, features passing selection could be defined as those scoring ≥ 80%); a feature with a higher score achieves greater specificity and lower sensitivity. Similar to feature selection in other machine-learning algorithms, the output of this step is a set of features that can be integrated to produce a predictor for the patient class of interest. Class prediction using selected features Selected features are combined by averaging their similarity scores to produce a single, integrated network for each class. GeneMANIA runs label propagation on each integrated network outward from the query nodes (“+” nodes) to rank all other nodes in the network by similarity to the query (Fig EV2). For each class, netDx generates a GeneMANIA database consisting of networks passing feature selection; for example, one database for high-risk of lung cancer and another for low-risk of lung cancer. It then runs a single GeneMANIA query on this database for each class, using the training samples as the query. This is equivalent to the query, “given predictive networks for patient class L, rank all patients by similarity to known examples of L”. netDx then assigns test patients to the class for which it has the highest similarity (Mostafavi et al, 2008; Figs EV1 and EV2). netDx output netDx returns the set of selected features for each class, predicted labels for all test patients, and standard performance measures including the area under the receiver operating characteristic curve (AUROC), area under the precision-recall curve (AUPR), and accuracy (Fig 1C). Class-specific scores for each feature are returned; if pathway features are used, they are visualized using an enrichment map (Fig 1D; Merico et al, 2011a). A final integrated patient similarity network is generated by integrating all selected features. Inter- and intra-class separation is measured using average shortest path methods over the integrated PSN (Fig 1C), and the network is visualized to aid interpretation of the strength of class separation. Benchmarking performance by predicting binarized survival in cancer To assess the classification performance of netDx, we use an established cancer survival prediction benchmark available for four tumor types, using data from The Cancer Genome Atlas (TCGA; http://cancergenome.nih.gov/) via the TCGA PanCancer Survival Prediction project website of Yuan et al (2014a), https://www.synapse.org/#!Synapse:syn1710282). These tumor types have been thoroughly analyzed using eight machine-learning methods, which provide extensive performance results that we can compare to (Yuan et al, 2014a). Data are for renal clear cell carcinoma (The Cancer Genome Atlas, 2013; KIRC, N = 150 patients; Data ref: Yuan et al, 2014b), ovarian serous cystadenocarcinoma (The Cancer Genome Atlas, 2011; OV, N = 252, Data ref: Yuan et al, 2014c), glioblastoma multiforme (The Cancer Genome Atlas, 2008; GBM, N = 155, Data ref: Yuan et al, 2014d), and lung squamous cell carcinoma (The Cancer Genome Atlas, 2012a; LUSC, N = 77, Data ref: Yuan et al, 2014e). Data for a given tumor type include clinical variables (e.g., age and tumor grade); mRNA, miRNA, and protein expression; DNA methylation; and somatic copy number aberrations. Binarization of survival and format of clinical variables followed previous work (Yuan et al, 2014a; Data ref: Yuan et al, 2014f); comparison of data processing and predictor methods to that of the PanCancer Survival project is shown in Fig EV3 and Appendix Table S1. Click here to expand this figure. Figure EV3. Comparison of workflow for netDx and PanCancer Survival for benchmarkingThe netDx workflow is shown on the left and PanCancer Survival on the right. To compare netDx to other methods used by the PanCancer Survival project, we downloaded processed data from Synapse (see Data Availability section). To keep all steps except the predictor identical to the PanCancer Survival project, clinical variables were coded identically to the PanCancer Survival project (Appendix Table S1). Processed variables were then provided to the predictor. Univariate prefiltering was performed within the cross-validation framework of the respective predictor (Appendix Table S1); the variability across this process is shown in Appendix Fig S1. Download figure Download PowerPoint For each tumor type, we classified samples into high and low survival categories using a range of models, each with different combinations of input data, following (Yuan et al, 2014a) for comparability (Fig 2A). Each model was independently trained (80:20 train:test) and optimized. For a given train/test split, feature selection was performed over 10 resamplings of the training samples for each class, and features robustly identified in at least 9 out of 10 resamplings were used to classify test samples. This process was repeated for multiple random splits of train and test until performance measures stabilized (20 splits; Fig 2A). Performance for each model was measured as the average of test classification (mean AUROC) across the 20 splits. Optimization of each model included varying choice of similarity metric, whether features were defined at the level of entire data types or of individual variables (genes), and by whether or not imputation was used (Appendix Table S1–S3). Where features were defined at the level of individual genes, variables were prefiltered using lasso regression to reduce the number of features considered. Prefiltering supplements the robust feature selection step described above and is performed within the resampling loop, by creating features only using selected variables. Appendix Fig S1 shows an example of variability produced in prefiltered features. For similarity metrics, we tested normalized similarity, Pearson correlation with and without exponential scaling, radial basis function, and Euclidean-distance-based similarity with exponential scaling. In total, 40 different models were trained, 9–11 per cancer type (Dataset EV1). Figure 2. Performance benchmarking with PanCancer Survival data Method. Various combinations of patient data types were provided as input to netDx, to predict binary survival (Good/Poor). Performance was compared for renal clear cell carcinoma (KIRC, N = 150 patients), ovarian cystadenocarcinoma (OV, N = 252), lung squamous carcinoma (LUSC, N = 77), and glioblastoma multiforme (GBM, N = 155). Average performance of netDx compared to other machine-learning methods. Each panel shows data for one tumor type, and each boxplot shows mean AUROC (20 train/test splits) for a given machine-learning method across different tested combinations of input data (Appendix Table S1–S3). Boxplot center indicates median; box bounds indicate 25th and 75th percentile, and whiskers mark 1.5 times the interquartile range. Dots indicate points that fall outside this range. netDx is shown in pink. Statistical significance is computed using a one-sided Wilcoxon-Mann-Whitney comparing netDx to all other methods combined; this was done to establish strictly higher performance of netDx relative to other methods. Bottom: Yellow box: As a reference point, Kaplan–Meier curves and hazard ratios are shown for predicted samples from a representative KIRC split (split with AUROC closest to the mean AUROC across 20 splits). The light shaded pink and blue areas in the Kaplan–Meier curve indicate 95% confidence intervals based on log-hazard; P-value from log-rank test. Error bars for the hazard ratio indicate 95% confidence intervals; P-value from Cox proportional hazards model. Download figure Download PowerPoint We use the AUROC to measure performance as this was the metric reported by the PanCancer Survival project (Yuan et al, 2014a; Appendix Table S3). Information on the exact samples used for the 10 train/test splits used by Yuan et al is not available, thus we used new random splits, but used more splits (n = 20) until we reached a stable AUROC estimate. For all tumor types, netDx demonstrates performance consistently better than, or at par with, other machine-learning methods (Fig 2B; Dataset EV1). Average netDx performance is significantly higher than that for all other methods for three of the tumors (one-tailed WMW to test strictly higher netDx performance; KIRC: P = 0.03; OV: P = 0.02; GBM: P = 0.02) and is not significant for the fourth (LUSC: P = 0.07). Furthermore, the top-performing netDx model outperforms all eight tested machine-learning algorithms for kidney, ovarian, and brain cancer (Fig 2B, Appendix Table S1), and outperforms all but one outlier model for lung cancer (netDx best = 0.72, Yuan et al second best = 0.71). Pairwise comparison of netDx broken down by machine-learning method shows that netDx has a significantly better median AUROC score in most (75%) cases (Appendix Table S2). Performance statistics reported by Yuan et al were the best performing models out of 320 tested for different data combinations with eight different machine-learning methods: diagonal discriminant analysis; K-nearest neighbor; discriminant analysis; logistic regression; nearest centroid; partial least squares; random forest; and support vector machine. Thus, netDx can perform as well as, or better than, a diverse panel of machine-learning methods. Pathway-level feature selection identifies cellular processes predictive of clinical condition Creating a single feature per data type identifies the general predictive value of that data layer but, without further work, does not provide insight into which genes or cellular processes are useful for classification. To better understand disease mechanisms, netDx supports the ability to group gene-level measures into pathways (gene sets) so that the predictive value of pathway features can be measured. To illustrate this ability, we classified breast tumors as being of the Luminal A subtype or not, using tumor-derived gene expression (N = 348 patients; 154 Luminal A, 194 non-Luminal A; The Cancer Genome Atlas, 2012b; Data ref: The Cancer Genome Atlas, 2012c,d). Gene expression was used as input, and features were defined at the level of curated pathways of cellular processes (i.e., 2,119 patient similarity networks, one per pathway; Data ref: Merico et al, 2011b). We split the data into train and test groups in an 80:20 ratio and ran netDx as described in the previous section to classify Luminal A vs. “other”. This process was repeated with 100 train/test splits to achieve stable pathway-level scores (Appendix Fig S2). Consistently predictive pathways, representing a high-confidence set of features, are visualized as an EnrichmentMap (Fig 3), with each pathway associated with a consistency score (highest netDx feature selection score in ≥ 70% of the 100 splits). Figure 3. Pathway-level feature selection in breast cancer and asthma netDx performance for binary classification of breast tumor as Luminal A subtype from tumor-derived gene expression (N = 384 patients). Pathways feature-selected by netDx in predicting Luminal A status. Nodes are pathways, and edges indicate shared genes. Nodes are colored by highest netDx score consistently achieved out of a maximum possible of 10, in ≥ 70% of 100 train/test splits. Themes identified by AutoAnnotate (Merico et al, 2011a; Kucera et al, 2016). Integrated patient similarity network. Nodes represent patients, and edges represent average similarity computed from pathways that scored 10 out of 10 in all splits. Nodes are colored by tumor type. Edges with weight < 0.7 were excluded, and the top 20% of edges per node were retained. The resulting network was visualized in Cytoscape (spring-embedded layout, spring strength = 5). Correlation of top-scoring pathway features (represented as the first three principal components of pathway-specific gene expression) with tumor type (Spearman's correlation). Table cells are colored by sign and magnitude of correlation (blue: Spearman corr. > 0; red, corr. < 0). Circled letters correspond to detailed panels on the right. Right: Projections of patient-level gene expression in feature-selected pathways onto the first two principal components (individual dots indicate patients). Points are colored by survival class. Decision boundaries were calculated using logistic regression on scatterplot data. Selected features for asthma case status in the case of asthma case/control prediction (N = 97 cases; N = 97 controls). Legend as in (B). Download figure Download PowerPoint Classification performance was high, with an average (SD) AUROC of 0.97 ± 0.01, average AUPR of 0.92 ± 0.02, and average accuracy of 89 ± 3% (100 train/test splits; Fig 3A). Performance for pathway-level features is slightly, but significantly, better than when gene expression is provided as a single feature (mean AUROC for single