TITLE: Reproducibility for Thistlethwaite et al. 2020
AUTHORS: Lillian Thistlethwaite, xiqi.li2 , Varduhipetrosyan 

[DESCRIPTION]
<div class = "text-blocks"><div class = "text-block">We consider the following algorithmic problem that arises in transcriptomics, metabolomics and other fields: given a weighted graph and a subset of its nodes, find subsets of perturbed variables that show significant connectedness within the graph. While solutions to this problem may be discovered by devising a variety of scoring functions, statistically rigorous discovery is obstructed by the high computational cost of permutation testing, required to establish p-values. In this work, we develop CTD, a novel information-theoretic method that evaluates the connectedness of perturbation signatures in a network context and assigns an upper bounds on their p-values without use of permutation testing. We apply CTD to interpret multi-metabolite perturbations due to inborn errors of metabolism and multi-gene perturbations associated with breast cancer in the context of disease-specific Gaussian Markov Random Field networks learned directly from respective molecular profiling data. </div></div>

[STEPS]
?. [Prepare dataset]
The CTD R package version 1.0.0 was used.
?. [Fig 1: Plot individual metabolomics profiles onto biochemical pathway maps.]
The CTDext R package version 1.0.0 was used. The CTDext R package holds many pre-computed files necessary to reproduce some results from Thistlethwaite et al. 2020, as well as includes some additional pathway visualization and plotting features.The CTDext R package can be downloaded using the devtools::install_github() function as follows:install_github("BRL-BCM/CTDext")
?. [Tuning parameters used in this analysis]
Before we start the analysis, we need to discuss some tuning parameters.Tuning parameters (defaults)p0: the probability uniformly distributed to all metabolites. default is 0.1.p1: the probability distributed preferentially based on edge weights and connectedness patterns in a network, G.default is 0.9.thresholdDiff: the probability at which the probability diffusion algorithm truncates. default = 0.01.kmx: the number of top-perturbed (up or down) metabolites considered in the network-based interpretation. default in this analysis = 15.
?. [Figure 3: Sensitivity and Specificity of CTD between 3 Network Learning Paradigms]
How to Replicate Figure 3In this result, we calculate the ROC-AUCs for all 5 CTD-defined diagnostic models. We also plot CTD scores across diagnostic categories as barplots to show the sensitivity and specificity of the models under three different network learning paradigms:i)   latent embedding + network pruning ("ind")ii)  latent embedding + no network pruning ("noPruning")iii) no latent embedding or network pruning ("noLatent")
?. [Figure 3: Sensitivity and Specificity of CTD between 3 Network Learning Paradigms]
The computational work in this step is required for downstream steps. In this analysis step, we estimate the probability and significance of patient-specific metabolite perturbations against different disease-specific network contexts using CTD.The CTDext package provides precomputed node rankings derived from all 5 disease networks in Thistlethwaite et al. 2020 in order to save time. You can find out how to precompute node rankings under a different Protocol: Precompute Node Rankings (DOI: dx.doi.org/10.17504/protocols.io.bkecktaw).You can find out how to learn partial correlation disease-specific networks under a different Protocol: Learn Partial Correlation Disease-Specific Networks (DOI: dx.doi.org/10.17504/protocols.io.bk7xkzpn).R package versioning:CTD v 1.0.0CTDext v 1.0.0R.utils v 2.9.2pROC v 1.16.2ggplot2 v 3.3.2gridExtra v 2.3.0
?. [Figure 3: Sensitivity and Specificity of CTD between 3 Network Learning Paradigms]
Now, we can visualize the results that generates Figure 3.
?. [Figure S1: Highly connected patient-specific modules called by CTD.]
How to Replicate Figure S1In this result, we interpreted disease signatures from selected individual patients representing each disease category. We also visualize selected patient's most connected modules in the relevant disease-specific network context.R package versioning:CTD v 1.0.0CTDext v 1.0.0R.utils v 2.9.2
?. [Power analysis]
How to Replicate Figure 4.In this result, we performed a power analysis on three simulated networks of size 50, with varying levels of connectedness.Network 1: 10% connectedNetwork 2: 30% connectedNetwork 3: 60% connectedThe simulation is broken up into 4 steps: 1. Build networks.2. Get number of edges connecting subsets of size 5.3. Use CTD to get subset probabilities by enumerating over all >2 million outcomes of choose(50, 5). This enumeration will allow you to establish a ground truth p-values by determining the distribution of probabilities assigned by CTD over all subset outcomes of size 5 in each of the three 50 node networks.4. For 2,500 node subsets, select equally amongst the number of edges between node subsets to assure that you get a variable level of connected subsets. Then, draw 2,000 permutations to establish a permutation-based p-value estimate.R package versioning:CTD v 1.0.0CTDext v 1.0.0igraph v 1.2.5R.utils v 2.9.2gmp v 0.6.0ggplot2 v 3.3.2
?. [Power analysis]
Build simulation networks: Build 3 undirected networks with 50 nodes and assign edge weights of 1 with a varying level of connectedness between them.
?. [Power analysis]
Establish ground truth: Count the number of modules of size 5 with parameter r number of edges between them. Note: We run 10,000 subsets at a time on a computing cluster, a form of parallelization, which significantly speeds up this step compared to enumerating using serial code. The parallel implementation is pasted below, following by a PBS job scheduler launcher script, and a wrapper script which automatically launches the PBS scripts.Each job running 10,000 subsets took about 1-1.5 minutes, and 100 jobs were running at a given time. It took about 5-7 minutes to run all 212 jobs. Collating the job output .RData files into one .RData files took another 5 minutes.
?. [Power analysis]
Run CTD on all possible node sets of size 5 for each network, collect p-values for each encoding algorithm (single-node). Note: We use the same approach to enumeration as we did in Step 6.2, where we launch individual jobs on a cluster that performs the CTD probability estimation for 10,000 subsets at a time. For over 2 million subsets of size 5 in a network of size 50, this amounted to 212 jobs per network. For 3 networks, this amounted to 212*3= 636 jobs.Each job took a variable amount of time, depending on how connected the 10,000 subsets being processed were and how efficient the cluster node was at the time. Job execution times ranged from 32 minutes to ~6.5 hours, and 100 jobs were running at any given time. 636 jobs were launched in total (212 jobs for 3 networks), resulting in an approximate execution time of about ~21 hours.
?. [Power analysis]
Get permutation-based pvalues for 2,500 node sets you calculated a CTD upper bounds estimate for.
?. [Table 5: CTD as a feature selection method]
How to Replicate Table 5In this result, we used CTD as a feature selection method and a covariate in 5 different disease-specific Partial Least Square (PLS) regression models.Here we compare CTD as a feature selection method to a basic top z-score feature selection method, and the FSFCN algorithm using three different network clustering algorithms: the Greedy Modularity Optimization, InfoMap, and WalkTrap.R package versioning:CTD v 1.0.0CTDext v 1.0.0igraph v 1.2.5pls v 2.7.3pROC v 1.16.2caret v 6.0.86
?. [Table 5: CTD as a feature selection method]
Define the FCFSN algorithm.
?. [Table 5: CTD as a feature selection method]
Apply and compare CTD as a feature selection method.
?. [Table 5: CTD as a feature selection method]
Estimate and plot the variable importance of every feature selected in each feature selection method.
?. [Table 5: CTD as a feature selection method]
Calculate the area under the curve (AUC) for each partial least sqaures (PLS) regression model.
?. [TCGA: topological pathway enrichment analysis]
Topological Pathway Enrichment Analysis for Breast Cancer Subtypes TCGA data was downloaded as shared through the UCSC via the Xena browser: HiSeqV2:https://xenabrowser.net/datapages/?dataset=TCGA.BRCA.sampleMap%2FHiSeqV2&host=https%3A%2F%2Ftcga.xenahubs.net&removeHub=https%3A%2F%2Fxena.treehouse.gi.ucsc.edu%3A443BRCA_clinicalMatrix:https://xenabrowser.net/datapages/?dataset=TCGA.BRCA.sampleMap%2FBRCA_clinicalMatrix&host=https%3A%2F%2Ftcga.xenahubs.net&removeHub=https%3A%2F%2Fxena.treehouse.gi.ucsc.edu%3A443Using the clinicalMatrix, we can select samples from HiSeqV2 that are breast cancer samples and identify the specific subtype of breast cancer for each sample.An important note: Since CePa, SPIA and PRS use permutation testing to estimate p-values, we noticed slightly different p-value estimates between runs. The same general results are reproduced across runs, but p-values outputted by these methods will differ slightly.R package versioning:CTD v 1.0.0CTDext v 1.0.0graphics (in base R version 4.0.2)dplyr v 1.0.2DESeq2 v 1.28.1graphite v 1.34.0igraph v 1.2.5graph v 1.66.0CePa v 0.7.0huge v 1.3.4.1org.Hs.eg.db v 3.11.4EnrichmentBrowser v 2.18.0ToPASeq v 1.22.0SPIA v 2.40.0DEGraph v 1.40.0fgsea v 1.14.0reshape2 v 1.4.4ggplot2 v 3.3.2
?. [TCGA: topological pathway enrichment analysis]
Normalize TGCA Breast Cancer and Normal Samples with the DESeq2 R package. Secondly, load the pathway knowledge using the graphite R package and build a pathway catalogue R object.
?. [TCGA: topological pathway enrichment analysis]
CTD for each subtype
?. [TCGA: topological pathway enrichment analysis]
CePa for each subtype
?. [TCGA: topological pathway enrichment analysis]
PRS for each subtype.If you are unhappy with the amount of time it takes PRS to run, set the nperm parameter to 100 instead of 1000. This will only allow a minimum p-value of 0.01, whereas 1000 permutations can estimate p-values at the resolution of 0.001.
?. [TCGA: topological pathway enrichment analysis]
SPIA for each subtype
?. [TCGA: topological pathway enrichment analysis]
DEGraph for each subtype
?. [TCGA: topological pathway enrichment analysis]
ORA for each subtype
?. [TCGA: topological pathway enrichment analysis]
GSEA for each subtype
?. [TCGA: topological pathway enrichment analysis]
Combine all pathway enrichment method's result by subtype
?. [TCGA: topological pathway enrichment analysis]
Generate Figure 5
?. [Set-based pathway enrichment methods for metabolomics]
Two set-based pathway enrichment methods are applied to the metabolomics data from Miller et al 2015, for 5 diagnostic categories (citrullinemia, maple syrup urine disease, methylmalonic aciduria, propionic aciduria, phenylketonuria).R package versioning:CTD v 1.0.0CTDext v 1.0.0
?. [Set-based pathway enrichment methods for metabolomics]
Over-representation analysis (ORA)
?. [Set-based pathway enrichment methods for metabolomics]
Metabolite set enrichment analysis (MSEA)
?. [Figure 7: Node set probability is based on network context.]
In this result, we show that the probability assigned to a node set varies significantly based on the network's specificity. For example, the top 5 perturbed metabolites in patients with citrullinemia are assigned the highest probability in the Citrullinemia disease context, compared to the top 5 metabolites perturbed in methylmalonic aciduria or phenylketonuria.Note, because surrogate profiles (generated by data.surrogateProfiles()) are generated using random draws from the standard normal distribution, results will vary when you run the code in this section, but the same general trends will be consistent.R package versioning:CTD v 1.0.0CTDext v 1.0.0huge v 1.3.4.1ggplot2 v 3.3.2
?. [Figure 7: Node set probability is based on network context.]
Separate data into cohorts
?. [Figure 7: Node set probability is based on network context.]
Learn disease-specific network folds.
?. [Figure 7: Node set probability is based on network context.]
Prune disease+control network folds to output disease-specific network folds.
?. [Figure 7: Node set probability is based on network context.]
This is the main driver script for this experiment. Now that we have all disease-specific network folds learned, we can calculate the probabilities of the top 5 perturbed metabolites for each disease state (citrullinemia, methylmalonic aciduria, phenylketonuria) in each network context.
?. [Figure 7: Node set probability is based on network context.]
Plot the results.