TITLE: Machine learning approach yields epigenetic biomarkers of food allergy: A novel 13-gene signature to diagnose clinical reactivity
AUTHORS: Ayush Alag

[DESCRIPTION]
<div class = "text-blocks"><div class = "text-block">Current laboratory tests have a less than 50% accuracy in distinguishing between people who have food allergies (FA) and those who are merely sensitized to foods, resulting in the use of expensive and potentially dangerous Oral Food Challenges. Our study presents a purely-computational machine learning approach, conducted using DNA Methylation (DNAm) data, to accurately diagnose food allergies and find genes that are strong biomarkers of the disease. </div><div class = "text-block">We built two deep learning classifiers with twelve CpG-input features each that achieved perfect accuracy and an AUROC of 1 on the completely hidden cross-validation cohort. In addition, 24 additional classifiers were created that each had an average cross-validation accuracy of 98.35%. These 26 classifiers yielded a total of 18 unique CpGs, which mapped to 13 genes that are strong epigenetic biomarkers of FA.</div><div class = "text-block">Biological enrichment on the 13-gene signature yielded new insights. Notably, our FA-discriminating genes were strongly associated with the immune system, which helps validate our findings. Seven of the 13 genes overlapped with previous food-allergy and DNAm studies.</div><div class = "text-block">Previous studies have also created a perfect classifier for this dataset, but they used a 96-CpG input feature set built on both data-driven and a priori biological insights. Our study is an improvement on previous work because it maintains a perfect classification accuracy using only 18 highly discriminating CpGs (0.005% of the total available features). In machine learning, simpler models, as used in our study, are preferred over more complex ones (all other things being equal).</div><div class = "text-block">In addition, our completely data-driven approach eliminates the need for \textit{a priori} information and allows for generalizability to DNAm classification problems in other disease areas, which may result in novel gene associations or accurate diagnostic tests for those diseases.</div></div>

[STEPS]
?. Download the Java code from BitBucket URL https://bitbucket.org/ayushalag/allergezy_plosone_src/src/master/
?. Create an Eclipse project using the src Java files downloaded. Download weka jar files from https://www.cs.waikato.ac.nz/ml/weka/downloading.html . The Eclipse project should have the src files with the Java code downloaded in the previous step. Create a directory extjar and put the five reference jar files from Weka. Download the data jar file and place it within the main project. This directory should have the input and output folders with many sub folders. The doc folder is optional and creates the generated JavaDoc for the project. JavaDoc can be generated by going to Eclipse --> Project --> Generate JavaDoc.
?. Next, we will validate that the CpG data that has already been separated into Food Allergy, Sensitized, and Non-Allergic samples have been downloaded correctly. Expand the data/input/gse59999 folder and verify that there are three files:allergy-59999-datasets.txt 215 MB contains 30 rows by 405,659 columnsnonallergic-59999-datasets.txt 98.9 MB contains 14 rows by 405,659 columnssensitized-59999-datasets.txt  215 MB contains 30 rows by 405,659 columnsFrom within Eclipse, run the main program for com.allergezy.fa.util.FileUtil. You should see the output shown below. Note the number of rows and columns and the validation check.The research was conducted using a dataset found in the Gene Expression Omnibus (GEO)under accession id GSE59999. The 71 patient samples in this dataset consisted of 29 patients with FA (tested positive on OFCs), 29 patients who were sensitized but not food-allergic, and 13 patients who were neither sensitized nor allergic. Each sample is associated with a DNAm profile that was taken from mononuclear blood cells and consisted of normalized Beta values ranging from 0 (completely unmethylated) to 1 (fully methylated) at 405,658 CpG islands distributed across the genome.
?. In this step, we will create  8-fold test/train/hidden samples. The 58 samples are randomly split into three datasets: 40 samples for training, 10 samples for testing, and 8 completely hidden samples for cross-validation. Half of the samples in each of the three datasets were children with food allergies and the other half were those that were food-sensitized but not food-allergic. To avoid potential bias, we created eight random splits,  shuffling the samples across the three datasets each time such that each of the 58 samples was in the hidden dataset at least once. The logic is implemented in com.allergezy.fa.kfold.KFoldDatasetGenerator. Run the main method in this class to see an output similar to below.This program produces an output file data/input/gse59999/k401008/gse59999-kfold-cases.txt as shown below.The file data/input/gse59999/k401008/gse59999-kfold-cases-final.txt is the final version of this 8-fold split that is used in the rest of the study.Below is the content of the data/input/gse59999/k401008/gse59999-kfold-cases-final.txt file.Each sample appears at least once in the hidden set. In addition, for every case there are 40 training samples. There is an equal number of food-allergic and food-sensitized samples in each dataset and in each case.To verify that this file can be accessed, run the main method for com.allergezy.fa.kfold.KFoldDatasetInfo. This file follows the Singleton design pattern and provides information on which samples are in the training, test, and hidden datasets for each of the eight cases. You should see the following output.The distribution of the samples can be seen in the following table.
?. DNAm datasets are characterized as having a small number of samples but a very high number of feature dimensions (HDLSS). To prevent overfitting and increase generalization, it is important to condense the feature list relative to the number of samples available. Computationally, it is  very expensive to evaluate the more than 400K CpG features individually. Therefore, in order to limit the evaluation size and begin with a list of potentially highly-relevant CpG points, we used the NCBI GEO2R  tool to obtain a prioritized list of CpG features differentially expressed across the two groups. For each of the eight independent cases, we split the forty training samples into two cohorts, one with allergic patients and the other with  sensitized patients.The GEO2R tool was used to derive eight lists of 100 CpGs each, one for each of the eight cases. The screenshot below shows the use of the GEO2R tool to create two cohorts of 20 samples each for the first case. Clicking on The full signature for the eight cases is in data/input/gse59999/geo2r/Geo2R-cpgs.txt file. Below are the first twenty lines of the file data/input/gse59999/geo2r/Geo2R-cpgs.txt file. This will be used in the next few steps.
?. Next, the eight lists were combined together, which resulted in 636 unique CpGs, the count being less than 800 as some of the CpGs were repeated across the eight cases.To map the CpGs to genes, we use the output from GEO2R contained in the file data/input/gse59999/geo2r/Geo2r-gse5999-full-sign.txt. The Singleton class com.allergezy.fa.geo2r.CPGAnnotations maps a CpG to genes using this file. Run the main method in the class com.allergezy.fa.geo2r.CPGAnnotations. You should see the output similar to the screen shot below.The class com.allergezy.fa.geo2r.Geo2rCpgInfo computes the overlap between the CpGs from the GEO2R output.  Running the main method produces the output shown below.The file  data/input/gse59999/geo2r/geo2RfeatureScoring.txt contains the result of this analysis. This is summarized as shown belowNote that this ranking really has no material significance in our methodology, since we evaluated all the unique 636 CpGs independently. It does provide some insights into features highlighted by GEO2R and those that appeared in our final CpG signature. The 20% overlap we found between the CpGs across the eight lists seems to suggest that the examples do have an effect on the CpGs that are differentially expressed between the two cohorts (when GEO2R is used). Thus, averaging the results across the eight independent runs, which is performed throughout the rest of the paper, should help in avoiding any potential biases, due to the distribution of the samples across the training, test, and hidden datasets.
?. Next, we will create smaller dataset files that contain the machine learning examples with 636 CpG values as columns. com.allergezy.fa.dataset.DatasetCreatorForEachCase -- generates the file that has the examples for each case. The output from the program is shown below.In the input/gse59999/ml-data there are eight files case-0.txt, case-1.txt, and so on for the eight folds.Attached is a screen shot of a part of the first file case-0.txt.
?. For each case, and for each of the 636 unique CpGs, we built four different machine learning models: Decision Trees (DT), Logistic Regression (LR), Radial Basis Function (RBF), and a Multi-Layer Perceptron (MLP), a deep learning network with two hidden layers of ten nodes each. Each predictive model only used one CpG, and was built on the training data. For each case and each feature, the classifier with the highest testing accuracy was selected. Finally, the average hidden data (cross-validation set) accuracy across the eight independent cases was computed. 636 of these accuracy scores (one for each CpG) and 20,352 models (8 independent cases x 636 features x 4 classifiers) were created in total using this process, for the single input case.com.allergezy.fa.feature.SingleFeatureScoringRunner is the class that runs this computation. A few class to notecom.allergezy.fa.weka.WekaHelper interfaces with the Weka APIs for creating modelsLogic for writing details as part of the learning are SummaryRunResultWriterImpl -- summary of resultsDetailedRunResultWriterImpl -- detailed output of each model, BestModelForCasesResultWriterImpl - best model.Run the main method com.allergezy.fa.feature.SingleFeatureScoringRunner. You should see an output similar to This program writes output to data/output/singlefeature/kfoldresult. Subdirectories case0, case1, …, case7 has results for the eight folds. Within each directory are four sub directories: DECISION_TREE, LOGISTIC_REGRESSION, MLP, RBF. Within each directory are detailed results for each model and results -- featscore0.txt, featscore1.txt, ….., The class com.allergezy.fa.feature.BestFeatureIdentifierUsingHiddenTestScores computes the average score for each feature across the eight cases. The following is a screen shot of the output file featureScoring.txt com.allergezy.fa.feature.BestFeatureIdentifierUsingHiddenTestScoresOutput from the consoleOutput file data/output/singlefeature/kfoldresult.bestFeatureScoresCpg.txtAttached is the screenshot of the final file, sorted by average hidden test accuracy.The following graph summarizes the results.The top CpGs with associated genes are
?. We sought to increase the number of features used by the classifier while still avoiding over-fitting and encouraging generalization.  Therefore, we selected the first eighteen CpGs, based on the accuracy scores computed above, and combined them two at a time, followed by three at a time, and so on until combinations of twelve were reached. Given the large number of potential combinations, each model was limited to a small subset of strong CpG-lists, to which a new input feature was added. On an average, we tried about 200 unique combinations for a given number of input features. Again, each unique input feature combination set was run 4 x 8 = 32 times, to account for the four different classifier methods and 8 independent sample-distribution "cases".The class com.allergezy.fa.multifeature.casegenerator.MultiFeatureCasesGenerator is used to create next set of cases to be tried.
?. For a given number of input features, the models were ranked using their accuracy scores on hidden data (described above). Odd numbers of models, starting from 1 to 101, were combined together using a simple voting scheme, i.e., each model independently predicted whether a sample was classified as FA or sensitized and the final prediction was the majority of predictions made across the different models.The class com.allergezy.fa.kfold.verify.VerifyWekaDatasetRunner runs the set of cases specified in the data/output/multifeature/setup/summary.txt file. A screen shot of an example file is shown next.This class has a visitor BestModelForCasesResultWriterImpl configured to calculate majority prediction.Iterate different combinations of CpG features using the summary.txt file as input, run the results, select the best models and try new combinations adding one feature at a time. The results for various combinations are in the directory data/output/multfeature/result/Final.
?. The file com.allergezy.fa.rawdata.RawCpGDataExtractor can extract raw CpG values for a CpG across the samples and can be used for generating data plots. Here is an example file.com.allergezy.fa.featureanalyzer.CpGFrequencyFeatureAnalyzerAnalyzes the frequency of various CpG across the summary file
?. Gene set analysis can provide biological context, as well as insights into disease mechanisms and possible treatments. Biological enrichment was performed by applying Illumina’s BaseSpace Correlation Engine to the 13-gene list. To understand these genes better we identified  tissues where the genes were expressed, found associated biological pathways, used gene ontology concepts to identify functionally-related gene sets, connected the 13-gene list to the Broad positional gene sets, and connected the gene signature to protein families.The 13-gene signature was imported into Illumina BaseSpace Correlation Engine
?. GO provides a framework for analyzing a group of co-expressed genes and provides insights into whether particular genes are involved in diseases. Similar to other ontologies, GO consists of terms and relationships represented in a directed acylic graph. GO focuses on using the terms in the ontology to describe gene functions. In GO, gene function is classified along three categories: molecular functions, cellular components, and biological processes.The GO Enrichment Analysis Tool was used to find GO terms associated with the 13-gene signature. Enter the following gene list as shown RNF213SARSZNF252TMED10PABCF2TIMP2MAFKCD7PANX1CTBP2SLC24A2ARID5BKIF13BFAM190BAfter computation the results are shown below.Follow the same procedure for the other two options: molecular function and cellular component, as shown below.The results from this analaysis is shown below.Next, the Generic Gene Ontology (GO) Term Mapper tool from Princeton University to map granular GO annotations to a higher-level set of terms, thus providing a broad set of categories. Make sure to select HomoSapiens for the Organism as shownThe results are shown below.The results are summarized in the following table.The resulting GO terms areGO:0030705GO:0007165GO:0048856GO:0006810GO:0009058GO:0030154 GO:0034641 GO:0007267 GO:0002376 GO:0006464GO:0022607GO:0065003GO:0048646GO:0006950GO:0016192GO:0008283GO:0055085 GO:0009056GO:0000003GO:0050877GO:0006605GO:0042592GO:0006520GO:0030198GO:0007010GO:0006412GO:0000278GO:0044281GO:0044403GO:0048870GO:0000902GO:0006399 GO:0051276GO:0007568GO:0040011  GO:0040007GO:0007049 Next, the REVIGO, an online tool that  summarizes and visualizes lists of gene ontology terms, to find a representative set of terms using a clustering algorithm.The frequency and uniqueness of the GO terms can be seen below.Click on the TreeMap tab.Download the csv file at the bottom of the page. You should see a csv file with the following content.This can be summarized by the following table.This same data can be represented in a treemap.The GO terms can be visualized using NaviGo.Results are shown below.Click on Open BP Visualizer to see the network of terms.Single GO terms can also be visualized, as shown below.See https://link.springer.com/protocol/10.1007/978-1-4939-3743-1_15 for a list of GO visualization tools.