Package: genoml2
GenoMLUpdated 2 March 2021: Latest Release on pip! v1.0.0.beta.11How to Get Started with GenoMLIntroductionGenoML is an Automated Machine Learning (AutoML) for genomics data. In general, use a Linux or Mac with Python >3.5 for best results.Thisrepositoryandpip packageare under active development!This README is a brief look into how to structure arguments and what arguments are available at each phase for the GenoML CLI.Installing + Downloading Example DataInstall this repository directly from GitHub (from source; master branch)git clone https://github.com/GenoML/genoml2.gitInstall using pip or upgrade using pippip install genoml2ORpip install genoml2 --upgradeTo install theexamples/directory (~315 KB), you can use SVN (pre-installed on most Macs)svn export https://github.com/GenoML/genoml2.git/trunk/examplesNote: When you pip install this package, the examples/ folder is also downloaded! However, if  you still want to download the directory and SVN is not pre-installed, you can download it via Homebrew if you have that installed usingbrew install svnTable of Contents0. (OPTIONAL) How to Set Up a Virtual Environment via Conda1. Munging with GenoML2. Training with GenoML3. Tuning with GenoML4. Testing/Validating with GenoML5. Experimental Features0. [OPTIONAL] How to Set Up a Virtual Environment via CondaYou can create a virtual environment to run GenoML, if you prefer.
If you already have theAnaconda Distribution, this is fairly simple.To create and activate a virtual environment:# To create a virtual environmentcondacreate-nGenoMLpython=3.7# To activate a virtual environmentcondaactivateGenoML# To install requirements via pippipinstall-rrequirements.txt# If issues installing xgboost from requirements - (3 options)# use Homebrew to# xcode-select --install# brew install gcc@7# conda install -c conda-forge xgboost# pip install xgboost==0.90# If issues installing umap# pip install umap-learn## MISC# To deactivate the virtual environment# conda deactivate GenoML# To delete your virtual environment# conda env remove -n GenoMLTo install the GenoML in the user's path in a virtual environment, you can do the following:# Install the package at this pathpipinstall.# MISC# To save out the environment requirements to a .txt file# pip freeze > requirements.txt# Removing a conda virtualenv# conda remove --name GenoML --all1. Munging with GenoMLMunging with GenoML will, at minimum, do the following:Prune your genotypes using PLINK v1.9 (if--genoflag is used)Impute per column using median or mean (can be changed with the--imputeflag)Z-scaling of features and removing columns with a std dev = 0Requiredarguments for GenoML munging are--prefixand--phenodata: Is the datacontinuousordiscrete?method: Do you want to usesupervisedorunsupervisedmachine learning?(unsupervised currently under development)mode:  would you like tomunge,train,tune, ortestyour model?--prefix: Where would you like your outputs to be saved?--pheno: Where is your phenotype file? This file only has 2 columns, ID in one, and PHENO in the other (0 for controls and 1 for cases)Be sure to have your files formatted the same as the examples, key points being:0=controls and 1=casein your phenotype fileYour phenotype file consistingonlyof the "ID" and "PHENO" columnsYour sample IDs matching across all filesYour sample IDs not consisting with only integers (add a prefix or suffix to all sample IDs ensuring they are alphanumeric if this is the case prior to running GenoML)Note:The following examples are for discrete data, but if you substitute following commands withcontinuousinstead of discrete, you can preprocess your continuous data!If you would like to munge just with genotypes (in PLINK binary format), the simplest command is the following:# Running GenoML munging on discrete data using PLINK genotype binary files and a phenotype filegenomldiscretesupervisedmunge\--prefixoutputs/test_discrete_geno\--genoexamples/discrete/training\--phenoexamples/discrete/training_pheno.csvIf you would like to control the pruning stringency in genotypes:# Running GenoML munging on discrete data using PLINK genotype binary files and a phenotype filegenomldiscretesupervisedmunge\--prefixoutputs/test_discrete_geno\--genoexamples/discrete/training\--r2_cutoff0.3\--phenoexamples/discrete/training_pheno.csvYou can choose to skip pruning your SNPs at this stage by changing the--skip_pruneflag to "yes" (default is "no")# Running GenoML munging on discrete data using PLINK genotype binary files and a phenotype filegenomldiscretesupervisedmunge\--prefixoutputs/test_discrete_geno\--genoexamples/discrete/training\--skip_pruneyes\--phenoexamples/discrete/training_pheno.csvYou can choose to impute onmeanormedianby modifying the--imputeflag, like so(default is median):# Running GenoML munging on discrete data using PLINK genotype binary files and a phenotype file and specifying imputegenomldiscretesupervisedmunge\--prefixoutputs/test_discrete_geno\--genoexamples/discrete/training\--phenoexamples/discrete/training_pheno.csv\--imputemeanIf you suspect collinear variables, and think this will be a problem for training the model moving forward, you can usevariance inflation factor(VIF) filtering:# Running GenoML munging on discrete data using PLINK genotype binary files and a phenotype file while using VIF to remove multicollinearitygenomldiscretesupervisedmunge\--prefixoutputs/test_discrete_geno\--genoexamples/discrete/training\--phenoexamples/discrete/training_pheno.csv\--vif5\--iter1The--vifflag specifies the VIF threshold you would like to use (5 is recommended)The number of iterations you'd like to run can be modified with the--iterflag (if you have or anticipate many collinear variables, it's a good idea to increase the iterations)Well, what if you had GWAS summary statistics handy, and would like to just use the same SNPs outlined in that file? You can do so by running the following:# Running GenoML munging on discrete data using PLINK genotype binary files, a phenotype file, and a GWAS summary statistics filegenomldiscretesupervisedmunge\--prefixoutputs/test_discrete_geno\--genoexamples/discrete/training\--phenoexamples/discrete/training_pheno.csv\--gwasexamples/discrete/example_GWAS.csvNote:When using the GWAS flag, the PLINK binaries will be pruned to include matching SNPs to the GWAS file....and if you wanted to add a p-value cut-off...# Running GenoML munging on discrete data using PLINK genotype binary files, a phenotype file, and a GWAS summary statistics file with a p-value cut-offgenomldiscretesupervisedmunge\--prefixoutputs/test_discrete_geno\--genoexamples/discrete/training\--phenoexamples/discrete/training_pheno.csv\--gwasexamples/discrete/example_GWAS.csv\--p0.01Do you have additional data you would like to incorporate? Perhaps clinical, demographic, or transcriptomics data? If coded and all numerical, these can be added as an--additfile by doing the following:# Running GenoML munging on discrete data using PLINK genotype binary files, a phenotype file, and an addit filegenomldiscretesupervisedmunge\--prefixoutputs/test_discrete_geno\--genoexamples/discrete/training\--phenoexamples/discrete/training_pheno.csv\--additexamples/discrete/training_addit.csvYou also have the option of not using PLINK binary files if you would like to just preprocess (and then, later train) on a phenotype and addit file by doing the following:# Running GenoML munging on discrete data using PLINK genotype binary files, a phenotype file, and an addit filegenomldiscretesupervisedmunge\--prefixoutputs/test_discrete_geno\--phenoexamples/discrete/training_pheno.csv\--additexamples/discrete/training_addit.csvAre you interested in selecting and ranking your features? If so, you can use the--feature_selectionflag and specify like so...:# Running GenoML munging on discrete data using PLINK genotype binary files, a phenotype file, and running feature selectiongenomldiscretesupervisedmunge\--prefixoutputs/test_discrete_geno\--genoexamples/discrete/training\--phenoexamples/discrete/training_pheno.csv\--additexamples/discrete/training_addit.csv\--feature_selection50The--feature_selectionflag uses extraTrees (classifierfor discrete data;regressorfor continuous data) to output a*.approx_feature_importance.txtfile with the features most contributing to your model at the top.Do you have additional covariates and confounders you would like to adjust for in the munging step prior to training your model and/or would like to reduce your data? To adjust, use the--adjust_dataflag with the following necessary flags:--adjust_normalize: Would you like to normalize your final adjusted data? (Default: yes)--target_features: A .txt file, one column, with a list of features to adjust (no header). These should correspond to features in the munged dataset--confounders: A .csv of confounders to adjust for with ID column and header. Numeric, with no missing data and the ID column is mandatory (this can be PCs, for example)To reduce your data prior to adjusting, use the--umap_reduce yesflag. This flag will also prompt you for if you want to adjust your data, normalize, and what your target features and confounders might be. We use theUniform Manifold Approximation and Projection for Dimension Reduction(UMAP) to reduce your data into 2D, adjust, and exports a plot and an adjusted dataframe moving forward. This can be done by running the following:# Running GenoML munging on discreate data using PLINK binary files, a phenotype file, using UMAP to reduce dimensions and account for features, and running feature selectiongenomldiscretesupervisedmunge\--prefixoutputs/test_discrete_geno\--genoexamples/discrete/training\--phenoexamples/discrete/training_pheno.csv\--additexamples/discrete/training_addit.csv\--umap_reduceyes\--adjust_datayes\--adjust_normalizeyes\--target_featuresexamples/discrete/to_adjust.txt\--confoundersexamples/discrete/training_addit_confounder_example.csv\--feature_selection50Here, the--confoundersflag takes in a dataset of features that should be accounted for. This is a .csv file with the ID column and header included and is numeric with no missing data.The ID column is mandatory.The--target_featuresflag takes in a .txt with a list of features (column names) you are adjusting for.2. Training with GenoMLTraining with GenoML competes a number of different algorithms and outputs the best algorithm based on a specific metric that can be tweaked using the--metric_maxflag(default is AUC).Requiredarguments for GenoML are the following:data: Is the datacontinuousordiscrete?method: Do you want to usesupervisedorunsupervisedmachine learning?(unsupervised currently under development)mode:  would you like tomunge,train,tune, ortestyour model?--prefix: Where would you like your outputs to be saved?The most basic command to train your model looks like the following, it looks for the*.dataForMLfile that was generated in the munging step:# Running GenoML supervised training after munging on discrete datagenomldiscretesupervisedtrain\--prefixoutputs/test_discrete_genoIf you would like to determine the best competing algorithm by something other than the AUC, you can do so by changing the--metric_maxflag (Options includeAUC,Balanced_Accuracy,Sensitivity, andSpecificity) :# Running GenoML supervised training after munging on discrete data and specifying the metric to maximize bygenomldiscretesupervisedtrain\--prefixoutputs/test_discrete_geno\--metric_maxSensitivityNote:The--metric_maxflag is only available for discrete datasets.3. Tuning with GenoMLThe most basic command to tune your model looks like the following, it looks for the file that was generated in the training step:# Running GenoML supervised tuning after munging and training on discrete datagenomldiscretesupervisedtune\--prefixoutputs/test_discrete_genoIf you are interested in changing the number of iterations the tuning process goes through by modifying--max_tune(default is 50), or the number of cross-validations by modifying--n_cv(default is 5), this is what the command would look like:# Running GenoML supervised tuning after munging and training on discrete data, modifying the number of iterations and cross-validationsgenomldiscretesupervisedtune\--prefixoutputs/test_discrete_geno\--max_tune10--n_cv3If you are interested in tuning on another metric other than AUC(default is AUC), you can modify--metric_tune(options areAUCorBalanced_Accuracy) by doing the following:# Running GenoML supervised tuning after munging and training on discrete data, modifying the metric to tune bygenomldiscretesupervisedtune\--prefixoutputs/test_discrete_geno\--metric_tuneBalanced_Accuracy4. Testing/Validation with GenoMLIn order to properly test how your model performs on a dataset it's never seen before (but you start with different PLINK binaries), we have created the harmonization step that will:Keep only the same SNPs between your reference dataset and the dataset you are using for validationForce the reference alleles in the validation dataset to match your reference datasetExport a.txtfile with the column names from your reference dataset to later use in the munging of your validation datasetUsing GenoML for both your reference dataset and then your validation dataset, the process will look like the following:Munge and train your first dataset
- That will be your “reference” modelUse the outputs of step 1's munge for your reference model to harmonize your incoming validation datasetRun through harmonization step with your validation datasetRun through munging with your newly harmonized datasetRetrain your reference model with only the matching columns of your unseen dataGiven the nature of ML algorithms, you cannot test a model on a set of data that does not have identical featuresTest your newly retrained reference model on the unseen dataHarmonizing your Validation/Test DatasetRequiredarguments for harmonizing with GenoML are the following:--test_geno_prefix: What is the prefix of your validation dataset PLINK binaries?--test_prefix: What do you want the output to be named?--ref_model_prefix:  What is the name of the previously GenoML-munged dataset you would like to use as your reference dataset? (Without the.dataForML.h5suffix)--training_snps_alleles: What are the SNPs and alleles you would like to use? (This is generated at the end of your previously-GenoML munged dataset with the suffixvariants_and_alleles.tab)To harmonize your incoming validation dataset to match the SNPs and alleles to your reference dataset, the command would look like the following:# Running GenoML harmonizegenomlharmonize\--test_geno_prefixexamples/discrete/validation\--test_prefixoutputs/validation_test_discrete_geno\--ref_model_prefixoutputs/test_discrete_geno\--training_snps_allelesoutputs/test_discrete_geno.variants_and_alleles.tabThis step will generate:a*_refColsHarmonize_toKeep.txtfile of columns to keep for the next step*_refSNPs_andAlleles.*PLINK binary files (.bed, .bim, and .fam) that have the SNPs and alleles match your reference datasetNow that you have harmonized your validation dataset to your reference dataset, you can now munge using a command similar to the following:# Running GenoML munge after GenoML harmonizegenomldiscretesupervisedmunge--prefixoutputs/validation_test_discrete_geno\--genooutputs/validation_test_discrete_geno_refSNPs_andAlleles\--phenoexamples/discrete/validation_pheno.csv\--additexamples/discrete/validation_addit.csv\--ref_cols_harmonizeoutputs/validation_test_discrete_geno_refColsHarmonize_toKeep.txtAll munging options discussed above are available at this step, the only difference here is you will add the--ref_cols_harmonizeflag to include the*.refColsHarmonize_toKeep.txtfile generated at the end of harmonizing to only keep the same columns that the reference dataset had.After munging and training your reference model and harmonizing and munging your unseen test data,you will retrain your reference model to include only matching features. Given the nature of ML algorithms, you cannot test a model on a set of data that does not have identical features.To retrain your model appropriately, after munging your test data with the--ref_cols_harmonizeflag, a final columns list will be generated at*.finalHarmonizedCols_toKeep.txt. This includes all the features that match between your unseen test data and your reference model. Use the--matching_columnsflag when retraining your reference model to use the appropriate features.When retraining of the reference model is complete, you are ready to test!A step-by-step guide on how to achieve this is listed below:# 0. MUNGE THE REFERENCE DATASETgenomldiscretesupervisedmunge\--prefixoutputs/test_discrete_geno\--genoexamples/discrete/training\--phenoexamples/discrete/training_pheno.csv# Files made:# outputs/test_discrete_geno.dataForML.h5# outputs/test_discrete_geno.list_features.txt# outputs/test_discrete_geno.variants_and_alleles.tab# 1. TRAIN THE REFERENCE DATASETgenomldiscretesupervisedtrain\--prefixoutputs/test_discrete_geno# Files made:# outputs/test_discrete_geno.best_algorithm.txt# outputs/test_discrete_geno.trainedModel.joblib# outputs/test_discrete_geno.trainedModel_trainingSample_Predictions.csv# outputs/test_discrete_geno.trainedModel_withheldSample_Predictions.csv# outputs/test_discrete_geno.trainedModel_withheldSample_ROC.png# outputs/test_discrete_geno.trainedModel_withheldSample_probabilities.png# outputs/test_discrete_geno.training_withheldSamples_performanceMetrics.csv# 2. HARMONIZE TEST DATASET IF USING PLINK/GENOTYPESgenomlharmonize\--test_geno_prefixexamples/discrete/validation\--test_prefixoutputs/validation_test_discrete_geno\--ref_model_prefixoutputs/test_discrete_geno\--training_snps_allelesoutputs/test_discrete_geno.variants_and_alleles.tab# Files made:# outputs/validation_test_discrete_geno.refColsHarmonize_toKeep.txt# outputs/validation_test_discrete_geno.refSNPs_andAlleles.bed# outputs/validation_test_discrete_geno.refSNPs_andAlleles.bim# outputs/validation_test_discrete_geno.refSNPs_andAlleles.fam# 3. MUNGE THE TEST DATASET ON REFERENCE MODEL COLUMNSgenomldiscretesupervisedmunge\--prefixoutputs/validation_test_discrete_geno\--genooutputs/validation_test_discrete_geno.refSNPs_andAlleles\--phenoexamples/discrete/validation_pheno.csv\--additexamples/discrete/validation_addit.csv\--ref_cols_harmonizeoutputs/validation_test_discrete_geno.refColsHarmonize_toKeep.txt# Files made:# outputs/validation_test_discrete_geno.finalHarmonizedCols_toKeep.txt# outputs/validation_test_discrete_geno.list_features.txt# outputs/test_discrete_geno.variants_and_alleles.tab# outputs/validation_test_discrete_geno.dataForML.h5# 4. RETRAIN REFERENCE MODEL ON INTERSECTING COLUMNS BETWEEN REFERENCE AND TESTgenomldiscretesupervisedtrain\--prefixoutputs/test_discrete_geno\--matching_columnsoutputs/validation_test_discrete_geno.finalHarmonizedCols_toKeep.txt# Note: This replaces the trained model you made in step 1!# Files made:# outputs/test_discrete_geno.best_algorithm.txt# outputs/test_discrete_geno.trainedModel.joblib# outputs/test_discrete_geno.trainedModel_trainingSample_Predictions.csv# outputs/test_discrete_geno.trainedModel_withheldSample_Predictions.csv# outputs/test_discrete_geno.trainedModel_withheldSample_ROC.png# outputs/test_discrete_geno.trainedModel_withheldSample_probabilities.png# outputs/test_discrete_geno.training_withheldSamples_performanceMetrics.csv# OPTIONAL: TUNING YOUR RETRAINED REFERENCE MODEL ON INTERSECTING COLUMNS BETWEEN REFERENCE AND TESTgenomldiscretesupervisedtune\--prefixoutputs/test_discrete_geno\--matching_columnsoutputs/validation_test_discrete_geno.finalHarmonizedCols_toKeep.txt# 5. TEST RETRAINED REFERENCE MODEL OR TUNED MODEL ON UNSEEN DATAgenomldiscretesupervisedtest\--prefixoutputs/validation_test_discrete_geno\--test_prefixoutputs/validation_test_discrete_geno\--ref_model_prefixoutputs/test_discrete_geno.trainedModel# If testing a tuned model, change suffix from .trainedModel to .tunedModel# Files made:# outputs/validation_test_discrete_geno.testedModel_allSample_predictions.csv# outputs/validation_test_discrete_geno.testedModel_allSample_probabilities.png# outputs/validation_test_discrete_geno.testedModel_allSample_ROC.png# outputs/validation_test_discrete_geno.testedModel_allSamples_performanceMetrics.csvNote:When munging the test dataset on the reference model columns using the --ref_cols_harmonize, be sure not to include the --feature_selection flag, as you have already specified the columns to keep moving forward.5. Experimental FeaturesUNDER ACTIVE DEVELOPMENTPlanned experimental features include, but are not limited to:Unsupervised munging, training, tuning, and testingGWAS QC and PipelineNetwork analysesMeta-learningFederated learningBiobank-scale supportCross-silo checks for genetic duplicatesOutlier detection...?