Package: biodem
DEMDual-extraction method for phenotypic prediction and functional gene miningTheDEMsoftware comprises 4 functional modules: data preprocessing, dual-extraction modeling, phenotypic prediction, and functional gene mining.Directory structure.
├──src# Python, Julia and R implementation for DEM├──data# Data files and example files├──tests# Easy tests for packages and pipelines├──pyproject.toml# Python package metadata├──LICENSE
└──README.md

./src/biodem
├──module1_predata.py# Module 1: Data preprocessing├──module2_model.py# Module 2: Dual-extraction modeling├──module3_predict.py# Module 3: Phenotypic prediction├──module4_mining.py# Module 4: Functional gene mining├──__init__.py# Initialize the Python package├──cli_dem.py# Command line interface for DEM└──utils_model.py# Utilities for modeling and prediction./src/DEMpre
├──Manifest.toml# Julia package metadata├──Project.toml# Julia package metadata└──src├──DEMpre.jl# Initialize the Julia package└──modules├──module_SearchSNP.jl# Search the nearest genic/intergenic region of SNP from GTF file├──module_DimsReduction.jl# Train the model and transform SNP data├──module_InterpretModel.jl# Get the parameters of the model for transformation interpretation├──models.jl# Model for SNP transformation├──types.jl# Self-defined data types├──utils_dl.jl# Utilities for modeling└──utils.jl# Utilities for IO and multithreadingDEM is mainly implemented in Python. You may use the Julia packageDEMpre.jlin some steps to process and transform SNP data.DEM has the Python packagebiodemwhich includes 4 functional modules:1. Data preprocessingSteps:Imputation & min-max scalingFeature selection using the variance threshold method, PCA, and random forests for multi-omics data.SNP transformationRelated functions:dem-imputedem-select-varpca,dem-select-rf(In Julia)search_gene_of_snp,train4embed,transform_snp2. Dual-extraction modelingIt takes preprocessed multi-omics data and phenotypic data as inputs for training and validation, based on which the DEM model is constructed. It is capable of performing both classification and regression tasks.Related function:dem-model3. Phenotypic predictionIt takes trained model and omics data of new/test samples as inputs and returns the predicted phenotypic values.Related function:dem-predict4. Functional gene miningIt performs functional gene mining based on the trained DEM model throughfeature permutation.Related function:dem-rankSystem requirementsPython 3.10 or 3.11.Graphics: GPU withPyTorchsupport.Recommended: NVIDIA graphics card with 12GB memory or larger.Install biodemConda/Mambaor virtualenv is recommended for installation.Create a conda environment:condacreate-ndempython=3.11
condaactivatedem# Install PyTorch with CUDA supportcondainstallpytorchtorchvisiontorchaudiopytorch-cuda=12.1-cpytorch-cnvidiaInstall biodem package:Simple installation fromPyPIpipinstallbiodemGet biodem package and install from local
Click to download:https://github.com/cma2015/DEM/blob/main/dist/biodem-0.2.0-py3-none-any.whlpipinstallbiodem-0.2.0-py3-none-any.whlA pipeline for testing is provided in./test/test_biodem.py.cdtestpythontest_biodem.pyInput and output file formatsPlease refer to the directory./dataforthe file formatsthat will be used in following methods.How to usebiodemWe provide bothcommand line toolsandimportable functionsforbiodem.Use command line toolsIn terminal, run the command likedem-impute --helpordem-impute -hfor help.dem-imputeThe tool contains 3 steps:Delete omics features with missing values exceeding 25%.Impute missing values with mean value of each feature.Min-max scaling.# Usage:dem-impute-I<an_omics_file_in>-O<an_omics_file_out>-i<a_phenotypes_file_in>-o<phenotypes_out>-p<NA_threshold>-m<is_minmax_omics>-z<is_zscore_phenotypes>ParametersTypeRequiredDescriptions-I--inomstringoptionalInput a path to an omics file-O--outomstringoptionalDefine your output omics file path-i--inphstringoptionalInput a path to a trait's phenotypes-o--outphstringoptionalDefine your output phenotypes path-p--propnafloatoptionalThe allowed max proportion of missing values in a feature (DEFAULT: 0.25)-m--minmaxint, 0 or 1optionalWhether min-max scaling for omics is required (0 denotes False, 1 denotes True, DEFAULT: 1)-z--zscoreint, 0 or 1optionalWhether z-score transformation for phenotypes is required (0 denotes False, 1 denotes True)dem-select-varpcaTo analyze the significance of each feature across different phenotype classes and save important features, the following steps were taken:Remove features with variance less than the given threshold.The ANOVA F value for each feature was calculated.Features with the lowest F values were sequentially removed.This process continued until the first principal component of the remaining features explained less than the given percent of the variance.# Usage:dem-select-varpca-I<an_omics_file_in>-i<a_phenotypes_file_in>-O<omics_file_output>-V<var_threshold>-P<target_variance_ratio>ParametersTypeRequiredDescriptions-I--inomstring*Input a path to an omics file-i--inphstring*Input a path to a trait's phenotypes-O--outomstring*Define your output omics file path-V--minvarfloatoptionalThe allowed minimum variance of a feature (DEFAULT: 0.0)-P--varpcfloatoptionalTarget variance of PC1 (DEFAULT: 0.5)dem-select-rfThe random forest (RF) algorithm, based on an ensemble of the given number of decision trees, was employed to screen out representative omics features for subsequent DEM construction.# Usage:dem-select-rf-I<an_omics_file_in>-i<a_phenotypes_file_in>-O<omics_file_output>-n<num_features_to_save>-p<validation_data_proportion>-N<num_trees>-S<random_seed_rf>-s<random_seed_sp>ParametersTypeRequiredDescriptions-I--inomstring*Input a path to an omics file-i--inphstring*Input a path to a trait's phenotypes-O--outomstring*Define your output omics file path-n--nfeatint*Number of features to save-p--propvfloatoptionalProportion of validation set (DEFAULT: 0.2)-k--kfoldintoptionalNumber of folds for k-fold cross validation (DEFAULT: 5)-N--ntreeintoptionalNumber of trees in the random forest (DEFAULT: 2500) (larger number of trees will result in more accurate results, but will take longer to run)-S--seedrflist[int]optionalRandom seeds for RF (DEFAULT: 1000, 1001, ..., 1009)-s--seedsplist[int]optionalRandom seeds for splitting (DEFAULT: 0, 1, ..., 4)dem-modelDEM aims to achieve high phenotypic prediction accuracy and identify the most informative omics features for the given trait.
This tool is used to construct a dual-extraction model based on the given omics data and phenotype, through cross validation or random sampling.# Usage example:dem-model-o<where_to_save_model>-r1-i<phenotype_file>-I<omics1_file><omics2_file><omics3_file>--seedsp12331234123512361237ParametersTypeRequiredDescriptions-I--inomlist[string]*Input path(s) to omics file(s)-i--inphstring*Input a path to a trait's phenotypes-o--outmdstring*The path to save your trained DEM model-r--regrclasint, 0 or 1*Regression or Classification task (1 denotes regression, 0 denotes classification)-p--propvfloatoptionalProportion of validation set (DEFAULT: 0.2)-k--kfoldintoptionalNumber of folds for k-fold cross validation (DEFAULT: 5)-s--seedsplist[int]optionalRandom seed(s) for data partition(s) (DEFAULT: 0...4)-b--batchsizeintoptionalBatch size (DEFAULT: 32)-l--learningratefloatoptionalLearning rate (DEFAULT: 0.0001)-e--patienceintoptionalEarly stopping patience (DEFAULT: 10)-N--nencintoptionalNumber of DEM encoders (DEFAULT: 4)-D--dropoutfloatoptionalDropout rate (DEFAULT: 0.1)dem-predictPredicts the phenotype of given omics data files using a trained DEM model.# Usage example:dem-predict-m<model_file_path>-o<phenotype_output>-I<omics_files>ParametersTypeRequiredDescriptions-m--inmdstring*The path to your trained DEM model-I--inomlist[string]*Input path(s) to omics file(s)-o--outphstring*The path to save predicted phenotypesdem-rankAssess the importance of features by contrasting the DEM model’s test performance on the actual feature values against those where the feature values have been randomly shuffled. Features that are ranked highly are then identified.# Usage example:dem-rank-m<model_file_path>-r1-i<phenotype_file>-o<rank_output>-s1000111111121113-I<omics1_file><omics2_file>ParametersTypeRequiredDescriptions-I--inomlist[string]*Input path(s) to omics file(s)-i--inphstring*Input a path to a trait's phenotypes-m--inmdstring*The path to your trained DEM model-o--outrankstring*The path to save importance scores and the order-r--regrclasint, 0 or 1*Regression or Classification task (1 denotes regression, 0 denotes classification)-s--seedrklist[int]optionalRandom seeds for ranking repeats (default: 0-9)Importbiodemin your Python projectThe main functions ofbiodemare listed below.
Their purposes and parameters are described in related sections ofUse command line tools.Python functionsCorresponding command line toolsfeat_impute_scaledem-imputefeat_select_varpcadem-select-varpcafeat_select_rfdem-select-rfdem_modeldem-modeldem_predictdem-predictdem_rankdem-rank# Example:# Use a trained DEM model to predict the phenotype of given omics data files# dem_predict is a main function in biodemfrombiodemimportdem_predictyour_model_path='~/model_dem.pth'your_omics_files_paths=['~/omics_1.csv','~/omics_2.csv']your_phenotype_output_path='~/phenotype_predicted.csv'# Run the predictiondem_predict(your_model_path,your_omics_files_paths,your_phenotype_output_path)DEM provides the Julia package DEMpre.jlwhich is intended to SNP transformationSearch for genomic regions associated with SNPsIt accepts SNP positions and the reference genome as inputs, and it returns the genomic regions containing the SNPs along with their associated genic/intergenic information.Transform SNP data to genomic embeddingThe sparse and discrete SNP features is transformed into dense and continuous features that represent genomic variation.The package is used to transform encoded SNP data to genomic embedding whose dimensionality is determined by the number of genomic regions (default: number of genes).System requirementsJulia1.9 or later.Graphics: GPU withFluxsupport.Recommended: NVIDIA graphics card with 12GB memory or larger.Input and output file formatsPlease refer tothe example filesin./data/examplesforthe file formatsthat will be used in following functions.Get DEMpre.jlgitclonehttps://github.com/cma2015/dem.gitcdDEMHow to useDEMpre.jlActivate the package# In shell, enter Julia REPLjulia# In Julia REPLusingPkgPkg.activate("./src/DEMpre")# It is relative path in this git repositoryinclude("./src/DEMpre/src/DEMpre.jl")# It is relative path in this git repositoryUse main functionssearch_gene_of_snpSearch the nearest genic/intergenic region of SNP from GTF file. It is accelerated by multithreading.ParametersTypeRequiredDescriptionspath_snpString*path of SNP filepath_gtfString*path of GTF filepath_outputString*path of output file where the nearest genic/intergenic regions will be savedinclude_intergenicBooloptionalwhether to include SNPs that are located in intergenic region (default: false)num_threadsInt64optionalnumber of threads to use (default: 8)# Example:path_snp="~/your_SNP_file.csv"path_gtf="~/your_GTF_file.gtf"path_output="~/your_output.csv"include_intergenic=falsenum_threads=8# RunDEMpre.search_gene_of_snp(path_snp,path_gtf,path_output,include_intergenic,num_threads)train4embedConstruct a locally connected neural network and train it with SNP data and phenotypes. It is intended to transform SNP data to genomic embedding.ParametersTypeRequiredDescriptionspath_snp2regionString*path of SNP2region file (the result of search_gene_of_snp)path_snp_matrixString*path of SNP matrix filepath_phenotypeString*path of phenotype fileoutput_directoryString*path of output directory for saving the trained model and recordscompress_modelBooloptionalwhether to compress the sparse model (default: true)prop_valFloat64optionalproportion of validation data (default: 0.2)batch_sizeInt64optionalbatch size (default: 32)max_epochsInt64optionalmaximum epochs for training (default: 1000)learning_rateFloat64optionallearning rate (default: 0.001)early_stop_patienceInt64optionalearly stop patience (default: 70)# Example:path_snp2region="~/your_snp2region_file.csv"path_snp_matrix="~/your_snp_matrix_file.csv"path_phenotype="~/your_phenotype_file.csv"output_directory="~/your_output_directory"compress_model=trueprop_val=0.2batch_size=32max_epochs=1000learning_rate=0.0001early_stop_patience=70# RunDEMpre.train4embed(path_snp2region,path_snp_matrix,path_phenotype,output_directory,compress_model,prop_val,batch_size,max_epochs,learning_rate,early_stop_patience)transform_snpTransform encoded SNP data to genomic embedding.ParametersTypeRequiredDescriptionspath_modelString*path of trained modelpath_snp_matrixString*path of new SNP data for testpath_outputString*path of genomic embedding# Example:path_model="~/your_model.jld2"path_snp_matrix="~/your_snp_matrix.csv"path_output="~/your_output_embedding.csv"# RunDEMpre.transform_snp(path_model,path_snp_matrix,path_output)Asking for helpIf you have any questions please:Contact us viaGitHub.Emailus.