Package: expert-mst
EXPERTExact and pervasive expert model for source tracking based on transfer learningHabitat specific patterns reflected by microbial communities, as well as complex interactions between the community and their environments or hosts' characteristics, have created obstacles for microbial source tracking: diverse and context-dependent applications are asking for quantification of the contributions of different niches (biomes), which have already overwhelmed existing methods. Moreover, existing source tracking methods could not extend well for source tracking samples from understudied biomes, as well as samples from longitudinal studies.Built upon the biome ontology information and transfer learning techniques, EXPERT has acquired the context-aware flexibility and could easily expand the supervised model's search scope to include the context-dependent community samples and understudied biomes. While at the same time, it is superior to current approaches in source tracking accuracy and speed. EXPERT's superiority has been demonstrated on multiple source tracking tasks, including source tracking samples collected at different disease stages and longitudinal samples. For example, when dealing with 635 samples from a recent study of colorectal cancer, EXPERT could achieve an AUROC of 0.977 when predicting the host's phenotypical status. In summary, EXPERT has unleashed the potential of model-based source tracking approaches, enabling source tracking in versatile context-dependent settings, accomplishing pervasive and in-depth knowledge discovery from microbiome.If you use EXPERT in a scientific publication, we would appreciate citations to the following paper:Enabling technology for microbial source tracking based on transfer learning: From ontology-aware general knowledge to context-aware expert systems
Hui Chong, Qingyang Yu, Yuguo Zha, Guangzhou Xiong, Nan Wang, Chuqing Sun, Sicheng Wu, Weihua Chen, Kang Ning
bioRxiv 2021.01.29.428751; doi: https://doi.org/10.1101/2021.01.29.428751If you have any question about our work, feel free to contactus.Thank you for using EXPERT.Current featuresContext-aware ability to adapt to microbiome studies viatransfer learningFast, accurate and interpretable source tracking viaontology-aware forward propagationSelective learning from training dataUltra-fast data cleaning & cleaning via in-memory NCBI taxonomy databaseParallelized feature encoding viatensorflow.kerasInstallationpythonsetup.pyinstallUsageOntology constructionconstruct a biome ontology usingmicrobiomes.txtexpertconstruct-imicrobiomes.txt-oontology.pkl# Also equivalent toexpertconstruct--inputmicrobiomes.txt--outputontology.pklInput:microbiomes.txtfile, contains path from "root" node to each leaf node of biome ontology.root:Environmental:Terrestrial:Soil
root:Environmental:Terrestrial:Soil:Agricultural
root:Environmental:Terrestrial:Soil:Boreal_forest
root:Environmental:Terrestrial:Soil:Contaminated
root:Environmental:Terrestrial:Soil:Crop
root:Environmental:Terrestrial:Soil:Crop:Agricultural_land
root:Environmental:Terrestrial:Soil:Desert
root:Environmental:Terrestrial:Soil:Forest_soil
root:Environmental:Terrestrial:Soil:Grasslands
root:Environmental:Terrestrial:Soil:Loam:Agricultural
root:Environmental:Terrestrial:Soil:Permafrost
root:Environmental:Terrestrial:Soil:Sand
root:Environmental:Terrestrial:Soil:Tropical_rainforest
root:Environmental:Terrestrial:Soil:Uranium_contaminated
root:Environmental:Terrestrial:Soil:Wetlands
root:Host-associated:Plants:Rhizosphere:SoilOutput: constructed biome ontology (pickle format, non-human-readable).Source mappingMapping their source environments to microbiome ontologyexpertmap--to-otlg-tontology.pkl-imapper.csv-olabels.h5# Also equivalent toexpertmap--to-otlg--otlgontology.pkl--inputmapper.csv--outputlabels.h5Input: the mapper file, contains biome source information for samples.EnvSampleID0root:Engineered:WastewaterERR22604421root:Engineered:WastewaterSRR9803222root:Engineered:WastewaterERR29852723root:Engineered:WastewaterERR28146484root:Engineered:WastewaterERR2985275Output: the labels for samples in each layer of the biome ontology (HDF format, non-human-readable).Data converting & cleaningConvert input data to a count matrix ingenuslevel.expertconvert-icountMatrices.txt-ocountMatrix.h5--in-cm# Also equivalent toexpertconvert--inputcountMatrices.txt--outputcountMatrix.h5--in-cmInput: a text file contains path to input count matrix file / OTU table.datasets/soil_dataset/root:Host-associated:Plants:Rhizosphere:Soil/MGYS00005146-ERR1690680.tsv
datasets/soil_dataset/root:Host-associated:Plants:Rhizosphere:Soil/MGYS00005146-ERR1689675.tsv
datasets/soil_dataset/root:Host-associated:Plants:Rhizosphere:Soil/MGYS00000513-ERR986792.tsv
datasets/soil_dataset/root:Host-associated:Plants:Rhizosphere:Soil/MGYS00005146-ERR1691198.tsv
datasets/soil_dataset/root:Host-associated:Plants:Rhizosphere:Soil/MGYS00001704-ERR1905845.tsv
datasets/soil_dataset/root:Host-associated:Plants:Rhizosphere:Soil/MGYS00005146-ERR1689214.tsv
datasets/soil_dataset/root:Host-associated:Plants:Rhizosphere:Soil/MGYS00005146-ERR1689910.tsvOutput: converted count matrix file ingenuslevel (HDF format, non-human-readable).Ab initio trainingBuild EXPERT model from scratch and trainingexperttrain-icountMatrix.h5-llabels.h5-tontology.pkl-omodel# Also equivalent toexperttrain--inputcountMatrix.h5--labelslabels.h5--otlgontology.pkl--outputmodelInput: biome ontology and converted count matrix in genus level (and also labels for samples involved in the count matrix).Output: trained model.Fast adaptationexperttransfer-icountMatrix.h5-llabels.h5-tontology.pkl-omodel# Also equivalent toexperttransfer--inputcountMatrix.h5--labelslabels.h5--otlgontology.pkl--outputmodelInput: biome ontology and converted count matrix in genus level (and also labels for samples involved in the count matrix).Output: trained model.Source trackingexpertsearch-icountMatrix.h5-osearchResult-mmodel# Also equivalent toexpertsearch--inputcountMatrix.h5--outputsearchResult--modelmodelInput: converted count matrix in genus level.Output: search result (multi-layer ).searchResult
├── layer-2.csv
├── layer-3.csv
├── layer-4.csv
├── layer-5.csv
└── layer-6.csvTakelayer-2.csvas an example.root:Engineeredroot:Environmentalroot:Host-associatedroot:MixedUnknownERR22787520.00414270160.263724180.686321260.000400036570.045411825ERR22787530.0028411790.079288960.917355240.000514631450.0ERR26668550.00067510480.00218035650.99705319.1493675e-050.0ERR26668600.00052277860.0139029890.985426250.000148039280.0ERR26668810.00095690570.00239577770.99654030.000106945660.0Evaluationexpertevaluate-isearchResultFolder-llabels.h5-oEvaluationReport-pNUMProcesses# Also equivalent toexpertevaluate--inputsearchResultFolder--labelslabels.h5--outputEvaluationReport--processorsNUMProcessesInput: multi-layer labels and search result (source contribution) for samples.Output: label-based evaluation report.EvaluationReport
├── layer-2
│   └── root:Host-associated.csv
├── layer-2.csv
├── layer-3
│   └── root:Host-associated:Human.csv
├── layer-3.csv
├── layer-4
│   ├── root:Host-associated:Human:Circulatory_system.csv 
│   ├── root:Host-associated:Human:Digestive_system.csv
│   ├── root:Host-associated:Human:Lympathic_system.csv
│   ├── root:Host-associated:Human:Reproductive_system.csv
│   ├── root:Host-associated:Human:Respiratory_system.csv
│   └── root:Host-associated:Human:Skin.csv
├── layer-4.csv
├── layer-5
│   ├── root:Host-associated:Human:Circulatory_system:Blood.csv
│   ├── ...
│   └── root:Host-associated:Human:Respiratory_system:Pulmonary_system.csv
├── layer-5.csv
├── layer-6
│   ├── root:Host-associated:Human:Digestive_system:Large_intestine:Fecal.csv
│   ├── ...
│   └── root:Host-associated:Human:Respiratory_system:Pulmonary_system:Sputum.csv
└── layer-6.csvTakelayer-4/root:Host-associated:Human:Skin.csvas an example.tTNFPFNTPAccSnSpTPRFPRRcPrF1ROC-AUCF-max0.0047688048470.09231.00.01.01.01.00.09230.16890.99510.93740.014479428933048160.94440.99380.93930.99380.06070.99380.62470.76720.99510.93740.024554521424448020.95840.99090.95510.99090.04490.99090.69150.81460.99510.93740.034593417535947870.96550.98780.96320.98780.03680.98780.7320.84090.99510.93740.044622814597347730.97080.98490.96940.98490.03060.98490.76590.86170.99510.9374Run the program with-hoption to see a detailed description on work modes & options.Input abundance dataEXPERT takes two kinds of **abundance data **as inputs.Taxonomic assignments result for a single sample (OTU table)Notice that here is a header "# Constructed from biom file" in the first line.# Constructed from biom file# OTU IDERR1754760taxonomy20711919.0sk__Archaea11809045.0sk__Archaea;k__;p__Thaumarchaeota;c__;o__Nitrosopumilales;f__Nitro...15315638.0sk__Archaea;k__;p__Thaumarchaeota;c__;o__Nitrosopumilales;f__Nitro...1317041.0sk__Archaea;k__;p__Thaumarchaeota;c__Nitrososphaeria;o__Nitrososp...1031815174.0sk__Bacteria1573619.0sk__Bacteria;k__;p__;c__;o__;f__;g__;s__agricultural_soil_bacterium_SC-I-11Taxonomic assignments result for multiple samples (count matrix)#SampleIDERR1844510ERR1844449ERR1844450ERR1844451sk__Archaea1.017.08.016.0sk__Archaea;k__;p__Crenarchaeota0000sk__Archaea;k__;p__Euryarchaeota8.02.03.01.0sk__Archaea;k__;p__Eury...;c__...0000sk__Archaea;k__;p__Eury...;c__...;o__...0000LicenseMaintainerNameEmailOrganizationHui Chonghuichong.me@gmail.comResearch Assistant, School of Life Science and Technology, Huazhong University of Science & TechnologyKang Ningningkang@hust.edu.cnProfessor, School of Life Science and Technology, Huazhong University of Science & Technology