Package: biodigest
biodigestThe python package forDIGEST(validation ofdisease andgenesets, clusterings or subnetworks). It greatly facilitates in silico validation of gene and disease sets, clusterings or subnetworks via fully automated validation pipelines comprising disease and gene ID mapping, enrichment
analysis, comparisons of shared genes and variants, and background distribution estimation. Moreover, functionality is provided to automatically update the external databases used by the pipelines.A light version excluding the subnetwork option end therefore reducing the needed
requirements for installing more complex python packages can be found asbiodigest-light.Source codeSetup for proper usageAfter installing biodigest, you need to installgraph-tools package.importbiodigestBefore you can run the validation, you need to download precalculated mappings and distance matrices. These can be downloaded in two ways:1. [Recommended] Get data from apiThe API keeps all data up to date and checks for updates at regular intervals. This process takes 1-5 minutes depending on the internet connection.frombiodigestimportsetupsetup.main(setup_type="api")2. Create data from scratchAny mappings are freshly fetched from the databases and the distance matrices are calculated. Be aware that this can take up to 3 hours.frombiodigestimportsetupsetup.main(setup_type="create")Run validationfrombiodigest.single_validationimportsingle_validationresults=single_validation(tar:Union[pd.DataFrame,set],tar_id:str,mode:str,distance:str="jaccard",ref:Union[str,set]=None,ref_id:str=None,enriched:bool=False,network_data:dict=None,mapper:Mapper=FileMapper(),runs:int=config.NUMBER_OF_RANDOM_RUNS,background_model:str="complete",replace=100,verbose:bool=False)All results that can later be saved and visualize are saved inresultsas data typedict().ParametersRequired parameterstar: Target input you want to be validateda cluster should be of typepd.DataFrame()withcolumns=["id","cluster"]a set should be of typeset()tar_id: Is the id type of the target (see possible options below)ref: Reference, to whichtarwill be compared (Only for mode id-set and set-set)a single id should be of typestra set should be of typeset()ref_id: Is the id type of the reference (see possible options below)mode: Desired mode. See possible options below.Optional parametersdistance: Distance measure used for pairwise comparisonenriched: SetTrue, if only enriched attributes of the reference set should be used (Only for set-set)network_data: Only for "subnetwork" and "subnetwork-set" mode. Dictionary consisting of {"network_file": path to network file,
"prop_name": name of vertex property with ids if network file of type graphml or gt,
"id_type": id type of network ids}background_model: Model defining how random values should be picked. See possible options below.mapper: Mapper object indicating where all files from the setup are saved.[Default=FileMapper()]runs: Number of runs with random target values for p-value calculationreplace: Percentage of how many of the original ids should be replaced with random idsverbose: get additional information during the runSupported typesgene types: entrez, ensembl, symbol, uniprotdisease types: mondo, omim, snomedct, umls, orpha, mesh, doid, ICD-10Modesset: Compare similarity inside the set using the mean of all pairwise comparisonsid-set: Compare target set to reference setset-set: Compare target set to reference idclustering: Compare cluster quality inside clustering. Either genes or diseasessubnetwork: Compare similarity inside the subnetwork nodes. Either genes or diseasessubnetwork-set: Compare target subnetwork to reference set. Both either genes or diseasesBackground modelscomplete: Random ids will be picked completely randomlyterm-pres: Random ids will preserve the number of mapped terms for the replaced idsnetwork: Random ids will preserve the number of connected components in given network.ResultThe method call returns the result in a json format of datatype dict which consists of
the following elements:result={'status':'Status text','input_values':{'values':dict(),'mapped_ids':list()},'random_values':{'values':dict()},'p_values':{'values':dict()}}status: contains either an error message if a mapping failed or "ok" if IDs could be mappedinput_values:values: table in dict format with the functional or genetic relevance score(s) determined for solely their inputmapped_ids: list containing the IDs with non empty annotations per functional or genetic annotation typerandom_values:values: table in dict format with the functional or genetic relevance score(s) determined for all random runsp_values: table in dict format with the calculated empirical P-values using the selected background model and other parameters that indicate the significance of the calculated relevance scores derived from the inputSave and visualize resultsfrombiodigest.single_validationimportsave_resultsfrombiodigest.evaluation.d_utils.plotting_utilsimportcreate_plots,create_extended_plots# Save results into json file and 2 .csv table filessave_results(results:dict,prefix:str,out_dir)# Generate and save plots based on results# Consisting of p-value plot and mappability plotcreate_plots(results,mode,tar,tar_id,out_dir,prefix,file_type:str="pdf")# Generate and save extended plots based on results# Consisting of distribution and sankey plotscreate_extended_plots(results,mode,tar,out_dir,prefix,file_type:str="pdf",mapper:Mapper=FileMapper())ParametersRequired parametersresults: Is the output created with methodsingle_validationas data typedict()prefix: Prefix for file namesout_dir: Output directory for the generated filesAdditional required parameters for create_plotstar: Target input you want to be validateda cluster should be of typepd.DataFrame()withcolumns=["id","cluster"]a set should be of typeset()tar_id: Is the id type of the target (see possible options above)mode: Desired mode. See possible options above.Optional parameters for create_plotsfile_type: Type of the plots image files.Run significance contribution calculationIf you are interested in how the single ids from the target set are contribution to the
final calculated empirical P-values, you can run the significance contribution
calculations. Keep in mind the runtime will increase in a linear way based
on the number of ids in the input target set.frombiodigest.single_validationimportsignificance_contributionsresults_sig=significance_contributions(results:dict,tar:Union[pd.DataFrame,set],tar_id:str,mode:str,distance:str="jaccard",ref:Union[str,set]=None,ref_id:str=None,enriched:bool=False,mapper:Mapper=FileMapper(),runs:int=config.NUMBER_OF_RANDOM_RUNS,background_model:str="complete",replace=100,verbose:bool=False)Parametersresults: This is the output generated fromsingle_validationon the full input set.
This will be used to calculate the significance contribution of the single ids from the input set.all other paramters:look up parameters under "Run validation"Save and visualize resultsfrombiodigest.single_validationimportsave_contribution_resultsfrombiodigest.evaluation.d_utils.plotting_utilsimportcreate_contribution_plots,create_contribution_graphs# Save results into json file and .csv table files for each validation typesave_contribution_results(results:dict,prefix:str,out_dir)# Generate and save plots based on results# Consisting an overview heatmap of the top 15 ids# with the largest absolute overall significance contribution# and top 10 largest positive and negative# significance contribution per annotation typecreate_contribution_plots(result_sig,out_dir,prefix,file_type:str="pdf")# Generate a graph visualization from subnetwork constructed# by the input set and the given (or default) network# with nodes colored by their significance contributioncreate_contribution_graphs(result_sig,tar_id,network_data,out_dir,prefix,file_type:str="pdf",mapper:Mapper=FileMapper())ParametersRequired parametersresults: Is the output created with methodsignificance_contributionsas data typedict()prefix: Prefix for file namesout_dir: Output directory for the generated filesAdditional required parametersinput_type: "genes" or "diseases" based on id type of target setnetwork_data: Only for "subnetwork" and "subnetwork-set" mode. Dictionary consisting of {"network_file": path to network file,
"prop_name": name of vertex property with ids if network file of type graphml or gt,
"id_type": id type of network ids}mapper: Mapper object indicating where all files from the setup are saved.[Default=FileMapper()]Optional parameters for create_plotsfile_type: Type of the plots image files.Example runsCheck out thetutorialto see examples of usage in a script.