Package: cazomevolve
cazomevolvecazomevolve('cazome-evolve') is an application and Python3 package for the automated annotation and exploratory analysis of CAZyme complements (CAZomes) for a set of species and/or genomes of interest. Carbohydrate Active enZymes are a subset of proteins that generate, modify and/or degrade carbohydrates. CAZy (www.cazy.org) is the most comprehensive CAZyme database, grouping proteins by sequence similarity into CAZy families.Usecazomevolveto explore:CAZome sizes:Compare the number of CAZymes and CAZy familiesCalculate the proportion of the proteome encompassed by the CAZomeCompute the CAZy family to CAZyme ratioCAZy class frequencies:Calculate the number of CAZymes per CAZy classPlot a proportional area plot of CAZy class frequency broken down by CAZy class and user defined group deliniations (e.g. by genus or species)CAZy families:Exploresequence diversitywithin a set of CAZy families:Run all-vs-all sequence comparison analysesCluster the sequences by degree of sequence similarityGenerate clustermaps of sequence identity, BLAST-score ratio, and coverageExploreCAZy family frequencies:Compute the number of CAZymes per CAZy familyIdentifylineage or group specific families- e.g. genus or species specific familiesIdentifycore CAZomes- families that appear in all genomesCluster genomes by CAZy family frequencies usinghierarchical clustering:Generate annotated clustermaps of CAZy family frequenciesBuild a dendogram using distances calculated from the CAZome compositionConstruct tanglegrams to compare CAZy family dendrogram to a ANI-dendrogram or phylogenetic treeAlways co-occurring families:Identity CAZy families that are always present in the CAZome togetherFind lineage or group specific groups of co-occurring familiesConstruct an upset plot of co-occurring familiesCalculate the number of genomes each group of co-occurring families appear inPrincipal component analysis (PCA):Use PCA to idenify overal trends in the large and complex data setProject genomes onto use selected principal components (PCs)Construct loadings plots to explore correlation between CAZy families and PCsIdentify relationships between groups of CAZy familiesExplore associations between CAZy families and lineage, phenotype, and niche adaptationCo-evolving CAZy families:Generate the input file tab delimited list of genomes and CAZy families required bycoinfinder(Whelanet al.)Optionally add taxonomic data to the tab delimited list, to include taxa in thecoinfinderoutputReconstruct phylogenetic trees to be used as input bycoinfinder:Reconstruct a multi-gene phylogenetic tree usingRaxML-ngConstruct an ANI-based dendrogramUsecoinfinderto identify CAZy families that appear together in a genomemore often than expected by lienage and chanceKozlov AM, Darriba D, Flouri T, Morel B, Stamatakis A. RAxML-NG: a fast, scalable and user-friendly tool for maximum likelihood phylogenetic inference. Bioinformatics. 2019 Nov 1;35(21):4453-4455. doi: 10.1093/bioinformatics/btz305.
Whelan FJ, Rusilowicz M, McInerney JO. Coinfinder: detecting significant associations and dissociations in pangenomes. Microb Genom. 2020 Mar;6(3):e000338. doi: 10.1099/mgen.0.000338. Epub 2020 Feb 24.Documentationcazomevolveuses bash-script based workflow management. A summary of this workflow is provided in this README.Please see thefull documeentation including tutorials at ReadTheDocs.An analysis usingcazomevolvecan be foundhere, which includes a README-walkthrough and all output files.ContentscazomevolveDocumentationInstallationRequirementsExplore sequence diversity in CAZy familiesConstruct a local CAZyme databaseGet sequencesRun all-versus-all analysisVisualise sequence diversityAnnotate the CAZomeDownload genomesUse genomic accessionsUsing taxonomiesAnnotateLocal CAZyme databaseCAZy annotationsdbCAN annotationsInvoke dbCANGet annotationsExplore the CAZomeNetworks of co-evolving CAZymesMulti-gene phylogenetic tree reconstructionANI distance-based treeNetworks of co-evolving CAZymesCAZome dendrogramsInstallationPyPiThe easiest way to installcazomeolveis via PyPipipinstallcazomevolveFrom sourceCreate a virtual environment with dependencies, then activate the environment -where venv_name is an chosen name for the virtual environmentcondacreate-ncazomevolvepython=3.9
condaactivatecazomevolveClone the repositorygitclonehttps://github.com/HobnobMancer/cazomevolve.git
pipinstallcazomevolve/.RequirementsEssentialPOISx or Mac OS, or linux emulatorPython version 3.8+
Miniconda3 or Anaconda managed microenvironment, incorporated code checkers are included in list form in 'requirements.txt'.Miniconda3 environment file is also available in the GitHub repository: 'environment.yml'.For all required Python libraries please read 'requirements.txt'.The following packages are required by the corecazomevolveapplication, and their installation is handled by thesetup.pyfile:adjustTextbiopythoncazy_webscraper`jupyterncbi-genome-downloadnumpypandassaintBioutilsseabornsklearn/scikit-learntqdmupsetplotOptionalThe following packages are optional can installation instructions can be found in their respective repositories:dbCAN:To expand the data set beyond those CAZomes only listed in CAZy (highly recommended!)dbCANmust also be installed.Note:dbCANandcazomevolvecan be installed in the same or separate virtual environments.dbCAN version 3 and 4:The CAZyme classifierdbCANversions >= 3.0.6 can be installed via Conda (recommended). The full installation instructions are foundhere, and must be followed to ensure all additional database files are downloaded and compiled correctly.dbCAN version 2:The installation instructions for dbCAN v==2.0.11 can be foundhere.coinfinder: Identify CAZy families that co-occurr more often than expected by lineage and chance:coinfinder>= v1.2.0Note: _coinfinderrequires Python v3.6, we recommend installing and runningcoinfinderin a separate venv tocazomevolveConstruct an ANI-based dendrogram:pyani>= version 0.3.0-alphaRReconstruct a multi-gene phylogenetic tree:MAFFTOrthofinderProdigalRaxML-ngT-coffeeExplore sequence diversity in CAZy familiesConstruct a local CAZyme databaseDownload all CAZyme records from CAZy, and compile the records into a local SQLite3 database usingcazy_webscraper:cazy_webscraper<user-email-address>-o<desired-path-for-db>Get protein sequencesPresuming a local CAZyme database has already been generated usingcazy_webscraper:Generate a multisequence FASTA file for each CAZy family of interest using the bash scriptget_fam_seqs.sh, which takes 4 positional arguments:email addresspath to a local cazyme dbname(s) of families of interest (separate with a single comma)path to an output dir (do not include terminal /)cazomevolveget_fam_seqs\<email>\<cazy_db>\<fam1,fam2,fam3>\<pathtooutdir>The output dir will be created bycazy_webscraper- it will not delete exsiting content in the outdir unless their is a FASTA file with the same nameCreates one output FASTA file per CAZy familyOr usecazy_webscraperdirectly to create a multisequence FASTA file containing the protein sequences of interstRun all-vs-all analysisRun all-vs-all sequence analysis for each multisequence FASTA file, using BLAST or DIAMOND (recommend for large families of >1000 proteins sequences).The output directories will be created bycazomevolve- existing data in the existing output directories willnotbe deleted.Using BLASTP:Use therun_fam_blastsubcommand, which takes 2 positional arguments:Path to the input FASTA filePath for the output TSV filecazomevolverun_fam_blast\<inputfastafile>\<outputTSVfile>Using DIAMOND: (recommended for large datasets):Use therun_fam_diamondsubcommand, which takes 3 positional arguments:Path to the input FASTA filePath where to create a diamond dbPath to write out output matrixcazomevolverun_fam_diamond\<inputfastafile>\<diamonddbpath>\<outputTSVfile>Visualise the sequence diversityVisualise the results using thejupyter notebooktemplate located atcazomevolve/seq_diversity/explore_seq_diversity.ipynb. This generates clustermaps and heatmaps that plot the proteins in the same order as the generated clustermap.All functions used in the notebook are available, and can be imported from,cazomevolve, specifically the modulecazomevolve.seq_diversity.explore.We recommend using theBLAST Score Ratio (BSR)to generate a clustermap, then generate heatmaps of the percentage identity (pident) and query coverage (qcov) so the proteins are plotted in the same order for the 3 plots and thus facilitates comparing between the three.Optionally, redundant protein sequences can be removed, and proteins of interest (mannually defined by the user) and functionally/structurally characterised proteins can be annotated on the plots, to facilitate identifying the degree of characterisation across a family.Annotate the CAZomeDownload genomesThe genomes to be download can be specified by [A] their genomic accessions, or [B] by specifying a taxa of interest (using a taxa of any level).[A] Genomic accessionsIf you have a list of genomic version accessions in a plain text file,cazomevolvecan use the Python packagencbi-genome-downloadto download the genomic assemblies genomic (.fna) and proteome (.faa) sequence files.Using thedownload_acc_genomessubcommand, which takes 5 positional arguments:Positional arguments:accessions- Path to file listing the accessions, with a unique genome accession per rowoutdir- Output directory to write out the genomes tofile_opts- File options - the file foramts to download the genomic assemblies in. Chose from:genbankfastarmfeaturesgffproteingenpeptwgscdsfastarnafnadatabase- NCBI database - the database to retrieve the assemblies from, GenBank or RefSeq:refseqorgenbankTo download the protein sequences of all annotated protein sequences, download the  assembles inproteinformat.If you are going to annotate the genomes, download the genomes infasta(genomic sequence) formate.Optional arguments:-A,--assembly_levels- limit the download to assemblies with the assembly status provided. A space-separate lists of assembly levels. Can provide multiple levels: Accepted levels:completechromosomescaffoldcontig-f,--forceForce file over writting (default: False)-n,--nodeleteenable/disable deletion of exisiting files (default: False)By default if the output directory exists,cazomevolvewill crash. To write to an existing output directory use the-f/--forceflag. By default,cazomevolvewill delete all existing data in the existing output directory. To retain the data available in the existing output directory use the-n/--nodeleteflag.[B] TaxaTo download load all genomic assemblies associated with a term of interest, such asPectobacteriaceae(so as to download all Pectobacteriaceae assemblies), use the subcommanddownload_genomes, which takes 4 arguments:Positional arguments:email - User email addressoutput_dir - Path to directory to write out genomic assembliesterms - Terms to search NCBI. Comma-separated listed, e.g, 'Pectobacterium,Dickeya'. To include spaces in terms, encapsulate the all terms in quotation marks, e.g. 'Pectobacterium wasabiae'file format: {genomic,protein}- Space-separated list of file formats to dowload. ['genomic' - downloads genomic.fna seq files, 'protein' - downloads protein.faa seq files]NCBI database: {genbank,refseq} - Choose which NCBI db to get genomes from: refseq or genbankOptional arguments:-A,--assembly_levels- limit the download to assemblies with the assembly status provided. A space-separate lists of assembly levels. Can provide multiple levels: Accepted levels:completechromosomescaffoldcontig-f,--force- Force file over writting (default: False)-n,--nodelete- enable/disable deletion of exisiting files (default: False)-l,--log- path to write out log file-v,--verbose- Set logger level to 'INFO' (default: False)--timeout- time in seconds before connection times out (default: 30)By default if the output directory exists,cazomevolvewill crash. To write to an existing output directory use the-f/--forceflag. By default,cazomevolvewill delete all existing data in the existing output directory. To retain the data available in the existing output directory use the-n/--nodeleteflag.Annotate CAZomesTo retrieve the most comprehensive annotation of the CAZome, we recommend using the (widely considered) canonical classifications from CAZy retrieved usingcazy_webscraper(Hobbset al.,2022), combined with predicted CAZy family annotations fromdbCAN(Zhanget al.2018).Emma E. M. Hobbs, Tracey M. Gloster, Leighton Pritchard; cazy_webscraper: local compilation and interrogation of comprehensive CAZyme datasets, BioRxiv, 3 December 2022,https://doi.org/10.1101/2022.12.02.518825Han Zhang, Tanner Yohe, Le Huang, Sarah Entwistle, Peizhi Wu, Zhenglu Yang, Peter K Busk, Ying Xu, Yanbin Yin; dbCAN2: a meta server for automated carbohydrate-active enzyme annotation, Nucleic Acids Research, Volume 46, Issue W1, 2 July 2018, Pages W95–W101,https://doi.org/10.1093/nar/gky418Build a local CAZyme database usingcazy_webscraperTo include 'canonical' CAZy family classifications from CAZy, download all data from the CAZy database and compile the data into a local CAZyme database usingcazy_webscraper(Hobbs _et al., 2022).cazy_webscraper: local compilation and interrogation of comprehensive CAZyme datasets
Emma E. M. Hobbs, Tracey M. Gloster, Leighton Pritchard
bioRxiv 2022.12.02.518825; doi:https://doi.org/10.1101/2022.12.02.518825Use thecazomevolvesubcommandbuild_cazy_dbto coordinate uisngcazy_webscraper:cazomevolvebuild_cazy_db\<email>\<desiredpathfordbFILE>Note the path needs to point to the target FILE path not DIR path.cazy_webscraperwill build all necessary parent directories.Or you can usecazy_webscraperdirectlycazy_webscraper\<email>\-o<dbfileoutputpath>Retrieve CAZy annotationsTheget_cazy_cazymessubcommand to coordinatecazomevolveto query the protein version accessions in the downloaded protein FASTA files against the local CAZyme db, to retrieve the 'canonical' CAZy family classifications:cazomevolveget_cazy_cazymes\<pathtodircontainingproteinFASTAfiles>\<pathtolocalcazymedatabase>\<pathtodirtowriteoutproteinsequencesNOTinthelocaldb>\<pathtowriteouttabdelimitedlistsofCAZyfamiliesandgenomicaccessions>\<pathtowriteouttabdelimitedlistsofCAZyfamilies,genomicaccessionsandproteinaccessions>\Two tab delimited lists are generated, containing:Listing the CAZy family accession and genomic accession per linefam1genome1
fam2genome1
fam1genome2
fam3genome2Listing the CAZy family, genomic accession and protein accession per linefam1genome1protein1
fam2genome1protein1
fam1genome2protein2
fam3genome2protein3Optional args:options:-h,--helpshowthishelpmessageandexit-f,--forceForcefileoverwritting(default:False)-llogfilename,--loglogfilenameDefineslogfilenameand/orpath(default:None)-n,--nodeleteenable/disabledeletionofexisitingfiles(default:False)--sql_echoSetverboseSQLite3logging(default:False)-v,--verboseSetloggerlevelto'INFO'(default:False)Invoke dbCANeCAMIis memory intensive. We recommend using the maximum availalbe RAM.dbCANcan be automated to parse all FASTA files in a directory (e.g. all download protein FASTA files or FASTA files of proteins not in a local CAZyme database), using thecazomveolve, subcommandrun_dbcancommand.cazomevolverun_dbcan\<pathtodircontainingFASTAfiles>\<pathtooutputdirectory>\<dbcanmajorversionnumber,2,3or4>cazomevolvewill which ever version of dbCAN is installed, but the commands and arguments between dbCAN version 2, 3 and 4 are different, socazomevolvemust be told which version to of dbCAN to communicate with.The ouput directory will be created byrun_dbcan.Inside the output directory, for each FASTA file parsed bydbCANan output subdirectory will be created (named after the genomic version accession) and will contain the output fromdbCANfor the respective protein FASTA file.Optional args:options:-h,--helpshowthishelpmessageandexit--cpuCPUNumberofCPUcorestouse.Defaultallavailablecores(default:allavilablecores)-f,--forceForcefileoverwritting(default:False)-llogfilename,--loglogfilenameDefineslogfilenameand/orpath(default:None)-n,--nodeleteenable/disabledeletionofexisitingfiles(default:False)-v,--verboseSetloggerlevelto'INFO'(default:False)Retrieve dbCAN annotationsAfter running dbCAN, use thecazomevolvesubcommandget_dbcan_cazymesto iterate through the output subdirectories created bycazomevolve run_dbcanand compile the data into two tab delimited lists, containing:Listing the CAZy family accession and genomic accession per linefam1genome1
fam2genome1
fam1genome2
fam3genome2Listing the CAZy family, genomic accession and protein accession per linefam1genome1protein1
fam2genome1protein1
fam1genome2protein2
fam3genome2protein3If paths to the tab delimited lists created bycazomevolve get_cazy_cazymesare provided, the dbCAN classifications will beaddedthe existing tab delimited lists, and will notoverwritethe data in the files (make sure to include the-f/--forceand-n/--nodeleteflags when wanting to add data to existing tab delimited files).cazevolve_get_dbcan_cazymes\<pathtodbCANoutputdir(continingoutputsubdirs)>\<pathtowriteouttabdelimitedlistsofCAZyfamiliesandgenomicaccessions>\<pathtowriteouttabdelimitedlistsofCAZyfamilies,genomicaccessionsandproteinaccessions>Optional args:options:-h,--helpshowthishelpmessageandexit-f,--forceForcefileoverwritting(default:False)-llogfilename,--loglogfilenameDefineslogfilenameand/orpath(default:None)-n,--nodeleteenable/disabledeletionofexisitingfiles(default:False)-v,--verboseSetloggerlevelto'INFO'(default:False)Add taxonomic informationTo include taxonomic information in the exploration of the CAZomes, taxonomic information needs to be added to the tab separated files
of CAZy families, genomic accessions and protein accessions.cazomevolveretrieves taxonomic classifications from NCBI or GTDB (as specified by the user), and
adds the taxonomic information to the respective genomic accession in the tab separated files.cazomevolveseparates the genomic accession and each rank of the taxonomic information with an underscore.
For example, if genus and species inforamtion was retrieved from NCBI, the output tab separated files would
contain:CBM50GCA_003382565.3UEM40323.1GT35GCA_003382565.3UEM39157.1GH5GCA_003382565.3UEM41238.1CBM3GCA_003382565.3UEM41238.1CE12GCA_003382565.3UEM40541.1GT2GCA_003382565.3UEM39295.1...and...CBM50GCA_003382565.3_Pectobacterium_aquaticumUEM40323.1GT35GCA_003382565.3_Pectobacterium_aquaticumUEM39157.1GH5GCA_003382565.3_Pectobacterium_aquaticumUEM41238.1CBM3GCA_003382565.3_Pectobacterium_aquaticumUEM41238.1CE12GCA_003382565.3_Pectobacterium_aquaticumUEM40541.1GT2GCA_003382565.3_Pectobacterium_aquaticumUEM39295.1cazomevolve add_taxsdoesnotoverwrite the existing tab separated lists.cazomevolve add_taxsextracts the data from the tab separated files, adds the taxonomic inforamtion
to the genomic accession in the files, and writes out the data to new files. These files are given the
same file path as the tab separated files, with the addition of_taxson the end.
Therefore, the input filedata/fams_genomesbecomesdata/fams_genomes_taxs.A CSV file listing the taxonomic information is also generated. By default this is written to the
same directory as the tab separated files and calledtaxonomies.csv. To specify a different file
path for the CSV file, use the `--outpath`` flag followed by the desired file path.Positional argument:Taxonomic information from NCBI or the Genome Taxonomy DatabaseGTDB, can be
added to the tab separated files using the subcommandadd_taxs.The only position argument is a user email address (which is required by NCBI).Tab separated files:Either the--FG_FILEand/or--FGP_FILEflags must be called:Use the--FG_FILEto provide a path to the tab separated file ofCAZy families and genomic accessions, to add taxonomic data to this file... code-block:: bashcazomevolve add_taxs dummy@domain.com \
    --FG_FILE data/fams_genomes_fileUse the--FGP_FILEto provide a path to the tab separated file ofCAZy families, genomic accessions and protein accessions, to add taxonomic data to this file... code-block:: bashcazomevolve add_taxs dummy@domain.com \
    --FGP_FILE data/fams_genomes_proteins_fileTaxonomic data can be added to both tab separated files by using the--FG_FILEand--FGP_FILEflags. For
example, if the tab separated files were stored in a directory calleddata/.cazomevolveadd_taxsdummy@domain.com\--FG_FILEdata/fams_genomes_file\--FGP_FILEdata/fams_genomes_proteins_file\Specify lineage ranks of interst:At least one rank or level of taxonomic lineage must be specified for inclusion in the tab separated files
of CAZy families and genomic accessions.To specify which ranks of lineage to retrieves and add to the tab separated files, add each respective
flag to the command:--kingdom--phylum--tax_class--tax_order--tax_family--genus--speciesFor example, to retrieve family, genus and species information for genomes listed
in a tab separated file, use the--tax_family,--genus, and--speciesflags:cazomevolveadd_taxsdummy@domain.com\--FG_FILEdata/fams_genomes_file\--FGP_FILEdata/fams_genomes_proteins_file\--tax_family\--genus\--speciesThe order the lineage ranks are specified does not matter.cazomevolve add_taxswill
always write out the lineage ranks in the true phylogenetic order: kingdom, phylum, class, order,
family, genus, and species.Note: 'Species' taxonomic information includes the strain information.GTDB or NCBI:By defaultcazomevolve add_taxsretrieves the latest taxonomic classification from NCBI for each genome
in each of the provided tab separated files.To instead use taxonomic classifications from the GTDB database (applicable for bacteria and archaea),
download a TSV database dump from theGTDB release server. Then
callcazomevolve add_taxsand include the--gtdbflag in the call, followed by the path to the TSV file
GTDB database dump. For example:cazomevolveadd_taxsdummy@domain.com\--FG_FILEdata/fams_genomes_file\--FGP_FILEdata/fams_genomes_proteins_file\--gtdbdownloads/gtdb/bac120_taxonomy.tsvOperational arguments-f,--force-  Force file over writting (default: False)-n,--nodelete- enable/disable deletion of exisiting files (default: False)-l,--log- path to write out log file-v,--verbose- Set logger level to 'INFO' (default: False)--retries- number of times to retry connection to NCBI if connection failsExplore the CAZome compositionThecazomevolvesubcommand provides a method for exploring CAZome compositions, calculating:The number of CAZymes per genome, and mean and SD calculated per user defined group (e.g. genus)Number of CAZy families per genome, and mean and SD calculated per user defined groupTotal number of proteins in each genome, per genome, and mean and SD calculated per user defined groupPercentage of the proteome that encapsulates or consistutes the CAZomes per genome, and mean and SD calculated per user defined groupNumber of CAZymes per CAZy class, per genome, and mean and SD calculated per user defined groupPercentage of the CAZome represented by each CAZy class, per genome, and mean and SD calculated per user defined groupNumber of CAZymes per CAZy family in each genomeAnalyse CAZy family frequencies using heirarchical clustering and generate a clustermapIdentify the core CAZome - CAZy families present in all genomes and per user defined groupIdentify CAZy families that always co-occur in the genome together, although each group of co-occurring CAZy families may not be present in all genomesRun Principal Component Analysis identify associations between user defined groups of genomes (e.g. genera), and CAZome compositionsPlots scatters plots projecting genomes onto all pairs of PCs from PC1-4, genomes are colour coded by user defined group (e.g. genus)Plots a corresponding loadings plot for each PCA scatter plotCommand line argumentsRequiredPath to tab delimited FGP file, listing CAZy families, genomic accessions, and protein IDsPath CSV file listing taxonomic data, containing a column called 'Genome', listing genomic accessions, and one column per taxonomic rank retrieved from NCBI and/or GTDB.Directory to write out all outputsEach of the taxonomic ranks included in the CSV file of taxonomic data must also be specified, by adding each of the relevant flags to command:--kingdom--phylum--tax_class--tax_order--tax_family--genus--speciesFor example, if genus and species inforamtion was listed in the CSV of taxonomy data:cazomevolveexplore_cazomes\data/cazomes/gfp_file.txt\data/taxs/tax.csv\results/\--genus\--speciesOptional--show_plots- Display plots generated as the program is executing (default: False)--round_by- ROUND_BY - Number of decimal places to round means and SDs to (default: 2)-f,--force- Force file over writting (default: False)-l,--loglog file name - Defines log file name and/or path (default: None)-n,--nodelete- enable/disable deletion of exisiting files (default: False)-v,--verbose- Set logger level to 'INFO' (default: False)Full customisation of CAZome explorationFor full customisation of the exploration import thecazomevolve.cazome.exploreinto a jupyter notebook.Full customisation includes:Adhock define groups of interestExclude specific groups of interest from specific analysesAlter figure sizesAdd additional and custom annotations (e.g. colour code by genomes by plant host)You can find an example notebook presented as awebsite hereand theraw notebook here.The modulecazomevolve.cazome.explorecontains functions for exploring the CAZome annotated bycazomevolve. These are:# loading and parsing datafromcazomevolve.cazome.explore.parse_dataimport(load_fgp_data,load_tax_data,add_tax_data_from_tax_df,add_tax_column_from_row_index,)# functions for exploring the sizes of CAZomesfromcazomevolve.cazome.explore.cazome_sizesimport(calc_proteome_representation,count_items_in_cazome,get_proteome_sizes,count_cazyme_fam_ratio,)# explore the frequency of CAZymes per CAZy classfromcazomevolve.cazome.explore.cazy_classesimportcalculate_class_sizes# explore the frequencies of CAZy families and identify the co-cazomefromcazomevolve.cazome.explore.cazy_familiesimport(build_fam_freq_df,build_row_colours,build_family_clustermap,identify_core_cazome,plot_fam_boxplot,build_fam_mean_freq_df,get_group_specific_fams,build_family_clustermap_multi_legend,)# functions to identify and explore CAZy families that are always present togetherfromcazomevolve.cazome.explore.cooccurring_familiesimport(identify_cooccurring_fams_corrM,calc_cooccuring_fam_freqs,identify_cooccurring_fam_pairs,add_to_upsetplot_membership,build_upsetplot,get_upsetplot_grps,add_upsetplot_grp_freqs,build_upsetplot_matrix,)# functions to perform PCAfromcazomevolve.cazome.explore.pcaimport(perform_pca,plot_explained_variance,plot_scree,plot_pca,plot_loadings,)Two example jupyter notebooks are available which demonstrate usingcazomevolveto explore, interogate, compare and visualise the CAZyme complement:Notebook exploring the CAZomes ofPectobacteriaceaeNotebookWebsiteNotebook exploring the CAZomes ofPectobacteriumandDickeyaNotebookWebsiteIdentify networks of co-evolving CAZy families usingcoinfinderAny phylogentic tree written in Newick format can be used bycoinfinder. To help out those new to phylogenetics,cazomevovleincludes two sets of bash scripts for two alternative methods for creating trees.Maximumlikelihood multi-gene treeTo reconstruct a multi-gene phylogenetic tree, we recommend following the method presented inHugouviux-Corre-Pattat et al.. The specific method they used can be found in theSI.To facilitate reconstructing the phylogenetic tree,cazomevolveincludes a series of bash scripts which are available in the repository to coordinate the process.An example of using very similar scripts can be found inHobbs et al.CDS predictionTo ensure consistency of nomenclature and support back threading the nucleotides sequences onto aligned single-copy orthologues, reannotate all prokaryotic and archaea genomes usingprodigalHyatt D, Chen GL, Locascio PF, Land ML, Larimer FW, Hauser LJ. Prodigal: prokaryotic gene recognition and translation initiation site identification. BMC Bioinformatics. 2010 Mar 8;11:119. doi: 10.1186/1471-2105-11-119. PMID: 20211023; PMCID: PMC2848648.scripts/tree/phylo/annotate_genomes.sh\<outputdir>The output directory is created by the bash scriptThe annotate features were written to the following directories:Proteins:.../proteinsCDS:.../cdsGBK:.../gbkIdentify Single Copy Orthologues (SCOs)Orthologues present in the genomes are identified usingorthofinder.Emms, D.M. and Kelly, S. (2019) OrthoFinder: phylogenetic orthology inference for comparative genomics.Genome Biology 20:238scripts/tree/phylo/find_orthologues.sh\<Pathto.../proteinsdircreatedbyannotate_genomes.sh>\<pathtooutputdir>The output dir will be created by orthofinder.Multiple Sequence AlignmentEach collection of single-copy orthologous was aligned usingMAFFT.The output fromMAFFT(the aligned files) are placed in thedata/pecto_dic/tree/sco_proteins_aligneddirectory.scripts/tree/phylo/align_scos.sh<pathtodircontainingSCOseqsfromorthofinder>Collect Single-Copy Orthologues CDS sequencesThe CDS sequences corresponding to each set of single-copy orthologues are identified and extracted with the Python scriptextract_cds.py.python3scripts/tree/phylo/extract_cds.py\<Pathto'Single_Copy_Orthologue_Sequences'intheorthofinderoutputdir>\<PathtoCDSsequenceannotatedbyProdigal>\<Outputdir>The output directory will be created by the script.The output is a set of unaligned CDS sequences corresponding to each single-copy orthologue, which are
placed in thedata/pecto_dic/tree/sco_cdsdirectoryBack-translate Aligned Single-Copy OrthologuesThe single-copy orthologue CDS sequences are threaded onto the corresponding aligned protein sequences usingt-coffee, coordinated using thebacktranslate.shscript.T-Coffee: A novel method for multiple sequence alignments. Notredame, Higgins, Heringa, JMB, 302(205-217)2000scripts/tree/phylo/backtranslate.sh\<pathtodircontainingproteinMSAfromMAFFT>\<outputdir-willcontaincodonMSA>The output dir will be made by the bash script.Concatenating CDS into a Multigene AlignmentThe threaded single-copy orthologue CDS sequences were concatenated into a single sequence per input organism using the Python scriptconcatenate_cds.py. To reproduce this, execute the script from this directory with:python3scripts/tree/phylo/concatenate_cds.py\<Pathtogenomedir>\<PathtoCDSsequenceannotatedbyProdigal>\<Outputdir>Two files are generated, a FASTA file with the concatenated multigene sequences, and a partition file allowing a different set of model parameters to be fit to each gene in phylogenetic reconstruction.Phylogenetic reconstructionTo reconstruct the phylogenetic tree, the bash scriptraxml_ng_build_tree.shis used, and is run from the root of this repository. This executes a series ofraxml-ngcommands.All genes are considered as separate partitions in the reconstuction,
with parameters estimated  for the model recommended byraxml-ng check.Tree reconstructions are placed in the output directory. The best estimate tree is listed in03_infer.raxml.bestTree.Alexey M. Kozlov, Diego Darriba, Tomáš Flouri, Benoit Morel, and Alexandros Stamatakis (2019) RAxML-NG: A fast, scalable, and user-friendly tool for maximum likelihood phylogenetic inference. Bioinformatics, btz305doi:10.1093/bioinformatics/btz305scripts/tree/phylo/raxml_ng_build_tree.sh\<pathtocontenatedfastafile>\<pathtopartitionfile>\<pathtooutputdir>The output directory is created by the scriptA distance based approachAn alternative approach is to calculate genome distances from the Average Nucleotide Identity (ANI).The software packagepyaniPritchard et al.can be used to calculate the ANI between all possible pairs of genomes, for a set of given genomes.Pritchard et al. (2016) "Genomics and taxonomy in diagnostics for food security: soft-rotting enterobacterial plant pathogens" Anal. Methods 8, 12-24An example of usingpyanito generate a ANI-based dendrogram can be found inHobbs et al.The installation instructions forpyanican be found in itsGitHub repo. We recommend using >= version 0.3.0-alpha.cazomevolveincludes bash scripts to coordinate usingpyani:scripts/tree/ani/run_anim.sh\<pyaniandlogfileoutputdirectory>\<dircontaininggenome.faaseqs>\<plotandmatrixoutputdir>The R scriptbuild_anim_tree.R(inscripts/tree/ani/) can be used and modified to create to infer genome distances from the all-vs-all ANI analysis and construct a dendrogram from the distances.Find networks of co-evolving CAZy familiesUse the Python packagecoinfinder(Whelan et al., 2020) to identify networks of co-evolving CAZy families.Fiona J. Whelan, Martin Rusilowicz, & James O. McInerney. "Coinfinder: detecting significant associations and dissociations in pangenomes." doi:https://doi.org/10.1099/mgen.0.000338See thecoinfinderdocumentationfor details.To customise the resulting phylogenetic tree and heatmap, edit the R scriptnetwork.Rincoinfinder.An example of where this is done can be found inHobbset al.SI information on the exploration ofPectobacteriaceaeCAZomes.Build dendograms based upon CAZome compositions, and compare against the phylogenetic tree....