Package: pygenprop
PygenpropBranchStatusMasterDevelopDocsPygenprop is a python library for programmatic exploration and usage of theEBI Genome Properties database.FeaturesAt its core, the library contains five major components:An object model for representing the Genome Properties database as an in-memory rooted direct acyclic graphA parser for Genome Properties database release flat filesA parser for Genome Properties assignment long-form filesA parser for InterProScan TSV filesA results class that is used to assign genome properties to one or more organisms and compare assignments across multiple organismsAn extended results class that is used to explore the InterProScan annotations and protein sequences that support genome properties assignments across multiple organisms.Code for generatingMicromeda files.InstallationPygenprop is compatible with Python 3.6 or higher (3.5 may work, but it is not tested). Requirements can be found inenvironment.yaml.To install from PyPipipinstallpygenpropTo install from Condacondainstall-cconda-forge-clbergstrandpygenpropTo install from source (for development)cd/path/to/pygenprop_source_dir
pipinstall.Acquiring Genome Properties DataBefore Pygenprop can assign genome properties to an organism, it first has to gather information from the Genome Properties database. The easiest way to gain access is through the parsing of a Genome Properties Database release file. This file is found in theEBI Genome Properties Github repositoryand is calledgenomeProperties.txt. The file is located in the repository'sflatfiles folder. For each release of Genome Properties, agenomeProperties.txtfile is generated from the description files of allpublicgenome properties.Acquiring Release FilesgenomeProperties.txtfiles can be found at URLs in the compatibility section below using a web browser or UNIX commands such aswgetorcurl. They can also be streamed directly into Jupyter notebooks using therequests python library. Code for streaming the database into a Jupyter notebook can be foundhere.CompatibilityPygenprop will be continually updated to take into account changes in the schema of the Genome Properties database. Below is a compatibility table that maps between Genome Properties and Pygenprop releases.Genome Properties ReleasegenomeProperties.txt URLCompatible Pygenprop Release1.1https://raw.githubusercontent.com/ebi-pf-team/genome-properties/rel1.1/flatfiles/genomeProperties.txt>= 0.62.0https://raw.githubusercontent.com/ebi-pf-team/genome-properties/rel2.0/flatfiles/genomeProperties.txt>= 0.6Latesthttps://raw.githubusercontent.com/ebi-pf-team/genome-properties/master/flatfiles/genomeProperties.txt>= 0.6Accessing Non-public PropertiesThe./datafolder of theEBI Genome Properties Github repositorycontains a series of folders with information about both public and non-public genome properties. Each folder contains both a description (DESC) file and a status (status) file. The status file contains information on whether a property is public or not (public:    0means that a property is not public). One can use these status files to find non-public properties. The description files for these non-public properties can be parsed using the same parser as used forgenomeProperties.txt. Eachgenome property objectthat results from the parsing of a description file has an object attribute calledpublicwhich can be set totrueorfalseto designate a property as public or not.property_one.public=FalseAcquiring Annotation DataPygenprop can assign genome properties to an organism fromInterProScan annotation TSV files,Genome Properties long-form assignment files(created by the Genome Properties Perl library) or a list of InterPro consortium signature accessions downloaded into a Jupyter Notebook. Pre-calculated InterProScan results for UniProt proteomes and taxonomies can be downloaded (in signature accession list format) from thebeta version of the InterPro website.Example DataAn InterProScan TSV fileA Genome Properties longform assignment fileRunning InterProScanInterProScan generates InterProScan annotation TSV files via domain annotation of an organism's proteins. Details and install instructions for InterProScan5 can be foundhere. For convenience, a Docker container for installing and running InterProScan5 can be foundhere.Integrating Protein SequencesPygenprop can be used to extract protein sequences that provide evidence for an organism possessing a genome property.  To use this feature, the organism's proteome FASTA files that were annotated by InterProScan must be opened and passed to Pygenprop. See the workflow below for more details on using this feature.Micromeda FilesPygenprop can generate Micromeda files, which are a new SQLite3-based pathway annotation storage format that allows for the simultaneous transfer of multiple organism's Genome Properties assignments and supporting information. Examples of supporting information include the InterProScan annotations and protein sequences that support assignments. These files allow for the transfer of complete Genome properties Datasets between researchers and software applications.UsageProgrammatic Property Comparison With Jupyter NotebooksA typical use case for Pygenprop will involve a researcher seeking to compute and compare Genome Properties between organisms of interest. For example, a researcher may have discovered a novel bacterium and would want to compare its functional capabilities to other bacteria within the same genus. The researcher could start the analysis by opening up a Jupyter Notebook and directly importing pre-calculated InterProScan annotations for novel and reference genomes within the same genus. Below is an example code for comparing virulence genome properties ofE. coliK12 and O157:H7.An interactive Jupyter Notebook with an extended version of this workflow, with outputs for each step, can be foundhere. Full API documentation is availablehere.fromsqlalchemyimportcreate_enginefrompygenprop.resultsimportGenomePropertiesResults,GenomePropertiesResultsWithMatches,\load_assignment_caches_from_database,load_assignment_caches_from_database_with_matchesfrompygenprop.database_file_parserimportparse_genome_properties_flat_filefrompygenprop.assignment_file_parserimportparse_interproscan_file,\parse_interproscan_file_and_fasta_file# Compare Properties and Steps Across Organisms# =============================================# Parse the flatfile databasewithopen('properties.txt')asfile:tree=parse_genome_properties_flat_file(file)# Parse InterProScan fileswithopen('E_coli_K12.tsv')asipr5_file_one:cache_1=parse_interproscan_file(ipr5_file_one)withopen('E_coli_O157_H7.tsv')asipr5_file_two:cache_2=parse_interproscan_file(ipr5_file_two)# Create results comparison objectresults=GenomePropertiesResults(cache_1,cache_2,properties_tree=tree)# Get properties with differing assignmentsdiffering_results=results.differing_property_results# Get property by identifiervirulence=tree['GenProp0074']# Iterate to get the identifiers of# child properties of virulencetypes_of_vir=[genprop.idforgenpropinvirulence.children]# Get assignments for virulence propertiesvirulence_assignments=results.get_results(*types_of_vir,steps=False)# Get percentages of virulence steps assigned# YES, NO, and PARTIAL per organismvirulence_summary=results.get_results_summary(*types_of_vir,steps=True,normalize=True)# Analyze InterProScan Annotations and Protein Sequences# That Support Genome Properties Across Organisms# ==================================================# Parse InterProScan files and FASTA fileswithopen('./E_coli_K12.tsv')asipr5_file_one:withopen('./E_coli_K12.faa')asfasta_file_one:extended_cache_one=parse_interproscan_file_and_fasta_file(ipr5_file_one,fasta_file_one)withopen('./E_coli_O157_H7.tsv')asipr5_file_two:withopen('./E_coli_O157_H7.faa')asfasta_file_two:extended_cache_two=parse_interproscan_file_and_fasta_file(ipr5_file_two,fasta_file_two)# Create results comparison object with InterProScan match information# and protein sequencesextended_results=GenomePropertiesResultsWithMatches(extended_cache_one,extended_cache_two,properties_tree=tree)# Get lowest E-value matches for each Type III Secretion System component for E_coli_O157_H7.extended_results.get_property_matches('GenProp0052',sample='E_coli_O157_H7',top=True)# Get all matches for step 22 of Type III Secretion for E. coli K12.extended_results.get_step_matches('GenProp0052',22,top=False,sample='E_coli_K12')# Write FASTA file containing the sequences of the lowest E-value matches for# Type III Secretion System component 22 across both organisms.withopen('type_3_step_22_top.faa','w')asout_put_fasta_file:extended_results.write_supporting_proteins_for_step_fasta(out_put_fasta_file,'GenProp0052',22,top=True)# Create a SQLAlchemy engine object for writing a Micromeda file.engine_proteins=create_engine('sqlite:///ecoli_compare.micro')# Write the results to the file.extended_results.to_assignment_database(engine_proteins)# Load results from a Micromeda file with proteins sequences.assignment_caches_with_proteins=load_assignment_caches_from_database_with_matches(engine_proteins)results_reconstituted_with_proteins=GenomePropertiesResultsWithMatches(*assignment_caches_with_proteins,properties_tree=tree)Command-line Interface (CLI)The command-line interface of Pygenprop is used primarily for generating and working withMicromeda files. It possesses three sub-commands and is installed when Pygenprop is installed.usage: pygenprop [-h] {build,merge,info,preprocess} ...

A command-line interface for generating and manipulating Micromeda pathway annotation files.

positional arguments:
  {build,merge,info,preprocess}
                        Available Sub-commands
    build               Generate a Micromeda file containing pathway annotations for one or more genomes. Supporting InterProScan and protein sequence information can also be optionally incorporated.
    merge               Merge multiple Micromeda files into a single output Micromeda file.
    info                Summarize the contents of a Micromeda file.
    preprocess          Replace FASTA header accessions with a numeric identifiers.

optional arguments:
  -h, --help            show this help message and exitThebuildcommand is used to generate Micromeda files. It requires a copy ofgenomeProperties.txt. InterProScan TSV files are used as input.pygenpropbuild-d./genomeProperties.txt-i*.tsv-oecoli_genomes_properties.microThebuildcommand has a-pflag that is used to add protein sequences to the output Micromeda file. With this flag active, Pygenprop searches the FASTA files that were scanned by InterProScan for proteins that support genome property steps and adds them to the output Micromeda file. The FASTA files must be in the same directory as the InterProScan files and share the same basename (e.g., filename without file extension).data/
├── ecoli_one.faa
├── ecoli_one.tsv
├── ecoli_two.faa
├── ecoli_two.tsvFor the above directory structure the following shell command would be used to generate a Micromeda file that integrates protein sequences:pygenpropbuild-d./genomeProperties.txt-i*.tsv-oecoli_genomes_properties.micro-pThemergecommand is used to merge multiple Micromeda files into a single output Micromeda file. It also requires a copy ofgenomeProperties.txt.pygenpropmerge-d./genomeProperties.txt-i*.micro-omerged_ecoli_genomes_properties.microTheinfocommand is used to get a summary of a Micromeda file's contents.pygenprop info -i merged_ecoli_genomes_properties.micro

    The Micromeda file contains the following:

    Samples: 2
    Property Assignments: 2572
    Step Assignments: 4644
    InterProScan Matches: 2843
    Protein Sequences: 1887DocumentationDocumentation can be found onRead the Docs.Trouble ShootingPlease report issues to theissues page.LicenceApache License 2.0Current ContributorsLee BergstrandPast ContributorsN/A