Package: opengenomebrowser-tools
OpenGenomeBrowser ToolsA set of scripts that helps to import genome data into the OpenGenomeBrowser folder structure.InstallationThis package requires at leastPython 3.9.pipinstallopengenomebrowser-toolsHelp functionAll scripts have a help function, for example:import_genome--helpinit_folder_structureCreates a basic OpenGenomeBrowser folders structure.More details:Once the folder structure has been initiated...useimport_genometo add genomes to the folder structureusedownload_ncbi_genomeandimport_genometo download and add genomes from NCBIwhen all genomes have been added, useinit_orthofinderandimport_orthofinderto calculate
orthologs (optional)Usage:exportFOLDER_STRUCTURE=/path/to/folder_structure
init_folder_structure# or --folder_structure_dir=/path/to/folder_structureResult:folder_structure
  ├── organisms
  ├── annotations.json
  ├── annotation-descriptions
  │   ├── SL.tsv
  │   ├── KO.tsv
  │   ├── KR.tsv
  │   ├── EC.tsv
  │   └── GO.tsv
  ├── orthologs
  └── pathway-maps
      ├── type_dictionary.json
      └── svgimport_genomeImport genome-associated files into OpenGenomeBrowser folder structure, automatically generate metadata files.More details:If the annotation was performed using the proper organism name, genome identifier and taxonomic information (recommended), the import is
straightforward because no files need to be renamed.exportFOLDER_STRUCTURE=/path/to/folder_structure# this directory contains the 'organisms' folderimport_genome--import_dir=/prokka/out/dir# optional: add "--organism STRAIN --genome STRAIN.1" as sanity checkHow to run prokka to get correct locus tagsSuppose the desired organism name isSTRAIN, the genome identifier isSTRAIN.1, this is how to run prokka:prokka\--strainSTRAIN\--locustagSTRAIN.1\--prefixSTRAIN.1\--genusMycoplasma--speciesgenitalium\# Optional. If set, this script can automatically detect the taxid.--out/prokka/out/dir\assembly.fastaHow to run PGAP to get correct locus tagsSuppose the desired organism name isSTRAIN, the genome identifier isSTRAIN.1, these are the lines in PGAPssubmol.yamlthat are relevant to
this script:organism:genus_species:'Mycoplasmagenitalium'# Optional. If set, this script can automatically detect the taxid.strain:'STRAIN'locus_tag_prefix:'STRAIN.1'bioproject:'PRJNA9999999'# Optional. If set, this script can automatically add it to bioproject_accession in genome.json.biosample:'SAMN99999999'# Optional. If set, this script can automatically add it to biosample_accession in genome.json.publications:# Optional. If set, this script can automatically add it to the literature_references in genome.json.-publication:pmid:16397293Rename files during importShould the locus tags not start with the genome identifier, the files need to be renamed accordingly. Theimport_genomecommand can do this
automatically sung the--renameflag.exportFOLDER_STRUCTURE=/path/to/folder_structure# this directory contains the 'organisms' folderimport_genome--import_dir=/prokka/out/dir--organismSTRAIN--genomeSTRAIN.1--renameThe renaming is provided as-is, and was only tested on files produced by certain versions of prokka and PGAP. If there is an error, you must rename
the files manually (with or without the help of myrenaming scripts) and then import them as
described in the previous section.Required filesThese files need to be inimport_dir:.fna: assembly (FASTA).gbk: GenBank file.gff: General feature format fileOptional files:.faa: protein sequences (FASTA). If non-existent, it will automatically be generated from the.gbkfile.ffn: nucleotides file (FASTA). If non-existent, it will automatically be generated from the.gbkfile.sqn: required for submission to GenBank, not really used by OpenGenomeBrowser.emapper.annotations: Eggnog annotation file.XX: custom annotation file (e.g.EC,.GO, etc.; any files with a suffix of two upper case letters are detected as custom annotations)_busco.txt: BUSCO output file, content will be added togenome.jsongenome.json: content will be added to finalgenome.json, may be as simple as{"restricted": true}organism.json: content will be added to finalorganism.json, may be as simple as{"assembly_tool": "SPAdes"}Example result:#### folder structure ####
folder_structure
└── organisms
    └── STRAIN
        ├── organism.json
        └── genomes
             └── STRAIN.1
         	     ├── genome.json
         	     ├── STRAIN.1.faa
         	     ├── STRAIN.1.ffn
         	     ├── STRAIN.1.fna
         	     ├── STRAIN.1.gbk
         	     ├── STRAIN.1.gff
         	     ├── STRAIN.1.sqn
         	     └── rest
         	      	 ├── PROKKA_08112021.err
         	      	 ├── PROKKA_08112021.fsa
        	      	 ├── PROKKA_08112021.log
         	      	 ├── PROKKA_08112021.tbl
         	      	 ├── PROKKA_08112021.tsv
         	      	 ├── PROKKA_08112021.txt
         	      	 └── short_summary.specific.lactobacillales_odb10.FAM3228-i1-1_busco.txtModify where files are moved toIt is possible to change where files end up in the folder structure. The behaviour is determined by a config file in json format that can be specified
with the --import_settings parameter or theOGB_IMPORT_SETTINGSenvironment variable.exportOGB_IMPORT_SETTINGS=/path/to/import_config.jsonThese are the default settings:{
    "organism_template": {},                           # use this to add metadata to all imported organism.json files, e.g. {"restricted": true}
    "genome_template": {},                             # use this to add metadata to all imported genome.json files, e.g. {"assembly_tool": "SPAdes"}
    "path_transformer": {
        ".*\\.fna": "{genome}.{suffix}",               # all files that match the regex will end up in organisms/STRAIN/genomes/STRAIN.1/STRAIN.1.fna
        ".*\\.faa": "{genome}.{suffix}",
        ".*\\.gbk": "{genome}.{suffix}",
        ".*\\.gff": "{genome}.{suffix}",
        ".*\\.sqn": "{genome}.{suffix}",
        ".*\\.ffn": "{genome}.{suffix}",
        ".*\\.emapper.annotations": "{genome}.eggnog",
        ".*\\.[A-Z]{2}": "{genome}.{suffix}",
        "genome.md": "genome.md", 
        "organism.md": "../../organism.md",            # this file will end up in /organisms/STRAIN/organism.md
        "genome.json": null,                           # this file will not be copied
        "organism.json": null,                         # this file will not be copied
        ".*": "rest/{original_path}"                   # this regex matches all files, thus all files that did not match any previous regex will
                                                       #   will end up in .../STRAIN.1/rest/
    }
}This is an example of an alternative configuration:{
    "organism_template": {},
    "genome_template": {},
    "path_transformer": {
        
        # raw reads
        ".*fastqc?\\..*": "0_raw_reads/{original_path}",
        
        # assembly
        ".*\\.fna": "1_assembly/{genome}.{suffix}",
        
        # coding sequence (CDS) calling
        ".*\\.faa": "2_cds/{genome}.{suffix}",
        ".*\\.gbk": "2_cds/{genome}.{suffix}",
        ".*\\.gff": "2_cds/{genome}.{suffix}",
        ".*\\.ffn": "2_cds/{genome}.{suffix}",
        ".*\\.sqn": "2_cds/{genome}.{suffix}",
        "PROKKA_.*": "2_cds/{original_path}",
        
        # functional annotations
        ".*\\.emapper.annotations": "3_annotation/{genome}.eggnog",
        ".*\\.[A-Z]{2}": "3_annotation/{genome}.{suffix}",
        ".*_busco\\.txt": "3_annotation/{original_path}",
        
        # special files
        "genome.md": "genome.md",
        "organism.md": "../../organism.md",
        "genome.json": null,
        "organism.json": null,
        
        # rest
        ".*": "rest/{original_path}"
    }
}Result:#### folder structure ####
folder_structure
└── organisms
    └── STRAIN
       ├── organism.json
       └── genomes
             └── STRAIN.1
                 ├── genome.json
                 ├── 1_assembly
                 │     └── STRAIN.1.fna
                 ├── 2_cds
                 │     ├── PROKKA_08112021.err
                 │     ├── PROKKA_08112021.fsa
                 │     ├── PROKKA_08112021.log
                 │     ├── PROKKA_08112021.tbl
                 │     ├── PROKKA_08112021.tsv
                 │     ├── PROKKA_08112021.txt
                 │     ├── STRAIN.1.faa
                 │     ├── STRAIN.1.ffn
                 │     ├── STRAIN.1.gbk
                 │     ├── STRAIN.1.gff
                 │     └── STRAIN.1.sqn
                 └── 3_annotation
                       └── short_summary_busco.txtAdd custom metadataThere are two ways to achieve this:Add aorganism.jsonand/orgenome.jsonfile intoimport_dir(seeimport_genome: Required files)Set a globalorganism.jsonand/orgenome.jsonfile that is used as a basis for all future imports (
seeimport_genome: Modify where files are moved to)rename_*The following scripts change the locus tags in the respective file formats.ScriptPurposerename_fastaChange locus tags of protein or nucleotide FASTA filesrename_genbankChange locus tags of GenBank files (tested with prokka and PGAP files)rename_gffChange locus tags of gff (general feature format) filesrename_eggnogChange locus tags of Eggnog files (.emapper.annotations)rename_custom_annotationsChange locus tags of custom annotations filesMore details:The syntax is always the same.rename_fasta\--file/path/to/input.file\--out/path/to/output.file\--new_locus_tag_prefixSTRAIN.2\--old_locus_tag_prefixSTRAIN.1# optional, good as sanity checkreindex_assemblyThis script changes the header of assembly FASTA (.fna) files.More details:reindex_assembly\--file/path/to/input.file\--out/path/to/output.file\--prefixSTRAIN_scf\--leading_zeroes5# optionalThis would transform a FASTA header like this>anything hereinto>STRAIN_scf_00001.genbank_to_fastaConvert GenBank to nucleotide (.ffn) or protein FASTA (.faa).More details:Usage:genbank_to_fasta\--gbk/path/to/input.gbk\--out/path/to/output.fasta\--formatfaa# or ffndownload_ncbi_genomeDownload genome-associated files (.fna,.gbk,.gff) from NCBI, rename the locus tags, and generate.ffnandfaafiles.More details:Usage:download_ncbi_genome\--assembly_nameGCF_005864195.1\--out_dir/path/to/outdir\--new_locus_tag_prefixFAM3257_Result:outdir
├── FAM3257.faa
├── FAM3257.ffn
├── FAM3257.fna
├── FAM3257.gbk
└── FAM3257.gffThe next step might be to import these genomes into the OpenGenomeBrowser folder structure like this:import_genome--import_dir=/path/to/outdir--organismFAM3257--genomeFAM3257init_orthofinderThis script collects the protein FASTAs infolder_structure/OrthoFinder/fastasand prints the command to run OrthoFinder.More details:Usage:exportFOLDER_STRUCTURE=/path/to/folder_structure
init_orthofinder--representatives_onlyResult:folder_structure
  ├── ...
  └── OrthoFinder
      └── fastas
          ├── GENOME1.faa
          ├── GENOME2.faa
          └── ...import_orthofinderThe output of OrthoFinder needs to be processed for OpenGenomeBrowser. This script creates two files:annotation-descriptions/OL.tsv: maps orthologs to the most common gene name, i.e.OG0000005->MFS transporterorthologs/orthologs.tsv: maps orthologs to genes, i.e.OG0000005->STRAIN1_000069, STRAIN2_000128, STRAIN2_000137More details:Usage:exportFOLDER_STRUCTURE=/path/to/folder_structure
import_orthofinder--whichhog# 'hog' for hierarchical orthogroups and 'og' for regular orthogroupsOnce these files exist, run the following command from within the OpenGenomeBrowser docker container:pythondb_setup/manage_ogb.pyimport-orthologsupdate_folder_structureFrom time to time, changes are made to the OpenGenomeBrowser folder structure. The current version of your folder structure is denoted
inversion.json. Use this script to upgrade to a new version.More details:1_to_2: addCOGto genome.json