Package: gff3toddbj
GFF3-to-DDBJ日本語版はこちら。[TOC]Table of ContentsWhat is this?How accurate is the conversion?Initial setupInstall with biocondaInstall with pipInstall from the sourceCreate DDBJ annotation from GFF3 and FASTARungff3-to-ddbjUnder the HoodCustomize the behaviorMetadata file[Advanced] Rename features and qualifiersRename types/feature keysRename attributes/qualifier keysTranslate GFF3 types to features with qualifiersTranslate (type, attribute) items to featuresRun with custom configuration[Advanced] Filter features and qualifiersTroubleshootingValidate GFF3Split FASTA from GFF3 (if needed)Normalize entry names (if needed)Known ProblemsCreditTable of contents generated with markdown-tocWhat is this?GFF3-to-DDBJ createsthe annotation file for submission to DDBJby taking GFF3 and FASTA files as input. It also works with FASTA alone.Analogous programs aretable2asnandGAGfor submissions to NCBI, andEMBLmyGFF3for submissions to EMBL.Please take a look at ourtest directoryfor examples. Files ending with .ann are the DDBJ annotation files produced by thie program.How accurate is the conversion?While there are many rules a DDBJ annotation file needs to comply with, it's difficult to tell what the correct GFF3→DDBJ conversion is.
There is no example of full-fledged GFF3 → DDBJ conversion available, either. So, we define GFF3-GenBank correspondence in RefSeq as the "correct" examples. To evaluate GFF3-to-DDBJ, we use RefSeq data and comparegff3-to-ddbjoutput with the other DDBJ annotation fromgenbank-to-ddbjusing the GenBank format. Please take a look at our evaluation dcoument for the detail as well as the current status. ([TODO] Add the page...)Heregenbank-to-ddbjis an executable included in this package. It shares codebase withgff3-to-ddbj, but we believe it does not bring any complexity to our evaluation due to its much simpler internals.Also note that we also use DDBJ'sParserfor checking the annotation files.Initial setupInstall with bioconda# Create a conda environment named "ddbj", and install relevant packages from bioconda channelcondacreate-nddbj-cbioconda-cconda-forgegff3toddbj# Activate the environment "ddbj"condaactivateddbjInstall with pip# Create a conda environment named "ddbj" and install pipcondacreate-nddbjpip# Activate the environment "ddbj"condaactivateddbj# Need bgzip executable in samtoolscondainstall-cbiocondasamtools# Install from pippipinstallgff3toddbjInstall from the source# Downloadwgethttps://github.com/yamaton/gff3_to_ddbj/archive/refs/heads/main.zip# Extract, rename, and change directoryunzipmain.zip&&mvgff3toddbj-maingff3toddbj&&cdgff3toddbj# Create a conda environment named "ddbj"condacreate-nddbj# Activate the environment "ddbj"condaactivateddbj# Install dependencies to "ddbj"condainstall-cbioconda-cconda-forgebiopythonbcbio-gfftomlpysamsamtoolspipbuild# Install gff3-to-ddbj and extra toolspython-mbuild&&pipinstall-e./Create DDBJ annotation from GFF3 and FASTARungff3-to-ddbjLet's run the main program to get some ideas.gff3-to-ddbj\--gff3myfile.gff3\# bare-minimum output if omitted--fastamyfile.fa\# <<REQUIRED>>--metadatamymetadata.toml\# example metadata used if omitted--locus_tag_prefixMYOWNPREFIX_\# default is "LOCUSTAGPREFIX_"--transl_table1\# default is 1--outputmyawesome_output.ann# standard output if omittedHere is the options:--gff3 <FILE>takes GFF3 file--fasta <FILE>takes FASTA file--metadata <FILE>takes the metadata file in TOML--locus_tag_prefix <STRING>takes the prefix of locus tagobtained from BioSample. You can skip this for now.--transl_table <INT>: Choose appropriate one fromThe Genetic Codes. The default value is 1 ("standard").--output <FILE>sets the path the annotation output.Under the HoodHere is the list of operationsgff3-to-ddbjwill do:Re-compress FASTA withbgzipif the FASTA input is compressed with gzipA bgzip file is created if absent likemyfile_bgzip.fa.gz.For indexing and saving memoryThe bgzip file should be compatible with gzipRename features and qualifiers following therenaming scheme.Search forassembly_gaps in FASTA, and add the feature.Add/transl_tableto each CDS.Insertsourceinformation from themetadata fie.InsertTOPOLOGYfeature if GFF3 hasIs_circular=truein an entry.Also handleorigin-spaning features.Join locations of features having the same parent withjoinnotation.CDS,exon,mat_peptide,V_segment,C_region,D-loop, andmisc_featuremay be joined.exons are NOT joined if havinggeneas the direct parent.Set the location of joined exons as its parent RNA's location, and discard the exons.Add partialness markup (<and>) toCDSlocations if start/stop codon is absent.SeeOffset of the frame at translation initiation by codon_startLet CDS have a single/productvalue: Set it to "hypothetical protein" if absent. Move the rest of exising values to/note.This is to conform theinstructionon/product.Even if there are multiple general names for the same product, do  not enter multiple names in 'product'. Do not use needless symbolic  letters as delimiter for multiple names. If you would like to describe  more than two names, please enter one of the most representative name in /product qualifier, and other(s) in /notequalifier.If the name and function are not known, we recommend to describe as "hypothetical protein".If agenefeature has/geneand/or/gene_synonym, copy these qualifiers to its children.Make/genehave a single value, and put the rest to/gene_synonym.Reference:Definition of Qualifier key: /gene.Remove duplicates in qualifier values.Sort lines in annotationFilter features and qualifiers followingthe matrix.genefeature will be discarded in this process.Customize the behaviorMetadata fileTo enter information missing in GFF3 or FASTA, such as submitter names and certain qualifier values, you need to feed a metadata file in TOML, saymymetadata.toml. Take a look atan examplematchingthe example annotation in the DDBJ page.The file accommodates following and they are all optional. That is, GFF3-to-DDBJ works even with an empty file.Basic features in theCOMMONentry... such asSUBMITTER,REFERENCE, andCOMMENT."meta-description" in the COMMON entryHere is an example with this notation:[COMMON.assembly_gap]estimated_length="unknown"gap_type="within scaffold"linkage_evidence="paired-ends"DDBJ annotation supports "meta" values with features under COMMON such that the items are inserted to each occurrencein the resulting flat fileproduced by DDBJ. Here is an example to insertassembly_gapfeature underCOMMONentry.Feature-qualifier items inserted to each occurrenceHere is an example: Difference from the previous case is only at[assembly_gap]as opposed to[COMMON.assembly_gap].[assembly_gap]estimated_length="unknown"# Set it "<COMPUTE>" to count the number of N'sgap_type="within scaffold"linkage_evidence="paired-ends"While this should work effectively the same as the "meta-description" item above, use this notation if you insert values repeatedlyin the annotation fileproduced by GFF3-to-DDBJ.Currently supporting[source]and[assembly_gap]only.For more examples, seeWGS in COMMONandWGSprovided by DDBJ as annotation examples, and corresponding metadata filesmetadata_WGS_COMMON.tomlandmetadata_WGS.tomlin this repository.[Advanced] Rename Features and QualifiersGFF3 and DDBJ annotation have rough correspondence like:GFF3 column 3 "type" →  DDBJ annotation column 2 as "Feature"GFF3 column 9 "attribute" →  DDBJ annotation column 4 and 5 as "Qualifier key", and "Qualifier value"but nomenclatures in GFF3 often do not conform the annotations set by INSDC. Furthermore, DDBJ lists up thefeature-qualifier pairs they accepts, a subset of the INSDC definitions.To meet convensions with the requirement, GFF3-to-DDBJ comes witha default configuration in TOMLto rename (or even translate) feature keys and qualifier keys/values. Note thatthe Sequence Ontologyis helpful in translating a type into a INSDC feature and qualifier(s).Here is how to customize the renaming configuration.Rename types/feature keysThe default setting renamesfive_prime_UTR"type" in GFF3 into5'UTR"feature key" in the annotation. This transformation is expressed in TOML as follows:[five_prime_UTR]feature_key="5'UTR"Rename attributes/qualifier keysThis is about renaming attributes under arbitrary types. By default,ID=foobar"attribute" in a GFF3 becomes/note="ID:foobar"qualifier in the annotation. (Here I follow the convention putting slash (like/note) to denote qualifier. But DDBJ annotation does NOT include slash hence no slash is used in any of TOML files.)Here is the TOML defining the transformation.__ANY__is the special name representing arbitrary types.IDis the original attribute key.noteis the name of corresponding qualifier key.ID:is attached as the prefix of the qualifier value.[__ANY__.ID]qualifier_key="note"qualifier_value_prefix="ID:"# optionalOne can also set a qualifier key and a value together. For example,/pseudoqualifier is discouraged by DDBJ regardless of features. We may enforce the replacement by,# /pseudo is always replaced by /pseudogene="unknown"[__ANY__.pseudo]qualifier_key="pseudogene"qualifier_value="unknown"Translate GFF3 types to features with qualifiersSometimes we want to replace a certain types with features WITH qualifiers. For example,snRNAis an invalid feature in INSDC/DDBJ hence we replace it withncRNAfeature with/ncRNA_class="snRNA"qualifier. Such transformation is written in TOML as following.[snRNA]feature_key="ncRNA"qualifier_key="ncRNA_class"qualifier_value="snRNA"Translate (type, attribute) items to featuresHere is a story in setting the default renaming scheme: Some annotation programs produce a GFF3 line containingRNAas the type andbiotype=misc_RNAas one of the attributes. But it should be treated asmisc_RNAfeature in DDBJ annoation. In such case, we join (feature key, qualifier key, qualifier value) with dot as delimiter, and write as follows.[RNA.biotype.misc_RNA]feature_key="misc_RNA"Run with custom configurationTo feed a custom translation table, use the CLI option:--config_rename <FILE>And here is an example call:gff3-to-ddbj\--gff3myfile.gff3\--fastamyfile.fa\--metadatamymetadata.toml\--locus_tag_prefixMYOWNPREFIX_\--transl_table1\--config_renamemy_translate_features_qualifiers.toml\# Set your customized file here--outputmyawesome_output.ann[Advanced] Filter features and qualifiersDDBJ specifies recommendedFeature/Qualifier usage matrix. To conform this rule, features and qualifiers appearing in the annotation output are filtered bythe filtering file in TOMLby default. The file is in TOML format with the structure like this:CDS=["EC_number","inference","locus_tag","note","product",]exon=["gene","locus_tag","note",]The left-hand side of the equal sign=represents an allowed feature key, and the right-hand side is a list of allowed qualifier keys. In this example, onlyCDSandexonfeatures will show up in the annotation, and qualifiers are limited to the listed items. To customize this filtering function, edit the TOML file first and pass the file with the CLI option:--config_filter <FILE>TroubleshootingValidate GFF3It might be a good practice to validate your GFF3 files.GFF3 online validatoris useful though the file size is limited to 50MB.Split FASTA from GFF3 (if needed)GFF3_to_DDBJ does not work when GFF3 contains FASTA information inside with##FASTAdirective. Attached toolsplit-fastareads a GFF3 file and saves GFF3 (without FASTA info) and FASTA.split-fastapath/to/myfile.gff3--suffix"_splitted"This creates two files,myfile_splitted.gff3andmyfile_splitted.fa.Normalize entry names (if needed)Letters like=|>" []are not allowed in the 1st column (= "Entry") of the DDBJ annotation.  The attached programnormalize-entry-namesrenames such entries. This program converts an ID likeERS324955|SC|contig000013intoERS324955:SC:contig000013for example.normalize-entry-namesmyannotation_output.txtThis command create as filesmyannotation_output_renamed.txtifthe invalid letters are found. Otherwise, you'll see no output.Known IssuesNeed to handle location correction and featurejoin()in presence of/trans_splicingNeed to handle location correction in presence of/transl_exceptat start/stop codonNeeds/translationwhen/exceptionexists.GFF3 handling when the flatfile is supposed to have "between-position" location like123^124Currently the development focuses on accuracy; the software runs slow using a single process.CreditGFF3-to-DDBJ's design is deeply indebted toEMBLmyGFF3, a versatile coversion for EMBL annotation format.