Package: gff3-parser
gff3_parsergithubThis is a simple python package to parsegff3( Generic Feature Format)
files into pandas dataframes. This file format is used for genetic annotation files and I couldn't find a parser
that worked with python so I wrote this. This is still a work in progress and I'll hopefully be adding
features soon.BackgroundWhat if gff3 file format?I store this from nice explanation fromNGS Analysis:General Feature Format (GFF) is a tab-delimited text file that holds information any and every feature that can be applied to a nucleic acid or protein sequence. Everything from CDS, microRNAs, binding domains, ORFs, and more can be handled by this format. Unfortunately there have been many variations of the original GFF format and many have since become incompatible with each other. The latest accepted format (GFF3) has attempted to address many of the issues that were missing from previous versions.GFF3 has 9 required fields, though not all are utilized (either blank or a default value of ‘.’).Sequence IDSource  - Describes the algorithm or the procedure that generated this feature. Typically Genescane or Genebank, respectively.Feature Type  - Describes what the feature is (mRNA, domain, exon, etc.). These terms are constrained to theSequence Ontology terms.Feature StartFeature EndScore  - Typically E-values for sequence similarity and P-values for predictions.Strand (+ or -)Phase  - Indicates where the feature begins with reference to the reading frame. The phase is one of the integers 0, 1, or 2, indicating the number of bases that should be removed from the beginning of this feature to reach the first base of the next codon.Atributes  A semicolon-separated list of tag-value pairs, providing additional information about each feature. Some of these tags are predefined, e.g. ID, Name, Alias, Parent . You can see the full listhere.Example File##gff-version 3
#description: evidence-based annotation of the human genome (GRCh38), version 38 (Ensembl 104)
#provider: GENCODE
#contact: gencode-help@ebi.ac.uk
#format: gff3
#date: 2021-03-12
##sequence-region chr1 1 248956422
chr1	HAVANA	gene	11869	14409	.	+	.	ID=ENSG00000223972.5;gene_id=ENSG00000223972.5;gene_type=transcribed_unprocessed_pseudogene;gene_name=DDX11L1;level=2;hgnc_id=HGNC:37102;havana_gene=OTTHUMG00000000961.2
chr1	HAVANA	transcript	11869	14409	.	+	.	ID=ENST00000456328.2;Parent=ENSG00000223972.5;gene_id=ENSG00000223972.5;transcript_id=ENST00000456328.2;gene_type=transcribed_unprocessed_pseudogene;gene_name=DDX11L1;transcript_type=processed_transcript;transcript_name=DDX11L1-202;level=2;transcript_support_level=1;hgnc_id=HGNC:37102;tag=basic;havana_gene=OTTHUMG00000000961.2;havana_transcript=OTTHUMT00000362751.1
chr1	HAVANA	exon	11869	12227	.	+	.	ID=exon:ENST00000456328.2:1;Parent=ENST00000456328.2;gene_id=ENSG00000223972.5;transcript_id=ENST00000456328.2;gene_type=transcribed_unprocessed_pseudogene;gene_name=DDX11L1;transcript_type=processed_transcript;transcript_name=DDX11L1-202;exon_number=1;exon_id=ENSE00002234944.1;level=2;transcript_support_level=1;hgnc_id=HGNC:37102;tag=basic;havana_gene=OTTHUMG00000000961.2;havana_transcript=OTTHUMT00000362751.1
chr1	HAVANA	exon	12613	12721	.	+	.	ID=exon:ENST00000456328.2:2;Parent=ENST00000456328.2;gene_id=ENSG00000223972.5;transcript_id=ENST00000456328.2;gene_type=transcribed_unprocessed_pseudogene;gene_name=DDX11L1;transcript_type=processed_transcript;transcript_name=DDX11L1-202;exon_number=2;exon_id=ENSE00003582793.1;level=2;transcript_support_level=1;hgnc_id=HGNC:37102;tag=basic;havana_gene=OTTHUMG00000000961.2;havana_transcript=OTTHUMT00000362751.1Why this is super annoying to parseBasically the first 8 columns are nicely structured tapular data but that last column has an arbitrary
number of new values. This is kind of similar to a SQL table and a paired noSQL db but the way these
files are distributed you can't use those tools.How to parseI just found every unique key in the last column and made it it's own column and then reorganized
data accordingly. It can be reasonably sparse and it does take a good amount of time and space
(the files are often pretty large) but the end result is a normal structured pandas dataframe.Installationpip install pip install gff3-parserI'd recommend updating this often as I find and fix issuesExample Usage>>>importgff3_parser>>>filepath="gencode.v38.annotation.gff3">>>just_tabular=gff3_parser.parse_gff3(filepath,verbose=True,parse_attributes=False)description:evidence-basedannotationofthehumangenome(GRCh38),version38(Ensembl104)provider:GENCODEcontact:gencode-help@ebi.ac.ukformat:gff3date:2021-03-12100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████|3148167/3148167[00:07<00:00,421099.43it/s]>>>just_tabular.head()SeqidSourceTypeStartEndScoreStrandPhase0chr1HAVANAgene1186914409NaN+NaN1chr1HAVANAtranscript1186914409NaN+NaN2chr1HAVANAexon1186912227NaN+NaN3chr1HAVANAexon1261312721NaN+NaN4chr1HAVANAexon1322114409NaN+NaN>>>full_data=gff3_parser.parse_gff3('gencode.v38.annotation.gff3',verbose=False,parse_attributes=True)>>>full_data.head()SeqidSourceTypeStartEndScoreStrandPhaseseqid...IDtranscript_support_levelParentonttranscript_typetaghavana_transcripttranscript_nameccdsid0chr1HAVANAgene1186914409NaN+NaNNaN...ENSG00000223972.5NaNNaNNaNNaNNaNNaNNaNNaN1chr1HAVANAtranscript1186914409NaN+NaNNaN...ENST00000456328.21ENSG00000223972.5NaNprocessed_transcriptbasicOTTHUMT00000362751.1\nDDX11L1-202NaN2chr1HAVANAexon1186912227NaN+NaNNaN...exon:ENST00000456328.2:11ENST00000456328.2NaNprocessed_transcriptbasicOTTHUMT00000362751.1\nDDX11L1-202NaN3chr1HAVANAexon1261312721NaN+NaNNaN...exon:ENST00000456328.2:21ENST00000456328.2NaNprocessed_transcriptbasicOTTHUMT00000362751.1\nDDX11L1-202NaN4chr1HAVANAexon1322114409NaN+NaNNaN...exon:ENST00000456328.2:31ENST00000456328.2NaNprocessed_transcriptbasicOTTHUMT00000362751.1\nDDX11L1-202NaN>>>full_data.columnsIndex(['Seqid','Source','Type','Start','End','Score','Strand','Phase','seqid','transcript_id','havana_gene','gene_type','gene_name','gene_id','exon_id','level','protein_id','hgnc_id','exon_number','ID','transcript_support_level','Parent','ont','transcript_type','tag','havana_transcript','transcript_name','ccdsid'],dtype='object')Full DocumentationThis whole project has literally one public function so far so I'm just going to document it here until
I feel like it needs more.Ill get around to it eventually. Theres only two uses and both are in the example.