Package: biobear
biobear is a Python library designed for reading and searching bioinformatic file formats, using Rust as its backend and producing Arrow Batch Readers and other downstream formats (like polars or duckdb).The python package has minimal dependencies and only requires Polars. Biobear can be used to read various bioinformatic file formats, including FASTA, FASTQ, VCF, BAM, and GFF locally or from an object store like S3. It can also query some indexed file formats locally like VCF and BAM.Please see thedocumentationfor information on how to get started using biobear.QuickstartTo install biobear, run:pipinstallbiobear
pipinstallpolars# needed for `to_polars` methodCreate a file with some GFF data:echo"chr1\t.\tgene\t1\t100\t.\t+\t.\tgene_id=1;gene_name=foo">test.gffecho"chr1\t.\tgene\t200\t300\t.\t+\t.\tgene_id=2;gene_name=bar">>test.gffThen you can use biobear to read a file:importbiobearasbbsession=bb.connect()df=session.sql("""SELECT * FROM gff_scan('test.gff')""").to_polars()print(df)This will print:┌─────────┬────────┬──────┬───────┬───┬───────┬────────┬───────┬───────────────────────────────────┐
│ seqname ┆ source ┆ type ┆ start ┆ … ┆ score ┆ strand ┆ phase ┆ attributes                        │
│ ---     ┆ ---    ┆ ---  ┆ ---   ┆   ┆ ---   ┆ ---    ┆ ---   ┆ ---                               │
│ str     ┆ str    ┆ str  ┆ i64   ┆   ┆ f32   ┆ str    ┆ str   ┆ list[struct[2]]                   │
╞═════════╪════════╪══════╪═══════╪═══╪═══════╪════════╪═══════╪═══════════════════════════════════╡
│ chr1    ┆ .      ┆ gene ┆ 1     ┆ … ┆ null  ┆ +      ┆ null  ┆ [{"gene_id","1"}, {"gene_name","… │
│ chr1    ┆ .      ┆ gene ┆ 200   ┆ … ┆ null  ┆ +      ┆ null  ┆ [{"gene_id","2"}, {"gene_name","… │
└─────────┴────────┴──────┴───────┴───┴───────┴────────┴───────┴───────────────────────────────────┘Using a Session w/ ExonBioBear exposes a session object that can be used withexonto work with files directly in SQL, then eventually convert them to a DataFrame if needed.See theBioBear Docsfor more information, but in short, you can use the session like this:importbiobearasbbsession=bb.connect()session.sql("""CREATE EXTERNAL TABLE gene_annotations_s3 STORED AS GFF LOCATION 's3://BUCKET/TenflaDSM28944/IMG_Data/Ga0451106_prodigal.gff'""")df=session.sql("""SELECT * FROM gene_annotations_s3 WHERE score > 50""").to_polars()df.head()# shape: (5, 9)# ┌──────────────┬─────────────────┬──────┬───────┬───┬────────────┬────────┬───────┬───────────────────────────────────┐# │ seqname      ┆ source          ┆ type ┆ start ┆ … ┆ score      ┆ strand ┆ phase ┆ attributes                        │# │ ---          ┆ ---             ┆ ---  ┆ ---   ┆   ┆ ---        ┆ ---    ┆ ---   ┆ ---                               │# │ str          ┆ str             ┆ str  ┆ i64   ┆   ┆ f32        ┆ str    ┆ str   ┆ list[struct[2]]                   │# ╞══════════════╪═════════════════╪══════╪═══════╪═══╪════════════╪════════╪═══════╪═══════════════════════════════════╡# │ Ga0451106_01 ┆ Prodigal v2.6.3 ┆ CDS  ┆ 2     ┆ … ┆ 54.5       ┆ -      ┆ 0     ┆ [{"ID",["Ga0451106_01_2_238"]}, … │# │ Ga0451106_01 ┆ Prodigal v2.6.3 ┆ CDS  ┆ 228   ┆ … ┆ 114.0      ┆ -      ┆ 0     ┆ [{"ID",["Ga0451106_01_228_941"]}… │# │ Ga0451106_01 ┆ Prodigal v2.6.3 ┆ CDS  ┆ 1097  ┆ … ┆ 224.399994 ┆ +      ┆ 0     ┆ [{"ID",["Ga0451106_01_1097_2257"… │# │ Ga0451106_01 ┆ Prodigal v2.6.3 ┆ CDS  ┆ 2261  ┆ … ┆ 237.699997 ┆ +      ┆ 0     ┆ [{"ID",["Ga0451106_01_2261_3787"… │# │ Ga0451106_01 ┆ Prodigal v2.6.3 ┆ CDS  ┆ 3784  ┆ … ┆ 114.400002 ┆ +      ┆ 0     ┆ [{"ID",["Ga0451106_01_3784_4548"… │# └──────────────┴─────────────────┴──────┴───────┴───┴────────────┴────────┴───────┴───────────────────────────────────┘EcosystemBioBear aims to make it simple to move easily to and from different prominent data tools in Python. Generally, if the tool can read Arrow, it can read BioBear's output. To call out a few examples here:PolarsThe session results and Reader objects can be converted to a Polars DataFrame.importbiobearasbbsession=bb.connect()df=session.sql("""SELECT * FROM gff_scan('test.gff')""").to_polars()Known IssuesFor GenBank and mzML, the naiveSELECT *will cause an error, because Polars doesn't support all Arrow types --Mapbeing the specific offender here. In these cases, select the fields from the map individually. Alternatively, you can first convert the table to a Pandas DataFrame.DuckDBBioBear can also be used to read files into aduckdbdatabase.importbiobearasbbimportduckdbsession=bb.connect()session.sql("""CREATE EXTERNAL TABLE gene_annotations STORED AS GFF LOCATION 'python/tests/data/test.gff'""")result=session.sql("""SELECT * FROM gene_annotations""")gff_table_arrow_table=result.to_arrow()duckdb_conn=duckdb.connect()result=duckdb_conn.execute('SELECT * FROM gff_table_arrow_table').fetchall()print(result)PerformancePlease see theexon's performance metrics for thorough benchmarks, but in short, biobear is generally faster than other Python libraries for reading bioinformatic file formats.For example, here's quick benchmarks for reading one FASTA file with 1 million records and reading 5 FASTA files each with 1 million records for the local file system on an M1 MacBook Pro:Library1 file (s)5 files (s)BioBear4.605 s ±  0.166 s6.420 s ±  0.113 sBioPython6.654 s ±  0.003 s34.254 s ±  0.053 sThe larger difference multiple files is due to biobear's ability to read multiple files in parallel.