Package: dbgz
dbgzDBGZ (Data block "GNU" zip) is a lightweight seekable compressed binary format for storing and retrieving data. The format is designed to hold schema and data comprising primary objects, such as numbers, strings and arrays, as well as more complicated instances via msgpack, such as dictionaries and python objects. It is based on the bgzip file format used in biology and other fields.This utility can be used to write and read DBGZ files in python.InstallationInstall using pippipinstalldbgzor from source:pipgit+https://github.com/filipinascimento/dbgz.gitUsageFirst import dbgz:importdbgzDefining a schemescheme=[("anInteger","i"),("aFloat","f"),("aString","s"),("anIntArray","I"),("aFloatArray","F"),("anStringArray","S"),("anyType","a"),#any data type]Writing some data to a dbgz filefromtqdm.autoimporttqdm# Optional, to print progress bar# pip install tqdmtotalCount=1000000;withdbgz.DBGZWriter("test.dbgz",scheme)asfd:# New entries can be added as:fd.write(anInteger=1,aString="1")fd.write(anInteger=2,aString="2",aFloat=5)fd.write(anInteger=3,aString="3",anIntArray=list(range(10)),aFloatArray=[0.1,0.2,0.3,0.5])# Here is a loop to write a lot of data:forindexintqdm(range(totalCount)):fd.write(anInteger=index,aFloat=index*0.01,anIntArray=list(range(index,index+10)),aString=str(index),aFloatArray=[index+0.1,index-0.2,index+0.3,index+0.4],anStringArray=[str(index),str(index+1),str(index+2),str(index+3)],anyType={"a":index,"b":index+1,"c":index+2})Reading the dbgz file sequencially:withdbgz.DBGZReader("test.dbgz")asfd:print(fd.scheme)forentryintqdm(fd.entries,total=fd.entriesCount):assertentry["anInteger"]==int(entry["aString"])Loading a dbgz file manually by using theread()method:withdbgz.DBGZReader("test.dbgz")asfd:pbar=tqdm(total=fd.entriesCount)print(fd.scheme)whileTrue:entries=fd.read(10)if(notentries):breakforentryinentries:assertentry["anInteger"]==int(entry["aString"])pbar.update(len(entries))pbar.refresh()pbar.close()Saving dictionary to file and loading it againwithdbgz.DBGZReader("test.dbgz")asfd:indexDictionary=fd.generateIndex("anInteger",indicesPath=None,filterFunction=lambdaentry:entry["anInteger"]<10,useDictionary=True,showProgressbar=True)forkey,valuesinindexDictionary.items():print(key,values)forvalueinvalues:assertint(key)==fd.readAt(value)[0]["anInteger"]Saving dictionary to file and loading it againwithdbgz.DBGZReader("test.dbgz")asfd:fd.generateIndex(key="anInteger",indicesPath="test_byAnInteger.idbgz",filterFunction=lambdaentry:entry["anInteger"]<10,useDictionary=True,showProgressbar=True)indexDictionary=dbgz.readIndicesDictionary("test_by.idbgz")forkey,valuesinindexDictionary.items():print(key,values)forvalueinvalues:assertint(key)==fd.readAt(value)[0]["anInteger"]Using a custom key generator for the index:withdbgz.DBGZReader("test.dbgz")asfd:fd.generateIndex(keyFunction=lambdaentry:entry["anyType"]["b"]ifentry["anyType"]elseNone,indicesPath="test_byAnyType_b.idbgz",filterFunction=lambdaentry:entry["anInteger"]<10,useDictionary=True,showProgressbar=True)indexDictionary=dbgz.readIndicesDictionary("test_byAnyType_b.idbgz")forkey,valuesinindexDictionary.items():print(key,values)forvalueinvalues:entry=fd.readAt(value)[0]assertint(key)==entry["anyType"]["b"]ifentry["anyType"]elseNone