Package: pyslow5
pyslow5 python libraryThe slow5 python library (pyslow5) allows a user to read and write slow5/blow5 files.InstallationInitial setup and example info for environmentslow5lib needs python3.4.2 or higher.If you only want to use the python library, then you can simply install using pipUsing a virtual environment (see below if you need to install python)Optional zstd compressionYou can optionally enablezstdcompressionsupport when buildingslow5lib/pyslow5. This requireszstd 1.3 or higher development librariesinstalled on your system:OnDebian/Ubuntu:sudoapt-getlibzstd1-dev
OnFedora/CentOS:sudoyumlibzstd-devel
OnOSX:brewinstallzstdBLOW5 files compressed withzstdoffer smaller file size and better performance compared to the defaultzlib. However,zlibruntime library is available by default on almost all distributions unlikezstdand thus files compressed withzlibwill be more 'portable'.Install from pypipython3-mvenvpath/to/slow5libvenvsourcepath/to/slow5libvenv/bin/activate
python3-mpipinstall--upgradepip# do this separately, after the libs above# zlib only buildpython3-mpipinstallpyslow5# for zstd build, run the followingexportPYSLOW5_ZSTD=1python3-mpipinstallpyslow5Dev install# If your native python3 meets this requirement, you can use that, or use a# specific version installed with deadsnakes below. If you install with deadsnakes,# you will need to call that specific python, such as python3.8 or python3.9,# in all the following commands until you create a virtual environment with venv.# Then once activated, you can just use python3.# To install a specific version of python, the deadsnakes ppa is a good place to start# This is an example for installing python3.8# you can then call that specific python version# > python3.8 -m pip --versionsudoadd-apt-repositoryppa:deadsnakes/ppa
sudoapt-getupdate
sudoaptinstallpython3.8python3.8-devpython3.8-venv# get zlib1g-devsudoapt-getupdate&&sudoapt-getinstall-yzlib1g-dev# Check withpython3--version# You will also need the python headers if you don't already have them installed.sudoapt-getinstallpython3-devBuilding and installing the python library.python3-mvenv/path/to/slow5libvenvsource/path/to/slow5libvenv/bin/activate
python3-mpipinstall--upgradepip

gitclonegit@github.com:hasindu2008/slow5lib.gitcdslow5lib# New build method to work with setuptools depricationpython3-mpipinstall.# This should not require sudo if using a python virtual environment/venv# confirm installation, and find pyslow5==<version>python3-mpipfreeze# Ensure slow5 library is working by running the basic testspython3./python/example.py# To Remove the librarypython3-mpipuninstallpyslow5# Legacy build methods - not recommended# CHOOSE A OR B:# (B is the cleanest method)# |=======================================================================|# |A. Install with pip if wheel is present, otherwise it uses setuptools  |python3-mpipinstall.--use-feature=in-tree-build# |=======================================================================|# |B. Or build and install manually with setup.py                         |# |build the package                                                      |python3setup.pybuild# |If all went well, install the package                                  |python3setup.pyinstall# |=======================================================================|UsageReading/writing a fileOpen(FILE, mode, rec_press="zlib", sig_press="svb-zd", DEBUG=0):The pyslow5 library has one main Class,pyslow5.Openwhich opens a slow5/blow5 (slow5 for easy reference) file for reading/writing.FILE: the file or filepath of the slow5 file to openmode: mode in which to open the file.r= read onlyw= write/overwritea= appendThis is designed to mimic Python's native Open() to help users remember the syntaxTo set the record and signal compression methods, use the followingrec_pressandsig_pressoptional args, however these are only used withmode='w'. Any append will use whatever is already set in the file.Compression Options:rec_press:"none""zlib" [default]"zstd" [requiresexport PYSLOW5_ZSTD=1when building]sig_press:"none""svb-zd" [default]Example:importpyslow5# open files5=pyslow5.Open('examples/example.slow5','r')When opening a slow5 file for the first time, and index will be created and saved in the same directory as the file being read. This index will then be loaded. For files that already have an index, that index will be loaded.get_read_ids():returns a list and total number of reads from the index.
If there is no index, it creates one first.Example:read_ids,num_reads=s5.get_read_ids()print(read_ids)print("number of reads:{}".format(num_reads))seq_reads(pA=False, aux=None):Access all reads sequentially in an opened slow5.If readID is not found,Noneis returned.pA = Bool for converting signal to picoamps.aux =str'<attr_name>'/'all' or list of names of auxiliary fields added to return dictionary,Noneif<attr_name>not foundreturnsdict= dictionary of main fields for read_id, with any aux fields addedExample:# create generatorreads=s5.seq_reads()# print all readIDsforreadinreads:print(read['read_id'])# or use directly in a for loopforreadins5.seq_reads(pA=True,aux='all'):print("read_id:",read['read_id'])print("read_group:",read['read_group'])print("digitisation:",read['digitisation'])print("offset:",read['offset'])print("range:",read['range'])print("sampling_rate:",read['sampling_rate'])print("len_raw_signal:",read['len_raw_signal'])print("signal:",read['signal'][:10])print("================================")seq_reads_multi(threads=4, batchsize=4096, pA=False, aux=None):Access all reads sequentially in an opened slow5, using multiple threads.If readID is not found,Noneis returned.threads = number of threads to use in C backend.batchsize = number of reads to fetch at a time. Higher numbers use more ram, but is more efficient with more threads.pA = Bool for converting signal to picoamps.aux =str'<attr_name>'/'all' or list of names of auxiliary fields added to return dictionary,Noneif<attr_name>not foundreturnsdict= dictionary of main fields for read_id, with any aux fields addedExample:# create generatorreads=s5.seq_reads_multi(threads=2,batchsize=3)# print all readIDsforreadinreads:print(read['read_id'])# or use directly in a for loopforreadins5.seq_reads_multi(threads=2,batchsize=3,pA=True,aux='all'):print("read_id:",read['read_id'])print("read_group:",read['read_group'])print("digitisation:",read['digitisation'])print("offset:",read['offset'])print("range:",read['range'])print("sampling_rate:",read['sampling_rate'])print("len_raw_signal:",read['len_raw_signal'])print("signal:",read['signal'][:10])print("================================")get_read(readID, pA=False, aux=None):Access a specific read using a unique readID. This is a ranom access method, using the index.If readID is not found,Noneis returned.pA = Bool for converting signal to picoamps.aux =str'<attr_name>'/'all' or list of names of auxiliary fields added to return dictionary,Noneif<attr_name>not foundreturnsdict= dictionary of main fields for read_id, with any aux fields addedExample:readID="r1"read=s5.get_read(readID,pA=True,aux=["read_number","start_mux"])ifreadisnotNone:print("read_id:",read['read_id'])print("len_raw_signal:",read['len_raw_signal'])get_read_list(read_list, pA=False, aux=None):Access a list of specific reads using a listread_listof unique readIDs. This is a random access method using the index. If an index does not exist, it will create one first.If readID is not found,Noneis returned.pA = Bool for converting signal to picoamps.aux =str'<attr_name>'/'all' or list of names of auxiliary fields added to return dictionary,Noneif<attr_name>not foundreturnsdict= dictionary of main fields for read_id, with any aux fields addedExample:read_list=["r1","r3","null_read","r5","r2","r1"]selected_reads=s5.get_read_list(read_list)forr,readinzip(read_list,selected_reads):ifreadisnotNone:print(r,read['read_id'])else:print(r,"read not found")get_read_list_multi(read_list, threads=4, batchsize=100, pA=False, aux=None)::Access a list of specific reads using a listread_listof unique readIDs using multiple threads. This is a random access method using the index. If an index does not exist, it will create one first.If readID is not found,Noneis returned.threads = number of threads to use in C backendbatchsize = number of reads to fetch at a time. Higher numbers use more ram, but is more efficient with more threads.pA = Bool for converting signal to picoamps.aux =str'<attr_name>'/'all' or list of names of auxiliary fields added to return dictionary,Noneif<attr_name>not foundreturnsdict= dictionary of main fields for read_id, with any aux fields added
Example:read_list=["r1","r3","null_read","r5","r2","r1"]selected_reads=s5.get_read_list_multi(read_list,threads=2,batchsize=3)forr,readinzip(read_list,selected_reads):ifreadisnotNone:print(r,read['read_id'])else:print(r,"read not found")get_num_read_groups():NEW: from version 1.1.0+Return an int for the number of read_groups present in fileget_header_names():Returns a list containing the uninon of header names from all read_groupsget_header_value(attr, read_group=0):Returns astrof the value of a header attribute (attr) for a particular read_group.
ReturnsNoneif value can't be foundget_all_headers(read_group=0):Returns a dictionary with all header attributes and values for a particular read_group
If there are values present for one read_group, and not for another, the attribute will still be returned for the read_group without, but with a value ofNone.get_aux_names():Returns an ordered list of auxiliary attribute names. (same order as get_aux_types())This is used for understanding which auxiliary attributes are available within the slow5 file, and providing selections to theauxkeyword argument in the above functoionsget_aux_types():Returns an ordered list of auxiliary attribute types (same order as get_aux_names())This can mostly be ignored, but will be used in error tracing in the future, as auxiliary field requests have multiple types, each with their own calls, and not all are used. It could be the case a call for an auxiliary filed fails, and knowing which type the field is requesting is very helpful in understanding which function in C is being called, that could be causing the error.get_aux_enum_labels(label):Returns an ordered list representing the values in the enum struct in the type header.The value in the read can then be used to access the labels as an index to the list.Example:s5=slow5.Open(file,'w')end_reason_labels=s5.get_aux_enum_labels('end_reason')print(end_reason_labels)>['unknown','partial','mux_change','unblock_mux_change','signal_positive','signal_negative']# or from newer datsets>["unknown","mux_change","unblock_mux_change","data_service_unblock_mux_change","signal_positive","signal_negative"]readID="r1"read=s5.get_read(readID,aux='all')er_index=read['end_reason']er=end_reason_labels[er_index]print("{}:{}".format(er_index,er))>4:signal_positiveWriting a fileTo write a file,modeinOpen()must be set to'w'and when appending,'a'get_empty_header(aux=False):Returns a dictionary containing all known header attributes with their values set toNone.User can modify each value, and add or remove attributes to be used has header items.
All values end up stored as strings, and anything left asNonewill be skipped.
To write header, seewrite_header()Ifaux=True, an ordered list of strings for the enumend_reasonwill be returned.
This can be modified depending on the end reason.Example:s5=slow5.Open(file,'w')header=s5.get_empty_header()end_reasonenum examples5=slow5.Open(file,w)header,end_reason_labels=s5.get_empty_header(aux=True)write_header(header, read_group=0, end_reason_labels=None):Write header to fileheader= populated dictionary fromget_empty_header()read_group = read group integer for when multiple runs are written to the same slow5 fileend_reason_labels = ordered list used for end_reason enumreturns 0 on success, <0 on error with error codeYou must writeread_group=0(default) first before writing any other read_groups, and it is advised to write read_groups in sequential order.Example:# Get some empty headersheader=s5.get_empty_header()header2=s5.get_empty_header()# Populate headers with some test datacounter=0foriinheader:header[i]="test_{}".format(counter)counter+=1foriinheader2:header2[i]="test_{}".format(counter)counter+=1# Write first read groupret=s5.write_header(header)print("ret: write_header():{}".format(ret))# Write second read group, etcret=s5.write_header(header2,read_group=1)print("ret: write_header():{}".format(ret))end_reasonexample:# Get some empty headersheader,end_reason_labels=s5.get_empty_header(aux=True)# Populate headers with some test datacounter=0foriinheader:header[i]="test_{}".format(counter)counter+=1# Write first read groupret=s5.write_header(header,end_reason_labels=end_reason_labels)print("ret: write_header():{}".format(ret))get_empty_record(aux=False):Get empty read record for populating with data. Use withwrite_record()aux = Bool for returning empty aux dictionary as well as read dictionaryreturns a single read dictionary or a read and aux dictionary depending on aux flagExample:# open some file to read. We will copy the data then write it# including aux fieldss5_read=slow5.Open(read_file,'r')reads=s5_read.seq_reads(aux='all')# For each read in s5_read...forreadinreads:# get an empty record and aux dictionaryrecord,aux=s5.get_empty_record(aux=True)# for each field in read...foriinread:# if the field is in the record dictionary...ifiinrecord:# copy the value over...record[i]=read[i]dosameforauxdictionaryifiinaux:aux[i]=read[i]# write the recordret=s5.write_record(record,aux)print("ret: write_record():{}".format(ret))write_record(record, aux=None):Write a record and optional aux fields.record = a populated dictionary fromget_empty_record()aux = an empty aux record returned byget_empty_record(aux=True)returns 0 on success and -1 on error/failureExample:record,aux=s5.get_empty_record(aux=True)# populate record, aux dictionaries#....# Write recordret=s5.write_record(record,aux)print("ret: write_record():{}".format(ret))write_record_batch(records, threads=4, batchsize=4096, aux=None):Write a record and optional aux fields, using multiple threadsrecords = a dictionary of dictionaries where each entry is a populated form ofget_empty_record()with the key of each being the read['read_id'].threads = number of threads to use in the C backend.batchsize = number of reads to write at a time. If parsing 1000 records, with batchsize=250 and threads=4, 4 threads will be spawned 4 times to write 250 records to the file before returningaux = an empty aux record returned byget_empty_record(aux=True)returns 0 on success and -1 on error/failureExample:record,aux=s5.get_empty_record(aux=True)# populate record, aux#....records[record['read_id']]=recordauxs[record['read_id']]=aux# Write recordret=s5.write_record_batch(records,threads=2,batchsize=3,aux=auxs)print("ret: write_record():{}".format(ret))close():Closes a record open for writing or appending, and writes an End Of File (EOF) flag.If not explicitly closed, when thes5object goes out of context in python, it will also trigger a close to attempt to avoid having a missing EOF.Please call this when you are finished writing a file.Example:s5=slow5.Open(file,'w')# do some writing....# Write's EOF and closes files5.close()CitationPlease cite the following in your publications when usingslow5lib/pyslow5:Gamaarachchi, H., Samarakoon, H., Jenner, S.P. et al. Fast nanopore sequencing data analysis with SLOW5. Nat Biotechnol 40, 1026-1029 (2022).https://doi.org/10.1038/s41587-021-01147-4@article{gamaarachchi2022fast,
  title={Fast nanopore sequencing data analysis with SLOW5},
  author={Gamaarachchi, Hasindu and Samarakoon, Hiruna and Jenner, Sasha P and Ferguson, James M and Amos, Timothy G and Hammond, Jillian M and Saadat, Hassaan and Smith, Martin A and Parameswaran, Sri and Deveson, Ira W},
  journal={Nature biotechnology},
  pages={1--4},
  year={2022},
  publisher={Nature Publishing Group}
}