Package: pod5
POD5 Python PackageThepod5Python package contains the tools and python API wrapping the compiled bindings
for the POD5 file format fromlib_pod5.InstallationThepod5package is available onpypiand is
installed usingpip:> pip install pod5UsageReading a POD5 FileTo read apod5file provide the theReaderclass with the inputpod5file path
and callReader.reads()to iterate over read records in the file. The example below
prints the read_id of every record in the inputpod5file.importpod5asp5withp5.Reader("example.pod5")asreader:forread_recordinreader.reads():print(read_record.read_id)To iterate over a selection of read_ids supplyReader.reads()with a collection
of read_ids which must beUUIDcompatible:importpod5asp5# Create a collection of read_id UUIDsread_ids:List[str]=["00445e58-3c58-4050-bacf-3411bb716cc3","00520473-4d3d-486b-86b5-f031c59f6591",]withp5.Reader("example.pod5")asreader:forread_recordinreader.reads(selection=read_ids):assertstr(read_record.read_id)inread_idsPlotting Signal Data ExampleHere is an example of how a user may plot a read’s signal data against time.importmatplotlib.pyplotaspltimportnumpyasnpimportpod5asp5# Using the example pod5 file providedexample_pod5="test_data/multi_fast5_zip.pod5"selected_read_id='0000173c-bf67-44e7-9a9c-1ad0bc728e74'withp5.Reader(example_pod5)asreader:# Read the selected read from the pod5 file# next() is required here as Reader.reads() returns a Generatorread=next(reader.reads(selection=[selected_read_id]))# Get the signal data and sample ratesample_rate=read.run_info.sample_ratesignal=read.signal# Compute the time steps over the sampling periodtime=np.arange(len(signal))/sample_rate# Plot using matplotlibplt.plot(time,signal)Writing a POD5 FileThepod5package provides the functionality to write POD5 files.It is strongly recommended that users first look at the available tools when
manipulating existing datasets, as there may already be a tool to meet your needs.
New tools may be added to support our users and if you have a suggestion for a
new tool or feature please submit a request on thepod5-file-format GitHub issues page.Below is an example of how one may add reads to a new POD5 file using theWriterand itsadd_read()method.importpod5asp5# Populate container classes for read metadatapore=p5.Pore(channel=123,well=3,pore_type="pore_type")calibration=p5.Calibration(offset=0.1,scale=1.1)end_reason=p5.EndReason(name=p5.EndReasonEnum.SIGNAL_POSITIVE,forced=False)run_info=p5.RunInfo(acquisition_id=...acquisition_start_time=...adc_max=......)signal=...# some signal data as numpy np.int16 arrayread=p5.Read(read_id=UUID("0000173c-bf67-44e7-9a9c-1ad0bc728e74"),end_reason=end_reason,calibration=calibration,pore=pore,run_info=run_info,...signal=signal,)withp5.Writer("example.pod5")aswriter:# Write the read objectwriter.add_read(read)Toolspod5 viewpod5 inspectpod5 mergepod5 filterpod5 subsetpod5 repackpod5 recoverpod5 convert fast5pod5 convert to_fast5pod5 updateThepod5package provides the following tools for inspecting and manipulating
POD5 files as well as converting between.pod5and.fast5file formats.To disable thetqdm <https://github.com/tqdm/tqdm>_  progress bar set the environment
variablePOD5_PBAR=0.To enable debugging output which may also output detailed log files, set the environment
variablePOD5_DEBUG=1Pod5 ViewThepod5 viewtool is used to produce a table similarr to a sequencing summary
from the contents of.pod5files. The default output is a tab-separated table
written to stdout with all available fields.This tools is indented to replacepod5 inspect readsand is over 200x faster.>pod5view--help# View the list of fields with a short description in-order (shortcut -L)>pod5view--list-fields# Write the summary to stdout>pod5viewinput.pod5# Write the summary of multiple pod5s to a file>pod5view*.pod5--outputsummary.tsv# Write the summary as a csv>pod5view*.pod5--outputsummary.csv--separator','# Write only the read_ids with no header (shorthand -IH)>pod5viewinput.pod5--ids--no-header# Write only the listed fields# Note: The field order is fixed the order shown in --list-fields>pod5viewinput.pod5--include"read_id, channel, num_samples, end_reason"# Exclude some unwanted fields>pod5viewinput.pod5--exclude"filename, pore_type"Pod5 inspectThepod5 inspecttool can be used to extract details and summaries of
the contents of.pod5files. There are two programs for users withinpod5 inspectand these are read and reads>pod5inspect--help
>pod5inspect{reads,read,summary}--helpPod5 inspect reads:warning: This tool is deprecated and has been replaced bypod5 viewwhich is significantly faster.Inspect all reads and print a csv table of the details of all reads in the given.pod5files.>pod5inspectreadspod5_file.pod5read_id,channel,well,pore_type,read_number,start_sample,end_reason,median_before,calibration_offset,calibration_scale,sample_count,byte_count,signal_compression_ratio00445e58-3c58-4050-bacf-3411bb716cc3,908,1,not_set,100776,374223800,signal_positive,205.3,-240.0,0.1,65582,58623,0.44700520473-4d3d-486b-86b5-f031c59f6591,220,1,not_set,7936,16135986,signal_positive,192.0,-233.0,0.1,167769,146495,0.437...Pod5 inspect readInspect the pod5 file, find a specific read and print its details.> pod5 inspect read pod5_file.pod5 00445e58-3c58-4050-bacf-3411bb716cc3File: out-tmp/output.pod5read_id: 0e5d6827-45f6-462c-9f6b-21540eef4426read_number:    129227start_sample:   367096601median_before:  171.889404296875channel data:channel: 2366well: 1pore_type: not_setend reason:name: signal_positiveforced Falsecalibration:offset: -243.0scale: 0.1462070643901825samples:sample_count: 81040byte_count: 71989compression ratio: 0.444run infoacquisition_id: 2ca00715f2e6d8455e5174cd20daa4c38f95fae2acquisition_start_time: 2021-07-23 13:48:59.780000adc_max: 0adc_min: 0context_tagsbarcoding_enabled: 0basecall_config_filename: dna_r10.3_450bps_hac_prom.cfgexperiment_duration_set: 2880...Pod5 mergepod5 mergeis a tool for merging multiple.pod5files into one monolithic pod5 file.The contents of the input files are checked for duplicate read_ids to avoid
accidentally merging identical reads. To override this check set the argument-D / --duplicate-ok# View help>pod5merge--help# Merge a pair of pod5 files>pod5mergeexample_1.pod5example_2.pod5--outputmerged.pod5# Merge a glob of pod5 files>pod5merge*.pod5-omerged.pod5# Merge a glob of pod5 files ignoring duplicate read ids>pod5merge*.pod5-omerged.pod5--duplicate-okPod5 filterpod5 filteris a simpler alternative topod5 subsetwhere reads are subset from
one or more input.pod5files using a list of read ids provided using the--idsargument
and writing those reads to asingle--outputfile.Seepod5 subsetfor more advanced subsetting.>pod5filterexample.pod5--outputfiltered.pod5--idsread_ids.txtThe--idsselection text file must be a simple list of valid UUID read_ids with
one read_id per line. Only records which match the UUID regex (lower-case) are used.
Lines beginning with a#(hash / pound symbol) are interpreted as comments.
Empty lines are not valid and may cause errors during parsing.Thefilterandsubsettools will assert that any requested read_ids are
present in the inputs. If a requested read_id is missing from the inputs
then the tool will issue the following error:POD5hasencounteredanerror:'Missing read_ids from inputs but --missing-ok not set'To disable this warning then the '-M / --missing-ok' argument.When supplying multiple input files to 'filter' or 'subset', the tools is
effectively performing amergeoperation. The 'merge' tool is better suited
for handling very large numbers of input files.Example filtering pipelineThis is a trivial example of how to select a random sample of 1000 read_ids from a
pod5 file usingpod5 viewandpod5 filter.# Get a random selection of read_ids>pod5viewall.pod5--ids--no-header--outputall_ids.txt
>all_ids.txtsort--random-sort|head--lines1000>1k_ids.txt# Filter to that selection>pod5filterall.pod5--ids1k_ids.txt--output1k.pod5# Check the output>pod5view1k.pod5-IH|wc-l1000Pod5 subsetpod5 subsetis a tool for subsetting reads in.pod5files into one or more
output.pod5files. See alsopod5 filterThepod5 subsettool requires amappingwhich defines which read_ids should be
written to which output. There are multiple ways of specifying this mapping which are
defined in either a.csvfile or by using a--table(csv or tsv)
and instructions on how to interpret it.pod5 subsetaims to be a generic tool to subset from multiple inputs to multiple outputs.
If your use-case is tofilterread_ids from one or more inputs into a single output
thenpod5 filtermight be a more appropriate tool as the only input is a list of read_ids.# View help>pod5subset--help# Subset input(s) using a pre-defined mapping>pod5subsetexample_1.pod5--csvmapping.csv# Subset input(s) using a dynamic mapping created at runtime>pod5subsetexample_1.pod5--tabletable.txt--columnsbarcodeCare should be taken to ensure that when providing multiple input.pod5files topod5 subsetthat there are no read_id UUID clashes. If a duplicate read_id is detected an exception
will be raised unless the--duplicate-okargument is set. If--duplicate-okis
set then both reads will be written to the output, although this is not recommended.Note on positional argumentsThe--columnsargument will greedily consume values and as such, care should be taken
with the placement of any positional arguments. The following line will result in an error
as the input pod5 file is consumed by--columnsresulting in no input file being set.# Invalid placement of positional argument example.pod5$pod5subset--tabletable.txt--columnsbarcodeexample.pod5Creating a Subset MappingTarget Mapping (.csv)The example below shows a.csvsubset target mapping. Any lines (e.g. header line)
which do not have a read_id which matches the UUID regex (lower-case) in the second
column is ignored.target, read_id
output_1.pod5,132b582c-56e8-4d46-9e3d-48a275646d3a
output_1.pod5,12a4d6b1-da6e-4136-8bb3-1470ef27e311
output_2.pod5,0ff4dc01-5fa4-4260-b54e-1d8716c7f225
output_2.pod5,0e359c40-296d-4edc-8f4a-cca135310ab2
output_2.pod5,0e9aa0f8-99ad-40b3-828a-45adbb4fd30cTarget Mapping from Tablepod5 subsetcan dynamically generate output targets and collect associated reads
based on a text file containing a table (csv or tsv) parsible bypolars.
This table file could be the output frompod5 viewor from a sequencing summary.
The table must contain a header row and a series of columns on which to group unique
collections of values. Internally this process uses thepolars.Dataframe.group_by <https://pola-rs.github.io/polars/py-polars/html/reference/dataframe/api/polars.DataFrame.group_by.html>_
function where thebyparameter is the sequence of column names specified with
the--columnsargument.Given the following example--tablefile, observe the resultant outputs given various
arguments:read_id    mux    barcode      length
read_a     1      barcode_a    4321
read_b     1      barcode_b    1000
read_c     2      barcode_b    1200
read_d     2      barcode_c    1234>pod5subsetexample_1.pod5--outputbarcode_subset--tabletable.txt--columnsbarcode
>lsbarcode_subset
barcode-barcode_a.pod5# Contains: read_abarcode-barcode_b.pod5# Contains: read_b, read_cbarcode-barcode_c.pod5# Contains: read_d>pod5subsetexample_1.pod5--outputmux_subset--tabletable.txt--columnsmux
>lsmux_subset
mux-1.pod5# Contains: read_a, read_bmus-2.pod5# Contains: read_c, read_d>pod5subsetexample_1.pod5--outputbarcode_mux_subset--tabletable.txt--columnsbarcodemux
>lsbarcode_mux_subset
barcode-barcode_a_mux-1.pod5# Contains: read_abarcode-barcode_b_mux-1.pod5# Contains: read_bbarcode-barcode_b_mux-2.pod5# Contains: read_cbarcode-barcode_c_mux-2.pod5# Contains: read_dOutput Filename TemplatingWhen subsetting using a table the output filename is generated from a template
string. The automatically generated template is the sequential concatenation ofcolumn_name-column_valuefollowed by the.pod5file extension.The user can set their own filename template using the--templateargument.
This argument accepts a string in thePython f-string style <https://docs.python.org/3/tutorial/inputoutput.html#formatted-string-literals>_
where the subsetting variables are used for keyword placeholder substitution.
Keywords should be placed within curly-braces. For example:# default template used = "barcode-{barcode}.pod5">pod5subsetexample_1.pod5--outputbarcode_subset--tabletable.txt--columnsbarcode# default template used = "barcode-{barcode}_mux-{mux}.pod5">pod5subsetexample_1.pod5--outputbarcode_mux_subset--tabletable.txt--columnsbarcodemux

>pod5subsetexample_1.pod5--outputbarcode_subset--tabletable.txt--columnsbarcode--template"{barcode}.subset.pod5">lsbarcode_subset
barcode_a.subset.pod5# Contains: read_abarcode_b.subset.pod5# Contains: read_b, read_cbarcode_c.subset.pod5# Contains: read_dExample subsetting frompod5 inspect readsThepod5 inspect readstool will output a csv table summarising the content of the
specified.pod5file which can be used for subsetting. The example below shows
how to split a.pod5file by the well field.# Create the csv table from inspect reads>pod5inspectreadsexample.pod5>table.csv
>pod5subsetexample.pod5--tabletable.csv--columnswellPod5 repackpod5 repackwill simply repack.pod5files into one-for-one output files of the same name.>pod5repackpod5s/*.pod5repacked_pods/Pod5 Recoverpod5 recoverwill attempt to recover data from corrupted or truncated.pod5files
by copying all valid table batches and cleanly closing the new files. New files are written
as siblings to the inputs with the_recovered.pod5suffix.>pod5recover--help
>pod5recoverbroken.pod5
>ls
broken.pod5broken_recovered.pod5pod5 convert fast5Thepod5 convert fast5tool takes one or more.fast5files and converts them
to one or more.pod5files.If the tool detects single-read fast5 files, please convert them into multi-read
fast5 files using the tools available in theont_fast5_apiproject.The progress bar shown during conversion assumes the number of reads in an input.fast5is 4000. The progress bar will update the total value during runtime if
required.Some content previously stored in.fast5files isnotcompatible with the POD5
format and will not be converted. This includes all analyses stored in the.fast5file.Please ensure that any other data is recovered from.fast5before deletion.By defaultpod5 convert fast5will show exceptions raised during conversion aswarningsto the user. This is to gracefully handle potentially corrupt input files or other
runtime errors in long-running conversion tasks. The--strictargument allows
users to opt-in to strict runtime assertions where any exception raised will promptly
stop the conversion process with an error.# View help>pod5convertfast5--help# Convert fast5 files into a monolithic output file>pod5convertfast5./input/*.fast5--outputconverted.pod5# Convert fast5 files into a monolithic output in an existing directory>pod5convertfast5./input/*.fast5--outputoutputs/
>lsoutputs/
output.pod5# default name# Convert each fast5 to its relative converted output. The output files are written# into the output directory at paths relatve to the path given to the# --one-to-one argument. Note: This path must be a relative parent to all# input paths.>lsinput/*.fast5
file_1.fast5file_2.fast5...file_N.fast5
>pod5convertfast5./input/*.fast5--outputoutput_pod5s/--one-to-one./input/
>lsoutput_pod5s/
file_1.pod5file_2.pod5...file_N.pod5# Note the different --one-to-one path which is now the current working directory.# The new sub-directory output_pod5/input is created.>pod5convertfast5./input/*.fast5output_pod5s--one-to-one./
>lsoutput_pod5s/
input/file_1.pod5input/file_2.pod5...input/file_N.pod5# Convert all inputs so that they have neibouring pod5 in current directory>pod5convertfast5*.fast5--output.--one-to-one.
>ls
file_1.fast5file_1.pod5file_2.fast5file_2.pod5...file_N.fast5file_N.pod5# Convert all inputs so that they have neibouring pod5 files from a parent directory>pod5convertfast5./input/*.fast5--output./input/--one-to-one./input/
>lsinput/*
file_1.fast5file_1.pod5file_2.fast5file_2.pod5...file_N.fast5file_N.pod5Pod5 convert to_fast5Thepod5 convert to_fast5tool takes one or more.pod5files and converts them
to multiple.fast5files. The default behaviour is to write 4000 reads per output file
but this can be controlled with the--file-read-countargument.# View help>pod5convertto_fast5--help# Convert pod5 files to fast5 files with default 4000 reads per file>pod5convertto_fast5example.pod5--outputpod5_to_fast5/
>lspod5_to_fast5/
output_1.fast5output_2.fast5...output_N.fast5Pod5 UpdateThepod5 updatetools is used to update old pod5 files to use the latest schema.
Currently the latest schema version is version 3.Files are written into the--outputdirectory with the same name.>pod5update--help# Update a named files>pod5updatemy.pod5--outputupdated/
>lsupdated
updated/my.pod5# Update an entire directory>pod5updateold/-oupdated/