Package: picopore
A tool for reducing the size of Oxford Nanopore Technologies’ datasets without losing information.If you find Picopore useful, please cite it athttp://dx.doi.org/10.12688/f1000research.11022.1Options:Raw compression: reduces footprint by removing event detection and basecall data, leaving only raw signal, configuration data and a choice of FASTQ data, basecall summary, both or neither;Lossless compression: reduces footprint without reducing the ability to use other nanopore tools by using HDF5’s inbuilt gzip functionality;(NOTE: as of May 2017, Oxford Nanopore Technologies implemented all compression strategies used in Picopore’s lossless compression. Recently basecalled files will therefore not benefit from this compression.)Deep lossless compression: reduces footprint without removing any data by indexing basecalled dataset to the event detection dataset.(NOTE: deep lossless compression will have the greatest impact on 2D datasets. Further work to implement 1D^2 compression is in progress.)Author: Scott Gigante, Walter & Eliza Hall Institute of Medical
Research. Contact:Email,TwitterInstallationInstall via pypiThe latest stable version of Picopore is available on PyPi. Install it using the following command:pip install picoporeInstall via condaPicopore and dependencies could also be installed using conda.conda install picopore -c bioconda -c conda-forgeInstall from sourceFor the bleeding edge, clone and install from GitHub.git clone https://www.github.com/scottgigante/picopore
cd picopore
python setup.py installCurrently,h5pyis only available on Windows viaconda.RequirementsPicopore runs on Python 2.7, 3.4, 3.5, 3.6 or 3.7 with development headers (python-devor similar).Picopore requiresh5repackfromhdf5-tools, which can be
downloaded fromhttps://support.hdfgroup.org/downloads/index.htmlor
usingsudoapt-getinstallhdf5-toolsor similar.Picopore requires the following Python packages:h5pyfuturewatchdog(for real-time compression)In addition,h5pyrequires HDF5 1.8.4 or later (libhdf5-devor similar). Difficulties resolving dependencies ofh5pycan be resolved by installing from your package manager, usingsudoapt-getinstallpython-h5pyor similar.Usagecommands: picopore
          picopore-realtime      monitors a directory for new reads and compresses them in real time
          picopore-test          compresses to temporary files and checks that all datasets and attributes are equal (lossless modes only)
          picopore-rename        renames groups and datasets within FAST5 filesusage: picopore [-h] --mode {lossless,deep-lossless,raw} [--revert] [--fastq]
                [--summary] [--manual STR] [-v] [-y] [-t INT] [--prefix STR]
                [--skip-root] [--print-every INT]
                [input [input ...]]positional arguments:
  input                 list of directories or fast5 files to shrink

optional arguments:
  -h, --help            show this help message and exit
  --mode {lossless,deep-lossless,raw}
                        choose compression mode
  --revert              reverts files to original size (lossless modes only)
  --fastq, --no-fastq   retain FASTQ data (raw mode only) (Default: --fastq)
  --summary, --no-summary
                        retain summary data (raw mode only) (Default: --no-
                        summary)
  --manual STR          manually remove only groups whose paths contain STR
                        (raw mode only, regular expressions permitted,
                        overrides defaults)
  -v, --version         show version number and exit
  -y                    skip confirm step
  -t INT, --threads INT
                        number of threads (Default: 1)
  --prefix STR          add prefix to output files to prevent overwrite
  --skip-root, --no-skip-root
                        ignore files in root input directories for albacore
                        realtime compression (Default: --no-skip-root)
  --print-every INT     print a dot every approximately INT files, or -1 to
                        silence (Default: 100)It is necessary to choose one compression mode out oflossless,deep-lossless, andraw.Note that onlylosslessanddeep-losslessare options for--revert.For--manualraw compression, the entire group path is used for matching. For example,
you could use the commandpicopore--moderaw--manual1D.*Events[...]to remove the
groups/Analyses/Basecall_1D_000/BaseCalled_template/Eventsand/Analyses/Basecall_1D_000/BaseCalled_complement/Events.Compression ModesPicopore compression allows most nanopore tools to operate unimpeded. We
provide a list of software tools which can operate on compressed files
unimpeded, and the process required to recover the necessary data if
this is not possible.FunctionalityLosslessDeep LosslessRawRaw--no-fastqMetrichoryespicopore--revertyesyesnanonetcallyespicopore--revertyesyesporetools fastqyespicopore--revertyesnanonetcall / MetrichorpoRe printfastqyespicopore--revertyesnanonetcall / Metrichornanopolish consensusyespicopore--revertnanonetcall / Metrichornanonetcall / MetrichorFAQsWhy would I want to shrink my fast5 files?Nanopore runs are big. Really big. Over a long period of time, the
storage footprint of a Nanopore lab will increase to unsustainable
levels.A large proportion of the data stored in ONT’s fast5 files is
unnecessary for the average end-user; during the basecalling process, a
large amount of intermediary data is generated, and for most users who
simply need the FASTQ, this data is useless.Picopore solves this problem. Without removing the raw signal or
configuration data used for basecalling, Picopore removes the
intermediary datasets to reduce the size of your Nanopore dataset.Do I lose functionality when using Picopore?LosslessLossless compression uses HDF5’s builtin compression, so all existing
fast5 tools will work seamlessly.Use case: power users who wish to reduce server storage footprintDeep LosslessDeep lossless compression modifies the structure of your fast5 file: any
data extraction tools will not work until you runpython picopore.py--revert--modedeep-lossless[input].Use case: power users who wish to reduce the size of their files during data transfer, or for long-term storageRawRaw compression removes the “squiggle-space” data. For most users, this
data is not critical; the only tools we know of which use the
squiggle-space data arenanopolish,nanorawandnanonettrain. If you do not intend on using these tools, your tools
will work as before. If you do intend to use these tools, the raw signal
is retained, and you can resubmit the files for basecalling to generate
new squiggle-space data.Use case: end users who are only interested in using the FASTQ dataUse case: power users running local basecalling with limited local disk space, who wish to use FASTQ immediately and will submit reads to Metrichor at a later dateRaw--no-fastqMinimal compression removes all data not required to rerun basecalling
on the fast5 files. This is only recommended for long-term storage, and
requires files to be re-basecalled for any data to be retrieved.Use case: users storing historical runs for archive purposes, with no short-term plans to use these readsDo I lose any data when using Picopore?Technically yes, but nothing that cannot be recovered. In the case where
you need to access the data which has been removed, you can regenerate
it using either picopore (on lossless compression) or using any
basecaller provided by ONT (for other methods.)Note that, since ONT’s base calling is continuously improving, the
basecalls generated when re-basecalling your data may not be the same,
but in fact higher quality than before. If it is important that you
retain the squiggle-space of the original called sequence, it is
recommended that you use a lossless compression method.