Package: blue-crab
blue-crabblue-crab is a conversion tool to convert from ONT's POD5 format to the community maintainedSLOW5/BLOW5 format. Maybe one day ONT will see the light and realise column-based file formats for row-based reading is a bad idea. Till then, Crab go snap snap!
Happy converting!SLOW5 specification:https://hasindu2008.github.io/slow5specsslow5tools:https://github.com/hasindu2008/slow5toolspyslow5:https://hasindu2008.github.io/slow5lib/pyslow5_api/pyslow5.htmlWARNINGWhile we test as much as we can and do our very best to ensure 100% data parity, we have no control over what ONT will do to pod5.Given their history of ad-hoc changes, there is bound to be cases in the future where this breaks the conversion.You may use commands likeslow5toolsquickcheck and index to verify the integrity of the created S/BLOW5 files.Quickstartpython3 -m venv ./blue-crab-venv
source ./blue-crab-venv/bin/activate
python3 -m pip install --upgrade pip

pip install blue-crab

blue-crab --helpSetupblue-crab requires python 3.8 or higher (limitation due to ONT's pod5 library). Using a virtual environment is recommended.Install zlib development libraries (and optionally zstd development libraries).The commands to zlibdevelopment librarieson some popular distributions :OnDebian/Ubuntu:sudoapt-getinstallzlib1g-dev
OnFedora/CentOS:sudodnf/yuminstallzlib-devel
OnOSX:brewinstallzlibSLOW5 files compressed withzstdoffer smaller file size and better performance compared to the defaultzlib. However,zlibruntime library is available by default on almost all distributions unlikezstdand thus files compressed withzlibwill be more 'portable'. Enabling optionalzstdsupport, requireszstd 1.3 or higher development librariesinstalled on your system:OnDebian/Ubuntu:sudoapt-getinstalllibzstd1-dev# libzstd-dev on newer distributions if libzstd1-dev is unavailableOnFedora/CentOS:sudoyumlibzstd-devel
OnOSX:brewinstallzstdpick option 2 or 3Create a virtual environment using Python 3.8+ and install blue-crab from pippython3 -m venv ./blue-crab-venv
source ./blue-crab-venv/bin/activate
python3 -m pip install --upgrade pip

# only if you want zstd support and have installed zstd development libraries for zstd build
export PYSLOW5_ZSTD=1

pip install blue-crab

blue-crab --helpCreate a virtual environment using Python 3.8+ and install blue-crab from source# clone the repo
git clone  https://github.com/Psy-Fer/blue-crab && cd blue-crab

# create venv
python3 -m venv ./blue-crab-venv
source ./blue-crab-venv/bin/activate
python3 -m pip install --upgrade pip

# only if you want zstd support and have installed zstd development libraries for zstd build
export PYSLOW5_ZSTD=1

# install blue-crab
python3 -m pip install .
blue-crab --helpYou can check your Python version by invokingpython3 --version. If your native python3 meets this requirement of >=3.8, you can use that, or use a
specific version installed with deadsnakes below. If you install with deadsnakes, you will need to call that specific python, such as python3.8 or python3.9, in all the following commands until you create a virtual environment with venv. Then once activated, you can just use python3. To install a specific version of python, the deadsnakes ppa is a good place to start:# This is an example for installing python3.8
# you can then call that specific python version
# > python3.8 -m pip --version
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt-get update
sudo apt install python3.8 python3.8-dev python3.8-venvUsagePlease visit themanual pagefor all the commands and options. Some examples are give below:# pod5 file -> slow5/blow5 file
blue-crab p2s example.pod5 -o example.blow5

# pod5 directory -> slow5/blow5 directory
blue-crab p2s pod5_dir -d blow5_dir

# slow5/blow5 -> pod5
blue-crab s2p example.blow5 -o example.pod5Note that default compression iszlibfor maximise compatibility. SLOW5 files compressed withzstdoffer smaller file size and better performance compared to the defaultzlib. If you installed blue-crab withzstdsupport, you can create zstd compressed BLOW5 as:# pod5 -> zstd compressed slow5/blow5
blue-crab p2s -c zstd pod5_dir -d blow5_dirNotesPOD5 has had a number of backward compatibility-breaking changes so far. This version of blue-crab is only tested on most recent pod5 files. blue-crab simply relies on ONT's POD5 API for reading and writing POD5 files, thus, leaving the burden of managing a library that can handle all the variants of POD5 and cleaning up the mess they create. We will not invest time to handle all these various idiosyncrasies in POD5, unlike we did for hundreds of different FAST5 formats when developing slow5tools. If your POD5 files are v0.1.5 or lower, you may checkthis old readmeout.Example comparisonThe following table compares an original 5khz pod5 file from the public zymo dataset (link below), containing 10k reads. Pod5 is using its default VBZ compression which is a mix of zstd and svb-zd for the signal.The blow5 files are conversions made using blue-crab and timed with/usr/bin/time -v <cmd>. They were carried out on an XPS 15 laptop with a modern SSD hard drive. They all have signal compression set to use svb-zd. Usingpython3.11.3.The table showspod5-vbzis slightly smaller than bothblow5-zstdandblow5-zlib. We prefer to default toblow5-zlibas it is more portable as zlib comes with most systems (as discussed above). If you want the best compression and faster conversion times however,blow5-zstdis the clear winner for blow5.methodsize (mb)time (s)pod5-vbz679-blow5-zstd6813.91blow5-zlib6897.86---blow5-xxx666-I have included an exampleblow5-xxxto show that we can make the files even smaller than pod5, and this work is under active development. However those compression techniques are currently not available in blue-crab.AcknowledgementGeorge Bourasfor providing some example becterial pod5 files.Rasmus Kirkegaardfor thispublic zymo pod5 dataset.Georgefrom ONT for help in understanding pod5 stuff.CitationGamaarachchi, H., Samarakoon, H., Jenner, S.P. et al. Fast nanopore sequencing data analysis with SLOW5. Nat Biotechnol 40, 1026-1029 (2022).https://doi.org/10.1038/s41587-021-01147-4@article{gamaarachchi2022fast,
  title={Fast nanopore sequencing data analysis with SLOW5},
  author={Gamaarachchi, Hasindu and Samarakoon, Hiruna and Jenner, Sasha P and Ferguson, James M and Amos, Timothy G and Hammond, Jillian M and Saadat, Hassaan and Smith, Martin A and Parameswaran, Sri and Deveson, Ira W},
  journal={Nature biotechnology},
  pages={1--4},
  year={2022},
  publisher={Nature Publishing Group}
}