Package: medaka-cpu
Medakamedakais a tool to create consensus sequences and variant calls from
nanopore sequencing data. This task is performed using neural networks applied
a pileup of individual sequencing reads against a draft assembly. It provides
state-of-the-art results outperforming sequence-graph based methods and
signal-based methods, whilst also being faster.© 2018- Oxford Nanopore Technologies Ltd.FeaturesRequires only basecalled data. (.fastaor.fastq)Improved accuracy over graph-based methods (e.g. Racon).50X faster than Nanopolish (and can run on GPUs).Includes extras for implementing and training bespoke correction
networks.Works on Linux and MacOS.Open source (Mozilla Public License 2.0).For creating draft assemblies we recommendFlye.InstallationMedaka can be installed in one of several ways.Installation with pipOfficial binary releases of medaka are available onPyPIand can be installed using pip:pip install medakaOn Linux platforms this will install a precompiled binary, on MacOS (and other)
platforms this will fetch and compile a source distribution.We recommend using medaka within a virtual environment, viz.:virtualenv medaka --python=python3 --prompt "(medaka) "
. medaka/bin/activate
pip install --upgrade pip
pip install medakaUsing this method requires the user to provide several binaries:samtools,minimap2,tabix, andbgzipand place these within thePATH.samtools/bgzip/tabixversion 1.14 andminimap2version 2.17 are recommended as these are those used in development
of medaka. (Newer versions are almost certainly fine).Installation with condaThe bioconda medaka packages are no longer supported by Oxford Nanopore Technologies.For those who prefer the conda package manager, medaka is available via thebiocondachannel:conda create -n medaka -c conda-forge -c bioconda medakaThe bioconda releases lag behind the source code andPyPIreleases.Installation from sourceThis method is useful for macOS M1 devices as it will assist in building
dependencies which will fail with the other methods above.Medaka can be installed from its source quite easily on most systems.Before installing medaka it may be required to install some
prerequisite libraries, best installed by a package manager. On Ubuntu
theses are:bzip2 g++ zlib1g-dev libbz2-dev liblzma-dev libffi-dev libncurses5-dev
libcurl4-gnutls-dev libssl-dev curl make cmake wget python3-all-dev
python-virtualenvIn addition it is required to install and set up git LFS before cloning
the repository.A Makefile is provided to fetch, compile and install all direct dependencies
into a python virtual environment. To set-up the environment run:# Note: certain files are stored in git-lfs, https://git-lfs.github.com/,
#       which must therefore be installed first.
git clone https://github.com/nanoporetech/medaka.git
cd medaka
make install
. ./venv/bin/activateUsing this method bothsamtoolsandminimap2are built from source and need
not be provided by the user.Using a GPUSince version 1.1.0medakauses Tensorflow 2, prior versions used Tensorflow 1.
Formedaka1.1.0 and higher installation from source or usingpipcan make
immediate use of GPUs. However, note that thetensorflowpackage is compiled against
specific versions of the NVIDIA CUDA and cuDNN libraries; users are directed to thetensorflow installationpages
for further information. cuDNN can be obtained from thecuDNN Archive, whilst CUDA
from theCUDA Toolkit Archive.Formedakaprior to version 1.1.0, to enable the use of GPU resource it is
necessary to install thetensorflow-gpupackage. Using the source code from github
a working GPU-poweredmedakacan be configured with:# Note: certain files are stored in git-lfs, https://git-lfs.github.com/,
#       which must therefore be installed first.
git clone https://github.com/nanoporetech/medaka.git
cd medaka
sed -i 's/tensorflow/tensorflow-gpu/' requirements.txt
make installGPU Usage notesDepending on your GPU,medakamay show out of memory errors when running.
To avoid these the inference batch size can be reduced from the default
value by setting the-boption when runningmedaka_consensus. A value-b 100is suitable for 11Gb GPUs.For users with RTX series GPUs it may be required to additionally set an
environment variable to havemedakarun without failure:export TF_FORCE_GPU_ALLOW_GROWTH=trueIn this situation a further reduction in batch size may be required.Using DockerThe source code repository contains aDockerfilewhich can be used to create
a GPU compatible Docker container image with the appropriate CUDA and cuDNN
library versions for running medaka. The image is built on top of imagesprovided by NVIDIAdesigned to run with theNVIDIA Container
Toolkit.
With the toolkit setup on your host computer the following command can be used
to run the latest version of medaka:docker run --rm --gpus 0 ontresearch/medaka:latest medaka --help(The--gpusoption can be amended as appropriate for your environment). Versioned
tags are also available.Usagemedakacan be run using its default settings through themedaka_consensusprogram. An assembly in.fastaformat and basecalls in.fastaor.fastqformats are required. The program uses bothsamtoolsandminimap2. If
medaka has been installed using the from-source method these will be present
within the medaka environment, otherwise they will need to be provided by
the user.source ${MEDAKA}  # i.e. medaka/venv/bin/activate
NPROC=$(nproc)
BASECALLS=basecalls.fa
DRAFT=draft_assm/assm_final.fa
OUTDIR=medaka_consensus
medaka_consensus -i ${BASECALLS} -d ${DRAFT} -o ${OUTDIR} -t ${NPROC} -m r941_min_high_g303The variablesBASECALLS,DRAFT, andOUTDIRin the above should be set
appropriately. For the selection of the model (-m r941_min_high_g303in the
example above) see the Model section following.Whenmedaka_consensushas finished running, the consensus will be saved to${OUTDIR}/consensus.fasta.Bacterial (ploidy-1) variant callingVariant calling for monoploid samples is enabled through themedaka_haploid_variantworkflow:medaka_haploid_variant -i <reads.fastq> -r <ref.fasta>which requires the reads as a.fastaor.fastqand a reference sequence as a.fastafile.Diploid variant callingThe diploid variant calling workflowmedaka_variantthat was historically implemented
within the medaka package has been surpassed in accuracy and compute performance by
other methods, it has therefore been deprecated. Our current recommendation for
performing this task is to useClair3either directly
or through the Oxford Nanopore Technologies provided Nextflow implementation available
throughEPI2ME Labs.ModelsFor best results it is important to specify the correct model,-min the
above, according to the basecaller used. Allowed values can be found by
runningmedaka tools list\_models.Recent basecallersRecent basecaller versions annotate their output with their model version.
In such cases medaka can inspect the files and attempt to select an appropriate
model for itself. This typically works best in the case of BAM output from
basecallers. It will work also for FASTQ input provided the FASTQ has been
created from basecaller output using:samtools fastq -T '*' dorado.bam | gzip -c > dorado.fastq.gzThe commandmedaka consensuswill attempt to automatically determine a
correct model by inspecting its BAM input file. The helper scriptsmedaka_consensusandmedaka_haploid_variantwill make similar attempts
from their FASTQ input.To inspect files for yourself, the command:medaka tools resolve_model --auto_model <consensus/variant> <input.bam/input.fastq>will print the model that automatic model selection will use.For older basecallers and when automatic selection is unsuccessfulIf the name of the basecaller model used is known, but has been lost from the input
files, the basecaller model can been provided to medaka directly. It must however
be appended with either:consensusor:variantaccording to whether the user
wishing to use the consensus or variant calling medaka model. For example:medaka consensus input.bam output.hdf \
    --model dna_r10.4.1_e8.2_400bps_hac@v4.1.0:variantwill use the medaka variant calling model appropriate for use with the basecaller
model nameddna_r10.4.1_e8.2_400bps_hac@v4.1.0.Medaka models are named to indicate i) the pore type, ii) the sequencing
device (MinION or PromethION), iii) the basecaller variant, and iv) the
basecaller version, with the format:{pore}_{device}_{caller variant}_{caller version}For example the model namedr941_min_fast_g303should be used with data from
MinION (or GridION) R9.4.1 flowcells using the fast Guppy basecaller version
3.0.3. By contrast the modelr941_prom_hac_g303should be used with PromethION
data and the high accuracy basecaller (termed "hac" in Guppy configuration
files). Where a version of Guppy has been used without an exactly corresponding
medaka model, the medaka model with the highest version equal to or less than
the guppy version should be selected.Improving parallelismThemedaka_consensusprogram is good for simple datasets but perhaps not
optimal for running large datasets at scale. A higher level of parallelism
can be achieved by running independently the component steps ofmedaka_consensus. The program performs three tasks:alignment of reads to input assembly (viamini_alignwhich is a thin
veil overminimap2)running of consensus algorithm across assembly regions
(medaka consensus, note no underscore!)aggregation of the results of 2. to create consensus sequences
(medaka stitch)The three steps are discrete, and can be split apart and run independently. In
most cases, Step 2. is the bottleneck and can be trivially parallelized. Themedaka consensusprogram can be supplied a--regionsargument which will restrict its action to particular assembly sequences from
the.bamfile output in Step 1. Therefore individual jobs can be run for batches
of assembly sequences simultaneously. In the final step,medaka stitchcan take as input one or more of the.hdffiles output by Step 2.So in summary something like this is possible:# align reads to assembly
mini_align -i basecalls.fasta -r assembly.fasta -P -m \
    -p calls_to_draft.bam -t <threads>
# run lots of jobs like this, change model as appropriate
mkdir results
medaka consensus calls_to_draft.bam results/contigs1-4.hdf \
    --model r941_min_fast_g303 --batch 200 --threads 8 \
    --region contig1 contig2 contig3 contig4
...
# wait for jobs, then collate results
medaka stitch results/*.hdf polished.assembly.fastaIt is not recommended to specify a value of--threadsgreater than 2 formedaka consensussince the compute scaling efficiency is poor beyond this.
Note also thatmedaka consensusmay been seen to use resources equivalent to<threads> + 4as an additional 4 threads are used for reading and preparing
input data.Origin of the draft sequenceMedaka has been trained to correct draft sequences output from theFlyeassembler.Processing a draft sequence from alternative sources (e.g. the output ofcanuorwtdbg2) may lead to different results.Historical correction
models in medaka were trained to correct draft sequences output from the canu
assembler withraconapplied either once,
or four times iteratively. For contemporary models this is not the case and
medaka should be used directly on the output of Flye.AcknowledgementsWe thankJoanna PinedaandJared Simpsonfor providing htslib code samples which aided
greatly development of the optimised feature generation code, and for testing the
version 0.4 release candidates.We thankDevin Drownforworking throughuse ofmedakawith his RTX 2080 GPU.HelpLicence and Copyright© 2018- Oxford Nanopore Technologies Ltd.medakais distributed under the terms of the Mozilla Public License 2.0.Research ReleaseResearch releases are provided as technology demonstrators to provide early
access to features or stimulate Community development of tools. Support for
this software will be minimal and is only provided directly by the developers.
Feature requests, improvements, and discussions are welcome and can be
implemented by forking and pull requests. However much as we would
like to rectify every issue and piece of feedback users may have, the
developers may have limited resource for support of this software. Research
releases may be unstable and subject to rapid iteration by Oxford Nanopore
Technologies.