Package: ont-remora
RemoraRemora models predict methylation/modified base status separated from basecalling.
The Remora repository is focused on the preparation of modified base training data and training modified base models.
Some functionality for running Remora models and investigation of raw signal is also provided.
For production modified base calling useDorado.
For recommended modified base downstream processing usemodkit.
For more advanced modified base data preparation from “randomers” see theBetta release community noteand reach out to customer support to inquire about access (customer.support@nanoporetech.com).InstallationInstall from pypi:pip install ont-remoraInstall from github source for development:git clone git@github.com:nanoporetech/remora.git
pip install -e remora/[tests]It is recommended that Remora be installed in a virtual environment.
For examplepython3-mvenv venv; source venv/bin/activate.See help for any Remora sub-command with the-hflag.Getting StartedRemora models predict modified bases anchored to canonical basecalls or reference sequence from a nanopore read.The Remora training/prediction input unit (referred to as a chunk) consists of:Section of normalized signalCanonical bases attributed to the section of signalMapping between these twoChunks have a fixed signal length defined at data preparation/model training time.
These values are saved with the Remora model to extract chunks in the same manner at inference.
A fixed position within the chunk is defined as the “focus position” around which the fixed signal chunk is extracted.
By default, this position is the center of the “focus base” being interrogated by the model.The canonical bases and mapping to signal (a.k.a. “move table”) are combined for input into the neural network in several steps.
First each base is expanded to the k-mer surrounding that base (as defined by the--kmer-context-baseshyper-parameter).
Then each k-mer is expanded according to the move table.
Finally each k-mer is one-hot encoded for input into the neural network.
This procedure is depicted in the figure below.Data PreparationRemora data preparation begins from a POD5 file containing signal data and a BAM file containing basecalls from the POD5 file.
Note that the BAM file must contain the move table (--emit-movesin Dorado) and the MD tag (default in Dorado with mapping and--MDargument for minimap2).
If using minimap2 for alignment usesamtools fastq-T"*" [in.bam] | minimap2-y-axmap-ont[ref.fa] - | samtools view-b-o[out.bam]in order to transfer the move table tags through the alignment step since minimap2 does not support SAM/BAM input.The following example generates training data from canonical (PCR) and modified (M.SssI treatment) samples in the same fashion as the released 5mC CG-context models.
Example reads can be found in the Remora repository (seetest/data/directory).K-mer tables for applicable conditions can be found in thekmer_models repository.remora\datasetprepare\can_reads.pod5\can_mappings.bam\--output-pathcan_chunks\--refine-kmer-level-tablelevels.txt\--refine-rough-rescale\--motifCG0\--mod-base-controlremora\datasetprepare\mod_reads.pod5\mod_mappings.bam\--output-pathmod_chunks\--refine-kmer-level-tablelevels.txt\--refine-rough-rescale\--motifCG0\--mod-basem5mCThe above commands each produce a core Remora dataset stored in the directory defined by--output-path.
Core datasets contain memory mapped numpy files for each core array (chunk data) and a JSON format metadata config file.
These memory mapped files allow efficient access to very large datasets.Before Remora, 3.0 datasets were stored as numpy array dictionaries.
Updating datasets can be accomplished with thescripts/update_dataset.pyscript included in the repository.Composing DatasetsCore datasets (or other composed datasets) can be composed to produce a new dataset.
Theremora dataset make_configcommand creates these config files specifying the composition of the new dataset.
When reading batches from these combined datasets, the default behavior will be to draw chunks randomly from the entire set of chunks.
This setting is useful for multiple flowcells of the same condition.The--dataset-weightsargument produces a config which generates batches with a fixed proportion of chunks from each input dataset.
This setting is useful when combining different data types, for example control and modified datasets.In addition, theremora dataset mergecommand is supplied to merge datasets, copying the data into a new core Remora dataset.
This may increase efficiency of data access for datasets composed of many core datasets, but only supports the default behavior from themake_configcommand.Composed dataset config files can also be specified manually.
Config files are JSON format files containing a single list, where each element is a list of two items.
The first is the path to the dataset and the second is the weight (must be a positive value).
Themake_configoutput config file will also contain the dataset hash to ensure the contents of a dataset are unchanged, but this is an optional third field in the config.Metadata attributes from each core dataset are checked for compatibility and merged where applicable.
Chunk raw data are loaded from each core dataset at specified proportions to construct batches at loading time.
In a break from Remora <3.0, datasets allow “infinite iteration”, where each core dataset is drawn from indefinitely and independently to supply training chunks.
For validation from a fixed set of chunks, finite iteration is also supported.To generate a dataset config from the datasets created above one can use the following command.remora\datasetmake_config\train_dataset.jsn\can_chunks\mod_chunks\--dataset-weights11\--log-filenametrain_dataset.logModel TrainingModels are trained with theremora model traincommand.
For example a model can be trained with the following command.remora\modeltrain\train_dataset.jsn\--modelremora/models/ConvLSTM_w_ref.py\--device0\--chunk-context5050\--output-pathtrain_resultsThis command will produce a “best” model in torchscript format for use in Bonito,remora infer, orremora validatecommands.
Models can be exported for use in Dorado with theremora model export train_results/model_best.ptcommand.Model InferenceFor testing purposes, inference within Remora is provided.
For large scale using the exported Dorado model during basecalling is recommended.remora\inferfrom_pod5_and_bam\can_signal.pod5\can_mappings.bam\--modeltrain_results/model_best.pt\--out-filecan_infer.bam\--log-filenamecan_infer.log\--device0remora\inferfrom_pod5_and_bam\mod_signal.pod5\mod_mappings.bam\--modeltrain_results/model_best.pt\--out-filemod_infer.bam\--log-filenamemod_infer.log\--device0Finally, Remora provides tools to validate these results.
Ground truthBED filesreference positions where each read should be called as the modified or canonical base listed in the BED name field.
Note in the test files where the control file has aCin the name field, while the modified BED file hasm(single letter code for 5mC) in the name field.WARNING: There is a bug in pysam which causes all-context (e.g.--motifC 0) modified model calls to produce invalid results with this command.
This issue is reportedhere.
We are investigating solutions to bypass this issue including dropping this command.remora\validatefrom_modbams\--bam-and-bedcan_infer.bamcan_ground_truth.bed\--bam-and-bedmod_infer.bammod_ground_truth.bed\--explicit-mod-tag-usedPre-trained ModelsSee the selection of current released models withremora model list_pretrained.
Pre-trained models are stored remotely and can be downloaded using theremora model downloadcommand or will be downloaded on demand when needed.Models may be run fromBonito.
See Bonito documentation to apply Remora models.More advanced research models may be supplied viaRerio.
These files require download from Rerio and then the path to this download must be provided to Remora.
Note that older ONNX format models require Remora version < 2.0.Python API and Raw Signal AnalysisRaw signal plotting is available via theremora analyze plot ref_regioncommand.Theplot ref_regioncommand is useful for gaining intuition into signal attributes and visualize signal shifts around modified bases.
As an example using the test data, the following command produces the plots below.
Note that only a single POD5 file per sample is allowed as input and that the BAM records must contain themvandMDtags (see the see “Data Preparation” section above for details).remora\analyzeplotref_region\--pod5-and-bamcan_reads.pod5can_mappings.bam\--pod5-and-bammod_reads.pod5mod_mappings.bam\--ref-regionsref_regions.bed\--highlight-rangesmod_gt.bed\--refine-kmer-level-tablelevels.txt\--refine-rough-rescale\--log-filenamelog.txtThe Remora API to access, manipulate and visualize nanopore reads including signal, basecalls, and reference mapping is described in more detail in thenotebookssection of this repository.Terms and LicenceThis is a research release provided under the terms of the Oxford Nanopore Technologies’ Public Licence.
Research releases are provided as technology demonstrators to provide early access to features or stimulate Community development of tools.
Support for this software will be minimal and is only provided directly by the developers. Feature requests, improvements, and discussions are welcome and can be implemented by forking and pull requests.
Much as we would like to rectify every issue, the developers may have limited resource for support of this software.
Research releases may be unstable and subject to rapid change by Oxford Nanopore Technologies.© 2021-2023 Oxford Nanopore Technologies Ltd.
Remora is distributed under the terms of the Oxford Nanopore Technologies’ Public Licence.Research ReleaseResearch releases are provided as technology demonstrators to provide early access to features or stimulate Community development of tools. Support for this software will be minimal and is only provided directly by the developers. Feature requests, improvements, and discussions are welcome and can be implemented by forking and pull requests. However much as we would like to rectify every issue and piece of feedback users may have, the developers may have limited resource for support of this software. Research releases may be unstable and subject to rapid iteration by Oxford Nanopore Technologies.