Nucleotide Transformer v3
Collection
10 items • Updated
Request access to the NTv3 models
One approval grants access to every gated NTv3 model. Requests are reviewed manually.
The NTv3 weights are released under the InstaDeep Open Model Licence for non-commercial use only, and its terms also cover the models' outputs. For commercial use, contact legal@instadeep.com.
Log in or Sign Up to review the conditions and access this model content.
NTv3 pretrained
The pretrained backbone of Nucleotide Transformer v3 (NTv3): a genomic foundation model that reads DNA at single-base resolution across long context windows. Use it for sequence embeddings, for masked-token prediction, and as the starting point for your own fine-tuning.
This checkpoint is the backbone the other NTv3 capabilities are built on. The task-adapted models — functional-track prediction and gene annotation — start from these weights and resolve their modelling code from this repository.
Architecture
A U-Net around a transformer bottleneck. A convolutional tower reads the sequence and downsamples it by 128, a transformer tower carries long-range context at that reduced resolution, and a deconvolutional tower returns predictions at base resolution, with skip connections from the encoder at every scale.
The transformer uses grouped-query attention with per-head query, key and value normalisation, rotary position embeddings on a frequency schedule chosen for genomic distances, and a GeGLU feed-forward block.
| Parameters | 703,349,146 |
| Resolution | 1 bp |
| Input length | any multiple of 128 |
| Vocabulary | 11 tokens |
| Precision | float32 |
Usage
import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer
repo = "InstaDeepAI/NTv3_pretrained"
model = AutoModelForMaskedLM.from_pretrained(repo, trust_remote_code=True).eval()
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
# The input length must be a multiple of 128.
sequence = "ACGT" * 256
# add_special_tokens=False: the model reads a genomic window, so one base is one
# token and nothing is prepended.
batch = tokenizer([sequence], return_tensors="pt", add_special_tokens=False)
with torch.no_grad():
out = model(**batch)
out.embedding.shape # (1, 1024, 1024) one vector per base
out.logits.shape # (1, 1024, 11) masked-token prediction
AutoModel works too, and outputs are reachable both as attributes
(out.logits) and as keys (out["logits"]).
trust_remote_code=True is required because the modelling code ships with the
repository rather than in transformers.
An attention_mask is accepted and ignored: the model derives its own mask
from the pad token at the resolution of its 128 bp blocks, so a per-token mask
has nowhere to apply. Pad with the pad token rather than masking.
Outputs
| Key | Shape | What it is |
|---|---|---|
embedding |
(batch, length, 1024) |
Decoder output, one vector per base |
logits |
(batch, length, 11) |
Masked-token prediction over the vocabulary |
after_transformer_embedding |
(batch, length / 128, 2048) |
Trunk output, one vector per 128 bp block |
before_transformer_embedding |
(batch, length / 128, 2048) |
Encoder output, before the transformer |
Use skip_lm_head=True to skip the language-modelling head when you only need
embeddings.
Input requirements
Two constraints, both of which fail quietly rather than loudly.
Length must be a multiple of 128. That is the total downsampling factor of the trunk. Pad or crop your window before calling the model.
The alphabet is uppercase. This checkpoint's vocabulary is A C G T N plus the
special tokens; lowercase is not in it. Most reference assemblies are distributed
soft-masked, with repeat regions in lowercase, and feeding one straight in turns nearly
every base into <unk>. The model will still return numbers. Upper-case your sequence
first, or use InstaDeepAI/NTv3_tracks, which reads a 16-symbol soft-masked alphabet.
Characters outside the vocabulary become <unk> and the sequence keeps its length.
Intended use
Embeddings and masked-token prediction on genomic DNA, and fine-tuning on your own task. The model was pretrained across a broad panel of species, so it is not specific to one genome.
Out of scope
Not a clinical or diagnostic tool, and not validated for variant interpretation in a medical setting. Not designed for RNA, protein, or non-genomic sequence. Predictions on sequence unlike anything in the pretraining distribution — synthetic constructs, assemblies from very distant clades — should be treated as unvalidated rather than as evidence.
Limitations
The model sees sequence only. It has no access to cell type, developmental stage, experimental condition, or any annotation, so nothing it outputs is conditioned on that context.
Masked-token prediction reflects the statistics of the training genomes. High confidence on a masked base means the surrounding sequence is predictive, not that the base is functionally important.
Accuracy varies by species and by genomic region, and is not uniform across the genome. Repetitive and low-complexity regions are harder, and this checkpoint cannot see the soft-masking that would tell it where they are.
Related models
| Model | For |
|---|---|
NTv3_tracks |
Functional-track prediction, species-conditioned, from 16 kb to 1 Mb inputs |
NTv3_tracks_1mb |
The same tracks from 1 Mb inputs only: the specialist behind the NTv3 paper's main results |
NTv3_annot: _mammals / _plants / _insects |
Ab initio gene annotation, one model per clade |
Licence
Released under the InstaDeep Open Model Licence. The licence restricts output as well as weights, so predictions made with this model carry the same terms. Read it before building on the model commercially.
Citation
Citation to be added on publication.
- Downloads last month
- 38