GB.DNA-7B

7B-parameter DNA foundation model from GenBio AI, trained on 10.6 billion nucleotides from 796 species. This is a standalone HuggingFace port that loads without the ModelGenerator package.

Renamed. GenBio AI renamed AIDO.DNA to GB.DNA; the upstream checkpoints now live at genbio-ai/GB.DNA-*. This repository was previously Taykhoom/AIDO.DNA-7B, and that id redirects here. The weights are unchanged.

Architecture

Parameter	Value
Layers	32
Attention heads	32
Embedding dimension	4352
FFN hidden dimension	11584 (SwiGLU)
Vocabulary size	16
Positional encoding	RoPE (rotary_percent=1.0)
Normalization	LayerNorm
MLP activation	SwiGLU
Architecture	Pre-LN Transformer (BERT-style encoder)
Max sequence length	4000 (training context; RoPE has no hard limit)

Vocabulary: [PAD], [MASK], [CLS], [SEP], [UNK], A, G, C, T, U, N, [BOS], [EOS], [UNUSED1], [UNUSED2], [UNUSED3]

DNA sequences use single-nucleotide tokenization over A, C, G, T, N. Each sequence is wrapped as [CLS] ... [SEP].

Note on U: the vocabulary is the shared GenBio RNABert vocabulary, so a U token exists (id 9) and the tokenizer will accept it. However, GB.DNA was pretrained on DNA (A, C, G, T, N) and never saw U during training - its embedding row is effectively untrained (embedding norm ~1.77, in line with the unused special tokens, versus ~0.69-0.97 for the trained nucleotides A/G/C/T). Do not feed U to this model; use T for thymine. The token is retained only to keep vocab_size=16 consistent with the original weights.

Pretraining

• Objective: Masked language modeling (MLM) on genomic DNA
• Data: 10.6B nucleotides from 796 genomes (Nucleotide Transformer dataset), single-nucleotide tokenization, 4000-nucleotide context
• Source checkpoint: genbio-ai/GB.DNA-7B

Checkpoint selection

The 7B model is the largest released GB.DNA checkpoint and the most accurate for functional genomics, genome mining, and unsupervised annotation. For lighter-weight deployment use Taykhoom/GB.DNA-300M.

Parity Verification

Hidden-state representations compared against the original genbio-ai/GB.DNA-7B weights (loaded into the genbio RNABertForMaskedLM reference) at all 33 representation levels (embedding + 32 transformer layers). The embedding layer matches exactly, and the final post-LayerNorm hidden state and MLM logits match within tight tolerance. Intermediate layer differences are floating-point accumulation noise in the un-normalized residual stream (relative error < 1e-6), normalized away by the final layer norm. Attention weights over valid positions sum to 1, and padded keys receive zero probability. Verified on PyTorch 2.7 / CUDA 12.

Related Models

See the full GB.DNA collection.

Model	Parameters	Notes
Taykhoom/GB.DNA-300M	300M	Smaller DNA variant
Taykhoom/GB.DNA-7B	7B	This model

Usage

Embedding generation

import torch
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("Taykhoom/GB.DNA-7B", trust_remote_code=True)
model = AutoModel.from_pretrained("Taykhoom/GB.DNA-7B", trust_remote_code=True)
model.eval()

sequences = ["ACGTACGTACGTACGT", "TTGCAACGTAGCTAGC"]
enc = tokenizer(sequences, return_tensors="pt", padding=True)

with torch.no_grad():
    out = model(**enc)

cls_emb   = out.last_hidden_state[:, 0, :]   # (batch, 4352) -- CLS token
token_emb = out.last_hidden_state             # (batch, seq_len, 4352)

# Intermediate layers
out_all = model(**enc, output_hidden_states=True)
layer3_emb = out_all.hidden_states[3]

MLM logits

from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("Taykhoom/GB.DNA-7B", trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained("Taykhoom/GB.DNA-7B", trust_remote_code=True)
model.eval()

enc = tokenizer(["ACGT[MASK]CGTA"], return_tensors="pt")
with torch.no_grad():
    logits = model(**enc).logits   # (1, seq_len, 16)

Fine-tuning

Standard HF conventions. Use cls_emb = out.last_hidden_state[:, 0, :] (CLS token) as input to a task-specific head for sequence-level tasks.

Implementation Notes

The original genbio-ai/GB.DNA-7B checkpoint requires the ModelGenerator package to load. This port is a clean standalone re-implementation:

• All model logic is contained in modeling_gbdna.py and configuration_gbdna.py.
• attn_implementation="sdpa" and attn_implementation="flash_attention_2" are added (not present in the original genbio-ai implementation).
• Architecture: pre-LN Transformer with SwiGLU MLP and RoPE positional embeddings, identical to the GB.RNA family (RNABertForMaskedLM).
• Weights are stored as 6 sharded model-0000X-of-00006.safetensors files.

Citation

@article{ellington2024_gbdna,
  title   = {Accurate and General {DNA} Representations Emerge from Genome Foundation Models at Scale},
  author  = {Ellington, Caleb N. and Sun, Ning and Ho, Nicholas and Tao, Tianhua and Mahbub, Sazan and Li, Dian and Zhuang, Yonghao and Wang, Hongyi and Song, Le and Xing, Eric P.},
  journal = {bioRxiv},
  year    = {2024},
  doi     = {10.1101/2024.12.01.625444}
}

Credits

Original model and code by Ellington et al. Source: GitHub. Hugging Face port maintained by Taykhoom Dalal.

License

GenBio AI Community License, following the original repository. See LICENSE for details.