GB.RNA-650M

650M-parameter base model trained on a broad ncRNA corpus. This is a standalone HuggingFace port that loads without the ModelGenerator package.

Renamed. GenBio AI renamed AIDO.RNA to GB.RNA; the upstream checkpoints now live at genbio-ai/GB.RNA-*. This repository was previously Taykhoom/AIDO.RNA-650M, and that id redirects here. The weights are unchanged.

Architecture

Parameter	Value
Layers	33
Attention heads	20
Embedding dimension	1280
FFN hidden dimension	3392 (SwiGLU)
Vocabulary size	16
Positional encoding	RoPE (rotary_percent=1.0)
Normalization	LayerNorm
MLP activation	SwiGLU
Architecture	Pre-LN Transformer
Max sequence length	1024 (training truncation; RoPE has no hard limit)

Vocabulary: [PAD], [MASK], [CLS], [SEP], [UNK], A, G, C, T, U, N, [BOS], [EOS], [UNUSED1], [UNUSED2], [UNUSED3]

Inputs should use DNA notation (A, C, G, T), as in the upstream implementation. Convert U to T before tokenization; the tokenizer treats them as distinct vocabulary entries.

Pretraining

• Objective: Masked language modeling (MLM) on RNA sequences
• Data: 42M non-coding RNA sequences
• Source checkpoint: genbio-ai/GB.RNA-650M

Checkpoint selection

General-purpose base model; recommended starting point for most RNA tasks.

Parity Verification

Hidden-state representations compared against the original genbio-ai/GB.RNA-650M weights at all 34 representation levels (embedding + 33 transformer layers). Intermediate layer differences are floating-point accumulation noise normalised away by the final layer norm; the final output matches the original within 1e-5. max diff = 5.04e-06 (final output), 6.10e-05 (intermediate layers). Verified on GPU with PyTorch 2.7 / CUDA 12.

Related Models

See the full GB.RNA collection.

Model	Parameters	Data	Notes
GB.RNA-1M-MARS	1M	MARS ncRNA	Smallest MARS variant
GB.RNA-25M-MARS	25M	MARS ncRNA	Mid-size MARS variant
GB.RNA-300M-MARS	300M	MARS ncRNA	Large MARS variant
GB.RNA-650M	650M	42M ncRNA	Base model
GB.RNA-650M-CDS	650M	42M ncRNA + CDS	CDS-adapted
GB.RNA-1.6B	1.6B	42M ncRNA	Largest base model
GB.RNA-1.6B-CDS	1.6B	42M ncRNA + CDS	Largest CDS-adapted

Usage

Embedding generation

import torch
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("Taykhoom/GB.RNA-650M", trust_remote_code=True)
model = AutoModel.from_pretrained("Taykhoom/GB.RNA-650M", trust_remote_code=True)
model.eval()

sequences = ["ACGTGCTAGCTAGCTA", "ATGCTAGCTAGCTAGC"]
enc = tokenizer(sequences, return_tensors="pt", padding=True)

with torch.no_grad():
    out = model(**enc)

cls_emb   = out.last_hidden_state[:, 0, :]   # (batch, 1280) -- CLS token
token_emb = out.last_hidden_state             # (batch, seq_len, 1280)

# Intermediate layers
out_all = model(**enc, output_hidden_states=True)
layer3_emb = out_all.hidden_states[3]

MLM logits

from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("Taykhoom/GB.RNA-650M", trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained("Taykhoom/GB.RNA-650M", trust_remote_code=True)
model.eval()

enc = tokenizer(["ACGT[MASK]GCTA"], return_tensors="pt")
with torch.no_grad():
    logits = model(**enc).logits   # (1, seq_len, 16)

Fine-tuning

Standard HF conventions. Use cls_emb = out.last_hidden_state[:, 0, :] (CLS token) as input to a task-specific head for sequence-level tasks.

Implementation Notes

The original genbio-ai/GB.RNA-650M checkpoint requires the ModelGenerator package to load. This port is a clean standalone re-implementation:

• All model logic is contained in modeling_gbrna.py and configuration_gbrna.py.
• The tokenizer normalizes U/u to T/t, matching the original checkpoint preprocessing.
• attn_implementation="sdpa" and attn_implementation="flash_attention_2" are added (not present in the original genbio-ai implementation).
• Architecture: pre-LN Transformer with SwiGLU MLP and RoPE positional embeddings.

Citation

@article{zou2024_gbrna,
  title   = {A Large-Scale Foundation Model for {RNA} Function and Structure Prediction},
  author  = {Zou, Shuxian and Tao, Tianhua and Mahbub, Sazan and Ellington, Caleb N. and Algayres, Robin and Li, Dian and Zhuang, Yonghao and Wang, Hongyi and Song, Le and Xing, Eric P.},
  journal = {bioRxiv},
  year    = {2024},
  doi     = {10.1101/2024.11.28.625345}
}

Credits

Original model and code by Zou et al. Source: GitHub. Hugging Face port maintained by Taykhoom Dalal.

License

GenBio AI Community License, following the original repository. See LICENSE for details.