MicroGlot

A taxonomy-informed sparse genomic language model for microbial DNA.

MicroGlot is a 23-layer decoder-only mixture-of-experts transformer pretrained on 378.3 billion nucleotides from 3.70 million sequences across 99,700 microbial species — bacteria, archaea, fungi, protists, viruses and plasmids. It encodes the taxonomic hierarchy as hyperbolic (Poincaré) embeddings learned independently of the language-modelling objective, and uses them both as an input token and to steer expert routing.

For details, see our manuscript, A Taxonomy-Informed Sparse DNA Foundation Model for Microbial Genomics.

Two checkpoints are released:

	Path	Parameters	Use
MicroGlot	repo root	2.98 B backbone (479 M activated) + 2.98 B species encoder	taxonomy-conditioned
MicroGlot-plain	plain/	2.98 B backbone (479 M activated)	no species information

Install

pip install "transformers==4.51.3" "torch>=2.7" "huggingface_hub[hf_xet]" accelerate

# microglot.py is a small helper in this repo, not a pip package: put it next to your script or notebook
hf download athanzli/MicroGlot microglot.py example.py --local-dir .
python example.py   # smoke test; the first run downloads ~18 GB (MicroGlot 11.9 GB, MicroGlot-plain 6.0 GB)

With an NVIDIA GPU, check that torch can use it:

python -c "import torch; print(torch.__version__, torch.cuda.is_available())"

If this prints False, install torch for your driver from https://pytorch.org/get-started/locally/. pip's default torch is CPU-only on Windows, and the default Linux build (CUDA 13.0) needs NVIDIA driver 580 or newer. Without a usable GPU, microglot.py runs on the CPU and says so.

flash-attn is optional. Install it for the fastest rotary kernel; without it the model falls back to an equivalent pure-PyTorch implementation.

Requirements

Python 3.10 or newer and torch 2.7 or newer (torch 2.5 and 2.6 normalise bfloat16 activations with a larger epsilon and give different outputs; torch 2.4 cannot run the model). Measured with transformers 4.51.3 and torch 2.14 on Linux:

	MicroGlot	MicroGlot-plain
Download	11.9 GB	6.0 GB
Host RAM while loading	12 GiB	5.5 GiB
GPU memory for the weights	11.1 GiB	5.6 GiB
GPU memory, one 8,192-token input	12.0 GiB	6.2 GiB

A 16 GB GPU holds one of the two models at a time. bfloat16 needs an NVIDIA Ampere or newer GPU; on older GPUs (T4, V100, RTX 20xx; untested) pass dtype=torch.float16 to MicroGlot.from_pretrained.

The CPU works, but slowly. The bfloat16 default is fast only on CPUs with native bfloat16 instructions (AVX512-BF16 or AMX, e.g. AMD Zen 4 or Intel Sapphire Rapids and newer). On other CPUs pass dtype=torch.float32, which is 2-5x faster there but needs about twice the RAM (~15 GiB peak for MicroGlot-plain, ~34 GiB for MicroGlot).

Quickstart

from microglot import MicroGlot   # microglot.py: see Install

model = MicroGlot.from_pretrained("athanzli/MicroGlot")

dna = "ATGAGTAAAGGAGAAGAACTTTTCACTGGAGTTGTCCCAATTCTTGTTGAATTAGATGGT"

# 1. species known -> use its precomputed taxonomy embedding
emb = model.embed(dna, species="Escherichia coli")      # [1, 1024]
print(emb.shape, emb.device)                            # torch.Size([1, 1024]) cuda:0

# species names are resolved loosely; the following are equivalent
emb = model.embed(dna, species="escherichia_coli")
emb = model.embed(dna, species="ESCHERICHIA-COLI")
emb = model.embed(dna, species="  Escherichia   coli  ")

# 2. species unknown -> the built-in encoder infers one from the sequence
emb = model.embed(dna)

# 3. no species information at all (both models loaded together take ~17 GiB of GPU memory;
#    on a 16 GB GPU, run `del model; import torch; torch.cuda.empty_cache()` first)
plain = MicroGlot.from_pretrained("athanzli/MicroGlot", variant="plain")
emb = plain.embed(dna)

Species names need not be formatted exactly: case, underscores, hyphens and repeated whitespace are ignored, so "bacillus_subtilis" resolves to "Bacillus subtilis". model.resolve_species(name) returns the canonical name a query resolves to; if no match exists, the resulting error lists the closest species names (for a strain, subspecies or serovar name, its species first).

Every hidden state, for layer-wise probing:

states, mask = model.hidden_states(dna, species="Escherichia coli")
len(states)          # 23, one per decoder layer: states[0] is layer 1, states[-1] layer 23
states[-1].shape     # [1, seq_len, 1024]

emb = model.embed(dna, species="Escherichia coli", layer=11)   # decoder layer 11

layer is the decoder layer number, 1 to 23, with -1 meaning the last. Intermediate layers frequently probe better than the final layer; sweeping layer on the target task is recommended.

Your own sequences

import numpy as np

def read_fasta(path):
    """Record names and bare sequences (header lines and line breaks removed)."""
    names, seqs = [], []
    with open(path) as fh:
        for line in fh:
            line = line.strip()
            if line.startswith(">"):
                names.append(line[1:].split()[0])
                seqs.append([])
            elif line:
                seqs[-1].append(line)
    return names, ["".join(s) for s in seqs]

names, seqs = read_fasta("my_sequences.fasta")
emb = model.embed(seqs, layer=11)       # [len(seqs), 1024], float32; runs 16 sequences at a time
np.save("embeddings.npy", emb.cpu().numpy())
np.savetxt("embedding_ids.txt", names, fmt="%s")   # row i of embeddings.npy is record names[i]

# a sequence longer than the context: embed pieces that fit and average them
long_seq = seqs[0]
pieces = model.split(long_seq)          # contiguous pieces of at most 8,192 tokens each
emb_long = model.embed(pieces).mean(0, keepdim=True)   # [1, 1024]

• embed runs batch_size sequences per forward pass (default 16, about 0.5 GiB of activations for 1.5 kb inputs); for inputs of tens of kb on a 16 GB GPU use batch_size=1. hidden_states runs its whole input as one batch, so give it a few sequences at a time.
• The context is 8,192 tokens: about 43 kb of typical DNA, but less for N-rich sequences, as each N or other ambiguity code is a token of its own. microglot.py truncates longer inputs and warns; model.split cuts them into pieces that fit, as above.
• Line breaks, carriage returns and spaces are removed. Other characters outside ACGTN and the IUPAC codes (a > header line, U in RNA) are each read as N, with a warning.
• Raw cosine similarities between embeddings are uniformly high, often above 0.9 for unrelated sequences. Mean-centre or standardise the embeddings across your dataset before comparing them.

Optionally, a first analysis with NumPy alone. my_labels.tsv stands for your own metadata, one line per record: its FASTA name and a label (e.g. the species), separated by a tab.

x = np.load("embeddings.npy")
ids = np.loadtxt("embedding_ids.txt", dtype=str, ndmin=1, comments=None)   # ids[i] is the record of row i
x = x - x.mean(0)                                  # mean-centre across the dataset
x /= np.linalg.norm(x, axis=1, keepdims=True)      # L2-normalise
cos = x @ x.T                                      # [n, n] cosine similarities

# nearest-centroid classifier: fit on a random three quarters of the records, checked on the rest
label_of = dict(np.loadtxt("my_labels.tsv", dtype=str, delimiter="\t", ndmin=2, comments=None))
labels = np.array([label_of[i] for i in ids])
test = np.random.default_rng(0).random(len(ids)) < 0.25
classes = np.unique(labels[~test])
centroids = np.stack([x[~test & (labels == c)].mean(0) for c in classes])
pred = classes[np.linalg.norm(x[test, None] - centroids, axis=-1).argmin(1)]
print(f"held-out accuracy: {(pred == labels[test]).mean():.2f}")

This checks the workflow, not the model: fragments of one genome, or near-identical sequences, on both sides of a random split make the accuracy optimistic. For a real evaluation, hold out whole genomes or clades.

Without the helper

microglot.py is a lightweight convenience layer. The standard interface may be used directly (the raw model is called lm here, so that it does not replace the helper's model):

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

device = "cuda" if torch.cuda.is_available() else "cpu"
tok = AutoTokenizer.from_pretrained("athanzli/MicroGlot", trust_remote_code=True)
lm = AutoModelForCausalLM.from_pretrained(
    "athanzli/MicroGlot", trust_remote_code=True, torch_dtype=torch.bfloat16
).to(device, torch.bfloat16).eval()

batch = tok("ATGAGTAAAGGAGAAGAACTTTTCACTGGAG", return_tensors="pt").to(device)
out = lm(**batch)
out.logits.shape     # [1, seq_len, 8192]

The tokenizer adds [BOS] and [EOS] automatically, matching how the model was pretrained.

MicroGlot-plain lives in the plain/ subfolder:

lm_plain = AutoModelForCausalLM.from_pretrained(
    "athanzli/MicroGlot", subfolder="plain", trust_remote_code=True, torch_dtype=torch.bfloat16
).to(device, torch.bfloat16).eval()

tok_plain = AutoTokenizer.from_pretrained(
    "athanzli/MicroGlot", subfolder="plain", trust_remote_code=True
)

To supply a taxonomy vector yourself, pass species_emb of shape [batch, 32], or [32] for a single vector applied to the whole batch.

Species assets

species/ contains the taxonomy resources for the 99,700 species:

• species_embeddings.pt — {"names": [...], "name_to_idx": {...}, "embeddings": float32[99700, 32]}, all unit-norm.
• species_taxonomy.tsv.gz — the lineage of each species over seven ranks (genus, family, order, class, phylum, kingdom, domain/realm), row-aligned with the embedding table.

model.has_species("Bacillus subtilis")   # True
model.species_embedding("Bacillus subtilis")   # unit-norm float32[32]
model.taxonomy()                          # 99,700 rows; a DataFrame if pandas is installed

Lookups in this section accept loosely formatted names on the same basis:

model.has_species("bacillus_subtilis")       # True
model.resolve_species("BACILLUS-SUBTILIS")   # -> 'Bacillus subtilis'

Species outside this set are handled by the built-in encoder; omit species= to invoke it:

novel = "ATGGCAACTGTTAAAGCGCTGGCTGAAGCGTTCGGTGAAACCG"   # stands in for an organism not in the table
model.has_species("Nonexistent species")  # False

emb = model.embed(novel)                # encoder infers the taxonomy embedding from DNA

# or supply your own 32-d taxonomy vector
import torch
emb = model.embed(novel, species_emb=torch.randn(32))

Model details

Architecture	decoder-only transformer, next-token prediction
Layers	23
Hidden size	1024
Attention	grouped-query, 16 query / 8 key-value heads, head dim 64
Feedforward	SwiGLU, intermediate size 2816
Normalisation / position	RMSNorm (pre-norm), RoPE with θ = 500,000
Mixture of experts	312 routed experts in a U-shaped schedule [4,64,4,32,4,16,4,8,4,8,4,8,4,8,4,8,4,16,4,32,4,64,4], top-1 routing plus an always-active shared feedforward
Species conditioning	32-d Poincaré embedding, as a token after [BOS] and as FiLM modulation of routing logits
Context	8,192 tokens (≈ 43 kb)
Tokenizer	byte-pair encoding, vocabulary 8,192
Precision	bfloat16

Pretraining used 70.6 billion tokens (one epoch, ~2.03 × 10²⁰ FLOPs) on 8 NVIDIA RTX PRO 6000 Blackwell GPUs. Sources: NCBI RefSeq release 232, ICTV VMR MSL40 v1, PLSDB 2024_05_31_v2, with NCBI Taxonomy of 2025-09-01.

Averaged across decoder layers, MicroGlot reaches a mean score of 0.816 over 13 frozen-probing tasks spanning microbial traits and taxonomic classification, against 0.798 for the strongest baseline evaluated (ProkBERT-mini-long).

Limitations

• Do not supply a species prior for taxonomy-prediction tasks. Giving the model the ground-truth species while asking it to predict taxonomy is circular. Use variant="plain", or omit species=.
• The context window is 8,192 tokens (about 43 kb). microglot.py truncates longer inputs and warns; the raw model runs them, but outside the trained range.
• Pin transformers==4.51.3 (4.50 to 4.57 give identical outputs). Under transformers 5, the copy of modeling_microglot.py published up to commit a3eafee leaves the RoPE inv_freq buffers uninitialised, which gives NaN or silently wrong outputs that change from run to run, in any dtype. The current file fills them; with it, outputs under transformers 5.17 matched 4.51.3 bit for bit in our tests. Restart the Python kernel after changing versions.

Licence and attribution

Released under CC BY 4.0.

Citation

If you use MicroGlot, please cite our manuscript, A Taxonomy-Informed Sparse DNA Foundation Model for Microbial Genomics:

Li, A. Z., Wang, S., Cheng, S., Du, Y. & Liu, R. A Taxonomy-Informed Sparse DNA Foundation Model for Microbial Genomics. bioRxiv (2026). https://doi.org/10.64898/2026.09.22.753215

@article{li2026microglot,
  author  = {Li, Athan Z. and Wang, Shiyuan and Cheng, Shupeng and Du, Yuxuan and Liu, Ruishan},
  title   = {A Taxonomy-Informed Sparse {DNA} Foundation Model for Microbial Genomics},
  journal = {bioRxiv},
  year    = {2026},
  doi     = {10.64898/2026.09.22.753215},
  url     = {https://www.biorxiv.org/content/10.64898/2026.09.22.753215v1}
}