LoCUS

Pretrained LoCUS models: DNA language models that read DNA as segments from locally consistent parsing (LCP). Code, fine-tuning recipes and the scripts that reproduce every result of the paper: https://github.com/STORMgroup/locus

Folder	Model
locus-0.5m/	LoCUS-0.5M: width 64, 4 kb context
locus-2m/	LoCUS-2M: width 128, 4 kb context
twin-fixed-2m/	Tokenizer study: LoCUS-2M with fixed 10 bp segments every 5 bp
twin-bpe-2m/	Tokenizer study: LoCUS-2M with BPE tokens (vocabulary 4,096) as segments
locus-0.5m-no-chr20-21/	Pretraining-split ablation: LoCUS-0.5M pretrained without chromosomes 20 and 21
locus-2m-no-chr20-21/	Pretraining-split ablation: LoCUS-2M pretrained without chromosomes 20 and 21
twin-fixed-2m-no-chr20-21/	Pretraining-split ablation: twin-fixed-2m pretrained without chromosomes 20 and 21
twin-bpe-2m-no-chr20-21/	Pretraining-split ablation: twin-bpe-2m pretrained without chromosomes 20 and 21 (its BPE vocabulary trained on those chromosomes)

Each folder has config.json (the pretraining configuration and step), model.safetensors (the weights) and a model card. The first four are pretrained on GRCh38 chromosomes 3-22. The four -no-chr20-21 models are the same models pretrained on chromosomes 1-19 and 22, without the chromosomes of the Nucleotide Transformer benchmark's test sets, everything else unchanged; the paper uses them to check that seeing the test chromosomes in pretraining does not inflate the test scores.

import locus

model = locus.load("locus-2m")          # downloads locus-2m/ from this repository on first use
vectors = model.embed(["ACGT" * 100])