Reviewer access request

This repository is reserved for peer reviewers. Please sign in to Hugging Face and submit an access request with your review purpose. MulinLab will review requests manually.

hEvo population-aware DNA foundation model: population diversity, genetic variation, and evolutionary history

hEVO-base-o

A haplotype-based DNA foundation model · 87.1M parameters · 12 layers · 768 hidden units · 1,024-nt context

Learning human genetic variation across populations and evolution. Use the final-layer nucleotide representations for downstream analyses, or score a single-nucleotide change from masked-base logits.

Model overview · Quick start · Nucleotide embeddings · SNV scoring · Benchmarks · Code

Model overview
Intended use
Training data and objective
Quick start
Nucleotide embeddings
Single-nucleotide variant scoring
Input and output
Evaluation and reproducibility
Provenance, access and citation

Model overview

hEVO-base-o is the base checkpoint in the hEvo public-data model family. It is a bidirectional BERT-style masked language model with one token per nucleotide and a maximum context of 1,024 nucleotides.

Configuration at a glance

Property	Value
Parameters	87.1M
Transformer layers	12
Hidden size	768
Attention heads	12
Maximum context	1,024 nucleotides
Input alphabet	A, C, G, T and N; 10-token vocabulary including special tokens
Tokenization	One token per nucleotide; no CLS or SEP tokens are added
Inference interface	`AutoModelForMaskedLM` with custom code
Frequency embeddings	Disabled by default; DNA-only inference
Stored precision	bfloat16 checkpoint; examples load float32

Checkpoint family

Checkpoint	Parameters	Layers × hidden size	Heads	Frequency embeddings
hEVO-base-o	87.1M	12 × 768	12	Disabled by default
hEVO-large-o	319.5M	16 × 1,280	20	Disabled by default

Both models use a 1,024-nt context and the same nucleotide tokenizer. The separate hEvo benchmark collection documents task inputs, sources and build workflows; GitHub contains the code and reproducible analysis workflows.

The base checkpoint contains optional frequency-branch tensors in its weight file. With the default configuration set to false, that branch is not instantiated or used; Transformers may report those tensors as unused checkpoint keys. The weights remain available in the checkpoint for provenance.

Intended use

The checkpoint provides contextual DNA representations and masked-nucleotide logits. The examples below show two direct uses:

Use	Output	Typical downstream step
Nucleotide representation	One hidden vector per input nucleotide	Feed position-level vectors to a task-specific model or analysis
Sequence summary	Optional mean of valid nucleotide vectors	Use as a simple sequence-level representation
Single-SNV comparison	ALT-minus-REF conditional log-likelihood ratio at a masked site	Compare the two alleles in the same sequence context

These are model outputs, not a clinical interpretation. The repository benchmark card describes the separately constructed evaluation tasks and data.

Training data and objective

The checkpoints were independently pretrained on public 1000 Genomes Project and Human Genome Diversity Project (HGDP) sequence data using self-supervised masked language modeling. The checkpoint provenance records no target benchmark label supervision. Dataset preparation and checkpoint provenance are linked in provenance.json and the benchmark repository.

Quick start

Install and authenticate

Use a Python environment with PyTorch and the checkpoint's tested Transformers version:

python -m pip install "torch>=2.6" "transformers==4.57.1" safetensors numpy webdataset huggingface_hub
hf auth login

Access is gated for peer reviewers and requires manual approval by MulinLab. Sign in to Hugging Face, submit an access request on this page, and wait for approval. After approval, run hf auth login with your own read token; token=True below uses your locally stored credential. The model and tokenizer load repository Python code through trust_remote_code=True; review model.py and data.py before loading.

Load once, then run either example

Both examples share this setup. It uses float32 and eager attention on CPU or CUDA, with Flash Attention disabled. Frequency conditioning is off: the model is constructed with use_frequency_embedding=False, and no frequency or ancestry values are passed. The pinned revision identifies the weights and custom code; the model card and artwork are maintained on main.

import torch
from transformers import AutoConfig, AutoModelForMaskedLM, AutoTokenizer

REPO = "MulinLab/hEVO-base-o"
# Freeze the checkpoint and its custom code for reproducible inference.
REVISION = "fa6139047c21d92b58bd8eda49ea2f89a884e33c"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

load_options = dict(revision=REVISION, token=True, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(REPO, **load_options)
config = AutoConfig.from_pretrained(
    REPO,
    use_frequency_embedding=False,
    use_flash_attention=False,
    **load_options,
)
assert config.use_frequency_embedding is False
model = AutoModelForMaskedLM.from_pretrained(
    REPO,
    config=config,
    attn_implementation="eager",
    dtype=torch.float32,
    **load_options,
).to(device).eval()

def encode_dna(sequence):
    sequence = sequence.strip().upper()
    if not 0 < len(sequence) <= config.max_position_embeddings:
        raise ValueError("Provide a sequence of 1 to 1,024 nucleotides.")
    if set(sequence) - set("ACGTN"):
        raise ValueError("DNA must contain only A, C, G, T or N.")
    encoded = tokenizer(sequence, return_tensors="pt", truncation=False)
    # This custom tokenizer returns 1D tensors for a single sequence.
    encoded = {
        key: value.unsqueeze(0) if value.ndim == 1 else value
        for key, value in encoded.items()
    }
    if "attention_mask" not in encoded:
        encoded["attention_mask"] = torch.ones_like(encoded["input_ids"])
    return {key: value.to(device) for key, value in encoded.items()}

encode_dna validates the input length before tokenization, adds the batch dimension, and supplies an attention mask. Inputs longer than 1,024 nt must be windowed explicitly. Frequency embeddings are explicitly disabled in these examples. The public configuration also defaults to disabled; the explicit override keeps the pinned revision consistent.

Nucleotide embeddings

Run after the shared setup. outputs.hidden_states[-1] is the final transformer layer, before the masked-LM prediction head.

sequence = "ACGTTGCATGTCGCATGATGCATGAGAGCT"
inputs = encode_dna(sequence)

with torch.inference_mode():
    outputs = model(**inputs, output_hidden_states=True, return_dict=True)
    token_embeddings = outputs.hidden_states[-1]  # [batch, length, hidden_size]

    # Optional: summarize the last-layer nucleotide embeddings by mean pooling.
    keep = inputs["attention_mask"].unsqueeze(-1).to(token_embeddings.dtype)
    sequence_embedding = (token_embeddings * keep).sum(dim=1) / keep.sum(
        dim=1
    ).clamp_min(1)

print("Last-layer token embeddings:", tuple(token_embeddings.shape))
print("Mean-pooled sequence embedding:", tuple(sequence_embedding.shape))

For this 30-nt example, the shapes are (1, 30, 768) for nucleotide embeddings and (1, 768) for the mean-pooled vector. Token index i corresponds to nucleotide i; the tokenizer adds no boundary tokens. Pooling averages valid positions only. For position-sensitive tasks, retain the token embeddings.

Single-nucleotide variant scoring

Run after the shared setup. The masked language model returns outputs.logits with shape [batch, length, vocab_size]. For a single-nucleotide variant, mask the target position and use that one shared context to score both alleles:

$\mathrm{LLR}_{\mathrm{ALT}-\mathrm{REF}} = \log P(\mathrm{ALT}\mid c_{\setminus i}) - \log P(\mathrm{REF}\mid c_{\setminus i}) = z_{i,\mathrm{ALT}} - z_{i,\mathrm{REF}}.$

Here i is the masked nucleotide index and z is its vocabulary-logit vector. The shared log-softmax normalization cancels, so the ALT-minus-REF logit difference equals the conditional LLR.

# Synthetic 1,023-bp example; replace with a REF-validated genomic window.
reference = "ACGT" * 255 + "ACG"
variant_index = 511  # Zero-based position within this window.
ref = reference[variant_index]  # T
alt = "A"
assert ref in "ACGT" and alt in "ACGT" and ref != alt

inputs = encode_dna(reference)
ref_id = tokenizer.convert_tokens_to_ids(ref)
alt_id = tokenizer.convert_tokens_to_ids(alt)
assert inputs["input_ids"][0, variant_index].item() == ref_id

# Mask the same site once; REF and ALT share exactly the same flanking context.
inputs["input_ids"][0, variant_index] = tokenizer.mask_token_id
with torch.inference_mode():
    logits = model(**inputs, return_dict=True).logits  # [1, 1023, 10]
    site_logits = logits[0, variant_index].float()    # [vocab_size]
    log_probs = torch.log_softmax(site_logits, dim=-1)
    llr = log_probs[alt_id] - log_probs[ref_id]

    # The shared log-softmax normalizer cancels: these are the same LLR.
    llr_from_logits = site_logits[alt_id] - site_logits[ref_id]
    torch.testing.assert_close(llr, llr_from_logits)

print("Logits:", tuple(logits.shape))
print(f"{ref}>{alt} LLR (ALT minus REF): {llr.item():.6f}")

Interpretation: positive LLR means ALT is more likely than REF in this context; negative LLR means REF is more likely. This is a model preference score, not a calibrated probability of pathogenicity.

For real variants, use the intended reference assembly, confirm that REF matches the genome, and place the SNV at the documented window position. A centered 1,023-bp window uses index 511; a centered 1,024-bp window uses index 512 under the benchmark convention. Set the mask token ID in the encoded tensor, as above; this character tokenizer does not parse the literal string [MASK] as a single nucleotide token. The example scores one SNV, not an indel or a full-sequence autoregressive likelihood.

The examples use PyTorch 2.6.0 and Transformers 4.57.1. The variant example is an API demonstration; its synthetic sequence is not a biological result.

Input and output

Item	Contract
Input	A string of 1–1,024 characters drawn from `A`, `C`, `G`, `T`, `N`
Token positions	One position per nucleotide; no boundary tokens
Hidden states	With `output_hidden_states=True`, the final layer has shape `[batch, length, hidden_size]`
Masked-LM logits	Shape `[batch, length, 10]`; logits at a masked position score the nucleotide vocabulary
Sequence pooling	The example computes a masked mean; there is no separate sequence-classification head in this interface
Frequency inputs	None in the default examples; the frequency embedding branch is disabled

Evaluation and reproducibility

The cards link to the shared benchmark data and task documentation and the code repository. The usage examples pin the checkpoint revision and use PyTorch 2.6.0 with Transformers 4.57.1. This card does not assert benchmark metrics; consult the task guides and the hEvo manuscript linked from the GitHub citation section for evaluation details.

Provenance, access and citation

Local evaluated-model alias: hEvo-open-base.

Its model.safetensors SHA256 matches the hEvo-open-base source weight record in repository table S47.

The -o suffix identifies the public-data training variant; it does not grant an open-source or open-weights license. Access is gated for peer reviewers and manually approved by MulinLab. Sign in, submit an access request on this page, and use your own Hugging Face token after approval.

For scientific use, cite the hEvo manuscript associated with the GitHub repository, the relevant source-data papers, and this model repository at the revision used. The GitHub README does not currently provide a DOI or full bibliographic record, so none is inferred here.

Related resources: hEVO-base-o · hEVO-large-o · hEvo benchmarks · GitHub workflows

Downloads last month: 9

Safetensors

Model size

87.1M params

Tensor type

BF16

MulinLab
/

hEVO-base-o