EvoLenTokenizer/evolen-100k-sft-checkpoints

EvoLen-100K — final SFT checkpoints

Fine-tuned classification checkpoints for all 56 downstream tasks, produced from the EvoLen pretrained model at pretraining step 100,000 (vocab 5,120).

These are the checkpoints behind the EvoLen row of Table 1 in EvoLen: Evolution-Guided Tokenization for DNA Language Model (arXiv:2604.08698).

Base model: EvoLenTokenizer/evolen-100k. Its tokenizer-matched control is EvoLenTokenizer/base-100k.

Verification against the paper

Every one of the 56 checkpoints was matched to its reported score, and the group averages reproduce Table 1 exactly (MCC ×100):

group	n	this repo	Table 1		group	n	this repo	Table 1
GUE EMP	10	46.81	46.8		NT His	10	54.95	55.0
GUE Mou	5	58.75	58.8		NT Enh	2	49.62	49.6
GUE P3	3	78.52	78.5		NT Pro	3	76.50	76.5
GUE PC	3	63.61	63.6		NT Spl	3	67.81	67.8
GUE Spl	1	75.22	75.2		cCRE	1	22.13	22.1
GUE TF	5	61.05	61.0		ATAC	1	13.30	13.3
GBM HR	5	70.29	70.3		GBM ME	1	63.84	63.8
GBM Inv	3	76.14	76.1

Layout

One folder per task, <SUITE>/<task>/:

GBM/demo_coding_vs_intergenomic_seqs/
GUE/EMP/H3K4me1/
NT/H3K27ac/
multi-SCREEN/allchr_csv/                 # the "cCRE" column of Table 1
multi-ATAC/human_mouse_superclass_allchr/ # the "ATAC" column of Table 1

Each folder holds model.safetensors, config.json, tokenizer.json, tokenizer_config.json, special_tokens_map.json, trainer_state.json and training_args.bin. Optimizer and scheduler state are stripped.

training_args.bin carries the complete TrainingArguments, so every hyperparameter — including model_max_length, which is per-task and not derivable from the task name — is recoverable from the checkpoint:

import torch, sys
class TrainingArguments: pass          # the pickle references train.py's subclass
sys.modules["__main__"].TrainingArguments = TrainingArguments
args = torch.load("training_args.bin", map_location="cpu", weights_only=False).__dict__

Loading

from transformers import AutoModelForSequenceClassification

REPO = "EvoLenTokenizer/evolen-100k-sft-checkpoints"
model = AutoModelForSequenceClassification.from_pretrained(
    REPO, subfolder="GUE/EMP/H3K4me1", trust_remote_code=True)

Reproducing the reported numbers

Evaluation only — no retraining. Four things must match or you will get close-but-wrong values:

1. fp16 autocast. Training used --fp16; evaluating in fp32 shifts MCC by ~3e-4.
2. Eval batch size 128. fp16 reduction order depends on batch shape.
3. model_max_length per task, read from training_args.bin.
4. Multiclass label ordering — string labels map via sorted(set(...)) over the split file. For multi-SCREEN/allchr_csv that is ['CA','CA-CTCF','CA-H3K4me3','CA-TF','PLS','TF','dELS','pELS'] → 0..7.

Tokenize with padding="longest", truncation=True, max_length=model_max_length; attention mask is input_ids.ne(pad_token_id). Metric is sklearn.metrics.matthews_corrcoef over argmax predictions.

Verified environment: Python 3.9.18, transformers 4.35.2, scikit-learn 1.6.1, torch 2.8.0, numpy 2.0.2, tokenizers 0.15.2, accelerate 0.25.0.

How these checkpoints were selected

Each run trained for num_train_epochs with per-epoch validation on dev.csv. load_best_model_at_end=True with metric_for_best_model=eval_f1 reloaded the checkpoint with the best validation F1, which was then evaluated once on test.csv; its MCC is the reported number. The selected epoch is recorded in trainer_state.json and is often earlier than the training budget.

Per-task hyperparameters were chosen as the maximum test MCC over the swept configurations. This is test-set model selection, so the reported values are an upper bound rather than an unbiased estimate.

Retraining from the base model will not reproduce these numbers exactly: fp16 training is nondeterministic and epoch-selection margins are small, so a rerun lands on a different epoch and a slightly different score. Evaluate these checkpoints instead.

Related

• EvoLenTokenizer/evolen-200k-sft-checkpoints — the same 56 tasks at pretraining step 200,000
• EvoLenTokenizer/evolen-100k / base-100k — pretrained models
• EvoLenTokenizer/evolen-200k / base-200k