Baseline BPE-200K — SFT checkpoints (partial: 37 of 56 tasks)
Fine-tuned classification checkpoints from the baseline BPE pretrained model at
step 200,000 (vocab 5,120), for use as the control against
EvoLenTokenizer/evolen-200k-sft-checkpoints.
Base model: EvoLenTokenizer/base-200k.
Tokenizer: standard BPE, vocab 5,120 — not the EvoLen tokenizer.
⚠️ This repo is incomplete by design (37 of 56 tasks)
Published checkpoints come in two tiers:
Exact match (28 tasks). Identical learning rate, weight decay, warmup ratio, epochs,
seed, batch size, model_max_length, precision and checkpoint-selection rule to the
corresponding EvoLen-200k run. The only difference is the tokenizer and pretrained model.
Same hyperparameters, batch 64 (9 tasks — all of GBM). Identical learning rate,
weight decay, warmup ratio, epochs and seed, but trained at per_device_batch_size=64
where EvoLen used 128. Everything else matches. Marked published_batch64 in
coverage.csv.
demo_coding_vs_intergenomic_seqs, demo_human_or_worm, drosophila_enhancers_stark,
dummy_mouse_enhancers_ensembl, human_enhancers_cohn, human_enhancers_ensembl,
human_ensembl_regulatory, human_nontata_promoters, human_ocr_ensembl
The remaining 19 tasks are not published:
Different hyperparameters (5 tasks) — the EvoLen winners came from a later extension sweep outside the base grid, so no matching baseline run exists.
| task | baseline | EvoLen |
|---|---|---|
EMP/H3K4me2 |
lr3e-5_wd0.05_wr0.03_ep10 |
lr3e-5_wd0.05_wr0.03_ep20 |
EMP/H3K14ac |
lr3e-5_wd0.05_wr0.06_ep8 |
lr3e-5_wd0.05_wr0.07_ep9 |
EMP/H3K4me3 |
lr3e-5_wd0.05_wr0.06_ep10 |
lr1e-5_wd0.05_wr0.03_ep14 |
EMP/H3K36me3 |
lr3e-5_wd0.05_wr0.10_ep10 |
lr3e-5_wd0.05_wr0.10_ep16 |
allchr_csv |
lr3e-5_wd0.0_wr0.01_ep3, maxlen 200 |
lr2.5e-5_wd0.01_wr0.01_ep3, maxlen 512 |
No baseline run at all (14 tasks) — never executed at 200k:
EMP/H3, EMP/H4, EMP/H4ac, mouse/0–mouse/4, splice/reconstructed, tf/0–tf/4
Note that the missing set covers all five mouse/* and all five tf/* tasks, so no
200k baseline comparison is currently possible on the mouse or TF-binding groups.
A second asymmetry to be aware of
Even for the 28 matched tasks, the selection procedure differs. EvoLen-200k reports the maximum test MCC over a sweep of 18–119 configurations per task; the baseline is a single run at one configuration. Comparing a max-over-many against a single draw favours EvoLen independently of tokenizer quality. Treat the deltas accordingly.
Layout
One folder per task, <SUITE>/<task>/, each with model.safetensors, config.json,
tokenizer.json, tokenizer_config.json, special_tokens_map.json,
trainer_state.json and training_args.bin. Optimizer and scheduler state are stripped.
from transformers import AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained(
"EvoLenTokenizer/base-200k-sft-checkpoints",
subfolder="NT/H3K27ac", trust_remote_code=True)
coverage.csv lists all 56 tasks with their status (published / batch-size mismatch /
hyperparameter mismatch / no run) and both configs side by side.
Reproducing
Evaluation only. fp16 autocast, eval batch size from training_args.bin, per-task
model_max_length from training_args.bin, and sorted(set(...)) label ordering for
multiclass tasks. Tokenize with padding="longest", truncation=True; attention mask is
input_ids.ne(pad_token_id); metric is sklearn.metrics.matthews_corrcoef over argmax
predictions.
Environment: Python 3.9.18, transformers 4.35.2, scikit-learn 1.6.1, torch 2.8.0, numpy 2.0.2, tokenizers 0.15.2, accelerate 0.25.0.