EvoLenTokenizer/evolen-200k-sft-checkpoints

EvoLen-200K — final SFT checkpoints

Fine-tuned classification checkpoints for all 56 downstream tasks, produced from the EvoLen pretrained model at pretraining step 200,000.

Base model: mtapiapacheco/len2_5120 (merge-len2 tokenizer, vocab 5120).

Layout

One folder per task, <SUITE>/<task>/:

GBM/demo_coding_vs_intergenomic_seqs/
GUE/EMP/H3K4me1/
NT/H3K27ac/
multi-ATAC/human_mouse_superclass_allchr/
multi-SCREEN/allchr_csv/

Each contains model.safetensors, config.json, tokenizer.json, tokenizer_config.json, special_tokens_map.json, trainer_state.json and training_args.bin. Optimizer and scheduler state are stripped.

training_args.bin holds the complete TrainingArguments for that run, so every hyperparameter is recoverable from the checkpoint itself:

import torch, sys
class TrainingArguments: pass          # the pickle references train.py's subclass
sys.modules["__main__"].TrainingArguments = TrainingArguments
args = torch.load("training_args.bin", map_location="cpu", weights_only=False).__dict__

Loading

from transformers import AutoModelForSequenceClassification, PreTrainedTokenizerFast

REPO = "EvoLenTokenizer/evolen-200k-sft-checkpoints"
model = AutoModelForSequenceClassification.from_pretrained(
    REPO, subfolder="GUE/EMP/H3K4me1", trust_remote_code=True)

Reproducing the reported numbers

Evaluation only — no retraining. Four things must match or you will get close-but-wrong values:

1. fp16 autocast. Training used --fp16; evaluating in fp32 shifts MCC by ~3e-4.
2. Eval batch size 128. fp16 reduction order depends on batch shape.
3. model_max_length per task — ranges from 20 to 512 and is not derivable from the task name. Read it from training_args.bin.
4. Multiclass label ordering — string labels map via sorted(set(...)) over the split file. For multi-SCREEN/allchr_csv that is ['CA','CA-CTCF','CA-H3K4me3','CA-TF','PLS','TF','dELS','pELS'] → 0..7.

Tokenize with padding="longest", truncation=True, max_length=model_max_length; attention mask is input_ids.ne(pad_token_id). Metric is sklearn.metrics.matthews_corrcoef over argmax predictions.

Verified environment: Python 3.9.18, transformers 4.35.2, scikit-learn 1.6.1, torch 2.8.0, numpy 2.0.2, tokenizers 0.15.2, accelerate 0.25.0.

How these checkpoints were selected

Each run trained for num_train_epochs with per-epoch validation on dev.csv. load_best_model_at_end=True with metric_for_best_model=eval_f1 reloaded the checkpoint with the best validation F1; that checkpoint was then evaluated once on test.csv, and its MCC is the reported number.

Two consequences worth knowing:

• The selected epoch is usually earlier than num_train_epochs — 40 of 56 runs stopped early, in one case at epoch 1 of 10. The real value is the epoch field in trainer_state.json, not the training budget.
• Checkpoints were selected on F1 while MCC is reported. Selecting on MCC instead is worth roughly +0.2 MCC on average across the suite.

Per-task hyperparameters were chosen as the maximum test MCC over the swept configurations (18 to 119 per task). This is test-set model selection, so the reported values are an upper bound rather than an unbiased estimate.

Retraining from the base model will not reproduce these numbers exactly. fp16 training is nondeterministic and the epoch-selection margins are ~0.002 in validation F1, so a rerun lands on a different epoch and a slightly different score. Evaluate these checkpoints to reproduce the reported values.