FunDLM-Medium-1024bp-Histone-Multilabel

LoRA adapter with three independent sigmoid outputs that predict whether a 1,024-bp DNA window carries a ChIP-seq peak of H3K27me3, H3K4me3 or H3K9me3. It was fine-tuned on top of the fungal DNA language model FunDLM-Medium-1024bp and released with the FunDLM paper.

One window can carry more than one mark, so the three outputs are trained with binary cross-entropy instead of a single softmax. H3K4me3 is associated with actively transcribed genes, H3K27me3 marks facultative heterochromatin, and H3K9me3 marks constitutive heterochromatin.

Model summary

Base architecture	FunDLM-Medium-1024bp (Caduceus; bidirectional Mamba-2 with reverse-complement parameter sharing; 64 layers, d_model 768, 471M unique parameters)
Max input length	1,024 bp
Task	Multi-label sequence classification (three independent sigmoid outputs)
Number of outputs	3
Output order	H3K27me3, H3K4me3, H3K9me3
Labels	0 = no peak, 1 = peak for each mark; -1 = unmeasured label (used during training only)
Adapter	LoRA, r = 8, alpha = 32, dropout = 0.1
Training checkpoint	checkpoint-12213 (12,213 optimization steps, 128 windows per step)

Repository contents

Path	Contents
adapter/adapter_config.json, adapter/adapter_model.safetensors	LoRA adapter weights
config.json, configuration_caduceus.py, modeling_caduceus.py, modeling_rcps.py	base FunDLM-Medium-1024bp configuration and custom model code
pytorch_model.bin	base FunDLM-Medium-1024bp weights
tokenizer.json, tokenizer_config.json, special_tokens_map.json	base tokenizer

The base model files are included so the repository is self-contained. The same base model is also available as gnikesh/FunDLM-Medium-1024bp.

Installation

Environment setup follows the same steps as PlantCAD2. This model uses a Mamba-based state-space backbone, so the Mamba kernels are required:

# Core dependencies (versions used for fine-tuning)
pip install transformers==4.49.0 peft==0.14.0 datasets==3.3.2
pip install torch  # choose the build that matches your CUDA version

# Mamba/Caduceus-specific dependencies
pip install mamba_ssm causal_conv1d --no-build-isolation

Running the classifier in Python

The example assumes a CUDA GPU (the base checkpoint is stored in bfloat16). On CPU, cast the model with model.float() and run the forward pass without the autocast block.

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from peft import PeftModel

base_id = "gnikesh/FunDLM-Medium-1024bp"
adapter_id = "PlantG3/FunDLM-Medium-1024bp-Histone-Multilabel"

tokenizer = AutoTokenizer.from_pretrained(base_id, trust_remote_code=True)
base = AutoModelForSequenceClassification.from_pretrained(
    base_id,
    trust_remote_code=True,
    num_labels=3,
    problem_type="multi_label_classification",
)
# Use the base classifier with the adapter applied, without merging it.
model = PeftModel.from_pretrained(base, adapter_id, subfolder="adapter").base_model.model
model = model.cuda().eval()

sequence = "ACGT" * 256  # replace with a 1,024-bp window
encoded = tokenizer(
    sequence,
    add_special_tokens=False,
    padding="max_length",
    truncation=True,
    max_length=1024,
)
input_ids = torch.tensor(encoded["input_ids"]).unsqueeze(0).cuda()
with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
    logits = model(input_ids=input_ids).logits
probs = torch.sigmoid(logits.float())[0]
print(f"H3K27me3: {probs[0]:.4f}  H3K4me3: {probs[1]:.4f}  H3K9me3: {probs[2]:.4f}")

Do not call merge_and_unload() on this adapter. The forward and reverse-complement Mamba branches carry separate LoRA weights, and merging folds both into the shared base weights and changes the predictions.

Batch prediction with the FunDLM code

The fine-tuning and prediction CLI is in the FunDLM repository. Clone this model repository and point --checkpoint_dir at its adapter/ directory:

git clone https://huggingface.co/PlantG3/FunDLM-Medium-1024bp-Histone-Multilabel

python src/finetune/lora_fine_tune.py predict \
    --checkpoint_dir FunDLM-Medium-1024bp-Histone-Multilabel/adapter \
    --data_dir windows.parquet \
    --output_file predictions.csv \
    --model_name gnikesh/FunDLM-Medium-1024bp \
    --task_type multi_label \
    --num_labels 3 \
    --batch_size 32 \
    --bf16 True \
    --seed 42

windows.parquet is the tokenized input produced by the tokenize subcommand of the same script. Inputs should be exactly 1,024 bp; the tokenizer pads shorter sequences and maps non-ACGT characters to [UNK].

Training data

The multi-label set pools the windows cut for the three marks and labels every window for every measured mark by at least 50% peak or control overlap. Windows with an undefined label for a mark measured in their isolate were discarded. O219 has no H3K4me3 data, so its H3K4me3 label was marked unmeasured and left out of the loss. O137 was held out entirely for testing, and the training windows come from Br48, Guy11, O135 and O219. The set contains 521,168 training and 27,430 validation windows and 154,002 O137 test windows. Positive fractions (H3K27me3 / H3K4me3 / H3K9me3) are 0.298 / 0.205 / 0.229 in training and 0.298 / 0.213 / 0.236 in the test set.

Training settings

• LoRA rank 8, alpha 32, dropout 0.1, applied to the Mamba in_proj and out_proj projections
• AdamW, three epochs at 128 windows per step (16 per device x 8 gradient-accumulation steps)
• Learning rate 1e-4 with cosine decay and 300 warm-up steps
• Weight decay 0.01, gradient clipping 1.0, bfloat16, seed 42
• Reported checkpoint: checkpoint-12213

Evaluation

Held-out test set: 154,002 O137 windows with 462,006 measured labels.

Metric	Value
Micro AUROC	0.972
Micro AP	0.942
Exact-match accuracy	0.872
Micro accuracy at probability 0.5	0.954
Micro precision at probability 0.5	0.939
Micro recall at probability 0.5	0.871
Micro F1 at probability 0.5	0.904

Per mark (full test set):

Mark	Prevalence	AUROC	AP
H3K27me3	29.8%	0.985	0.957
H3K4me3	21.3%	0.995	0.983
H3K9me3	23.6%	0.944	0.909

On the windows shared with the three single-mark test sets, this adapter performs as well as the mark-specific adapters: the average-precision difference is within 0.002 for every mark.

Genome-wide O137 scan (43,159 non-overlapping 1,024-bp windows): H3K27me3 AUROC 0.953 and AP 0.727; H3K4me3 AUROC 0.982 and AP 0.918; H3K9me3 AUROC 0.933 and AP 0.471.

Limitations

The multi-label and single-mark adapters were trained and tested on different window sets, because the multi-label set needs windows that can be labeled for every measured mark. Compare them only on matched windows. The O137 test windows were sampled around peaks, so the positive fractions are much higher than in a genome and the genome-wide AP is lower than the test-window AP. A probability threshold of 0.5 overestimates the extent of the marks genome-wide; use the probabilities to rank windows and calibrate a threshold on a related genome with ChIP-seq data. Genome-wide, this adapter ranked H3K4me3 slightly better than the single-mark adapter (AP 0.918 versus 0.910), but ranked H3K27me3 (0.727 versus 0.746) and especially H3K9me3 (0.471 versus 0.538) less well. H3K9me3 is largely predictable from local sequence composition.

Contact

PlantG3 lab, Kansas State University. Please open an issue at PlantG3/funDLM for questions about the model.