FunDLM-Small-1024bp-Mini

LoRA adapter that classifies 1,024-bp DNA windows as mini-chromosomal (1) or core-chromosomal (0). It was fine-tuned on top of the fungal DNA language model FunDLM-Small-1024bp and released with the FunDLM paper.

Mini-chromosomes (also called supernumerary or accessory chromosomes) are small, repeat-rich, gene-poor chromosomes that can carry effector genes in blast fungi. This adapter recognizes mini-chromosomal sequence in isolates that were not used for fine-tuning.

Model summary

Base architecture	FunDLM-Small-1024bp (Caduceus; bidirectional Mamba-2 with reverse-complement parameter sharing; 24 layers, d_model 768, 88M unique parameters)
Max input length	1,024 bp
Task	Sequence classification (softmax over two classes)
Number of classes	2
Labels	0 = core chromosome, 1 = mini-chromosome
Adapter	LoRA, r = 8, alpha = 32, dropout = 0.1
Training checkpoint	checkpoint-27000 (27,000 steps, 512 windows per step)

Repository contents

Path	Contents
adapter/adapter_config.json, adapter/adapter_model.safetensors	LoRA adapter weights
config.json, configuration_caduceus.py, modeling_caduceus.py, modeling_rcps.py	base FunDLM-Small-1024bp configuration and custom model code
pytorch_model.bin	base FunDLM-Small-1024bp weights
tokenizer.json, tokenizer_config.json, special_tokens_map.json	base tokenizer

The base model files are included so the repository is self-contained. The same base model is also available as gnikesh/FunDLM-Small-1024bp.

Installation

Environment setup follows the same steps as PlantCAD2. This model uses a Mamba-based state-space backbone, so the Mamba kernels are required:

# Core dependencies (versions used for fine-tuning)
pip install transformers==4.49.0 peft==0.14.0 datasets==3.3.2
pip install torch  # choose the build that matches your CUDA version

# Mamba/Caduceus-specific dependencies
pip install mamba_ssm causal_conv1d --no-build-isolation

Running the classifier in Python

The example assumes a CUDA GPU (the base checkpoint is stored in bfloat16). On CPU, cast the model with model.float() and run the forward pass without the autocast block.

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from peft import PeftModel

base_id = "gnikesh/FunDLM-Small-1024bp"
adapter_id = "PlantG3/FunDLM-Small-1024bp-Mini"

tokenizer = AutoTokenizer.from_pretrained(base_id, trust_remote_code=True)
base = AutoModelForSequenceClassification.from_pretrained(
    base_id,
    trust_remote_code=True,
    num_labels=2,
    id2label={0: "core", 1: "mini"},
    label2id={"core": 0, "mini": 1},
)
# Use the base classifier with the adapter applied, without merging it.
model = PeftModel.from_pretrained(base, adapter_id, subfolder="adapter").base_model.model
model = model.cuda().eval()

sequence = "ACGT" * 256  # replace with a 1,024-bp window
encoded = tokenizer(
    sequence,
    add_special_tokens=False,
    padding="max_length",
    truncation=True,
    max_length=1024,
)
input_ids = torch.tensor(encoded["input_ids"]).unsqueeze(0).cuda()
with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
    logits = model(input_ids=input_ids).logits
prob_mini = torch.softmax(logits.float(), dim=-1)[0, 1].item()
print(f"mini-chromosome probability: {prob_mini:.4f}")

Do not call merge_and_unload() on this adapter. The forward and reverse-complement Mamba branches carry separate LoRA weights, and merging folds both into the shared base weights and changes the predictions.

Batch prediction with the FunDLM code

The fine-tuning and prediction CLI is in the FunDLM repository. Clone this model repository and point --checkpoint_dir at its adapter/ directory:

git clone https://huggingface.co/PlantG3/FunDLM-Small-1024bp-Mini

python src/finetune/lora_fine_tune.py predict \
    --checkpoint_dir FunDLM-Small-1024bp-Mini/adapter \
    --data_dir windows.parquet \
    --output_file predictions.csv \
    --model_name gnikesh/FunDLM-Small-1024bp \
    --task_type classification \
    --batch_size 32 \
    --bf16 True \
    --seed 42

windows.parquet is the tokenized input produced by the tokenize subcommand of the same script. Inputs should be exactly 1,024 bp; the tokenizer pads shorter sequences and maps non-ACGT characters to [UNK].

Training data

The adapter was trained on 1,024-bp windows from the six Pyricularia assemblies used to train MiniC: 70-15, B71Ref2, MZ5-1-6, LpKY97, O135 and TF05. Core and mini-chromosomal sequences were tiled with different step sizes to balance the classes, and windows with a confident alignment to sequence shared between the two compartments were removed. This gave 13,655,061 training and 718,688 validation windows.

Training settings

• LoRA rank 8, alpha 32, dropout 0.1, applied to the Mamba in_proj and out_proj projections
• AdamW, learning rate 1e-4 with cosine decay, 2,700 warm-up steps
• Weight decay 0.01, gradient clipping 1.0, bfloat16, seed 42
• 16 windows per device x 32 gradient-accumulation steps = 512 windows per optimization step
• 27,000 optimization steps (one pass over the training windows)
• Reported checkpoint: checkpoint-27000

Evaluation

Held-out test set: 162,080 non-overlapping 1,024-bp windows from four isolates that were not used for fine-tuning (Br62, G239, O137 and JP37), of which 4,551 (2.8%) are mini-chromosomal.

Metric	Value
AUROC	0.986
Average precision (AP)	0.884
Precision at probability 0.5	0.886
Recall at probability 0.5	0.828
F1 at probability 0.5	0.856

Average precision by isolate: Br62 0.883, G239 0.996, O137 0.799, JP37 0.557. This is the adapter behind the paper's mini-chromosome results for the Small model; it performs slightly better than the Medium-1024bp adapter (AP 0.866).

Limitations

The held-out-isolate test largely measures recognition of sequence related to the training isolates: 76% of the test windows are at least 99.9% identical to training sequence over their best alignment. On windows without a close homolog in the training genomes, performance drops for this adapter and for the baselines tested in the paper. The adapter is intended for fungal mini-chromosome discovery and should not be interpreted as a general repeat or accessory-chromosome detector without validation on the target lineage.

Contact

PlantG3 lab, Kansas State University. Please open an issue at PlantG3/funDLM for questions about the model.