qwen3.8-27b-gsm8k-lora

LoRA adapter for grade-school math word problems.

Fine-tuned from Qwen/Qwen3.8-27B on openai/gsm8k with QLORA.

Results

Model	GSM8K exact match
Qwen3.8-27B (base)	83.6% (209/250)
this model	93.6% (234/250)

Training setup

Method	QLORA
LoRA rank / alpha	32 / 64
Adapted modules	attention q,k,v,o on the full-attention layers; FFN gate,up,down on all 64 layers
Gated DeltaNet projections	left frozen
Vision tower and MTP head	left frozen, carried over from the base checkpoint
Precision	bf16 (fp16 produces NaN gradients in the Gated DeltaNet blocks)
Max sequence length	1024
Hardware	1 node, 8 GPUs, Slurm partition batch-xdr

Training data is GSM8K rationales as visible (non-thinking) content, so evaluate thinking-mode behaviour separately if you depend on it.

Usage

from transformers import AutoModelForImageTextToText, AutoTokenizer
from peft import PeftModel
import torch

base = AutoModelForImageTextToText.from_pretrained(
    "Qwen/Qwen3.8-27B", dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(base, "Balab2021/qwen3.8-27b-gsm8k-lora")
tokenizer = AutoTokenizer.from_pretrained("Balab2021/qwen3.8-27b-gsm8k-lora")

Reproducing

Scripts: cluster-activities/qwen38-gsm8k-sft in the source repo. Requires transformers>=5.2.0 and flash-linear-attention for the qwen3_5 architecture.