HMk100 — Handwritten Math Expression LoRA (Qwen2.5-VL-7B)

LoRA adapter that fine-tunes Qwen/Qwen2.5-VL-7B-Instruct to transcribe handwritten mathematical expressions. Trained on the HME100K dataset only (experiment e2 of THIRA — Thai Handwriting Intelligence Recognition & Analysis).

Model Details

• Base model: Qwen/Qwen2.5-VL-7B-Instruct, revision cc594898137f460bfe9f0759e9844b3ce807cfb5
• Adapter: LoRA, r=16, alpha=32, dropout=0.05
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Task: read a handwritten mathematical expression from an image and transcribe it

Dataset

Trained on HME100K, a large-scale handwritten mathematical expression recognition dataset, via the Kaggle mirror used by the training pipeline (kagglehub.dataset_download("prajwalchy/hme100k-dataset")):

• Source: https://www.kaggle.com/datasets/prajwalchy/hme100k-dataset
• Check the Kaggle listing for the dataset's current license/usage terms before redistributing data or derived artifacts.

This repository is kept private for now.

Evaluation

From the project's internal eval harness (runs/e2, see results.json in this repo for full detail).

Test set	N	CER	char F1	Exact match
hme (in-domain)	24,607	0.0425	0.9766	65.53%
burapha (out-of-domain Thai)	13,600	3.1985	0.0005	0.10%
thai_sentence (out-of-domain prose)	2,032	1.3775	0.2088	0.05%
combined	38,207	0.0800	0.9525	42.24%

Findings

1. Catastrophic forgetting, and it is severe. A CER of 3.1985 on BURAPHA is not "poor accuracy" — a CER above 1.0 means the model emits more erroneous characters than the reference contains. Character F1 of 0.0005 says the output shares essentially no characters with the target. Training on HME100K alone did not merely fail to teach Thai; it destroyed the base model's ability to produce it. The same collapse appears on Thai sentences (CER 1.3775).

This is the single most important result from the e2 run, and it is why the project never considered a math-only adapter for a Thai worksheet product.

2. e3 dominates it outright. The jointly-trained adapter matches e2 on its own home turf while keeping Thai intact:

	hme CER	burapha CER	thai_sentence CER
e2 (this model)	0.0425	3.1985	1.3775
e3 (joint)	0.0421	0.0400	0.6753

e3 is better on maths (0.0421 vs 0.0425) and ~80× better on Thai. There is no measured task on which e2 is the right choice over e3. e2 is superseded.

3. Merging cannot repair it. Weight-space merges of e2 with the Thai adapter e1 were tried at three ratios; every one gives up math accuracy without reaching e3's Thai:

merge (e1–e2 weight)	hme CER	burapha CER
0.3 – 0.7	0.0483	0.5144
0.5 – 0.5	0.0592	0.1215
0.7 – 0.3	0.1134	0.0480
e3 (joint training)	0.0421	0.0400

A three-way merge (e7) is worse still on maths (0.369–0.443). Retrain jointly; do not merge.

4. The decoder emits space-separated tokens, which inflates its CER against normally spaced references. e2 and e3 write maths as \frac { \log _ { 2 } 6 } — one space between every token. Any character-level comparison against a normally spaced reference charges each of those spaces as an edit. In this project that artifact was, for a time, read as evidence that the math adapters were worse than they are: on real worksheets a whitespace-keeping CER ranked a prose adapter first, while the same data with whitespace removed ranked the math-routed configuration first. Report symbol-level (whitespace- stripped) CER alongside raw CER whenever comparing this adapter to a non-space-separated decoder.

5. Benchmark strength does not transfer to real worksheets. On real Thai student homework, symbol-level CER on mathematical regions was 0.451 (n=20) for the math-routed configuration, against 0.042 here. HME100K is a much easier distribution than photographed student work. Treat 0.0425 as an in-domain figure only.

6. Not selected for the production pipeline — see e3, which is the deployed mathematics recogniser in the Learnly MVP's routed configuration. e2 is retained as the single-dataset baseline that establishes the HME100K ceiling and documents the forgetting result.

How to Use

import torch
from peft import PeftModel
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor

base = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2.5-VL-7B-Instruct",
    revision="cc594898137f460bfe9f0759e9844b3ce807cfb5",
    torch_dtype=torch.bfloat16,
)
model = PeftModel.from_pretrained(base, "TLDRKKU/HMk100")
processor = AutoProcessor.from_pretrained("TLDRKKU/HMk100")

Training Procedure

• 1 epoch, cosine LR schedule, learning rate 1e-4, warmup ratio 0.03
• Effective batch size 32 (per-device 8 × grad accumulation 4)
• bf16, gradient checkpointing enabled
• Framework: PEFT 0.20.0

Related

• TLDRKKU/burapha — BURAPHA-only adapter (e1)
• TLDRKKU/burapha-HMk100 — joint adapter (e3), the one to prefer

---

Part of THIRA — Thai Handwriting Intelligence Recognition & Analysis.