NeurIPS 2025

Uni-MuMER: Unified Multi-Task Fine-Tuning of
Vision-Language Model for Handwritten
Mathematical Expression Recognition
Yu Li∗
Jin Jiang∗
Jianhua Zhu
Shuai Peng
Baole Wei
Yuxuan Zhou
Liangcai GaoB
Wangxuan Institute of Computer Technology, Peking University, Beijing, China
liyu@stu.pku.edu.cn
jiangjin@stu.pku.edu.cn
zhujianhuapku@pku.edu.cn
glc@pku.edu.cn
Abstract
Handwritten Mathematical Expression Recognition (HMER) remains a persistent
challenge in Optical Character Recognition (OCR) due to the inherent freedom
of symbol layouts and variability in handwriting styles. Prior methods have faced
performance bottlenecks by proposing isolated architectural modifications, making
them difficult to integrate coherently into a unified framework. Meanwhile, recent
advances in pretrained vision-language models (VLMs) have demonstrated strong
cross-task generalization, offering a promising foundation for developing unified
solutions. In this paper, we introduce Uni-MuMER, which fully fine-tunes a VLM
for the HMER task without modifying its architecture, effectively injecting domain-
specific knowledge into a generalist framework. Our method integrates three data-
driven tasks: Tree-Aware Chain-of-Thought (Tree-CoT) for structured spatial rea-
soning, Error-Driven Learning (EDL) for reducing confusion among visually simi-
lar characters, and Symbol Counting (SC) for improving recognition consistency
in long expressions. Experiments on the CROHME and HME100K datasets show
that Uni-MuMER achieves super state-of-the-art performance, outperforming the
best lightweight specialized model SSAN by 16.31% and the top-performing VLM
Gemini2.5-flash by 24.42% under zero-shot setting. Our datasets, models, and
code are open-sourced at: https://github.com/BFlameSwift/Uni-MuMER
1
Introduction
38.64
37.75
37.45
62.58
62.51
65.30 
72.62
64.26
69.14 
82.05
77.94
79.23
30
40
50
60
70
80
90
100
CROHME 2014
CROHME 2016
CROHME 2019
ExpRate(%)
SSAN(2025)
Qwen2.5VL Zero-Shot
Qwen2.5VL SFT(Vanilla)
Un...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000064. Conference bucket: NeurIPS.