Uni-MuMER: Unified Multi-Task Fine-Tuning of Vision-Language Model for Handwritten Mathematical Expression Recognition
Authors: Yu Li
Jin Jiang
Jianhua Zhu
Shuai Peng
Baole Wei
Yuxuan Zhou
Liangcai Gao
Venue: NeurIPS 2025 spotlight ()
Keywords: Handwritten Mathematical Expression Recognition
Vision-Language Models
Tree-of-Thought Reasoning
Multi-Task Learning
Optical Character Recognition
Primary area: deep_learning
TL;DR: We propose Uni-MuMER, full fine-tuning a VLM for handwritten math expression recognition via structured spatial reasoning (Tree-CoT), error-driven learning (EDL), and symbol counting (SC), achieving state-of-the-art results.

Abstract: Handwritten Mathematical Expression Recognition (HMER) remains a persistent challenge in Optical Character Recognition (OCR) due to the inherent freedom of symbol layouts and variability in handwriting styles.
Prior methods have faced performance bottlenecks by proposing isolated architectural modifications, making them difficult to integrate coherently into a unified framework.
Meanwhile, recent advances in pretrained vision-language models (VLMs) have demonstrated strong cross-task generalization, offering a promising foundation for developing unified solutions. 
In this paper, we introduce Uni-MuMER, which fully fine-tunes a VLM for the HMER task without modifying its architecture, effectively injecting domain-specific knowledge into a generalist framework. 
Our method integrates three data-driven tasks: Tree-Aware Chain-of-Thought (Tree-CoT) for structured spatial reasoning, Error-Driven Learning (EDL) for reducing confusion among visually similar
characters, and Symbol Counting (SC) for improving recognition consistency in long expressions.
Experiments on the CROHME and HME100K datasets show that Uni-MuMER achieves super state-of-the-art performance, 
outperforming the best lightweight specialized model SSAN by 16.31\% and the top-performing VLM Gemini2.5-flash by 24.42\% under zero-shot setting. 
Our datasets, models, and code are open-sourced at: https://github.com/BFlameSwift/Uni-MuMER

Uni-MuMER: Unified Multi-Task Fine-Tuning of Vision-Language Model for Handwritten Mathematical Expression Recognition
Yu Li" Jin Jiang" Jianhua Zhu Shuai Peng Baole Wei Yuxuan Zhou
Liangcai Gao
Wangxuan Institute of Computer Technology, Peking University, Beijing, China
liyudstu. pku.edu edu.
jiangj inčatu. .uu. pku.edu. zhujtanhuapkuuhpuuccmm edw. cn
glc0pku. glcapku.edu.co eecikuueddu.cm edu.cn
Abstract
Handwritten Mathematical Expression Recognition (HMER) remains a persistent challenge in Optical Character Recognition (OCR) due to the inherent freedom of symbol layouts and variability in handwriting styles. Prior methods have faced performance bottlenecks by proposing isolated architectural modifications, making them difficult to integrate coherently into a unified framework. Meanwhile, recent advances pretrained vision-language models (VLMs) have demonstrated strong cross-task generalization, offering promising foundation for developing unified solutions. this paper, we introduce Uni-MuMER, which fully fine-tunes a VLM for the HMER task without modifying its architecture, effectively injecting domain- specific knowledge into a generalist framework. Our method integrates three data- driven tasks: Tree-Aware Chain-of-Thougee soning. Error-Driven Learning (EDL) for reducing confusion among visually simi- lar characters, and Symbol Counting (SC) for improving recognition consistency in long expressions. Experiments on the CROHME and HME1001 datasets show that Uni-MuMER achieves super state-of-the art performance, outperforming the best lightweight specialized model SSAN by 16.31 % and the top-performing VLM Gemini2.5-fiar by 24.42% under pero-shot setting. Our datasets, models, and code are open-sourced at: https: // thub. com/ BBFl ameewameweee t/ -uMER
(Tree-CoT) for structured spatial rea-
1 Introduction
Handwritten Mathematical Expression Recog- nition (HMER) seeks to translate handwrit- markup, supporting document understanding and dig-
SSANDOLS) Qwen29V) Zem-Shrn (Qwn0t.9) SPI/Vanlll Uhi SuMER!
90
ten expressions into
80
82,06
77.91
79.23
ExpRate(%
72.50
70
65.30
ital preservation of scientific material. Un- like standard Optical Character Recognition (OCR), HMER involves parsing complex 2D structures [15], ambiguous symbols, and the inherent freedom of symbol layout and variabil- ity handwriting styles [68], requiring not just recognizing individual symbols but also layout reasoning and parsing their complex spatial lationships. Recent approaches have primarily leveraged RNNs [23] Transformers [48] due
62.58
62.51
60
50
40
38.6
$7.75
37.42
30
CROHME 2014
CROHME 2016
CROHME 2019
Figure 1: Performance comparison CROHME
sets ( (%). Our Uni-MuMER achieves significant improvements compared to the previous SOTA (SSAN) and Qwen2.5VL Zero-Shot.
Equal contribution, "CCoreeponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

to their powerful sequential modeling capabilities. Additional modules, such as tree-structured decoders [64, 52, 70] and siriiie--osition--ve aware mechanisms [18, 62, 53, 54], have been introduced to further enhance performance. These dominant research directions in HMER have focused on in- corporating prior human knowledge into model architectures, particularly through manually designed structural modules or attention mechanisms [63, 65, 66, 67,30, 18, 4].
Despite the importance of HMER and numerous proposed improvements, the field has seen only marginal progress in recent years. As is shown in Tab. 1, performance on the CROHME datasets [38, 39, 37] has improved by merely 3% (from CoMER [66] to SSAN [62]), underscoring an urgent need for a novel paradigm. The limited progress of these models arose from three constraints: (1)
Improvements are isolated and model sppecifcc making them hard integrate or scale. (2) Optimizing across multiple auxiliary tasks remains challenging, with many approaches focusing on singular priors rather than adopting a unified, multi-faceted enhancement [7]; and (3) Models trained on single-domain datasets, struggling to generalize to other datasets, lack essential scalability and transferability [68, 46). Additionally, widely standard metrics like ExpRate fail to capture visual equivalence in L outputs, and the scarcity of diverse data exacerbates these constraints. Recently introduced visual metrics such as CDM and large-scale datasets like Mathwriting offer opportunities for large-scale cross-dataset training and multiple LTTH-X syntax style evaluation.
Rapid advancements in pretrained vision-language models (VLMs) have significantly enhanced foundational recognition capabilities and generalization across related tasks [8, 25, 28, 31, 32). While early benchmarks like OCRBench [34] reported poor HMER performance for generalist VLMs, Tab. presents our evaluation of recent open-source and closed-source VI.Ms, such as Qwen2.5- VL. [3], Dou.ao-o..-1ppp 1.5-pro [45], (emini2.55-flash [11], and GPT4o [41]. Our results indicate that these large-scale models have exhibited unexpectedly strong capabilities in handling structured recognition tasks. However, these high-performance closed-source models trained on massive amounts of undisclosed data [1] make it difficult to pinpoint how to systematically improve performance on
HMER. Consequently, empowering open-source VLMs to achieve comparable or superior HMER performance remains an open and urgent challenge.
To bridge this gap, we propose Uni-MuMER, a unified multi-task fine-tuning framework for enhancing open-source VLMs in HMER. Unlike previous methods constrained by single-domain datasets or isolated architectural improvements, Uni-MuMER fully exploits existing data resources through data-driven fine-tuning. Another motivation of our work is to unify previously fragmented HMER tasks into a unified framework, shifting the focus from incremental architectural modifications toward generalizable recognition capabilities. Concretely, we introduce specialized training data and learning objectives, employing novel tasks such as Tree-Aware Srain-of-Thhugle for explicit structural reasoning [51, 57], Error-driven learning to reduce symbol confusion [36], and Symbol Counting [30] to enhance parsing expressions capabilities. Fig. nicely illustrates the jump with Uni-MuMER: it exceeds Qwen-2.5-VL in the zero-shot setting and the vanilla SFT setting across multiple CROHME datasets. Besides, owing to advancements in open-source inference frameworks (c.g., vllm [29]), our method achieves superior inference speeds compared to traditional methods, enhancing its practical applicability. Our contributions are listed below:
We propose new unified paradigm for HMER. Unlike prior methods that heavily rely on special- ized networks and single-task training, our data-driven multi-tast method injects domain knowledge into generalist VLM, yielding cumulative performance gains.
We introduce three data-driven tasks: Tree-Chain-ol Thoughh,, Error-Driven Learning, and Symbol Counting. These systematically address the challenges of HMER, which are two-dimensional structure, ambiguous handwriting, and output consistency.
Our Uni-MuMER method achieves new SOTA results on the CROHME and HME100K datasets. Notably, it attains 79.74% averaged across CROHME datasets (+41.79% over VW +24.42% over Gemini2.5-fias in zero-shot setting, and +16.31% % over specialized models SSAN).
Related Work
2.1 HMER
Early rule-based approaches attempted to address the challenges of HMER through symbol segmenta-
tion, recognition, and syntactic parsing based on handcrafted grammars [6, 60, 10, 2, 22, 47]. These methods struggled with generalization to diverse handwriting styles and complex layout structures.

---

Sequence-based decoding methods The emergence of deep learning shifted HMER towards end- to-end sequence modeling tasks. Early encoder-decodes architectures based on RNNs, WAP [65] first applied seaeeneett-sequennee learning to HMER. Subsequent RNN-based models improved visual encoding [63] and robustness to handwriting style [53, 54). Inspired by the Transformer's success [48], BTTR [67] introduced the first Transformer-basee model for HMER. To enhance alignment during decoding, CoMER [66] introduced a coverage attention mechanism.
Multi-task Learning Beyond sequence modeling, researchers have explored structural decoders and multi-task models to better capture 2D structure. One central line of work focuses on modeling the hierarchical tree structure of expressions. TreeDecoder (TD) [64] and its improved version TDv2 directly predict a structuuu representation of the expression. SAN [58] introduced syntactic constraints into the decoding process, and TAMER [70] jointly optimizes both sequence and tree decoding within a unified Transformer framework. Various auxiliary tasks have also been proposed to inject domain knowledge: ABM [4] adds dual-direction decoder loss to enhance context modeling. RLFN [9] fuses
gegguage-model-assse module with the recognizer to leverage 18TEX syntax and context. ICAL [69] proposed implicit character construction to capture latent symbol-level semantics. To address symbol omission and repetition, CAN [30] incorporated an auxiliary symbol- counting task. UniMERNet [49] adds Length-Aware Module targeting real-world expressions' extreme length variance. PosFormer [18] and SSAN [62] explicitly model spatial relationships among symbols to guide the network's 's attention.
2.2 Vision-Language Models
Vision-Language Models (VLMs), initially popularized by contrastive learning frameworks like CLIP [43], laid the groundwork for powerful zero-shot multimodal recognition. Donut [27] and LayouLMMv3 [24] extended this capability specifically to OCR tasks, effectively recognizing diverse textual content. High-resolution,
decument-centris VLMs, notably Monkey [33] and TextMon- key [35], enhanced OCR performance by combining patch-based image encoding with explicit textual supervision. For mathematical OCR, models like Im2LaTeX [12] and Nougat [5] focused explicitly on end-to-end LaTTeX reconstruction from printed equations and scientific documents. The recent FERMAT benchmark [40] explicitly evaluated multiple SOTA VLMs on handwritten mathe- matics recognition, revealing critical gaps between general text OCR and handwritten mathematical recognition. Domain-specifte adaptations such as VLPG [20] and HIE-VL [21] propose graph-based or hicrarchical adapters to enhance mathematical recognition, although their extensive architectural modifications and modest performance limit broader adoption. In contrast, recent general-purpor frameworks like Qwen2.5-VL. [3], equipped with dynamic-resolution ViTs and structured outputs, demonstrate promising potential for adaptation to mathematical OCR tasks.
3 Method
We introduce Uni-MuMER, a unified multi-task fine-tuning framework for enhancing open-source VLMs in HMER. The overall pipeline is shown in Fig. 2. Given input image of handwritten expression and a task-specific instruction, the model is trained to produce the corresponding output. Uni-MuMER integrates four tasks: Vanilla HMER, Tree-aware Chain-of-Thhughy Error-Dret -rivve Learning, and Symbol Counting, into unified training paradigm. The primary goal is to adapt a general-purposs purpose VLM to the highly structured and domain -emain--peciffc knowledge of HMER without modifying any original architecture. Subsequent subsections provide additional details for each task.
3.1 Vanilla HMER
Base Model We adopt Qwen2.5- VL-3B [3] as the VLM Backbone. It comprises ViT-based visual encoder and a transformer-base language decoder, pre-trained to perform various image-to-text tasks, which provide robust visual grounding and structured sequence-generation capabilities, rendering it an effective foundation for end-to-end fine-tuning in HMER tasks.
Vanilla HMER As is shown at the top of Fig. 2, Vanilla HMER involves providing an image of mathematical expression alongside a textual instruction, prompting the model to directly output the corresponding 1TEX formatted expression. Traditional lightweight specialized models took images alone as input and generated the expressions, whereas VLMS now exhibit strong generalization capabilities across related tasks. We also utilize models fine-tuned specifically on the Vanilla HMER
3

---

Vanilia HMER
Reuhht
on image handwritten mathematical expression, Please write out expression the formulo in the inage using Lo TeXeX formot.
Uni-MuMER
Ciyy b
Count: 1A Lak. 2.0. wet
Please
image mathaematical and give the result latex
A A A
Tree GoT:
Symbol Counting
Qwen-VL
[leshe riggt diboniittt insooiiee truatjine mossture
vieble symbol in imgg, and then provide corresponding fornat.
image... identiff and
nigidiezy onghtfalr pudj
maskilai Srughthie lape thains eelic (ebeugaradt reggi ietrr onghoaanzc
Tree-Awaar Aware Chain-of- Chaln-of-Thought
on image... Please generate the abstract syntex tree (AST) formula in the inage, and then provide its corresponding LaTeX format.
(ighilaattr 2 sogels (inght
Final
Error Detection and Correction
(22--2222277
have on image and its OCR recogrition resuft. Please help to detect possible ervors the places where
recognition result and
end- and thar jas (8)-by ba (9) fagey (22--m)2322 a2)2212
with
Errer Detection
1i (ox (0)-by (5]=0]
<deletech. Expression:
(aad oar Crance w2-- (122111
I have or image.. and its GCR necognition resuit Please correct the errers marked vemor__strrte saroor_eeee rdeleted Expression:
Correction
expression.
RETLASES-
Lowejex 10510y c8/1e)1 oen come cant some ond (22))(222111) an125331
Figure 2: Overview of our Uni-MuMER Framework: which augments VLM base with four distinct integrated tasks: Vanilla HMER, Awee-Aware Chain-of- Thouuhhhh, Error-Driven Learning, and Symbol Counting to generate robust and accurate LTTęX from input image of a handwritten expression.
task as baselines for subsequent comparison. Through Vanilla HMER, the model acquires essential recognition capabilities and ensures accurate and structured outputs.
3.2 Tree-Aware Chain-of-Thougee
We propose Tree-CoT task that employs pars- ing Abstract Syntax Trees (ASTs) as interme- diate cognitive structures for expression recog- nition. The model accepts an input comprising an image and an instruction, subsequently gen- erating both serialized tree and the final rec- ognized 18TX expression. Inspired by ASTs widely utilized in mathematical information re- trieval [61, 42] and previous tree-decoding meth- ods in HMER [64, 52, 70], we incorporate ex- plicit tree-structured decoding into the textual output. This explicit intermediate representation enables the model to directly reason the inherent 2D spatial relationships among symbols. Conse- quently, Tree-CoT guides the model to produce structurally coherent and semantically accurate ыяバΕ outputs by bridging visual input process- ing and linear textual serialization. Below, describe in detail the construction.
Abstract Syntax Tree
Tree-CoT
Dopth-Firet Traversal
Vrac
Serialized AST
Formula
Ifac
2
(ebove)
a² + 1 2
(sup)
(right)
(below)
Input
Output
Figure 3: Illustration of the proposed Tree-Co'T construction and serialization procedure. For input expression, parse it into AST, then traverse in depth-first order and serialize into a structured textual format.
Tree-CoT The construction procedure of Tree-CoT is illustrated in Fig. 3. We first parse the input expression to derive its Abstract Syntax Tree (AST), explicitly capturing hierarchical and spatial relationships among symbols. We subsequently perform depth-first traversal (DFS) of the AST to sequentially linearize the tree structure. To encode this structured representation, we introduce a specific serialization scheme, using tab-based indentation to reflect tree depth and newline separation to distinguish individual nodes clearly, and the raw text is in Fig. 2. Each serialized line explicitly contains the symbol label and its spatial relation its immediate parent node, resulting in coherent textual representation of the AST.
3.3 Error-Driven Learning
Exror-Driven Learning (EDL) leverages a learning-froms soimiingfrrommmsstteee mistakes paradigm to enhance model accu- racy. Its input consists of an error corpus generated by the model itself, structured into two distinct subtasks: error detection and error correction. Inspired by self- self-improvement strategics and previous

---

work indicating the suitability of language models for correcting HMER errors [9], we integrate these subtasks explicitly. The subsequent sections detail the generation of the error corpus and the definitions of error detection and correction subtasks.
Error Corpus Generation As is shown in the Fig. 4. Initially, the complete dataset (e.g., CROHME) is randomly partitioned into multi- ple distinct subsets denoted as F1, ..,,.... Fn. Subsequently, we perform cross- volsddaaliotion train- ing, where each VLM is trained on 1 folds and evaluated on the remaining held-out fold F, During this, multiple candidate predictions are generated through multiple sampling for each in- put image to collect potential model outputs. By comparing these candidate predictions to their corresponding ground-trun -ruth labels, explicitly identify erroneous outputs and compile these er- roneous predictions into the final error corpus.
Partihioingg
Out Fald Training
Mulit Sample Corpus
Teak
F1
Somple
Error Corpus
F2
Fols Truin Data
Sample
:
F
scmple or Tootie
Sample &
Buiid
F
Fold Fş
Somple
Emarr Detection Correction Data
Figure 4: Illustration of Error Corpus Genera- tion for Error-Driven Learning Task (EDL), where erroneous model outputs are collected through cross-fold training and multiple sampling from held-out evaluation subsets.
Error Detection and Correction Based on this error corpus, we formulate two related subtasks: error detection and error correction. As shown in the Fig. 2, the error-detection task takes as input the original expression image alongside potentially erroneous predicted expression, outputting marked expression wherein errors are explicitly enclosed by <error_ sttrt>> and <error end>> tags, and omissions are explicitly indicated by <deleted> tags. The subsequent error-correction subtask receives this masked expression as input and produces the correction log and corresponding ыTTX expression. The scale of the error corpus obtained for each training set used in Sec. 4.1 approximately equal to that of the original training data.
3.4 Symbol Counting Auxiliary Task
We introduce a Symbol Counting (SC) auxiliary task designed to encourage the model to explicitly account for all symbols appearing in expressions. The input to this task comprises an expression image along with a counting instruction, and the expected output consists of the total count of visible symbols in the expression alongside the corresponding ыЯТІ expression. Inspired by the CAN [30], we observe that models frequently produce locally coherent yet globally inconsistent outputs, especially involving repeated symbols. To mitigate this issue, explicitly integrate symbol- counting information into the textual output representation. This task compels the model to accurately predict symbol counts, thus reducing the likelihood of symbol omissions or hallucinations during final LTTXX expression generation.
Symbol Counting. We construct training targets that prepend the count string to the actual K8TX.. Specifically, for a given handwritten expression image, the target output becomes <Count
string?. For example: For the expression a1+1 the textual count string is: \frac: a,:1 2:2,,+11
4 Experiments
4.1
Experimental Setup
This section details the experimental setup used to evaluate the Uni-MuMER, including datasets, evaluation metries, training settings, and basclines.
Datasets The CROHME series [37, 38, 39] is the most widely used dataset for HMER. It comprises 8836 training samples, and three test sets contain 986, 1147, and 1 1199 images for the CROHME 2014, 2016, 2019. Recently, the CROHME 2023 [56] iteration provided an expanded training set of 10979 images and a test set of 2300 images. HME100K [59] is a large-scale, real-world dataset. It consists of 74502 training and 24607 testing images, encompassing various writing styles and conditions. Math BathWriting [17], released by Google Research in 2024, is the largest HME corpus to date with 230K human-written and 400K synthetically-geneee generated expressions. Finally, Im21.atex22 [44] builds upon the printed expression dataset Im21 Im21.atex- a 100k[12] and enhances it with improved LTRX normalization and 59 diverse rendering styles for more realistic expression images.
5

---

Table I: Performance comparison on CROHME datasets( %).. We evaluate our model and four baselines using expression recognition rate (Exp) and ExpRate@CDD CDM (@ (@CDM) on the CROHME 2014, 2016 and 2019 test sets. We use the following notation for reported results: All models are evaluated under zero-shot inference settings, denotes training with data augmentation. P denotes
reproduced by and denotes the use of external data.
Method
CROHME14
CROHME16
CROHME19
Average
Expt
@CDM
Expt
@CDM
Exp
@CDM
Expt
@CDM
Closed MoureeSouuce Source Models
GPT-4o[41]
50.61
60.85
46.03
49.79
57.13
48.81
56.21
Doubao-L.5 [.5-pr][45]
50.41
69.17
49.00
64.08
46.87
64.05
48.79
65.77
Geus [41]
58.01
69.98
52.74
61.34
55.21
65,38
55.32
65.57
Source Models
3wen255-VL-3B[]]
38.64
44.01
37.75
43.33
37.45
43.04
37,95
43,46
Qwen2.5-VL-7BB33
55.98
65,51
50.92
56.32
49.62
56.55
52.17
59.46
Qwen2.5-VL-72B([]
59.74
68.56
54.32
59.37
55.13
61.05
56.40
62.99
Speciolized Models
DenseWAP[63]
50.1
47.5
BTTR[67]
$3.96
52.31
52.96
53.08
ABM[4]
56.85
52.92
53.96
54.58
CAN-ABM[30]
57.26
56.15
55.96
56,46
CoMER[66]
58.38
61,66
56.98
63.03
59.12
62.71
58.16
62.47
ICAL
60.63
58.79
60.51
59.98
PosFormer[1
60.45
60.94
61.20
TA
61.36
63.28
59.54
62.94
60.13
63,46
60.34
63.23
SSAN[62]
60.85
60.02
61.83
60.90
CoMER [66]
59.33
59.81
62.97
60.70
CoMER*21
46.96
53.55
43.33
48.47
48.25
54.84
46.18
52.29
Postormer [18]
62,74
65.88
61.03
65.39
64.97
69.30
62.91
66.85
SSAN*[62]
62.58
62.51
65.30
63.43
UniMERNet*
67.4
68.4
65.4
67.07
VLPG'(200
60.41
60.51
62.34
61.09
HiE-VL"
73.30
70.70
69.3
71.13
Vanilla (baseline)
72.62
75.66
64.26
67.83
69.14
71.98
68.64
71.82
Uni-MuMER
75.36
79.11 1945
70.79
75.24-744
73.73
77.23
73.29
77.19-5.32
Uni-MuMER!
82.05 99.4
85.09 9.43
77.94 13,68
81.08 13:25
79.23*
10.00
82.40
10442
79.74
82.86 11.4
Evaluation Metries The standard metric for HMER is Expression Recognition Rate (ExpRate), which measures exact-match accuracy. However, Exprate unfairly penalizes visually correct pre- dictions that differ only in IaTEX syntax styles. Character Detection Matching (CDM) offers visual evaluation alternative. renders both prediction and ground-truth into images, comparing them via character detection that considers spatial layout. This yields two metrics: the CDM Score, quantifying visual similarity, and Exprate CDM, a visually-based accuracy measure. These CDM metrics provide evaluation robustness against variations in ETTX syntax.
Training We fine-tune the Qwen2.5-VL. VL-3B model on all datasets with three data-driven tasks simultaneously for a single epoch. Further details can be found in the Appendix B.4.
Baselines We compare methods against four types of baselines: (i) Closed-source VLMs, including Gemini2.5-flash, etc; (ii) Open-source VL.Ms, such as the Qwen-2.5VL. family. Both closed-source VLMs and open-source VLMs are evaluated under zero-shot inference settings; (iii) Previous SOTA methods on HMER and their data augmentation versions, for example, PosFormer and SSAN, etc; and (iv) A Vanilla(baseline) and Uni-MuMER model, which is only fine-tuned on the CROHME training set to ensure a fair comparison with previous SOTA methods.
4.2 Main Results
Tab. 1 displays the strong performance of our Uni-MuMER and previous methods on the CROHME dataset. As CDM is recently proposed metric, we reproduced all reported results involving CDM. For a fair comparison, the Vanilla (baseline) reported in Tab. and Tab. 2 was trained exclusively on either CROHME or HME100K dataset. The Uni-MuMER variant incorporated training samples constructed from the corresponding images for three tasks. Further utilizing external data, UniMuMER1 was trained on approximately 1.6M samples, comprising data from 386K images across the three tasks and all datasets. Due to space limitations, additional results presented in Appendix D.

---

Comparison with Previous SOTA Methods Data augmentation has traditionally improved perfor- mance in lightweight models, yet inherent limitations still restrict overall effectiveness. Recently, VLPG [20] and VI. [21]. have integrated VLM into HMER. HiE-VL. enhances performance through non-end-to-end training, architectural refinements, and extra data. However, both Uni- MuMER and Uni-MuMER (use external data) maintain a notable performance advantage over these VLM-based methods. Notably, we also trained CoMER [66] using the same external data (386K images) and observed a significant performance degradation. This finding indicates that the inherent limitations of lightweight models hinder their ability to effectively utilize large and diverse datasets.
Comparison with Large VLMs (Zero-Shot) Next, we investigate the zero-shot performance of both open-source (Qwen-2.5VL family) and closed-source large VLMs on CROHME. Although open-source models exhibit nontrivial results, they tend to lag behind previous SOTA methods when no fine-tuning involved. Contrastingly, certain closed-source VL.Ms like Gemini2.5 exhibit remarkable accuracy, surpassing lightweight specialized models in a purely zero-shot setting. They still underperform relative to our proposed Uni-MuMER and Uni-MuMER method.
Incremental Gains Building upon the Qwen2.5- VL-3B VLM base, our Vanilla (baseline) model already surpasses most published SOTA meth- ods. After integrating three data-driven tasks, Tree-CoT, EDL. and SC, our method, Uni- MuMER further improves both ExpRate and Ex- pRate @ CDM, as shown in Tab. 1. For fair comparison, both the Vanilla (baseline) and Uni- MuMER are trained solely on the CROHME train- ing set, without using any external data. Intro- ducting Upon incorporating external data along with these tasks (1.6M samples), our final model, Uni-MuMERR! achieves even greater performance, notably reaching an ExpRate of 79.74% and Ex- pRate CDM of 82.86% on the CROHME Aver- age. Both Uni- MMMMER and Uni-MuMER! sig- nificantly outperform both the Vanilla (baseline) model and prior methods, underscoring the vital role of data-driven tasks and diverse external data in enhancing model accuracy.
Table 2: Performance on HME100K dataset (%), evaluated by Exp. @CDM, and CDM. The denotes the use of external data.
НМЕ100K
Method
Exp7
@CDMf
GPT-4o [41]
22.96
27.02
Gemini2.5 flash [11]
28.14
34.32
Doubao1.S-pro [45]
43,90
55.08
Qwen2.5-VL-3B [3]
44,42
47.41
Qwen2.5-VL-7B [3]
54.57
58.84
Qwen2,5-VL-72B [3]
49.59
55.09
DenseWAP [63]
61.85
CoMER [66]
68.12
70.36
TAMER [70]
69.50
71.30
HiE-VI. [21]
64.20
Vanilla (baseline)
70.15
71.80
Uni- Mui-MuMER
71.62
73.40
Performance on HME100K To focus the evalu- ation on mathematical expression recognition, fol- lowing [16], we exclude all test instances contain- ing CJK characters. Detailed impacts are discussed in Appendix C.5. Tab. 2 shows that spenesource and dlosed-soure VLMs exhibit modest performance, which is due to the challenging real-world con- ditions in HME100K dataset. In contrast, several prior lightweight specialized methods exclusively trained on HME100K, such TAMER, achieve strong performance. Although HiE-VL utilizes external training data, it achieves limited results on HME100K, even underperforming compared to prior models. Our Uni-MuMER achieves SOTA performance. Additionally, by using external data, Uni-MuMER further enhances its performance, clearly surpassing HiE-VL.
Uni- MuMERT
72.66
74.30
Table CROHME-Average ablation results (ExpRate %).
5 Analysis
5.1
Ablation Study
Tree-CoT
EDL
SC
ExpRatet
x
x
68.64
In this section, we present an ablation study to quantify the contribution of the three proposed tasks, Tree-CoT, EDL, and SC. As is shown in Tab. 3, each individual task enhances performance compared to the Vanilla(baseline) The Uni-MuMER combined integration of all tasks results in the best overall performance, while removing any mod- ule leads noticeable performance degradation, highlight- ing their complementary roles and collective importance.
/
x
X
70.85
X
70.30 1.66
X
x
l
69.86 1.22
x
V
71.95 33.31
X
71.06 2.42
71.76*312
73.29 +1.5
7

---

Tree-CoT enhances comprehension of structurally complex expressions Fig. 5a presents comparative analysis of accuracy across models with varying structural complexities of expressions. The Vanilla model experiences significant degradation in accuracy as structural complexity increases. Introducing the Tree-CoT strategy notably alleviates this performance decline. Specifically, for structurally complex expressions, Vanilla+TTree-Ca demonstrates approximately 5-6% over the Vanilla baseline, closely matching the performance of the Full model. Conversely, Tree-CoT offers marginal improvements for simpler expressions. suggesting its primary utility is enhancing model robustness and accuracy when handling structurally intricate expressions.
accuracy improvement of
EDL reduces confusion among visually similar characters Tab. enumerates the top five most fre- quent alphabet-number confusion pairs CROHME- Average for the Vanilla model and Vanilla+EDI. The Vanilla model frequently misclassifies pairs such as 2 resulting in 5.24 misrecognitions per expres- sion in all SUB1 cases. After applying EDL, the number misclassifications decreases to 3.31. This implies that EDL effectively guides the model in distinguishing fine-gr gree-graied differences among com- monly confused symbols, enhancing symbol--evel accuracy and overall recognition performance.
Table 4: Top-5 alphabet-number confu- sions (SUB1) on CROHME-Average, com- paring the Vanilla vs. Vanilla EDL.
Top-S misrecognition pair (%)
Type
2++2
0++.
3++2
1++i
1++
Vanilla
1.40
1.48
0.74
0.89
0.74
5.25
EDL
1.23
1.16
0.15
0.62
0.15
3.31
(J)
0.17
0.32
0.58
0.27
0.59
1.94
SC improves recognition consistency in long expressions Fig. 5b compares model accuracy with increasing symbol repetition. The Vanilla baseline accuracy deteriorates as repetition frequency rises, when symbols repeat five or more times. Incorporating the SC task substantially mitigates this decline, closely matching the Uni-MuMER model's stable performance under high repetition conditions. However, for simpler expressions with minimal repetition, SC slightly reduces accuracy compared to the Vanilla baseline, suggesting it occasionally diverts attention from primary recognition tasks.
90
80
84,00
77.34
77,53
75.68
74.51
80
75.09
72,81
72,56
60.21
72 "r
72.34
70
21.91
72.64
20.97
ExpRatoo%)
70
70.50%
67.07
67.07
ExpRattoo))
68.94
89.17
67.57
09:06
60.02
60
63664
64.63
64.20
60
Varilla
51.54
Vanilla
368.0
50
Uni-MaMER
Varilla+Tree-COf
51.87
50.22
V VyylllaSSymbol Counting
Uni-MuMER
53.33
40
50
Expression Structural Complexity Level
M Maximum Symhol Occurrence Count
(a) Model performance relative to structural complexity. The Tree-CoT task notably reduces accuracy loss as structural complexity increases.
(b) Model performance concerning symbol repetition count. Symbol Counting (SC) task mitigates accuracy degradation caused by frequent symbol repetitions.
Figure 5: Model accuracy comparison under varying (a) structural complexities and (b) symbol repetition counts. Both models significantly outperform the Vanilla baseline, approaching Uni- MuMER model performance in challenging scenarios.
5.2 Effects Data Diversity
In this section, we examine the influence of incrementally scaling training datasets on model per- formance. As illustrated in Fig. 6, start with the CROHME dataset, which serves as the standard benchmark in HMER. We then sequentially incorporate increasingly diverse datasets, including the updated CROHME-2023 set, the real-scene HME100K dataset, the extensive handwritten corpus Math Writing, and finally the printed-expression Im2LaTeXv2 dataset. At each step, the model retrained from scratch on the cumulative training data and evaluated across multiple distinct test sets.

---

To effectively account for variations in 1TEX notation styles, we employ ExpRate CDM as the standardized metric for consistent performance comparison.
Benefits of Enhanced Data Diversity Each in- cremental expansion of the training sets with more varied handwritten expressions yields con-
140
90.29
Tet Set Expitare CDMP))
75.01
sistent gains all evaluation sets, confirming that performance scales with data diversity.
70.10
-0
67.55
Out-of-Domaie Generalization Accuracy printed expression images gradually improves with increased handwritten training data, reflect- ing partial transfer of structural knowledge for ex- pression recognition. Nevertheless, its accuracy improves significantly after adding Im21.'TeXv2, reflecting the value of in-domain printed data for that target.
50.47
41990
HMEE
Mattweiing
In2taTekie Text Sot
CROUME Vrin
+ CHOCIMB-23
MaltWerizz (220K)
(IK))
(22))
(HKK pretadi
Figure 6: Impact of incrementally training data scaling on ExpRate @ CDM. Starting from the CROHME training set, we incrementally incor- porate four additional datasets. The model is re- trained from scratch and evaluated on five distinct test sets at each expansion step.
Ceave-One-Out Ablation To further investigate the
generalization performance, conducted an additional "Leave- Caee--Oe--'r ex- periment, training Uni-MuMER on N-1 datasets and evaluating it the held-out N-th dataset. The results below present the expression recognition accuracy in each scenario. For comparison, we include the baseline CoMER and Doubao-1.5 1.5-rr under zero-shot settings.
Table 5: Leave-one-out ablation of training datasets. (ExpRate @ CDM% %) We train Uni-MuMER in six configurations: using all data (all five sources), and using all-minus-one for each major dataset. Results are reported on CROHME-Average CROHME 2023, HME100K, Math MathWriting, and Im2LaTeXv2. For reference, we also show a specialized model (CoMER) trained with and without CROHME, and closed-source VLM (Doubao-1.5 -pro) for reference.
Model
CROHME-Avg CROHME 2023
НМЕ10оK
Math Writig
Im2l.a/TeXv2
COMER (wfo CROHME)
45.99
58.28
39.49
23.23
40.39
COMERT
52.29
59,91
44.63
28.45
53.37
Doubuo-1.5 1.5-pro
65.77
58.18
55.08
26. 36.34
27.66
Uni-MuMERR (wfo CROHME)
78.22
72.19
74.47
68.08
89.18
Uni MuMERT (wfo CROHME-2023)
Uni- Uni-MuMER! (wfo HMET0OK)
82.30
72.01
52.38
68.94
91.19
Uni-MuMER' (wfo MathWriting)
79.58
73.51
74.29
32.55
93.31
Uni MuMER (w/ Im2LaTeeXv2)
83.20
73.33
74.51
68.93
75.17
Uni-MuMER
82.89
74.42
74.30
69.11
93.29
The leave-one-out evaluation confirms that our unified model can generalize to an unseen dataset reasonably well. When the unseen domain is very different, performance drops more markedly, which highlights an area for improvement. HME100K for real-life, low-quality images, Math MathWriting Writing for densely structured expression, and Im2LaTeXvv2 for printed expression images, each of which represents a markedly different and challenging domain.
Crucially, after introducing even modest amounts of domain-relevant data, our unified approach easily achieves top-tic performance. Thus, our results indicate that, compared to existing specialized models and general VLM baselines, our approach provides superior generalization with minimal domain- specific data requirements, highlighting a clear advantage and direction for future improvement.
5.3 Mixing General-Purposs VLM Data
From the perspective of cumpros-purpose LLMs and VLMs, mixing data from different domains can yield substantial performance gains. Although our Uni-MuMER addresses previous limitations by introducing unified solution, achieving super SOTA performance, it's necessary to validate whether Uni-MuMER benefits from general domain data beyond HMER-only training and how Uni-MuMER-Daan affects the model's general capabilities. To this end, we introduce Uni-MuMER- ILLVA, tee-tued using an equal 1:1 mixture HMER and II VA-AOne data, and compare its performance with Uni-Me ni-MuMERR across benchmarks assessing HMER-specific and general capabilities.

---

Table Comparison among Uni-MuMER, Uni-MuMER-LLAVA and Qwen2.5-VL on multi-
modal reasoning benchmarks. Results are reported on MMMU val, With-n and Math-
Vista_testmini, covering general multimodal understanding and mathematical reasoning tasks.
Model
MMMU_va Math-Vision
Math MaahWista_festie festmini
Uni-MuMER Uni-MuMER-LLANE
47.89
24.01
47.8
48.67
24.34
51.1
Qwen2.5-VL VL-BBB I-Intruuct
52.78
21.38
47.9
Table 7: Cross-henchmark comparison between Uni-MuMER and Uni-MuMER-LLAN across
benchmarks. Results reported on CROHME-AVg, CROHME-2023, HME100K, Math Writing, and Im2VaTeXv2.
Model
CROHME-AvE CROHME-2223
НME100K MathWriting
Im2LaTeXv2
Uni-MuMER
82.89
74.42
74.30
69.11
93.29
Uni NUMER-LLAVE
81.99
69.10
73.48
70.48
93.44
As shown in Tab. 6 and Tab. 7, Uni-MuMER exhibits strong generalization capabilities despite being fine-tuned exclusively on HMER-specific data, achieving performance comparable to Dwen2.5VVL-3B Notably, M-M-MMER--LLAA further enhances overall performance, surpassing Uni-MuMER on general tasks like MMMU and Math, while incurring marginal performance decreases on specific HMER test sets, It's promising integrate our method and Uni-MuMER- Data to empower VLMs.
Discussion
Larger Model, Better Performance, Faster Speed Previous HMER models were lightweight and relied on sask-spec architectural designs (e.g., tree-structured modules). Uni-MuMER builds upon both these task-specific insights and current VLM, taking a significant step forward with a unified and more powerful model. Benefiting from recent advances in efficient inference frameworks, Uni-MuMER not only achieves substantially improved performance but also delivers faster inference compared to smaller models. Further detail is provided in Appendix C.2.
For Small Models:
For VLM: A Chain-of-Thought Perspective Expression Recognition Unlike direct expression recognition, Uni-MuMER adopts a chain-of-thought perspective by formulating three tasks: expres- sion tree construction, error detection and correction, and symbol counting. These tasks guide the model step by step toward the final prediction, enhancing both interpretability and generalization. This approach offers novel direction for advancing expression recognition with current VLM.
Self-Correction for Expression Recognition Uni-MuMER provides new perspective for enabling self-correction during inference in expression recognition. By integrating the Tree-CoT mechanism with an explicit error correction task, the model can exhibit self-corrective behavior within CoT process. Moreover, incorporating colloquial, human-like like expressions (c.g.. "wait, there's left parenthesis earlier-thi this is more likely right parenthesis, not "2"") can further induce R1-like spontaneous self-correction [19], enhancing the upper limit of the model's performance.
Future Direction:
Limitations Despite the strong results, our method has several limitations. Due computational resource constraints, 400k synthetic subset of the MathWriting dataset was unused, potentially limiting the diversity benefits. Additionally, exploring optimal task mixture ratios within Uni-MuMER and optimal domain mixture ratios between TMMER-speciffr data and general-domain data is crucial; however, these experiments were omitted due to significant resource constraints. Investigating these ratios could yield further enhance performance and generalizability.
7 Conclusion
This paper introduces Uni-MuMER, unified multi-task fine-tuning framework for HMER, estab- lishing new paradigm by leveraging large-scale VLMs integrating multiple HMER tasks unified textual modality. Without architectural modifications. Uni-MuMER integrates Tree-aware Chain- ohoThh, Error-Driven Learning, and Symbol Counting tasks to jointly address structured spatial reasoning, error correction, and long-expression consistency in expression recognition, achieving super SOTA performance. This unified method provides enhanced accuracy, scalability, and faster inference speed, establishing promising new research direction for future advancements in HMER.
10

---

8 Acknowledgement
This work is supported by the projects of Beijing Nova Interdisciplinary Program (20240484647) and National Natural Science Foundation of China (No. 62376012), which is also a research achievement of State Key Laboratory of Multimedia Information Processing and Key Laboratory of Science, Technology and Standard in Press Industry (Key Laboratory of Intelligent Press Media Technology).
References
[1] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt,
S. Altman, S. Anadkat, al. Gpt-4 technical report. arXiv preprint urXiv2233308772 2023.
[2] F. Álvaro, J.-A. Sánchez, and J.-M. Benedí. Recognition of on on-line handwritten mathematical expressions using 2d stochastic context-free grammars and hidden markov models. Pattern Recognition Letters,
35:58-67, 2014. 2
[3] S. Bai, K. Chen, X. Liu, J. Wang. W. Ge, S. Song. K. Dang, P. Wang. S. Wang, J. Tang. H. Zhong, Y. Zhu, M. Yang. Z. L., J. Wan, P. Wang. W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang. T. Xie, Z. Cheng, H. Zhang,
7. Yang. H. Xu, and J. I Lin. Qwen2.5-VT. technical report, Feb, 2025. 2, 6,7
[4] X. Bian, B. Qin, X. Xin, J. Li. X. Su, and Y. Wang. Handwritten mathematical expression recognition via attention aggregation based bi-directional mutual learning. In Proceedings of the AAAI Conference on
Artificial Intelligence, volume 36, pages 113-121, 2022. 2, 3,
[5] L. Blecher, G. Cucurull, T. Scialom, and R. Stojnic. Nougat: Neural optical understanding for academic documents, 2023. 3
[6] K.-F. Chan and D.-Y. Yeung. Mathematical expression recognition: survey. International Journal on Document Analysis and Recognition, 3:3-15, 2000. 2
[7] K. Chen, H. Shen, L.. Zhong, and M. Chen. Mmhmer: Multi-viewer and multi-task for handwritten mathematical expression recognition. arXiv preprint arXiv:2502.055 05557, 2025.
[8] X. Chen, X. Wang. S. Changpinyo. A. J. Piergiovanni, P. Padlewski, D. Salz, S. Goodman, A. Gryener, B. Mustafa, I.. Beyer, A. Kolesnikov, J. Puigcerver, N. Ding, K. Rong, H. Akhari, G. Mishra, I.. Xue, A. V. Thapliyal, Bradbury, W. Kuo, M. Seyedhosseini, C. Jia, B. K. Ayan, C. R. Ruiz, A. P. Steiner, A. Angelova, X. Zhai, N. Houlsby, and R. Soricut. PaI.:: A jointly-scaled multilingual language-image model. In The Eleventh International Conference on Learning Reprexentations, Sept. 2022. 2
[9] 7. Chen, J. Han, C. Yang, and Y. Zhou. Language model is suitable for correction of handwritten mathematical expressions recognition. In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4057-4068, Singapore, Dec. 2023. Association for Computational Linguistics. 3.5
[10] P. A. Chou. Recognition of equations using two-dimensional stochastic fortext-free grammar. In Visual Communications and Image Processing IV, volume 1199, pages 852 865. SPIE, 1989. 2
[11] G. DeepMind. Gemini 2.5 Flash: Enhanced multimodal model. https://blog gowgle/ google/products/ producto/
gemini/ Seataningemniii2-----e f-assh wtw-----eaaa--peiiiee//ee Apr. 2025. Public preview announced 17 Apr 2025. 2, [12] Y. Deng, A. Kanervisto, J. Ling, and A. M. Rush. Image-to-markup generation with coarse-to-fnn attention.
In International Conference on Machine Learning, pages 980-989. PMLR, 2017. 3,5
[13] D. Dricss, F. Xia, M. S. Sajaadi, C. Lyneh, A. Chowdhery, Wahid, J. Tompson, Q. Vuunng Yu, W. Huang, et al. Palm-e: An embodied multimodal language model, Mar. 2023. 32
[14] A. Fadeeva, P. Schlattner, A. Maksai, M. Collier, F. Kokiopoulou, J. Berent, and C. Musat. Representing online handwriting for recognition in large vision- Iasion-language models. arXiv preprint 4rXiv:2402.15307,
2024. 32
[15] S. A. Firdaus and K. Vaidehi. Handwritten mathematical symbol recognition using machine learning techniques: Review. In S. C. Satapathy, K. S. Raju, K. Shyamala, D. R. Krishna, and M. N. Favorskaya, editors, Advances in Decision Sciences, Image Processing, Security and Computer Vision, volume 4, pages 658-671. Springer International Publishing. Cham, 2020.
[16] Y. Gao, W. Wang, and H. Ney. Uniblock: Scoring and filtering corpus with unicode block information. In K. Inui, J. Jiang, V. Ng. and X. Wan, editors, Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1324-1329, Hong Kong, China, Nov. 2019. Association for Computational Linguistics. 7
[17] P. Gervais, A. Fadeeva, and A. Maksai. Mathwriting: A dataset for handwritten mathematical expression recognition. arXiv preprint arXiv:2404. 10690, 2024. 25
[18] T. Guan, C. Lin, W. Shen, and X. Yang. Posformer: recognizing complex handwritten mathematical expression with position forest transformer. In European Conference on Computer Vision, pages 130-147. Springer, 2024. 2, 3, 6, 26
11

---

[19] D. Guo, D. Yang, H. Zhang. J. Song, R. Zhang, R. Xu, Q. Zhu. S. Ma, P. Wang, X. Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprini arXiv:2501.12945.
2025. 10
[20] H.-Y. Guo, C. Wang, F. Yin, X.-H. Li, and C.-L. Liu. Vasion-language pre-training for graph-based handwritten mathematical expression recognition. Pattern Recognition, 162:111344 11346, 2025. 3, 6, [21] H.-Y. Guo, F. Yin, J. Xu, and C.-I.. Liu. HiE-VI.: A large vision-language model with hierarchical adapter for handwritten mathematical expression recognition. In loassp 2025 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (Icasxp), pages 1-5, Hyderabad, India, Apr. 2025. IEEE. 3, 6.
7.25
[22] Ha, R. M. Haralick, and T. Phillips. Understanding mathematical expressions from document images. In Proceedings of 3rd International Conference on Document Analysis and Recognition, volume 2. pages 956-959, IEEE, 1995. 2
[23] J. J. Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8)-255--2558, 1982.
[24] Y. Huang. I.v, I.. Cui, Y. Lu, and F. Wei. Layouthmv3: Pre-training for document ai with unified text and image masking. In Proceedings of the 30th ACM international conference on multimedia, pages
4083-4091, 2022.
[25] C. Jia, Y. Yang. Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. Sung. Z. Li, and T. Duerig, Scaling up visual and usion-language representation learning with noisy text supervision. In International conference
machine learning, pages 4904-4916. PMLR, 2021. 2
[26] N. Jiang, S. Liang, C. Wang, J. Wang, and L. Tan. Latte: Improving latex recognition for tables and formulae with iterative refinement. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 4030-4038, 2025. 26
[27] G. Kim, T. Hong, M. Yim, J. Nam, J. Park, J. Yim, W. Hwang. S. Yun, D. Han, and S. Park. Ocr-fire free document understanding transformer. In European Conference Computer Vision, pages 598-517.
Springer, 2022.
[28] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 61 1-626, 2023.
[29] W. Kwon, Z. Li, S. Zhuang, Y. Sheng. L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th
Symposium on Operating Systems Principles, pages 61 626, 2023. 2
[30] B. Li, Y. Yuan, D. Liang, X. Liu, Z. Ji, J. Bai, W. Liu, and X. Bai. When counting mects hmer: counting- aware network for handwritten mathematical expression recognition. In European Conference on Computer Vision, pages 197-214. Springer, 2022. 2, 3, 5,
[31] J. Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping aagguage-image pre-training with frozen image encoders and large language models. In International conference on muchine learning, pages 19730-19742.
PMLR, 2023. 2
[32] J. Li, D. Li, C. Xiong, and S. Hoi. BLIP: Bootstrapping language-image image pre-training for unified vision- language understanding and generation. In Proceedings of the 39th International Conference on Machine Learning, pages 12888- 12900. PMLR, June 2022. 2
[33] Z. Li, B. Yang, Q. Liu, Z. Ma, S. Zhang, J. Yang, Y. Sun, Y. Liu, and X. Bai. Monkey: Image resolution and text label are important things for large multi-modal models. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26763-26773, 2024. 3
[34] Y. Liu, Z. Li, M. Huang. B. Yang, W. Yu, C. Li, X.-C Yin. C.-L. Liu, L. Jin, and X. Bai. Oerbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 67(12)220010, 2024. 2
[35] Y. Liu, B. Yang, Q Liu, 7. Li, 7. Ma, S. Zhang, and X. Bai. Textmonkey: An ocr-free large multimodal model for understanding document. arXiv preprint 2iix:2403300443 2024.
[36] A. Madaan, N. Tandon, P. Gupta, S. Hallinan, I. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye,
Y. Yang. et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information
Processing Systems, 11146534445994 2023. 2
[37] M. Mahdavi, R. Zanibbi, H. Mouchere, C. Viard Hard-Gaudin, and U. Garain. ICDAR 2019 CROHME+ TFD: Competition on recognition handwritten mathematical expressions and typeset formula detection. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1533-1538. IEBE, 2019. 2,5
[38] H. Mouchere, C. Viard-Gaudin, R. Zanibbi, and U. Garain. Iefhr 2014 competition on recognition of on-line handwritten mathematical expressions (crohme 2014). In 2014 14th International Conference on Frontiers in Handwriting Recognition. pages 791-796, 2014. 2.5
[39] H. Mouchère, C. Hand-GGudin,, R. Zanibbi, and U, Garain. lefhr2016 crohme: Competition on recognition of online handwritten mathematical expressions. In 2016 15th International Conference on Frontiers in Handwriting Recognition (ICFHR), pages 607-612, 2016. 5.5
12

---

[40] O. Nath. H. Bathina, M. S. U. R. Khan, and M. M. Khapra. Can vision-language models evaluate handwritten math? arXiv preprint arXi::2501.02244 2025. 3
[41] OpenAI. GPT-40: Multimodal generative transformer. httpe://ppenaa..ccm///aeee/e///
2024. Flagship model unveiled 13 May 2024. 6,
[42] S. Peng, I.. Gao, K. Yuan, and Z. Tang. Image to latex with graph neural network for mathematical formula recognition. In Document Analysis and Recognition-ICDAAR 2021: 16th International Conference, Lausanne, Swätzerland, September 5-10, 2021, Proceedings. Part II 16, pages 648-663. Springer, 2021. 4 [43] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and L. Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, pages 8748-8763.
PMLRR, July 2021. 3
[44] F. M. Schmitt- Schmitt-Koopman Koopmann, M. Huang, H.-P, Hutter, T. Stadelmann, and A. Darvishy. MathNet: data-centric approach for printed mathematical expression recognition. IEEE Access, 12.76963-66974,
2024. 5, 25, 34
[45] B. D. Team. Doubao 1.5 Vision Pro: Multimodal large language model. https: : / /sed. bytedance. com/en en/ eel/e///pppeee /ppp//aa//dubbaaaooooooo 1_______________ __p_, Jan. 2025, Official release reported 22 Jan 2025. 2, 6,
[46] T.-N. Truong, C. T. Nguyen, R. Zanibbi, H. Mouchère, and M. Nakagawa. A survey on handwritten
mathematical expression recognition: The rise of encoder-decoder and GNN models. Pattern Recognition,
153:110531, Sept. 2024. 2
[47] H. M. Twaakyondo and Okamoto. Structure analysis and recognition of mathematical expressions. In Proceedings of 3rd International Conference on Document Analysis and Recognition, volume 1, pages
430-437. IEFE, 1995.
[48] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L.. Jones, A. N. Gomez, E.. Kaiser, and I, Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 1,3
[49] B. Wang. Z. Gu, G. Liang, C. Xu, B. Zhang, B. Shi, and C. He. Unimernet: A universal network for real- world mathematical expression recognition. arXiv preprint arXiv:2404.15255 2024. 3
[50] B. Wang. F. Wu, L. Ouyang, Z. Gu, R. Zhang, R. Xia, B. Shi, Zhang, and C. He. Image over text: Transforming formula recognition evaluation with character detection matching. In Proceedings of the Computer Vision and Puttern Recognition Conference, pages 1968 1688-19690,, 2025. 35
[51] J. Wei, X. Wang. D. Schuurmans, M. Bosma, F. Xia, E. Chi. Q. V. Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing syxtems,
35:24824-244 22837,44277773 2022. 2
[52] C. Wu, J. Du, Y. Li, J. Zhang, C. Yang, B. Ren, and Y. Hu. TDv2: A Novel Tree- Structrrdd Decoder for Offline Mathematical Expression Recognition. Proceedings of the AAAI Conference on Artificial Intelligence, 36(3):2694-2702. June 2022. 2.4
[53] J.W.. Wu, F. Yin, Y.M. Zhang. X.-Y. Zhang, and C.-L. Liu. Image-to-markes generation via paired adversarial learning. In Machine learning and knowledge discovery in databases: European conference, ECML PKDD 2018, Dublin, Ireland, September 10- 14, 2018, Proceedings, Part I 18, pages 18 18-34.
Springer, 2019. 2,
[54] J.-W. Wu, F. Yin, Y-MM. Zhang, X.-Y. Zhang, and C.-L. Liu. Handwritten mathematical expression recognition via paired adversarial learning. International Journal of Computer Vision, 128:2386-2401,
2020. 2,
[55] Y. Xie and H. Mouchère. Tst: Tree structured transformer for handwritten mathematical expression recognition. International Conference on Document Analysis and Recognition, pages 236-252. Springer,
2025. 33
[56] Y. Xie, H. Mouchère, F. Simistira Liwicki, S. Rakesh, R. Saini, M. Nakagawa, C. T. Nguyen, and T.-N. Truong. ledar 2023 crohme:
Competition on recognition handwritten mathematical expressions. In International Conference on Document Analysis and Recognition, pages 553-565. Springer, 2023. 5, 25 [57] S. Yao. D. Yu, J. Zhao, I. Shafran, T. Griffiths. Y. Cao, and K. Narasimhan. Tree thoughts: Deliberate problem solving with large language models. Advances in neural information
systems, 36: 11891
11822, 2023. 2
[58] Y. Yuan, X. Liu, W. Dikuhab, H. I Liu, 7. Ji. 7. Wu, and X. Bai. Syntax-aware network for handwritten mathematical expression recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3553-4562, 2022. 3
[59] Y. Yuan, X. Liu, W. Dikubab, H. Liu, Z. Ji, Z. Wu, and X. Bai. Syntax-aware network for handwritten mathematical expression recognition. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pages 4553-4562, 2022. 5
[60] R. Zanibbi, D. Blostein, and J. R. Cordy. Recognizing mathematical expressions using tree transformation.
IEEE Transactions on pattern analysis and machine intelligence, 24(11):1455-144) 2002. 2
[61] R. Zanibbi, H. Mouchere, and C. Faarin Evaluating structural pattern recognition for handwritten math via primitive label graphs. In Document Recognition and Retrieval XX. volume 8658, pages 411 411-444 1-411. SPIE, 2013. 4
13

---

[62] Н. Zhang, X. Su, X. Zhou, and G. Gao. SSAN: A symbol spatial-aware network for handwritten mathematical expression recognition. Proceedings of the AAAI Conference on Artificial Intelligence. 39(21);2239-222406 Apr. 2025. 2,
[63] J. Zhang, J. Du, and L. Dai. Multi-scale attention with dense encoder for handwritten mathematical expression recognition. In 2018 24th international conference on pattern recognition (ICPR), pages
2245-2250, 2018. 2, 6, [64] J. Zhangg, J. Du. Y. Yang. Y.-Z. Song. S. Wei, and L. Dai. A tree-structured decoder for image-to-markup generation. In ICML, page Press, 2020. 2,
[65] J. Zhang. J. Du, Zhang. D. Liu, Y. Hu, Hu, S. Wei, and L. Dai. Watch, attend and parse: An end-to-end neural network based approach handwritten mathematical expression recognition. Puttern Recognition.
71:196-206, 2017. 2,
[66] W. Zhao and I.. Gao. Comer: Modeling coverage for transformer-bssed handwritten mathematical
expression recognition. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel,
October 23-27. 2022, Proceedings. Part XXVIII, pages 392-408. Springer, 2022. 2, 3, 6,
[67] Zhao, L. Gao, Yan, Peng. L. Du, Zhang. Handwritten mathematical expression recognition with bidirectionally trained transformer. In Document Analysis and Recognition ICDAR 2021: 16th International Conference, Lausamne, Switzerland, September 5-10, 2021, Proceedings, Part II 16, pages
570-584. Springer, 2021. 2, 6
[68] D. Zhelezniakov, V. 7aytsev, and O. Radyvonenko. Online handwritten mathematical expression recogni-
tion and applications: A survey. IEFe Access, 9:38352-38373 2021. 1,
[69] Zhu, L. Gao, and W. Zhao. Ical: Implicit character- learning for enhanced handwritten mathematical
expression recognition. arXiv preprint arXiv:2405 09002, 2024. 3
[70] Zhu, W. Zhao, Y. Li, X. Hu, and L. Gao. Tamer: Tree-aware transformer for handwritten mathematical expression recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 10950-10958, 2025. 2, 3,4, 7,
9 Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and Introduction enumerate three contributions, which are uni- fied multi-task fine-tuning, three data-driven tasks, and new SOTA results, all verified experimentally in Tab. and Tab. 2.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: dedicated "Limitations" paragraph See. 6, which discusses domain- specific scope, compute constraints, and unused synthetic data.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
14

---

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms
and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't ' acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
The work is empirical and introduces no formal theorems proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Thcorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have taken steps to ensure the reproducibility of our result. Sec. 4.1 detail datasets, and Appendix introduce how to preprocess datasets and hyper-parameters
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
15

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We have not included the actual code or data with the submission. However, we provide thorough instructions and descriptions the paper and the appendix to reproduce all main experiments. We will publicly release code, model checkpoints, and detailed usage instructions in the future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.co
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. wttpccccpppcccceeeeeee.e.c.cccte wwwpc//ggss///e//eu Cooeeuumisssii ssionPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
16

---

Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: All data are public, and provide the split training sets and test sets. Hyperpa- rameters and optimizer are introduced in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We did not report explicit error bars, confidence intervals, or statistical signifi- cance tests for our experimental results. Due to the high cost fine-tuning large VLMs, each configuration
once; therefore report point estimates without confidence
intervals.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Answer: [Yes]
Justification: Appendix introduces training used 4 A800-80 GB GPUs. 80 GB VRAM each, single-epoch on 1.6M train samples, which cost 96 hours, and describes that training run for one epoch on a combined dataset of around 1.6M images.
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http///eeeippp.eeeee cttpps.cc//poee htcpppulic/tthhcchieeeellel//ell
Answer: [Yes]
Justification: We have read the NeurlPS Code of Ethics and confirm that our research adheres to it. Our work involves improving handwriting recognition model using public, licensable datasets and does not present unethical content methodology.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: We did not include dedicated broader societal impacts section because work focuses on a specialized technical application with limited direct societal implications. The primary impact of our research is positive, improving tools for digitizing and under- standing handwritten mathematics, which can benefit education and scientific document processing.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
18

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The released fine-tuned model targets mathematical OCR and poses minimal dual-use risk; no additional safeguards are required.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Sec. 4.1 cites CROHME, HME100K, MathWriting, and Im2LaTeXv2 with their original papers and open licences; code bases (e.g.. Qwen2.5-VL, are Apache-2.0.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 'ss creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We do not introduce new proprietary assets in this paper. Our contributions lie in the unified fine-tuning approach and tasks, applied to existing datasets and an existing open-souree model.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
19

---

The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This work involves only publicly available handwriting datasets; no human subjects were recruited.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: IRB approval is not applicable because the study does not collect human- subject data.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions NeurIPS Code of Ethics and
and locations, and we expect authors to adhere guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: We clearly describe our use of large-scale pre-trained model in the paper. In particular, Sec. 3, explains that our method is built the Qwen2.5-VL-3B vision-language model, which we fine-tune for HMER. The role of this L.LM/LLM in our approach explicitly documented as core component of the methodology.
Guidelines:
20

---

The answer NA means that the core method development in this research does not involve LLMs as any important. original, non non-standard components.
Please refer to our LLM policy (https : /ttp///ee.ppp/eeeeee/eee/eeeeee. cc/Conferences///20 /LLM) for what should or should not be described.
21

---

Appendix
Table of Contents
Prompt Design for Multi-Tasi Training
23
B Expreimental Details
25
B.1 Uni-MuMER-Dade -Dta Preprocess...
25
B.2 Evaluation Datasets and Metrics
26
B.3 Evaluation Prompting Protocol
27
B.4 Training Detail
27
C Extended Analysis
28
C.I Effects of Data Diversity Compared with Specialized HMER Model
28
C.2 Inference Efficiency
28
C,3 Alternative explorations format
28
C.4 Hallucination for HMER
30
C.5 CJK influence for HMER
31
More Experimental Results
32
D.1 Results on Handwritten Mathematical Expression
................................
32
D.2 Results on Printed Mathematical Expressions
33
E Case Study
35
22

---

A Prompt Design for Multi-Task Training
We design distinct prompts for each of the four training tasks in Uni-MuMER. Each prompt guides the model specific sub-task, and the model's expected output format is tailored to that task. Each prompt consists fixed system prompt: You are a helpful assistant. Below we list the prompts used for Vanilla HMER, Tree-CoTT, Error-Driven Learning, and Symbol Counting (SC), along with example prompts and the form of expected outputs. It a detailed expansion the content in Fig. 2.
Example A.1: Prompt for Vanilla HMER
Prompt: have an image of handwritten mathematical expression. Please write out the expression of the formula in the image using BTEX format.
Answer:
| |fraac {ax-
+ c \sqrt a { + {2)}}1
Example A.2: Prompt for Tree-CoT
Prompt: have an image a handwritten mathematical expression. Please generate the abstract syntax tree (AST) of the formula the image, and then provide corresponding BTJ format. Answer:
Afrac (right)
a (above)
(right)
(aub)
(right)
b (right)
(right)
0 (sub)
(right)
(right)
Naqrt (below)
(above)
2 (sup)
(right)
b (right)
2 (sup)
(right)
\frac { b {0) c { Asqrt a - { (2)331
Example A.3: Prompt for Error Detection
have an image of handwritten mathematical expression and its OCR recognition result. Please help me to detect possible errors in the recognition result and mark the places where errors occur with Cerror_starts <error_ cerror_end> and <deleted>.
Prompt:
erroneous formula: Airac a ( (() Isqrt | + (2)))
Marked formula:
Answer:
|frrac {ax_ by (0) c
(sqrrt {
<<rmor_start>> d cerror_ends [2) (2)))
Example A.4: Prompt for Error Correction
have an image handwritten mathematical expression and a predicted formula with error tags, correct the formula by modifying the parts marked with serror_starts and <error_end and inserting content where <deleted> are present. Output the modifications in a single string using the following format: REPLACE:old new for errors be replaced. INSERT:content after the position for missing content. DELETE:to_ded _ecleteeeeeeeee for parts be removed.
Prompt:
Marked formula: Mfrae a ( 0 by (0) +c] \sqrt cerror_stare <error_end>> (2)+b
} }}}
Correction log:
Answer:
REPLACE:
Iffrac x (0)+ (sqit (2)+b*(2))
23

---

Example A.5: Prompt for Symbol Counting
T have an image of a handwritten mathematical expression. Please identify and count each distinct visible mathematical symbol in the image, and then provide its corresponding 1 format.
Prompt:
Answer:
I: \frac: a: 2, x:
3. 2. y: e: 1, lsqi:: 1 2.2: |Mrrac [ax_ b [0) + lsqrt (2 + (211111
These carefully designed prompts ensure that during fine-tuning, Uni-MuMER learns not only to recognize the expressions but also to reason about structure, identify and correct errors, and pay attention to every symbol. By training on this multi-task prompt set, the model acquires a richer understanding of handwritten math, which cumulatively improves accuracy and robustness.
24

---

B Expreimental Details
B.1 Uni-MuMER-DDan Preprocess
We perform comprehensive preprocessing on all datasets prepare the training and evaluation data for Uni-MuMER, This involves cleaning noisy data, tokenizing 1TEX strings into standardized tokens, and normalizing different LTE notations to consistent style. BTEX expressions often be written in syntactically different ways while conveying the same meaning, so unifying these representations crucial for effective cross-dataset training.
Data Cleaning First, we exclude all instances containing CJK characters. For example, the HME100K dataset occasionally contains Chinese characters in annotations; these were removed to avoid confusing the model. Additionally, we remove characters or symbols that lack semantic value and negatively impact recognition accuracy, such Aunderline (\quad) and empty brace pairs ((), which not contribute to the math content.
Data Tokenization Since the I&TEX expressions provided by datasets like CROHME 2023 [56] and Mathwriting [17] are not inherently tokenized. To train a sequence model effectively, we convert each expression into standardized sequence of LTTEX tokens. Specifically, we adopt the tokenization procedure from the preprocess. searrccessformuleee as script of the open-source im2markup repository, leveraging the KaTeX JavaScript library to perform semantic-level tokenization. During tokenization, also remove any syntactically invalid expressions that fail to parse. Finally, the expression \frac a 2 is tokenized as \frac 2 2, where each LTTEX token is separated by space.
Data Normalization After tokenization, we normalize the ground-truth 1АTΕX sequences across all training datasets to ensure consis- tent format. Different datasets might represent the same mathematical concept in slightly dif- ferent ыТІК forms. Such variations include no- tation differences (e.g., using Veq and Mee,, op- tional braces (a 2 and
Table 8: Examples of BTEX math normaliza- tion. Comparison of original math expressions, normalized versions, and their rendered figures. Following the configurations of HiE-VL. [21] and MathNet [44], apply these transformations to make training more consistent.
2 ), stylistic markup (e.g., using (ex vs. just a), which typically not visible handwritten notes. If left unnormalized, these variations could con- fuse the model or reduce its ability to generalize between datasets. We therefore systematically unify these to a single canonical form. For exam- ple, the expression \frac a 2 2 is normalized
Transform
Original
Normalized
Figure
Brace
HTT Vint [int[[b][aa
H*[
H' So
SubSup
Vint_ [nt_[a][[bb
Root
1sqrt[2]a
[sqrt[][[a]
Vo
Stylized
Mtextbf(a)
a
as Afrac a 2 } { 2 Detailed normalization transformations
summarized
in Tab. 8, illustrating the specific QАТК syntax standardization rules we applied, extending and optimizing the practices initially established by MathNet [44].
Table 9: Dataset filtering summary and multi-task sample counts.
Category
CROHME
CROHME2023
НМE100K
Mathwriting
Im2LaTeXV2
raw
8834
12204
74502
229864
74244
rendering
587
1166
624
1153
normalization step
590
4749
3954
brackets invalid
0
0
65
82
rendering errors
0
0
293
Total (after deletion)
8834
11616
72746
224133
69054
Tree-CoT
8834
11616
72746
224133
69054
Error Detection
9010
6287
49959
125316
8828
Error Correction
11434
8860
58884
166387
17047
Symbol Counting
8834
11616
72746
224133
69054
Total (with four tasks)
38112
38379
254335
739969
163983
25

---

Dataset Filtering Each dataset contributed a different number of valid expressions after prepro- cessing. Tab. summarizes the filtering outcomes for the major datasets we used: CROHME. CROHME2023, HME100K, MathWriting. and Im22a For each dataset, we list the original number of expressions and how many were removed for various reasons, such as rendering errors or invalid syntax detected during normalization. The final row gives the count of expressions remaining in each dataset after all filtering steps.
Task Data Construction With the cleaned datasets in hand, we construct the multi-task training samples. Each image can yield multiple training samples, one for each task. The lower part of Tab, lists the number of training samples per task for each dataset. For the Tree-CoT, Symbol Counting, and Vanilla HMER tasks, we typically generate one sample per image, so their counts per dataset equal to the number of remaining images. For the Error Detection and Error Correction subtasks, we collect all error corpus generated by each dataset. Further inspired by recent work on incremental refinement LATTE [26], our error-correction task is specifically constrained to correcting one error per round, which is more effective than inputting the whole incorrect expression. In total, across all datasets, our training set comprises roughly 1.6 million samples derived from about 392k unique images, spanning the three tasks.
B.2 Evaluation Datasets and Metrics
Datasets The MNE (Multi-level Nested Expression) dataset is specifically designed evaluate models on recognizing complex handwritten mathematical expressions [18]. The dataset is organized into subsets designated as N1, N2, and N3, based on the structural complexity of the expressions, determined by the maximum nested levels within their substructures. Fig. 7 illustrates various mathematical expressions along with their corresponding tree-based structural complexities. For
instance, an expression such as +1+1 possesses structural complexity level of 2, indicating two hierarchical nesting layers.
The Character Error Rate (CER) CER is computed via Levenshtein distance the Unicode character level, normalized by ground truth length. It is widely used in online handwriting datasets (e.g.. Math Writing) to detect localization errors, brace imbalances. and missing superscripts. EditScore Following MathNer, we adopt EditScore, defined as Fdir distance EditScore, the
Evalution Metric
max by 3
sequence-level counterpart to CER, strongly correlates with human judgments printed Mathematical Expression Recognition (MER) tasks and tolerates minor reordering and length variations while preserving semantics. BLEU-4 For consistency and comparability with existing corpus in the im2latexv2 domain, we incorporate the BLEU-4 metric, calculated based on 4- 4-gram overlap after WTEX normalization, which effectively quantifies n-gram overlaps and remains valuable metric.
a +
a +1 1
a + 1 = b b
a² + 1
Σ
2
2
a
ffaac
sum
right
above
below
right
2
+
2
Afrac
right
right
sup
right
1
2
2
sap
right
right
+
right
right
b
Complexity
Complexity =
Complexity = 2
Complexity
Figure 7: Examples of structural complexity for different expressions
26

---

B.3 Evaluation Prompting Protocol
When evaluating the fine-tuned Uni-MuMER model and comparing it to various baselines, we standardize the prompting method to ensure a fair and consistent evaluation across models: Below, we describe the exact prompts and settings used in different evaluation scenarios.
Uni-MuMER (Ours) We used the Vanilla HMER prompt to elicit the final IME output when evaluating our fine-tuned model the test sets. Instead of explicitly prompting the model to show its chain-of-thought or to count symbols during this evaluation phase, we directly request the final LTTEX output. The rationale that our multi-task training has already imbued the model with better internal reasoning and understanding; at test time, we just want the final answer, not the intermediate steps. Thus, for each test image, we simply provided the vanilla prompt like: "I have an image of handwritten mathematical expression. Please write out the expression of the formula in the image using BTEX format." " to the model, along with the handwritten expression image.
Baseline VLMs (Open-Source and Closed- -Source) We evaluate several open-source VLMs and closed-source VLMS in a zero-shot setting on the CROHME and HME100K datasets. However, one challenge was that some general models might produce additional explanatory text or not output the expression alone. To mitigate this, minimally revise the prompt ask the model to output the LTEX expression in the \boxed ). The prompt is "I have an image of a handwritten mathematical expression. Please write out the expression of the formula in the image using BTEX format and place it inside \boxed]"". We then post-processed the output by removing the \boxed!] wrapper to retrieve the raw ЕЛЕК.
Output Normalization After obtaining outputs from all VLMs in zero-shot setting, we apply the same cleaning and tokenization steps used in training each output before evaluation. This is important so that metrics like ExpRate or CER treat all outputs on equal footing. In particular, for the HME100K test set, if a model produced any non-18TeX characters CJK text, we dropped those outputs entirely.
Reproducibility of Baselines To ensure transparency and reproducibility, explicitly detail the exact versions of the models evaluated in this study. Closed-source models were accessed through their respective APIs or publicly available releases at specific reference dates: Gemini 2.5-flash (version: gemini-2.5 2-5-ini-.5--lash-ppp pwet--rreviii---e ), Doubao- 1.5-pro (version: doubao- 1iabao-I..-visionn 1.5-vision-pre pro-250328) ), and GPT-4o (version: 111-4--2024-11-22 ). For open-source models such as Owen2.55VVL (3B, 7B, and 72B). we adopted the latest publicly accessible checkpoints available at the time of experimentation. All models underwent evaluation on identical test datasets, specifically the CROHME 2014, 4, 2016, and 2019 datasets, as well as the HME100K test set. We meticulously ensured consistent preprocessing procedures and uniformly applied evaluation metrics across all model outputs.
By enforcing this consistent prompting and processing protocol, we aimed to make the comparison as fair possible. All outputs were judged by exactly the same criteria. This way, the results reflect the true capability of each model on HMER, without artifacts from prompting differences.
B.4 Training Detail
Implementation Details We fine-tune Qwen2.5-VL-3B using full-parameen aall-parrmeter update. Besides, we employ a cosine decay schedule to gradually reduce the learning rate to 10% of its initial value (1 x 10-5) by the end of training. Training is conducted for a single epoch using all tasks and mixed dataset batches. Due to memory constraints, we adopt fit batch size of 512 within approximately 80 GB of VRAM per GPU (8xA800). This large batch size was specifically chosen to accommodate the diversity of tasks and stabilize the training process.
precision training, which enables us to
Training time We trained for single epoch over the combined multi-task dataset. One epoch in this context amounted roughly 1.6 million training samples, since each image contributes multiple task-specific samples. Iterating over 1.6M samples with batch 512 yields about 3152 optimization steps. In practice, this single epoch took 20 hours to complete on the 8x A800 setup. Although this runtime slightly exceeds that of training lightweight specialized models from scratch, our approach achieves convergence to a gegh-performance level within just a single epoch.
27

---

C Extended Analysis
C.1 Effects of Data Diversity Compared with Specialized HMER Model
In this subsection, we examine how incrementally scaling training data impacts ExpRate CDM. As noted in Sec. previous specialized models typically train on individual datasets, limited by the available training data and rarely assessed using visual metrics. Building upon the analysis in Sec. 5.2, we start with the CROHME dataset alone and then progressively add more training data from four additional sources. At each stage, we train three models from scratch on the combined data up to that point: (i) a lightweight specialized model, COMER, which a strong transformer- based architecture representative of specialized models; (ii) our vanilla baseline, which is only fine-tuned on 392k unique images; (iii) Uni-MuMER which multi-task fine-tuned on the 1.6M samples).
The results, summarized in the Tab. 10, indicate that CoMER initially benefits from increased data diversity but subsequently exhibits significant performance degradation, settling modest perfor- mance. Conversely, the vanilla baseline and Uni-MuMER show a much more robust improvement curve. Finally, Uni-MuMER consistently outperforms the Vanilla model at every stage and shows continuous gains. This demonstrates that the multi-task fine-tuning is effective at harnessing diverse data. Furthermore, specialized models like CoMER typically require extensive hyperparameter tuning, including adjustments to learning rates and training epochs. In contrast, Uni-Muti MiiMM attains superior performance after only single training epoch without additional optimization efforts. This underscores Uni-MuMERR 's robustness and efficiency.
Table 10: Effect of progressively adding training datasets on CROHME-Arg ExpRate CDM. At each step, models are trained from scratch on the cumulative data. Uni-MuMER improves monotonically and consistently outperforms the Vanilla (single-task) and CoMER baselines.
Model
CROHME Train CROHME-23
HME100K
Math MathWriting
+ 1m21 L
COMER
58.14
60.08
63.10
52.17
52.29
Vanilla (baseline)
71.14
75.07
75.13
80.87
80.35
Uni-MuMER
75.36
76.27
78.01
83.20
82.89
Uni-MuMER-TT
77.45
81.17
80.35
83.35
83.34
C.2 Inference Efficiency
We evaluated the inference speed and model size of our proposed Uni-MuMER model against the base- line methods CoMER and TAMER, utilizing a sin- gle NVIDIA A800 GPU. To ensure consistent and fair comparison, we used the HME100K test dataset, comprising 24,607 real-scene handwritten mathemat- ical expression images. Additionally, we leveraged vLLM, an optimized transformer-basee inference engine known for accelerating decoding processes through efficient batch processing and caching mech- anisms. The comparative efficiency metrics are sum- marized Table 11. As demonstrated, Uni-MuMER significantly surpasses previous HMER models in terms of inference speed, achieving an FPS ap- proximately eightfold higher than existing baselines under identical hardware conditions. As result, Uni-MuMER not only accurate but also efficient enough for practical use.
Table 11: Comparison of methods on effi- ciency metries. Throughput reported in frames per second (FPS, higher is better) and model capacity in trainable parameters.
Method
FPS
Parameters
COMER TAMER
5.23
6.39 M
4.50
8.23 M
Uni-MuMER
43.40
3.75
C.3 Alternative explorations format
In developing the Tree-CoT and Symbol Counting tasks, we explored various output representation formats and their impact model performance. Our goal was to find format that is expressive enough to capture structure/counts but concise enough for the model learn effectively. Here we report experiments with alternative formats for the expression tree serialization in Tree-CoT, as well as for the symbol counting task. We compare each alternative to the format ultimately used in Uni-MuMER (referred to as "our format"). All experiments in this section were conducted using
28

---

only CROHME data for training (no external data), so that differences reflect format efficacy rather than additional data. We evaluate both in a Vanill+tX setting and in the full Uni-MuMER setting (all three tasks, but substituting the alternative format for the relevant task) to assess each format's influence. Below, we outline each representation with an example.
Example C.1: S-expression foramt for Tree CoT
(mu_(pi)
Example C.2: Json AST foramt for Tree CoTT'
['type": 'supsub', "value": 'base": "type": 'mathord", "value": "Imu', 'sub': "type": "ordgroup", "value": ['type': 'mathord', 'value": "p". "type": 'mathord', "value": "J]]
Example C.3: Simplified Json AST foramt for Tree Co'T
(supsub: base: "Imu', sub:
sub: ordgroup: ['p'.
The S-expression is merely the formula itself and adds no metadata. Its nested structure also hurts readability and organization. The JSON AST is excessively complex-it reflects JavaScript's 's internal LaTeX format-and includes substantial redundancy. The "Siiplliied JSON AST" was manually trimmed and lacks a consistent standard. contrast, our AST format (used in Uni-MuMER) strikes balance between structural clarity and brevity with special delimiters to indicate hierarchical relationships. Our format avoids the heavy overhead of full JSON while preserving the necessary structure of the expression tree.
Tab. 12 reports the model performance using each of these tree formats, which shows that the choice of format significantly impacts accuracy. We attribute this gain to our format's compactness and structured guidance: it provides clear structural cues (like ISON) but with far fewer extraneous tokens, making it easier for the decoder to learn the task.
Table 12: Performance impact of different expression tree formats for Tree-CoT. Bold numbers
best in column. Setup: Models trained the CROHME dataset and evaluated on CROHME 2014, 2016, and 2019 test sets and their average. Results are reported in terms of ExpRate and ExpRate CDM. We compare the baseline Vanilla model against variants that incorporate the Tree- CoT task using various AST output formats, and likewise compare full Uni-MuMER models using those formats.
CROHME 2014
CROHME 2016
CROHME 2019
CROHME Average
Model
Exp
@CDM
Exp
@CDM
Exp
@CDM
Exp
@CDM
Vanilla Vanilla S-expression
72.62
75.66
64.26
67.83
69.14
71.98
68.67
71.82
72.21
75.56
66.87
70.97
68.06
71.73
69.05
72.75
Vanilla Json AST
71.70
75.66
66.70
70.51
70.56
74.04
69.65
73.40
Vanilla Simplified Json AST
72.62
76.47
66.96
70.18
69.89
73.06
69.82
73.24
Vanilla AST
73.33
75.76
68.00
70.97
71.23
74.40
70.85
73.71
Uni-MuMER (w/ S-expression)
74.85
78.07
68.07
73.14
72.56
76.14
71.83
75.78
Uni-MuMER (w/ Json AST)
75,46
79.51
70.10
74.28
72.98
76.06
72.85
76.62
Uni-MuMER (w/ Simplified Json AST)
74.44
78.27
70.36
74.63
73.90
77.48
72.90
76.79
Uni-MuMER (w/ our AST)
75.36
79.11
70.79
75.24
73.73
77.23
73.29
77.19
In addition, investigated alternative formats for the Symbol Counting (SC) auxiliary task. The default format in Uni-MuMER expresses the symbol counts as simple list of symbol-count pairs, separated by commas. We compared this with more explicit map/dictionary format. For example, consider the expression x [ 2 +y f 1
Example C.4: Map Counting format for Symbol Counting
(xx": 1, 107; 2, ; 2, 24?; 1, Py": 1, 1, '1' 13
Example C.5: Our Counting format for Symbol Counting
x: 1, 7: 2, 2: 2, +: 1, y: 1, <: 1, 1:
29

---

The Map Counting format is semantically equivalent to ours but clearly more redundant (it introduces additional punctuation and quotation characters). We found that this extra overhead can make the sequence longer and potentially confuse the model (especially because symbols like braces and quotes have no direct meaning in the counting task itself). As a result, as shown in Tab. 13, the map format yielded slightly worse performance. The differences are modest, but consistent.
Table 13: Impact of Symbol Counting formats. Bold numbers best in column. Setup: Models are trained on the CROHME dataset and evaluated on CROHME 2014, 2016, and 2019 test sets and their average. Results are reported in terms of ExpRate and ExpRate CDM. We compare using JSON-like Map Counting format versus our simple listing format for the SC task.
CROHME 2014
CROHME 2016
CROHME 2019
CROHME Average
Model
Exp
@CDM
Exp
@CDM
Exp
@CDM
Exp
@CDM
Vanilla
72.62
75.66
64.26
67.83
69.14
71.98
68.67
71.82
Vanilla + Map Counting
72.92
76.67
65.39
70.01
69.89
73.65
69.40
73.44
Vanilla + our Counting
71.91
76.85
67.04
72.28
70.64
75.15
69.86
74.76
Uni-MuMER (w/ Map Counting)
75.05
78
70.62
74.43
72.48
76.21
72.72
76.48
Uni-MuMER (w/ Counting)
75.36
79.11
70.79
75.24
73.73
77.23
73.29
77.19
C.4 Hallucination for HMER
Large generative models are known to occasionally hallucinate output tokens that do not belong or omit tokens that should be present. In the context of HMER, both big VLMs and prior specialized models can produce spurious symbols. To quantitatively analyze this, we preliminarily adopt two metrics: Hallucination Token Rate (HTR) (percentage of predicted tokens not in the ground truth) and Missing Token Rate (MTR) (percentage ground truth tokens not in the pred). Additionally, we categorized hallucinations by token type (numerical((-9), variable(a-zA-22 structural delimiters (e-g., D. (). O). This gives finer-grained error counts: _Tm__ HTR_var HTR HTR_delim for hallucinated numbers, letters, and delimiters, respectively. We evaluated these metrics across the CROHME 2014, 2016, and 2019 test sets, ensuring that no external data was used during training. The results are summarized below:
Table 14: Error analysis of hallucinations and omissions on CROHME. Results include ExpRate (Exp), ExpRate @DM,, overall HTR and MTR errors, well as specific error types (HTR_num, _nmm, HTR var, LITR_delime We compare two previous SOTA models ( CoMER and TAMER the Vanilla baseline, ablations adding each of our tasks individually (Tree-CoT, EDL, SC), and the full
Uni-MuMER.
Model
Exp
CDM
HTR
MTR
HTR num
HTR var
HTR delim
CoMER (S0TA'2022)
59.18
62.47
2.55
3.91
0.37
0.33
1.18
TAMER (SOTA'2024)
60.29
63.18
2.17
4.02
0.29
0.29
1.05
Vanilla (baseline)
68.64
71.74
4.08
2.42
1.08
0.29
1.68
Vanilla + Tree-CoT
70.85
73.71
1.67
1.56
0.14
0.21
0.85
Vanilla + EDL
70.30
73.76
2.58
1.94
1.09
0.22
0.83
Vanilla CAN
69.86
74.76
2.15
2.06
0.23
0.27
1.08
Uni-MuMER (w/ tasks)
73.29
77.19
1.18
1.39
0.12
0.15
0.60
Tab. 14 shows that full model yields the highest accuracy while also having the lowest HTR and MTR of all methods. Besides, incorporating each auxiliary task individually already helps: adding the Tree-CoT task slashes hallucinations to 1.67% HTR, adding the Symbol Counting task alone significantly lowers omissions (MTR), and adding the Error-Drive Learning task reduces hallucinations of delim and var. These improvements suggest that Tree-CoT primarily mitigates structural hallucinations, EDL corrects symbol-level mistakes, and SC enforces global consistency.
30

---

C.5 CJK influence for HMER
In Sec. 4.2, we excluded samples containing CJK characters of HME100K. This is primarily for two reasons:
Table 13: Examples of CJJ-ccotaining im- age-caption pairs in HME100K that lie outside pure expression recognition, which influence the robustness of expression recog- nition and are outside the scope of math ex- pression recognition.
1. Our prompt specifically targets recognizing math- ematical expressions, not general textual content such as CJK characters. This filtering ensures a clear evaluation of expression recognition capabil- ity.
image id
caption
2. Some CJK-containing samples introduced prob- lematic annotations unrelated to pure expressions (e.g., textual descriptions or numeric lists)
train_63827
=88 5
train_
1
14 x4.4 3.14)
train_ 70305
390 1,3,
Having justified the filtering strategy, we additionally examine the effect of CJK characters. We first list typical CJK-containing captions illustrate the distributional shift (Tab. 15), then compare performance under two evaluation settings: (i) test set filtered to remove CJK, and (ii) test set retaining CJK. We also test training variants that include exclude CJK samples. For reference, we also include results for CoMER and TAMER (previous specialized models) evaluated on the same splits.
Table 16: Comparison of recognition performance HME100K test sets with and without CJK characters. Results are reported in terms of ExpRate and ExpRate @ CDM. We evaluate three models each subset: CoMER (specialized, SOTA 2022), TAMER (specialized, SOTA 2024), and Uni-MuMER, For Uni-MuMER, we report two training conditions: w/o CJK and w/ CJK. For CoMER and TAMER, we evaluated open-sourced models under zero-sho shot inference settings.
HME100K Test w/ CJK
HME100K Test w/o CJK
Model
ExpRate
ExpRate@CDM
CDM
ExpRate
ExpRate@CDM
CDM
CoMER TAMER
68.47
70.19
96.79
68.70
70.29
96.81
69.50
71.12
97.02
69.74
71.25
97.03
Uni-MuMERI (w/o CJK)
69.46
72.49
97.08
72.66
74.30
97.38
Uni-MuMER| (w/ CJK) 71.63
73.60
97.32
70.98
74.32
97.38
As is shown in Tab. 16, the performance of CoMER and TAMER remained nearly identical before and after filtering. Our model still outperforms previous methods, with the CJK characters the same in the training set, which suggests minimal effect on comparability, and including vs. excluding the CJK samples in training had minimal effect on overall HMER accuracy.
We further examine whether including CJK characters in training has any noticeable effect the cross-domain performance of Uni-MuMER. Tab. 16 and Tab. 17 summarizes ExpRate CDM results for model variants across benchmarks. These results indicate that adding the small set of CJK- containing samples introduces tiny trade-off: it may slightly compromise performance on the main handwriting benchmarks, while providing negligible benefit certain out-of-domain cases. We conclude that excluding the CJK data is reasonable choice to keep the training focused, with no meaningful loss in generality.
Table 17: Effect of including CJK during training evaluated on ExpRate CDM across bench- marks. Results reported CROHME-Average CROHME 2023 test, Mathwriting test, and 1m2Latexv2 test. The best results for each column are highlighted bold.
Model
CROHME-Avg
CROHME 2023
Mathwriting
Im2Latexv2
Uni-MuMERY (w/o CJK)
82.89
74.42
69.11
88.99
Uni- MuMER (w/ CJK)
82.21
72.14
68.91
90.28
31

---

D More Experimental Results
D.1 Results on Handwritten Mathematical Expression
Tab. I and Tab. 2 in the main paper. we focused on the strict ExpRate and visual CDM metrics. Following past lightweight specialized models in HMER, we report Σ 1 and tolerating up to one or two token prediction errors within the generated 15TEX sequence. Here, we provide additional metries and comparisons on multiple datasets to fully characterize Uni-MuMER's strengths.
Tab. 18 is an extended version of Tab. 1. Specifically, Tab, 18 reports not only ExpRate but also the lenient metries allowing up < 1 and < 2 token errors. We compare Uni-MuMER against broad range of baselines: open-source VLMs, closed-source VLMs, previous SOTA methods on HMER, and their data augmentation versions.
Tab. 19 presents an extended evaluation of our method on the CROHME2023 test set, providing metrics including ExpRate, <. 1 and < 2 token errors, well as CDM-based evaluations. We benchmark Uni-MuMER and Uni-MuMER against various baselines. Our baseline model already demonstrates competitive performance against previous SOTA methods. Notably, after incorporating external data, Uni-MuMER exhibits a drop ExpRate due to changes in LaTeX-style outputs but achieves outstanding performance in ExpRate CDM.
Tab. and Tab. 21 shows the results on the MNE dataset. This shows that the vanilla model already competitive with the prior SOTA on these challenging sets. By introducing three data-driven tasks and external data, the Uni-MuMER and Uni-MuMER outperform the prior best methods.
Tab. 22 shows a comparison of Math MathWniting using CER. Mathwriting is dataset of hand writwritten expressions paired with online handwriting traces and offline rendered images. With its introduction, Google released two foundation models, PaL.I [14] and PaLM-E [13]. which are capable of reading both images and sequences of pen strokes as input [14]. While online handwriting traces typically aid models in mitigating ambiguity among visually similar characters, resulting in enhanced performance, our proposed Uni Mni-MuMER model demonstrates superior results despite relying exclusively on offline image data for both training and evaluation.
Tab. 23 an extended version of Tab. 2. presenting the performance of various models on the
HME100K test set. We report Exprate, 1, s and the CDM-based metrics for this dataset.
Table 18: Performance comparison on CROHME datasets (%). We evaluate our model and four baselines using expression recognition rate (Exp), < 1, and < 2 on the CROHME 2014, 2016, and 2019 test sets. We the following notation for reported results: All results zero-shot inference; a, data augmentation, P: reproduced results, f; external data used.
CROHME 2014
CROHME 2016
CROHME 2019
Method
Expt
Sit
21
Expt
$17
2
Exp t
S <1f
S
GPT-4o
50.61
65.21
77.69
61.99
73.85
49.79
64.79
75.00 06
Doubao-1.5-ppo
50.41
68.15
77.69
49.00
61.99
73.85
46.87
68.81
78.65
Gemini2.5 5 flash
58.01
71.60
81.14
52.74
667.28
76.24
55.21
70.14
79.48
Qwen2.5 Swen2.5-VL-33. L-BL-3
38.64
50.81
60.55
37.75
$1.00
61.38
37.45
53.88
63.39
Qwen2.5 VL.
55,98
68.76
77.28
50.92
67.13
76.37
49.62
67.22
76.73
Qwen2.5.V1.-72 5-
59.74
70.39
78.40
54.32
69.53
77.51
55.13
69.81
78.65
WAP
50.1
47.5
BTTR
53.96
66.02
70.28
52.31
63.90
68.61
$2.96
65.97
69.14
ABM
56.85
73.73
80.61
52.92
69.66
78.73
53.96
71.06
78.65
CAN-ABM
57.26
74.52
82.03
72.71
CoMER
58.38
74.48
81.14
56.98
74.44
81.87
59.12
77.45
83.87
PosFormer
60.45
77.28
83.68
79.40
TAMER
61.36
76.77
83.25
60.26
76.91
84.05
61.97
78.97
85. 80
SSAN
60.85
75.56
82.25
60.02
76.22
83.28
61.83
79.08
86.08
CoMER
59.33
71.70
75.66
74.37
80.30
62.97
77,40
81.40 40
CoMER PPT
60.65
64
PosFormer
62.68
79.01
84.69
61.03
77.86
84,74
64.97
82,49
87.24
SSAN=1
62.58
62.51
65.30
YLPG! HIE-V
60.41
74,14
78.90
60.51
75.50
79.86
62.34
76.81
81.15
73,30
84.00
87.00
70.70
82,10
86.40
69.30
81.90
86.20
Vanilla (haseline)
72.62
81.34
86.82
88.07
Uni- Mui-MuMERR
75.36
85.09-101
89.05-233
70.79
83.44-1.4
89 89.54 553
73
9776
82.05
89.45
93.00 16.18
77.94
87.45
92.07
79.23
90,91
93.83
32

---

Table 19: Performance on CROHME2023 Test set (%). Evaluated by ExpRate, ExpRate @ CDM, and CDM. The denotes the of external data. The denotes the results reproduced by [55].
CROHME 2023
Method
ExpRate
S11
21
ExpRate@ CDM f CDM 1
VLPG
56.47
68.54
73.63
CoMER P
59.82
69.23
74.12
TST
60.72
70.51
75.00
TST!
62.59
73.59
77.51
Vanilla (baseline)
68.39
79.13
87.57
69.24
94.43
Uni-MuMER 70.26
81.13*2.00
88.35+078
71,47
95.12-0d6
Uni Uni-MuMER 1.61
79.78
$7.00
74.42 15.8
95.33 0050
Table 20: Performance comparison the MNE dataset (%). We compare the expression recognition rate (ExpRate) and CDM-based metries between our model and previous SOTA models on the MNE N1/N2/N3 test The denotes the use of external data.
N2 @CDM1 CDM
N3 @CDM1 CDM
Method
Exp
@CDM CDM
Exprt
Expt
CoMER
59.73
37.17
24.04
PosFormer
60.59
38.82
36.82
MMHMER
62.03
39.47
34.29
Vanilla (baseline)
64.37
68.21
93.94
51.65
58.22
92.89
35.66
46.59
91.75
Uni-MuMER
70.08
74.19
95.33
55.59-184
63
94.64
38.53-288
50.96
93.63-1.8
Uni-MuMER
76.00
81.44
96.37
58.88
71.38
96.08
46.45
10.7
66.67
95.13
Table 21: Performance comparison on the MNE dataset (%). We compare the expression recogmition rate (ExpRate). S 1, and < between our model and previous SOTA models on the MNE N1/N2/N3 test sets. The denotes the use of external data.
NI
N2
N3 S 1
Method
Expt
<1f
21
Exp
s 1
22
Exp
2
CoMER
59.73
77.55
84.11
37.17
53.95
65.13
24.04
32.31
36.34
PosFormer
60.59
77.97
84,32
38.82
56.91
66.12
36.82
40.30
43.10
ММHМER
62.03
78.13
84.32
39.47
56.57
65.46
34.29
36.68
40.51
Vanilla (buseline)
64.37
78.61
85.65
51.65
59.54
69.08
35.66
44.81
52.12
Uni-MuMER
70.08 5771
83.73
90.29
1464
55.59
64.80
73.36
4.28
38.53
49.18
1.37
57.24
Uni-MuMER'
76.00 11.63
87.31
(8.70
92.16 (85)
58.88
7.23
74.34
11450
82.90
13:02
46.45
10.79
54.37
99.56
63.66
11.54
Table 22: Comparison with multi-modal models PaLI and PaLM-E on Math Writing, CER is the character error rate (lower is better). The denotes the use of external data.
Model
Input
CER 4
Exp t
@ CDMM f
CDM 1
PaLI
Image
8.07
PaLI
Ink
4.64
PaLI
Ink Image
4.55
PaLM-E
Image
4.87
PaLM-E
Ink
6.46
PaLM-E
Ink + Image
4.22
Vanilla
Image
4.15
50.51
66.46
94,20
Uni-MuMER
Image
4.01
51.19
68.73
94.80
Uni-MuMER
Image
4.00
51.45
69.11
94.84
D.2 Results on Printed Mathematical Expressions
An important aspect our approach is whether model fine-tuned primarily for handwritten ex- pressions also performs well on printed mathematical expressions. To test this, we evaluate on the Im21aTeXv22 dataset, a dataset of printed formulas rendered from ЕЛІХ. Minor normalization adjust- ments were applied to align the dataset with our preprocessing standards. Specifically, we performed synonym replacements such as changing expressions from \le \leq, inserted spaces to sepa- rate commands clearly (c.g., converting (begin(align) \begin {align)), and standardized
33

---

Table 23: Performance on HME100K dataset (%). denotes the use of external data.
Evaluated by Exp, @CDM, and CDM. The
HME100K
Method
Exp 1
S
2t
@CDM f
CDM
GPT-4o
22.96
37.32
47.40
27.02
83.13
Gemini2.5-flash
28.14
44.23
54.75
34.32
85.43
Doubaol.5-pp
43.90
64.52
74.09
55.08
91.28
-wen2.5-VI--3B
44.42
61.47
70.38
47.41
89.96
Qwen2.5-VL-7B
54.57
71.05
78.44
58.84
92.90
Qwen2.5-VI.-228
49.59
66.21
73.79
55.09
93.08
WenseWAP
61.85
70.63
77.14
CoMER
68.12
84.20
89.71
70.36
96.83
TAMER
69.50
85.48
90.80
71.30
97.07
HiE-VL'
64.20
Vanilla (baseline)
70.15
85.81
91.25
71.80
96.97
Uni-MuMER
71.62
86.92
92.16
73.40
97.30
Uni-MuMER 72.66 2.551
87.67 1.86
92.60 1.35
74.30 22.00
97.381041
notation by enclosing single-element subscripts like a_i within braces a (.]. Other than these normalization steps, no further modifications were made, and the mathematical semantics remained entirely unaffected. These normalization procedures consistent with the preprocessing approach detailed Appendix B. For transparency, we also include results from evaluations performed on the original, unnormalized set. Such normalization adjustments were applied other test sets in our study, thus preserving fairness and comparability across datasets.
Tab. 24 summarizes these results using common metrics from the Im2LaTeXvv: Edit Score, BLEU-4, and ExpRate. We also include ExpRate@CDD) for completeness, especially since our models output ETEE that could be visually equivalent even if not exactly matching. Our Vanilla baseline performed slightly better than the previous SOTA. After completing three data-driven tasks, Uni-MuMER further reached an ExpRate of 88.52. We also check our Uni-MuMERI, which is primarily optimized for handwriting. Interestingly, its performance on purely printed data a bit lower than Uni-MuMER, perhaps because training such broad mix causes slight domain shift. Nonetheless, all models outperform MathNet [44].
Table 24: Comparison of model performance the im2latexv2 test set. We compare our models to prior Im2LaTeX systems. Metrics shown are Edit Score, BLEU-4, ExpRate, and ExpRate@ CDM. All metrics are percentages (%), and higher values indicate better performance. The lower half reports results on the original, unnormalized test set. Missing values indicated by
Model
Train Dataset
Edit
Bleu-4
ExpRate
ExpRate@ CDM
WYGIWYS
im2latex- 100k
37.2
23.9
0.0
121-strips
im2latex-1444 140k
75.9
65.9
10.3
121-nopool
am2latex-1400
76.0
66.4
10.4
MathNet
im2latexv2
97.2
96.8
83.9
Vanilla (baseline)
im2latexv2
99.12
98.44
88,08
93.19
Uni-Mutt MuMER
im2latexv2
99.16
98.70
88.52
93.46
MniMMuMERi
Uni-MuMER-Dare
99.33
98.71
88.19
93.29
Evaluation on Unnormalized Test Set
Vanilla (baseline)
im2latexv2
98.71
97.59
79.85
93.19
Uni-MuMii
im2latexv2
98.75
97.85
80,27
93.46
Uni-MuMER'
Uni-MuMER-Daie
98.54
96.72
76.36
93.29
34

---

E
Case Study
We present a qualitative comparison illustrate how Uni-MuMER handles challenging expression versus other models. As shown in Fig. 8, we compare the outputs of closed-source VLMs, prior lightweight HMER model (CoMER), our fine-tuned baseline (vanilla), and our Uni-MuMER. The figure displays each model's 's predicted 18TEX, highlighting any incorrect tokens in red. These case studies reinforce our quantitative findings: Uni-MuMER not only quantitatively better but qualitatively produces significantly cleaner and more complete outputs, especially expressions that involve several levels of nesting and a variety of symbols.
(go0))oooo))
(A(x)=
4Bm c Va
2 2- 8 (
Imago
V 3x
aITi
(Vntty Vifty Vintay Vintty) (Wnfty liffly Vinfty Vinfty)
(oort(t-ak]- - -WVeftgeammm Vogge//right
2404 bo (frackeB,aan222art22-l
forht Vied Viaed (hhae, 1241 (ndh tain Mreed Shhte 1331 (1B) ban (meee) tshera, (DA) (22 (na
flah
(ooo)ooooo)
2000) b Mrrac44 1oit yoort [Lax]]
[sqr((t-x]]--a 4/1gammm Magg(41
Viset (2 thew Eirtzorettieennn eeeee yourtiitt (wrtnrend)
Sprr
(1la 12ta (31a (4)) [__ [216_ [R]o__]] [4]]
= Vfrec (4 B
Flgrt (1 xx]
(saart [1+3x]1 3x 4 (aaammm + Vog 3)
(c0000][00000]
2h(x)= be Mrae (4 B
Tugrt (1 }
View 1123 showe En how Visr 1EB tiee Mier 1121 12
Deert [1 x 3x)] (1gamma 1 (
(laeiinol
(000D)(0000)
= (4 B\ppp loorrt
Negrt + (y Vog [4]
Caa Maat 1-34 Md
(42 Mowa
Mekres
(C000)(0000)
Bipi /oort Ossgr[1 + 3- lagammm [4))
Vinc 21123 Moote (1.1) 13-11/ (13 Wher 1ichea
Figure 8: Qualitative comparison on a challenging handwritten expression. Compared with closed-source VLMs, CoMER, Vanilla(baseline), and Uni-MuMERI, The red symbols represent incorrect predictions.
CDM is not the ultimate evaluation metric yet Although CDM metries [50] quantify the visual equivalence between predicted and ground ыÓT₃X sequences beyond ExpRate, they not definitive for evaluating иЛІІ recognition ac- curacy. CDM tends to prioritize local charac- ter matching, potentially neglecting significant global structural discrepancies. This oversight can lead to inaccurate recognition assessments, is shown in Fig. 9, CDM fails detect the incorrect subscript structure of the character ug mistakenly equating it with wuzz due to relying solely on local character matching without considering global hierarchical relationships.
6uu
6u
Figure 9: An example of CDM leading to incor- rect recognition assessment.
35


Reviews:
[{"review_id": "zeGETmuyut", "reviewer": "Reviewer_DUPV", "summary": "This paper explores the use of a general-purpose large language model (vLLM), specifically Qwen, fine-tuned for the task of handwritten mathematical expression recognition. The authors introduce three targeted fine-tuning tasks designed to inject task-specific knowledge into the model: (1) generation of abstract syntax trees (ASTs), (2) self-error detection and correction, and (3) symbol counting.\n\nThe proposed approach is evaluated against both state-of-the-art specialized models and larger generalist models on several databases. Experimental results show that the fine-tuned Qwen model outperforms these baselines across several benchmarks. The paper also presents an ablation study assessing the individual contribution of each fine-tuning task as well as an analysis of the model's out-of-domain generalization capabilities.", "questions": "- Why choosing Qwen-3B for fine-tuning, considering that the 7B version already achieves significantly better performance even without fine-tuning?\n- In Table 8, inference time comparisons suggest that smaller specialized models are slower than Qwen. How is this possible? \n- Will the source code and trained models be made publicly available?\n- What value of k was used in the k-fold Error Corpus Generation process?\n- In Table 3, the dataset version is not specified. Moreover, I could not locate the reported result for CoT+EDL+SC (73.29) in other tables, particularly Table 1. Could you clarify this point?\n- The experiments in Section 5.2 suggest that the specialization of the Qwen model is only effective when fine-tuning is performed with in-domain data. Simple task-specific fine-tuning (for mathematical expression recognition) appears insufficient unless the fine-tuning data is closely aligned with the test set. This observation seems to partially contradict the goal of the proposed specialization tasks, which aim to introduce general task knowledge into the model, independent of the dataset. Could the authors clarify their position on this point and discuss whether in-domain fine-tuning remains, in practice, the most effective way to improve model performance?\n- The proposed future direction of Self-Correction for Expression Recognition remains somewhat unclear. It is difficult to see how \"colloquial expressions,\" such as the example provided, could be effectively integrated, given that they depend on the specific expression being recognized. Could the authors provide more concrete examples or clarify how this type of contextual, expression-dependent correction could be implemented?", "limitations": "The authors have addressed some of the limitations of their approach, notably by discussing how their model could be evaluated on other tasks, such as OCR. However, the paper does not adequately address the potential implications of using a significantly larger generalist language model for a relatively narrow, well-defined task such as handwritten mathematical expression recognition. The increased computational resources and environmental costs associated with large-scale vLLMs could limit the practicality of the approach compared to smaller, task-specific models. A more explicit discussion of these issues would improve the paper.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths:\n- First systematic evaluation of fine-tuning a generalist vLLM for handwritten mathematical expression recognition.\n- Well-motivated fine-tuning tasks (AST generation, self-correction, symbol counting) that are relevant and can be adapted to other domains.\n- The model outperforms both specialized state-of-the-art models and larger generalist models.\n- Ablation study (Section 5) is carefully designed and informative, clearly linking fine-tuning tasks to specific recognition improvements.\n\n## Weaknesses:\n- The scientific novelty of using a vLLM for this task is limited, even though the experimental exploration is solid.\n- The effectiveness of the approach seems heavily dependent on access to in-domain data, which limits its general applicability.\n- The Error-Driven Learning approach requires fine-tuning multiple models, making it potentially expensive to apply.\n- The future research direction on self-correction lacks clarity, especially regarding handling of context-dependent expressions.", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reiewer DUPV\n\nWe sincerely thank the reviewer for the thoughtful feedback, which is helpful to improve the paper. \n\n## **Weakness 1:** **Limited Novelty of Using a vLLM**\n\n> The scientific novelty of using a vLLM for this task is limited, even though the experimental exploration is solid.\n\nDue to space limitations, please refer to Reviewer QkdQ’s comments regarding Originality for further discussion on this issue.\n\n\n## **Weakness 2: Dependence on In-Domain Data**\n\n> The effectiveness of the approach seems heavily dependent on access to in-domain data, which limits its general applicability.\n\nThank you for highlighting this point. To investigate our model’s generalization, we conducted an additional “Odd-One-Out” experiment (training on N-1 datasets, testing on the held-out dataset), evaluted with ExpRate@CDMmailto:ExpRate@CDM %.\n\n\n|Model|CROHME-Avg|CROHME 2023|HME100K|MathWriting|Im2LaTeXv2|\n|-|-|-|-|-|-|\n|CoMER⁺(w/oCROHME)|45.99|58.28|39.49|23.23||\n|CoMER⁺|52.29|59.91|44.63|28.45||\n|Uni-MuMER⁺*(w/oCROHME)*|*78.22*|72.19|74.47|68.08|89.18|\n|Uni-MuMER⁺*(w/oC-2023)*|82.31|*71.66*|74.68|68.85|89.77|\n|Uni-MuMER⁺*(w/oHME100K)*|82.30|72.01|*52.38*|68.94|91.19|\n|Uni-MuMER⁺*(w/oMathWrt)*|79.58|73.51|**74.29**|*32.55*|**93.31**|\n|Uni-MuMER⁺*(w/oIm2LTX)*|**83.20**|73.33|72.40|68.93|*70.06*|\n|Uni-MuMER⁺|82.89|**74.42**|72.31|**69.11**|88.99|\n\nThe odd-one-out evaluation confirms that our unified model can **generalize to an unseen dataset reasonably well**. When the unseen domain is very different, performance drops more markedly, which highlights an area for improvement. HME100K for **real-life, low-quality** images, MathWriting for **densely structured expression**, and Im2LaTeXv2 for **printed expression images**, each of which represents a markedly different and challenging domain.\n\nCrucially, after introducing even modest amounts of domain-relevant data, **our unified approach easily achieves top-tier performance.** Thus, our results indicate that, compared to existing specialized models and general VLM baselines, our approach provides superior generalization with **minimal domain-specific data requirements**, highlighting a clear advantage and direction for future improvement.\n\nWe will include these findings in the paper, discuss the implications for generalization and acknowledge the limitations.\n\n## **Weakness 3 : Computational Cost of Error-Driven Learning (EDL) with Multiple Models**\n\nThank you for raising this important point. To balance effectiveness and computational efficiency, we chose a modest value (k=3) for the k-fold setup, considering both data quality and training costs. While this does increase computational cost, we believe it is justified because:\n\n1. **Error-Driven Learning** provides a general, reusable strategy enabling continuous model improvement.\n2. The resulting **error corpus**, which will be open-sourced, contains inherently valuable, challenging examples tailored specifically to our strongest current model (Uni-MuMER), offering lasting benefits for future HMER research.\n\n## **Weekness 4 : Clarity of Future Work on Self-Correction**\n\n> The future research direction on self-correction lacks clarity, especially regarding handling of context-dependent expressions.\n\nThank you for your interest in our proposed future direction on self-correction.\n\nUni-MuMER introduces a novel perspective on enabling **self-reflective behavior** for mathematical expression recognition:\n\n1. Uni-MuMER brings a **Tree-aware Chain-of-Thought** mechanism and basic error-correction capabilities. Combining these aspects can naturally facilitate self-reflection behaviors, similar in spirit to **Deepseek-R1**.\n\n2. Specifically, we can obtain data with error correction behavior through synthesis or manual annotation to train the model to acquire similar capabilities, and then enhance it through RL. (Similar to the process of Deepseek-R1)\n\n3. To illustrate how our envisioned method handles context-dependent expressions, consider the example expression `\\lim _ { n \\to \\infty } ( \\sum _ { k = 1 } ^ n \\frac 1 { k ^ 2 } )`: $lim _ { n \\to \\infty } ( \\sum _ { k = 1 } ^ n \\frac { 1 } { k ^ 2 } )$\n\n- Without Tree-CoT and error-correction, the model might mistakenly decode the final token as “2”.\n\n- With Tree-CoT and self-correction capabilities, upon initially decoding the token as “2”, the model can reflect and identify an unclosed left parenthesis earlier in the expression. By systematically reviewing the tree structure (as enabled by Tree-CoT), it can correct this token, determining that a right parenthesis is contextually more appropriate than “2”.\n\nThis example highlights the practical benefit of our envisioned approach for handling context-dependent expressions, and we will include a clearer description of this idea in the revised manuscript.\n\n\n\n## Question 1 **Base Model Selection (Why Qwen-2.5-VL-3B?)**\n\n> Why choosing Qwen-3B for fine-tuning, considering that the 7B version already achieves better than Uni-MuMER?\n\nThank you for raising this question. We chose **Qwen2.5-VL-3B** to strike a balance between **performance and computational cost**, ensuring our approach remains suitable for real-world deployment.\n\nImportantly, we also **trained a 7B version** of Uni-MuMER for comparison and observed only **marginal improvements:**\n\n\n|Model|CROHME-Average|CROHME 2023 test|HME100K|Mathwriting test|Im2Latexv2 test|\n|--|--|--|--|--|--|\n|Uni-MuMER†|80.45|**46.35**|72.72|51.41|88.03|\n|Uni-MuMER-7B†|**80.55**|46.04|**73.25**|**53.33**|**91.32**|\n\n\n Additionally, as noted in Table 1, **even the larger zero-shot models** (7B, 72B, Gemini2.5-flash) significantly **underperform compared to specialized HMER methods**, while our fine-tuned 3B model surpasses them by **over 20%**. This strongly indicates that effective domain-specific fine-tuning matters more for HMER than the base model size, further justifying our choice of the smaller, more efficient **3B** version.\n\n\n## Question 2 **Inference Time in Table 8**\n\n> In Table 8, inference time comparisons suggest that smaller models are slower than Qwen. How is this possible?\n\nThe higher inference speed of Qwen-based Uni-MuMER is due to the **vLLM inference engine**, which provides **highly optimized batched decoding and caching mechanisms**, including optimized caching mechanisms such as PagedAttention for efficient KV caching. With access to large-memory GPUs (e.g., A800 80GB), vLLM can cache more key-value pairs, further accelerating inference. In contrast, smaller models are often compute-bound rather than memory-bound.\n\nAdditionally, previous specialized models typically rely on **beam search**, which is inherently slower than the **nearly greedy decoding** used in VLMs. vLLM performs an **optimized prefill stage**, amortizing the initial encoding cost across multiple subsequent decoding steps and boosting per-sample inference speed in batched processing.\n\nMany specialized models employ **custom architectural modifications** (e.g., tree decoders), which **prevent them from leveraging these general-purpose inference optimizations** available in developer groups.\n\nOur approach **retains standard VLMs architecture** without structural changes **highlighting its key strength.**\n\n## Question 3 **Open-Sourcing**\n\n> Will the source code and trained models be made publicly available?\n\nYes, we will make the code, trained models, training data and reproducible training scripts publicly available.\n\n\n\n## Question 4 **k-Fold Value in Error-Driven Learning**\n\n\nThank you for pointing out this oversight. We employed a **3-fold** on the training set to generate corpus, which was **chosen as a balance** **between generating** error corpus and **keeping computation** feasible.\n\nWe will clarify this detail in Section 3.3 in the final version.\n\n\n\n## Question 5 **Clarification on Table 3 (Ablation Results)**\n\n> In Table 3, the dataset version is not specified. Moreover, I could not locate the reported result for CoT+EDL+SC (73.29) in other tables, particularly Table 1. Could you clarify this point?\n\n\n\nThe result of 73.29% ExpRate corresponds to the Uni-MuMER (w/o †) row in Table 1, under the “Average Exp” column (**second-to-last row, second-to-last column**). \n\nThis version of **Uni-MuMER** includes all three proposed tasks (Tree-CoT, EDL, and SC) and **is aligned with the “Vanilla+CoT+EDL+SC” setting in Table 3**. To **ensure a fair comparison with previous work**, this model was fine-tuned only on the CROHME training set, without any external data. \n\n\n\n## Question 6 **Generalization vs. In-Domain Performance**\n\n> The experiments in Section 5.2 suggest that the specialization of the Qwen model is only effective when fine-tuning is performed with in-domain data...\n\n\n\nThank you for your comment. We appreciate the opportunity to clarify.\n\nOur intended message was that fine-tuning effectiveness indeed varies based on the closeness of the training and test datasets:\n\n- **In-domain fine-tuning consistently achieves the highest performance**. For example, printed-expression data fine-tuning significantly outperforms using handwritten data alone (90% vs. 70% ExpRate@CDM).\n\n- **Partial out-of-domain data can still improve performance**, as illustrated in Figure 6, where printed data enhanced some handwritten datasets.\n\n- **Mismatched datasets may degrade performance** due to substantial domain gaps, such as training on clean expressions and testing on low-quality real-life images (e.g., HME100K), may degrade performance due to substantial domain gaps.\n\nAdditionally, we are actively **exploring the integration of general SFT datasets** (e.g., LLaVA’s One Vision data) **with our HMER-specific training data to assess broader generalization performance.** Due to computational constraints, this evaluation is ongoing, and we plan to discuss these findings further in future revisions.\n\n## Limitaions\n\nDue to space limitations, detailed explanations of future work and limitations will be provided later."}, {"review_id": "Tyt4wLryxO", "reviewer": "Reviewer_QkdQ", "summary": "The paper proposes a multi-task training setup for handwritten math expression recognition which involves basic HMER, syntax tree parsing, symbol counting, and error identification in the OCR output from other models. Authors finetune QWEN2.5VL-3B on these tasks on a range on datasets (CROHME14/16/19, NME, HME100k, MathWriting) and show state-of-the-art results, particularly highlighting CROHME datasets. Authors perform an ablation study to show that the effect of multi-task training helps performance.", "questions": "You mention that the performance does rely heavily on having the target dataset in the training mixture, but that this does depend on the variety in the data. One thing that could strengthen the work could be showing the generalization ability, ex. by looking how does the model perform in an odd-one-out setting, where it is trained on N-1 datasets and is evaluated on the Nth - do you have some observations around this?", "limitations": "--", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 1, "strengths_and_weaknesses": "Quality & Clarity: The paper is well-written and easy to understand. The appendix is quite comprehensive with additional results, explanations, and code. However, the comparison section lacks comparisons with works like UNIMer and ICAL (Authors do mention the see works in the related work section, and, IIUC, the numbers for those will be lower for CROHME datasets than the ones reported by the authors, but potentially higher than other baselines reported by them, but it would nevertheless strengthen the paper to include them in the table 1). Furthermore, authors only report results on MNE, HME100k, and Mathwriting in the appendix, and on a limited subset of papers - it would be beneficial to see a wider comparison, whenever available, and in the main manuscript.\n\nOriginality: The core of the work is a multi-task training setup on a number of datasets, with the observation that multi-task training helps, and having the training dataset corresponding to the test is beneficial for performance (l. 341) - the novelty is quite limited, with the exception of the specific task setup.\n\nSignificance: The model does obtain state-of-the-art results and the multi-task training setup is useful for practitioners. This is also complemented by the code in the supplementary material related to data management, training and inference, fostering reproducibility.", "was_revised": true, "final_justification": "I thank the authors for their response, specifically the odd-one-out-experiment and the additional results with UNIMerNET and ICAL, which strengthen the paper. I do not believe that the observation that \"more data yields better results\" is unique or surprising in the HMER case, but given other points addressed by the authors, I am increasing my rating to accept.", "rebuttal": "# Response to Reviewer QkdQ\n\nWe thank the reviewer for the thoughtful and constructive feedback. We address each concern and question in detail below, aiming to clarify our design choices and highlight the contributions of our work.\n\n## Weakness1 **Comparisons with UNIMer and ICAL in Table 1**\n\n> However, the comparison section lacks comparisons with works like UNIMer and ICAL (Authors do mention the see works in the related work section, and, IIUC, the numbers for those will be lower for CROHME datasets than the ones reported by the authors, but potentially higher than other baselines reported by them, but it would nevertheless strengthen the paper to include them in the table 1).\n\nWe appreciate the reviewer highlighting these missing comparisons. **Due to page constraints**, we were initially unable to include these results. In the revised manuscript, **we will incorporate comparisons with UnimerNet and ICAL in the updated Table 1** to strengthen our evaluation:\n\n|Model|CROHME 2014 ExpRate|CROHME 2014 ExpRate@CDM|CROHME 2016 ExpRate|CROHME 2016 ExpRate@CDM|CROHME 2019 ExpRate|CROHME 2019 ExpRate@CDM|CROHME Average ExpRate|CROHME Average ExpRate@CDM|\n|--|--|--|--|--|--|--|--|--|\n|ICAL|60.63|-|58.79|-|60.51|-|59.98|-|\n|UnimerNet†|67.4|-|68.4|-|65.4|-|67.07|-|\n|Vanilla|72.62|75.66|64.26|67.83|69.14|71.98|68.67|71.82|\n|Uni-MuMER|75.36|79.11|70.79|75.24|73.73|77.23|73.29|77.19|\n|Uni-MuMER†|**82.05**|**85.09**|**77.94**|**81.08**|**79.23**|**82.40**|**79.74**|**82.86**|\n\n\n## Weakness 2 **Limited Dataset Evaluation Reporting in the Main Text**\n\n> Furthermore, authors only report results on MNE, HME100k, and Mathwriting in the appendix, and on a limited subset of papers - it would be beneficial to see a wider comparison, whenever available, and in the main manuscript.\n\nWe appreciate the reviewer’s suggestion. Due to **strict page limits** and the fact that **some test sets** (e.g., MNE, MathWriting) **are not yet widely tested in prior relative work**, we included these results in the appendix. **We agree** that presenting a broader set of comparisons in the main text will strengthen the paper, and **we will move key results from these datasets into the main paper** in the camera-ready version.\n\n## Originality: **Novelty is Limited**\n\n> The core of the work is a multi-task training setup on a number of datasets, with the observation that multi-task training helps, and having the training dataset corresponding to the test is beneficial for performance (l. 341) - the novelty is quite limited, with the exception of the specific task setup.\n\nWe appreciate the reviewer’s comment regarding novelty.\n\n### **(1) Novelty from Paradigm Shift**:\n\nFrom past viewpoint with smaller models, framework innovation was essential; But for the **current generation** of **large-scale VLMs**, we argue that **data-centric and unified multi-task fine-tuning itself constitutes a significant innovation.**\n\nAlthough each innovation individually offers distinct advantages, as discussed in the paper (line 41), **prior architectural innovations are fragmented and difficult to integrate, which limits the development of HMER domain.**\n\n\nPrevious works primarily experimented with smaller models. Their effectiveness when applied to modern large-scale pretrained VLMs has not been validated, and their improvements may not translate to current VLMs\n\n### **(2) Empirical Novelty (Data Utilization Efficiency)**:\n\nOur method enhances **data utilization efficiency**. Such advantages are broadly applicable to other domains facing challenges with limited or diverse training resources.\n\n### **(3) Generalization beyond Task-Specific Datasets**:\n\nMoreover, the reviewer **correctly observes that more data generally improves performance**. Yet, we stress that this outcome is **non-trivial and unique** in the context of HMER:\n\n- traditional lightweight models (e.g., CoMER) have **limited benefits or even performance degradation**. As shown explicitly in our ablation (Appendix C.1 **Effects of Data Diversity Compared with Specialized HMER Model**, Table 7), incremental dataset scaling substantially benefits our model (Uni-MuMER), whereas the performance of traditional specialized models is unstable or declines:\n  \n    |Model|CROHME Train|+ CROHME-23|+ HME100K|+ MathWriting|+ Im2Latexv2|\n  |-|--|--|--|--|--|\n  |CoMER|58.14|60.08|63.10|52.17|52.29|\n  |Uni-MuMER|**75.36**|**76.27**|**78.01**|**83.20**|**82.86**|\n\n- Additionally, prior studies typically experimented on limited datasets. We found that the performance of these lightweight models often decreases significantly (as demonstrated in Appendix C.1 Table 7).\n\n## Questions: **Cross-Dataset Generalization – Odd-One-Out Experiment**\n\n> You mention that the performance does rely heavily on having the target dataset in the training mixture, but that this does depend on the variety in the data. One thing that could strengthen the work could be showing the generalization ability, ex. by looking how does the model **perform in an odd-one-out setting**, where it is trained on N-1 datasets and is evaluated on the Nth - do you have some observations around this?\n\nWe thank the reviewer for this insightful suggestion. As suggested, **we conducted an “odd-one-out” generalization experiment**, training Uni-MuMER on N-1 datasets and evaluating it on the held-out N-th dataset.\n\nThe results below present the expression recognition accuracy (ExpRate@CDM%) in each scenario. For comparison, we include the baseline CoMER trained on all datasets:\n\n\n| Model                               | CROHME-Avg | CROHME 2023 | HME100K | MathWriting | Im2LaTeXv2 |\n|-------------------------------------|------------|-------------|---------|-------------|------------|\n| Doubao-1.5-pro                      | 65.77      | 58.18       | 55.08   | 26.34       | 27.66      |\n| CoMER⁺ (w/o CROHME)                 | 45.99      | 58.28       | 39.49   | 23.23       |            |\n| CoMER⁺                              | 52.29      | 59.91       | 44.63   | 28.45       |            |\n| Uni-MuMER⁺ *(w/o CROHME)*           | *78.22*      | 72.19       | 74.47   | 68.08       | 89.18      |\n| Uni-MuMER⁺ *(w/o CROHME-2023)*      | 82.31      | *71.66*      | 74.68   | 68.85       | 89.77      |\n| Uni-MuMER⁺ *(w/o HME100K)*          | 82.30      | 72.01       | *52.38*   | 68.94       | 91.19      |\n| Uni-MuMER⁺ *(w/o MathWriting)*      | 79.58      | 73.51       | **74.29** | *32.55*       | **93.31**  |\n| Uni-MuMER⁺ *(w/o Im2LaTeXv2)*       | **83.20**  | 73.33       | 72.40   | 68.93       | *70.06*      |\n| Uni-MuMER⁺                          | 82.89      | **74.42**   | 72.31   | **69.11**   | 88.99      |\n\n\n**Observations:**\n\nOur results clearly demonstrate the strong generalization capability of Uni-MuMER compared to the baseline model (CoMER):\n\n- **Even without training on the target domain**, Uni-MuMER consistently achieves substantially higher accuracy than the baseline CoMER† trained on **all datasets**. For example, Uni-MuMER without MathWriting achieves 32.55% accuracy on MathWriting, significantly surpassing CoMER’s 26.34% accuracy (trained on MathWriting explicitly).\n\nThis finding confirms that Uni-MuMER **effectively leverages shared knowledge across domains and generalizes well to unseen datasets**, validating its strong multi-task learning capabilities.\n\nHowever, we also notice that the degree of generalization depends on data distribution similarity:\n\n- For datasets closely related to others (e.g., CROHME and CROHME2023), accuracy drops are minimal (generally within 1-3%), further indicating good cross-domain transfer.\n- For datasets with distinct characteristics (e.g., HME100K for **real-life, low-quality**, MathWriting for **densely structured expression**, and Im2LaTeXv2 for **printed expression**), accuracy drops are more pronounced. Nevertheless, even in these challenging cases, Uni-MuMER outperforms baseline methods significantly.\n\nIn summary, our “odd-one-out” experiments demonstrate that Uni-MuMER exhibits remarkable generalization capability and robustness compared to previous methods, **validating the benefits of the proposed multi-domain modeling approach**.\n\nWe will clearly highlight these findings in the revised manuscript and further discuss the implications and limitations of our approach."}, {"review_id": "2pLCWWcuIX", "reviewer": "Reviewer_uB1H", "summary": "The paper addresses the challenge of Handwritten Mathematical Expression Recognition (HMER) by introducing Uni-MuMER, a framework that fine-tunes the Qwen2.5-VL-3B vision-language model (VLM) with three data-driven tasks: Tree-CoT, Error-Driven Learning (EDL), Symbol Counting (SC). The experiment result shows the Uni-MuMER achieves significant improvement and establishes a new SOTA on the HMER task.", "questions": "Have you done some error analysis? Which types of symbols remain most challenging for Uni-MuMER?\nAre there more training tasks that can be used to further improve the model？", "limitations": "yes. The authors have adequately addressed the limitations.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "【strengthes】\n1. Unified and efficient framework: Leveraging existing pre-trained VLMs is efficient and convenient to unify multi-tasks to enhance the HMER.\n2. Strong performance: the three training tasks work well and largely improve the VLM's HMER capability.\n【weaknesses】\nFor the easy, clear, reasonable, and effective method proposed in the paper, there are not many weaknesses. The dominating weakness is the inference speed and the computational cost increment introduced by the general-purpose large language model when compared with the previous small and specific models.", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer uB1H\n\nWe thank the reviewer for the positive assessment of our work. We are glad that the reviewer found our unified and efficient framework, as well as the strong performance improvements of Uni-MuMER. We address each weakness, question, and point raised by the reviewer in detail below.\n\n## Weakness: Inference Speed and Computational Cost\n\n> For the easy, clear, reasonable, and effective method proposed in the paper, there are not many weaknesses. The dominating weakness is the inference speed and the computational cost increment introduced by the general-purpose large language model when compared with the previous small and specific models.\n\nWe appreciate the reviewer’s valid concern about inference speed and computational cost. Indeed, **using a general-purpose VLM incurs higher computational overhead** compared to prior lightweight models.\n\nOur approach, which leverages the optimized **vLLM inference engine**, significantly mitigates this issue. By using some GPU (like A800 80GB), our method achieves **8.29× faster inference speed** compared to previous smaller, specialized models, as mention in Appendix C.2 Inference Efficiency Table 8.\n\nBy **retaining a standard VLM architecture without structural modifications**, we fully exploit community-optimized inference framework, which futher highlighting a key strength of our work.\n\nBesides, our empirical results demonstrate that this cost brings significant **performance and speed improvement.**\n\n## Question1: Error Analysis\n\n> Have you done some error analysis?\n\nThank you for the suggestion. To deepen our analysis of error types in mathematical expression recognition, we have introduced definitions for hallucination and omission at the token level:\n\n- Hallucination Token Rate (**HTR**): the rate of tokens predicted that do not appear in the ground truth.\n\n  - _HTR_num:_ hallucinated numeric digits (0–9)\n  - _HTR_var:_ hallucinated alphabetic variables (a–z, A–Z)\n  - _HTR_delimiter:_ hallucinated structural delimiters (e.g., [], {}, ())\n- Missing Token Rate (**MTR**): the rate of tokens ground truth that do not appear in the predicted to quantify omitted tokens\n\nWe evaluated these metrics across the CROHME 2014, 2016, and 2019 test sets, ensuring that no external data was used during training. The results are summarized below:\n\n| Model                 | ExpRate   | @CDM      | HTR      | MTR      | HTR_num  | HTR_var  | HTR_delim |\n| --------------------- | --------- | --------- | -------- | -------- | -------- | -------- | --------- |\n| CoMER (SOTA 2022)     | 59.18     | 62.47     | 2.55     | 3.91     | 0.37     | 0.33     | 1.18      |\n| TAMER (SOTA 2024)     | 60.29     | 63.18     | 2.17     | 4.02     | 0.29     | 0.29     | 1.05      |\n| Vanilla (baseline)    | 68.64     | 71.74     | 4.08     | 2.42     | 1.08     | 0.29     | 1.68      |\n| Vanilla+Tree-CoT      | **70.85** | 73.71     | **1.67** | **1.56** | 0.14     | **0.21** | 0.85      |\n| Vanilla+EDL           | 70.30     | 73.76     | 2.58     | 1.94     | 1.09     | 0.22     | **0.83**  |\n| Vanilla+CAN           | 69.86     | **74.76** | 2.15     | 2.06     | **0.23** | 0.27     | 1.08      |\n| Uni-MuMER (w/3 tasks) | 73.29     | **77.19** | **1.39** | **1.18** | **0.12** | **0.15** | **0.6**   |\n\n\n\n\n**Uni-MuMER significantly reduces hallucinated tokens and omissions** compared to both the vanilla baseline and prior SOTA specialized models.\n\n- **Tree-CoT** improves structural reasoning and reduces structural hallucinations\n- **EDL** corrects frequent symbol-level mistakes,\n- **SC** enforces token consistency, reducing omissions and hallucinated insertions.\n\nWe will incorporate this analysis in Section 5 of the final paper to highlight Uni-MuMER’s robustness.\n\n## Question2: Which types of Symbols are Most Challenging?\n\n> Which types of symbols remain most challenging for Uni-MuMER?\n\nThank you for raising this insightful question, as **it pertains directly to inherent challenges within the HMER domain and future work** (response Question 3).\n\nWe have conducted a detailed analysis of the error cases from the CROHME datasets. Our observations indicate that the majority of **remaining errors can be attributed primarily to two factors**:\n\n1. **Hallucination**: As discussed previously (Response to Question 1), **hallucination is a known issue inherent to VLMs**. While **Uni-MuMER substantially alleviates this issue**—as demonstrated by the significantly reduced Hallucination Token Rate (HTR) shown previously—it is challenging to entirely eliminate due to inherent ambiguities in handwritten data.\n2. **Similar-character (Token-level) Errors**: Another primary error source stems from visually similar or ambiguous characters. Without explicit context, distinguishing certain symbol pairs—such as:\n\n   - Lowercase \"s\" and uppercase \"S\" (`s_a` vs. `S_a`),\n   - Digit \"0\" and uppercase letter \"O\",\n   - Lowercase letter \"a\" and Greek letter (`\\alpha`),\n   - Digit \"1\" and lowercase \"l\",\n     is particularly challenging. These symbol pairs often share substantial visual similarity, making isolated recognition inherently ambiguous even when trained on extensive datasets containing diverse handwriting styles.\n\nAddressing these challenges transcends the traditional exact-match metrics (e.g., ExpRate) and relies on more sophisticated context-aware evaluation methods beyond token-level accuracy.\n\n## Question3: Additional Training Tasks for Improvement\n\n> Are there more training tasks that can be used to further improve the model？\n\nLooking ahead, we envision that improvements in the model’s integration of broader world knowledge could significantly alleviate issues mentioned in Response Question 2. As the model gains more extensive and nuanced reasoning capabilities, it can better infer plausible characters within ambiguous contexts. Moreover, as mentioned in Section 6 (Discussion), promising directions include:\n\n- **R1-like Reflective Self-Correction:**\n  Allowing the model to iteratively reflect on and correct its predictions, thereby reducing errors due to ambiguous or visually similar symbols.\n- **O3-like Functional Call Visual Reasoning:**\n  Enabling explicit visual reasoning steps, such as zooming into particular regions of an expression image to resolve fine-grained details and mitigate visual ambiguity.\n- **Hierarchical Recognition with Integrated Reasoning:**\n  Structuring the recognition task hierarchically—first counting symbol, then constructing the Abstract Syntax Tree (AST), and embedding reflective and functional-call reasoning (R1-like and O3-like) within the AST construction process to ensure accuracy and structural consistency.\n\nWe plan to explore these exciting avenues in future research, extending Uni-MuMER’s capabilities while leveraging the strength and flexibility of our existing framework."}, {"review_id": "c5b2QEXQX2", "reviewer": "Reviewer_Yrc6", "summary": "The paper proposes Uni-MuMER, a unified multi-task fine-tuning framework for Handwritten Mathematical Expression Recognition (HMER), leveraging a generalist vision-language model (Qwen2.5-VL-3B). The authors introduce three auxiliary tasks: Tree-Aware Chain-of-Thought (Tree-CoT) for structural reasoning, Error-Driven Learning (EDL) for self-correction, and Symbol Counting (SC) for improving recognition consistency. Experiments on the CROHME and HME100K datasets demonstrate strong improvements over existing models, including both zero-shot and fine-tuned baselines.", "questions": "1) In Table 1, several prior methods were trained solely on the CROHME training set. Could the authors report the performance of Uni-MuMER when fine-tuned only on the CROHME training set, without incorporating external data? This would help establish a fairer comparison under the same training conditions.\n\n2) For the HME100K evaluation, the paper mentions excluding samples containing CJK characters. Could the authors clarify the rationale behind this filtering step, and whether it significantly affects the reported performance or comparability with prior work?\n\n3) “External data” is mentioned in both Table 1 and Table 2. Could the authors clarify which datasets or sources are included in this category?", "limitations": "Compared to specialized models, do large fine-tuned VLMs like Uni-MuMER exhibit hallucination or repetition issues when generating mathematical expressions? This is a common concern with large models and should be discussed in the limitations section.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1) The paper addresses a relevant and long-standing challenge in OCR for mathematical expressions, with a novel use of large-scale vision-language models.\n2) The integration of three auxiliary tasks is well-motivated, and the experimental results show that they complement each other.\n3) Ablation studies and comparisons with both open- and closed-source baselines are fairly comprehensive.\n\nWeaknesses:\n1) The method relies solely on fine-tuning and input reformulation without architectural innovation, which may limit its perceived novelty.\n2) The Tree-CoT serialization and symbol counting formats are hand-designed and lack systematic justification or alternative explorations.\n3) While the integration of multiple tasks into a unified framework is valuable, the individual tasks themselves (e.g., symbol counting and AST-based reasoning) are not new to the HMER domain. Prior works such as CAN and tree-decoder-based models have already explored similar directions.\n4) The Tree-CoT task assumes expressions can be parsed into clean hierarchical trees, which limits its applicability to more complex layouts like matrices, piecewise functions, or multi-line expressions that follow grid-like or block structures rather than tree-like ones.", "was_revised": true, "final_justification": "After the rebuttals, the authors have addressed and clairified most of my concerns.", "rebuttal": "# Response to Reviewer Yrc6\n\nWe thank reviewer for thoughtful and constructive feedback. We address each concern and question in detail below.\n\n## Weakness\n\n1. **Lack of Architectural Innovation**\n\n> The method relies solely on fine-tuning and input reformulation without architectural innovation, potentially limiting perceived novelty.\n\nFrom past viewpoint with smaller models, framework innovation was essential; But for the **current generation** of **large-scale VLMs**, we argue that **data-centric and unified multi-task fine-tuning itself constitutes a significant innovation.**\n\nAlthough each innovation individually offers distinct advantages, as discussed in the paper (line 41), **prior architectural innovations are fragmented and difficult to integrate, which limits the development of HMER domain.**\n\nIn contrast, **our Uni-MuMER leverages VLM input-output to integrate multiple architectural strengths, achieving SOTA performance**. This novel recognition paradigm can further generalize to related tasks and enhance robustness by integrating diverse data sources, providing new insights for applying VLMs in expression recognition tasks.\n\nLooking ahead, our approach extends in two promising directions:\n\n- enhancing general VLM capabilities through richer, domain-specific data resources;\n- integrating reinforcement learning strategies, such as R1-like self-reflective error correction,\n\n2. **Auxiliary Tasks (AST Reasoning and Symbol Counting) Are Not New**\n\n> \"...the tasks are not new to the HMER. Prior works such as CAN and tree-based models have explored similar directions.\"\n\nWe appreciate the reviewer’s insightful comment.\n\n1. Our contribution is not about proposing any single auxiliary task in isolation. Rather, we introduce a **unified modeling paradigm** capable of jointly addressing multiple HMER-related tasks (as detailed in our response to Weakness 1).\n2. Specifically, the proposed Error Detection and Correction tasks represent a **novel direction** for HMER, laying the foundation for self-reflective capabilities in future HMER.\n3. Previous works primarily experimented with smaller models. Their effectiveness when applied to modern large-scale pretrained VLMs has not been validated, and their improvements may not translate to current VLMs\n4. Additionally, prior studies typically experimented on limited datasets. We found that the performance of these lightweight models often decreases significantly (as demonstrated in Appendix C.1 Table 7).\n\nTable 7: Impact of incrementally training data scaling on CROHME evaluaed with ExpRate@CDMmailto:ExpRate@CDM %\n||CROHME |+ CROHME-23|+ HME100K|+ MathWriting|+ Im2Latexv2|\n|-|-|-|-|-|-|\n|CoMER|58.14|60.08|63.10|52.17|52.29|\n|Uni-MuMER|**75.36**|**76.27**|**78.01**|**83.20**|**82.86**|\n\n\n3. **Alternative explorations format**\n\n> The Tree-CoT serialization and symbol counting formats are hand-designed and lack alternative explorations.\n\nFor Tree-CoT serialization formats, including **S-expression**, **JSON AST** and our proposed **AST format**.\n\n- **S-expression** provides minimal structural detail.\n\n  - mu_{pj}\n- **JSON AST** is overly verbose and redundant.\n\n  - [{'type': 'supsub', 'value': {'base': {'type': 'mathord', 'value': '\\\\mu'}, 'sub': {'type': 'ordgroup', 'value': [{'type': 'mathord', 'value': 'p'}, {'type': 'mathord', 'value': 'j'}]}}}]\n- **our chosen AST** format offers a clear and concise hierarchical representation, balancing minimal redundancy with structural clarity.\n\n```\n\\mu\n    p (sub)\n    j (right)\n```\n\nWe evaluated our method across those formats, observing improved performance in all cases, with the our **AST format achieving the best results**\n\n||14-Exp|14-CDM|16-Exp|16-CDM|19-Exp|19-CDM|Avg-Exp|Avg-CDM|\n|-|-|-|-|-|-|-|-|-|\n|Vanilla|72.62|75.66|64.26|67.83|69.14|71.98|68.67|71.82|\n|w/ S-exp|72.21|75.56|66.87|70.97|68.06|71.73|69.05|72.75|\n|w/ Json AST|71.70|75.66|66.70|70.51|70.56|**74.04**|69.65|73.40|\n|w/ Sim AST|72.62|**76.47**|66.96|70.18|69.89|73.06|69.82|73.24|\n|w/ our AST|73.33|75.76|68.00|70.97|71.23|74.40|70.85|73.71|\n\nWe explored another counting format, “Map Counting” with higher redundancy. Experiments confirm our format are better.\n\nFor `x + 1`\n\n**Map Counting Format**\n\n```\n{'x':1,'+':1,'1' 1}\n```\n\n**Our Counting Format**\n\n```\nx: 1,+: 1,1: 1\n```\n\n||14Exp|14@CDM|16Exp|16@CDM|19Exp|19@CDM|AvgExp|Avg@CDM|\n|-|-|-|-|-|-|-|-|-|\n|Vanilla|72.62|75.66|64.26|67.83|69.14|71.98|68.67|71.82|\n|Van+Map|**72.92**|76.67|65.39|70.01|69.89|73.65|69.40|73.44|\n|Van+our|71.91|76.85|67.04|72.28|70.64|75.15|69.86|74.76|\n\n\n\n4. **Applicability of Tree-CoT to Non-Tree Structures:**\n\n> The Tree-CoT task ... which limits its applicability to more complex layouts like matrices, piecewise functions, or multi-line expressions that follow grid-like or block structures rather than tree-like ones.\n\nWe clarify that complex layouts **can indeed be transformed into hierarchical tree structures**, including these multi-line expressions. While earlier HMER datasets, such as CROHME, omitted these formats, recent datasets, such as **Mathwriting include them**. We address this gap by introducing relation tokens (e.g., nextline).\n\nFor instance, a piece wise function:\n\n$$\ny = \\begin{cases}  0 & x < 0 \\\\\\\\  1 & x \\ge 0  \\end{cases} \n$$\n\n```\ny\n= (right)\n\\begin (right)\n    0 (begin_cases)\n    & (right)\n    x (right)\n    < (right)\n    0 (right)\n          1 (nextline)\n          & (right)\n          x (right)\n          \\ge (right)\n          0 (right)\n    \\end (end_cases)\n```\n\n\n## Questions\n\n1. **CROHME-Only Fine-Tuning Results**\n\n> Could you provide Uni-MuMER’s results when fine-tuned only on CROHME (without external data) for fair comparison with prior methods (Table 1)?\n\nIn fact, We stated in Section 4.1 (lines 240), and our model surpassing previous SOTA.\n\n||Score|\n|-|-|\n|SSAN(SOTA'25)|63.43|\n|Uni-MuMER|**73.29**|\n\nThis setup applies consistently across Tables 2, 9–12. Both Vanilla and Uni-MuMER **were trained solely on the respective training sets**. We only employed external data in models explicitly marked with a dagger ($\\dagger$), such as Uni-MuMER$^\\dagger$.\n\n\n2. **Filtering out CJK in HME100K Evaluation**\n\n> For HME100K, Could the authors clarify the rationale behind filtering CJK characters, and whether it significantly affects the reported performance or comparability with prior work?\n\nThank you for raising this point. We excluded samples containing CJK characters primarily for two reasons:\n\n1. Our prompt **specifically targets recognizing mathematical expressions**, not general textual content such as CJK characters. This filtering ensures a clear evaluation of expression recognition capability.\n2. Some CJK-containing samples introduced problematic annotations unrelated to pure expressions (e.g., textual descriptions or numeric lists)\n\n|image_id|caption|\n|-|-|\n|train_63827|=88、5|\n|train_70305|39的因数：1,3,|\n\nWe evaluated the **impact of this filtering and found minimal effect on comparability**.\n\nTo further address your concern, we **retrained Uni-MuMER† with the CJK characters in training set** and compared its performance with w/o CJK version, evaluate on ExpRate (%):\n\n- For instance, the performance of CoMER and TAMER remained nearly identical before and after filtering.\n- **our model still outperforms previous methods**, with the CJK characters same in training set\n\n||w/CJK|w/o CJK|\n|-|-|-|\n|CoMER|68.47|68.70|\n|TAMER|69.50|69.74|\n|Uni-MuMER⁺(w/o CJK)|69.46|72.72|\n|Uni-MuMER⁺(w/CJK)|**71.63**|70.98|\n\nHowever, **a decrease in expression recognition performance when add CJK into training set.**\n\n||CROHME-Avg|\n|-|-|\n|Uni-MuMER⁺(w/CJK)|79.46|\n|Uni-MuMER⁺(w/o CJK)|80.45|\n\n**We will include the above details** in Section 4.2 Main Results in the final version.\n\n\n3. **Clarification of “External Data” Definition**\n\n> “External data” is mentioned in both Table 1 and Table 2. Could the authors clarify which datasets or sources are included in this category?\n\n**Although we mentioned** in Section 4.2 (lines 247), we realize that it was unclear.\n\nAs detailed in Appendix B.1 (Paragraph Data Construction, lines 1007–1014), the “external data” collectively includes CROHME, CROHME 2023, HME100K, MathWriting, and Im2Latexv2, totaling 392K images. These datasets, in combination with three tasks (Tree-CoT, EDL, and SC), form the 1.6M samples.\n\nWe will clarify this in Section 4.2 to avoid confusion, and open-source our dataset.\n\n\n## Limitations\n\n**Hallucination and Repetition Issues**\n\n> Do fine-tuned large VLMs like Uni-MuMER suffer from hallucination compared to specialized models? This limitation should be discussed.\n\nBoth fine-tuned large VLMs and specialized models are probabilistic model, making them possible to  hallucination and repetition issues.\n\nTo quantitatively analyze this, we preliminary adopt two metrics: **Hallucination Token Rate (HTR)** (percentage of predicted tokens not in the ground truth) and **Missing Token Rate (MTR)** (percentage of ground truth tokens not in the pred). Additionally, we categorized hallucinations by token type (numerical(0-9), variable(a-zA-Z).\n\n\nExperimental results (without external data) on the CROHME 2014, 2016, and 2019 datasets clearly demonstrate that Uni-MuMER (HTR=1.18%) **significantly reduces hallucination** compared to both baseline and specialized models.\n\n||ExpRate|HTR|MTR|HTR_num|HTR_var|\n|-|-|-|-|-|-|\n|CoMER (SOTA'22)|59.18|2.55|3.91|0.37|0.33|\n|TAMER (SOTA'24)|60.29|2.17|4.02|0.29|0.29|\n|Vanilla|68.64|4.08|2.42|1.08|0.29|\n|+Tree-CoT|**70.85**|**1.67**|**1.56**|0.14|0.21|\n|+EDL|70.30|2.58|1.94|1.09|0.22|\n|+CAN|69.86|2.15|2.06|**0.23**|0.27|\n|Uni-MuMER |73.29|1.18|1.39|0.12|0.15|\n\n**We will include the above details** in Section 5 Analysis in the final version.\n\nWe sincerely appreciate the reviewer’s constructive suggestions. The additional experiments conducted based on your feedback have strengthened our paper. We hope these improvements provide clear grounds for reconsidering your evaluation and potentially raising your score."}]

Author rebuttal:

Meta-review: (a) This paper introduces Uni-MuMER, a framework that fine-tunes a general-purpose vision-language model (VLM), Qwen-3B, for the specialized task of Handwritten Mathematical Expression Recognition (HMER). The central claim is that a VLM's capabilities for this task can be significantly enhanced through a unified, multi-task fine-tuning approach. The authors propose three novel auxiliary tasks designed to inject domain-specific knowledge: Tree-Aware Chain-of-Thought (Tree-CoT) to improve structural reasoning, Error-Driven Learning (EDL) to enable self-correction capabilities by learning from an error corpus, and Symbol Counting (SC) to enforce recognition consistency. The paper's key finding is that this multi-task fine-tuning paradigm allows the moderately-sized VLM to establish new state-of-the-art performance, surpassing not only larger, generalist zero-shot models but also highly specialized, purpose-built HMER models on a range of standard benchmarks.

(b) The paper's primary strength, as noted by the reviewers, is its systematic and well-motivated approach to adapting a generalist VLM for a complex, specialized domain. The three proposed fine-tuning tasks were consistently praised as being relevant, creative, and adaptable to other fields. The experimental validation is another major strength; the paper demonstrates state-of-the-art performance against a comprehensive set of baselines and includes a carefully designed ablation study that clearly attributes performance gains to the specific tasks. The work is also highly regarded for its clarity, reproducibility, and the practical utility of its multi-task training setup, which provides a strong new baseline for the HMER community.

(c) The main initial weakness identified by reviewers was the perceived limited novelty, as the core idea involves fine-tuning an existing VLM rather than proposing a new model architecture. Some reviewers also raised concerns that the method's effectiveness appeared heavily dependent on access to in-domain training data, potentially limiting its out-of-the-box generalization. Other initial critiques included the potential computational expense of the Error-Driven Learning approach and a lack of clarity in some of the experimental details and future work descriptions, though most of these points were successfully addressed during the rebuttal period.

(d) Recommendation: Accept (Spotlight)
The paper is a clear Accept due to its technical soundness, state-of-the-art results, and an exemplary rebuttal that addressed all major reviewer concerns. It warrants a Spotlight because it represents a compelling case study for a significant paradigm shift in specialized recognition tasks. While many papers apply large models to new tasks, this work stands out for three reasons. First, it successfully argues that in the era of VLMs, innovation is shifting from bespoke architectures to intelligent, data-centric fine-tuning paradigms, and provides a highly effective and generalizable recipe for doing so. Second, the paper demonstrates strong rigor in ablation studies, particularly through the comprehensive "odd-one-out" generalization experiments and detailed error analysis conducted during the rebuttal, setting a high standard for empirical validation. Finally, it delivers a high-impact practical outcome: a unified model that is not only more accurate but also—counter-intuitively—faster at inference than previous smaller, specialized models, providing a powerful and accessible new baseline that will likely drive future research in the HMER field and Handwriting Recognition field in general.
Decision: Accept (spotlight)