Title: Uni-MuMER: Unified Multi-Task Fine-Tuning of Vision-Language Model for Handwritten Mathematical Expression Recognition
Venue: NeurIPS 2025
Accepted: yes
Keywords: Handwritten Mathematical Expression Recognition

Vision-Language Models

Tree-of-Thought Reasoning

Multi-Task Learning

Optical Character Recognition
Abstract: Handwritten Mathematical Expression Recognition (HMER) remains a persistent challenge in Optical Character Recognition (OCR) due to the inherent freedom of symbol layouts and variability in handwriting styles.
Prior methods have faced performance bottlenecks by proposing isolated architectural modifications, making them difficult to integrate coherently into a unified framework.
Meanwhile, recent advances in pretrained vision-language models (VLMs) have demonstrated strong cross-task generalization, offering a promising foundation for developing unified solutions. 
In this paper, we introduce Uni-MuMER, which fully fine-tunes a VLM for the HMER task without modifying its architecture, effectively injecting domain-specific knowledge into a generalist framework. 
Our method integrates three data-driven tasks: Tree-Aware Chain-of-Thought (Tree-CoT) for structured spatial reasoning, Error-Driven Learning (EDL) for reducing confusion among visually similar
characters, and Symbol Counting (SC) for improving recognition consistency in long expressions.
Experiments on the CROHME and HME100K datasets show that Uni-MuMER achieves super state-of-the-art performance, 
outperforming the best lightweight specialized model SSAN by 16.31\% and the top-performing VLM Gemini2.5-flash by 24.42\% under zero-shot setting. 
Our datasets, models, and code are open-sourced at: https://github.com/BFlameSwift/Uni-MuMER


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Uni-MuMER, a unified multi-task fine-tuning framework for Handwritten Mathematical Expression Recognition (HMER), leveraging a generalist vision-language model (Qwen2.5-VL-3B). The authors introduce three auxiliary tasks: Tree-Aware Chain-of-Thought (Tree-CoT) for structural reasoning, Error-Driven Learning (EDL) for self-correction, and Symbol Counting (SC) for improving recognition consistency. Experiments on the CROHME and HME100K datasets demonstrate strong improvements over existing models, including both zero-shot and fine-tuned baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1) The paper addresses a relevant and long-standing challenge in OCR for mathematical expressions, with a novel use of large-scale vision-language models.\n2) The integration of three auxiliary tasks is well-motivated, and the experimental results show that they complement each other.\n3) Ablation studies and comparisons with both open- and closed-source baselines are fairly comprehensive.\n\nWeaknesses:\n1) The method relies solely on fine-tuning and input reformulation without architectural innovation, which may limit its perceived novelty.\n2) The Tree-CoT serialization and symbol counting formats are hand-designed and lack systematic justification or alternative explorations.\n3) While the integration of multiple tasks into a unified framework is valuable, the individual tasks themselves (e.g., symbol counting and AST-based reasoning) are not new to the HMER domain. Prior works such as CAN and tree-decoder-based models have already explored similar directions.\n4) The Tree-CoT task assumes expressions can be parsed into clean hierarchical trees, which limits its applicability to more complex layouts like matrices, piecewise functions, or multi-line expressions that follow grid-like or block structures rather than tree-like ones.\n\nQUESTIONS:\n1) In Table 1, several prior methods were trained solely on the CROHME training set. Could the authors report the performance of Uni-MuMER when fine-tuned only on the CROHME training set, without incorporating external data? This would help establish a fairer comparison under the same training conditions.\n\n2) For the HME100K evaluation, the paper mentions excluding samples containing CJK characters. Could the authors clarify the rationale behind this filtering step, and whether it significantly affects the reported performance or comparability with prior work?\n\n3) “External data” is mentioned in both Table 1 and Table 2. Could the authors clarify which datasets or sources are included in this category?\n\nLIMITATIONS:\nCompared to specialized models, do large fine-tuned VLMs like Uni-MuMER exhibit hallucination or repetition issues when generating mathematical expressions? This is a common concern with large models and should be discussed in the limitations section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

comments: SUMMARY:
The paper addresses the challenge of Handwritten Mathematical Expression Recognition (HMER) by introducing Uni-MuMER, a framework that fine-tunes the Qwen2.5-VL-3B vision-language model (VLM) with three data-driven tasks: Tree-CoT, Error-Driven Learning (EDL), Symbol Counting (SC). The experiment result shows the Uni-MuMER achieves significant improvement and establishes a new SOTA on the HMER task.

STRENGTHS AND WEAKNESSES:
【strengthes】
1. Unified and efficient framework: Leveraging existing pre-trained VLMs is efficient and convenient to unify multi-tasks to enhance the HMER.
2. Strong performance: the three training tasks work well and largely improve the VLM's HMER capability.
【weaknesses】
For the easy, clear, reasonable, and effective method proposed in the paper, there are not many weaknesses. The dominating weakness is the inference speed and the computational cost increment introduced by the general-purpose large language model when compared with the previous small and specific models.

QUESTIONS:
Have you done some error analysis? Which types of symbols remain most challenging for Uni-MuMER?
Are there more training tasks that can be used to further improve the model？

LIMITATIONS:
yes. The authors have adequately addressed the limitations.
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a multi-task training setup for handwritten math expression recognition which involves basic HMER, syntax tree parsing, symbol counting, and error identification in the OCR output from other models. Authors finetune QWEN2.5VL-3B on these tasks on a range on datasets (CROHME14/16/19, NME, HME100k, MathWriting) and show state-of-the-art results, particularly highlighting CROHME datasets. Authors perform an ablation study to show that the effect of multi-task training helps performance.\n\nSTRENGTHS AND WEAKNESSES:\nQuality & Clarity: The paper is well-written and easy to understand. The appendix is quite comprehensive with additional results, explanations, and code. However, the comparison section lacks comparisons with works like UNIMer and ICAL (Authors do mention the see works in the related work section, and, IIUC, the numbers for those will be lower for CROHME datasets than the ones reported by the authors, but potentially higher than other baselines reported by them, but it would nevertheless strengthen the paper to include them in the table 1). Furthermore, authors only report results on MNE, HME100k, and Mathwriting in the appendix, and on a limited subset of papers - it would be beneficial to see a wider comparison, whenever available, and in the main manuscript.\n\nOriginality: The core of the work is a multi-task training setup on a number of datasets, with the observation that multi-task training helps, and having the training dataset corresponding to the test is beneficial for performance (l. 341) - the novelty is quite limited, with the exception of the specific task setup.\n\nSignificance: The model does obtain state-of-the-art results and the multi-task training setup is useful for practitioners. This is also complemented by the code in the supplementary material related to data management, training and inference, fostering reproducibility.\n\nQUESTIONS:\nYou mention that the performance does rely heavily on having the target dataset in the training mixture, but that this does depend on the variety in the data. One thing that could strengthen the work could be showing the generalization ability, ex. by looking how does the model perform in an odd-one-out setting, where it is trained on N-1 datasets and is evaluated on the Nth - do you have some observations around this?\n\nLIMITATIONS:\n--", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 1, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the use of a general-purpose large language model (vLLM), specifically Qwen, fine-tuned for the task of handwritten mathematical expression recognition. The authors introduce three targeted fine-tuning tasks designed to inject task-specific knowledge into the model: (1) generation of abstract syntax trees (ASTs), (2) self-error detection and correction, and (3) symbol counting.\n\nThe proposed approach is evaluated against both state-of-the-art specialized models and larger generalist models on several databases. Experimental results show that the fine-tuned Qwen model outperforms these baselines across several benchmarks. The paper also presents an ablation study assessing the individual contribution of each fine-tuning task as well as an analysis of the model's out-of-domain generalization capabilities.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n- First systematic evaluation of fine-tuning a generalist vLLM for handwritten mathematical expression recognition.\n- Well-motivated fine-tuning tasks (AST generation, self-correction, symbol counting) that are relevant and can be adapted to other domains.\n- The model outperforms both specialized state-of-the-art models and larger generalist models.\n- Ablation study (Section 5) is carefully designed and informative, clearly linking fine-tuning tasks to specific recognition improvements.\n\n## Weaknesses:\n- The scientific novelty of using a vLLM for this task is limited, even though the experimental exploration is solid.\n- The effectiveness of the approach seems heavily dependent on access to in-domain data, which limits its general applicability.\n- The Error-Driven Learning approach requires fine-tuning multiple models, making it potentially expensive to apply.\n- The future research direction on self-correction lacks clarity, especially regarding handling of context-dependent expressions.\n\nQUESTIONS:\n- Why choosing Qwen-3B for fine-tuning, considering that the 7B version already achieves significantly better performance even without fine-tuning?\n- In Table 8, inference time comparisons suggest that smaller specialized models are slower than Qwen. How is this possible? \n- Will the source code and trained models be made publicly available?\n- What value of k was used in the k-fold Error Corpus Generation process?\n- In Table 3, the dataset version is not specified. Moreover, I could not locate the reported result for CoT+EDL+SC (73.29) in other tables, particularly Table 1. Could you clarify this point?\n- The experiments in Section 5.2 suggest that the specialization of the Qwen model is only effective when fine-tuning is performed with in-domain data. Simple task-specific fine-tuning (for mathematical expression recognition) appears insufficient unless the fine-tuning data is closely aligned with the test set. This observation seems to partially contradict the goal of the proposed specialization tasks, which aim to introduce general task knowledge into the model, independent of the dataset. Could the authors clarify their position on this point and discuss whether in-domain fine-tuning remains, in practice, the most effective way to improve model performance?\n- The proposed future direction of Self-Correction for Expression Recognition remains somewhat unclear. It is difficult to see how \"colloquial expressions,\" such as the example provided, could be effectively integrated, given that they depend on the specific expression being recognized. Could the authors provide more concrete examples or clarify how this type of contextual, expression-dependent correction could be implemented?\n\nLIMITATIONS:\nThe authors have addressed some of the limitations of their approach, notably by discussing how their model could be evaluated on other tasks, such as OCR. However, the paper does not adequately address the potential implications of using a significantly larger generalist language model for a relatively narrow, well-defined task such as handwritten mathematical expression recognition. The increased computational resources and environmental costs associated with large-scale vLLMs could limit the practicality of the approach compared to smaller, task-specific models. A more explicit discussion of these issues would improve the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}