Language Model is Suitable for Correction of Handwritten Mathematical Expressions Recognition
Authors: Zui Chen
Jiaqi Han
Chaofan Yang
Yi Zhou
Venue: EMNLP 2023 Main (2023)
Keywords: LaTeX mathematical expression
handwritten mathematical expression recognition
language model
LaTeX language property
TL;DR: Study language characteristics of LaTeX mathematical expressions, highlight LM is suitable for correction of HMER and propose an architecture to validate

Abstract: Handwritten mathematical expression recognition (HMER) is a multidisciplinary task that generates LaTeX sequences from images. Existing approaches, employing tree decoders within attention-based encoder-decoder architectures, aim to capture the hierarchical tree structure, but are limited by CFGs and pre-generated triplet data, hindering expandability and neglecting visual ambiguity challenges. This article investigates the distinctive language characteristics of LaTeX mathematical expressions, revealing two key observations: 1) the presence of explicit structural symbols, and 2) the treatment of symbols, particularly letters, as minimal units with context-dependent semantics, representing variables or constants. Rooted in these properties, we propose that language models have the potential to synchronously and complementarily provide both structural and semantic information, making them suitable for correction of HMER. To validate our proposition, we propose an architecture called Recognize and Language Fusion Network (RLFN), which integrates recognition and language features to output corrected sequences while jointly optimizing with a string decoder recognition model. Experiments show that RLFN outperforms existing state-of-the-art methods on the CROHME 2014/2016/2019 datasets.

Language Model is Suitable for Correction of Handwritten Mathematical Expressions Recognition
Zui Chen 1.2 Jiaqi Han 1.2 Chaofan Yang', Yi Zhou 'Shanghaitech University,
Shanghai Innovation Center for Processor Technologies University of Science and Technology of China 1 "[chenzui, hanjq2022, yangchf tech. edu. cn 8yi yyi_zhuu@ustc. edu. cn
Abstract
inherent ambiguities in handwritten characters and the complexity of mathematical formulas. These hurdles underscore the pivotal role that NLP could play in enhancing the robustness of current visual models grappling with these issues.
Handwritten mathematical expression recogni- tion (HMER) is multidisciplinary task that generates LaTeX sequences from images. Ex- isting approaches, employing tree decoders within attention-based encoder-decoder archi- tectures, aim to capture the hierarchical tree structure, but are limited by CFGs and pre- generated triplet data, hindering expandabil- ity and neglecting visual ambiguity challenges. This article investigates the distinctive lan- guage characteristies of LaTeX mathematical expressions, revealing two key observations: 1) the presence of explicit structural symbols. and 2) the treatment of symbols as minimal units, each directly assigned specific seman- tics. Rooted in these properties, we propose that language models have the potential to syn- chronously and complementarily provide both structural and semantic information, making them suitable for correction of HMER. val- idate our proposition, propose architec- ture called Recogmition and Language Fusion Network (RLFN), which integrates recognition and language features to output corrected se- quences while jointly optimizing with a string decoder recogmition model. Experiments show that RL.NN outperforms existing state-of-the the- art methods on the CROHME 2014/201//2019 !atasets.
The encoder-decoder architecture is the preva- lent method for HMER, which recasts the prob- lem as an image-to-sequemt translation task, con- verting handwritten formula image into LaTeX markup sequence. In contrast to traditional OCR tasks, the two-dimensional structure of handwritten formulas necessitates approach that doesn't rely on direct segmentation. Since Zhang et al. (2017) introduces a decoder using RNN with attention, subsequent work has concentrated on enhancing the accuracy of the visual attention (Zhao et al., 2021; Bian et al., 2022; Li et al., 2022). Currently, various tree decoders and methods of syntactic anal- ysis, such Zhang et al. (2020) and Yuan et al. (2022), are employed to focus analyzing the expression structure and the relations of symbols.
While ceuuctre-focussed methods have undeni- ably enriched recognition model capabilities, they have also precipitated two notable challenges: 1) They rely on complex Context-Free Grammars (CFGs), necessitating the prestransformation of the LaTeX markup sequence into specific tuple rep- resentations, which limits their extensibility. 2) The issue of visual ambiguity is left behind. Tree decoders pay less attention to context when predict- ing triples, often unable to distinguish differences such as '2' and 'z'. Ung et al. (2021) try to em- ploy language model (LM) for post-correction, but Gupta et al. (2021) underline the inherent risk of wholly depending on a LM for the correction of low-redundancy information, such as numbers. which is particularly susceptible to biases intro- duced by probabilistic skewing.
1 Introduction
Handwritten Mathematical Expression Recognition (HMER), a demanding subsection of optical char- acter recognition (OCR), constitutes an interdisci- plinary crossroad of computer vision, pattern recog- nition, and natural language processing (NLP). The unfolding of deep learning advancements has no- tably enhanced the effectiveness of HMER, ush- ering its adoption in diverse arenas, including in- telligent education. Nonetheless, the precision of these technologies is continuously challenged by
However, as formal language designed for mathematical structures, LaTeX mathematical ex- pressions possess unique language characteristics.

---

We believe that advance structural analysis sepa- rately following normal NLP methods may not be a prerequisite to catch complicated structures of LaTeX mathemattcal expressions.
methods treat symbol recognition and structure analysis separately, several global methods aim tackle them simultaneously. Awal al. (2014) con- sider HMER as a simultaneous optimization prob- lem encompassing expression recognition, symbol recognition, and structure analysis. Then Álvaro et al. (2016) further extend the methodology by incorporating 2D-PCFG to integrate stochastic information from multiple sources.
Two key characteristics of LaTeX mathematical expressions are: 1) They have explicit structural symbols. 2) Minimal units are symbols, and they are directly assigned specific semantics. Based on this, we propose that LMs can proffer both struc- tural and semantic information, making them suit- able for correction of HMER. An in-depth theoreti- cal and statistical exploration of this perspective is articulated in Section 3.
Encoder-Decoder based methods are led by Deng et al. (2017) and Zhang et al. (2017). Based on CNN encoder and RNN decoder, Deng et al. (201 7) design coarse-to-fine process, while Zhang et al. (2017) design coverage attention to avoid over-parsing and under-parsing. Zhang et al. (2018) use DenseNet (Huang et al.. 2017) as encoder and introduce a decoder with multi-scale attention. Wu et al. (2020) integrate left-to-right attention to sim- ulate the progressive nature of human perception. Wang et al. (2019) use multi-modal attention aim to fully utilize both online and offline information. Zhao et al. (2021) replace RNN-based decoder with bidirectionally trained transformer, leading to en- hance global coverage and parallelization capabil- ities. Bian et al. (2022) apply mutual learning to enhance bidirectional learning and design a multi- scale coverage attention for longer expressions.
Specifically, regarding the current limitations of structure-focused methods, we believe that: 1) The structural information can be represented by struc- tural symbols' semantic with the context provided by mathematical notations. This circumvents the necessity for complex CFGs to generate triplet data. 2) Character ambiguity between different types can be rectified with contexts provided by structural symbols and mathematical notations.
We substantiate our propositions through exper- iments HMER correction. Specifieally, we de- ploy math LM rectify an HMER model reliant on unstructured- -based methods, demonstrating the suitability LM in addressing current limitations. Additionally, we argue against the sole reliance on LMs in a post-correction method. By leveraging information from the recognition model, we can constrict the correction space.
Several works focus on the tree structure of math expressions. Zhang et al. (2020) regarded the ex- pression as a tree represented by triples that include parent, children, and relation; then designed a tree decoder to predict each triple. Based on this work, Zhong et al. (2022) expanded prediction of sym- bols into attribute prediction and position predic- tion, then purposed transformer-based decoder predict triples. Yuan et al. (2022) utilized grammar constrained attention to transform the whole image into a parse tree. Wu et al. (2022a) added thinking attention to tree decoder, assisted by pixel-level auxiliary loss to improve recognition of complex expressions. Wu et al. (2022b) designed structural string representation, attempting to utilize both lan- guage model and tree structure. These structured representations are specifically designed, limiting their extensibility.
Finally, we propose our architecture called Recognition and Language Fusion Network (RLFN), which integrates recognition and language feature to output correct sequences and optimizes jointly with the recognition model. Experiments show that RLFN outperforms existing state-of of-th- art methods and achieves expression recognition rates (ExpRate)s of 000/54.3//541133 on the CROHME 2014/201 datasets.
2
Related works
2.1 HMER
Many traditional methods utilize specially designed grammars, including Chan and Yeung (2001) ) that employ definite clause grammar, MacLean and Labahn (2013) that propose a fuzzy relational gram- mar for handling ambiguous and non-linear inputs, Álvaro et al. (2014) that apply hidden Markov mod- cls to CFG, and Noya et al. (2021) that integrate hypergraph into CFG prediction. While the above
2.2 HMER & OCR Correction
Limited research are done on HMER correction. Chan and Yeung (2001) detect and correct errors based on grammar and heuristics rules. Ung et al. (2021) train and apply a language model for post-

---

correction tasks.
analyze its component parts
the stort of molecule
More correction works are done on OCR. Litman ct al. (2020) repetitively correct recurrent block out- puts by fusing it with visual features each step in training. Other works use LM to assist correction. Nguyen et al. (2020) use BERT for error detection, and then use neural machine translator to correct er- rors. Qiao et al. (2020) use the pre-trained FastText (Joulin al., 2017) to supervise the generation of semantic features, fusing it with encoder visual fea- tures to capture global semantic information. Gupta et al. (2021) utilize perplexity of language model to choose output among multiple aligned models, and stress that correction of numbers requires extra reliable information source. Yasunaga et al. (2021) adopt unsupervised correction by comparing log- its of LM output with local perturbations of the text. Fang et al. (2021, 2023) explicitly use built-in bidirectional LM to iteratively correct the output. Several math LMs are pretrained jointly with text and LaTeX expressions, potentially beneficial for HMER. Novotný and Štefánik (2022) design MathBERTa based on RoBERTa (Liu et al., 2019), with soft vector space model to capture the seman- tic similarity between tokens. Peng et al. (2021) is designed to improve the prediction of masked formula substructures extracted from the Operator Tree (OPT). Scarlatos and Lan (2023) conduct mul- tiple modifications on the GPT-2 (Radford et al., 2019) model, resulting in MathGPT, which exhibits strong performance in generating mathematical ex- pressions. Our method utilizes MathBERTa to pro- vide auxiliary information for correction.
Parse a sentence give description word Parse words
Afrac {1} (2)
the start of radicand
Asqrt (2)
Figure 1: Cases of structural symbols explicitly convey their structure in the context mathematical notations. Like in English, the word has different semantics in different contexts.
(wheeled vehicle)
rost-index (2)
is-a
has das--port
[sqrt].
radicand
1s-0
has-part
((x)
(wegon)
(vchicle)
has-part (brake)
(wheel)
lagrt (2]
xyz
(splesher)
Asgrt [3] (x)
xyx
Wheeled vehicle has brake, wheel, splasher.
Wheeled vehicle has a vahicle, wagon. Vegrt [x]
xy2
Cases of the semantic problems within dif- ferent types of symbols can be found with the structure and mathematical notation. Like in English, given the semantic relationship in WordNet, whether these types of words are correct or incorrect.
Figure 2:
larities with the mechanism of how words within phrases in English explicitly communicate their se- manties. And this characteristic enables math LMs to synchronously provide structural information just as they provide semantic information.
As figure shows, '\frac' and "\sqrt' repre- sent the fraction line and radical symbol itself while providing context. Then the following 'C' respec- tively represent the beginning of the numerator and radicand. In the English case, 'parse' relies on different context to express different semantics in sentence 'parse a sentence' and 'parse words'.
3 Why LM is Suitable for HMER Correction?
2) LaTeX mathematical expressions treat sym- bols as minimal semantic units. And based on contextual semantics, visual ambiguities between categories be corrected. Though using 'x' 'y' as unknown variables has no difference, the seman- tic distinction between an unknown variable and a number significant. For instance, V/ is rea- sonable, but V/ is not in line with convention. We will represent it as xt/= instead. Similarly, when we trust z, we tend to believe that the expression is not radical expression.
3.1 Theoretical Analysis
As formal language designed specifically for the representation of complex mathematical symbols and formulas, symbols in LaTeX mathematical ex- pressions can be broadly divided into four distinct categories: 1) Structural symbols _. C. MG \} 2) Mathematical notations (e.g., \frac, \sqrt, +, -). 3) Latin and Greek alphabets (e.g., A, a, a), and 4) Numbers. The key differences from English can be summarized in the following two points:
1) The structural symbols in LaTeX mathemat- ical expressions explicitly convey their structure, and certain mathematical notions serve to assist in this structural representation. This mechanism of structural representation shares fundamental simi-
From the perspective of analogy with English, Figure illustrates why different category sym- bols in LaTeX mathematical expressions have semantic differences when given structural rela- tionships. English, sentence semantic errors

---

caused by certain types of words can be detected through semantic relationships. According to WordNet (Fellbaum, 2005) of "wheeled vehicle'. 'wagon' and 'vehicle' have an 'is-a' relation with it. While given 'has-part' relationship, semantic errors in sentence 'wheeled vehicle has vehicle' can be discovered. Similarly, the symbol "[ I' represents the 'root-index' in the case of "[sqrt[2](x) and ")sqrt[3](x)'. "ssqrtz2[x33' uses an unknown variable as the 'root-index' which is generally unconventional. Morcover, words that represent semantic relation- ships, such as 'multiply' and 'mul mipled''' are not explicitly stated in the case of 'xyz" and 'xyx'. In general, given the multiplicand (the left term of multiplication), it's expected to use an unknown number the multiplier, semantic errors may occur in the case of 'xy2'.
Type
SS
MN
LGA
Num
Total
Perplexity Counts
2.905
3.396
3.723
3.282
3.262
51030
35112
28275
22267
136684
Categories
39
54
109
Table I: Perplexity, counts and categories of structural symbols (SS), mathematical notations (MN), latin and greek alphabets (LGA), and numbers (Num).
Type z++2 9--2
0--0
9
5+18
Top 5
sl-+52 1.926%
2.627%
1.226%
1.576%
0.876%
8.231
s2-s1 1.226%
1.226%
0.350%
0
2.803%
Total 3.152% 2.627%
2.452%
1.927%
0.876%
11.034%
Table 2: Percentage of top alphabet- number mis- recognition pairs among all SUB1 cases. s1 + s2 indicates that symbol s1 is mis-recognized as s2.
distribution, the reciprocal of which, corresponding to the actual word, signifies its perplexity. Subse- quently, the mean perplexity, according to category, is designated as the perplexity of this particular category of symbols.
In addition to language characteristic that make LM suitable for HMER correction, the task itself is also suitable. While OCR employs letters as the smallest unit for correcting words, HMER uti- lizes symbols as the minimum unit for amending expressions. The former represents morphological correction, while the latter is semantic correction.
Results in Table I reveal that among the four types of symbols, the structural symbols exhibit the lowest perplexity. This finding aligns with our the- oretical analysis that the explicit structural symbols suggest a comparatively robust semantic signal.
3.2 Statistical Analysis
We conducted statistical analysis collateral evi- dence the CROHME dataset (Mouchère al., 2014). The CROHME dataset, byproduct of the Competition on Recognition of Online Handwrit- ten Mathematical Expressions (CROHME), is uni- versally recognized as the principal public dataset within HMER field. A comprehensive collection, the CROHME training set comprises 8835 hand- written mathematical expressions. In addition, it includes three distinct testing subsets, namely CROHME 2014, 2016, and 2019, containing 986, 1147, and 1199 handwritten mathematical expres- sions respectively. Noteworthy the inclusion of a total of 111 symbol classes, which encompasses the "sos" and "eos" symbols.
The visual ambiguity between numbers and alphabets do exist. We conducted an analysis of the results from the string decoder recogni- tion model, DWAP (Zhang et al., 2018), using the CROHME 2014/2016/20 6/2019 datasets. During this analysis, we employed a reliable metric called Substitute-by-On (SUB1), which identifies cases where the model's predictions deviate from the ground truth by only one substitution. Within the SUB1 cases, the mis-recognition of one character as another is verified, avoiding character substitu- tion indeterminacy in evaluation.
The outcomes are shown in Table Among all SUB1 instances, the top 5 pairs of misrrecognition between numbers and alphabets contribute to I 1% of the total misrrecognition, while the overall mis- recognition between numbers and alphabets con- tribute to 26%. These statistical findings highlight that mis-recognition between numbers and letters not only exists considerably but also tends con- centrate on visual ambiguity. Thus overcoming the visual ambiguity issues as mentioned in theoretical analysis is significant.
The explicit structural symbols do express their semantics. This is affirmed via an application of the Math-aware BERT model (Reusch et al., 2022), where the calculated perplexity acts as an index for semantic strength, applied to those four categories of symbols in CROHME training set.
In detail, we engage an individualized masking operation, followed by a model prediction of the obfuscated symbol. The outcome is a probability

---

4 Method
Iet
In an endeavor to empirically corroborate our hypotheses, propose novel architecture, the Recognition and Language Fusion Network (RLFN), engineered specifically to address the dual challenge of visual ambiguity and the com- plex structural issues that are inherent to string de- coder recognition models. The RLFN is built with a string decoder recognition module, language module that extracts language information, and fusion module to refine the recognition output by utilizing the language information.
Av
cony
Linsor
Tonk
Linear & Seftmax
A
Linegr
Lioor
Lineor
Ein
Saftrea
4.1 Recognition Module
Figure 3: Recognition module
Our recognition module basically follows DWAP (Zhang et al., 2018), using DenseNet (Huang et al., 2017) extract visual feature F HB''>xxx from the single-channel input image.
4.2 Language Module
We utilize MathBERTa (Novotný and Štefánik, 2022) to extract language information, which is RoBERTa (Liu et al., 2019) model specifically fine-tuned on LaTeX expressions.
As shown in Figure 3, each step in the decoder, we iteratively update two state weights: the GRU (Cho et al., 2014) hidden state h2 and the coverage attention (cumulative attention map) At.
As a variant of BERT (Devlin et al., 2019), RoBERTa solely focuses on masked language mod- (1) eling (MLM) task, uses larger mini-batches and employs dynamic masking, which all contribute to improve bidirectional semantic language modeling. Building upon RoBERTa, MathBERTa further
h4 CRUU(B7-,,1-1
e We (anh(FF + WAAr + Whht 1)
(2)
exp(eiia maxi,j
(3) focuses on language processing of LaTeX expres- sions. It undergoes fine-tuning on an extensive dataset containing both text and LaTeX expressions. This specialized training enhances MathBERTa's comprehension of semantic and syntactical proper- ties of LaTeX mathematical expressions.
04.jst Σ Cexp(ci.jt max; (())t)
A = A,1 +a
(4)
Here, e, represents the attention score which pro- duces the attention map at, where i, denote the coordinate on the feature map. Ey-- represents the embedding of the last symbol, F3 corresponds to the position encoded feature (Parmar et al., 2018). W.. WA. and Wi, represent the trainable weights. After that, we generate the content vector o with element-wise multiplication of a with vi- sual features F. Then & is combined with h, and embedding of 1 obtain the symbol state st and the recognition prediction symbol 7t.
Considering that our recognition output does not need tokenization, and LaTeXX mathematical nota- tions are prone to problems, we manually associate the vocabulary with the one-hot encoding in Math- BERTa instead of using the tokenizer. Then given the recognition prediction sequence 7, MathBERTa outputs the language feature Fi.
4.3 RLEN
As shown Figure 4. RLFN, the input image is sent to recognition module to extract recognition feature and the prediction sequence. The latter is passed to language module to form language feature. Both features are fused in fusion module to output the conrected prediction sequence.
8t Wec + W + he
(5)
= softmax(w'
(6)
W., W.,, W w, b are trainable weights. Lastly, recognition module outputs the total symbol states s as recognition feature FR. along with the recog- nition prediction sequence 77.
The main objective of the fusion module is integrate the information from the recognition mod- ule with the semantic information obtained through

---

in the fusion module is follows:
Vfroc (8 bx Vircz (3 Mimes 1)
35
5
d
Fn FW=WLF
(7)
Linear Softmax
(8)
Recognition Module
Language Module
bR = tanh(aR)) hy = tanh(a
(9)
DenseNet Encoder
MathBERTa
hn + 2) h1.
(10)
Transformen
Encoder
y softmax("'
(11)
Transformer
Encoder
Attention Decoder
Tronsformer
where a refers to sigmoid activation function, WR. WL, WF, w', V are weights to learned.
Encoder
4.4 Parameter Learning
FR
We jointly optimize our RCFN with the recognition module through a linear classification layer, and the loss function as follows:
Fusion Module
Linear
Linear
L CR + CF
(12)
tonh
fanh
where La and Lp are the cross cross-entropy loss of the recognition prediction sequence probability and the corrected prediction sequence probability with respect to the ground-truth.
Linear & Softmax
Ln aims to guide the the recognition module and the additional linear classification layer, while Lf focus on guiding the fusion of the two features. To mitigate the influence of the training set's proba- bility bias and the large number of parameters, we have frozen the language model's parameters. Fur- thermore, gradient separation has been employed to enhance the focus of the loss functions on their respective optimization goals within the recogni- tion model. Nevertheless, joint optimization still affects each other's ' updates of problematic parts through the optimizers and other means.
Vfrac (8 Vtimms 5 3)
1](6+1)
Figure 4: Architecture Recognition and Language Fusion Network (RLEN)
the language module, to generate the cor- rected prediction sequence.
In the fusion module, the recognition feature FR and language feature Fr. are first dimensionally aligned through linear operation. Then we follow Yao et al. (2017) incorporate gating neuron, de- noted This neuron allows us assign weights based contributions of two features during the calculation of output. Within the gating neuron, the two aligned features are horizontally concatenated. The resulting concatenated vector is then adjusted to match the size of the aligned features. Subse- quently, the resized vector is fed into sigmoid function to generate weights. These weights are then utilized to modulate the output of the aligned features, which first processed through tanh activation function.
Experiments
5.1 Implement Details
Our RCFN is implemented in Py Torch with a sin- gle NVIDIA Gelfrree RTX 3090. We use Adadelta optimizer (Zeiler, 2012) with the learning rate in- creases from 0 to 1 at the first epoch and decays to 0 following the cosine schedules (Zhang et al., 2019b). No data augmentation for fair compari- The batch size is set 8. All images within batch filled in the upper left corner of the canvas of the same size. Due memory limitations, the canvas size does not exceed 1280 * 280, and any excess images will discarded. The total training epoch set to 200 epochs taking around 16 hours.
The process of generating the corrected sequence

---

CROHME 2014
CROHME 2016
CROHME 2019
Method
ExpRatet
21
ExpRatef
SIT
221
ExpRatet
SIT
22
UPV (Mouchère et al., 2014)
37.22
44.22
47.26
WAP (Zhang ct ul., 2017)
46.55
61.16
65.21
44.55
57.10
61.55
(Wu PAL. et al., 2019)
39.66
56.80
65.11
TAP (Zhang et al., 2019a)
48.47
63.28
67.34
44.81
59.72
62.77
DWAP (Zhang et al., 2018)
50.10
47.50
MAN (Wang et al., 2019)
54.05
68.76
72.21
50.56
64.78
67.13
PAL-V2 (Wu al., 2020)
48.88
64.50
69.78
49.61
64.08
70.27
RBR (Truong et al., 2020)
53.40
65,20
70.30
52.10
63.20
69.40
$3.10
63,90
68.50
DLA 2020)
49.85
47.34
DWAP-TD (Zhang et al., 2020)
49.10
64.20
67.80
48.50
62.30
65,30
$1.40
66.10
69.10
WS-WAP (Truong ct ul., 2020)
53.65
51.96
64.34
70.10
BTTR (Zhao et al., 2021)
53.96
66.02
70.28
52.31
63.90
68.61
$2.96
65.97
69.14
ABM (Bian et al., 2022)
56.85
73.73
81.24
52.92
69.66
78.73
53.96
71.06
78.65
SAN (Yuan ct al., 2022)
56.20
72.60
79.20
53.60
69.60
76.80
53.50
69.30
70.10
GPT-4V (Yang et ol., 2023)
31.85
49.09
60.45
DWAP (Baseline)'
51.72
69.47
77.99
48.82
67.13
75.41
50.79
69.64
76.81
RLFN-DWAP (Ours)
57.00
72.01
80.73
54.23
70.10
78.47
54.13
72.56
80.07
Table 3: Results on the CROHME dataset without any data augmentation. 'indicates that we reproduce DWAP as shown in figure RLFN-DWAP represents we the reproduced DWAP as our recognition module.
5.2 Evaluation
Method
ExpRate
117
S 2
DWAP (Baseline)
51.72
69.47
77.99
The metric expression recognition rate (Ex- pRate) utilized, defined as the proportion of accu- rately recognized expressions. Additional measure- ments, denoted as < and < 2, are also employed, where the ExpRate accommodates most two symbol-level errors, respectively.
Language Module
53.96
70.18
78.80
Fusion Module
57.00
72.01
80.73
Table 4: Ablation study on CROHME 2014
fusion module. To separate the impact of the lan- guage module, we did not use the fusion module in RLFN. Instead, in order to generate the corrected prediction sequence, we treat it as translation task and add decoder with transformer layers in the language module. Other settings are all identi- cal RLFNN Results on CROHME 201 are shown in table 4, we can only tell that the language module and the fusion module both have their impact.
We experiment on CROHME datasets mentioned Scction 3.1. Consistent with previous methods, we use CROHME 2014 as the validation set and test on CROHME 2016 and 2019 to compare with previous state-of-the-an (SOTA) methods.
As shown in Table 3, take the reconstructed DWAP as our baseline. And our RLFN-DWAP using it as the recognition module achieves SOTA on the ExpRate indicator. In addition, can be ob- served that the improvement of model on ExpRate is higher than on < 1 or < 2, which is consistent with the intuition that sentences with fewer errors have more complete semantics information.
5,4 Improvement Study
In this section, we explore whether the improve- ment comes from the correction of complex struc- ture and visual ambiguity to validate our propo- sition that LM do obtain semantic and structural information synchronously and complementarily.
Inspired by the LaTeX code generation capa- bility reported (Yang et al., 2023), we conduct an experiment using GPT-4V on CROHME 2014 dataset with the the prompt 'generate latex code and output without compile." The outputs are post- processed to align CROHME vocabulary, and the ExpRate of GPT-4V is 31.85. Given that it is not finetuned the CROHME dataset, its performance is acceptable.
We conduct an incremental comparison on struc- tural complexity to assess the improvement of our model on complex expressions across all datasets. We define the structural complexity of an expres- sion as the count of six structural symbols men- tioned in Section 3.1. The results are presented in Table 5. Models perform worse when facing more complex expressions, suggesting they are more challenging to recognize. Interestingly, the rela- tive improvement becomes progressively higher for more complex expressions. The improvement
5.3 Ablation Study
this subscction, we perform ablation study to analyze the impact of the language module and the

---

DWAP
RIFN
4x4+4-0
ExpRate
0.2965
0.3469
17.00%
Complex 25%
0.4454
0.4766
7.01%
4
0.5378
0.5954
10.71%
DIAAP bi lpn luget [ Minnc [1] Saptt (1-46)1))
DWAh 4 4
< 3
0.6074
0.6543
7.71%
RLPNI by 1pm lugrt Effrac (33 Sagrt (1-4c]]]
ExpRate
0,3908
0.4322
10.60%
Complex 50% %
< 1
0.5672
0.6014
6.03%
27 J
F DWAP (3) (R)
2
0.6507
0.6963
7.01%
S 3
0.7209
0.7539
4.58%
(1sr 12733 17.31 (31)19)
ExpRate
0,5039
0.5501
9.17%
RLPN:
$ 0.6873
0.7155
4.10%
Total
0.7668
0.7971
3.95%
Figure Cases with complex structure (A) and visual
0.8235
0.8457
2.70%
ambiguity (B) that RLFN outperforms our baseline.
Table Incremental examination of the Top 25%, 50%, and total expressions based structural complexity in the unified CROHME 2014/201 6/2019 dataset. RI denotes the relative improvement from DWAP to RLFN.
Uirma
1.00
0.42
0.04
1.00
1.00
1.00
2.00
1.00
0.03
2.00
1.00
1.00
1.00
1.00
00
0.00
0:20
0.00
0.00
0.00
8.00
0.00
0.00
0.03
0.00
0.00
0.00
0.00
Z++2
g++9
9++9
04-0
b-+6
Top-5
-44%
-19%
0%
-24%
10%
-21%
8.00
6.00
0.00
0000
8.00
0.00
0.01
0.00
0.00
0.00
0.00
Table Relative change of top mis-recognition.
Figure 6: Visualization of top-5 probability
for the top 25% most complex expressions is nearly twice that of the improvement among all expres- sions. These observations indicate that RLFN out- performs our baseline especially in recognizing complex structures which might be because our RLFN can extract explicit structural information through LM, serving the similar role to tree de- coders.
ture modeling.
In group B, cases possess visual ambiguity be- tween variables and numbers. The baseline model relies solely on visual appearance and cannot distin- guish visually resemble symbols. This is likely due to its lack of architecture to effectively utilize and comprehend contextual and structural information. RLFN, on the contrary, can correctly recognize 'Ntimes' with surrounding numbers and recognize '1' and '0' based on their superscript and subscript structural relations. This indicates that RLFN can complement each other's 's semantic and structural information when recognizing visually ambiguous symbols.
Regarding RLFN's performance in handling vi- sual ambiguity, we conduct replicated analysis same to the one described in Section 3.1. Specifi- cally, we compare the frequency of top 5 number- alphabet pairs mis-recognition with our baseline and study the difference, which is shown in Table 6. We observe that RLFN effectively reduces the occurrence of the top-5 mis-recognition by 21 % compared to our bascline. This shows the capabil- ity of RLFN reduce visual ambiguity between alphabets and numbers using contextual informa- tion provided by language modeling.
As depicted in Figure 6, delve into particu- lar case about its top-5 probability. The probability derived from the baseline demonstrates visual ambi- guity concerning the symbols 'x', 'X', and ' 'Vtimes' without understanding their semantics. After deter- mining that this may be multiplication structure, RLFN reduces the probability associated with 'x' and 'X', while recognizing the correct \times sym- bol with high confidence.
5.5 Case Study
As shown in Figure 5, we present two complex structure cases in group A, along with another two visual ambiguity cases in group B.
Conclusion
In group A, the baseline model recognizes an additional structure symbol "I'in case and misplaces "VJ' the wrong position in the other. In contrast, RLFN gets both expressions correctly, This is not a symbol recognition problem, which indicates that RLFN has learned the semantics of structure symbols and gained the ability of struc-
In this research, we have examined the unique lan- guage characteristics intrinsic LaTeX mathemat- ical expressions, with keen focus on the minimal semantic unit and explicit structural symbols. Our investigation underscores that these characteristics give HMER systems the potential to obtain both se-

---

mantic and structural information through language models. We subsequently propose an innovative ar- chitecture that harmoniously integrates recognition and language features to yield corrected sequences. This framework eliminates the requirement to con- struct complex CFGs for resolving structural issues, and serves to ameliorate the challenge of visual am- biguities. This integrative approach offers fresh insights and promising theoretical groundwork for the development of HMER and related mathemati- cal endeavors.
the significance of excavating model architectures for specific tasks. However, is there real need for the involvement of general large model in a specific task? When the accuracy requirements are stringent, how do the upper limits of a general large model compare with that of small model tailored for specific task? Or is it the case that data is truly everything? These questions still require deep consideration.
A Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grant No.62250057.
Limitations
The limitations of our theoretical assessment war- rant acknowledgment. In our analysis, we scruti- nized the language characteristics of LaTeX math- ematical expressions, drawing parallels between their expression mechanisms and those of the En- glish language. This led to posit that LM adept at handling English semantics should, in theory, be equally proficient with LaTeX mathematical expressions. However, our methodology, rooted inferential analogy, is weaker than directly an- alyzing how LMs handles La'TeX mathematical expressions and cannot be further extended, such customizing a L. suitable for LaTeX mathemat- ical expressions.
References
Ahmad-Montane Awal, Harold Mouchère, and Chris- tian Viard-Gaudin. 2014. A global learning approach for an online handwritten mathematical expression recognition system. Puttern Recognition Letters, 35:68-77, 77. Frontiers in Handwriting Processing.
Xiaohang Bian, Bo Qin, Xiaozhe Xin, Jianwu Li, Xue- feng Su, and Yanfeng Wang. 2022. Handwritten mathematical expression recognition via attention ag- gregation based bi-directional mutual learning. Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, 36(1) 1): 13-121.
Kam-Fai Chan and Dit-Yan Yan Yeung. 2001. Error detec- tion, eтror correction and performance evaluation in on-line mathematical expression recognition. Pattern Recognition, 34(8): 167 1-1684
Our proposed model architecture is not devoid of certain limitations. The architectural design broadly follows a late fusion strategy, which, when contrasted with the early fusion approach seen in semantic modeling and modal fusion during the decoding phase of the recognition module, exhibits lack of thorough information interaction. This shortfall is exemplified by our model's disregard for the prediction probability of the recognition se- quence input to the language module, resulting in some information loss.
Kyunghyun Cho, Bart van Merriënboer, Caglar Gul- cehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference Empirical Methods in Natural Longuage Processing (EMNLP), paages 1724 1734, Doha, Qatar. Association for Computational Linguistics.
Besides, given the current state of the field, where most existing recognition models rely heav- ily tree decoders and bidirectional training ar- chitectures, triplet data and reverse sequences are not suitable for language modeling. This limitation confines the range of selectable baseline models. Notwithstanding, one of our overarching goals in this endeavor is to maneuver around this intrinsic constraint that inherently stifles expansion.
Yuntian Deng, Anssi Kanervisto, Jeffrey Ling, and Alexander M. Rush. 2017. Image-to-markup genera- tion with coarse-to-fine fine attention. In Proceedings of the 34th International Conference on Machine Learn- ing. volume 70 of Proceedings of Machine Learning Research, pages 980-989. PMLR.
Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Furthermore, the formidable capabilities of LLM and LMM/MLLM are widely researched, some methods can even achieve an OCR-free un- derstanding of text images. This casts doubt on

---

Shancheng Fang, Zhendong Mao, Hongtao Xie, Yuxin Wang, Chenggang Yan, and Yongdong Zhang. 2023. Abinet++: Autonomous, bidirectional and iterative language modeling for scene text spotting. IEEE Trunsactions on Puttern Analysis and Machine Intel- ligence, 45(6))7123-7141.
Scott MacLean and George Labahn. 2013. new ap- proach for recognizing handwritten mathematics us- ing relational grammars and fuzzy sets. International Journal on Document Analysis and Recognition (IJ- DAR), 16(2):139-163.
H. Mouchère, C. Viard-Gaudin, R. Zanibbi, and U. Garain. 2014. Icfhr 2014 competition on recogni- tion of on-line handwritten mathematical expressions (crohme 2014). In 2014 14th International Confer- ence on Frontiers in Handwriting Recognition, pages 791-796.
Shancheng Fang. Hongtao Xie, Yuxin Wang, Zhendong Mao, and Yongdong Zhang. 2021. Read like hu- mans: Autonomous, bidirectional and iterative lan- guage modeling for scene text recognition. In 2021 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 7094-7103.
Thi Tuyet Hai Nguyen, Adam Jatowt, Nhu- Van Nguyen, Mickael Coustaty, and Antoine Doucet. 2020. Neu- ral machine translation with bert for post-ocr error detection and correction. In Proceedings of the A CM/IEEE Joint Conference on Digital Libraries in 2020, JCDL '20, pages 333-336, New York, NY, USA. Association for Computing Machinery.
Christiane Fellbaum. 2005. Wordnet and wordnets. In Alex Barber, editor, Encyclopedia of Language and Linguistics, pages 2-665. Elsevier.
Harsh Gupta, Luciano Del Corro, Samuel Broscheit, Johannes Hoffart, and Eliot Brenner. 2021. Unsu- pervised multi-view post-OCR error correction with language models. In Proceedings of the 2021 Confer- ence on Empirical Methods Natural Language Pro- cessing, pages 8647-8652, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Vít Novotný and Michal Štefánik. 2022. Combining sparse and dense information retrieval. In Proceed- ings of the Working Notes of CLEF 2022- Conference and Labs of the Evaluation Forum, pages 104-118. Bologna. CEUR-WS.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. 2017. Densely connected con- volutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pages 2261-2262
Ernesto Noya, Joan Andreu Sánchez, and José Miguel Benedí. 2021. Generation of hypergraphs from the n-best parsing of 2d- éd-probabillistic context-free grammars for mathematical expression recognition. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 5696-5703.
Armand Joulin, Edouard Grave Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Con- ference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Pu- pers, pages 427-431, Valencia, Spain. Association for Computational Linguistics.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. 2018. Image transformer. In Proceedings of the 35th International Conference on Machine Learn- ing. volume 80 of Proceedings of Machine Learning Research, pages 4055 PMLR.
Anh Due Le. 2020. Recognizing handwritten mathe- matical expressions via paired dual loss attention net- work and printed mathematical expressions. In 2020 IEEE/CVF Conference on Computer Vision and Pat- lern Recognition Workshops (CVPRW), pages 2413- 2418.
Shuai Peng, Ke Yuan, Liangcai Gao, and Zhi Tang. 2021. Mathbert: A pre-trained model for mathematical for- mula understanding. CoRR, abs/2105.00377.
Zhi Qiao, Yu Zhou, Dongbao Yang. Yucan Zhou, and Weiping Wang. 2020. Seed: Semantics enhanced mcoder-decode framework for scene text recogni- tion. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13525- 13534.
Bohan Li, Ye Yuan, Dingkang Liang, Xiao Liu, Zhilong Ji, Jinfeng Bai, Wenyu Liu, and Xiang Bai. 2022. When counting meets hmer: Counting-aware network for handwritten mathematical expression recognition. In Computer Vision ECCV 2022, pages 197-214.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 1(8):
Ron Litman, Oron Anschel, Shabar Tsiper, Roee Lit- man, Shai Mazor, and R. Manmatha. 2020. Scatter:
Selective context attentional scene text recognizer. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11959-11969
Anja Reusch, Maik Thiele, and Wolfgang Lehner, 2022. Transformer-encodee and decoder models for ques- tions on math. Proceedings of the Working Notes of CLEF 2022, pages 5-8.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.
Alexander Scarlatos and Andrew Lan. 2023. Tree-based representation and generation natural and mathe- matical language. CoRR, abs/2302.07974.
Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1 16692.

---

Thanh-Nghia Truong, Cuong Tuan Nguyen, Khanh Minh Phan, and Masaki Nakagawa. 2020. Improvement of end-to-end offline handwrit- ten mathematical expression recognition by weakly supervised learning. In 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFII)) pages 181-186.
Ye Yuan, Xiao Liu, Wondimu Dikubab, Hui Liu, Zhi- long Ji, Zhongqin Wu, and Xiang Bai. 2022. Syntax- aware network for handwritten mathematical expres- sion recognition. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4543-4552
Matthew D. Zeiler. 2012. ADADELTA: an adaptive learning rate method. CoRR, abs/ 22125221..
Huy Quang Ung, Cuong Tuan Nguyen, Hung Tuan Nguyen, Thanh-Nghia Truong, and Masaki Naka- gawa. 2021. A transformer- based math language model for handwritten math expression recognition. In Document Analysis and Recognition ICDAR 202/ Workshops, pages 403-415, Cham. Springer International Publishing.
Jianshu Zhang, Jun Du, and Lirong Dai. 2018. Multi- scale attention with dense encoder for handwritten mathematical expression recognition. In 2018 24th International Conference on Pattern Recognition (ICPR), pages 2245-2250.
Jaaming Wang, Jun Du, Jianshu Zhang, and Zi-Rui Wang. 2019. Multi-mode attention network for hand- written mathematical expression recognition. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1181-1186.
Jianshu Zhang, Jun Du, and Lirong Dai. 2019a. Track, attend, and parse (tap): An end-to-end framework for online handwritten mathematical expression recogni- tion. IEEE Transactions on Multimedia, 21(1):221- 233.
Changjie Wu, Jun Du, Yunqing Li, Jianshu Zhang, Chen Yang. Bo Ren, and Yiqing Hu. 2022a. Tdv2: A novel tree-structured decoder for offline mathematical ex- pression recognition. Proceedings of the AAAI Con- ference on Artificial Intelligence, 2693)2269
Jianshu Zhang, Jun Du Yongxin Yang. Yi-Zhe Song, Si Wei, and Lirong Dai. 2020. A tree-structured decoder for image-to-markup generation. Pro- ceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 11085. PMIR.
Jiajia Wu, Jinshui Hu, Mingjun Chen, Lirong Dai, Xue- jing Niu, and Ning Wang. 2022b. Structural string de- coder for handwritten mathematical expression recog- nition. In 2022 26th International Conference on Puttern Recognition (ICPR), pages 3246-3251.
Jianshu Zhang, Jun Du, Shiliang Zhang, Dan Liu, Yu- long Hu andJ inshui Hu, Si Wei, and Lirong Dai. 2017. Watch, attend and parse: An end-to-end neural network based approach to handwritten mathematical expression recognition. Pattern Recognition, 71:196- 206.
Jin- Wen Wu, Fei Yin. Yan-Ming Zhang, Xu- Yao Zhang, and Cheng Lin Liu. 2019. Image-to to taaae-to-markup gen- eration via paired adversarial learning. In Machine Learning and Knowledge Discovery in Databases, pages 18-34, Cham. Springer International Publish- ing.
7hi Zhang, Tong He, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. 2019b. Bag of freebies for training object detection neural networks. CoRR, 0bs//992.04411
Jin- Wen Wu, Fei Yin, Yan-Ming Zhang, Yao Zhang, and Cheng-Lin Liu. 2020. Handwritten mathemat- ical expression recognition via paired adversarial learning. International Journal of Computer Vision 128(10):2386-24)) 2401.
Wenqi Zhao, Liangeai Gao, Zuoyu Yan, Shuai Peng. Lin Du, and Ziyin Zhang. 2021. Handwritten mathe- matical expression recognition with bidirectionally trained transformer. In Document Analysis and Recognition ICDAR 2021, pages 570-584, Cham. Springer International Publishing.
Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Shung-Ching Lin, Zicheng Liu, and Lijuan Wang. 2023. The dawn of Imms: Preliminary explorations with gissin)
Shuhan Zhong, Sizhe Song, Guanyao Li, and S.-H. Gary Chan. 2022. A tree-based structure-ad aware trans- former decoder for image-to-markup generation. Proceedings of the 30th ACM International Confer- ence Multimedia, MM '22, page 5751-5760, New York, NY, USA. Association for Computing Machin- ery.
Lili Yao, Yaoyuan Zhang, Yansong Feng, Dongyan Zhao, and Rui Yan. 2017. Towards implicit content- introducing for generative chort-text conversation sys- tems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2 2190-2199, Copenhagen, Denmark. Associa- tion for Computational Linguistics.
Francisco Álvaro, Joan-Andreu Sánchez, and José- Miguel Benedí, 2014. Recognition of on-line hand- written mathematical expressions using 2d stochastic context-free grammars and hidden markov models. Pattern Recognition Letters, 35:58 --65 Frontiers Handwriting Processing.
Michihiro Yasunaga, Jure Leskovec, and Percy Liang. 2021. Lm-critic: Language models for unsu- pervised grammatical error correction. CoRR, abs/2109.06822.

---

Francisco Álvaro. Joan-Andreu Sánchez, and José- Miguel Benedí. 2016. An integrated grammar- based approach for mathematical expression recognition. Pattern Recognition, 51:135-147.


Reviews:
[{"review_id": "CYu1cS24Zp", "reviewer": "Reviewer_jmUa", "summary": "", "questions": "1) The evaluation can be made more exhaustive by adding more sources other than CROHME. How will the designed system evaluate with low resolution scanned expressions or those written by less experienced people. How will it recognize wrong expressions. Will it autocorrect.\n2) Comparison with frameworks/platforms like Google Lens, Socratic etc. which has made solving math problems easy for people would also be appreciated.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review.\n\n\n\nFirstly, I'd like to address the practical application concerns you've highlighted. While these may not directly tie into the central thrust of my paper, my response stems primarily from our experiences in trying to use HMER as the input for automated grading systems.\n\nTo the best of my knowledge, current HMER systems still lag behind human performance. This includes the Google Lens you mentioned, as well as the frequently used Mathpix (which excels in recognizing individual formula images) and TAL (which excels in recognizing full answer sheet images) APIs. Despite them potentially being trained on vast amounts of data, their accuracy is still insufficient for applications demanding high precision, such as automated grading. Human intervention remains necessary in most scenarios. \n\nThe CROHME dataset discussed in the paper is different. CROHME is artificially created and includes some exceedingly complex formulas that might rarely be encountered in real-world scenarios. The evaluation requires training on 8,835 images from the training set without any data augmentation. On one hand, this is to ensure a fair comparison; on the other hand, it measures the unique ability of HMER systems to handle complex structures, which typically represents the upper limit of the model.\n\nWe've tested the Mathpix and TAL APIs on CROHME2014, and the accuracy rates for Ours RLFN/Mathpix/TAL are 0.5700/0.6187/0.2667 respectively. Regarding Google Lens, since we couldn't locate a dependable API, we evaluate the initial 100 test samples from CROHME2014 using its app, with accuracy rates of 63/100 for our RLFN and 61/100 for Google Lens. It's essential to note that this isn't a fair comparison. For such applications, on one hand, they probably have tenfold or even a hundredfold more training data, undoubtedly enhancing their capability. On the other hand, the diverse sources of their data introduce additional noise, potentially reducing performance.\n\nOutside of CROHME, real-world datasets, those containing low-resolution images or shadows, are seldom made public. We could only find an open dataset called HME100K, which covers segmented mathematical and chemical formulas. However, they don't provide data cleaning or augmentation methods. Compared to the reported 67% accuracy by the dataset provider, our RLFN achieved around 62% without data preprocessing. This discrepancy may be due to our exclusion of chemical formulas and the lack of additional data cleaning and augmentation.\n\nIn our practice, we have labeled scanned images from the same source (with different writers). The performance metrics are as follows: RLFN training on HME100K/API(TAL)/RLFN fintune on ~1k data, 69%/ 78% / 80%. This indicates that providing data tailored to the scenario can result in a significant performance boost. However, for individual daily use, the previously mentioned APIs or software are undoubtedly more robust.\n\nAdditionally, the HMER task doesn't specifically cater to less experienced people, and our model won't autonomously correct formulas devoid of image information. On the contrary, this is what we aim to avoid. The problem you've described seems more aligned with the requirements of an auxiliary input system, rather than the core objective of HMER – precise image recognition suited for high-precision tasks like automated grading.  Our choice of employing a fusion module over post-correction methods is driven by our intent to provide semantic context for visually ambiguous terms and to avoid dependency on semantic cues for high-confidence visual terms, thus preventing illusions or extreme corrections due to factors like low redundancy, such as the output x=1 being corrected to x=2.\n\n\n\n\nLastly, we review the introduction section and the comments of another reviewer W2zF, I think it may be unclear when trying to introduce our viewpoints about what characteristics of mathematical formulas make LM suitable for HMER, resulting in confusion in paragraphs 4, 5, and 6 of the introduction.\n\nWe have identified two main arguments in Section 3.1 and plan to make corresponding adjustments to this section and the introduction in the future.\n\n1. The presence of an explicit symbolic structure allows the LM to directly model the structure.\n2. Based on contextual semantics, visual ambiguities between categories can be corrected.\n\nCorresponding content of Statistical Analysis in Section 3.2:\n\n1. The explicit structural symbols do express their semantics.\n2. The visual ambiguity between numbers and alphabets do exist.\n\nCorresponding content of Improvement Study in Section 5.4:\n\n1. Complex structural expressions get more improvements.\n2. Visual ambiguity mis-recognition has been effectively reduced.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper helps in making use of language characteristics of LaTeX way of writing mathematical expressions and use this to resolve conflicts. Mathematical expressions can get complex and typical OCR which involved segmentation and recognition steps are not applicable for this. Hence the paper could be useful to scientific community and mathematicians.", "reasons_to_reject": "The paper is not easy to understand. The language can be simplified and made more crisp. While i was reading the introduction it was not clear to me that what is the problem to be solved exactly and what is the novelty the paper is adding. Only after i read the entire paper i was able to understand the work entirely. Introduction should get the reader inquisitive and give an overall view of paper and make the reader interested in reading the paper entirely.", "questions_to_authors": "1) The evaluation can be made more exhaustive by adding more sources other than CROHME. How will the designed system evaluate with low resolution scanned expressions or those written by less experienced people. How will it recognize wrong expressions. Will it autocorrect.\n2) Comparison with frameworks/platforms like Google Lens, Socratic etc. which has made solving math problems easy for people would also be appreciated.", "paper_topic_and_main_contributions": "The problem being addressed is recognition of mathematical expressions which are handwritten into latex statements. The paper addresses structural and visual ambiguities and builds an architecture RLFN to solve this. Though the architecture is an extension to DWAP however by making use of recognition and language module there is improvement in expression recognition than the baseline method.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "None"}, "extra_text": {}}, {"review_id": "9lGXqEmGNr", "reviewer": "Reviewer_W2zF", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review.\n\nI largely agree with the issues you pointed out regarding the theoretical analysis in Section 3.1. In the limitations section, we also mentioned this method based on analogical reasoning, comparing the expression mechanism of LaTeX with English language to demonstrate that the LM is effective for the HMER task is not sufficient.\n\nIntuitively, LMs might be applicable to most natural languages, but for mathematics, it isn’t as straightforward.\n\nThere's a skepticism in the field, mirrored in studies focusing on post-correction for OCR. They argue that solely relying on an LM for correcting low-redundancy information, such as numerical data, might expose the process to biases, particularly from the probabilistic distortion inherent in models.\n\nRecent trends offer further evidence. Following the surge in LLM's popularity, tasks like mathQA, which bridge natural language to mathematical expressions, have shown progress. Yet, when directly grappling with mathematical formulas, LLM continues to face challenges, especially in areas like mathematical reasoning.\n\nWe couldn't find an appropriate method to explain 'Why' from an overall LM perspective. Subsequently, we shifted our focus to identify what characteristics of mathematical formulas make LM suitable for HMER. Apart from the model, a significant portion of our paper, including Sections 3.1, 3.2, and 5.4, is dedicated to explanations from theoretical, statistical, and result improvement perspectives.\n\nReflecting on the feedback, including issues highlighted by another reviewer concerning our introduction, we acknowledge that our paper's viewpoints might come across as somewhat muddled. We have identified two main arguments in Section 3.1 and plan to make corresponding adjustments to this section and the introduction in the future.\n\n1. The presence of an explicit symbolic structure allows the LM to directly model the structure.\n2. Based on contextual semantics, visual ambiguities between categories can be corrected.\n\nCorresponding content of Statistical Analysis in Section 3.2:\n\n1. The explicit structural symbols do express their semantics.\n2. The visual ambiguity between numbers and alphabets do exist.\n\nCorresponding content of Improvement Study in Section 5.4:\n\n1. Complex structural expressions get more improvements.\n2. Visual ambiguity mis-recognition has been effectively reduced.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A well written paper, interesting idea, well-done experiments, promising results, wide comparison with other methods.\nI appreciate the attempt to motivate the use of LM by a series of analyses of the exiting datasets and common errors obtained by the current SOTA. The ablation study showing the actual contribution of the LM component and the fusion part. ", "reasons_to_reject": "The paper might be not so much relevant to the conference. The only relevant part is the use of the LM component and the motivation for using the LM component by comparing the Latex language to English. \nI am not were convinced by the theoretical analysis in Section 3.1 which tries to explain why LM can help in the task, e.g. the argument that \nbecause \\sqrt[z]{x} is less common than x^{1/z} than the output should be rather \\sqrt[2]{x}. I trust the results that LM can help the task, but is such situations are typical, some quantitative analysis should be done.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents a novel method for recognition of handwritten math expressions. The input is an image of such an expression, the output is latex code that generates such expression. The problems has been studied relatively lot recently, there is a series of CROHME datasets from three years and several models were proposed. The contribution of this paper is to incorporate a language model component in the model, which is a quite novel approach. The language model is adopted from another work and it is a RoBERTa based model fine-tuned on latex match expressions. The results are quite promising outperforming SOTA by a large margin. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1y2iBakMQI", "reviewer": "Reviewer_Gsfw", "summary": "", "questions": "A: The paper mentions that existing approaches in HMER are limited by CFGs and pregenerated triplet data. Could you provide more insights into how these limitations affect the scalability and adaptability of current methods, and how RLFN overcomes these issues?\n\nB: Could you discuss the training process and data requirements for the Recognition and Language Fusion Network (RLFN)? How does the integration of recognition and language features affect the overall training complexity and computational demands?\n\nC: In the limitations you mention the late fusion of the recognition module and the language module. Is there any proposal how to improve the interaction between both?\n\nD: The final loss function is defined as the sum of the cross-entropy loss of the recognition prediction sequence probability and the the corrected prediction sequence probability with respect to the ground truth. Could you explain if you could observe a prioritization of one of them, or if a weighted with hyperparameters could help improve the  performance of your approach?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review.\n\n\n\n**Responses regarding code, data, question B, and question D:**\n\nData, code, and weight files were packaged within the **Supplementary Materials** when the article was submitted. After downloading MathBERTa from huggingface, the provided weight files can be used to reproduce the results on the three test sets.\n\nIn Section 5.1, we detailed critical parameters and specifics that have a direct bearing on the results. Notably, the GPU/batch size/epoch/time/optimizer settings are: one 3090/8/200/16 hours/Adadelta. The learning rate ascends from 0 to 1 during the first epoch and thereafter decays to 0, following the cosine schedules. No data augmentation for fair comparison. And all images within a batch are filled in the upper left corner of the canvas of the same size. Due to memory limitations, the canvas size does not exceed 1280 * 280, and any excess images will be discarded. Other hyperparameters can also be directly viewed in the code.\n\nFor detailed discussions related to datasets and the application layer, please refer to our response to Reviewer jmUa. On the whole, the CROHME dataset is characterized by its limited training data and intricate structures, factors that may contribute to overfitting and weak generalization. Theoretically, furnishing supplementary information might be a viable solution. Our RLFN incorporates two means to provide supplementary information. On one hand, we introduce domain knowledge through MathBERTa, freezing its parameters and treating it as prior knowledge. On the other, we jointly optimize the recognition prediction sequence and the predicted sequence resulting from combining visual and linguistic features, hoping the model can strike a balance between these two sources of information.\n\nPractically speaking, the added training parameters come only from the fusion module. When the hidden layer dimension of the fusion module is set at 256, the increase in parameters is limited, increasing the model's size from ~4.7M to ~5.1M. The actual training time for a single epoch increases by about 20%, but the actual training difficulty decreases, making it easier to fit the training set. Regarding the training results, we noticed that the primary enhancement over the baseline is attributed to the joint optimization, directly enhancing the recognition module's predictive accuracy. The gains from the fusion module's prediction corrections vis-à-vis the recognition model's outputs are comparatively marginal.  We attempted to improve the effectiveness of 'correction' by using methods including weighting loss with a ratio of 1:2 and increasing the hidden layer dimension of the fusion module to increase model capacity. The experimental accuracy of several parameter combinations remained between 55%-57%, and we also failed to see a significant improvement in the fusion module prediction comparing to the recognition module prediction.\n\nMoreover, in ablation study, where we abandoned the fusion module and jointly optimized the recognition module predictions and language modules' ensuing predictions. We noticed a similar trend, where the primary enhancement is attributed to the joint optimization, directly enhancing the recognition module's predictive accuracy.\n\nIn summary, regarding the source of model improvement, we can essentially conclude that semantic information plays a role similar to 'aware', akin to those methods based on tree decoders. It serves as auxiliary information or a means of regularization to jointly optimize the recognition model, rather than enhancing it through a two-stage approach of first recognizing then correcting. This aligns with the aim to avoid the hallucination issue of post-correction in mathematical formulas being often questioned.\n\nHowever, it's worth noting that the fusion module uses visual features as input. Removing the loss from the recognition module means it can't guide its classification head and serve as input to the language module. This makes it impossible to directly ablate the effects of joint optimization to validate our perspective.\n\n\n\n**Response for question A:**\n\nFirstly, current tree decoders and works related to CFG develop independently, making continuous exploration challenging. The reasons for this include that the complex CFG design prevents them from learning from each other themselves; The pre constructed triplet of data conversion makes it difficult to apply other methods to experiments due to non-standard data.\n\nFurthermore, they neglect challenges related to visual ambiguity. This stems both from the inherent nature of being context-free and their approach of translating structural symbols into relations and distinguishing them from general symbols, directly limiting their capacity for contextual modeling.\n\nOur RLFN, being based on a linear decoder, offers a certain degree of scalability. However, as discussed in our limitations section, there are certain challenges related to bidirectional decoders.\n\nRegarding the issue of visual ambiguity, it involves our viewpoint in Section 3.1:\n\n1. The presence of an explicit symbolic structure allows the LM to directly model the structure.\n2. Based on contextual semantics, visual ambiguities between categories can be corrected.\n\nAdditionally, it's worth noting that structural symbols are not directly represented in images and are regarded as a special kind of symbol. This brings attention alignment issues into play during the decoding process. Both the current tree decoders and linear decoders don't differentiate in the context of images, which could be a primary reason for attention drift. However, this hypothesis has not yet been substantiated in our research.\n\n\n\n**Response for question C:**\n\nLate fusion methods, in comparison to post-correction, merge features instead of sequences, retaining some information from the predicted sequences of the recognition module. Similarly, during language modeling, while the predicted sequence is still fed as input, the predicted probabilities of the recognition sequence inputted into the language module are overlooked, leading to information loss.\n\nHowever, fusing information at every step of decoding can evidently prevent this loss of information, but it would incur a prohibitive computational cost.\n\nThis might immediately bring to mind a series of Visual Language Processing (VLP) tasks, an area I haven't delved deeply into.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well structured, well written and it provides a clear explanation of the proposed approach. It also contains ablation studies, case and improvement studies to demonstrate the effectiveness of the RLFN method in HWER.\n\nThe tables, summaries and graphs are very helpful to understand the paper.\n\nThe model tackles a relevant challenge which is handwritten text recognition applied to mathematical expressions and LaTeX, achieving SOTA results in various COHRME datasets.\n\nThe potential of this approach could be beneficial for other areas, such as the standard handwritten text recognition or image captioning.", "reasons_to_reject": "The data, code, method are neither made publicly available nor do the authors state that they plan to do so. So there is no possibility to check the claims the authors made about the effectiveness of their method.\n\nMore details about the hyperparameters would have been helpful to better understand the details.", "questions_to_authors": "A: The paper mentions that existing approaches in HMER are limited by CFGs and pregenerated triplet data. Could you provide more insights into how these limitations affect the scalability and adaptability of current methods, and how RLFN overcomes these issues?\n\nB: Could you discuss the training process and data requirements for the Recognition and Language Fusion Network (RLFN)? How does the integration of recognition and language features affect the overall training complexity and computational demands?\n\nC: In the limitations you mention the late fusion of the recognition module and the language module. Is there any proposal how to improve the interaction between both?\n\nD: The final loss function is defined as the sum of the cross-entropy loss of the recognition prediction sequence probability and the the corrected prediction sequence probability with respect to the ground truth. Could you explain if you could observe a prioritization of one of them, or if a weighted with hyperparameters could help improve the  performance of your approach?", "paper_topic_and_main_contributions": "The paper focuses on Handwritten Mathematical Expression Recognition (HMER), converting images of handwritten math expressions into LaTeX sequences. Current HMER methods employ attention-based encoder-decoder architectures with tree decoders to capture expression structure. However, these methods rely on Context-Free Grammars (CFGs) and pregenerated triplet data, which, based on the authors claims, limit scalability and neglect visual ambiguity challenges.\n\nThe study examines unique language traits in LaTeX math expressions: explicit structural symbols and symbols (especially letters) as minimal units with context-dependent semantics. The authors suggest language models can provide both structural and semantic info, aiding HMER correction.\n\nThey propose a new architecture, Recognition and Language Fusion Network (RLFN), that integrates recognition and language features to generate corrected sequences. \n\nReported experimental results on CROHME datasets highlight RLFN's higher performance. This research's key contribution lies in the fusion of recognition and language features, which could advance HMER and lays groundwork for broader math-related handwriting recognition and language understanding.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "l. 035: crossroad\nl. 255: not a radical expression\nl. 394: the recognition module\nl. 549: wich might be because"}, "extra_text": {}}, {"review_id": "C0SihChNps", "reviewer": "Reviewer_xgEb", "summary": "", "questions": "I have no questions.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review.\n\nIn the article, we only discussed the issues related to post-correction-like architectures and addressed the potential information loss of our current framework in the limitations section.\n\nMore article space has been devoted to the characteristics of Latex mathematical expressions and their impact on LM.\n\nUndoubtedly, superior architectures exist, and we hope to further explore them in subsequent research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written, combines SOTA models in an innovative approach and improves partially the SOTA performance with respect to this special task. It avoids very specialized methods which have been proposed earlier and which are complicated to enhance to other situations. The new proposed network, its implementation and the optimization process are also well described.", "reasons_to_reject": "Both components of the proposed fusion network are already known. The combination is novel but may be other possible combinations and architectures may perform even better but have not been discussed by the authors.", "questions_to_authors": "I have no questions.", "paper_topic_and_main_contributions": "The paper adresses the problem of correcting the output of handwritten mathematical text recognition (which generates LaTeX output from images). The authors develop a model which does not rely on complicated structure focused methods but instead use an already introduced recognition module and a fine-tuned language model for LaTeX expressions (MathBERTa). A novel fusion model is introduced which is able to correct the output of the LaTeX sequences by utilizing structural and semantic information inherent in the language model and thus tries to solve the problem of visual ambiguity.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "-", "Typos_Grammar_Style_and_Presentation_Improvements": "-"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All four reviewers felt generally positively about the paper, citing the usefulness of the task, the interestingness of the idea, promising results, and wide comparison with other methods., and potential application to other tasks. One reviewer, jmUa, had a concern about presentation -- that the paper was only clear after reading it in entirely -- but other reviewers felt the paper was well-written. Reviewer W2zF had a concern about relevance of the task (parsing an image into latex code) to the conference, but seemed convinced by the author response and still felt positively about the paper. Overall this paper looks to make an interesting contribution on a well-motivated task, with thorough experiments and strong results.
Decision: Accept-Main