A TWO-STAGE FRAMEWORK FOR MATHEMATICAL EXPRESSION RECOGNITION
Authors: Jin Zhang
Weipeng Ming
Pengfei Liu
Venue:  (2020)
Keywords: mathematical expressions recognition
seq2seq model

Abstract: 
Although mathematical expressions (MEs) recognition have achieved great progress, the development of MEs recognition in real scenes is still unsatisfactory. Inspired by the recent work of neutral network, this paper proposes a novel two-stage approach which takes a printed mathematical expression image as input and generates LaTeX sequence as output. In the first stage, this method locates and recognizes the math symbols of input image by object detection algorithm. In the second stage, it translates math symbols with position information into LaTeX sequences by seq2seq model equipped with attention mechanism. In particular, the detection of mathematical symbols and the structural analysis of mathematical formulas are carried out separately in two steps, which effectively improves the recognition accuracy and enhances the generalization ability. The experiment demonstrates that the two-stage method significantly outperforms the end-to-end method. Especially, the ExpRate(expression recognition rate) of our model is 74.1%, 20.3 percentage points higher than that of the end-to-end model on the test data that doesn’t come from the same source as training data.

Under review as a conference paper at ICLR 2020
A TWO-STAGE FRAMEWORK FOR MATHEMATICAL EXPRESSION RECOGNITION
Anonymous authors
Paper under double-blind review
ABSTRACT
Although mathematical expressions (MEs) recognition have achieved great progress, the development of MEs recogmition in real scenes still unsatisfac- tory. Inspired by the recent work of neutral network, this paper proposes a novel two-stage approach which takes printed mathematical expression image as input and generates LaTeX sequence as output. In the first stage, this method locates and recognizes the math symbols of input image by object detection algorithm. the second stage, it translates math symbols with position information into LaTeX sequences by seq2seq model equipped with attention mechanism. In particular, the detection of mathematical symbols and the structural analysis of mathemati- cal formulas are carried out separately in two steps, which effectively improves the recognition accuracy and enhances the generalization ability. The experiment demonstrates that the two-stage method significantly outperforms the end-to-end method. Especially, the ExpRate(expression recognition rate) of our model is 74. %,, 20.3 percentage points higher than that of the end-to-end model on the test data that doesn't come from the same source training data.
INTRODUCTION
Mathematical expressions (MEs) play an essential role in math, physics and many other fields. Recognizing mathematical expressions is receiving increasing attentions for application in digiti- zation and retrieval printed documents. The process of recognizing mathematical expressions is to convert mathematical expressions into LaTeX strings, which includes three stages: symbol segmentation, symbol recognition and structural analysis. We usually divide recognition of MEs into handwritten and printed domains. In the domain of printed MEs, researchers face three chal- lenges{Anderson) (967); Belaid & Haton (1984)): the complicated two-dimensional structures, var- ious styles of images in printed input and strong dependency contextual information.
Three major problems are involved in MEs recognition (Zanibbi et al. (001)); Mouchre et al. (016)): symbol segmentation, symbol recognition, structural analysis. These problems can be solved sequentially or globally. [Deng et al. (2016a) proposed an end-to-end formula recognition method for images generated directly from LaTeX code. For their method, CNN is applied to tract visual features from the input images and then every row is encoded using a recurrent neural network (RNN). These encoded features are then used by an RNN decoder with visual attention mechanism to produce final outputs.
+
6
m
(130 (1+ 70%)
2
3
b
a
b
3
x
2
1
n
(a)
(b)
(c)
(d)
Figure I: ME images from real world
Their model achieved 77. 77.46% accuracy test data which had the same distribution the training data. For convenience, we define homologous test data and non-homologous test data here. is called homologous test data if the test data comes from the same source the training data.

---

Under review as a conference paper at ICLR 2020
decoder
second stage:
Attention
transkation
encoder
25
The first staghs location and dlsviliattion
YOLO3
pu
+ 12--22--3
Figure 2: Two-stage method: the first stage to use YOLOv3 to detect the symbols, the second to translate the symbols with position information into LaTeX strings by seq2seq model.
Otherwise, is called non-homologous test data. The result demonstrated that the model had good performance homologous test data. However, in practice we found that the model had poor performance on non-homologous test data, that is, the generalization ability of the above model is very weak. In real scenes, the images may be of great variety, like wide range of backgrounds, as is shown in Figure [ It is impossible to predict all input cases and the test data may diverge from what the system has seen before. Therefore, the method above to recognize images from real scenes. It seems necessary task to design model that has strong generalization ability to recognize real MEs images.
appropriate to be applied
To improve the generalization ability, we decouple the feature extraction process and the translation process. In the feature extraction process, we use YOLOv3(Redmon & Farhadi 2018) to locate and classify the symbols of images. Then the class and location information of each symbol is vectorized, which is used as input for the seq2seq model and translated into LaTeX strings.
The two-stage approach has several benefits. Firstly, changes in the input images styles would have no influence on the encoder-decoder model, and YOLOv3 has good generalization ability (Redmon) al. 20155 Redmon & Farhadi, 2018), the two-stage method would have better generalization ability than the end-to-end (enn et al., (0166). Secondly, the feature vectors composed by position and classification information are much more concise than the feature vectors extracted directly by the convolutional Mhyers(enn et al. 2016b), which are easier to learn and get higher recognition rate.
The main contributions of this paper can be summarized as:
(1) A two-stage method for MEs recognition is proposed to decouple the feature extraction process and the translation process, which has better generalization ability and achieve better accuracy.
(2) By concatenating position information and classification information into feature vectors, we successfully translate symbols with position information into LaTeX strings by the q2seq model. To the best of our knowledge, we use the seq2seq model to solve structural analysis problem for the first time and achieve satisfactory results, which may accelerate progress in machine recognition of other two-dimensional languages.
(3) We propose a method automatically generate MEs images with position and classifica- tion information of the symbols, which avoid expensive manual annotation.
2 RELATED WORK
The detection framework based on tfate-of-the-an convolutional neural network (CNN) can be di- vided into two categories: one-stage method and two-stage method. The typical two-stage methods include R-CNN(Girshick (0044)) Fast R-CNN(Uijlnngs al. (013)), Faster RCNN (Ren et al.
2

---

Under review as a conference paper at ICLR 2020
2017 and R- R-FCN al. (2016)), which firstly utilize the region proposal generation algorithms to generate potential symbol regions and then perform classification on the proposed regions. These methods improve the accuracy, but yield slow processing speed. On the contrary, single-stage meth- ods like SSD(Liu et al. (2016)) and (Remon et al. (2015). ). apply predefined sliding default boxes of different scales accuracy.
multiple feature maps, thus keep balance between speed and
Aiming to structural analysis, some papers prefer approaches based on predefined grammars solve the problem. For example, [avirotte| (1998) investigated graph grammar and graph rewriting as a solution to recognize two dimensional mathematical notations. Chan & Yeung (200T) developed system based on definite clause grammars(DCG), which incorporated an error detection and cor- rection mechanism into a parser. Huang (0007) proposed structural analysis approach based on the Attribute String Grammar and the Baseline Tree Transformation approaches. [Ivaro et al. (2014) introduced hidden Markov models to recognize mathematical symbols and stochastic context-free grammar to model the relationship between these symbols. Yamamoto et al. (2006) et al. pro- posed a new two-dimensional structure model for mathematical expressions by the new concept of Hidden Writing Area (HWA). Maclean & Labahn (2015) presented system, which captured all recognizable interpretations of the input and organized them in parse trees by a Bayesian scoring model. [Ivaro al. (2016) described statistical framework of grammar-based approach that deal with structured problems by two-dimensional grammars.
Some papers introduced an end-to-end (Tao et al. (2033); Cirean et al. (2000): Deng al. (2016a) framework totally based on neutral networks for mathematical expressions which did not require a predefined math grammar. Zhang et al. (2017b) proposed a model namely Watch, Attend and Parse (WAP), which had two components: a watcher and a parser. In Zhang et al. (2017a) the authors improved the CNN encoder by employing a novel architecture called densely connected convolutional networks (DenseNet)(Gao et al. (2016)).
NETWORK ARCHITECTURE
In this section, we will give detailed introduction about the two-stage framework proposed in this paper for printed mathematical expressions recognition. In the first stage, the main task is to locate and recognize the math symbols of input images by YOLOv3, In the second stage, the task is to translate the vector containing symbols categories and positional information into LaTeX strings. the classic attention mechanism based on encoder-decoder framework and
We will elaborate explain the process of parsing. As illustrated in Fig. [ the detection is YOLOv3, which generates series of symbol sequences with location information from the whole MEs images. Then these information including symbols' location and classes are processed and converted into vectors as input to the seq2seq model. Finally, the output of the decoder is LaTeX strings.
3.1 DETECTION: DOLLC
Symbol detection in formula images are more complex than other natural scene object detection due its smaller contours and various shapes. There are many small symbols. like '., '-', and subscript and superscript symbols, which may be lossed in symbol detection. Besides, some symbols are very similar, like digit '1' and letter 'I', which might cause error classification. Error detection will seriously affect the parsing in next stage, thus reduce the final formula recognition accuracy.
Among many object detection algorithms, YOLOv3 has proved to be the most effective in detecting small objects. YOLOvv3 has integrated advanced conception from other object detection algorithms like feature pyramid networks (L.in et al. 2016) and anchor box prediction mechanism (Redmon & Farhadi 2017). It uses entire image input of the network and draknet-53 without fully connected layer to extract feature, and finally predicts the bounding box of object and its category directly at the output layer. In the part of predicting boxes, it extracts features from three different scales and merges up-sampled feature map with earlier feature map, thus get better performance in small object detection.
We adjust the size of the anchor box to reduce missed detection. Especially, anchors like (10,10), (6,40) and (40.6) are added to detect points, horizontal or vertical lines. For similar symbols like
3

---

Under review as a conference paper at ICLR 2020
digit 'I' and letter 'I', it is almost impossible to be distinguished by the model. Such errors might be corrected if extra rules are added as postprocessing.
3.2 PARSE:ENCODER-DECODER
lot of grammar- based parsing algorithms have been proposed for structural analysis and per- formed well in several systems (Ivaro et al. (2014); Awal et al. (2014); lvvao et al. (2016)). But grammar-baser methods requires priori knowledge deñine ME grammar and its complexity in- creases exponentially with the size of the predefined grammar.
The encoder-decoder framework was first presented in the paper(Choo et al. (2014)) which was used to translate one language into another. Structural analysis is to parse symbols with position infor- mation into LaTeX sequence, which similar to translation. The only difference between them is that the former is two-dimensional and the latter is one-dimensional. By creatively concatenat- ing position information and classification information as input vectors, we successfully employed encoder-decoder model to perform structural analysis of MEs.
3.2. ENCODER-DECODER
The LSTM is an improved version of simple RNN, so we employ LSTM as the encoder for al- leviating the vanishing and exploding gradient problems(Bengio et al. (2002)). In encoding part, the input of each time step is symbol with position information, represented by feature vector (x, y: w. h, o-h) where (z. y, w, h) are the coordinates of the top-left corner, width and height of the rectangle that contains the symbol, and o-h is one-hot vector to distinguish the symbol from others.
Similarly, we employ lstm as the encoder. In the decoding stage, we aim to generate a most likely LaTeX string given the input feature vectors:
arg max log P(y|æ)
(1)
In fact, the sequence is closely related to the semantic vector C generated by the encoder, and y one time also is affected by y generated other times, which can be described as the following formula:
y/2 argmaxP (y.) II p (vel (V1. .............. M.....,,,.)..
In addition, it intuitive to adopt the ensemble peethod(Ditteriii (2000)) for improving the perfor- mance of beam algorithm.
3.2.2 ATTENTION
Although, the encoder-decoder model has been widely applied to many problems, its biggest lim- itation is that the only link between encoding and decoding is a fixed-length semantic vector C. However, this semantic vector may not fully represent the information of the whole sequence, and the information carried by the first input will be diluted by the later input information. The accu- racy of model is also affected by inadequate input sequence information. To solve this problem, Bahdanau al. (2014) proposed attention model that generates an "attention range" when output generated, indicating which parts of the input sequence should be paid attention to in the next output, and then generates the next output according to the region of interest.
Between encoder and decoder, we introduce attention mechanism. In standard attention Bahddnnau et al. 20144)), we compute a context vector candidate è as:
Where hy is the hidden state encoder and aij is the correlation coefficient. We use neural network to approximate the attention distribution any
4

---

Under review as a conference paper at ICLR 2020
exp (ees)
Oij = softmax (wt)
Ek= exp (e(k)
(4)
Ct = u tanh (W (Watthi aattaa)
Where ezi denotes the energy of annotation vector a, at time step conditioned on the current Istm hidden state prediction ht
EXPERIMENTS
To validate the effectiveness of the proposed method for printed mathematical expression recogni- tion, we design set of experiments to evaluate the answers of the following questions:
Q1 Is the improved YOLOv3 for locating and recognizing mathematical symbols effective?
Q2 How does incodr-ddccoer analyze the 2D structure of MEs?
Q3 Does the proposed approach outperform state-of-the-arts?
4.1 METRIC
Metric of detection model: For detection tasks, IoU (Intersection-over-Unniese is a standard metric to measure performance. In this paper, detections are considered true or false positives based on IoU. A detection is to be considered correct if the loU exceeds 50% and the classification is right.
Metric of translation model: For the translation model, we use accuracy and Word Error Rate( (ER Klakow & Peters (2002) metries to measure the performance. The accuracy is caleu- lated based on expression recognition rates (ExpRate). i.e., the percentage of predicted mathematical expressions exactly matching the ground truth, which gives a useful global performance metric.
4.2 DATA
Data with classification and position information expensive annotate and rare. To address this problem, tool is developed to provide detailed ground-trute annotations, which are cheap and more precise than data annotated manually. After filtering some inappropriate equations IM2LATEX- 100K (Deng et al., 2016b), data set containing 81214 equations is obtained to generate mathe- matical expression images. Besides, 6000 background images are collected and split into two parts: one part contains 4000 images, which are used as background of training set, validation set and the first test set; the other part 2000 images are used as background of the second test set. The two test sets generated by the same LaTeX equations but with different background. The first test data set, called homologous test set, has the same distribution as the training set. The second test set, called oon-homologous test set, doesn't have the same distribution as the training set, which used to evaluate the generalization ability of the models. For details to generate data, see Appendix A
4.3 DETECTION MODEL EXPERIMENTS
We train the model using minibatch SGD with four GPUs, 0.9 momentum, 0.0005 weight decay, and batch size 64. We first train the model with 0.0005 learning rate for 20000 iterations, and then continue training for 4000 iterations with 0.0001 and 1000 iterations with 0.00001. The anchors we use are (10, 100,0)) (6,40), (40,40), (40.6), (80,80) (02, (120,120), (100,12) 12), (228,228). Especially, anchor (10,10) is designed to detect small symbol like '." and '., and the anchors (6,40), (40,6), (00,, 120 to detect horizontal or vertical lines, like "-",
or 'l'
Figure 3 gives example of YOLOv3. Testing on 5000 images of the homologous test set, there are 4822 images that all the symbols are correctly located and recognized. Besides, location emrors occurs in 125 images and classification
occur in 62 images, among which there are images that contain both location and classification errors. e Testing on 5000 images of the non-homologous test set, there are 4813 images that all the symbols are correctly located and recognized. In addition,
5

---

Under review as a conference paper at ICLR 2020
A V
A
S
S
-
Figure 3: An example of YOLOv3.
location errors and classification emrors occur in 126 images and 72 images, respectively, The images that contain both location and classification errors are 1
Table 1 shows the testing results of both homologous and non-homologous test set, which are al- most identical. Although the detection model has never 'seen' the background images of the non- homologous test set, it still achieves good performance. This demonstrates the detection model has good generalization ability, which are very important in practical applications.
Table 1: Testing result on homologoussnonn-oooollgg test set
Test Set
NO Error
Loe Errors
Cls Errors
Loc and Cls Errors
Homologous
96.44%
2.5% 2.52%
1.24% 1.44%
0.18% 0.22%
Non-homologous
96.26%
4.4 TRANSLATION MODEL. EXPERIMENTS
We regard the conversion of symbol sequences with location information into LaTeX sequences as a translation process and solve this problem with the encoder-devoder model, which is of great significance. As far know,
the first to the encoder-decode model to solve the two- dimensional translation problem. As is shown Section 4.2] we have 81214 different LaTeX math equations, 71214 expressions for training, 5000 for validation and 5000 for test. In order train the encoder-decoder model, we run our YOLOv3 model on the training dataset, validation dataset and test dataset, respectively and store the symbols' position and classification information of every image. After filtering the wrong cases, we obtained 69405 training expressions, 4826 validation expressions and 4822 test expressions.
The encoder is bi-directional LSTM. The hidden state of the decoder is of size 512, encoder LSTM of 256, which are the same the configuration the RNN layers in Im2LaTex (Deng et al. 20166). The model was trained using SGD with batch size 64 for 120 epoches. The initial learning rate is 0.001, and reduced to 0.0001 after 100 epoches. Besides, the input symbols are ordered by the upper-left corners of their boxes from left to right, top to bottom. For example, the input order of the symbols is m, k, c, =, v. 3, T for expression me? = /31.
Since have only 69405 mathematical expressions train the encoder-decoder model, how to avoid overfitting is great challenge. To address the problem, we use extensive data augmentation. Firstly, we replace symbols with other symbols to get more mathematical expressions. For example, we replace the 'a', 'b', "c' and '2' in a² 12 = e² with four other random chosen symbols 'c', ''g, 'I' and '4', thus obtain new equation e + g 14, For position information, we introduce random scaling and translations. For each training epoch, up to 80% of the 69405 mathematical expressions would be augmented by the strategy above.
Data augmentation plays an important role in training the encoder-decoder model. The experiment shows that we can improve12.88 12.8% ExpRate with the augmentation strategy above. Why does the encoder-decoder model benefit so much from the augmentation strategy? One reason may be that the data is not enough to train a good seq2seq model and the feature vectors one-hot vectors) too concise, which don't contain any noise compared to the feature vectors extracted by the convolution layers in Im2LaTex (Deng et al. 2016b).
In Table the performance of standard attention model is clear to be observed. For homologous test set, the ExpRate 78.0% and the wer is 4.4%. Besides, the result of non- non-hooologous test set is almost the same as that of homologous test set. It understandable because the input of the encoder-

---

Under review as a conference paper at ICLR 2020
Table 2: Testing result of the encoder-decoder model
Test Set
WER
ExpRate
1%
S 2%
Homologous 0.044
78.0%
83.1%
85.9%
Non- Non-homologous
0.045
77.9%
83.1%
85.7%
decoder model is the position and classification information of symbols, not the image. Obviously, changes in image style has little influences on the encoder-decoder model.
(a) (x) = 2Φ(x) + 26( +
(b) Lo ((w) = (
Stag
104r
(c)
A-B A+
<
C
A+ A
(d) me?
/T
Figure 4: Four sample alignments by the encoder-decoder model. The x-axis and y-axis of each plot the symbols with position information and the generated translation (LaTeX),, respectively. Note that we omit the position information here. Each pixel shows the weight a, of the annotation of the j-th source word for the i-th target word, in grayscale (0: black, 1: white)
We would inspect the alignment between the symbols (with position information) in source tence and the words generated LaTeX by visualizing the annotation weights æ; from Eq. (4), in Fig. Each row of matrix in each plot indicates the weights associated with the annota-

---

Under review as a conference paper at ICLR 2020
tions. From the matrix generating the target LaTeX.
which positions in the source sentence
important when
From the Fig. 4 we can conclude that the alignment between the symbols with position information and the target LaTeX is monotonic. In general, weights along the diagonal of each matrix are closer to However, there are also many non-trivial, non-monotonic alignments. It's obvious that the alignments depend on the spatial relationships between the mathematical symbols, which might be horizontal, vertical, subscript, superscript or inside. Fig. 4a shows a strictly monotonic alignment since all the symbols are horizonal relationship. Vertical, subscript, superscript or inside relationships are usually ordered differently between the imput symbols and generated LaTeX. When
me² (vertical and superscript), a target LaTeX contains several different spatial relationships, like à: the weights seem disorderly in some degree. In Fig- Fd,, the symbol 'k' of the input sequence is ahead of the symbol 'c', but the encoder-decoder model was able to correctly align the symbol "c" with numerator and the symbol 'k' with denominator, which show that the model successfully captured the position information of the symbols. Fig. (c))(d) demonstrated the encoder-decoder model can handle complicated too-dieensional structure.
4.5 EVALUATE AS A WHOLE
The current state-of-art OCR-based mathematical expression recognition system is Im2LaTex(Dent al. 2016b), an end-to-end recognition system, which combines symbol recognition and structural analysis as a whole. In this part we run experiments comparing our model to Im2LaTex. To make a fair comparison, we downloaded the code they exposed GitHub and trained with the same data set.
Table 3: Testing result of the homologous test dataset
Model
WER
ExpRate
< 1%
< 2%
Ours
0.048
74.3%
79.7%
82.8%
Im2LaTex
0.054
73.37%
78.5%
80.9%
Table 4: Testing result of the non-homologous test dataset
Model
WER
ExpRate
< 1%
2%
Ours
0.050
74.1%
78.21 %
81.5%
Im2LaTex
0.1355
53.9%
60.9%
64.56%
In T we can observe that our model achieved an ExpRate of 74.3%, while its WER was only 0.048 for homologous test set, which is slightly better than Im2LaTexx However, Table shows the results of the non-homologous test dataset. The ExpRate of our model 74.1 %, 20.3 percentage points higher than Im2LaTex. The < 1% eттоr percentages 78.21%) and < 2% error percent- ages (8.5%) of our model is also much higer than Im2LaTex. From these indicators, our system outperforms Im2LaaTex for homologous test dataset, and is evidently better than Im2La'Tex for non- homologous test dataset. So our model has better generalization, which is superior to Im2LaTexx for recognizing MEs with real backgrounds.
CONCLUSIONS
In this study, we introduce two-stage framework to recognize mathematic expressions. It is the first work that employs YOLOv3 to detect mathematic symbols and uses the seq2seq model based attention to perform structural analysis. We demonstrate through experiment results that the novel two-stage framework has better generalization ability and performs better than the state-of-the-ar methods in real scenario. In future work, plan to further improve the accuracy of parser by adding more data and introducing more advanced mechanisms, thereby improving the accuracy of the whole mathematical formula recognition system.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Robert H. Anderson. Syntax-directee recognition of hand-printed two-dimensional mathematics.
Interactive Systems for Experimental Applied Mathematics, pp. 436-459, 1967.
Ahmad Montaser Awal. Harold Mouchre, and Christian Maud-n. A global learning approach for an online handwritten mathematical expression recognition system. Puttern Recognition Let- ters, 35(1):68-77) 2014.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. Computer Science, 2014.
Belaid, A. and J P Haton. A syntactic approach for handwritten mathematical formula recognition. IEEE Transactions on Pattern Analysis & Machine Intelligence, 6(1):105-11 1984.
Bengio, Y... Simard, P., and Frasconi, P. Learning long-term dependencies with gradient descent is difficult. IEEE Trans Neural Netw, 5(2):157-166, 2002.
Kam Fai Chan and Dit Yan Yeung. Error detection, error correction and performance evaluation in on-line mathematical expression recognition. Puttern Recognition, 3461):1671-1684 2001.
Kyunghyun Cho, Bart Van Merrienboer, Caglar Gulcehre, Fethi Bougares, and Yoshua Bengio. Learning phrase representations using mn encoder-decoder for statistical machine translation. Computer Science, 2014.
D. C. Cirean, U Meier, L. M. Gambardella, and Schmidhuber. Deep, big, simple neural nets for handwritten digit recognition. Neural Computation, 22(12):3207-3220, 2010.
Jifeng Dai, Li Yi, Kaiming He, and Sun Jian. R-fen: Object detection via region-based fully convo- lutional networks. 2016.
Yuntian Deng, Anssi Kanervisto, and Alexander M. Rush. What you get what you see: visual markup decompiler. 2016a.
Yuntian Deng, Anssi Kanervisto, and Alexander M Rush. What you get is what you see: A visual markup decompiler. arXiv preprint arXiv:1609.04938 10:32-37, 2016b.
Thomas G. Dietterich. Ensemble methods machine learning. Proc International Workshgp on Multiple Classifier Systems, 1857(1):1-15, 2000. 2000.
Huang Gao, Liu Zhuang, and Kilian Q Weinberger. Densely connected convolutional networks. 2016.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for ac- curate object detection and semantic segmentation. In IEEE Conference on Computer Vision & Pattern Recognition, 2014.
B. Q. Huang. A structural analysis approach for online handwritten mathematical expressions. 2007.
Dietrich Klakow and Jochen Peters. Testing the correlation of word error rate and perplexity. Speech Communication, 38(1):19-28) 2002.
Stphane Lavirotte. Mathematical formula recognition using graph grammar. Spie Proceedings, 3305:44-52, 1998.
Tsung Yi Lin, Piotr Dollr, Ross Girshick, Kaiming He, and Serge Belongie. Feature pyramid net- works for object detection. 2016.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng Yang Fu, and Alexander C. Berg. Ssd: Single shot multibox detector. In European Conference on Computer Vision, 2016.
Francisco Ivaro. Joan Andreu Snchez, and Jos Miguel Bened. Recognition of on-line handwritten mathematical expressions using 2d stochastic context-free grammars and hidden markov models. Patlern Recognition Letters, 35(1):58-66) 133887677606 2014.

---

Under review as a conference paper at ICLR 2020
Francisco Ivaro, Joan Andreu Snchez, and Jos Miguel Bened. An integrated grammar craamar-bbseed ap- proach for mathematical expression recognition. Pattern Recognition, 51:135-1477 147, 2016.
Scott Maclean and George Labahn. Bayesian model for recognizing handwritten mathematical expressions. 2015.
Harold Mouchre, Richard Zanibbi, Utpal Garain, and Christian Viard-Gaudin. Advancing the state of the art for handwritten math recognition: the crohme competitions, 201 12014. International
Journal on Document Analysis & Recognition, 19(2):13-188, 2016.
Joseph Redmon and Ali Farhadi. Yolo9000: Better, faster, stronger. In IEEE Conference Com- puter Vision Puttern Recognition, 2017.
Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv, 2018.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. 2015.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnm: Towards real-time object detection with region proposal networks. IEEE Transactions on Patlern Analysis & Machine Intelligence, 39(6):1177-1149 2017.
Patrick Rez, Michel Gangnet, and Andrew Blake. Poisson image editing. Acm Transactions on Graphics, 223))313-318 2003,
Wang Tao, David Wu, Adam Coates, and Andrew Y Ng. End-to-end text recognition with convo- lutional neural networks. In International Conference on Pattern Recognition, 2013.
Uijlings, R. R J., Van De Sande, E. K., Gevers, Smeulders, and W. M A. Selective search for object recognition. International Journal of Computer Vision, 104())154-171, 2013.
Ryo Yamamoto, Shinji Sako, Takuya Nishimoto, and Shigeki Sagayama. On-line recognition of handwritten mathematical expressions based on stroke- based stochastic context-free grammar. 2006.
Proc.int.workffoonters Handwriting Recognition Oct, pp.
Richard Zanibbi, Dorothea Blostein, R. Zanibbi, and D. Blostein. Recognition and Retrieval of Mathematical Expressions. 2012.
Jianshu Zhang, Jun Du, and Lirong Dai. A gru-based encoder-decoder approach with attention for online handwritten mathematical expression recognition. 2017a.
Jianshu Zhang, Jun Du Shiliang Zhang, Dan Liu, Yulong Hu, Jinshui Hu, Si Wei, and Lirong Dai. Watch, attend and parse: An end-to-end neural network based approach to handwritten mathematical expression recognition. Pattern Recognition, 71:196-2000 2017b.
A DATA
A.1 BACKGROUND IMAGES
(a) bgl.
(b) bg2.
(c) bg6.
Figure 5: Background images
To favour variety, 6,000 background images are collected from printed articles, books, examination papers and We collected lot of printed articles, books, examination papers and then took photos of them. We only cropped the non-text areas of the photos as background images. Figure shows examples of the background images.
10

---

Under review as a conference paper at ICLR 2020
A.2 DATA GENERATION
There are two kinds of mathematical symbols: single-component (like 'a', 'b', 'c', etc) symbols and multi-components symbols (like 'i', J". '=', etc). To get the position information, the multi- components symbols are rendered with specific colors. Then the single-component symbols are segmented by connected components and the colpi-ccmm symbols by colors. After obtaining the position information, the symbols were cropped and then classified by a neural network. Eventu- ally, we obtained a dataset with 81214 images and corresponding location and category information of symbols. Finally, the images are binarized and blended into the background images using Poisson image editing(Res et al. 2003). Figure 6 gives an example to show the process to generate data.
x + y*2 125 \div
generate: Image
155
location
binaryzation
L + = 125
adi background
125
x y 5
Figure 6: The process to generate data


Reviews:
[{"review_id": "r1lT36Q5cS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to recognise a mathmatical expression using a two-stage framework, including object detection by YOLOv3, and encoder-decoder based translation. The paper is written well, and easy to read. In the experiments, the recognition and translation methods both work well on Homologous and non-Homologous test data. In particular, the proposed method improved the performance over the state-of-the-art method Im2LaTex. Addtionally, the authors visualised the sample alignments of encoder-decoder model, which is helpful for understanding the method.\n\nA few comments are as follow:\n1) On Page 1, the first and second paragraphs both contain \"symbol segmentation, symbol recognition and structural analysis\". It looks this framework is repeated again and again.\n2) In formula (4), the reviewer did not see the explanation of \"v_{att}^T\" and \"u_T\".\n3) Some references have only authors and title information, without conference/journal information.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Syle2m0aKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the problem of converting an image of a math expression into LaTeX.  They note that while the model proposed in Deng et al works well on the IM2LATEX-100K dataset, is doesn't generalize well to equations in real-world settings that you'd have in a photograph or a scan of an equation.  They propose an approach that breaks the problem into two steps.  In the first step they detect all the characters in the image, identifying the  character type and bounding box for each.  In the second step they use an encoder/decoder (LSTM/LSTM with attention) model to translate this sequence of character encodings into a LaTeX sequence.  They create a new dataset of LaTeX equations rendered on backgrounds sampled from real photographs of books and papers.  They split this into a training and \"homologous\" test set.   They find performance of their two stage model is a bit better than Deng's model on this test set.  They then create a \"non-homologous\" test set, in which the test equations are rendered on a new set of backgrounds, unseen in training.  On this test set, the model in the paper performs essentially the same as on the homologous dataset, while the Deng model performs substantially worse.  The conclusion is that the two stage approach creates a model that is much more robust to the appearance of the equation.\n\nI think the main takeaway from this paper is that there can be disadvantages to an end-to-end approach.  To achieve their improvement, authors used their insight that there is an intermediate representation that summarizes all relevant information (the sequence of characters and positions), together with the ability to generate a new training set automatically to learn this intermediate representation.\n\nI think this is an interesting case study in applied machine learning, but I don't think it will be of enough general use or interest to the ICLR community to merit acceptance.\n\nAs an applications paper, I think there are several aspects that can be improved.  Here are some specific questions and comments that may help a further iteration of this paper:\n- You have examples of ME images from the real world, but you don't have any examples of your artificial \"real world\" equations, overlayed on your sampled backgrounds.  Those would be helpful. Are any other modifications done to the equation to simulate a real-life picture or scan, such as color or darkness distortions, angles, etc?\n- How does your proposed model perform on the original IM2LATEX-100K problem?\n- How well does the encoder-decoder model do an a gold encoding of the input?  It would be nice to separate the errors into translation errors vs object detection errors.\n- Can you give examples of scenarios where your model got things right and Deng's model did not, and vice versa?  Is there any interpretation to why each model does better or worse for various example?  I imagine the model may have more difficulty with equations where one needs to refer to characters that are on the left of the image quite late in the LaTeX expression.  For example, fractions with long numerator expressions and and cases environments.\n- Did you try or consider using a soft encoding of the character identity, instead of a one-hot?  Perhaps there are context clues that the encoder/decoder model could use to disambiguate between a 1 and an l, for example.  \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BJlB2ry9cB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper the authors propose a two stage pipeline that aims to solve for mathematical expression recognition. The main approach uses the following stages, a detection stage that is based on YoloV3 and a sequence to sequence approach. The authors compare their method against Image2Latex approach (2016) that is an end to end pipeline and show that there is significant improvement compared to this approach. \n\nHowever, this problem has been a standard task and solved both in the handwritten math expression problem (CROHME challenge of ICDAR and typeset formula detection and recognition. There have been much progress through these challenges with various teams competing. A variety of approaches have been tried for this task and unfortunately the present work has not compared nor evaluated against these approaches. \n\nIm2Latex work is quite old benchmark and there have been numerous works as have been cited by the authors and more as can be available from the challenge. The methods presented are also not novel. Using Yolov3 for detection and sequence2sequence for parsing expressions are more or less standard approaches. Hence, the proposed work does not add a significant insight in solving the problem. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: One reviewer is positive, while the others recommend rejection. The authors did not submit a rebuttal, thus the reviewers kept their original assessment.
Decision: Reject