Handwriting Interface for Computer Algebra
Systems
H. Okamura, T. Kanahori, M. Suzuki
Graduate School of Mathematics, Kyushu University
suzuki@math.kyushu-u.ac.jp
R. Fukuda
Faculty of Engineering, Oita University
rfukuda@cc.oita-u.ac.jp
W. Cong, F. Tamari
Faculty of Education, Fukuoka University of Education
tamari@fukuoka-edu.ac.jp
Abstract
This paper describes our experimental real time recognition system of handwritten mathematical expressions, linked to a computer
algebra system Mathematica by Mathlink. This system is titled as“INFTY”. The fundamental operations of INFTY are (1) input ofmathematical expressions using handwriting input system, (2) editingof them by basic editing operation (cut, copy, paste and delete) and(3) calculations and plotting graphs using Mathematica. INFTY also
supports the input and output by TeX format.
In the handwriting window, we adopt the automatic character
rewriting method. This method largely improves the accuracy of thethe structure analysis of the written mathematical expressions.

1Introduction

The recent developments in network and computer technology is allowing
us the prospect of adopting new approaches to both the research and theeducation of mathematics. For example, an electric white board connected1

by network to a computer algebra system or some mathematical data baseenables us to deliver a new style of lecture or discussion on mathematics.
However, it is true that the user interfaces of the current computer systems are not convenient to input mathematical expressions ([1][2]). There
are two typical methods to input mathematical expressions into a computer.
The ﬁrst one is based on menu operations. The user selects mathematicalsymbols from panels and change modes to input subscripts, superscripts, etc.
This method is intuitive and easy to understand for the user, but repetitious
menu operations sometimes prevent the smooth input of mathematical expressions and can disturb the train of thought of the user. The second method
uses only the keyboard to input mathematical expressions using syntax based
data formats, such as TeX notations or command formulations of computer
algebra systems, etc. This latter method often results in a smoother inputof mathematical expressions. However, some training is required to masterthe notations and it is not easy to understand the meaning of the written
expressions at a glance.
To solve this problem, we are developing a tentative real time recognition
system of handwritten mathematical expressions([3][4]). In order to test
the eﬃciency of our system, we linked it to the computer algebra system“Mathematica” by Mathlink. This experimental system is named “Infty”.
The fundamental operations of Infty are the followings. First, users write
mathematical expressions in a dialog box by hand using a mouse or a datatablet. Each handwritten mathematical expression is recognized and put into
the main board (display area) of Infty, transformed into the type character
form. The mathematical expressions written on the display area can be editedby the usual basic editing operations such as cut, copy, paste and delete, andthey can be evaluated, factorized, expanded or presented by graph, etc. using
Mathematica linked by Mathlink. The results of the calculation are put into
the clipping board of Infty, and can be pasted at any place of the display
area of Infty and edited freely. Infty also supports the input and output
by TeX format texts and can be used, for example, to edit mathematicalexpressions of TeX source by handwriting, making use of calculation resultsby Mathematica as the need arises.
In the handwriting window, each character is recognized at each step andautomatically rewritten in a regular form. This automatic rewriting method
considerably improves the accuracy of the structure analysis of the written
mathematical expressions.
Presently, Infty recognizes all mathematical expressions, except for matrices, at the high school level. The recognition of complicated expressions
at university level is left to future researche.
As our system is developed using object oriented programing designs,

it can be easily extended to support inputs and outputs using any otherformats of mathematical expressions, such as Maple format, MathML format
or Braille codes, etc. by overriding some primitive functions.

2Infty System

Infty is a handwriting interface for Mathematica, and by using this system
we can easily execute calculations or plot graphs. We are developing thissystem using Visual C++ compiler system (ver. 6.0). Therefore it can beused in Windows 95/98 systems in which Mathematica is installed.
Its main window consists of two areas: Display Area and Handwriting
Area. Mathematical expressions for the executions are inputted to DisplayArea, using a handwriting input system in Handwriting Area. Expressions inDisplay Area are displayed in type setting form. The calculation results arealso displayed in Display Area and these are also objects for the executions.
Details for Handwriting Area are described in Section 3. We explain theremainder parts of this system in this section.

Figure 1: Infty

2.1

Display Area and Clipping Area

In Display Area, we can edit mathematical expressions using ordinary editing
operations: cut, copy, paste and delete. To select a mathematical expression,

we surround the corresponding expression with the rectangle which is displayed when the mouse is dragged. We may also click the left edge of a line
to select the whole expression in the line. When the function cut or copy isexecuted, selected expressions are stored in the buﬀer of Infty, and displayedin the dialog box called ’Clipping Area’. The Clipping Area gets visible when
the ﬁrst expression is stored in the buﬀer. The Clipping Area is also used torepresent calculation results by Mathematica. Mathematical expressions in
Clipping Area, which are stored in the buﬀer, can be pasted to anywhere inDisplay Area.

2.2

Execution

Let us explain the menu items in the pull down menu ‘Execute’ following
the Figure 1. First, the expression is input by hand in Handwriting Area (a)Pushing the button OK, the recognized expression is displayed in Display
Area .
1
Suppose that the user selected whole of the ﬁrst line. Next, selecting
5 in Figure 1), the calculation‘Expand’ in the pull down menu ‘Execute’ (see 2If the user wants toresult by Mathematica is displayed in Clipping Area ().
continue the same operation ‘Expand’, the user may select ‘Execute’ buttonin the pull down menu or double click the left button on the expression inDisplay Area.
Each menu item in this pull down menu, except for ‘Execute’, ‘Add
Graph’ and ‘New Graph’, directly stands for the corresponding Mathematica
command. Figure 2 shows the results of the execution noted below “→” forthe expressions on the left hand side written in Display Area.

Figure 2: Examples of Execute2.3

Graphic Display Area

Suppose that the expression “2x3 + x2 − 1” ((c) in Figure 1) is selected.
Executing ‘Add Graph’ or ‘New Graph’, the graph of “y = 2x3 + x2 − 1”
4 in
is plotted in Graphic Display Area, which is a dialog box for graphics(Figure 1). We can change the range of the variable x, rewriting two numerals
in both side of ‘< x <’ in the upper side of Graphic Display Area (see theleft dialog box in Figure 1).
If there are already some graphs in Graphic Display Area, we use ‘New
Graph’ and ‘Add Graph’ as follows. When ‘New Graph’ is selected, the oldgraphs are cleared and only the new graph is plotted in the area. On theother hand, when ‘Add Graph’ is selected, the old ones remain plotted anda new graph is plotted on the same area. In both cases, the expression ofthe function will be listed in Graphic Display Area. In general, there areseveral functions in this list. We can select some of them by clicking on thecorresponding expressions, and remove them by ‘Delete’ button.
Graphic Display Area is not visible at ﬁrst. Using the menu item ‘Graphics Visible/Invisible’ in the pull down menu ‘Edit’, we can make this dialog
box visible or invisible.

Figure 3: Dialog box for Solve

2.4Solve

Let us explain the execution ‘Solve’ following Figure 3. Assume that theexpression “ax2 + bx + c = 0” is input and selected in Display Area. ‘Solve’
is also in the pull down menu ‘Execute’. Followed by the click of ‘Solve’, adialog box appears to determine the variable to solve the equation for (see 1in Figure 3). We can select the variable by clicking on the character of thevariable in the dialog box. Clicking OK button, we can obtain the solution
2 in Figure 3).
in Clipping Area (see 3

Handwriting Interface

User can input mathematical expressions by hand using a mouse or a datatablet in the lower window of the INFTY system. We call this window
Handwriting Area.
Our Handwriting Area has four menu buttons Clear, Delete, Undoand OK. When a user pushes the OK button, the mathematical expressions written on the Handwriting Area are recognized and inserted to theDisplay Area of Infty at the current position of its cursor. The behaviors forthe other buttons are described in a subsection below.

3.1

Automatic Rewriting Method

One of the characteristic points of our handwriting input system of mathematical expressions is the automatic character rewriting method. Each written character1 is rewritten by neat strokes in an appropriate size and position,
immediately and automatically.
There are several reasons for adopting this method. The distortion of theinput characters and the turbulence of the positions or the scales of charactersusually cause serious diﬃculties in the structure analysis of mathematicalexpressions, in which the positions and the scales of the characters havespecial meanings. An error of a character recognition or an error of thesegmentation of the strokes into character units leads sometimes to a fatal
error of the structure analysis of the mathematical expression. The labors
for the correction of this kind of errors disturb seriously the smooth input ofmathematical expressions.
Therefore, our automatic character rewriting method is introduced inorder to overcome this diﬃculty. In this interface, the user can identify each
recognition error immediately it occurs, and can correct it easily.

The rewriting of characters proceeds as follows.
1

Alphabets, numerals, Greek letters and other mathematical symbols are included.

The input characters are classiﬁed into two groups. One is a group ofcharacters which can be extended to other characters by adding some strokes(=extendable character), and the other is the group of unextendable characters. For example, ’F’ and ’C’ are extendable since these are transformed
into ’E’ and ’G’ by adding a stroke or strokes.
When a stroke is written, it is recognized as a character together with theuntreated strokes if they exist. In the case where the recognition result is an
unextendable character and the recognition cost is low (good) enough, theresult of the recognition is determined and the character is rewritten neatly
by computer. The strokes of the character are marked as treated strokes.
Otherwise, the system waits for the input of the next stroke.
Extendable characters are also determined either when some prescribedtime interval has elapsed with no input of new stroke2 , or the next stroke
is pushed down to a new character position3 . In this way, the determination of the recognition result proceeds automatically with no supplementary
operation of the user.
Since the determined character is rewritten using neat strokes into standard character form (and with the regulated size), the user can know therecognition result immediately. If the recognition result is wrong, the user
can choose the correct one among three candidates of the recognition resultsof the character by clicking the mouse. If the correct one is not in the listof candidates, the user can rewrite them after pushing the Undo button orthe Delete button described below.
Another advantage of this rewriting method is that the rewritten characters may serve as a model of the feasible character forms and the sizes
for the recognition. The user learns them subconsciously. Apart from thispedagogic eﬀect, it also leads the user to a neater writing style.

3.2

Menu Buttons of Handwriting Area

The Handwriting Area has four menu buttons Clear, Delete, Undo andOK. The operations corresponding to these buttons are as follows:
1. clear. Clear all the characters and undetermined strokes written on
the board.
2. delete. Delete last one character. If there is an undetermined character(untreated strokes), all the untreated strokes are cleared.
2
3The actual default value of this interval is 3 seconds.
namely, the pushed down position contradicts to the character extension

3. undo. If there is an undetermined character, delete last one stroke.
Otherwise, back the last character to the undetermined state.
4. ok. Recognize the structure of mathematical expression written on theboard, and send the result to the display area. The received result isinserted into the line of the display area at the current cursor position.

3.3

Restrictions of Expressions

The mathematical expressions which can be recognized by our system arelimited to the elementary ones such as fractions, square roots, subscripts,superscripts, integrals, limits, summations, etc., the expressions used in highschool mathematics or in the ﬁrst course of university mathematics. Thematrices are excluded at the moment.
These structure of mathematical expressions may have the nested structures: The denominator and numerator of the fraction, for example, may
have the structure of fractions, square roots, etc., the superscripts also may
include fractions, and so on. However, deeply nested structure leads to theincrease of small size characters and naturally increases the errors of therecognition.
As for the order of writings in mathematical expressions, almost all naturally used writing orders are allowed. For example, either the numerator orthe denominator can be written ﬁrst in the fractions, and either the superindex or sub index may be written ﬁrst in the integrals, summations etc.,
and so on.
The characters recognized in our system are listed below. It includes
all the alphanumeric characters, some Greek letters, and other symbols frequently used in mathematical expressions.
Alphabets
Numerals
Greek letters
Special symbols
Operators
Relational operators
Function names
Parenthesis

3.4

a,b,. . . ,z,A,B,. . . , Z
0,1,2,. . . ,9α, β, γ, δ, ,θ, π, etc.
√  , , , ∞, ∂+, −, ×, ÷,/=, ≡,=, ≡, <, ≤, >, ≥, →lim, log,sin, cos, tan(, ), {, }, [, ]Distinction of Similar Characters

In the characters and symbols used in mathematical expressions, there areseveral pairs which can not be distinguished from each other by their own

shapes. Besides the cases of capital letters and lower case letters of thealphabet such as C and c, S and s, Z and z, etc., there are, for example, qand 9, O(alphabet) and 0(number), 1(number) and |(symbol), etc.
In our system, in order to “avoid” the confusion of these symbols, theusers are requested to write these special capital letters and some other characters in a designated way, as it is described below, the same way as it is usedwidely in Japan when a mathematical expression is written on blackboards.
A model of user interface improves considerably the performance of thesystem to the level of practical use, by asking for some cooperation by theuser.
Character
C, SU, V, WPXZq(1-stroke)
1(numeral)

Writing method
Start with the vertical rod
Attach a crossbar on the top
Put a crossbar at the bottom
Attach a crossbar both on thetop and at the bottom.
Start and end by a vertical rod
Give a serif to backStart with an oblique line.
Stress marks for some characters

The distinction of the capital letter O and the number 0 is diﬃcult without entering into the context. In our system, any circle is recognized as thenumeral 0 if it is vertically long, and as ’o’ if the aspect ratio is near to 1.
When the recognition result is the numeral 0, the system returns always thecapital letter O as its second candidate. Therefore, the user can input thecapital O by the mouse clicking on the character image of 0.
The lower case letter ”s” and the integral symbol has the same shape ifthey are normalized into a square box, but no special method is introduced
in order to distinguish them, since they can be distinguished by the aspectratio.

4Conclusions

In this paper, we introduced our tentative on-line recognition system of handwritten mathematical expressions. We proposed the ”automatic rewriting
method” to improve the accuracy of the structure analysis of handwritten mathematical expressions and to realize an easy and prompt correction

method of recognition errors. We emphasized that an easy correction methodof the recognition results is extremely important to realize smooth writing ofmathematical expressions.
In order to show the possibility of its practical use, we proposed also afront tool for a computer algebra system Mathematica, having a handwriting
input interface using our on-line recognition system of handwritten mathematical expressions and making use of Mathlink as the protocol. The system
is implemented by Visual C++ and works on Windows 95/98. In our experimental system ”Infty”, we can edit mathematical expressions by hand,making use of calculation results by Mathematica as the need arises. The
edited mathematical expressions can be saved into a ﬁle in the notations ofLaTeX and Mathematica. Output format can be changed easily to Maple V,
MathML, etc. by replacing some ﬁltering primitive functions.
Presently, there is no convenient, inexpensive hardware appropriate toinput characters by hand into a computer. Widely used data tablets to inputfree lines into a computer is not convenient to write characters. Hopefully,
hardware to input characters by hand into computers progress further.

5References[1] D. Blostein and A. Grbavec, “Recognition of Mathematical notaion”,
Handbook of Character Recgniyon and Document Image Analysis (1997) 557582.
[2] T. Sakurai, Y.Zhao, H. Suguura nat T. Torii, “A Front-end Tool for Mathematical Computation and Education in a Network Environment”, Proc.
3rd. Asian Technology Conference in Mathematics, Springer (1998) 197-205.
[3] Y. Sakamoto, M. Xie, R. Fukuda and M. Suzuki, “On-Line Recognition of Handwriting Mathematical Expression Via Network”, Proc. 3rd.
Asian Technology Conference in Mathematics, Springer (1998) 271-279.
[4] I. Sou, M. Xie, R. Fukuda and M. Suzuki, “On Line Recognition of Handwritten Mathematical Fomulae Using Stroke Relations” (in Japanese), Technical Report of IEICE. PRMU98-137 (1998) 9-16