Silent training by error correction for on-line handwritting recognition systems

A method for using information provided during error correction for modifying character prototypes in an on-line handwriting recognition system is disclosed. The method allows a user to correct misrecognized handwritten characters by overwriting directly on the displayed ASCII representation of the recognition result for a given character. The rewritten character is then used to silently retrain the system so as to adapt it to the user's particular handwriting style.

FIELD OF THE INVENTION 
This invention relates generally to handwriting recognition systems and, in 
particular, to a method and apparatus for training such systems to 
minimize the occurrence of repeat errors. 
BACKGROUND OF THE INVENTION 
Many on-line, handwriting recognition systems employ curve matching methods 
to match an unknown character against prototype, or template, characters. 
One example is T. Fujisaki, T. F. Chefalas, J. Kim and C. C. Tappert, 
"Online recognizer for runon handprinted characters," Proc. 10th Int. 
Conf. Pattern Recognition, pp. 450-454, June 1990. In general, the 
recognition accuracy of such prototype-based handwriting recognition 
systems is a function of the quality of the prototypes, which in turn, is 
dependent upon the efficiency and accuracy of system training. 
Occasionally, errors are made by the recognition system and, when they 
occur, the user corrects them. However, in current systems there is no 
provision for training during error correction to prevent future similar 
errors. The training that is available requires the user to enter a 
special training mode. This mode switching interrupts the normal work flow 
of the user. 
The following U.S. and Japanese Patents all teach various aspects of 
handwriting recognition systems. 
U.S. Pat. No. 4,561,105, issued Dec. 24, 1985, entitled "Complex Pattern 
Recognition Method and System" to Crane et al describes an on-line 
character recognition system that uses template or model strokes (col 3, 
lines 1-52). 
U.S. Pat. No, 5,285,505, issued Feb. 8, 1994, entitled "Method and 
Apparatus for Improving Prototypes of Similar Characters in On-Line 
Recognition" to Kim et al, describes a character recognition system for 
improving the prototypes of similar characters by emphasizing the 
dissimilarities of prototypes of similar characters and deemphasizing the 
similarities. 
Japanese Patent No. 62-24382 entitled "Method for Recognizing Handwritten 
Character" describes, in the abstract, a method for recognizing 
handwritten characters in which input strokes are compared with reference 
strokes stored in a template memory. 
C. Chiang and H. Fu, "Using Neural Nets to Recognize Handwritten/Printed 
Characters", IEEE Proc. Advanced Computer Technology, Reliable Systems and 
Applications, pp. 492-96, 1991, describes a handwritten character 
recognition system implemented by a stochastical neural network which can 
provide on-line training of a handwriting of the system. 
What is not taught by the foregoing references, and what is thus an object 
of the invention to provide, is a method and apparatus for improving or 
optimizing a set of character prototypes, by training the system to 
recognize a particular user's style of writing. 
A further object of the invention is to provide a method, and a system for 
accomplishing the method, for silently training (i.e., training performed 
without explicit knowledge of, or direction by, the user) the system to 
recognize a particular user's handwriting through user feedback, and 
without the user having to specifically request training. 
Another object of the invention is to provide an efficient method of using 
the information provided during error correction to modify the prototypes 
so as to reduce the likelihood of similar errors occurring in the future. 
Such a method permits training to occur over time (that is, the prototype 
set to evolve) as a natural process without having the user switch modes. 
The foregoing and other problems are overcome and the objects of the 
invention are realized by a method for silent training by error correction 
for an on-line handwriting recognition system. 
SUMMARY OF THE INVENTION 
The invention provides a method of training a handwriting recognition 
system to recognize handwritten characters of a particular user. The 
method includes storing a plurality of prototype characters in a memory, 
and entering a string of user-generated handwritten characters on an input 
device. The user generated string of characters is compared with the 
prototype characters stored in memory. Next, for each user generated 
handwritten character, a corresponding character is displayed, the 
corresponding character of a particular user-generated character being a 
prototype character most closely resembling the particular user-generated 
handwritten character. The user then enters on the input device, for each 
incorrectly recognized user-input character, a user-generated handwritten 
correction character. The user generated correction character is compared 
with the prototype characters stored in memory, excluding the incorrectly 
returned character. For each correction character, a corresponding 
correction character is displayed, the corresponding correction character 
of a particular user-generated correction character being a prototype 
character most closely resembling the user-generated handwritten 
correction character. The correction steps are then repeated if the system 
again returns an incorrect recognition result. 
In one embodiment, the invention provides a method for using information 
provided during error correction for modifying character prototypes in an 
on-line handwriting recognition system. The method allows a user to 
correct misrecognized handwritten characters by overwriting directly on 
the displayed ASCII representation of the recognition result for a given 
character. The rewritten character is then used to silently retrain the 
system so as to adapt it to the user's particular handwriting style.

DETAILED DESCRIPTION OF THE INVENTION 
Referring to FIG. 1, there is shown an on-line handwriting character 
recognition system 10 constructed and operated in accordance with the 
invention. System 10 includes a handwriting transducer comprised of an 
electronic table 12 having a writing implement, such as a stylus 14. In 
response to handwritten characters, the transducer outputs a serial stream 
of (x,y) coordinate pairs corresponding to a position of the stylus 14 
relative to a tablet 12 coordinate system. The coordinate pairs are 
provided to a stroke segmenter 16 that identifies a beginning and an end 
of discrete handwritten strokes. Strokes are provided to a character 
matcher 18. Character matcher 18 is coupled to a store 20 of predetermined 
character prototypes for comparing points of input strokes to points of 
stored character prototypes so as to identify, to within some probability, 
that an input stroke or assemblage of strokes represents a specific 
character. Recognized characters are output from the matcher 18 and may be 
displayed to a user or input to an application, such as a word processing 
program. In some systems, the functionality of the segmenter 16 and the 
character matcher 18 may be combined within one handwritten stroke 
processing unit. 
Character prototypes may be provided to the store 20 during a training 
session held with a user of the system 10. The character prototypes may be 
entered by a specific user of the system and thereafter used only when the 
specific user enters handwritten characters. Also, a global or universal 
set of character prototypes may be generated for use with more than one 
user. 
In this regard, a starter prototype set may be designed as an initial 
prototype set to be used by any writer on first using the system 10. If 
desired, a writer may then add character prototypes in his/her own writing 
style to obtain a prototype set tailored to the writer, also referred to 
as an individualized prototype set. In that a starter prototype set is 
typically created by an expert in handwriting recognition systems, there 
is no burden placed on a user of the system 10 to follow any writing 
suggestions or constraints. 
In any case, during a training session, characters from character matcher 
18 are routed through a physical or logical switch 22 to a prototype 
processor 24. Prototype processor 24 is also coupled to a user input and 
output device, such as a keyboard 26 and a display 28, for controlling the 
operation thereof, as will be described in detail below. The output of 
prototype processor 24 is a series of prototype characters that are 
supplied to store 20 for subsequent use during a handwriting recognition 
session conducted with a user of the system 10. 
In accordance with the invention, there is provided a method of "silently" 
training the system during operation to more accurately recognize the 
handwriting of the particular writer. This training method is "silent" 
because it is performed without explicit knowledge of, or direction by the 
user. Through such silent training, the system adapts to the writing of 
the user and becomes more accurate with use over time. 
In general, the method of the present invention operates to update the 
prototype set by using user-provided feedback to silently retrain the 
system. This is accomplished by disabling in the candidate store the 
individual character recognition result indicated to be erroneous by the 
user. The redrawn handwritten character is then used to select the closest 
match from among the remaining candidates. A preferred embodiment of the 
invention will now be described in detail. 
The user starts with an empty handwriting pad, as shown in FIG. 2. The 
handwriting pad is displayed on the tablet 12 of FIG. 1. The user then 
writes strokes on the tablet-display, (FIG. 3). Each stroke is sent to 
segmenter 16 to begin processing. The user then taps the "OK button", and 
the system sends a completion notification to the segmenter to indicate 
that processing should begin. A completion notification can also be sent 
automatically via a time-out (user does not write for a period of time). 
The completion notification causes the recognition engine to return a 
translation result in the form of ASCII characters. The original ink 
corresponding to the handwriting is replaced by the recognized ASCII 
characters, as shown in FIG. 4. The original mapping of the input 
characters to the recognized characters is shown in FIG. 5. 
When the user notices a recognition error, for instance an "0" incorrectly 
recognized as a "U", he over-writes the incorrect ASCII character with a 
handwritten correct one, i.e., another "0" in the present example, as 
shown in FIG. 6. The recognition system then tries to recognize the newly 
written character, and re-displays the resulting ASCII character, FIG. 8. 
During this recognition, the ASCII character previously indicated to be 
incorrect is eliminated from contention so that the same, erroneous 
character cannot be chosen. FIG. 7 shows a mapping of the handwritten 
character which was originally misrecognized to the intended result as 
obtained from the step depicted in FIG. 8. After all errors are corrected, 
the user finally accepts the results by touching "OK" or issuing an 
alternate "completion" command. 
The method of silent training will now be described, with reference to the 
flow chart of FIG. 9. After the user initially enters a string of text 
(FIG. 3), and an initial recognition result is returned (FIG. 4), the 
recognition system internally saves the original handwritten strokes and 
the mapping to the translated ASCII characters (FIG. 5). When the user 
rewrites the misrecognized character directly on the displayed image of 
the incorrect ASCII character as shown in FIG. 6, the recognition system 
automatically eliminates the prior ASCII character from contention. This 
is achieved automatically by using the internally saved original strokes 
and mapping to the original translated ASCII characters. The recognition 
engine also internally saves the error correction strokes and the mapping 
to the translated ASCII characters, FIG. 7. 
When the user finally accepts the translation, the recognition system is 
notified of final acceptance and performs a "silent training." The final 
error correction strokes and error correction mappings are assumed to be a 
legitimately written version of the recognized ASCII character. If the 
final error correction strokes are similar to the original strokes within 
a user settable threshold (similarity is determined by performing a 
match), both the error correction strokes and original strokes are used to 
silently train the recognition engine. This training is conducted in the 
same manner as original system training, with no intervention or direction 
required of the user. The strokes will be learned as the finally accepted 
ASCII results. The subsequently written characters alone can be used for 
training if not similar (within a matching threshold) to the accepted 
handwritten character. 
After final acceptance and silent training, the recognition system destroys 
the internal strokes and mappings that were being saved. 
While the system has been described with respect to a "boxed mode" 
handwriting pad for the original stroke input, it should be understood by 
those skilled in the art that other modes of original stroke input pads 
can be used, such as "Runon", "Cursive", "Symbols", etc., without 
departing from the spirit of the invention. The "boxed mode" is the 
preferred and simplest form because it automatically implies the mapping 
of strokes to translated ASCII results. For some of the other forms (e.g., 
cursive) of error correction, the mapping would need to be displayed back 
to the user for acceptance.