Recovering dynamic information from static, handwritten word images: bridging the gap between on-line and off-line handwriting recognition

The eld of handwriting recognition is divided into two distinctive areas: o -line recognition and on-line recognition. A typical example of o -line recognition is postal mail sorting where addresses on envelopes are automatically scanned, processed in an OCR-system, and, according to the recognition results, mechanically routed to the appropriate sorting bins. O -line recognition rates vary with the size of the lexicon, the country, etc. However, it can be said that the state of the art for writer independent recognition is around 95% for lexica of American city names containing a few hundred words. The redundancy between names and zip codes is used to improve recognition rates. In general, the recognition rates of machine printed addresses are higher than the recognition rates of handwritten addresses. A typical example of on-line recognition are Personal Digital Assistants (PDAs). These are small handheld computers accepting pen-based input and combining agenda, address book and telecommunication facilities. On-line recognition methods exploit dynamic information such as the position of the pen at a given time, the velocity of the pen, or the pressure on the writing surface. They require hardware that can record the data of a written text during writing, e.g. an electronic tablet. On-line recognition rates are higher than o -line recognition rates because the dynamic information improves recognition accuracy. Writer independent recognition rates of 98% for lexica with 1000 entries have been reported. The main goal of this thesis is to extract dynamic information from static word images. For static word images written on a paper-like medium no information about the production process of writing is available. However, extracting dynamic information from static images allows the application of powerful online methods to o -line problems. Conversely, o -line methods help to improve on-line recognition rates because temporal information complicates recognition with variations that are not apparent in the static images. Hence, extracting dynamic information and combining on-line methods with o -line methods helps to get a uni ed view of handwriting recognition which is independent of a speci c application. In the framework of this thesis, extracting dynamic information is equivalent to recovering information about sequences of pen movements. The extraction of vi pen pressure and velocity is neglected. A sequence of pen movements is de ned by a sequence of edges in the skeleton of a binary word image. The skeleton is a mathematical construct comprising nodes and edges. It can be regarded as a mathematical abstraction of the written word coinciding in most areas with the centerline of the writing trace, where edges correspond to parts of strokes and nodes correspond to intersections or end points. With the help of the skeleton, the extraction of dynamic information can be de ned as a graph-theoretical problem. In particular, the extraction of dynamic information is understood as a global optimization problem that can be solved using graph-theoretical methods. Two optimization problems are investigated in this thesis: the minimization of length and the minimization of curvature. An important constraint is that traces which are composed of several intersecting strokes, where a stroke is the trace written between a pen-down and a pen-up, can only be approximated. The minimization of length is based on the assumption that writers tend to write a word with minimal length, i.e. without traversing long edges more than once. The search for the shortest path is equivalent to the search for the shortest path in the skeleton of the static word image. This search is called Chinese postman problem in graph-theoretical terms. The minimization of curvature is based on the assumption that writers tend to minimize the deviation from the straight writing line at nodes of the skeleton. This problem is equivalent to the search for a traveling salesman tour in a graph that can be derived from the skeleton in a straightforward manner. While the Chinese postman problem can be solved in polynomial time, the traveling salesman problem is a well-known NP-hard problem. It is shown that the restriction to graphs derived from planar skeletons and the use of a speci c cost function cannot reduce the complexity. For that reason, we present a method that detects splitting edges dividing a word into subparts. The optimization problems can be independently solved for each subpart without changing the global solution. Substantial processing steps of the graph-theoretical methods utilized in this thesis can be described using a discrete, linear mathematical model. This model resembles a speci c cybernetic model that is based on recursion and feedback. It can be implemented by a simple multiplication of matrices. Practical examples show the good performance of the presented methods. However, they illustrate that some writing traces cannot be recovered using length minimization. O -line images that have been generated from genuine on-line data are used for quantitative evaluations. The edges of their skeletons are ordered using the on-line data. This provides a set of known writing traces which can be compared to recovered writing traces using the Levenshtein distance. The vii measured error is moderate and length minimization together with a proper local continuity criterion renders similarly good results as curvature minimization (average Levenshtein distance: 4, average length of sequences: > 17). In another experiment, an existing on-line recognizer is trained and tested with recovered on-line information extracted from addresses containing handwritten American city names. The results are compared to the rates of an o -line recognizer (94%). Though the o -line recognition rates are still higher, the on-line recognition rates based on the recovered on-line data (73%) reach the state of the art in postal automation in the early nineties. In particular, 50% of the words that are wrongly classi ed by the o -line recognizer are correctly classi ed by the on-line recognizer. Hence, the combination of o -line and on-line recognition shows great promise. We have shown that recovering dynamic information from static word images is not only of theoretical signi cance but also feasible and of practical importance. The presented approach is the rst method that provides satisfactory recognition rates of recovered information and facilitates a uni ed view of handwriting recognition. It is to be expected that the recovery of dynamic information will be adopted by other researchers and also tested in other applications, e.g. signature veri cation. Acknowledgments I would like to thank all my colleagues from the Department of Text Understanding Systems (F3 M/T) at the Daimler-Benz Research Center in Ulm. This thesis would not have been possible without their support and advice over the last three years. Particular thanks to Eberhard Mandler for giving advice on all problems concerning handwriting and for providing many important ideas. Thanks to Matthias Oberl ander for the inspiring discussions we had. Thanks also to Jochen Gloger, Torsten Caesar, Ulrich Bohnacker, and Hanno Walischewski for their hardware and software support. I am much obliged to Ulrich Kressel who explained so many problems to me and improved my understanding of classi cation. And thanks to Alfred Kaltenmeier who gave me insight into hidden Markov models. Not forgetting J urgen Franke, Claus Aufmuth, Thomas Bayer, Ingrid Renz, Heike Mogg-Schneider, Tilo Schwarz, Hartmut Schafer, Ralf Specht, Armin Jahn who made my three years enjoyable. I would like to thank J urgen Sch urmann and Bernhard Nebel for supervising my work. Their suggestions and advice were a great deal of help. Special thanks are due to the head of our department, Norbert Bartneck, for giving me so much room and letting me realize my own ideas. Thank you for that! Particular thanks to Stefan Manke for doing the on-line experiments at the University of Karlsruhe. Last but but not least, I would like to thank my roommate, Rainer Portscht. Though he was not directly involved in my work, he was an appreciated adviser in all kinds of problems. Without him, many things would have remained misty.