Abstract: We describe an online handwriting system that is able to support 102 languages using a deep neural network architecture. This new system has completely replaced our previous Segment-and-Decode-based system and reduced the error rate by 20%-40% relative for most languages. Further, we report new state-of-the-art results on IAM-OnDB for both the open and closed dataset setting. The system combines methods from sequence recognition with a new input encoding using B\'ezier curves. This leads to up to 10x faster recognition times compared to our previous system. Through a series of experiments we determine the optimal configuration of our models and report the results of our setup on a number of additional public datasets.

Introduction: In this paper we discuss online handwriting recognition: Given a user input in the form of an ink, i.e. a list of touch or pen strokes, output the textual interpretation of this input. A stroke is a sequence of points INLINEFORM0 with position INLINEFORM1 and timestamp INLINEFORM2 . Figure FIGREF1 illustrates example inputs to our online handwriting recognition system in different languages and scripts. The left column shows examples in English with different writing styles, with different types of content, and that may be written on one or multiple lines. The center column shows examples from five different alphabetic languages similar in structure to English: German, Russian, Vietnamese, Greek, and Georgian. The right column shows scripts that are significantly different from English: Chinese has a much larger set of more complex characters, and users often overlap characters with one another. Korean, while an alphabetic language, groups letters in syllables leading to a large “alphabet” of syllables. Hindi writing often contains a connecting ‘Shirorekha’ line and characters can form larger structures (grapheme clusters) which influence the written shape of the components. Arabic is written right-to-left (with embedded left-to-right sequences used for numbers or English names) and characters change shape depending on their position within a word. Emoji are non-text Unicode symbols that we also recognize. Online handwriting recognition has recently been gaining importance for multiple reasons: (a) An increasing number of people in emerging markets are obtaining access to computing devices, many exclusively using mobile devices with touchscreens. Many of these users have native languages and scripts that are not as easily typed as English, e.g. due to the size of the alphabet or the use of grapheme clusters which make it difficult to design an intuitive keyboard layout BIBREF0 . (b) More and more large mobile devices with styluses are becoming available, such as the iPad Pro, Microsoft Surface devices, and Chromebooks with styluses. Early work in online handwriting recognition looked at segment-and-decode classifiers, such as the Newton BIBREF1 . Another line of work BIBREF2 focused on solving online handwriting recognition by making use of Hidden Markov Models (HMMs) BIBREF3 or hybrid approaches combining HMMs and Feed-forward Neural Networks BIBREF4 . The first HMM-free models were based on Time Delay Neural Networks (TDNNs) BIBREF5 , BIBREF6 , BIBREF7 , and more recent work focuses on Recurrent Neural Network (RNN) variants such as Long-Short-Term-Memory networks (LSTMs) BIBREF8 , BIBREF9 . How to represent online handwriting data has been a research topic for a long time. Early approaches were feature-based, where each point is represented using a set of features BIBREF6 , BIBREF10 , BIBREF1 , or using global features to represent entire characters BIBREF6 . More recently, the deep learning revolution has swept away most feature engineering efforts and replaced them with learned representations in many domains, e.g. speech BIBREF11 , computer vision BIBREF12 , and natural language processing BIBREF13 . Together with architecture changes, training methodologies also changed, moving from relying on explicit segmentation BIBREF7 , BIBREF1 , BIBREF14 to implicit segmentation using the Connectionist Temporal Classification (CTC) loss BIBREF15 , or Encoder-Decoder approaches trained with Maximum Likelihood Estimation BIBREF16 . Further recent work is also described in BIBREF17 . The transition to more complex network architectures and end-to-end training can be associated with breakthroughs in related fields focused on sequence understanding where deep learning methods have outperformed “traditional” pattern recognition methods, e.g. in speech recognition BIBREF18 , BIBREF19 , OCR BIBREF20 , BIBREF21 , offline handwriting recognition BIBREF22 , and computer vision BIBREF23 . In this paper we describe our new online handwriting recognition system based on deep learning methods. It replaces our previous segment-and-decode system BIBREF14 , which first over-segments the ink, then groups the segments into character hypotheses, and computes features for each character hypothesis which are then classified as characters using a rather shallow neural network. The recognition result is then obtained using a best path search decoding algorithm on the lattice of hypotheses incorporating additional knowledge sources such as language models. This system relies on numerous pre-processing, segmentation, and feature extraction heuristics which are no longer present in our new system. The new system reduces the amount of customization required, and consists of a simple stack of bidirectional LSTMs (BLSTMs), a single Logits layer, and the CTC loss BIBREF24 (Sec. SECREF2 ) trained for each script (Sec. SECREF3 ). To support potentially many languages per script (see Table TABREF5 ), language-specific language models and feature functions are used during decoding (Sec. SECREF38 ). E.g. we have a single recognition model for Arabic script which is combined with specific language models and feature functions for our Arabic, Persian, and Urdu language recognizers. Table TABREF5 shows the full list of scripts and languages that we currently support. The new models are more accurate (Sec. SECREF4 ), smaller, and faster (Table TABREF68 ) than our previous segment-and-decode models and eliminate the need for a large number of engineered features and heuristics. We present an extensive comparison of the differences in recognition accuracy for eight languages (Sec. SECREF5 ) and compare the accuracy of models trained on publicly available datasets where available (Sec. SECREF4 ). In addition, we propose a new standard experimental protocol for the IBM-UB-1 dataset BIBREF25 (Sec. SECREF50 ) to enable easier comparison between approaches in the future. The main contributions of our paper are as follows:

Question: Which language has the lowest error rate reduction?
Answer: thai