# Handwriting Recognition Based on 3D Accelerometer Data by Deep Learning
## 1 Introduction  
Online handwriting recognition has been the subject of research for a long time, and part of the technology has found commercial application. However, the limited number of success stories from the market suggests that more research on the pen interface and the recognition methods is still necessary [1].  
Currently, it is common to find wearable devices with a wide range of embedded sensors, which can be used in human activity and behavior studies. Such devices have contributed to the development of new smart applications to use that kind of data. The number of wearables with 3D accelerometer sensors available in the market, and their low cost makes it possible to develop new friendly and non-invasive human-computer interfaces. Such technology allows users to write customarily and freely on smartphones, smartwatches, smart TVs, computers, etc. In order to attract the user's attention to the pen interface, fast and accurate handwriting recognition interfaces are highly required.  
Some systems using 3D accelerometer data for human activity, handwriting and sign language recognition have been developed [2; 3; 4; 5; 6; 7; 8]. Online handwriting recognition is stillchallenging due to the variability of stroke order, shape and style of handwriting [1]. Some approaches recover the trajectory of the handwritten character to convert it to image. In this case, to cope with variability, robust feature extraction methods can be employed [9; 10]. Another option is to perform feature extraction directly in the time domain using a Convolutional Neural Network [11].  
This paper proposes a novel deep learning approach to online handwriting character recognition. The method is focused on efficient feature extraction, classification, and evaluation modules using 3D accelerometer data. The recognition system is based on combining the CNN and LSTM architectures. CNN architecture was used for feature extraction, which allows transforming 3D raw accelerometer data into a feature vector sequence. Finally, an LSTM architecture is trained to recognize the patterns delivered by the CNN. In this way, the classical ad hoc techniques in feature extraction and classification are avoided, permitting artificial intelligence (AI) to make these tasks.  
The main contributions reported in this paper comprise:  
1. A proprietary dataset of 3D accelerometer data corresponding to multi-stroke freestyle handwritten lowercase letters and digits. Unlike previous approaches, we do not impose restrictions on handwriting style and number and order of the strokes.
2. Three neural network architectures (CNN, LSTM, and CNN-LSTM) were proposed. In the last architecture, a CNN was used for feature extraction to encode the global characteristics of raw 3D accelerometer data together with an LSTM for sequence processing and classification.  
The remainder of this paper is organized as follows. Section 2 provides a summary of representative works on online handwritten character recognition. Section 3 includes a detailed description of the proposed method. Section 4 presents the designed experiments and numerical results. Finally, Section 5 announces the conclusions derived from this study.