# TARNet: An Efficient and Lightweight Trajectory-Based Air-Writing Recognition Model Using a CNN and LSTM Network

Authors: Md. Shahinur Alam; Ki-Chul Kwon; Shariar Md Imtiaz; Md Biddut Hossain; Bong-Gyun Kang; Nam Kim
Journal: Human Behavior and Emerging Technologies
 (2022)

---

## Abstract

Air-writing is a growing research topic in the field of gesture-based writing systems. This research proposes a unified, lightweight, and general-purpose deep learning algorithm for a trajectory-based air-writing recognition network (TARNet). We combine a convolutional neural network (CNN) with a long short-term memory (LSTM) network. The architecture and applications of CNN and LSTM networks differ. LSTM is good for time series prediction yet time-consuming; on the other hand, CNN is superior in feature generation but comparatively faster. In this network, the CNN and LSTM serve as a feature generator and a recognizer, optimizing the time and accuracy, respectively. The TARNet utilizes 1-dimensional separable convolution in the first part to obtain local contextual features from low-level data (trajectories). The second part employs the recurrent algorithm to acquire the dependency of high-level output. Four publicly available air-writing digit (RealSense trajectory digit), character (RealSense trajectory character), smart-band, and Abas datasets were employed to verify the accuracy. Both the normalized and nonnormalized conditions were considered. The use of normalized data required longer training times but provided better accuracy. The test time was the same as those for nonnormalized data. The accuracy for RTD, RTC, smart-band, and Abas datasets were 99.63%, 98.74%, 95.62%, and 99.92%, respectively.


---

## Body

## 1. Introduction

Touchless writing systems are becoming popular with the advent of augmented reality (AR)/virtual reality (VR), especially when conventional writing is not possible or extremely inconvenient. The emerging technique of AR/VR appears to lead the next generation of interaction. However, the input method of these technologies has not yet developed at its mature level. Speech recognition can be the possible solution; nevertheless, it does not fit in a noisy environment or location where privacy is important [1]. Conventional input method varies across techniques such as mouse or keyboard. However, gesture-based writing can be performed well. It allows users to cooperate with an electronic device without physical interaction; this is especially helpful during the COVID-19 pandemic. In terms of gesture numbers, existing gesture-based writing systems are limited [2–7]. Conventionally, a specific combination of a gesture and a posture symbolizes a number or character, but not all characters are often represented. To overcome this issue, we introduce a novel air-writing technique. It can be counted as complex motion gestures. However, it can significantly express arbitrary character/text input for intelligent control systems.Air-writing employs a finger or a specific device to write a character or word in a virtual environment. Characters or words are written in an imaginary box (a virtual window) by moving the finger or device [8–14] creating a trajectory representing a specific letter or number in the air. In contrast to gesture recognition, air-writing enables a wide range of interaction options; additionally, the user does not require to memorize them. Alternatively, it can work as a natural solution. It can also be used for wearable devices and egocentric cameras such as Microsoft HoloLens and Google Glass. As writing is based only on the trajectory, it suffers fewer spatial constraints and lacks haptic feedback, providing an unconstrained writing experience. Air-writing is similar to pen-and-paper-based writing; the only difference is the writing medium (air or paper). However, the bounding box varies among users. Moreover, it is not easy to maintain the same virtual window even for the same user. Users may lose orientation in space due to the absence of visual and haptic feedback. These issues can be resolved by applying specific normalization/transformation techniques to the temporal and trajectory data.Various three-dimensional (3D) devices track trajectories effortlessly, particularly the Microsoft Kinect, Intel RealSense, and Leap Motion. All have an elaborate application programming interfaces (APIs); the user requires only minimal knowledge of tracking algorithms. Recently, some researchers have been focusing on Wi-Fi [15, 16] and radar [17–19] technology. The Kinect and RealSense are mid-to-long-range devices; the Leap Motion is a short-range device but affords millimetre-level accuracy. Thus, the Kinect and RealSense find most applications on desktops; Leap Motion is preferred for AR/VR [20, 21]. All these devices are excellent in their specific purpose and platforms. Here, we focus on PC-based midrange interaction; hence, Intel RealSense camera is utilized.Recognition is currently the most challenging task in the field. However, deep learning-based recognition systems afford excellent results without requiring feature generation. Such networks include convolutional neural networks (CNNs) and long short-term memory (LSTM). A CNN is optimal for feature generation, whereas an LSTM is a gold standard for sequential data processing. In this research, we propose a versatile deep learning model that fuses both CNN and LSTM features and achieves outstanding results. The model was trained and tested using the publicly available RealSense trajectory digit (RTD) [22], RealSense trajectory character (RTC) [23], smart-band [24], and Abas datasets [25]. The variety of datasets proves that the proposed model is invariant to a different dataset. The benchmark results verify that the model performs better than a standalone CNN or LSTM network.The rest of the paper is organized as follows. In Section2, the background and related works are described. The details of the RTD, RTC, smart-band, and Abas datasets are discussed in Section 3. The proposed TARNet is explained in Section 4. The experimental setup and results are discussed in Section 5. Finally, Section 6 contains the conclusion, summary, and future work.


## 2. Related Work and the Dataset of Air-Writing

Air-writing is a new form of writing; several novel somatosensory sensors and devices have recently emerged. Some are simple and very cheap ($1) [26]; some are expensive [14, 22, 27–29]. There are some special sensors based on accelerometer [30–32], gyroscope [33], wearable gloves [34, 35], and wrist-worn devices [36–39]. Wobbrock et al. developed a $1 device that can easily handle under 100 lines of code [26]. Most of the accelerometer-based research is conducted with cheap accelerometer sensors equipped with wireless communication [30, 31]. However, the easiest way to access such a sensor is via a mobile phone [32]. Accuracy improves when gyroscopic, motion, and acceleration sensors are combined [33]. The support vector machine (SVM) was first introduced to identify the gestures; later, the HMM was applied to recognize sentences. However, the dataset contains only 720 sentences from 9 participants. More sensors increase the accuracy of trajectory information. However, excessive sensors are burdensome for the user to employ in real-life situations. Also, such devices are bulky and difficult to design. Thus, some off-the-shelf ready-to-go devices have been developed. An earlier version of these devices is H.P. slate [40] and Wiimote [14]. These wearable devices were difficult (and sometimes impossible) to handle. The recent Kinect [27, 29, 41], RealSense [22, 23], and Leap Motion [12, 21, 25, 28, 42, 43] devices are vision-based and very user-friendly and do not require additional devices. The RealSense camera is straightforward to ease.Earlier air-writing recognition methods relied on hand-crafted feature matching or classical machine learning-based algorithms. The SVM [21, 29, 44] and hidden Markov model (HMM) [8, 12, 14, 27, 28, 44] were the most widely used machine learning algorithms requiring predefined feature generation techniques. Kumar et al. proposed SVM for sign language recognition and achieved 63.57%; the accuracy was improved then by applying the BLSTM algorithm [21]. They collected 2240 sign language and 28 finger writing sentences from 10 users. Similarly, Qu et al. proposed a digit recognition system using SVM but added dynamic time warping (DTW) for better accuracy [29]. The collected digit vocabulary was 6104 collected by 10 participants. Amma et al. resolved the dataset issue by collecting 60,000 words from speech data [44]. The highest accuracy was 83%. Fu et al. applied HMM algorithm and achieved more than 86% accuracy [8]. A motion gesture concept was introduced for uninterrupted data processing using Leap Motion and HMM algorithm, which achieved more than 98% recognition accuracy [12, 14]. In most cases, the HMM was more accurate than the SVM. However, neither was completely perfect.The classical machine learning algorithm suffers from recognition accuracy and is solved by recent state-of-the-art deep learning-based algorithms. CNN [15, 16, 24, 42, 43, 45], LSTM [19, 42, 43, 46–49], and BLSTM [19, 21, 28, 50, 51] algorithms have pioneered air-writing and gesture identification. Yanay and Shmueli employed a CNN algorithm and achieved more than 83% accuracy from 21450 sample data [24]. Some researchers focused on technique rather than the data or participants; hence, accuracy suffers in some cases for the CNN algorithm [15, 16]. In contrast to the accuracy, LSTM performs better for the low volume of the dataset [19, 46]. Since the algorithms are designed and developed for specific purposes and reasons, the standalone module cannot get the full essence of the deep learning features. To overcome this situation, we combined the CNN and LSTM algorithms and TARNet, for better feature generation and detection. Recognition accuracy thus improved.


## 3. The Datasets

We used four air-writing datasets: the RTD [22], RTC [23], smart-band [24], and Abas datasets [25]. The RTD and RTC vision-based datasets were collected using an Intel RealSense sr300 camera. The smart-band is a sensor-based dataset collected using a smartphone and a wrist-worn smart band. On the other hand, the Abas dataset is collected using a Leap Motion device. Although visualization of air-writing characters differs somewhat by how the data are collected, the writing process is the same as the traditional handwriting method.


### 3.1. The RTD

The RTD dataset is an air-writing digit dataset collected by an Intel RealSense sr300 camera [22]. Graduate students aged between 23 and 30 were employed to collect the dataset. There were 10 participants, and each collected 2100-digit data. They were primarily male (eight). The instructions were similar to those for box-writing [12, 14]. Digits were stored as a sequence of coordinates. The average trajectory length was between 13 and 265. Digits 4 and 1 exhibited the maximum and minimum average lengths, respectively. However, the standard deviation ranged from 15 to 27, indicating that the dataset was consistent. The total RTD dataset size is 21,000, including train and test data. RTD raw data sample is shown in Figure 1(a) (sample digit “5”).Figure 1
Raw dataset samples. (a) An RTD dataset sample (digit “5”). (b) An RTC dataset sample (“A”). (c) A smart-band dataset sample (“A”). (d) An Abas dataset sample (“A”).
(a)(b)(c)(d)


### 3.2. RTC

The RTC is an extension of the RTD dataset containing 26 English uppercase letters [23]. Ten graduate students participated; 3,000 trajectory data were collected from each. The sample total is 30,000, including test and training samples. The average trajectory length ranged from 21 to 173. Characters M and C exhibited the maximum and minimum average lengths, respectively. An RTC raw data sample is shown in Figure 1(b) (sample character “A”).


### 3.3. Smart-Band

The smart-band dataset was collected by a “Microsoft band 2” using the Samsung Galaxy S8 smartphone [24]. Accelerometer and gyroscope signals were recorded during air-writing using a dedicated Android app running on a smartphone. Motions were recorded by smart-band sensors. Each sample provided a total of six different dimensions. There were 55 subjects (28 females and 17 males; 46 right-handed) and total 21450 samples. A raw data sample is shown in Figure 1(c) (sample character “A”).


### 3.4. Abas Dataset

Abas dataset contains 22,000 and 8,000 samples for training and testing, respectively [25]. A Leap Motion device was employed to collect the dataset containing a sequence of hand trajectory information. The 3D data are then converted to the tile-based image matrix dimension of 28×28. Hence, each sample data is a 784-size vector data. There were 10 participants aged between 20 and 21 years who participated in this project; each produced around 3,000 handwriting characters throughout a month. The sample data is shown in Figure 1(d). It is already normalized.


## 3.1. The RTD

The RTD dataset is an air-writing digit dataset collected by an Intel RealSense sr300 camera [22]. Graduate students aged between 23 and 30 were employed to collect the dataset. There were 10 participants, and each collected 2100-digit data. They were primarily male (eight). The instructions were similar to those for box-writing [12, 14]. Digits were stored as a sequence of coordinates. The average trajectory length was between 13 and 265. Digits 4 and 1 exhibited the maximum and minimum average lengths, respectively. However, the standard deviation ranged from 15 to 27, indicating that the dataset was consistent. The total RTD dataset size is 21,000, including train and test data. RTD raw data sample is shown in Figure 1(a) (sample digit “5”).Figure 1
Raw dataset samples. (a) An RTD dataset sample (digit “5”). (b) An RTC dataset sample (“A”). (c) A smart-band dataset sample (“A”). (d) An Abas dataset sample (“A”).
(a)(b)(c)(d)


## 3.2. RTC

The RTC is an extension of the RTD dataset containing 26 English uppercase letters [23]. Ten graduate students participated; 3,000 trajectory data were collected from each. The sample total is 30,000, including test and training samples. The average trajectory length ranged from 21 to 173. Characters M and C exhibited the maximum and minimum average lengths, respectively. An RTC raw data sample is shown in Figure 1(b) (sample character “A”).


## 3.3. Smart-Band

The smart-band dataset was collected by a “Microsoft band 2” using the Samsung Galaxy S8 smartphone [24]. Accelerometer and gyroscope signals were recorded during air-writing using a dedicated Android app running on a smartphone. Motions were recorded by smart-band sensors. Each sample provided a total of six different dimensions. There were 55 subjects (28 females and 17 males; 46 right-handed) and total 21450 samples. A raw data sample is shown in Figure 1(c) (sample character “A”).


## 3.4. Abas Dataset

Abas dataset contains 22,000 and 8,000 samples for training and testing, respectively [25]. A Leap Motion device was employed to collect the dataset containing a sequence of hand trajectory information. The 3D data are then converted to the tile-based image matrix dimension of 28×28. Hence, each sample data is a 784-size vector data. There were 10 participants aged between 20 and 21 years who participated in this project; each produced around 3,000 handwriting characters throughout a month. The sample data is shown in Figure 1(d). It is already normalized.


## 4. Proposed Air-Writing Recognition Method

This article proposes a unified general-purpose deep learning algorithm for air-writing recognition. The datasets used are publicly available and were processed to feed the deep learning model. Each dataset was divided into training and test portions. Then, the network was designed using a fusion CNN-LSTM algorithm, trained using the training datasets, and a model generated to test unknown data. Finally, the model was verified by examining the test datasets.


### 4.1. Dataset Normalization

The raw data collected for air-writing from the sensor or camera is uneven due to its virtual environment (shown in Figure1); there is no pen-up pen-down system. Hence, normalization is required (except for the Abas dataset). To normalize the dataset, two predominant techniques are used—nearest neighbour and root point normalization. Root point normalization is simple and easy to calculate. Equations (1) to (3) are employed. As there is no virtual window, the trajectories are scattered. Root point normalization transforms all trajectories to the origin:
(1)xi=x0−xi,(2)yi=y0−yi,(3)zi=z0−zi.Here,x0,y0,z0 and xi,yi,zi are the initial and instantaneous points, respectively.Nearest neighbour point normalization averages the adjacent value from the trajectories. The purpose of the nearest neighbour normalization is to transform the trajectories to a human-understandable format. Equations (4) to (6) are applied for the nearest neighbour normalization.
(4)xi=1n∑i=0nxi,(5)yi=1n∑i=0nyi,(6)zi=1n∑i=0nzi.Herei and n refer to any individual point and the number of points considered over a trajectory, respectively. Consider more points shrink the original shape; hence, it is found that five or six numbers were optimal. In this experiment, six points are considered.The normalized datasets are shown in Figure2, which is visually far clear than Figure 1. Especially, dramatic changes were found for the smart-band dataset. In Figure 2, all trajectories are started from the origin that comes from the root point normalization. Besides, the smoothness comes from the nearest neighbour point normalization.Figure 2
Normalized dataset samples. (a) The RTD dataset for sample digit “5.” (b) The RTC dataset for sample character “A.” (c) The smart-band dataset for sample character “A.” Normalization is not required in case of Abas dataset.
(a)(b)(c)


### 4.2. Separable Convolution Layer

In this work, a fusion of CNN and the LSTM network is utilized. Due to the wide use of CNN, different variants of CNNs are proposed. It is mainly used for image processing and feature detection [52]. A conventional CNN network is very expensive in terms of resources and time, requiring massive calculations. However, the recent depthwise separable convolution processes allow efficient calculation and processing of one-dimensional information and sequential data [53]. Separable convolution features two steps; the first is depthwise convolution and the second 1×1 pointwise convolution to convolve the channels.Figure3 shows a schematic diagram of the depthwise separable convolution. A depthwise separable convolution follows the same feature as spatially separable convolutions; splitting the kernels into two smaller ones produces the same result with lesser multiplications. The total multiplications in the depthwise and pointwise convolution are H×W×n×f×f and H×W×n, respectively. The principal advantage of pointwise convolution is that the filter is the only multiplicand lacking an extra dimension; hence, the calculation burden falls dramatically.Figure 3
Schematic diagram of the separable convolution layer.


### 4.3. The LSTM Network

A recurrent neural network (RNN) is widely used for time series predictions (activities, audio, and text) [53]. An RNN is a generalization of a feedforward neural network that uses memory to store local information. Unlike feedforward networks, RNN uses internal memory (known as the state) to process sequences of inputs. An LSTM is a modified version of RNN that resolves the vanishing gradient problem. It features input, forget, and output gates, of which the most important are the forget and output gates that define the data that should be removed and passed to the output, respectively.


### 4.4. The TARNet Architecture

The proposed TARNet exploits the advantages of both the CNN and LSTM networks. A CNN and an LSTM are used for feature extraction and sequence recognition, respectively. The complete network diagram is shown in Figure4. The network features an input, convolution, pooling, an LSTM unit, and dense layers. The inputs are one-dimensional sequential data. The input layer dimensions are set to the maximum lengths of the trajectory sequences, which are 300, 800, 850, and 785 for the RTD, RTC, smart-band, and Abas data, respectively. The first convolution layer features a filter of size 256, followed by a max-pooling layer that downsamples the CNN network. This reduces computational complexity and cost. A filter of size 512 is used for the following convolution layer (which again includes a max-pooling layer). The principal purpose of CNN is feature generation.Figure 4
The proposed TARNet architecture.The LSTM features two layers, and sizes are 256 and 512 for the first and second layers, respectively. Similar to the previous convolution layers, max-pooling is used in the first LSTM layer. An LSTM layer consisted of sigmoid, tanh, pointwise addition, and multiplications. Flattening is performed before feeding the dense layer. The first and second dense layer features 128 and 256 neurons, respectively. The dense layer is fully connected and associated with the ReLU activation function (7). It helps the network by removing the negative part and keeping only the positive value from the neuron. A dropout rate of 0.4 is used in both dense layers to prevent overfitting. Adam optimizer is used in this network with a learning rate of 0.0005. The output layer is variable here; the output size is 10, 26, 26, and 26 for RTD, RTC, smart-band, and Abas datasets, respectively. To generalize the results, the softmax activation function is employed in the output layer (8). Softmax is a probabilistic method to regularize an output.
(7)fx=max0,x,(8)σZi=eZi∑k=1KeZk.Here,σ represents the softmax function. An input vector Z is assigned to all classes by the multiclass classifier k.


## 4.1. Dataset Normalization

The raw data collected for air-writing from the sensor or camera is uneven due to its virtual environment (shown in Figure1); there is no pen-up pen-down system. Hence, normalization is required (except for the Abas dataset). To normalize the dataset, two predominant techniques are used—nearest neighbour and root point normalization. Root point normalization is simple and easy to calculate. Equations (1) to (3) are employed. As there is no virtual window, the trajectories are scattered. Root point normalization transforms all trajectories to the origin:
(1)xi=x0−xi,(2)yi=y0−yi,(3)zi=z0−zi.Here,x0,y0,z0 and xi,yi,zi are the initial and instantaneous points, respectively.Nearest neighbour point normalization averages the adjacent value from the trajectories. The purpose of the nearest neighbour normalization is to transform the trajectories to a human-understandable format. Equations (4) to (6) are applied for the nearest neighbour normalization.
(4)xi=1n∑i=0nxi,(5)yi=1n∑i=0nyi,(6)zi=1n∑i=0nzi.Herei and n refer to any individual point and the number of points considered over a trajectory, respectively. Consider more points shrink the original shape; hence, it is found that five or six numbers were optimal. In this experiment, six points are considered.The normalized datasets are shown in Figure2, which is visually far clear than Figure 1. Especially, dramatic changes were found for the smart-band dataset. In Figure 2, all trajectories are started from the origin that comes from the root point normalization. Besides, the smoothness comes from the nearest neighbour point normalization.Figure 2
Normalized dataset samples. (a) The RTD dataset for sample digit “5.” (b) The RTC dataset for sample character “A.” (c) The smart-band dataset for sample character “A.” Normalization is not required in case of Abas dataset.
(a)(b)(c)


## 4.2. Separable Convolution Layer

In this work, a fusion of CNN and the LSTM network is utilized. Due to the wide use of CNN, different variants of CNNs are proposed. It is mainly used for image processing and feature detection [52]. A conventional CNN network is very expensive in terms of resources and time, requiring massive calculations. However, the recent depthwise separable convolution processes allow efficient calculation and processing of one-dimensional information and sequential data [53]. Separable convolution features two steps; the first is depthwise convolution and the second 1×1 pointwise convolution to convolve the channels.Figure3 shows a schematic diagram of the depthwise separable convolution. A depthwise separable convolution follows the same feature as spatially separable convolutions; splitting the kernels into two smaller ones produces the same result with lesser multiplications. The total multiplications in the depthwise and pointwise convolution are H×W×n×f×f and H×W×n, respectively. The principal advantage of pointwise convolution is that the filter is the only multiplicand lacking an extra dimension; hence, the calculation burden falls dramatically.Figure 3
Schematic diagram of the separable convolution layer.


## 4.3. The LSTM Network

A recurrent neural network (RNN) is widely used for time series predictions (activities, audio, and text) [53]. An RNN is a generalization of a feedforward neural network that uses memory to store local information. Unlike feedforward networks, RNN uses internal memory (known as the state) to process sequences of inputs. An LSTM is a modified version of RNN that resolves the vanishing gradient problem. It features input, forget, and output gates, of which the most important are the forget and output gates that define the data that should be removed and passed to the output, respectively.


## 4.4. The TARNet Architecture

The proposed TARNet exploits the advantages of both the CNN and LSTM networks. A CNN and an LSTM are used for feature extraction and sequence recognition, respectively. The complete network diagram is shown in Figure4. The network features an input, convolution, pooling, an LSTM unit, and dense layers. The inputs are one-dimensional sequential data. The input layer dimensions are set to the maximum lengths of the trajectory sequences, which are 300, 800, 850, and 785 for the RTD, RTC, smart-band, and Abas data, respectively. The first convolution layer features a filter of size 256, followed by a max-pooling layer that downsamples the CNN network. This reduces computational complexity and cost. A filter of size 512 is used for the following convolution layer (which again includes a max-pooling layer). The principal purpose of CNN is feature generation.Figure 4
The proposed TARNet architecture.The LSTM features two layers, and sizes are 256 and 512 for the first and second layers, respectively. Similar to the previous convolution layers, max-pooling is used in the first LSTM layer. An LSTM layer consisted of sigmoid, tanh, pointwise addition, and multiplications. Flattening is performed before feeding the dense layer. The first and second dense layer features 128 and 256 neurons, respectively. The dense layer is fully connected and associated with the ReLU activation function (7). It helps the network by removing the negative part and keeping only the positive value from the neuron. A dropout rate of 0.4 is used in both dense layers to prevent overfitting. Adam optimizer is used in this network with a learning rate of 0.0005. The output layer is variable here; the output size is 10, 26, 26, and 26 for RTD, RTC, smart-band, and Abas datasets, respectively. To generalize the results, the softmax activation function is employed in the output layer (8). Softmax is a probabilistic method to regularize an output.
(7)fx=max0,x,(8)σZi=eZi∑k=1KeZk.Here,σ represents the softmax function. An input vector Z is assigned to all classes by the multiclass classifier k.


## 5. Experimental Setup and Performance Analysis

The experimental setup is straightforward and does not require any high-end devices. The specifications are listed in Table1. An Intel Core i5 processor (clock speed 3.40 GHz), Windows 10, and a 64-bit operating system were employed to train the network. The main memory was 16 GB. The popular deep learning-based library Keras ran on top of TensorFlow. We employed the Spyder integrated development environment (IDE) of the Anaconda system to run the network. All code was written in Python. The code is available on GitHub: https://github.com/shahinur-alam/TARNetTable 1
Technical specification of the hardware and the development environment.
ContextSpecificationCPUIntel Core i5-7500 3.40 GHzOperating systemWindows 10 professional, 64-bitMemory16 GBEnvironmentAnaconda 64 bitIDESpyderProgramming languagePythonDeep learning libraryTensorFlow, KerasThe model was trained using RTD, RTC, smart-band, and Abas datasets containing data on 20,000, 30,000, 21,450, and 30,000 trajectories, respectively. During training, we employed 10-fold cross-validation. Different batch sizes and iterations are performed. However, a batch size of 256 is optimal for the RTD, RTC, smart-band, and Abas datasets. The training accuracies and losses are shown in Figure5 (number of iterations: 50). However, optimal accuracy was achieved within 15 iterations for all datasets. Figure 5(a) shows the training and validation accuracies by iteration number. The validation accuracy after the first step was almost 0.9. Figure 5(b) shows the losses over the iterations. As for accuracy, the expected outcome was found within 15 iterations. Note that the training and validation accuracies follow the same curve; this indicates network stability, fidelity, and minimal overfitting. Both accuracy and loss verify that the model is a good fit for both RTD, RTC, smart-band, and Abas datasets.Figure 5
Model accuracy and loss curves for RTD dataset. (a) Accuracy over iterations. (b) Loss over iterations.
(a)(b)The confusion matrices for the RTD dataset are shown in Figure6. For the RTD dataset, 0 and 9 were detected with the maximum accuracy and 5 and 2 with the poorest accuracy because they are challenging to write, ambiguous, and similar in shape.Figure 6
Confusion matrix for the RTD dataset.Table2 and Figure 7 show that the RTC dataset exhibited uniformly good performance. The best results were found for the character I, J, L, T, Y, and Z. All are distinguishable and unique characters. The worst result was that for F. There were 14 false positives for R. This is because of the similar writing patterns. During air-writing, the start and endpoints are the same for F and R. Similarly, there were three false-positive results for O. However, these results are reasonable. All other characters were reliably detected.Table 2
Character-wise performance comparison between different datasets and algorithms. The accuracies are in %.
CharacterRTC [23]Smart-band [24]Abas [25]CNN [22]LSTM [22]TARNetTARNetMLP-BpMLP-RpropDBN-BpDBN-RpropTARNetA99.8710099.1097.5399.3510010010099.88B99.6199.3599.8798.27928699.6899.03100100C99.7499.8799.8798.2794.8110099.3510099.88D10010099.8677.7792.8810098.7110099.88E99.7499.7499.6198.0292.5110099.3510099.76F98.7199.2297.9497.5397.7399.6899.0399.3599.76G99.4899.7499.8797.7777.6710098.3899.6899.88H99.8799.8799.8797.1690.2999.6899.6899.6899.88I99.6099.7310098.1490.5897.4095.4598.38100J10010099.9196.0496.7410010010099.76K99.8799.8799.8795.6794.4899.0398.7010099.64L10010010097.4099.6810010010099.88M99.7399.4798.8698.5191.5399.6799.0299.67100N99.4899.4899.7498.0298.3899.3599.3599.3599.40O99.7499.8799.8797.1695.4599.0398.3899.3599.52P99.3599.3599.4894.0795.4499.6799.6710099.64Q99.8799.7499.8798.0291.5399.6799.3599.6799.88R99.4899.7499.7479.9091.8699.6795.4499.67100S99.7499.8799.7497.1693.1699.3599.3510099.76T99.6199.8710097.7793.1810099.0310099.88U99.2299.8799.6198.2794.1610010010099.88V99.3499.2199.7397.9099.03100100100100W99.8798.0699.8798.2795.4499.3510010099.88X99.8799.8799.8798.0295.1199.3510010099.76Y10010010095.0698.0599.3599.6799.67100Z10010099.9298.7677.2099.3598.3798.05100Figure 7
Confusion matrix for the RTC dataset for the proposed CNN-LSTM network.The false-positive rate was higher for the smart-band dataset shown in Table2 and Figure 8. However, usually, accuracy was satisfactory. The highest false positives were P (163), D (35), and T (12) for D, J, and K, respectively. Except for character D, other results are satisfactory. A more complex algorithm and a better normalization technique might solve the problem.Figure 8
Confusion matrix for the smart-band dataset for the proposed CNN-LSTM network.In the case of the Abas dataset, the best result was found for the character B, M, V, Y, and Z; the details are shown in Table2 and Figure 9. Some character shows better result for DBN-Rprop than the proposed method. However, the overall result is better than theirs (shown in Table 3).Figure 9
Confusion matrix for the Abas dataset for the proposed CNN-LSTM network.Table 3
Performance comparisons.
DatasetNormalizedAlgorithmAccuracyRTD [22]YesCNN [22]99.06LSTM [22]99.17CNN [43]98.5TARNet99.63NoCNN [22]98.26LSTM [22]98.68CNN [43]98.2TARNet99.4RTC [23]YesCNN [22]97.73LSTM [22]97.12CNN [43]97.3TARNet98.74NoCNN [22]97.71LSTM [22]96.98CNN [43]97.64TARNet98.4Smart-band [24]YesCNN [22]75.97LSTM [22]67.11CNN [43]73.17CNN (KNN-DTW)83.2TARNet95.62NoCNN [22]75.83LSTM [22]67.11CNN [43]75.45TARNet80.72Abas [25]Normalization is not necessaryMLP-Bp93.43MLP-Rprop99.59DBN-Bp99.05DBN-Rprop99.71CNN [43]79.15TARNet99.92The character-wise performance of the TARNet is shown in Table2. We compared the method with previously implemented CNN and LSTM networks [22]. Different algorithms and normalization conditions were considered; the TARNet afforded superior accuracy for all normalized RTD, RTC, smart-band, and Abas data.The overall accuracy of the RTD dataset for normalized and nonnormalized for CNN [22], LSTM [22], CNN [43], and proposed TARNet is 99.06, 99.17, 98.5, and 99.63 and 98.26, 96.68, 98.2, and 99.4, respectively. Similarly, the accuracy for the RTC normalized and nonnormalized is 97.73, 97.12, 97.3, and 98.4 and 97.71, 96.98, 97.64, and 98.74, respectively. The accuracy is almost similar for CNN and TARNet algorithms for RTC normalized data. However, the accuracy significantly varies for nonnormalized data. The CNN [22], LSTM [22], and CNN [43] accuracies for the smart-band dataset were much lower, perhaps because the data were noisy (Figure 1). The actual result is better than CNN [22], LSTM [22], and CNN [43] networks, because the dataset was modelled with the KNN-DTW algorithm [24]. However, the TARNet algorithm still performed well (95.62%). The normalized and nonnormalized data accuracies were 95.62 and 80.72%, respectively. We found that Abas Leap Motion dataset is very organized. Hence, an excellent result is found. The accuracy of DBN-Rprop was 99.71%; on the other hand, the proposed algorithm achieved 99.92%. Abas dataset is a little different than the trajectory dataset. Abas et al. used a special algorithm during preprocessing to convert 3D trajectory data into a 2D binary image so that it can be compared with the MNIST dataset (Figure 1(d)). That is why normalization is not necessary and has no impact on it.


## 6. Conclusion

In this work, a combination of CNN and LSTM named TARNet is developed where the CNN and the LSTM are simultaneously used for feature generation and recognition, respectively. As a result, the network is time-efficient and exhibits high performance. Accuracy was verified using the publicly available RTD, RTC, smart-band, and Abas datasets. The benchmark result outperforms those of existing standalone CNN and LSTM models. The accuracies were 99.63, 98.74, 95.62%, and 99.92% for the RTD, RTC, smart-band, and Abas datasets. The overall accuracy is remarkable; however, some digits and characters exhibited higher error rates because of differences in writing styles. In the future, we will resolve these issues. Also, more datasets will be collected and tested.



---

*Source: 6063779-2022-09-24.xml*