# Intelligent Writing System for Patients with Upper Limb Disabilities

Authors: Mihaela Hnatiuc; Domnica Alpetri; Muhamad Arif; Dragos Vicoveanu; Iuliana Chiuchisan; Oana Geman
Journal: Journal of Sensors
 (2022)

---

## Abstract

Subjects with neurological problems and neurological disorders can be recovered in the dedicated recovery centers and at home. The paper presents a smart writing system that uses dedicated movements to recover specific problems of the hands. The system is based on a smart pen equipped with vibration, acceleration, and gyroscope sensors that give remote data to a computer via an ESP Arduino board via Bluetooth. The purpose of the system design is to serve in applications such as recognizing the writing, drawings, or emotional and physiological states of the writer. The data is preprocessed by extracting the characteristics. They will be used in a prediction and classification system using machine learning (ML) algorithms. The paper proposes a new method of using the dynamic processing of certain geometric shapes that are recognized and analyzed to diagnose the mobility of the subjects’ hands. Using methods of clustering data as K-means and classification as support vector machine (SVM), the results obtained from the data analysis using the TensorFlow platform have an accuracy of over 70%. Due to the adaptation algorithms used, the system can be customized, learning the hand movement of the subject.


---

## Body

## 1. Introduction

With the growing use of computers in society, human-computer interaction has become part of everyday life. One of the existing barriers, according to [1], in human-computer interaction is the way subjects interact with the high flow of information. The solution is to develop “natural” computer interaction techniques, similar to those by which people interact with each other. Through nonverbal forms of communication, made by moving hands, gestures become part of a computer dialog language used for information exchange [2]. Motion detection is a used term that simplifies many processes, reducing consumption and real-time response time. The most common motion sensors used are the accelerometer, gyroscope, and magnetometer to measure linear and rotational movements. For example, MPU 6050 is a motion sensor used in movement identification applications that has an accelerometer and a gyroscope on a single chip called an inertial measurement sensor (IMU) [3, 4]. An online handwriting solution, where data is achieved through depth sensors, is presented in [5]. Users can write in air and the algorithms can recognize in real time using the proposed representation. The method uses an effective fingertip tracking approach and reduces the need for a pen-up/pen-down shift, obtaining a 97,59% recognition accuracy for character recognition.A new method of designing and implementing of an infrared sensor system, InfraNotes, is described in [6], to automatically record grade notes on the board by detection and the analysis of the teacher’s hand. Compared to existing techniques, the system does not need special accessories, such as sensor-enabled pixels, surface writing, or video recording infrastructure. Similar research is presented in [7] using the Kinect sensor where it is possible to recognize numeric and alphabetical characters written by hand in the air. The Kinect sensor can capture motion without the sensor device being attached to the user’s body. Most people are not used to writing in the air and are not familiar with the Kinect sensor, and it takes some time to control both of them. Firstly, the user must get used to using the sensor; the average recognition rate is 95,0% and 98,9%, obtained for numeric and alphabetical characters. Studies to identify finger movements have been carried out to identify hand-made forms [8–10]. In [11], a vector machine support (SVM) using a hand gesture identifier (Figure 1) is used in the observation phase to identify those data segments containing handwritten characters. The recognition stage uses the hidden Markov models (HMM) to generate a text representation from motion sensor data. Individual letters are processed using HMM and concatenated to a word. The system can continuously recognize traditional sentences based on a free vocabulary. A statistical language model is used to improve the recognition of performance and to restrict the search space.Figure 1
Gesture identifier device [11].A Wi-Fi data acquisition system [12] has been defined in e-health research, and the results have been recently published. A fairly complex system with both hardware and software components has been developed and tested to allow physiological tremors to be investigated. This device includes the Wi-Fi module and a 3-axis acceleration sensor, and data received from the sensors are processed into an application developed in the LabVIEW environment. The signal acquired from the acceleration sensors is processed with a Butterworth filter, and an FFT (fast Fourier transform) analysis is performed. The data acquisition system can communicate directly with a router and transmit data to a server or smartphone. The software architecture of the system is presented in detail, considering the different situations in which it is used. A group of healthy subjects who offered themselves as volunteers are monitored during trials, and data analysis is performed online using the developed programs. Comparative data for signal amplitude and frequency for each axis of the accelerometers are presented in the paper, following different scenarios. A low-power Wi-Fi-based system has been experimentally developed and tested, designed to monitor physiological and pathological tremors. The prototype delivers performance features such as low-power consumption, battery life, Wi-Fi data transmission possibility, and relatively small size. The identification of human emotions using inertial sensors is another study presented in [13]. The task of recognizing a subject’s condition by writing is actively investigated, due to the high variety of possible types of writing (in terms of its appearance), when the subject is in different emotional states. In [14], a tool has been developed to effectively classify the emotions and gestures of a human subject into “typical” and “atypical” during a certain type of interactions.Two types of neural network architectures have been used to classify human gestures and emotions resulting from infrared cameras. These architectures can be used in parallel (for faster and more robust processing) or only convolutional neural network (CNN) for processing the characteristic features. Therefore, two types of modular hierarchical architectures are applied, one for the task of recognizing emotions and the other being represented by human actions and used to solve a real problem, as regards the classification of human behavior in “typical” and “atypical” relative to a particular task. Various projects have been developed, using accelerometer sensors, to identify the person after walking using the inertial sensor in mobile phones [15–17]. Other applications have the basis idea of developing an intelligent pen that can identify the person’s emotions [18, 19] and also to be used as a medical recovery tool.Taking into account the research presented above, this paper describes an intelligent writing system based on sensor-based handwriting recognition methods that allow the continuous observation and recognition of a drawing, considered a special type of gesture. Three drawing gestures are recognized: horizontal, vertical lines, and circles. Gestures can be used to develop interfaces that allow the recovery of people with hand disabilities for integration into the activities of daily living. Gestures are identified using relevant signal segments in the continuous data stream.The paper is organized into four sections. The first section presents the state of art and the second section describes the methods and algorithms used in signal processing. In the third section are presented the details of the intelligent writing system, and the results and discussion are presented in the fourth section, followed by conclusions.


## 2. Standalone Models

Based on the type of techniques, the independent models were divided into two categories: statistics and machine learning (ML).


### 2.1. Statistics

Several conventional statistical time series forecasting models have been widely used in nonlinear time series resolving problems and showing outstanding performance. In this paper, we focus on the most recent approaches, such as the Savitzky-Golay (i.e., SG) filter which is a noise elimination method widely used in different domains. The SG filter is a digital filter that has two design parameters, window length and filter order. As the length of the window increases, the estimation variance decreases, but the bias error increases at the same time [20]. The signals are preprocessed statistically for the extraction of the characteristics to identify the type of hand movement. The data after the preprocessing step is normalized using the standard min-max procedure.
(1)xinorm=xi−xminxmax−xmin.One of the parameters used in signal processing is entropy. If an experiment is conducted only once, producing eventi, it shall provide a quantity of information Ii. If the experiment is repeated several times and each possible event is likely to occur pi, total information will be obtained which depends on the information provided by each event [21].Thus, inN rehearsals of the experiment, with the frequency of occurrence of the event i and
(2)∑ini=N,the amount of information for each event is
(3)Ii=−logpi.The total information will be(4)It=∑iniIi.The average information will be written as(5)Im=∑niNIi.IfN is very large (theoretical N), limn⟶∞ni/N=pi, therefore,
(6)Im=∑pilog1p1=−∑pilogpi.The average information, noted withH, is called the Shannon informational entropy because its expression is analogous to the Boltzmann formula of thermodynamic entropy.
(7)S=−k∑i=1npilogpi.The difference between the maximum and actual informational entropy of an experiment, as noted withR, is called the absolute redundancy, R=Hmax−H. In practice, it is often used the size called relative redundancy, Rr.
(8)Rr=RHmax=1−HHmax.In the statistics are used the standard deviation and variance to measure dispersion. Therefore, the coefficient of variation (CV), which is defined as the ratio of the standard deviation to the mean, is used to measure the relative dispersion. The coefficient of variation is lacking the unit of measurement and is useful in comparing the variability between observation groups. Many researchers have proposed confidence intervals for the coefficient of variation [22]. Other parameters are the skewness and kurtosis coefficients described in the diagram illustrated in Figure 2.Figure 2
Brief description of kurtosis and skewness algorithms.The clustering algorithms are based on K-means, Silhouette, Silhouette score, Calinski-Harabasz (CH) index, and Davies-Bouldin (DB) index [21]. The Calinski-Harabasz index validates the number of classes based on the average of the square sum of the values inside and outside the class. The index (I) considers the separation based on the maximum distance between the center of the classes and measures the compactness based on the sum of the distances between objects and their cluster’s center. The Silhouette (S) index [13] validates the clustering performance based on the pairwise difference of between and within-cluster differences. In addition, the optimal cluster number is determined by maximizing the value of this index. The Davies-Bouldin (DB) index is calculated as such, for each C cluster, the similarities between C and all other clusters are calculated, and the highest value is attributed to C as similarity to its cluster. The DB index can then be obtained by averaging all the similarities of the cluster. The smaller the index, the better the grouping result.For signal, characteristics have been calculated the standard deviation, max peak, and min prominence peaks. The results were analyzed only as preliminary results and they were not used in the classification.


### 2.2. Machine Learning

A sequential neuronal network has been tested for the data classification system. In ML, the radial basis function kernel (RBF kernel) is a popular kernel function used in various kernel-based learning algorithms. In particular, it is commonly used in support vector machine (SVM) classification [23].The RBF kernel on two samplesx and x’, represented as feature vectors in some input space, is defined in [23, 24].
(9)Kx,x′=exp−∥x−x′∥22σ2,where ∥x−x′∥2 may be recognized as squared Euclidean distance between the two feature vectors. The σ is a free parameter. An equivalent definition involves the parameter
(10)γ=12σ2.Therefore,(11)Kx,x′=exp−γ∥x−x′∥2.Since the value of the RBF kernel decreases with distance and ranges between zero (in the limit) and one (whenx=x’), it has a ready interpretation as a similarity measure. The feature space of the kernel has an infinite number of dimensions. For σ=1, its expansion is
(12)exp−12∥x−x′∥2=exp22xTx′−12∥x∥2−12∥x′∥2=expxTx′exp−12∥x∥2exp−12∥x′∥2=∑j=0∞xTx′jj!exp−12∥x∥2exp−12∥x′∥2=∑j=0∞∑ni=jexp−12∥x∥2x1n1⋯xknkn1!⋯nk!exp−12∥x′∥2x1′n1⋯xk′nkn1!⋯nk!.For the accuracy score, it shows the percentage of the true positive and true negative to all data points. So, it is useful when the data set is balanced. For theF1 score, it calculates the harmonic mean between precision and recall, and both depend on the false positive and false negative. So, it is useful to calculate the F1 score when the data set is not balanced [25]. The spectral entropy (SE) of a signal is a measure of its spectral power distribution. The concept is based on the Shannon entropy or information entropy.


## 2.1. Statistics

Several conventional statistical time series forecasting models have been widely used in nonlinear time series resolving problems and showing outstanding performance. In this paper, we focus on the most recent approaches, such as the Savitzky-Golay (i.e., SG) filter which is a noise elimination method widely used in different domains. The SG filter is a digital filter that has two design parameters, window length and filter order. As the length of the window increases, the estimation variance decreases, but the bias error increases at the same time [20]. The signals are preprocessed statistically for the extraction of the characteristics to identify the type of hand movement. The data after the preprocessing step is normalized using the standard min-max procedure.
(1)xinorm=xi−xminxmax−xmin.One of the parameters used in signal processing is entropy. If an experiment is conducted only once, producing eventi, it shall provide a quantity of information Ii. If the experiment is repeated several times and each possible event is likely to occur pi, total information will be obtained which depends on the information provided by each event [21].Thus, inN rehearsals of the experiment, with the frequency of occurrence of the event i and
(2)∑ini=N,the amount of information for each event is
(3)Ii=−logpi.The total information will be(4)It=∑iniIi.The average information will be written as(5)Im=∑niNIi.IfN is very large (theoretical N), limn⟶∞ni/N=pi, therefore,
(6)Im=∑pilog1p1=−∑pilogpi.The average information, noted withH, is called the Shannon informational entropy because its expression is analogous to the Boltzmann formula of thermodynamic entropy.
(7)S=−k∑i=1npilogpi.The difference between the maximum and actual informational entropy of an experiment, as noted withR, is called the absolute redundancy, R=Hmax−H. In practice, it is often used the size called relative redundancy, Rr.
(8)Rr=RHmax=1−HHmax.In the statistics are used the standard deviation and variance to measure dispersion. Therefore, the coefficient of variation (CV), which is defined as the ratio of the standard deviation to the mean, is used to measure the relative dispersion. The coefficient of variation is lacking the unit of measurement and is useful in comparing the variability between observation groups. Many researchers have proposed confidence intervals for the coefficient of variation [22]. Other parameters are the skewness and kurtosis coefficients described in the diagram illustrated in Figure 2.Figure 2
Brief description of kurtosis and skewness algorithms.The clustering algorithms are based on K-means, Silhouette, Silhouette score, Calinski-Harabasz (CH) index, and Davies-Bouldin (DB) index [21]. The Calinski-Harabasz index validates the number of classes based on the average of the square sum of the values inside and outside the class. The index (I) considers the separation based on the maximum distance between the center of the classes and measures the compactness based on the sum of the distances between objects and their cluster’s center. The Silhouette (S) index [13] validates the clustering performance based on the pairwise difference of between and within-cluster differences. In addition, the optimal cluster number is determined by maximizing the value of this index. The Davies-Bouldin (DB) index is calculated as such, for each C cluster, the similarities between C and all other clusters are calculated, and the highest value is attributed to C as similarity to its cluster. The DB index can then be obtained by averaging all the similarities of the cluster. The smaller the index, the better the grouping result.For signal, characteristics have been calculated the standard deviation, max peak, and min prominence peaks. The results were analyzed only as preliminary results and they were not used in the classification.


## 2.2. Machine Learning

A sequential neuronal network has been tested for the data classification system. In ML, the radial basis function kernel (RBF kernel) is a popular kernel function used in various kernel-based learning algorithms. In particular, it is commonly used in support vector machine (SVM) classification [23].The RBF kernel on two samplesx and x’, represented as feature vectors in some input space, is defined in [23, 24].
(9)Kx,x′=exp−∥x−x′∥22σ2,where ∥x−x′∥2 may be recognized as squared Euclidean distance between the two feature vectors. The σ is a free parameter. An equivalent definition involves the parameter
(10)γ=12σ2.Therefore,(11)Kx,x′=exp−γ∥x−x′∥2.Since the value of the RBF kernel decreases with distance and ranges between zero (in the limit) and one (whenx=x’), it has a ready interpretation as a similarity measure. The feature space of the kernel has an infinite number of dimensions. For σ=1, its expansion is
(12)exp−12∥x−x′∥2=exp22xTx′−12∥x∥2−12∥x′∥2=expxTx′exp−12∥x∥2exp−12∥x′∥2=∑j=0∞xTx′jj!exp−12∥x∥2exp−12∥x′∥2=∑j=0∞∑ni=jexp−12∥x∥2x1n1⋯xknkn1!⋯nk!exp−12∥x′∥2x1′n1⋯xk′nkn1!⋯nk!.For the accuracy score, it shows the percentage of the true positive and true negative to all data points. So, it is useful when the data set is balanced. For theF1 score, it calculates the harmonic mean between precision and recall, and both depend on the false positive and false negative. So, it is useful to calculate the F1 score when the data set is not balanced [25]. The spectral entropy (SE) of a signal is a measure of its spectral power distribution. The concept is based on the Shannon entropy or information entropy.


## 3. The Proposed System

The system proposed in this paper is based on the sensors and acquisition card placed on the pen to identify the hand movement. The smart system uses a vibration sensor to identify the hand’s movements and an inertial sensor with an accelerometer and gyroscope (MPU 6050) to identify the movements in the writing process. The signal acquisition is performed with an Arduino ESP-32 board that collects data from the sensors and transmits it to the computer, for storage and analysis. The Arduino ESP-32 has a frequency of 80 MHz, and communication with the sensors is made via the I2C serial port (Figures3(a) and 3(b). The acquisition board communicates with the computer via Bluetooth.Figure 3
The intelligent writing system: (a) lateral view and (b) top view.
(a)(b)The MPU6050 sensor is a device used to monitor the movements on the three coordinated axes. It combines the gyro signals on the three axes with the accelerometer signals on the three axes, respectively, with the angular speed. The SW-420 vibration sensor transmits two logic signals (low or high/0 logic or 1 logic) depending on motion. If the motion is sudden, the sensor will send a high signal (1 logic), and if the movement is slow, the signal will be low. The lateral view of the system can be visualized in Figure4. The inertial sensor (IMU) is positioned under the vibration sensors on the writing tool, being responsible for recordings of the space motions in the 3D coordinates.Figure 4
Lateral view of the intelligent writing system with 3D axis representation.The vibration sensor data are not used in the signal’s classification because when the signal is “1 logic,” the system shows as the subject begins to draw. The trajectory used for the tests is presented in Figure5. The signals resulting from testing the system are shown in Figure 6. The vertical, horizontal, and circle lines are identified separately, and then, the shapes in the drawing are identified.Figure 5
The trajectory of the pen: the horizontal and vertical lines with the size of 7 cm and circle with the diameter of 7 cm.Figure 6
The signal from the accelerometer, gyroscope, and vibration sensors during the drawing of the circle and horizontal and vertical lines.Figure6 illustrates the signals acquired from the accelerometer, gyroscope, and vibration sensors during the drawing of the circle and horizontal and vertical lines. OX axis signals have higher amplitudes and are chosen for analysis.The signals acquired from the inertial accelerometer and gyroscope sensor during the 5 cm horizontal line representation with slight breaks between the drawings are presented in Figure7. The largest amplitude is the signal from the accelerator and gyroscope sensor in the OY axis. The OX and OY signals are synchronous; the OZ signals are not significant in this test.Figure 7
Representation of the signals from (a) the accelerometer and (b) gyroscope on the three axes for a horizontal line of 5 cm.
(a)(b)Analyzing the signal presented in Figure8, it can be observed that, during the drawing of a horizontal line with a size of 7 cm, the signal has a frequency in which after 2 small amplitudes a larger amplitude follows. The signals will not be the same because the person’s movements are not usually regular. The signals are acquired from testing the intelligent writing tool on the following drawings: (1) horizontal line of 5, 10, 15, and 20 cm; (2) vertical line of 5, 10, 15, and 20 cm; and (3) circle of 5 and 10 cm in diameter.Figure 8
Accelerometer signal in the OX axis during the drawing of the 7 cm horizontal line.Also, the following statistical parameters have been used: entropy, standard deviation, skewness, kurtosis, peak detection, max prominence peaks, min prominence peaks, min peaks, and half-width max peaks. The signals are filtered using the Savitzky-Golay (SG) filter. The identification of the writing process and the resting is achieved by isolating the signals acquired during drawing with the resting time (Figure9).Figure 9
Signals from the accelerometer on the OX axis: (a) the noisy and filtered signals for the horizontal line of 5 cm length; (b) the normalized and filtered signals with the Savitzky-Golay filter for the horizontal line of 5 cm length; (c) the noisy and filtered signals for the vertical line of 15 cm length; (d) the normalized and filtered signals with the Savitzky-Golay filter for the vertical line of 15 cm length; (e) the noisy and filtered signals for the horizontal line of 20 cm length; (f) the normalized and filtered signals with the Savitzky-Golay filter for the horizontal line of 20 cm length; (g) the noisy and filtered signals for the circle of 5 cm in diameter; and (h) the normalized and filtered signals with the Savitzky-Golay filter for the circle of 5 cm in diameter. Note: skewness ACCY: skewness coefficient of accelerometer signal OY axis; kurtosis ACCY: kurtosis coefficient of accelerometer signal OY axis; skewness ACCX: skewness coefficient of accelerometer signal OX axis; kurtosis ACCX: kurtosis coefficient of accelerometer signal OX axis; entropy ACCY: entropy coefficient of accelerometer signal OY axis; entropy ACCX: kurtosis coefficient of accelerometer signal OX axis; standard deviation ACCY: standard deviation of accelerometer signal OY axis; standard deviation ACCX: standard deviation of accelerometer signal OX axis.
(a)(b)(c)(d)(e)(f)(g)(h)The histograms are performed to observe the distribution of signals (Figure10). The standard deviation of the signals collected from the accelerometer on the OX axis is in the range of 0,1 to 0,2.Figure 10
The distribution of signals.It can be observed that the entropy for both OX and OY accelerations is about 6. The OX and OY signals are symmetrical, and the OX flattening is better than OY. The standard deviation is around 0,1 which shows a spread of values.The correlation matrix shows how much the characteristics extracted from the analyzed signals are related. The entropies on the OX and OY axes are related (0,92), which is also shown from the signal shape, confirming that the signals are synchronized. The characteristics of the OX axis signals are related to a coefficient with a value of -0,6. In order to draw the geometric figures, 71 files were used with an average of 1000 samples per file.After applying the K-mean and Silhouette index, it was observed that all signals (feature parameters) can be classified into three classes (Figure11). The Silhouette parameter has the highest value for these three classes. The number of classes used for trace identification was 3, corresponding with the classes resulting for the clustering algorithm.Figure 11
Silhouette score resulting from data clustering presented for a class number from 1 to 10 for the following features: (a) Silhouette score: skewness ACCY and kurtosis ACCY; (b) Silhouette score: skewness ACCY, kurtosis ACCY, and kurtosis ACCX; (c) Silhouette score: entropy ACCY and kurtosis ACCY; and (d) Silhouette score: entropy ACCX, entropy ACCY, and kurtosis ACCY.
(a)(b)(c)(d)Data analysis shows that the parameters most specified in the classification are entropy ACCY and kurtosis ACCY because the Silhouette score is the largest and the CH and DB indexes are the lowest (Table1). The number of samples used in the classification was 68.Table 1
Algorithm performance analysis using K-means.
ClassCentroidInertiaSilhouette scoreCalinski-Harabasz indexDavies-Bouldin indexSkewness ACCY and kurtosis ACCYHorizontal line0,02819,63Vertical line0,0846,471388,80,56175,960,57Circle-2,8635,81Entropy ACCY and kurtosis ACCYHorizontal line6,656,47Vertical line6,2319,90910,160,63265,80,41Circle6,1836,54Skewness ACCX, kurtosis ACCY, and kurtosis ACCXHorizontal line-7,781-3,460,047Vertical line13,90-2,350,470,4873,360,70Circle0,17614,32-0,929Skewness ACC, kurtosis ACCY, and kurtosis ACCXHorizontal line-10,796-0,10,01Vertical line2,640,150,0160,63264,620,42Circle19,28-0,160,015The correlation matrix is illustrated in Figure12 and shows how closely the parameters extracted from the signals are related. The entropy on the OX and OY axes is related (0,92), which is also extracted from the signal shape, confirming that these two signals are synchronized. The characteristics of the signals on the OX axis are related to a coefficient with a value of -0,6. In the classification, 71 files were used with an average of 1000 samples to draw the geometrical figures.Figure 12
The correlation matrix of the statistical parameters.


## 4. Results and Discussion

Different machine learning methods, such as decision trees, SVM, neural networks, k-nearest neighbor method, and the Bayesian classifier, can be used for classification. The SVM is the most used method because it removes overfitting and is not influenced by the noise. However, a nonlinear classification problem cannot be resolved by a linear classifier. It is possible to transform the nonlinear dataset in a way that allows it to be classified linearly. An approach to this problem is to use different kernels of different types of functions (i.e., polynomial).Several methods can be used for classification, including the SVM clustering algorithm, which is unsupervised classification. In this paper, data has been classified using the SVM clustering algorithm for drawing identification (Table2). Feature sets 2 and 4 give the best results in classification with an accuracy of more than 60 percent. The SVM was used with polynomial and RBF kernel. In both cases, the best classification accuracy was obtained by entropy ACCY and kurtosis ACCY data for Gamma=0,7 and C=0,9. Gamma is a scalar that defines how much influence has a single training point. Precision in each class is defined as the number of correct classified position data split by the total number date. Recall in each class is the ratio of the number of correctly classified position data to the number of positive values. The F1 score in each class is obtained as a measure of precision and recall in each class. For the F1 score, it can be observed that the horizontal lines are best classified using both SVC modes with polynomial and RBF kernel for the entire ACCY and kurtosis ACCY characteristics. If we analyze the number of forms used in the identification, it can be observed that for 12 horizontal lines, the F1 score is 0,7 which demonstrates that drawings can be identified using the proposed intelligent writing sensor-based system. These signals can be translated into a graphical mode that can be viewed remotely by a therapist.Table 2
Algorithm performance analysis.
ClassPrecisionRecallF1 scoreAccuracySVM with polynomial kernelEntropy ACCX and kurtosis ACCYHorizontal line0,730,670,70,57Vertical line0,50,380,43Circle0,2510,4SVM with polynomial kernelEntropy ACCX and kurtosis ACCYHorizontal line0,750,750,750,71Vertical line10,50,67Circle0,510,67SVM with RBF kernelEntropy ACCX and kurtosis ACCYHorizontal line0,560,830,670,64Vertical line0,750,430,55Circle111SVM with RBF kernelEntropy ACCY and kurtosis ACCYHorizontal line0,560,830,670,71Vertical line0,750,430,55Circle111In testing process, four data feature sets are used for clustering: (1) “skewness ACCY” and “kurtosis ACCY”; (2) “entropy ACCY” and “kurtosis ACCY”; (3) “kurtosis ACCX,” “skewness ACCY,” and “kurtosis ACCY”; and (4) “entropy ACCX,” “entropy ACCY,” and “kurtosis ACCY.”For classification, it was chosen as features of the entropy ACCY and kurtosis ACCY signals. A simple feed-forward full-connected artificial neural network has been developed. The classification task was achieved by using the TensorFlow 2.0 library along with the Keras module. The total number of parameters used in the dense 2 hidden layers of “relu” type, together with the output layer, was 323. On the output, the activation function is “softmax” and three neurons are considered, each representing a class—horizontal line, vertical line, and circle. In the first hidden layer, there are 20 neurons and in the second 10 neurons. The training is done for 200 epochs with a loss function of 0,59 and an accuracy of 0,79. The average drive time was 7 ms over an epoch.The loss function of 0,6737 and the accuracy of 0,7142 were obtained during system testing (Figure13). In future research, other types of CNN will be tested due to the obtained results which are more than 50% on the classification to identify the object drawn.Figure 13
The loss function and the accuracy represented against the epoch number.
(a)(b)Compared with other studies where the hand gesture identification/classification was made by image processing techniques, our idea is fairly simpler to implement, keeping at the same time the subject anonymous by using only sensors and signal processing algorithms. Comparing the results obtained by smart pen in [26] with the results obtained with the system developed in the laboratory, it is observed that the first system allows identifying the characteristics of writing and tremor with age. In the case of our system, it helps in recovering the subjects’ hand problems by making specific drawings indicated by the therapist.


## 5. Conclusions

The present paper describes a handwriting recognition method based on inertial sensors that allow continuous observation and recognition of a drawing, constructed from simple geometric figures. The architecture of the system comprises hardware and software elements to detect the hand of a subject’s normal (i.e., physiological) movements by detecting the proposed written gestures and solving the correct classification by various algorithms.The results obtained after the signal processing techniques lead us in two study directions. Firstly, they can give to the observer a good classification of the signals considering a set of their statistical parameters for different geometric tracks (i.e., simple or combined). Secondly, it can provide a valuable clinical tool regarding the hand movement evolution in a recovery process. Therefore, we intend to improve the actual system using an acquisition board that can be used to send the data to the cloud where it will be processed, and the results provide a basis for the therapists. The classification problem can be improved and therefore deep learning methods will be used. Data (i.e., physiological and pathological) will be analyzed to have good accuracy of the sketch identification. The system proposed can be used in the medical rehabilitation clinics. Because the system is based on adaptive algorithms, the therapist may suggest that the subjects draw certain figures based on the required recovery needs.



---

*Source: 1005061-2022-04-15.xml*