"SECTION II.Methodology
The whole process of hand gesture recognition contains five elements: data acquisition, dataset collection, preprocessing, feature extraction and classification. The workflow for hand gesture recognition is illustrated in Fig. 1.
A. Data Acquisition
The collected data are used for model training. The configuration of the computation unit uses Anaconda Spyder, Python 3.8 version and runs on MacOS PC with a dual-core Intel i5 1.6 GHz CPU and 16.0 GB of RAM. Data acquisition must be conducted with a leap motion controller for training the model for hand gesture recognition. By connecting a leap motion controller to the computer, it offers an API that enables the user to acquire information about hands and fingers, such as palm center position, fingertips position, hand sphere radius and so on. As the input sensor device, 3D real time hand gestures performed and captured by the controller will then translate into a 3D spatial skeleton [1]. The geometry, entire movement of the detected hand posture, vectors like the rotation and position of fingers and hands and joints in 3D coordinates can be contained.

B. Dataset Collection
A dataset is collected under unconstrained environments such as different backgrounds, varying light intensity, skin color, hand shape, size, and geometry. The preformed hand gestures are obtained with the leap motion controller hardware. The custom dataset is loaded with 1600 samples with 5 different classes shown in Figure 4.1, where each class has an average of 320 images. 4-fold cross validation is applied, and the pixel size is 100×100. The train and test data account for 67% and 33% of the dataset, respectively.

C. Pre-Processing
Pre-processing may have an impact on the recognition result. It can partly remove the noise from the obtained data in order to increase the accuracy of hand gesture recognition. In this phase, the images from the selected dataset should be applied for image segmentation and hand extraction that aims to prepare for further handling. Image processing, like converting images to grayscale images, skeletonize and resizing, are performed by the algorithms.

Skeletonization is also recognized as a result of the thinning processing. It is a significant procedure in the pre-processing stage that is enabled to remove pixels on the object margins in a binary image. The action of peeling the object contour continues until it achieves the most medial 1 pixel width or no more pixels can be removed [6]. In order to facilitate the accurate and fast image processing, skeletonization should be used.

The skeletonization algorithms are from OpenCV. OpenCV which is the Open-Source Computer Vision Library to conduct pre-processing. It is a library of programming functions basically aimed at real-time computer vision. A binary image is an image with the pixel value of between 0 to 255, where 0 indicates pure black and 255 represents pure white.

Images from the dataset are resized into 100 x100 pixels. In most of the cases, a digital image is a 2D plane of pixels that obtains the width and height. In the hope of resizing an image, the code of resize is (x,x), given the specified width and height.

Threshold value filtering is adopted for rectifying the images. After finishing the process of thresholding, grayscale images convert into binary images. Background noise or the non-hand part in the image can mostly be removed and the hand part can be retained. Based on a threshold value, the pixels with a value less than the threshold will be modified to 0 in black and those larger than the threshold will be modified to 255 in white.

The images from the dataset are converted to the skeleton and saved in tag image file format. Tiff is the abbreviation of this format, and is a computer file format for storing raster graphics images and handling images within a single file. Thus, the stage of pre-processing plays a vital role in improving hand gesture recognition rate. It is necessary to implement data preparation before classification.

D. Feature Extraction
The information captured by the leap motion controller should be analyzed for extracting robust characteristics. By applying the obtained information, hand gestures can be distinguished. The center of the hand and the spot of each nonoverlapped fingertip is provided by a leap motion controller. However, merely acquiring six points of data is greatly exiguous. Fortunately, the useful features can be gained from the limited data through calculation. Lastly, the return data from the leap motion controller are extracted with specific features. The suggested feature extraction approach developed by Hisham & Hamouda [7] are utilized in this study.

Vector of finger i is calculated by:
v⃗ i=pi−c(1)
View SourceRight-click on figure for MathML and additional features.where i∈{thumb,index,middle,ring,little} and indicates the index of a finger. Additionally, c denotes the center of hand, and pi denotes the fingertip location.

There are totally four connecting vectors used to generate the feature vector F=[f1,f2,f3,f4], where f1, f2, f3 and f4, defined by equations (2) to (5), denote criterion of the finger's vector, angles between vectors & adjacent fingers, angles between finger vector and the normal of hand, and curvature of hand (r) and number of fingers (n) detected in generated sphere's radius, respectively.
f1=[∥v⃗ thumb∥,∥v⃗ index∥,…,∥v⃗ little∥]f2=⎡⎣⎢⎢⎢⎢⟨v^thumb,v^index⟩⟨v^index,v^middle⟩⟨v^middle,v^ring⟩⟨v^ring,v^little⟩  ⎤⎦⎥⎥⎥⎥f3=⎡⎣⎢⎢⎢⎢⟨v^thumb,v^normal⟩⋅⋅⟨v^little,v^normal⟩⎤⎦⎥⎥⎥⎥f4=[r,n](2)(3)(4)(5)
View SourceRight-click on figure for MathML and additional features.

E. Classification
Images can be classified by extracted features using different classifiers. In this study, five classifiers are used to train and test the hand gesture recognition accuracy rate. These classifiers include support vector machine (SVM), K-nearest neighbors (KNN), convolutional neural network (CNN), deep neural network (DNN) and decision tree. SVM is a robust mathematical supervised machine learning algorithm, which is applied to classify binary data and linearly separable data. It can be used to make classifications in a 2-dimensional plane and even a multidimensional hyperplane, and is vastly applied in predicting the outcome after training. For the KNN classifier, the k nearest neighbor acts in an essential role in the accomplishment of KNN. The learning algorithm is based on the k value's attribute in classifying the hand gestures. It should be counted as the critical tuning parameter in KNN [8]. In this study, k values are set in a range of 1 to 20 for choosing an ideal parameter of the KNN algorithm. For the CNN classifier, loss function should be used. The performance of CNN for comparing with validation data is assessed by the loss function. Once the differences between the training and validation data are evaluated, the result of errors is transmitted back for making improvements in the model. Finally, the model's setting can be adjusted and consummated. DNN is an artificial neural network with multiple hidden layers between the input and output layers. It has more hidden layers than shallow neural networks, and is typically a feedforward network in which data flows from the input layer to the output layer without looping back. Decision tree is commonly used for classification and regression problems. They primarily study the hierarchy of if or else questions in order to do decision making.

The fluctuation of accuracy and loss rate are the indicators of the CNN model performance in the training process [9]. Training loss and validation loss, training accuracy and validation accuracy are the parameters for estimation. The difference between loss and accuracy is that loss is not a percentage. It is the sum of errors made during the training processes. Training loss is measured during each epoch and the validation loss is measured after each epoch. Epochs are the number of times that the algorithms work through the whole training dataset. Theoretically, the larger the number of epochs leads to minimize the error and achieve better performance of the model. This is because running the model for more epochs enables it to learn more from the given information and obtains less loss rate. However, the risks of overfitting should be borne. Overfitting may occur when the model fits the training data too closely. Training loss keeps declining whereas the validation loss remains unchanged or increasing. The learning curves assist to diagnose whether the classification model is over-fitted, optimized to the training data or under-fitted. Thus, the ideal number of epochs is not that essential. The training and validation loss should be more significant. During training a model, the number of epochs should be increased until the loss function has reached a plateau to gain better performance.

For classification, machine learning algorithms from scikit-learn are adopted to perform the hand gesture recognition evaluation. Sklearn is the abbreviation for scikit-learn. Scikit-learn is an open-source software ML library for the Python Programming language. It contains a large number of advanced ML algorithms and is commonly used in the academic world and industry [10]. Scikit-learn is designed for interactive operation with Python numerical and scientific libraries and matplotlib. It should be considered as a simple and effective tool for predictive data analysis."
From the above paper, summarize the hand gesture recognition work flow.