## 4 Results
### Convolutional Neural Network (CNN) Result  
As a preliminary experiment, we tested the multi-class classification on the LeNet first. The mean and standard deviations were calculated for the image set. The restored signature was first transformed to the necessary pixel size and gray-scale pictures as seen in Figure 6 because LeNet [15] only accepts 32 by 32 one-channel gray-scale images. We had 22 participants and 44 categories; thus, the final fully connected layer was 84 to 44, as shown in Figure 7. For training and testing, the model employed cross-entropy loss and ReLU activation functions [48]. For AlexNet [16], similar to the LeNet, the mean, and standard deviations were calculated for the signature image set. However, AlexNet took in three-channel images, so the mean and standard deviations are in a list format, which is [0.88680416, 0.9199834, 0.95062685] for the mean and [0.22626287, 0.16366783, 0.103204496] for the standard deviations. Different from LeNet, AlexNet took in three-channel colored images of 224 by 224 pixels as shown in Figure 6. The model used in this experiment is identical to the original structure, except for the last output layer. The original paper had 1000 categories, it changed to 44 to fit the smartwatch in-air signature dataset. The VGG [17] structure was modified from 1000 for the last fully connected layer that was initially used for ImageNet [49], which contained 1000 classes and 44 classifications. The size of the input did not change as it traveled through the convolutional layer but it was cut in half by the max-pooling layer since the max-pooling size was two and the stride size is two. To better fine-tune the model, here are the results for each model with respect to the learning rate. The cross-entropy loss and stochastic gradient descent (SGD) [47] were chosen for each model, and we ran 1000 epochs for each model. For each epoch, the batch size was selected to be 5 to best minimize the loss oscillation. Given the dataset is relatively small, the model used hold-out validation with 50% for training and 50% for testing to reduce the possible bias in the testing process. As shown in Table 4, to evaluate the effect of the learning rate on the model and dataset, we kept the loss function fixed to cross-entropy loss and the optimizer fixed to be stochastic gradient descent (SGD); the bold numbers show the highest testing score for each of the learning rates with the corresponding model. The highest testing score was 99.77% for ResNet with a learning rate of \(1\times 10^{-4}\), which is reasonable because ResNet uses a much deeper network and batch normalization to adjust the distribution of the feature map. Additionally, the best testing scores for AlexNet and VGG all exist in the \(1\times 10^{-4}\) learning rate field. [PERSON]'s research [50] also supports that for a relatively small dataset, choosing a lower learning rate can be more suitable for the model. The accuracy is expected to be lower if we expand our dataset, which is also one of the problems in this proposed recognition method if we want to apply the technique to industries. If more users are registered into the recognition system, and more classification categories are needed to be created, it may result in an unbalanced dataset with lower accuracy [51]. With the learning rate fixed to \(1\times 10^{-4}\) and the loss function fixed to cross-entropy loss, Table 5 shows the model performance with respect to different optimizers, in which 10 optimizers are tested for the performance. We ran 1000 epochs for each optimizer and a batch size of 5 to best evaluate the optimizer performance. As shown in Table 5, the training loss, train score, test loss, and test score were evaluated with respect to the optimizers. The 10 optimizers covered in the experiment were Adadelta [38], Adagrad [39], Adam [40], AdamW [41], Adamax [40], ASGD [42], NAdam [44], RAdam [45], RMSprop [46], and SGD [47].