## 3 Proposal of Restoration of In-Air Signature
### Comparative Study of CNN  
LeNet [15] was one of the earliest proposed image recognition convolutional neural networks; it was first used for letter recognition. The LeNet structure, as shown in Figure 7, can be broken down into three fully linked layers, two sub-sampling layers, and two convolutional layers. (The LeNet concept is from [15] and the figure was created based on NN-SVG). One channel, six kernels, a stride of one, and a kernel size of five make up the first convolutional layer. Using a pool size of 2 and a stride of 2, we can construct the first pooling layer. The second convolutional layer has a stride of 1, 16 channels, 32 kernels, and a kernel size of 5. A pool size of 2 and a stride of 2 are likewise employed by the second pooling layer. The first fully connected layer, which follows the LeNet structure, is \(32\times 5\times 5\) to 120. The second fully linked layer is 120 to 84.  
The AlexNet [16] model structure is illustrated in Figure 8. (The AlexNet concept is from the original paper [16] and the figure was created based on NN-SVG.) For the original AlexNet structure, two GPUs are used, and the top and bottom parts are identical structures. It mainly contains 5 convolutional layers and 3 max-pooling layers.  
VGG [17] was first proposed in 2014, which illustrates 6 different structures of the VGG with subtle differences; the mostly used configuration is the D structure. Figure 9 shows the convolutional layer, max pooling layer, and fully connected layer, which give a better illustration of the structure. (The VGG concept is from the original paper [17] and the figure is finished based on NN-SVG). By substituting the large-size kernels with much smaller 3 by 3 kernels, which have the same receptive field, VGG can minimize the number of parameters [17]. We have 16 layers in the D architecture, comprising 3 fully connected layers and 13 convolutional layers. For the convolutional layers, the stride is 1 and the padding is 1. The max-pooling size and stride both equal 2. In a manner similar to AlexNet, we use 224 by 224 RGB input signature images, which are then passed into two 3 by 3 kernel layers with 64 kernels, max-pooling, two 3 by 3 kernel layers with 128 kernels, max-pooling,  
Figure 10: The in-air signature three-dimensional static restoration of a smartwatch.  
three 3 by 3 kernel layers with 256 kernels, max-pooling, three 3 by 3 kernel layers with 512 kernels, and a max-pooling layer. There are finally three fully connected layers.  
ResNet [18], which employs extremely deep convolutional neural networks, was initially suggested in 2015. To address the degradation issue, it suggested using residual blocks, and to speed up training, it proposed using batch normalization rather than a dropout. The residual block sums up the input matrix through the shortcut together with the feature matrix following convolution before passing it into the ReLU activation function [48]. A much deeper convolutional neural network may be used without deterioration because of the residual block. For ResNet, batch normalization can be used to adjust the feature map distribution after the convolution, which satisfies the average to be zero and the variance to be one. The transformer [11] consists of multiple sets of encoders and decoders. Each of the encoders has the same structure but does not share the same parameters. For the encoder, it consists of input embedding, multi-head attention, and a feed-forward network. For the decoder, it consists of masked multi-head attention, multi-head attention, and a feed-forward network. In our experiment, the performance of the transformer is mainly used as the baseline reference for the rest of the convolutional neural network models.