# Human Skeleton Data Augmentation for Person Identification over Deep Neural Network
## 3 Motivation  
As reviewed in Section 2, the three machine-learning algorithms, \(k\)-NN, SVM, and MLP, have been widely used for person identification. In general, \(k\)-NN demonstrates better performance than the other methods [22; 23; 24; 25]. For verification of the performance, we implemented the MLP as in previousstudies [22; 23; 24; 25], which included a single hidden layer. The number of hidden units was set to 10, 20, and 40. We attempted to train the MLP using the simulation parameters used previously [22; 23; 24; 25], but failed to achieve the desired accuracy. From the investigation, we found that the numbers of hidden layers and hidden units of the MLP were too small to learn person identification tasks. To overcome this issue, we used a DNN that consists of two dense layers with 512 full connections in each layer. With the DNN, the training set can achieve an accuracy of up to almost 100% over the same datasets used previously [22; 23; 24; 25]. Figure 1 shows the training loss and training set accuracy of the DNN. As shown in the figure, after approximately 360 epochs, the training loss converges and the training set accuracy reaches approximately 100%. Nevertheless, the maximum test accuracy of the DNN is 57.5%. From this simulation, we found that overfitting occurs during the training phase, which in turn leads to significant performance degradation in the testing phase. To overcome the overfitting issue, we applied several techniques such as batch normalization [38], dropout [39], and L2 regularization [40] according to the number of hidden layers and the number of hidden units in each layer. However, we failed to avoid overfitting due to the shortage in the dataset. The dataset contains 160 subjects in total with 5 sequences for each. Therefore, this dataset contains a total number of 800 sequences \((=160\) subjects \(\times\) 5 sequences per subject). Assuming that 5-fold cross-validation is used, 640 sequences \((=160\) subjects \(\times\) 4 sequences per subject) can be used for the training, which is insufficient to train the DNN without overfitting. Therefore, to overcome the lack of data, we propose a novel data augmentation method to achieve both training set enrichment and overfitting prevention effectively. For this augmentation, we focus on increasing the number of anthropometric feature vectors rather than the number of 3D skeleton sequences based on the fact that the human body has bilateral symmetry. The main motivation for the approach is as follows: in general, the synthetic skeleton sequence can be generated by rotation and translation in 3D coordinates without loss of information [41; 42]. However, even if the number of skeleton sequences increases by generating such personal synthetic skeleton sequences, the anthropometric features extracted from the original and synthetic skeleton sequences are similar, as shown in Figure 2, so that there is a limitation to improve the performance by increasing the  
Figure 1: Training loss and training set accuracy of deep neural network. synthetic skeleton sequence only. Therefore, it is essential to augment anthropometric feature vectors for training the DNN more accurately while preventing the DNN from overfitting. Hence, we note that some of the anthropometric features, such as the average lengths of the left and right arms, forearms, thighs, and shanks, are reflected by the characteristics of bilateral symmetry of the human body. Inspired by this, we utilize this bilateral symmetry for data augmentation, which is highly effective to improve the performance on small-scale datasets. Figure 3 shows a schematic overview of our proposed method for skeleton-based person identification. In the proposed method, for each anthropometric feature vector, we augment new vectors by exchanging the anthropometric features extracted from the left (or right) side of the human body with the corresponding right (or left) features.