# Identification of 3D Lip Shape during Japanese Vowel Pronunciation Using Deep Learning
## 3 Results
### Experimental Method  
K-fold cross-validation was used in the experiment to examine the relationship between 3D mouth shape and vowel class identification. The prepared dataset was divided into training, validation, and test sets. The number of records of training, validation, and test sets is presented in Table 4. In k-fold cross-validation, k = 10, the training set and validation set were interchanged. The training set, validation set, and test set were divided such that each vowel had the same number of records. The record no. used as the test data is shown in Table 5. Records other than those in Table 5 were used for k-fold cross-validation. Additionally, two types of training and testing were conducted: mixed-gender and male-only datasets. The combination of formant frequencies of male and female vowels is very different, and it is possible to discriminate between males and females according to frequency of pitch [20]; the purpose of this experiment was to determine the effect of gender on pitch frequency. The training epoch was set at 500, and the bit size was set at 16. The optimizer for the machine learning model was Adam [21], and the loss function was F.cross_entropy() from PyTorch. The training data should be used without padding. In order to compare the results with conventional methods, 2D images taken simultaneously during the acquisition of the 3D point cloud dataset were used to compare the results with those obtained using Xception [22]. Xception is considered to be one of the most accurate methods for classifying 2D images.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline  & **Training** & **Verification** & **Test** \\ \hline Mixed-gender & 180 & 20 & 45 \\ Male-only & 153 & 17 & 25 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Number of records used for training, validation, and testing.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Contents** & **Specifications** \\ \hline OS & Windows Server 2016 Datacenter x64 \\ CPU & Intel i7-7820X @ 3.6 GHz \\ Memory & 48 GB \\ GPU & NVIDIA GeForce GTX 1080 Ti 11 GB \\ Python & Version 3.6.11 \\ PyTorch & Version 1.7.0 \\ CUDA Toolkit & Version 10.1.243 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Computer environment in which learning and identification took place.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline  & **Test Data** & **Number** \\ \hline Mixed-gender & “A” & 1,21,36,56,71,111,116,121,141 & 9 \\  & “I” & 37,67,87,127,132,147,157,177,197 & 9 \\  & “U” & 38,43,48,53,73,123,168,208,228 & 9 \\  & “E” & 4,9,49,69,114,124,164,179,184 & 9 \\  & “O” & 35,75,95,165,170,180,190,215,225 & 9 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Record no. combinations of test data used in each experiment (training and verification data used for k-fold cross-validation used records other than those in this table).  
Figure 5: Progress of learning of training and validation data (accuracy).  
Figure 6: Progress of learning training and validation data (loss).  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & & \multicolumn{5}{c}{**Predicted**} \\ \hline  & \multicolumn{2}{c}{\({}^{\textbf{a}}\)**A\({}^{\textbf{a}}\)**} & \multicolumn{2}{c}{\({}^{\textbf{a}}\)**I\({}^{\textbf{a}}\)**} & \multicolumn{2}{c}{\({}^{\textbf{a}}\)**U\({}^{\textbf{a}}\)**} & \multicolumn{2}{c}{\({}^{\textbf{a}}\)**E\({}^{\textbf{a}}\)**} & \multicolumn{2}{c}{\({}^{\textbf{a}}\)**O\({}^{\textbf{a}}\)**} \\ \hline  & \({}^{\textbf{a}}\)A\({}^{\textbf{a}}\) & 78 & 7 & 0 & 1 & 4 \\  & \({}^{\textbf{a}}\)I\({}^{\textbf{a}}\) & 3 & 74 & 11 & 1 & 1 \\ Actual & \({}^{\textbf{a}}\)U\({}^{\textbf{a}}\) & 0 & 4 & 86 & 0 & 0 \\  & \({}^{\textbf{a}}\)E\({}^{\textbf{a}}\) & 4 & 5 & 3 & 68 & 10 \\  & \({}^{\textbf{a}}\)O\({}^{\textbf{a}}\) & 7 & 0 & 6 & 11 & 66 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Confusion matrix using test data (mixed-gender).