# Efficient DNN Model for Word Lip-Reading
## 5 Evaluation Experiment
### Lrw  
Table 2 shows the top-1 accuracy and the number of parameters for each model described in Section 4. In the table, the upper row is the accuracy of other representative papers, and the lower row is the accuracy of this paper. \"--\" means that the reference does not show the number of parameters in this model. Although the recognition accuracy of this research did not reach the recognition accuracy of SOTA, the following things were clarified by examining various models.  
As a feature extractor, it can be confirmed that ResNet18 is superior to other models. The analysis of mouth movements, which is the target of our experiment, has a smaller  
\begin{table}
\begin{tabular}{l c c} \hline \hline \multirow{2}{*}{**Model**} & **Top-1** & **Params** \\  & **Acc. (\%)** & \(\times 10^{6}\) \\ \hline Multi-Tower 3D-CNN [4] & 61.1 & — \\ WLAS [10] & 76.2 & — \\
3D-Conv + ResNet34 + Bi-LSTM [40] & 83.0 & — \\
3D-Conv + ResNet34 + Bi-GRU [41] & 83.39 & — \\
3D-Conv + ResNet18 + MS-TCNN [6] & 85.3 & — \\
3D-Conv + ResNet18 + MS-TCNN + MVM [20] & 88.5 & — \\
3D-Conv + ResNet18 + MS-TCNN + KD [8] & 88.5 & 36.4 \\ Alternating ALSOS + ResNet18 + MS-TCN [42] & 87.0 & 41.2 \\ Voxk + MediaPipe + LS + MixUp + SA & 88.7 & — \\
* 3D-Conv + ResNet-18 + BiLSTM + Cosine WR [22] & 88.7 & — \\
3D-Conv + EfficientNetV2 + Transformer + TCN [21] & 89.5 & — \\
* 3D-Conv + ResNet18 + [DC-TCN, MS-TCN, BGRU] (ensemble) & \\ + KD + Word Boundary [18] & 94.1 & — \\ \hline
3D-Conv + ResNet18 + MS-TCNN (ours) & 87.4 & 36.0 \\
3D-Conv + ResNet18 + MS-TCNN + RA (ours) & 85.3 & 36.0 \\
3D-Conv + ResNet18 + MS-TCNN + ArcFace (ours) & 86.7 & 36.0 \\
3D-Conv + ResNet18 + ViT (ours) & 83.8 & 30.1 \\
3D-Conv + WideResNet18 + MS-TCNN (ours) & 86.8 & 36.0 \\
3D-Conv + WideResNet18 + Transformer (ours) & 79.2 & 11.2 \\
3D-Conv + EfficientNet-b0 + MS-TCNN (ours) & 80.6 & 32.3 \\ ViT + MS-TCN (ours) & 79.9 & 24.0 \\ ViViT (ours) & 72.4 & 3.9 \\ ViViT + RA (ours) & 75.6 & 3.9 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Recognition results (LRW).  
Figure 4: Extracted lipROIs (**left**: OuluVS, **center**: CUAVE, **right**: SSSD).  
difference in movements than in other video classification tasks, such as human action recognition. In addition, other feature extractors did not obtain the expected accuracy because the image's resolution was small. The model used in this experiment is a model that achieves high accuracy in image recognition, but these are usually input with high-resolution images such as \(224\times 224\) pixels. This research is video recognition, and it is necessary to put many images in the memory at once, so the experiment was performed at a low resolution due to memory constraints. Therefore, effective feature values could not be obtained in deep layers.  
It was found that MS-TCN tends to obtain recognition accuracy in the inference part of the latter stage. As for MS-TCN, we conducted experiments using RA and ArcFace, but the recognition accuracy decreased. We suspect that this is because the application of the RA-generated image data is unsuitable for the task or the model's generalization performance deteriorated due to the excessive application to the training data.  
We consider the details of the recognition results for 3D-Conv + ResNet18 + MS-TCN, which had the highest accuracy among the models investigated in this paper. Among 500 words, 15 words with low recognition rates are listed in Table 3. In the table, the 4 th column shows words with many mistakes, and the numbers in parentheses are the misrecognition rate. From the table, it can be confirmed that words with a low recognition rate are misrecognized as similar words. Among the 500 words, 252 had a recognition rate of 90.0% or higher.