# Automatic Lip Reading System Based on a Fusion Lightweight Neural Network with Raspberry Pi
## 3 Experimental Dataset and Results
### Results and Discussion  
In this section, we evaluate the designed mobile lip reading recognition system, and analyze and compare the results on our dataset. We randomly disrupt the dataset and divide the training set and the test set according to 90% and 10%. We built MTCNN and LSTM networks with PyTorch. The random gradient drop method is used to train the network. The training model is inputted in 64 units. The learning rate of the first 100 iterations is 0.1, and then changed to 0.001 (in order to speed up the convergence rate).  
We choose Raspberry Pi (Raspberry Pi 4, 4 GB of LPDDR4 SDRAM, Dual monitor support, at resolutions up to 4K) based on the Linux system to realize dynamic lip reading recognition on the mobile end, as shown in Figure 8. Compared with the general PC computer platform, Raspberry Pi has the advantages of small size, low power consumption, and low cost. It can complete some tasks and applications that a PC platform can normally realize.  
Figure 7: Neural network model based on weight analysis.  
In order to evaluate the performance of the mobile lip reading system, we compared the mainstream research methods [28; 29] through a large number of experiments, and the results are shown in Table 1. The proposed method can reduce a large number of parameters and reduce the complexity of the model and does not significantly degrade the performance of the model. We propose that the recognition time of the model is the time of video recognition, including decision- making. It can be seen that the recognition accuracy of the lightweight model proposed by us is smaller than that of the deep convolution hybrid network, and the recognition speed is greatly improved, which can meet the deployment and application of the mobile terminal.  
The proposed system can be adapted well to the real environment without excessive degradation of model performance.  
The training dataset and the test dataset are input into two MobileNets respectively, and then the sequence features of \(4096\times 10\) are extracted with the same LSTM model. Loss, accuracy, and recall of each period are shown in Figures 9 and 10.  
In Figure 9, when the period (epochs) is about 19, the loss tends to be stable, indicating that the optimal solution has been reached at this time. The accuracy of the proposed network model in the test dataset is 86.5%. The test set performs very well, and the accuracy and loss eventually tend to balance, which shows that the spatial and temporal characteristics have been learnt.  
As we want to identify the results as accurately as possible and reduce the situation of confusion recognition, we therefore pay more attention to the recall evaluation index of the model. Figure 10 below is the recall of our proposed model and the recall of the comparative model. It can be seen that our model performs well in pronunciation 2, 7, and 8, which is of great significance compared with previous studies. Considering the above experimental considerations, our research can be deployed well in the mobile terminal, and the efficiency is high.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Network** & **Accuracy** & **Time** &
\begin{tabular}{c} **Model Parameter** \\ **(Million)** \\ \end{tabular} \\ \hline BiLSTM + AlexNet (No data expanded) & 85.7\% & 10.0 s & 61 \\ AttentionLSTM+VGG16 (No data expanded) & 88.2\% & 16.3 s & 139 \\ LSTM + MobileNets (Data expanded) & 86.5\% & 7.3 s & 5.2 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Performance comparison of the mainstream research methods.  
Figure 8: Physical photo of Lip Reading System on Raspberry Pi.