## 3 Experimental Dataset and Results
### Results and Discussions  
In this section, we evaluated our proposed neural network model, and the results were analyzed and compared in our dataset. The out-of-order dataset was randomly divided into 80% training dataset and 20% test dataset. We used pytorch toolkit to carry out the CNN and the attention-based LSTM network. It used a random gradient descent method to train the network in small batches of 50 units with the learning rate of 0.001. The weight of CNN was based on the parameters of the pre-trained model of VGG19, and the weight of attention-based LSTM and the full connection were randomly initialized. The visualization of CNN model was shown in Figure 6.  
In order to evaluate the improvement performance influenced by the attention mechanism, we tested and compared the general CNN-RNN architecture. As shown in Figure 7, the proposed network only added one calculation step (gray part) of the attention layer, and the other parts of the two architectures were identical to the initial parameters of the architecture. Figure 5: VGG19 network structure. The training dataset and the test dataset were respectively input into two CNN-RNN networks, then we used the same CNN (VGG19) to extract the sequence features of 4096 \(\times\) 10 and input them into two different RNN networks. The losses, accuracies, and visualizations of the attention mechanism for each period were shown in Figures 8 and 9. Figure 8: Comparison of our proposed attention-based model (blue) with CNN-LSTM model (red). (**a**) Comparison of losses between the two networks and (**b**) comparison of the accuracies between the two networks. Figure 6: The visualization processes of feature extraction. Figure 7: Our proposed architecture and the compared network architecture. In Figure 8, it could be seen that the recognition accuracy of the model with the attention mechanism was higher. When the epochs were around 15 times, the loss tended to be stable and it indicated that the optimal solution had been reached at this time. The accuracy of the proposed network model was 88.2% in the test dataset and 84.9% for the contrastive network (CNN-LSTM). Figure 9 shows the visualization of the attention mechanism, each line shows the weight of attention at each moment from zero to nine, a deeper color represents a greater weight of attention. It could be inferred that the attention area of each pronunciation was concentrated near the third and seventh frames, because the third, fourth, seventh, and eighth frames contained the main information of the lip motion. These frames were related to the video theme and contained a certain chronological order, which were considered to be important video frames, the model assigned a large amount of attention weight. This attention distribution showed that the attention mechanism optimized the proportion of keyframe and achieved the requirement of allocating more information processing resources to important parts. Therefore, our research used an architecture which was a fusion of CNN and attention-based LSTM. Obviously, the performance of our proposed architecture was more powerful. We tested the English words in ten independent utterances in two models. The experimental results were shown in Figure 10. The results show that the proposed model had higher accuracy in all independent digital word utterances and it was stronger than the general CNN-LSTM model. Furthermore, according to the analysis of the recognition results for each individual utterance of English words, the identification of \"two\" was the easiest, \"zero\" was the most difficult to identify. Complex pronunciation was difficult to recognize because of errors in individual pronunciation. The best-recognized utterance was \"two\" because the speakers' movements were consistent without regional differences, and the worst identified utterance was \"zero\" because the lip movements of speakers were complicated and a part of pronunciation required tongue to control syllables.