# An Investigation into Audio-Visual Speech Recognition under a Realistic Home-TV Scenario
## 4 Experiments and Analysis
### Experimental Results  
#### 4.2.1 Results of the Pre-Training Strategies  
**The results of hybrid system (HS) pre-training. ** The results of pre-training with force alignment (FA) from the hybrid ASR system are listed in Table 1. From this table, we can make several observations: (1) the splicing of continuous segments (CAT) can effectively expand the audio data and improve the performance of ASR, as shown in the first and second rows of Table 1; (2) when the training data size is not large enough (i.e., the 100 h original far-field audio-visual training data), the AVSR model using the E2E framework (CER = 52.26%) is worse than the conventional hybrid AVSR model (CER = 44.12%), which indicates that the AVSR system based on the E2E architecture overfits when there are not enough training data. This is also our motivation to explore the pre-training strategies; (3) through utilizing the force alignment (FA) from the hybrid ASR system for pre-training, the AVSR system has been significantly improved and under the same architectures the CER decreased by 18.84% (from 52.26% to 33.78%); (4) after using the HS pre-training strategies, the recognition performance of end-to-end AVSR (CER = 33.78%) was significantly better than that of hybrid AVSR (CER = 44.12%), which also shows the effectiveness of our pre-training strategies. **The results of end-to-end system (ES) pre-training. Table 2 presents the results of the end-to-end system (ES) pre-training. Note that, for the ASR-E2E system, we also use the hybrid system pre-training methods to perform the system initialization. From this table, we can draw the following conclusions: (1) Through utilizing the HS pre-training methods, the ASR-E2E system (CER = 36.81%) can achieve better performance than the ASR-Hybrid system (CER = 41.69%) in Table 1. (2) The recognition performance of the ASR-E2E system will be further improved by applying both SP and CAT methods at the same time. When we only use the CAT method, the CER is 36.81%. When we only use the SP method, the CER is 37.44%. When we use both SP and CAT methods simultaneously, the CER is 35.90%, indicating better performance. (3) The ES pre-training (CER = 32.21%) can achieve better results than HS pre-training (CER = 33.78%) in Table 1, which may have two reasons. First, the ASR-E2E model achieved better performance than the ASR-Hybrid model. Second, the ES pre-training initialized more modules in the AVSR system, as shown in Figures 2 and 3. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Task** & **Model** & **Pre-Training** & **Data Processing** & **Eval** \\ \hline \multirow{3}{*}{ASR} & \multirow{3}{*}{Hybrid} & \multirow{3}{*}{N} & N & 45.92 \\ \cline{3-5}  & & & CAT & 41.69 \\ \hline \multirow{3}{*}{AVSR} & Hybrid & N & N & 44.12 \\ \cline{2-5}  & E2E [36] & N & N & 52.26 \\ \cline{1-1} \cline{2-5}  & E2E & HS (CAT) & N & 35.93 \\ \cline{1-1} \cline{2-5}  & E2E & HS (CAT) & CAT & 33.78 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: The results (CER %) of hybrid system pre-training. HS means the hybrid system. CAT means splicing the continuous segments. Similarly, combining CAT and SP techniques for data augmentation still yields the best results in AVSR systems. **The results of pre-trained audio-visual representation. Table 3 presents the results of utilizing the pre-trained audio-visual representation. Due to the system with pre-trained audio-visual representation being unable to converge on the model with the original configuration (M = 12, N = 6), as shown in the last row in Table 3, we also used other configurations (M = 6, N = 3 and M = 10, N = 4). As can be seen, the introduction of pre-trained audio-visual representation brought performance degradation to the AVSR system (the CER increased from 38.27% to 42.78%). Furthermore, increasing the number of model parameters (i.e., M = 10, N = 4) only leads to marginal improvement (the CER decreased from 42.78% to 41.85%).