# Audio-Visual Speech Recognition Based on Dual Cross-Modality Attentions with the Transformer Model
## 5 Conclusions  
In this paper, we proposed an AVSR model based on the transformer with the DCM attention and a hybrid CTC/attention architecture. We constructed the DCM attention for proper alignment information between audio and visual modality even with noisy reverberant audio data, and applied a hybrid CTC/attention structure to enhance monotonic alignments. In general, our model provided better recognition performance than the compared models based on the transformer, even for out-of-sync data, and the hybrid CTC/attention loss further improved the performance. In the  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline \multicolumn{2}{c}{**Fusion Method**} & \multicolumn{2}{c}{**Concatenation**} & \multicolumn{2}{c}{**TM-DCM**} \\ \hline \multicolumn{2}{c}{**Objective**} & \multicolumn{2}{c}{**CE**} & \multicolumn{2}{c}{**CE**} & \multicolumn{1}{c}{**H**} \\ \hline \multicolumn{2}{c}{**Clean**} & \multicolumn{2}{c}{**9.6**} & \multicolumn{1}{c}{**8.8**} & \multicolumn{1}{c}{**8.7**} \\ \hline \multirow{5}{*}{
\begin{tabular}{c} Noisy \\ reverberant \\ \end{tabular} } & 20 & 18.2 & 17.5 & **16.9** \\  & 15 & 18.9 & 17.7 & **17.0** \\ \cline{1-1}  & SNR & 10 & 21.2 & 19.4 & **18.9** \\ \cline{1-1}  & (dB) & 5 & 25.2 & 22.3 & **22.1** \\ \cline{1-1}  & & 0 & 33.5 & 30.0 & **29.5** \\ \cline{1-1}  & & \(-\)5 & 47.3 & 42.7 & **41.7** \\ \hline \multicolumn{2}{c}{Avg.} & \multicolumn{2}{c}{24.8} & \multicolumn{1}{c}{22.6} & \multicolumn{1}{c}{**22.1**} \\ \hline \hline \end{tabular}
\end{table}
Table 4: WERs (%) for our model and decoding on simple concatenation of audio and video encoder outputs on the test sets of LRS2-BBC and LRS3-TED datasets. The boldface WERs denote the best performance in each condition. Abbreviation to understand the table: CE, cross-entropy loss only; H, hybrid CTC/attention loss.  
Figure 10: WERs for various model sizes with the numbers of (**a**) trainable model parameters and (**b**) multiply-accumulate operations (MACs). The data used to measure the MACs were “PSIBIZ3 hQKc/00011.mp4” in the test set of the LRS3-TED dataset.  
future, we will focus on more efficient fusion strategy of audio and video information and extend to audio-visual speech recognition including a speech enhancement model.  
Conceptualization, Y.-H.L., D.-W.J., R.-H.P., and H.-M.P.; methodology, Y.-H.L., D.-W.J., and J.-B.K.; software, Y.-H.L., D.-W.J., and J.-B.K.; validation, Y.-H.L. and J.-B.K.; formal analysis, Y.-H.L. and D.-W.J.; investigation, Y.-H.L., D.-W.J., R.-H.P., and H.-M.P.; resources, R.-H.P. and H.-M.P.; data curation, Y.L. and H.-M.P.; writing--original draft preparation, Y.-H.L., R.-H.P., and H.-M.P.; writing--review and editing, Y.-H.L., D.-W.J., J.-B.K., R.-H.P., and H.-M.P.; visualization, Y.-H.L.; supervision, R.-H.P. and H.-M.P.; project administration, H.-M.P.; funding acquisition, H.-M.P. All authors have read and agreed to the published version of the manuscript.  
This research was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (NRF-2017R1A2B4009964 and NRF-2020R1A2B5B01002398).  
The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.