# Audio-Visual Speech Recognition Based on Dual Cross-Modality Attentions with the Transformer Model
## 4 Experimental Results and Discussions
### WER Results  
Table 1 summarizes the word error rates (WERs) for our proposed model (TM-DCM), the TM-seq2 seq, and the AV align implemented on the transformer model (TM-av_align). The TM-seq2 seq model was implemented using modality-independent encoder-decoder attention as described in [20]. For fair comparison, the TM-av_align model was implemented by applying the cross-modal alignment structure in [13] to the TM-seq2 seq model used as the common baseline model and performed modality fusion with attention using audio as a query on the encoder side while our proposed model added the DCM to the baseline model. In the modality column, 'A', 'V', and  
Figure 6: DCM attention weight maps between audio and video encoders using audio or video features as queries in the TM-DCM for clean and noisy reverberant data. The used clean utterance was “PSBIZ3 hqKc/00011.mp4” in the test set of the LRS3-TED dataset, whose character label was “that is the real world and unless we find a way to globalize democracy or”. The input SNR for noisy reverberant data was 0 dB. The weights were computed by averaging over all the cross-modality attention heads. The weight maps for (**a**) clean audio query and video key/value; (**b**) noisy reverberant audio query and video key/value; (**c**) video query and clean audio key/value; and (**d**) video query and noisy reverberant audio key/value. Figure 7: Encoder–decoder attention weight maps using the DCM attention outputs as keys and values in the TM-DCM for the cross-entropy and hybrid CTC/attention losses. The weight maps were displayed for both clean and noisy reverberant data, and the used clean and noisy reverberant data were the same as in Figure 6. The weights were computed by averaging over all the encoder–decoder attention heads at all decoder layers. The weight maps for (**a–d**) the cross-entropy loss and (**e–h**) the hybrid CTC/attention loss; the weight maps between (**a,e**) audio encoders and decoders using clean audio input; (**b,f**) audio encoders and decoders using noisy reverberant audio input; (**c,g**) video encoders and decoders using clean audio input; and (**d,h**) video encoders and decoders using noisy reverberant audio input. 'AV' denote that audio-only, video-only, and audio-visual modalities were used, respectively. In the objective column, 'CE' and 'H' mean the objective functions based on the cross-entropy only and the hybrid CTC/attention loss, respectively. In addition, the numbers of parameters in the used models are presented, and our model requiring two additional attentions for DCM has 2.5% more parameters than the TM-seq2 seq. Table 2 describes architectural differences in cross-modality attentions between the three models since the cross-modality attention architectures are mainly different with audio-visual modalities. Regardless of the used methods, the WERs increased as the SNR decreased. The relatively significant differences in WERs between clean and 20-dB noisy reverberant data were possibly caused by the reverberation to simulate realistic situations. In the case of using the cross-entropy only as a loss function, our model consistently provided better recognition performance than the TM-seq2 seq and the TM-av_align regardless of the input SNRs. In particular, the WER of our model averaged over the two datasets achieved a relative improvement of about 16.9% for clean data compared to that of the TM-seq2 seq (much larger than the parameter growth rate) whereas the TM-av_align showed comparable or slightly deteriorated performance than the TM-seq2 seq. These results were because our model could effectively fuse the modality information by the video query cross-modality attention in addition to the audio query cross-modality attention similar to the AV align model. As shown in Figure 6, clean weight maps of the video query cross-modality attention in our model were helpful for recognition of noisy reverberant audio data as well as clean data.