# Audio-Visual Speech Recognition Based on Dual Cross-Modality Attentions with the Transformer Model
## 1 Introduction  
Automatic speech recognition (ASR) has attracted much interest because speech is the most convenient, natural, and user-friendly interface to various kinds of devices. Unfortunately, a speech signal acquired in real-world noisy environments is significantly contaminated, and the performance of ASR systems with the contaminated speech signal is seriously degraded due to the mismatch between the training and testing environments. Although many approaches have been developed to accomplish robustness by compensating for the mismatch under specific conditions, most of them fail to attain robustness in real-world environments with various types of noise (e.g., [1; 2; 3; 4; 5]). Therefore, robust recognition remains a challenging but important issue in the field of ASR. As a result that visual information is not distorted by acoustic noise, visual speech recognition (known as lip reading) may play an important role in ASR in acoustically adverse environments [6]. Thus, visual speech recognition generally provides consistent recognition accuracies regardless of the signal-to-noise ratios (SNRs) of acquired acoustic speech whereas audio speech recognition getsworse recognition accuracies for speech with lower SNRs. However, it is well known that audio speech recognition with clean speech typically achieves higher recognition accuracy than visual speech recognition because speech without distortion may provide more sufficient and clear cues to classify phonemes than visual movement in a lip in addition to its face. Therefore, audio-visual speech recognition (AVSR) fuses audio and visual information acquired from a talking face with audio to achieve comparable or possibly higher recognition performance than audio speech recognition with clean speech and visual speech recognition in acoustically adverse environments [7]. As a result that features for visual speech recognition are not well-established in contrast to acoustic speech features, such as the logarithmic mel-frequency power spectral coefficients or cepstral coefficients, conventional visual features including histogram of oriented gradients [8], local binary patterns [9], and features with scale-invariant transforms [10] have been commonly used. As a result that these are rather general-purpose image features, the performance of visual speech recognition may be improved by devising effective features [11]. Influenced by impressive success of deep learning in not only diverse object detection and recognition tasks but also action recognition tasks, deep architectures such as convolutional neural networks (CNNs) and long short-term memory (LSTM) have been applied to AVSR [12]. In addition, deep learning provides ways to learn end-to-end recognition models without developing and training separated acoustic and language models indispensable for conventional speech recognition (e.g., [7; 13; 14; 15]). However, a mechanism fusing audio and visual information in AVSR should be still developed to achieve successful recognition performance in both acoustically clean and noisy environments. Intuitively, it is better to rely more on audio features than on visual features in clean environments. Therefore, AVSR is hard to train attentions with balanced modalities. In [16], modality attention computes scores for modality space at a certain time whereas conventional attention [17] computes scores for time space using a specific modality (query). However, the modality attention is applied assuming that all modalities have the same time length. As a result that audio and visual features are usually generated at different time steps, they have to be resampled to apply the modality attention. [PERSON] et al. proposed cross-modality attention that computed the video context using audio query (AV align) although conventional attention computes the video context using video query [13]. Although [PERSON] et al.