# Audio-Visual Speech Recognition Based on Dual Cross-Modality Attentions with the Transformer Model
## 2 Related Work
### Attention Mechanism  
Additive attention and dot-product attention are typical attention mechanisms. In case of the additive attention, [PERSON] et al. [33] applied Bahdanau attention [17] to LSTM-based audio speech recognition model. First, an encoder generates high-level representation \(\mathbf{h}=(h_{1},\dots,h_{U})\) for input \(\mathbf{x}=(x_{1},\dots,x_{T})\) with the number of frames \(T\geq U\), which is expressed as [33]  
\[\mathbf{h}=\text{Encoder}(\mathbf{x}). \tag{1}\]  
At decoder time step \(i\), the attention module computes the context \(c_{i}\) using the scalar energy \(e_{i,u}\) at every encoder-output time step \(u\) that is subsequently calculated by the decoder state \(s_{i}\) and the encoder output \(h_{u}\in\mathbf{h}\), \(1\leq u\leq U\) as [33]  
\[e_{i,u}=\left\langle\phi(s_{i}),\psi(h_{u})\right\rangle, \tag{2}\]  
Figure 1: An overview of our proposed audio–visual speech recognition (AVSR) architecture. It consists of four modules: encoders, dual cross-modality (DCM) attentions, an attention decoder, and a connectionist-temporal-classification (CTC) block. The encoders receive each modality and compress their information. DCM attentions are calculated by using different modalities for the queries and the keys. Finally, character probabilities are calculated by using both the attention decoder and the CTC block.  
\[\alpha_{i,u}=\frac{\exp(e_{i,u})}{\sum_{u^{\prime}}\exp(e_{i,u^{\prime}})}, \tag{3}\]  
\[c_{i}=\sum_{u}\alpha_{i,u}h_{u}, \tag{4}\]  
where \(\phi\) and \(\psi\) denote processing functions by multi-layer perceptron (MLP). Finally, the decoder state \(s_{i}\) and the output character \(y_{i}\) at decoder time step \(i\) are computed by [33]  
\[s_{i}=\text{Decoder}(s_{i-1},y_{i-1},c_{i-1}), \tag{5}\]  
\[P(y_{i}\mathbf{x},y_{<i})=\text{CharacterDistribution}(s_{i},c_{i}), \tag{6}\]  
where Decoder is composed of two-layer LSTM and CharacterDistribution is an MLP with softmax outputs over characters.  
On the other hand, [PERSON] et al. [34] used attentional vector \(\tilde{s_{i}}\) at decoder time step \(i\) to produce the predictive character distribution, computed from the decoder state \(s_{i}\) and source-side context vector \(c_{i}\), represented as  
\[\tilde{s_{i}}=\tanh(W_{c}[c_{i},s_{i}]), \tag{7}\]  
\[P(y_{i}\mathbf{x},y_{<i})=\text{softmax}(W_{s}\tilde{s_{i}}), \tag{8}\]  
where \([\cdot,\cdot]\) indicates concatenation of the two components and \(W_{c}\) and \(W_{s}\) are trainable parameters.  
The dot-product attention applied to the transformer model [24] calculates the dot products of the query with all keys, divides each by the square-root of the key dimension \(\sqrt{d_{k}}\), and applies a softmax function to get the weights on the values. The matrix of attention outputs for a set of queries \(Q\) with the keys and values packed into matrices \(K\) and \(V\) is computed by  
\[\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{QK^{T}}{\sqrt{d_{k}}} \right)V. \tag{9}\]  
To efficiently combine information from different representation subspaces at different positions, they proposed multi-head attention mechanism using \(Q\), \(K\), and \(V\) as inputs. The output of the \(n\)-th attention head is expressed as [24]  
\[\text{Attention}_{n}(Q,K,V)=\text{softmax}\left(\frac{\left(W_{n}^{q}Q^{T} \right)^{T}\left(W_{n}^{k}K^{T}\right)}{\sqrt{d_{k}}}\right)\left(W_{n}^{v}V^{ T}\right)^{T}, \tag{10}\]  
where \(W_{n}^{q}\), \(W_{n}^{k}\), and \(W_{n}^{v}\) denote the linear projection parameters for \(Q\), \(K\), and \(V\), respectively.