## 2 The ANN Models--A Brief Review
### Transformer Networks  
The transformer, introduced by [PERSON], et al. [59] in their paper called \"Attention is All You Need\", revolutionized sequence modeling using an attention mechanism instead of the bidirectional LSTM enabling parallel processing of inputs, and handling longer dependencies in sequences. Unlike RNNs, which naturally encode the position of inputs in the sequence, transformers require positional encoding to identify the positions of the inputs in the sequence. The transformer model, initially designed for Natural Language Processing (NLP) tasks (see Figure 6), has been further developed for various applications, including image [60], and video [61] classifications.  
The attention mechanism was introduced by [62] and initially utilized bidirectional LSTM to establish similar weights. Eventually, various methods were developed to compute dynamic similarity weights between input elements (keys) with respect to specific outputs (queries). These methods include additive, multiplicative (dot product), scaled multiplicative, general, biased general, activated general, and similarity [63]. The scaled dot product approach scales the dot product of the query and the transposed key values (\(Q_{i}\)\(K^{T}_{i}\)) based on the input dimensions (\(d_{ki}\) in Equation (17)). These similarity weights are then multiplied by the input sequence (values) to emphasize specific elements within the sequence. To introduce nonlinearity, fully connected units (FC) are used in the first and last linear layers, as depicted in Figure 7. A multi-head scaled dot product attention mechanism with a mask layer for selectively masking (filter future values commonly) certain parts of the input is shown in Figure 7.  
Figure 6: A transformer architecture.  
Figure 7: A multi-head attention units’ architecture.  
Given a set of queries (\(Q\)), keys (\(K\)), and values (\(V\)), and the number of attention heads (\(l\)), the scaled dot product attention scores (_Attention_ (\(Q\), \(K\), \(V\))) for each head (\(i\)) are calculated as:  
\[Attention(Q,\ K,\ V)=SoftMax\bigg{(}\frac{Q_{i}K^{T}{}_{i}}{\sqrt{d_{ki}}}\bigg{)} \,V_{i} \tag{17}\]  
The output for each head:  
\[head_{i}=Attention\Big{(}Q\ W_{i}^{Q},\ K\ W_{i}^{K},\ V\ W_{i}^{V}\Big{)} \tag{18}\]  
Concatenate the outputs of all heads:  
\[MultiHead(Q,\ K,\ V)=Concat\ (head_{1},\ldots,head_{i})W^{O} \tag{19}\]  
where \(SoftMax\) is the SoftMax function, which normalizes the attention scores, \(\sqrt{d_{ki}}\) is the square root of the dimension of the key vectors, used for scaling to prevent gradients from becoming too small, and \(W_{i}^{Q},\ W_{i}^{K},\ W_{i}^{V},and\ W^{O}\) are the learnable weight matrix for queries (\(Q\)), keys (\(K\)), and values (\(V\)), and the output projection.  
The collection of attention units (represented by red blocks in Figure 7) integrates information from different perspectives of queries, keys, and values, like the use of multiple kernels in CNNs. This mechanism allows the model to focus on different parts of the input for different tasks or aspects, making it more flexible and capable of capturing various patterns in the data.