## Appendix A Deep Learning Background
### Attention and Transformer  
The attention mechanism [42] was introduced to enhance encoder-decoder models for machine translation by dynamically weighting input elements instead of treating them equally. Traditional models generate a latent representation from an input sequence and use it to produce an output sequence. However, they lack a mechanism to prioritize relevant parts of the input.  
Attention introduces a _context vector_, computed at each decoding step to focus on the most relevant input elements. This involves calculating alignment scores \(e_{t,i}\) using a function \(a(.)\), applying softmax normalization to obtain weights \(\alpha_{t,i}\), and computing \(c_{t}\) as a weighted sum of encoder hidden states:  
\[\begin{split} e_{t,i}&=a(s_{t-1},h_{i})\\ \alpha_{t,i}&=softmax(e_{t,i})\\ c_{t}&=\sum_{i=1}^{T}\alpha_{t,i}h_{i}\end{split} \tag{12}\]  
This mechanism has been generalized [43] for broader sequence-to-sequence tasks using _queries_, _keys_, and _values_. Here, queries correspond to previous decoder outputs, while keys and values derive from encoded inputs. The goal is to identify which keys are most relevant to a query and assign alignment scores accordingly. The context vector is computed as \[Attention(Q,K,V)=softmax(\frac{QK^{T}}{\sqrt{d_{k}}})V \tag{10}\]  
where \(d_{k}\) is the dimension of queries and keys. This formulation enables efficient computation across multiple queries, keys, and values simultaneously, significantly improving performance in tasks like machine translation and beyond.  
Based on the attention mechanism, the _Transformer_ architecture was introduced [43], extending it significantly to handle long-range dependencies and process sequences in a parallelizable manner, making it much faster and more efficient than previous models. The Transformer uses a stack of encoders and decoders, each composed of self-attention layers and FFNN, to transform the input sequence into the output sequence without relying on any recurrent connections.  
Indeed, in the Transformer, both the _encoder_ and _decoder_ are composed of a stack of identical layers (typically 6). Each layer in the encoder has two primary sub-layers:  
* A _multi-head self-attention mechanism_, which allows the model to focus on different parts of the input sequence simultaneously and can be expressed as: \[\begin{split}& MultiHead(Q,K,V)=Concat(head_{1},\dots,head_{h})W ^{O}\\ & head_{i}=Attention(QW^{O}_{i},KW^{K}_{i},VW^{V}_{i})\end{split}\] (11) Here, \(W^{O}_{i},W^{K}_{i},W^{V}_{i}\) are learned projection matrices, and \(W^{O}\) is the final output projection matrix.
* A _position-wise fully connected FFNN_ that applies non-linear transformations independently to each position in the sequence, thus taking care of the position of tokens in the input sequence. The positional encoding for position _pos_ and dimension \(i\) is given by: \[\begin{split}&\text{PE}(pos,2i)=\text{sin}\bigg{(}\frac{pos}{10000 ^{2i/d_{\text{model}}}}\bigg{)}\\ &\text{PE}(pos,2i+1)=\text{cos}\bigg{(}\frac{pos}{10000^{2i/d_{ \text{model}}}}\bigg{)}\end{split}\] (12) This sinusoidal function enables the model to learn position-sensitive patterns and can generalize to sequences longer than those encountered during training.  
The decoder, in addition to these two sub-layers, also contains an additional sub-layer that performs multi-head attention over the encoder's output. This third layer is essential as it enables the decoder to attend to the relevant portions of the input sequence.  
Both the encoder and decoder use _residual connections_ around each of the sub-layers, followed by layer normalization to ensure stable learning. Each sub-layer can be expressed as \(\text{LayerNorm}(x+\text{Sublayer}(x))\) where \(x\) is the input to the sub-layer, and _Sublayer_(\(x\)) represents the function applied by the sub-layer (either self-attention or feedforward).