## 3 Background  
### Transformer  
Transformer [37] currently represents the state-of-the-art in MT research. It is an encoder-decoder architecture based on stacked self-attention and position-wise fully connected layers in both its encoder and decoder components. Figure 1 shows the transformer architecture. The left side of Figure 1 depicts the Nth layer of the encoder, while the right side shows the Nth layer of the decoder. Each encoder layer contains two sub-layers: a multi-head self-attention mechanism layer, and a position-wise fully connected feedforward network layer. Residual connections [38] are employed around each sub-layer, followed by layer normalization [39]. This configuration results in the output of each sub-layer being expressed as \(LayerNorm(x+Sublayer(x))\), where \(Sublayer(x)\) denotes the function implemented by the specific sub-layer. Both the sub-layers and the embedding layers are designed to produce outputs of dimension \(dmodel\) to accommodate the residual connections. The decoder structure is similar to the encoder but includes an additional third sub-layer. This sub-layer applies multihead attention over the output of the encoder stack, known as encoder-decoder attention. Multihead attention enables the model to process information from different representation subspaces across different positions. The multihead attention sub-layer masks the received output from the encoder and offsets the embeddings by one position, ensuring that predictions for position \(i\) can only depend on the known outputs at positions smaller than \(i\). As in the encoder, a residual connection which carries gradients from the input layers to output layers is added around each sub-layer within the decoder layer to prevent vanishing gradients, followed by layer normalisation. Furthermore, ref. [37] proposed scaled dot-product attention, which is significantly faster and more memory-efficient than additive attention when implemented via optimized matrix multiplication. By scaling the computed dot products, dot-product attention maintains its advantage over additive attention even as the dot product values become larger. The attention function is simultaneously computed on a set of queries, keys, and values packed into matrices Q, K, and V, respectively. The matrix of outputs that is fed into the decoder is computed as shown below in Equation (1), where \(d_{k}\) denotes the dimension of  
Figure 1: Transformer model architecture as in [37]. the query and key vectors. Scaling is performed by dividing the dot product by \(\sqrt{d_{k}}\). This process is illustrated in Figure 2.  
\[Attention(Q,K,V)=softmax(\frac{QK^{T}}{\sqrt{d_{k}}})V \tag{1}\]  
Multi-head attention is a mechanism enabling neural network models to simultaneously process information from distinct representation subspaces across various positions within the input. This process begins by linearly projecting the query, key, and value (Q, K, and V) h times using unique learned linear transformations to specific dimensions: \(d_{k}\) for Q and K, and \(d_{v}\) for V. The core attention function is then applied in parallel to each of these projected Q, K, V sets, yielding \(d_{v}\)-dimensional output values for each parallel head. These outputs are subsequently concatenated and subjected to a final linear projection to produce the ultimate values. The attention mechanism is applied independently for each of the h attention heads, as shown in Equation (2). The resulting outputs from each head are then concatenated and passed through a final linear transformation, as shown in Equation (3).