# Enhancement of English-Bengali Machine Translation Leveraging Back-Translation
## 2 Related Work
### Neural Machine Translation (NMT) Architecture
#### 2.1.3 Transformer Based Model  
To address the aforementioned problems, the Transformer [8] is introduced, which receives the entire input sequence in only one step and uses self-attention to leverage context information for encoding the input sequence. For the details of the Transformer, we recommend referring to the original paper by [PERSON] et al. [8]. Figure 2 shows the architecture of the Transformer. In the encoder phase, given the embedded input sequence (denoted as matrix \(X=[x_{1},\ldots,x_{l}]\)), it is first added by a positional encoding matrix, then goes through a stack of \(N\) encoder blocks (\(N=6\) in practice). In each encoder block, it is further encoded by a multi-head self-attention layer, which leverages context information. The production of head \(n\) (\(n=1,\ldots,8\) in practice) is matrix \(H_{\text{tr}}\), as is shown by Equations (6)-(11).  
\[q_{i} =W_{Q}x_{i} \tag{6}\] \[k_{i} =W_{K}x_{i}\] (7) \[v_{i} =W_{V}x_{i}\] (8) \[e_{ij} =\frac{\exp(q_{i}\cdot k_{j}/\sqrt{d_{k}})}{\sum_{j=1}^{l}\exp(q_{ i}\cdot k_{j}/\sqrt{d_{k}})}\] (9) \[h_{i} =\sum_{j=1}^{l}e_{ij}v_{i}\] (10) \[H_{n} =[h_{1},\ldots,h_{1}] \tag{11}\]  
where \(W_{Q}\), \(W_{K}\), and \(W_{V}\) are linear transformations that transfer embedded input \(x_{i}\) into 3 vectors that play as query, key, and value, respectively in constructing context-considered representations, \(d_{k}\) is the dimension of \(q_{i}\) and \(k_{j}\), \(i,j=1,\ldots,l\). Then the output of this layer \(O\) has the same size as the input, as shown in Equation (12).  
\[O=W_{O}\texttt{Concatenate}(H_{1},\ldots,H_{8}) \tag{12}\]  
Then it is passed to a Feed Forward network (2 FC layers in practice) to obtain better representation. After each step, there is a residual connection and a layer normalization operation to keep the value of the output in a reasonable range. The output of the encoder is the context matrix \(C=[c_{1},\ldots,c_{l}]\), which is similar to the hidden state matrix in RNN. The decoder of the Transformer consists of \(N=6\) blocks. It uses teacher forcing in the training stage; that is, the embedded input of first decoder block is right shifted ground truth \(\hat{Y}=[zero\_vec,\hat{y}_{1},\ldots,\hat{y}_{M}]\) where column \(i\) is similar to the input of decoder in step \(i\) for RNN. After positional encoding, it is passed to a masked multi-head self-attention layer, where \(\exp(q_{i}\cdot k_{j}/\sqrt{d_{k}})\) is masked (set to 0) for all \(j>i\) to prevent the contribution from future information when constructing a representation of \(\hat{Y}\). Another attention layer in the decoder block is similar to that in the encoder, with some differences are shown in Equations (13)-(15).  
\[q_{i} =W_{Q}\hat{Y}_{i}, \tag{13}\] \[k_{j} =W_{K}c_{j},\] (14) \[v_{j} =W_{V}c_{j}, \tag{15}\]  
where \(c_{j}\) is from \(C\), the output of encoder, \(i=1,\ldots,N+1\), \(j=1,\ldots,l\). The output of the decoder in training (after (softmax)) is matrix \(Y=[y_{1},\ldots,y_{M}]\), where \(y_{i}\) is the vector of the probability distribution of candidate tokens for position \(i\) in the output sequence. The Transformer can be trained at high speed through parallel computation and has high capacity. It is the basis of many large language models (LLM), including chat generative pre-trained transformer (ChatGPT (ChatGPT [[https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt)]([https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt)) accessed on 1 August 2024)). Also, as stated earlier that BUT CSE NLP Group from Bangladesh, who are consistently working on optimizing the NLP tasks in Bengali, they also use Transformer as the basis of their study [14; 15; 16; 17]. We further observe that prominent machine translation models [63; 64; 65] are built on top of the base transformer [8].