# Enhancement of English-Bengali Machine Translation Leveraging Back-Translation
## 3 Methodology
### Back-Translation with OpenNMT  
The framework of back-translation we conduct is as follows:  
* We train an English-Bengali model and a Bengali-English model on the original parallel corpus. * We introduce a Bengali monolingual corpus and use the trained Bn-En model to translate it into English; thus, we have another English-Bengali corpus (pseudo-corpus). * We use the two corpora to train a new English-Bengali model. Notably, when preparing the pseudo-parallel data by back translation, we use two decoding methods: top-\(k\) random sampling and random sampling with temperature \(T\). To demonstrate them, we observe the last steps of predicting an output word. The direct output from the decoder is a vector. The official documentation of OpenNMT-py (OpenNMT-py Official Documentations [[https://opennmt.net/OpenNMT-py/options/translate.html](https://opennmt.net/OpenNMT-py/options/translate.html)]([https://opennmt.net/OpenNMT-py/options/translate.html](https://opennmt.net/OpenNMT-py/options/translate.html)) accessed on 1 August 2024) [67, 68, 69] use the term \"logits\" for it. The logits vector is then processed by softmax and becomes a probability distribution, according to which the model samples the output word. Therefore, top-\(k\) sampling is easy to understand. Supposing that \(k=3\), it simply picks up the 3 words with the highest probabilities and sets the probabilities of the other words to \(-\infty\) (\(-10000\) in practice). Another strategy is to introduce temperature \(T\), a positive float variable, to modify the probability distribution. That is, we divide the logits by \(T\) before feeding them into softmax. In this strategy, all words in the vocabulary of the model are regarded as candidates. For better understanding, we consider logits with 2 elements: {\(T\):0, 'you':\(x\)}, notice that \(x\) can be any value, positive or negative. After softmax with temperature \(T\) (can be any positive value), \(P(\textit{you'})\) becomes Equation (16):  
\[P(\textit{you'})=\frac{\exp(x/T)}{1+\exp\left(x/T\right)}. \tag{16}\]  
Figure 3 shows the result of \(P(\textit{you'})\) at different temperatures. At a low temperature (\(T=0.1\)), if the probability of 'you' (\(x\)) is lower than that of \(T\)' (zero), then the final probability is \(P(\textit{you'})=0\), and thus \(P(\textit{'}T)=1\) and vice versa. Therefore, at a low temperature, the probability distribution becomes sharp, and the sampling algorithm concentrates more on the top words. Meanwhile, at a high temperature (\(T=50\)), the probability distribution becomes flat, and all words have similar probabilities for sampling, despite the raw output (\(x\)) from the decoder. Using these two decoding methods, with different values of \(k\) or \(T\), we investigate the performance of models trained on data with different levels of noise. First, we compare the performance of LSTM and Transformer networks in both translation directions. Then we choose the better architecture and back-translate (Bengali-English) the Bengali part of the training data to create a pseudo-parallel corpus and add it to the original training set. We keep the validation set invariant for consistency. At this stage, we generate different translations by setting the random sampling parameter, \(k=1\) (the argmax), 5, and 10. We  
Figure 3: The relation between \(x\) and \(P(\textit{you'})\) at different temperatures \(T\). also generate translations with \(T=0.5,1\). Note that if we set \(T=0\), then the top 1 word with the maximum probability will have \(P(word)=1\) and will be sampled. Therefore, this is equivalent to top-\(k=1\) random sampling. In addition, setting \(k=+\infty\) in top-\(k\) allows the model to sample from all tokens in the vocabulary, while setting temperature \(T=1\) leads to the unmodified probability distribution for all tokens in the vocabulary. Both of the cases are equivalent to the setup where neither top-k nor temperature sampling is used. It is denoted as \"no strategy\" in this article.