# Fully Attentional Network for Low-Resource Academic Machine Translation and Post Editing
## 6 Discussion and Future Works  
In this study, pre-trained language models were used to improve the translation quality in low-source language pairs. Turkish-English language pair was chosen for the translation system. For the translation system, first of all, a parallel corpus of academic translations were created. While creating the parallel corpus, the summary sections of the theses prepared in Turkey were used. For the translation system, a new model consisting of an encoder-decoder network with transformer architecture, a pre-trained language model, and a FAN architecture is proposed. In the proposed model, the last layer of the basic transformer decoder network is combined with the last layer of the pre-trained language  
Figure 5: Perplexity score of the models created. \begin{table}
\begin{tabular}{c c} \hline \hline  & **Parameter Size** \\ \hline Conv & 96.841.408 \\ LSTM & 31.066.208 \\ Transformer (6 layers) & 73.033.074 \\ Transformer Big (12 layers) & 98.278.770 \\ Transformer Big + FFN & 136.303.045 \\
**Transformer Big + FAN** & **158.523.484** \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Created model trainable parameter size. Figure 6: TER, BLEU, and METEOR scores of the models created. model using FAN. In this study, unlike the known shallow fusion method, FAN architecture was used for the first time instead of FFN. In addition, Byte Pair Encoding was used to overcome the rare word problem during the training. After the model was trained, the best translation was tried to be selected among the alternative translations with the Beam Search algorithm. During the testing phase of the proposed model, TER, BLEU, and METEOR scores were calculated (Figure 5). The proposed model achieved (+2 BLEU, +4 METEOR) higher scores than the base transformer method. The BLEU score of 40 and above is known as high-quality translation [[https://cloud.google.com/translate/automl/docs/evaluate](https://cloud.google.com/translate/automl/docs/evaluate)]([https://cloud.google.com/translate/automl/docs/evaluate](https://cloud.google.com/translate/automl/docs/evaluate)) (Accessed on 3 November 2022). In addition, the proposed model also achieves higher scores than the Google Translate translation system. The main reason for this situation may be the academic sentence structure and the excess of academic terms in the test set. Using the proposed model, the translation was made using the zero-shot method on the WMT17 and WMT18 test sets. The model used current settings. In other words, extra parallel corpus, and monolingual data were not used in the proposed model while making these translations. The proposed model achieved a BLEU score of 20.12 and 20.56 on these test sets, respectively. In the latest studies, [47] achieved a 21.9 BLEU score for the WMT17 test set and [48] achieved a 23.1 BLEU score for the WMT18 test set. It is observed that the parallel corpus and monolingual data exclusive to the translation type were used in the mentioned studies. The proposed translation model reached competitive BLEU scores while the highest scores on these test sets were recorded in prior studies. Furthermore, instead of the SciBert model used in the translation systems, language models like BERT, DistilBert, and GPT were tried to translate WMT test sets. However, the results were much higher than expected. This may be due to the use of unlabelled data while training the pre-trained language models. It is estimated that WMT test sets were used while training the pre-trained language models. The model can be used as a helpful resource for researchers preparing articles in English. Although the model achieves sufficient scores, it has less success compared to translation systems that are widely used and continue to be developed (En-De, En-Fr, etc...). The main reason for these low scores is insufficient parallel corpus. Other language pairs currently have 50 M parallel corpus [9]. There is no corpus of this size among the Tr-En language pair yet.