## 3 The Architecture of the NMT System
### B.4. B.4. B.5.
\begin{table}
\begin{tabular}{c c} \hline \hline
**Vocabulary** & **Number of Words in the Vocabulary** \\ \hline Size of English vocabulary before BPE & 165,233 \\ Size of English vocabulary after BPE & 45,231 \\ Size of Punjabi vocabulary before BPE & 189,231 \\ Size of Punjabi vocabulary after BPE & 56,123 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: The Size of Vocabulary with the use of BPE. \begin{table}
\begin{tabular}{c c} \hline \hline
**NMT Model** & **Configuration** \\ \hline Baseline model & BiLSTM as encoder \\ (NMT model 1) & LSTM as decoder \\ BPE to reduce vocabulary size & BiLSTM as encoder \\ Baseline model + word embedding & LSTM as decoder \\ (NMT model 2) & BPE to reduce vocabulary size \\  & WE to handle OOV words \\  & BiLSTM as encoder \\ Baseline model + MWE corpus & LSTM as decoder \\ (NMT model 3) & BPE to reduce vocabulary size \\  & MWEs are used in the training set \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Configuration of different NMT models. Figure 3: NMT model 4 architecture. ### Evaluation of the NMT System

The performance of the MT system can be evaluated by using two methods: human assessment and automated assessment. #### 5.2.1 Human Assessment of MT System

Human assessment was undertaken by using linguistic experts. The linguistic expert knows both the source and target language. There are three parameters to assess the prediction of the MT system: adequacy, fluency, and overall rating. Both parameters are scored on a scale of 1 to 5, with 1 indicating mediocre performance and 5 indicating excellent performance. Figure 4 shows the scale of the adequacy, fluency, and overall rating of translation. #### 5.2.2 Automated Assessment of MT System

When opposed to the human assessment of MT systems, the automated assessment of MT systems is more cost-effective. There are multiple tools available to automate the evaluation of MT systems such as BLEU, NIST, TER, METEOR, etc. In this research, two tools were used: BLEU and TER. The most often used metric for the automated assessment of the MT method is the Bilingual Measurement Understudy (BLEU) [39]. The BLEU score was computed using the mean of the precision for the unigram, bigram, trigram, and 4-g, and a length penalty was also applied if the MT output sentence had a shorter length than the best matching reference translation. \[BLEU=BP\cdot exp\sum_{m=1}^{N}\frac{1}{n}logp_{n} \tag{3}\]

The translation error rate (TER) is an MT error metric that computes the amount of post-editing used to convert the prediction obtained from the model to the given reference translation. \[TER=\frac{number\ of\ edits}{average\ number\ of\ reference\ words} \tag{4}\]

The test set data was split into three sets based on the number of words in the source sentence to validate the NMT models. The small test set contained all sentences having a maximum token of five. The medium test set contained all sentences with tokens between six and fifteen. The large test set contained all sentences with tokens more than or equal to sixteen. ### Human Evaluation Score of NMT Models

One linguistic expert was hired to evaluate all four NMT models. The individual score of each sentence was averaged to obtain the final score of the parameter. The human evaluation score of different models is shown in Tables 9-12. Figure 4: Human assessment scale of adequacy, fluency and overall rating. ### Automated Evaluation Score of NMT Models

The BLEU and TER scores were used to evaluate all four NMT models. Table 13 shows the BLEU performance of different NMT models. The precision of the NMT model is proportional to the BLEU score. The TER scores of all four NMT models are shown in Table 14. The lower the TER%, the higher the accuracy of the NMT model. ### Analysis of NMT Models

In this subsection, many distinct NMT models are assessed depending on the numerous parameters that are available.