## 3 The Architecture of the NMT System
### B.4. B.4. B.5. The NMT system has a limited vocabulary issue. To overcome this problem, researchers have used character embeddings and the subword algorithm. The subword is in between the word and the character. There are different subword algorithms: byte pair encoding (BPE), WordPiece, and SentencePiece. From these three, the BPE was selected to build a subword dictionary. BPE is a simple technique for compressing data. This substitutes a single unused byte for the most common bytes in a sequence. The algorithm to develop a subword using BPE is given below (see Algorithm 1). ``` Input: large training corpus, vocabulary size Output: training corpus containing subword units
1. Create a large training repository. 2. Define the vocabulary size of the subword
3. Divide the word into a character series and add the suffix \"\(<\)/w\(>\)\" to the end of the word with the word frequency. For example, if \"fast\" is 4 in frequency, then it is recreated as \"f a s t \(<\)/w\(>\)\"-4. 4. Create a new subword based on the occurrence's high frequency. 5. Repeat step 4 until you achieve the size of the subword vocabulary as defined in step 2, or the next highest frequency pair is 1. Figure 2: Bidirectional LSTM model. The independent encodings for the Punjabi and English corpus with 35 K words were learned with the use of the BPE technique. After learning, it was applied to different sets of Punjabi and English corpus. ### Word Embedding

Word embedding is a technique for learning various characteristics in which words that have the same meaning are given the same representation in a vector space that is continuous. Word embedding may be learned in a variety of different ways. Word2 vec, Glove, and fastText are the three approaches that are used the most often to learn how to embed. Facebook's Artificial Intelligence Research (FAIR) team developed a neural network-based framework called fastText to categorize text based on word representations [27]. It is written in C++ and supports multiprocessing during training. The fastest supports are the CBOW and Skip Gram models. Facebook also provides a pre-trained word vector for various languages that are trained using fastText [28]. Punjabi and English pre-trained word vectors were downloaded from the fastText website and used in this research to overcome the OOV problem of NMT. ### Multi-Word Expressions

MWEs play an important role in the NLP and MT. MWEs consist of two or more words but are treated as a single word [29]. Each word of the MWE has a specific meaning and it is different from the collective meaning of MWEs. Different types of MWEs exist in the Punjabi language. #### 3.3.1 Replicated Words

The Punjabi language has replicated words that can be treated as MWEs. The replicated word may contain a particle or hyphen sign in between. Table 2 displays the replicated word in Punjabi. #### 3.3.2 Waala Morpheme Construct

The term \"waala\" acts as a morpheme construct to create MWEs in Punjabi. \"Waala\" has many forms such as \"waalaa\", \"waalii\", \"waale\", or \"waalaen\". Table 3 shows the waala morpheme construct in Punjabi. #### 3.3.3 Combination of Word with Synonym, Antonym, Hyponym, or Number

MWEs are also constructed in Punjabi by combining a word with its synonym, antonym, hyponym, or number. Table 4 shows the combination of Punjabi words with their synonym, antonym, hyponym, or number. \begin{table}
\begin{tabular}{c c} \hline \hline
** Punjabi Replicated Word** & **Translation in English** \\ \hline \hline
**Punjabi** (“Roz róz”) & Every day \\
**Punjabi** (“Holi höli”) & Quite slowly \\
**Punjabi** (“Pani hï pain”) & Water all over \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Replicated words in Punjabi. \begin{table}
\begin{tabular}{c c} \hline \hline
**Waala Morpheme Construct** & **Translation in English** \\ \hline \hline
**Punjabi** (“Kam wäli”) & Maid \\
**Punjabi** (“Dudha väla”) & Milkman \\
**Punjabi** (“Pani väli balatí”) & Water Bucket \\ \hline \hline \end{tabular}
\end{table} 
Table 3: The waala morpheme constructed in Punjabi.