Context Based Machine Translation With Recurrent Neural Network For English-Amharic Translation 

1 INTRODUCTION . Context based machine translation ( CBMT ) is a phrase-based machine translation ( PBMT ) approach proposed by Miller et al . ( 2006 ) . Unlike most PBMT approaches that rely on statistical occurrence of the phrases , CBMT works on the contextual occurrence of the phrases . CBMT uses bilingual dictionary as its main translator and produces phrases to be flooded into a large target corpus . The CBMT approach addresses the problem of parallel corpus scarcity between language pairs . The parallel corpus set for English-Amharic language pair , for instance , composes of the Bible , the Ethiopian constitution and international documents . These sources use words specific to their domain and overlook phrases and words used by novels , news and similar literary documents . The CBMT uses synonyms of words in place of rare words and rely on large target corpus and a bilingual dictionary to help with data scarcity ( Miller et al. , 2006 ) . It is not dependent on large parallel corpus like most PBMT such as the statistical machine translation ( SMT ) ( Brown et al. , 1990 ) and the example-based machine translation EBMT ( Gangadharaiah , 2011 ) . The CBMT , however , fails in fluently translating texts compared to the neural machine translation ( NMT ) . The NMT learns the pattern of humans ’ translation using human translated parallel corpus . Its translations are more fluent and accurate than all the rest so far when evaluated individually ( Popovic , 2017 ) . However , NMT struggles to translate properly rare words and words not commonly used ( Wu et al. , 2016 ) . In addition , NMT requires large parallel corpus for training . The aim of this research is to build a system by combining the CBMT with the NMT for English to Amharic translation . The combination of PBMT and NMT is the future and most promising than the individual approaches themselves ( Popovic , 2017 ) . CBMT ’ s ability to address rare words and the NMT ’ s ability to produce fluent translation along with their context awareness makes them complementary couple . The combination is done by providing the NMT with two inputs , one from the source language and the other from the output of the CBMT to produces the final target sentence . In this paper , we show that this approach utilizes the strength of each method to achieve a significant translation performance improvement over simple NMT . The improvement is mostly dependent on the performance of the CBMT and mostly on the bilingual dictionary of the CBMT . 2 RELATED WORKS . PBMT approaches are mostly used to translate English to Amharic as in the case of Gasser ( 2012 ) , Tadesse & Mekuria ( 2000 ) , Teshome ( 2000 ) , Besacier et al . ( 2000 ) , Zewgneh ( 2017 ) and Taye et al . ( 2015 ) . Below we summarize the researches with most significance to ours . The SMT approach takes a parallel corpus as an input and it selects the most frequent target phrase based on statistical analysis for each searched source phrase ( Brown et al. , 1990 ) . The SMT approach applied to the English-Amharic pair has produced 18.74 % BLEU score ( Tadesse & Mekuria , 2000 ) . The SMT has good accuracy in translating all the words in a sentence but it is not fluent ( Oladosu et al. , 2016 ) . Hybrid of SMT and rule based machine translation ( RBMT ) translates and orders the source text based on the grammar rules of the target language and sends it to the SMT for final translation ( Yulianti et al. , 2011 ) ( Labaka et al. , 2014 ) . The hybrid approach for English-Amharic pair has achieved a 15 % improvement over SMT on simple sentence and 20 % improvement for complex sentence ( Zewgneh , 2017 ) . Hybrid of RBMT and SMT gets fluency from RBMT and accuracy from SMT but for longer sentences , the reordering fails ( Oladosu et al. , 2016 ) . The CBMT approach has been implemented for the language pair Spanish-English . In CBMT , the source phrases are translated using bilingual dictionary and flooded to target corpus . It has achieved 64.62 % BLEU score for the researchers ’ dataset ( Miller et al. , 2006 ) . The CBMT outperforms SMT in accuracy and fluency but translation of phrases with words not in the bilingual dictionary is weak ( Miller et al. , 2006 ) . The NMT has been researched by different groups and here the research by Googles ’ researchers on the language pair English-French is presented . The NMT model is trained using parallel corpus . The source sentence is encoded as a vector and then decoded with the help of an attention model . Googles ’ NMT model has achieved 38.95 % BLEU score ( Wu et al. , 2016 ) . The NMT has accuracy and fluency but it fails to translate the whole sentence and also fails to perform well with rare words ( Wu et al. , 2016 ) . To solve this using sub-word units has been suggested ( Sennrich et al. , 2016 ) but Amharic has unique treats like ” Tebko-lalto ” , one word with two different meanings , which can only be addressed by using context . The NMT has been modified to translate low resourced languages . One approach uses universal lexical representation ( ULR ) were a word is represented using universal word embeddings . This benefits low resource languages which have semantic similarity with high resourced languages ( Gu et al. , 2018 ) . This achieved 5 % BLEU score improvement over normal NMT . However , most southern Semitic languages like Amharic do not have a strong semantic relative with large resource . NMT has also been modified to work with monolingual corpus instead of parallel corpus using cross-lingual word embedding ( Artetxe et al. , 2017 ) . Such an approach achieved a 15.56 % BLEU score which was less that the semi-supervised and supervised which achieved 21.81 % BLEU score and 20.48 % BLEU score respectively . Combination of NMT and PBMT which takes the output of SMT ( a PBMT ) and the source sentence to train the NMT model has been used for the language pair English-German . It has achieved 2 BLEU points over basic NMT and PBMT ( Niehues et al. , 2016 ) . Combination of NMT and PBMT which takes three inputs ; output of basic NMT , output of SMT and output of Hierarchical PBMT ( HPBMT ) of SMT has been implemented for English-Chinese language pair . It achieved 6 BLEU points over basic NMT and 5.3 BLEU points over HPBMT ( Zhang et al. , 2017 ) . The combination of PBMT and NMT performs better ( Popovic , 2017 ) in terms of accuracy and fluency but it is dependent on the performance of the chosen PBMT approach . 3 METHODOLOGY . In this research , we have selected CBMT and NMT to form a combinational system . This approach addresses the limitation of context unawareness of some PBMT approaches like SMT and the need for large parallel corpus of simple NMT . In our approach , the source sentence in English and the translation output of the CBMT in Amharic has been fed to the NMT ’ s encoder-decoder model as shown in Figure 1 . The NMT model then produces the final Amharic translation . The combination of the CBMT and the NMT follows the mixed approach proposed by Niehues et al . ( 2016 ) . Their mixed approach feeds the NMT with the source sentence and the output of the PBMT . The research by Zhang et al . ( 2017 ) also supports this way of combining different systems . 3.1 CBMT SYSTEM . The CBMT outperforms RBMT , SMT and EBMT when it comes to languages with less parallel corpora ( Miller et al. , 2006 ) . It uses a bilingual dictionary , a large target corpus and smaller source corpus , which is optional . In the context based machine translation , there are different components working together to produce the translation . Figure 2 shows the flow of data in the different components of the CBMT . The source sentence is converted into N-gram phrases and then it is translated using bilingual dictionary . CBMT ’ s performance is mostly dependent on the efficiency of the dictionary . We have manually built a phrase based dictionary aided by Google translate . A synonym finder helps the dictionary ’ s search using WordNet ( Soergel , 1998 ) . WordNet is a library with large lexical database of English words . It provides synonyms of the English words whose Amharic translations are not in dictionary . In this paper , a maximum of four N-grams has been used . Most phrases of English that are translated to a single word in Amharic have a length of four or less words . For example the English phrase ” everyone who calls on the name of the lord will be saved ” has the translations in Output 1 using our dictionary . Output 1 : The translated output of the N-grams These translations have been combined into sentences in the same order as the source sentence . Then each sentence is converted into N-grams of variable length . The maximum flooded N-grams length is len ( Ngram ) 2 + 1 if len ( Ngram ) ≥4 else it is equal to len ( Ngram ) . This provides a good variable range to capture neighboring words in a single N-gram . Output 2 shows sentences formed using the translations in Output 1 . The N-grams to be flooded are formed by sliding one word from left to right through the combined sentence . Output 2 : The translated output combined into sentence Output 3 shows the N-grams for the translated sentences shown in Output 2 . Output 3 : The N-grams for the translated sentences The flooder is then responsible for searching the translated phrases in the target corpus and finding the longest N-gram match . For each phrase to be flooded , it selects a phrase in the target corpus with the most translated words and least in-between words amongst the words matched . The flooder produces the result in Output 4 with the Book of Romans as the target corpus to be flooded . Output 4 : Final output of flooder for single flooded file The N-gram connector combines the flooded text to find the longest overlap of the translated target text . The overlapping system favors those with the least number of not searched words found in between the searched N-grams when calculating the overlap . Output 5 shows the final outcome of the N-gram connector . The system selects the maximum or longest overlapping phrases from the combiner and merges them to form the final target sentence . So finally , the translation for the example English phrase ” everyone who calls on the name of the lord will be saved ” is . 3.2 NMT SYSTEM . In this paper , we have used RNN ( recurrent neural networks ) for the NMT . In RNN , the output is fed back to the neuron to learn from both the fresh input and its previous output . This improves RNN ’ s performance because it learns from its errors while training . The neural cell used is the LSTM ( long short-term memory ) , introduced by Hochreiter & Schmidhuber ( 1997 ) . We have used LSTM cells for both the encoding and decoding of the sentences . For the decoding , a greedy algorithm has been used . The algorithm selects the first fit word that has the highest probability of occurrence . Probability refers to the probability of being translated and appearing next to the word before itself . The system has an attention layer between the encoder layer and the decoder layer . We have used the Luong attention model ( Luong et al. , 2015 ) . Equation 1 through Equation 4 shows Luong attention model ’ s computation . αts = exp ( score ( ht , hs ) ) S∑ s′=1 exp ( score ( ht , hs′ ) ) [ Attention Weights ] ( 1 ) Output 5 : Final Output of N-gram connector ct = ∑ s αtshs [ Context vector ] ( 2 ) at = f ( ct , ht ) = tanh ( Wc [ ct : ht ] ) [ Attention Vector ] ( 3 ) score ( ht , hs ) = h T t Whs [ Luong ’ s multiplicative style ] ( 4 ) The score function , calculated using Equation 4 , is used to compare the output of the decoder ( ht ) with the output of the encoder ( hs ) in order to find the attention weight calculated using Equation 1 . The Attention weights ( alphats ) are then used for the context vector ( ct ) calculated by Equation 2 . This context vector as well as the output of the decoder is then used to produce the final output of the decoder using Equation 3 .

Summary: This paper presents a machine translation system based on a combination of a neural machine translation system (NMT) and a context-based machine translation (CBMT). The method is evaluated on a small parallel corpus application of English-Amharic translation. The idea is that in the small corpus setting, the CBMT can leverage a manually built bilingual dictionary