Context Based Machine Translation With Recurrent Neural Network For English-Amharic Translation 

1 INTRODUCTION . Context based machine translation ( CBMT ) is a phrase-based machine translation ( PBMT ) approach proposed by Miller et al . ( 2006 ) . Unlike most PBMT approaches that rely on statistical occurrence of the phrases , CBMT works on the contextual occurrence of the phrases . CBMT uses bilingual dictionary as its main translator and produces phrases to be flooded into a large target corpus . The CBMT approach addresses the problem of parallel corpus scarcity between language pairs . The parallel corpus set for English-Amharic language pair , for instance , composes of the Bible , the Ethiopian constitution and international documents . These sources use words specific to their domain and overlook phrases and words used by novels , news and similar literary documents . The CBMT uses synonyms of words in place of rare words and rely on large target corpus and a bilingual dictionary to help with data scarcity ( Miller et al. , 2006 ) . It is not dependent on large parallel corpus like most PBMT such as the statistical machine translation ( SMT ) ( Brown et al. , 1990 ) and the example-based machine translation EBMT ( Gangadharaiah , 2011 ) . The CBMT , however , fails in fluently translating texts compared to the neural machine translation ( NMT ) . The NMT learns the pattern of humans ’ translation using human translated parallel corpus . Its translations are more fluent and accurate than all the rest so far when evaluated individually ( Popovic , 2017 ) . However , NMT struggles to translate properly rare words and words not commonly used ( Wu et al. , 2016 ) . In addition , NMT requires large parallel corpus for training . The aim of this research is to build a system by combining the CBMT with the NMT for English to Amharic translation . The combination of PBMT and NMT is the future and most promising than the individual approaches themselves ( Popovic , 2017 ) . CBMT ’ s ability to address rare words and the NMT ’ s ability to produce fluent translation along with their context awareness makes them complementary couple . The combination is done by providing the NMT with two inputs , one from the source language and the other from the output of the CBMT to produces the final target sentence . In this paper , we show that this approach utilizes the strength of each method to achieve a significant translation performance improvement over simple NMT . The improvement is mostly dependent on the performance of the CBMT and mostly on the bilingual dictionary of the CBMT . 2 RELATED WORKS . PBMT approaches are mostly used to translate English to Amharic as in the case of Gasser ( 2012 ) , Tadesse & Mekuria ( 2000 ) , Teshome ( 2000 ) , Besacier et al . ( 2000 ) , Zewgneh ( 2017 ) and Taye et al . ( 2015 ) . Below we summarize the researches with most significance to ours . The SMT approach takes a parallel corpus as an input and it selects the most frequent target phrase based on statistical analysis for each searched source phrase ( Brown et al. , 1990 ) . The SMT approach applied to the English-Amharic pair has produced 18.74 % BLEU score ( Tadesse & Mekuria , 2000 ) . The SMT has good accuracy in translating all the words in a sentence but it is not fluent ( Oladosu et al. , 2016 ) . Hybrid of SMT and rule based machine translation ( RBMT ) translates and orders the source text based on the grammar rules of the target language and sends it to the SMT for final translation ( Yulianti et al. , 2011 ) ( Labaka et al. , 2014 ) . The hybrid approach for English-Amharic pair has achieved a 15 % improvement over SMT on simple sentence and 20 % improvement for complex sentence ( Zewgneh , 2017 ) . Hybrid of RBMT and SMT gets fluency from RBMT and accuracy from SMT but for longer sentences , the reordering fails ( Oladosu et al. , 2016 ) . The CBMT approach has been implemented for the language pair Spanish-English . In CBMT , the source phrases are translated using bilingual dictionary and flooded to target corpus . It has achieved 64.62 % BLEU score for the researchers ’ dataset ( Miller et al. , 2006 ) . The CBMT outperforms SMT in accuracy and fluency but translation of phrases with words not in the bilingual dictionary is weak ( Miller et al. , 2006 ) . The NMT has been researched by different groups and here the research by Googles ’ researchers on the language pair English-French is presented . The NMT model is trained using parallel corpus . The source sentence is encoded as a vector and then decoded with the help of an attention model . Googles ’ NMT model has achieved 38.95 % BLEU score ( Wu et al. , 2016 ) . The NMT has accuracy and fluency but it fails to translate the whole sentence and also fails to perform well with rare words ( Wu et al. , 2016 ) . To solve this using sub-word units has been suggested ( Sennrich et al. , 2016 ) but Amharic has unique treats like ” Tebko-lalto ” , one word with two different meanings , which can only be addressed by using context . The NMT has been modified to translate low resourced languages . One approach uses universal lexical representation ( ULR ) were a word is represented using universal word embeddings . This benefits low resource languages which have semantic similarity with high resourced languages ( Gu et al. , 2018 ) . This achieved 5 % BLEU score improvement over normal NMT . However , most southern Semitic languages like Amharic do not have a strong semantic relative with large resource . NMT has also been modified to work with monolingual corpus instead of parallel corpus using cross-lingual word embedding ( Artetxe et al. , 2017 ) . Such an approach achieved a 15.56 % BLEU score which was less that the semi-supervised and supervised which achieved 21.81 % BLEU score and 20.48 % BLEU score respectively . Combination of NMT and PBMT which takes the output of SMT ( a PBMT ) and the source sentence to train the NMT model has been used for the language pair English-German . It has achieved 2 BLEU points over basic NMT and PBMT ( Niehues et al. , 2016 ) . Combination of NMT and PBMT which takes three inputs ; output of basic NMT , output of SMT and output of Hierarchical PBMT ( HPBMT ) of SMT has been implemented for English-Chinese language pair . It achieved 6 BLEU points over basic NMT and 5.3 BLEU points over HPBMT ( Zhang et al. , 2017 ) . The combination of PBMT and NMT performs better ( Popovic , 2017 ) in terms of accuracy and fluency but it is dependent on the performance of the chosen PBMT approach . 3 METHODOLOGY . In this research , we have selected CBMT and NMT to form a combinational system . This approach addresses the limitation of context unawareness of some PBMT approaches like SMT and the need for large parallel corpus of simple NMT . In our approach , the source sentence in English and the translation output of the CBMT in Amharic has been fed to the NMT ’ s encoder-decoder model as shown in Figure 1 . The NMT model then produces the final Amharic translation . The combination of the CBMT and the NMT follows the mixed approach proposed by Niehues et al . ( 2016 ) . Their mixed approach feeds the NMT with the source sentence and the output of the PBMT . The research by Zhang et al . ( 2017 ) also supports this way of combining different systems . 3.1 CBMT SYSTEM . The CBMT outperforms RBMT , SMT and EBMT when it comes to languages with less parallel corpora ( Miller et al. , 2006 ) . It uses a bilingual dictionary , a large target corpus and smaller source corpus , which is optional . In the context based machine translation , there are different components working together to produce the translation . Figure 2 shows the flow of data in the different components of the CBMT . The source sentence is converted into N-gram phrases and then it is translated using bilingual dictionary . CBMT ’ s performance is mostly dependent on the efficiency of the dictionary . We have manually built a phrase based dictionary aided by Google translate . A synonym finder helps the dictionary ’ s search using WordNet ( Soergel , 1998 ) . WordNet is a library with large lexical database of English words . It provides synonyms of the English words whose Amharic translations are not in dictionary . In this paper , a maximum of four N-grams has been used . Most phrases of English that are translated to a single word in Amharic have a length of four or less words . For example the English phrase ” everyone who calls on the name of the lord will be saved ” has the translations in Output 1 using our dictionary . Output 1 : The translated output of the N-grams These translations have been combined into sentences in the same order as the source sentence . Then each sentence is converted into N-grams of variable length . The maximum flooded N-grams length is len ( Ngram ) 2 + 1 if len ( Ngram ) ≥4 else it is equal to len ( Ngram ) . This provides a good variable range to capture neighboring words in a single N-gram . Output 2 shows sentences formed using the translations in Output 1 . The N-grams to be flooded are formed by sliding one word from left to right through the combined sentence . Output 2 : The translated output combined into sentence Output 3 shows the N-grams for the translated sentences shown in Output 2 . Output 3 : The N-grams for the translated sentences The flooder is then responsible for searching the translated phrases in the target corpus and finding the longest N-gram match . For each phrase to be flooded , it selects a phrase in the target corpus with the most translated words and least in-between words amongst the words matched . The flooder produces the result in Output 4 with the Book of Romans as the target corpus to be flooded . Output 4 : Final output of flooder for single flooded file The N-gram connector combines the flooded text to find the longest overlap of the translated target text . The overlapping system favors those with the least number of not searched words found in between the searched N-grams when calculating the overlap . Output 5 shows the final outcome of the N-gram connector . The system selects the maximum or longest overlapping phrases from the combiner and merges them to form the final target sentence . So finally , the translation for the example English phrase ” everyone who calls on the name of the lord will be saved ” is . 3.2 NMT SYSTEM . In this paper , we have used RNN ( recurrent neural networks ) for the NMT . In RNN , the output is fed back to the neuron to learn from both the fresh input and its previous output . This improves RNN ’ s performance because it learns from its errors while training . The neural cell used is the LSTM ( long short-term memory ) , introduced by Hochreiter & Schmidhuber ( 1997 ) . We have used LSTM cells for both the encoding and decoding of the sentences . For the decoding , a greedy algorithm has been used . The algorithm selects the first fit word that has the highest probability of occurrence . Probability refers to the probability of being translated and appearing next to the word before itself . The system has an attention layer between the encoder layer and the decoder layer . We have used the Luong attention model ( Luong et al. , 2015 ) . Equation 1 through Equation 4 shows Luong attention model ’ s computation . αts = exp ( score ( ht , hs ) ) S∑ s′=1 exp ( score ( ht , hs′ ) ) [ Attention Weights ] ( 1 ) Output 5 : Final Output of N-gram connector ct = ∑ s αtshs [ Context vector ] ( 2 ) at = f ( ct , ht ) = tanh ( Wc [ ct : ht ] ) [ Attention Vector ] ( 3 ) score ( ht , hs ) = h T t Whs [ Luong ’ s multiplicative style ] ( 4 ) The score function , calculated using Equation 4 , is used to compare the output of the decoder ( ht ) with the output of the encoder ( hs ) in order to find the attention weight calculated using Equation 1 . The Attention weights ( alphats ) are then used for the context vector ( ct ) calculated by Equation 2 . This context vector as well as the output of the decoder is then used to produce the final output of the decoder using Equation 3 .

Summary: This paper aims to combine a traditional CBMT system with an NMT system. The core idea of the paper is to use the output of the CBMT system as a second source to a multi-source NMT system. The first source of the system is CBMT, the second source is the original source and the output is the translation in the target language. All the experiments are conducted for English-Amharic with a small amount of parallel data from the Bible. A lot of details are provided about the generation of the CBMT system using Google Translate and details are provided about the approach to create such a system.