Question: What is the essence of their approach?
Evidence: Inspired by the multi-source NMT as additional parallel data in several languages are expected to benefit single translations, we aim to develop a NMT-based approach toward an universal framework to perform multilingual translation. Our solution features two treatments: 1) Coding the words in different languages as different words in the language-mixed vocabularies and 2) Forcing the NMT to translating a representation of source sentences into the sentences in a desired target language.
Language-specific Coding. When the encoder of a NMT system considers words across languages as different words, with a well-chosen architecture, it is expected to be able to learn a good representation of the source words in an embedding space in which words carrying similar meaning would have a closer distance to each others than those are semantically different. This should hold true when the words have the same or similar surface form, such as (@de@Obama; @en@Obama) or (@de@Projektion; @en@projection). This should also hold true when the words have the same or similar meaning across languages, such as (@en@car; @en@automobile) or (@de@Flussufer; @en@bank). Our encoder then acts similarly to the one of multi-source approach BIBREF10 , collecting additional information from other sources for better translations, but with a much simpler embedding function. Unlike them, we need only one encoder, so we could reduce the number of parameters to learn. Furthermore, we neither need to change the network architecture nor depend on which recurrent unit (GRU, LSTM or simple RNN) is currently using in the encoder.
We could apply the same trick to the target sentences and thus enable many-to-many translation capability of our NMT system. Similar to the multi-target translation BIBREF8 , we exploit further the correlation in semantics of those target sentences across different languages. The main difference between our approach and the work of BIBREF8 is that we need only one decoder for all target languages. Given one encoder for multiple source languages and one decoder for multiple target languages, it is trivial to incorporate the attention mechanism as in the case of a regular NMT for single language translation. In training, the attention layers were directed to learn relevant alignments between words in specific language pair and forward the produced context vector to the decoder. Now we rely totally on the network to learn good alignments between source and target sides. In fact, giving more information, our system are able to form nice alignments.
In comparison to other research that could perform complete multi-task learning, e.g. the work from BIBREF9 or the approach proposed by BIBREF11 , our method is able to accommodate the attention layers seemlessly and easily. It also draws a clear distinction from those works in term of the complexity of the whole network: considerably less parameters to learn, thus reduces overfitting, with a conventional attention mechanism and a standard training procedure.
Target Forcing. While language-specific coding allows us to implement a multilingual attention-based NMT, there are two issues we have to consider before training the network. The first is that the number of rare words would increase in proportion with the number of languages involved. This might be solved by applying a rare word treatment method with appropriate awareness of the vocabularies' size. The second one is more problematic: Ambiguity level in the translation process definitely increases due to the additional introduction of words having the same or similar meaning across languages at both source and target sides. We deal with the problem by explicitly forcing the attention and translation to the direction that we prefer, expecting the information would limit the ambiguity to the scope of one language instead of all target languages. We realize this idea by adding at the beginning and at the end of every source sentences a special symbol indicating the language they would be translated into. For example, in a multilingual NMT, when a source sentence is German and the target language is English, the original sentence (already language-specific coded) is:
@de@darum @de@geht @de@es @de@in @de@meinem @de@Vortrag
Now when we force it to be translated into English, the target-forced sentence becomes:
<E> @de@darum @de@geht @de@es @de@in @de@meinem @de@Vortrag <E>
Due to the nature of recurrent units used in the encoder and decoder, in training, those starting symbols encourage the network learning the translation of following target words in a particular language pair. In testing time, information of the target language we provided help to limit the translated candidates, hence forming the translation in the desired language.
Figure FIGREF8 illustrates the essence of our approach. With two steps in the preprocessing phase, namely language-specific coding and target forcing, we are able to employ multilingual attention-based NMT without any special treatment in training such a standard architecture. Our encoder and attention-enable decoder can be seen as a shared encoder and decoder across languages, or an universal encoder and decoder. The flexibitily of our approach allow us to integrate any language into source or target side. As we will see in Section SECREF4 , it has proven to be extremely helpful not only in low-resourced scenarios but also in translation of well-resourced language pairs as it provides a novel way to make use of large monolingual corpora in NMT.
Figure 1 illustrates the essence of our approach. With two steps in the preprocessing phase, namely language-specific coding and target forcing, we are able to employ multilingual attention-based NMT without any special treatment in training such a standard architecture. Our encoder and attention-enable decoder can be seen as a shared encoder and decoder across languages, or an universal encoder and decoder. The flexibitily of our approach allow us to integrate any language into source or target side.
Answer: Employ multilingual attention-based NMT without any special treatment in training such a standard architecture with language-specific coding and target forcing. And it can integrate any language into the source or target side because of the flexibity.