Title: Deep Character-Level Neural Machine Translation By Learning Morphology
Authors: Shenjian Zhao
Zhihua Zhang
Venue: ICLR 2017
Keywords: Natural language processing
Deep learning

Abstract: Neural machine translation aims at building a single large neural network that can be trained to maximize translation performance. The encoder-decoder architecture with an attention mechanism achieves a translation performance comparable to the existing state-of-the-art phrase-based systems. However, the use of large vocabulary becomes the bottleneck in both training and improving the performance. In this paper, we propose a novel architecture which learns morphology by using two recurrent networks and a hierarchical decoder which translates at character level. This gives rise to a deep character-level model consisting of six recurrent networks. Such a deep model has two major advantages. It avoids the large vocabulary issue radically; at the same time, it is more efficient in training than word-based models. Our model obtains a higher BLEU score than the bpe-based model after training for one epoch on En-Fr and En-Cs translation tasks. Further analyses show that our model is able to learn morphology.



TL;DR: We devise a character-level neural machine translation built on six recurrent networks, and obtain a  BLEU score comparable to the state-of-the-art NMT on En-Fr and Cs-En translation tasks. 

Review discussion:
Hi, interesting paper!
I have two questions.
First, are you still planning to compare your results to the ones of Chung et al, 2016 and Lee et al, 2016? These seem to be absolutely necessary comparisons for your paper.
Second, the first paragraph of Subsection 5.3 where you say that "learning morphology could speed up learning" leaves me confused. You would have to compare to the performance of the bpe baseline after one epoch, and I don't think you have this number in Table 1. Can you please clarify?
Thanks for your comments.
After having used the similar setting and training for 1,280,000 iterations on Cs-En task (Lee et al, 2016 trained for 1,500,000 iterations), we achieve a comparable performance. Please refer to the updated paper for details.
In the revision, we have changed our source word encoder to use a bidirectional RNN to compute the energy (weight) as shown in Figure 1.
Additionally, we compare our results with "Hierarchical Multiscale Recurrent Neural Networks" (J Chung et al, 2016) in Figure 5, which may be interesting.
As for "learning morphology could speed up learning", it is mainly based on the analysis in Section 5.2. This has also been shown in Table 1 (En-Fr and En-Cs task) from which we see that when we train our model just for one epoch, the obtained result even outperforms the final result with the bpe baseline. We have changed this description and Table 1 to avoid misunderstanding.
Sorry for the delay in the pre-review questions, overall the paper is well-written.
- One of the first question, popped up in my mind as the other reviewer raised was to compare against to the previously published results.
However, in the revised version of the paper those results seemed to be added.
That is very encouraging.
- Have you tried using hierarchical attention on the decoder[1]? Do you think it makes sense for this task?
[1]Nallapati, Ramesh, Bowen Zhou, Çaglar Gulçehre, and Bing Xiang. "Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond."
Thank you for your insightful comments!
1. Lee et al, 2016 [1] submitted their paper to arxiv on October 10th and released the code on October 29th. We found that they trained with longer sentences and much more iterations.
Thus we re-trained our model with similar settings, and encouragingly we obtained a comparable performance as shown in the revision.
2. We haven't tried the hierarchical attention [2]. After having used the attention mechanism [3] in the decoder, we found it better to enhance the encoder based on the experiments (and other publications [1] also enhanced the encoder).
Thus we have devised such an architecture which encodes the source word and source sentence more detailed.
Besides, it is possible to apply our approach to text summarization (as mentioned in the conclusion); we could use the architecture in Figure 1 to encode the sentence and the relevant words may have high energies (weights). It is under developing and testing.
It is a good idea to try the hierarchical attention [2] which may further improve the performance. Thanks for your suggestion. We will try this architecture.
[1] Lee, Jason, Kyunghyun Cho, and Thomas Hofmann. "Fully Character-Level Neural Machine Translation without Explicit Segmentation."
[2] Nallapati, Ramesh, Bowen Zhou, Çaglar Gulçehre, and Bing Xiang. "Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond."
[3] Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. "Neural machine translation by jointly learning to align and translate."
The paper presents one of the first neural translation systems that operates purely at the character-level, another one being https://arxiv.org/abs/1610.03017 , which can be considered a concurrent work. The system is rather complicated and consists of a lot of recurrent networks.  The quantitative results are quite good and the qualitative results are quite encouraging.
First, a few words about the quality of presentation.
Despite being an expert in the area, it is hard for me to be sure that I exactly understood what is being done. The Subsections 3.1 and 3.2 sketch two main features of the architecture at a rather high-level.
For example, does the RNN sentence encoder receive one vector per word as input or more? Figure 2 suggests that it’s just one. The notation h_t is overloaded, used in both Subsection 3.1 and 3.2 with clearly different meaning. An Appendix that explains unambiguously how the model works would be in order.
Also, the approach appears to be limited by its reliance on the availability of blanks between words, a trait which not all languages possess.
Second, the results seem to be quite good.
However, no significant improvement over bpe2char systems is reported.
Also, I would be curious to know how long it takes to train such a model,  because from the description it seems like the model would be very slow to train (400 steps of BiNNN). On a related note, normally an ablation test is a must for such papers, to show that the architectural enhancements applied were actually necessary. I can imagine that this would take a lot of GPU time for such a complex model.
On the bright side, Figure 3 presents some really interesting properties that of the embeddings that the model learnt.
Likewise interesting is Figure 5.
To conclude, I think that this an interesting application paper, but the execution quality could be improved. I am ready to increase my score if an ablation test confirms that the considered encoder is better than a trivial baseline, that e.g. takes the last hidden state for each RNN.
Thanks for your nice comments!
We think our system is less complicated compared to [1] which uses thousands of filters, and it's more natural to encode words with RNN.
Below are the one-to-one responses to the comments.
Comment 1: Presentation.
Response: As Figure 2 suggests, the RNN sentence encoder receives one vector per word. Notation h_t indicates the hidden state of RNN unit, thus we used it in both Subsection 3.1 and 3.2. We have clarified it in the revision and we will add an Appendix to explain unambiguously how the model works. Thanks for your suggestion.
Comment 2: Training time.
Response: En-Cs model is trained for 15 days and Cs-En model is trained for about 22 days, so the training speed is similar to Lee et al, 2016 [1]. We have added the comparison in Table 1 in the revision. As mentioned in the paper, all the models are trained on a single NVIDIA TITAN X GPU. You could test the code which released on github (https://github.com/swordyork/dcnmt). Actually, it takes less memory and much easier for back-propagation because of the hierarchical architecture.
Comment 3: Trivial baseline.
Response: As mentioned in the introduction, the encoder in Ling et al, 2015 [2] has encountered some problems; they just take the last hidden state for each RNN (C2W model) and their results are not competitive (the result in their paper and the result in Table 1).
Besides, our previous version (https://arxiv.org/pdf/1608.04738.pdf and the corresponding code https://github.com/swordyork/dcnmt/tree/old-version) also could be considered as a trivial baseline which takes the last hidden state of each RNN.
However, the results are not such competitive (En-Fr BLEU on newstest2014 is 31.76, and this version is 32.85.).
Thus we have devised such an architecture which encodes the source word and source sentence more detailed.
Thanks!
[1] Lee, Jason, Kyunghyun Cho, and Thomas Hofmann. "Fully Character-Level Neural Machine Translation without Explicit Segmentation."
[2] Ling, Wang, et al. "Character-based Neural Machine Translation."
Dear reviewer,
We have added an appendix that described the model in detail. If it is still unclear, please let us know.
Thanks.
Update after reading the authors' responses & the paper revision dated Dec 21:
I have removed the comment "insufficient comparison to past work" in the title & update the score from 3 -> 5.
The main reason for the score is on novelty. The proposal of HGRU & the use of the R matrix are basically just to achieve the effect of "whether to continue from character-level states or using word-level states". It seems that these solutions are specific to symbolic frameworks like Theano (which the authors used) and TensorFlow. This, however, is not a problem for languages like Matlab (which Luong & Manning used) or Torch.
This is a well-written paper with good analysis in which I especially like Figure 5.
However I think there is little novelty in this work. The title is about learning morphology but there is nothing specifically enforced in the model to learn morphemes or subword units.
For example, maybe some constraints can be put on the weights in w_i in Figure 1 to detect morpheme boundaries or some additional objective like MDL can be used (though it's not clear how these constraints can be incorporated cleanly).
Moreover, I'm very surprised that litte comparison (only a brief mention) was given to the work of (Luong & Manning, 2016) [1], which trains deep 8-layer word-character models and achieves much better results on English-Czech, e.g., 19.6 BLEU compared to 17.0 BLEU achieved in the paper. I think the HGRU thing is over-complicated in terms of presentation. If I read correctly, what HGRU does is basically either continue the character decoder or reset using word-level states at boundaries, which is what was done in [1]. Luong & Manning (2016) even make it more efficient by not having to decode all target words at the morpheme level & it would be good to know the speed of the model proposed in this ICLR submission. What end up new in this paper are perhaps different analyses on what a character-based model learns & adding an additional RNN layer in the encoder.
One minor comment: annotate h_t in Figure 1.
[1] Minh-Thang Luong and Christopher D. Manning. 2016. Achieving Open Vocabulary Neural Machine Translation
with Hybrid Word-Character Models. ACL. https://arxiv.org/pdf/1604.00788v2.pdf
Thanks for your comments!
Below are the one-to-one responses to the comments.
Comment 1: Comparison with Luong & Manning, 2016 [1].
Response: We think it is unfair to compare with such a deep model, neither Lee et al, 2016 [2] nor Chung et al 2016 [3] compared with Luong & Manning, 2016 [1].
Besides, our model is much efficient than [1], because their model is trained for 3 months (you could check the training time in their paper).
However, our En-Cs model is trained for only 15 days on a single NVIDIA TITAN X GPU. We have added the detailed training time in the revision. Thanks for your suggestion!
Comment 2: The HGRU thing is over-complicated in terms of presentation.
Response: It is correct that HGRU is either continue the character decoder or reset using word-level states at boundaries.
However, it is tricky when implementing. We have presented the trick (by using an auxiliary sequence and the matrix R) of our implementation in the submission.
Besides, it is not only more elegant but also more efficient (trained for only 15 days) to decode all target words at the character level with HGRU. Please refer to the revision for the training time in Table 1.
Comment 3: Novelty and specific constraints.
Response: We think another major novelty of our work is the word encoder. As you can see, the attention mechanism also does not specifically enforce the model to align on one word, the alignment is learnt from data.
Similarly, the morphemes or subword units are learnt from data. Actually, it does provide the nice embedding (Figure 3) and segmentation (Figure 5).  We have tried several constraints like sparsity, but less useful. It is better to model the words as an energy model (like the attention mechanism) which is used in the submission.
Comment 4: Annotate h_t in Figure 1.
Response: The h_t is the hidden state of RNN which is omitted in Figure 1. To clarify we have added it to Figure 1 in the revision.
Thanks!
[1] Minh-Thang Luong and Christopher D. Manning. 2016. Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models. ACL. https://arxiv.org/pdf/1604.00788v2.pdf
[2] Lee, Jason, Kyunghyun Cho, and Thomas Hofmann. "Fully Character-Level Neural Machine Translation without Explicit Segmentation." arXiv preprint arXiv:1610.03017 (2016).
[3] Chung, Junyoung, Kyunghyun Cho, and Yoshua Bengio. "A character-level decoder without explicit segmentation for neural machine translation." arXiv preprint arXiv:1603.06147 (2016).
Thanks for your responses.
My point in referring to the work of (Luong & Manning, 2016) is to point out that their work shares some similarity with your work, e.g., having two levels of decoders & testing on WMT'15 En-cs. Adding comparison to their system doesn't bring down the value of your work; it helps readers understand more the difference and you can in fact highlight why your models are more efficient given that your revised version now includes training time. Btw, I checked Luong & Manning's paper, the hybrid system with 19.6 BLEU only takes 3 weeks to train though you're right that their purely character models took 3 months.
I have tried to reread the description of HGRU below Eq. (7) but am having some difficulty in understanding the text. There are two decoders and it wasn't clear which one these phrases, "outputs of decoder", "outputs from the decoder", "outputs of the decoder"are referring to. It would be beneficial to revise that paragraph to make things easier to understand.
Even so, at the high-level, I can't see why it is tricky to implement the following behavior of HGRU "either continue the character decoder or reset using word-level states at boundaries": once you have chosen which states (word or character) are used for different examples in a mini-batch as input, batch training with matrix-matrix multiplication should be possible?
Thank you for your patience and insightful comments.
Below are the one-to-one responses to the comments.
Comment 1: Comparison with Luong & Manning, 2016 [1].
Response: We have added the comparison with Luong & Manning, 2016 [1] in Table 1 and Section 3.2, to help readers more clearly understand the difference and efficiency of the hierarchical decoder in our model.
Comment 2: Implementing the behavior of HGRU.
Response: When training in a batch manner, different sentences have different word boundaries.
For example, when the target sequence batch contains the following sentences:
" Hello world !
I am fine . "
the number of words and boundaries are different for these two sentences. At the framework-level, it will be intractable to conditionally pick outputs from the first-level decoder when training in batch manner. As far as we know, it is impossible for Theano and other symbolic deep learning frameworks, because we need to build the symbolic expression before training. The second-level decoder doesn't know when to reset using word-level states at boundaries for each sentences during training.
Thus, it is tricky. Luong & Manning, 2016 [1] uses two forward passes (one for word-level and another for character-level) in batch training, which is less efficient (we have checked their code [https://github.com/lmthang/nmt.hybrid/blob/master/code/lstmCostGrad.m] and Section 4.3 in their paper [https://arxiv.org/pdf/1604.00788v1.pdf]).
However, in our model, we use a matrix R to unfold the outputs of the first-level decoder and utilize the auxiliary sequence to build the symbolic expression. It makes the batch training process more efficient (single forward pass as usual).  We think it is the main reason why our model is much faster than their purely character model.
Besides, the vector used to seed second-level decoder in our model (see, Eqn. (8) in the revision) is very different from that in Luong & Manning, 2016 [1] (see, Eqn. (6) in their paper). The source word encoder in our model (see, Figure 1) is completely different from that in Luong & Manning, 2016 [1] (see, Figure 1 in their paper).
We have clarified the "decoder" when describing HGRU in the revision. Thanks for your suggestion.
If you have any further comments, please let us know. Thanks.
[1] Minh-Thang Luong and Christopher D. Manning. 2016. Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models. ACL. https://arxiv.org/pdf/1604.00788v2.pdf
Thanks for your comments and paper revisions. I'll take a look later and reevaluate the paper again.
Dear reviewer,
We have added the comparison of the model size to Table 1 in the revision. From Table 1, we could find that Luong's model is nearly five times as large as ours, thus it is not surprised that their model outperforms the others.
Thanks for your time and effort on our paper.
Thanks.
Dear reviewers,
We uploaded a slightly modified version. We have added the training time for each model to Table 1 and clarified some notations.
Thanks!
Dear reviewers,
We uploaded a slightly modified version.
We have further explained the novelty of the hierarchical decoder in Section 3.2.
We have added the comparison with Luong & Manning, 2016 [1] and the trivial baseline (CNMT) to Table 1.
The trivial baseline is the old version of this submission which takes the last hidden state of RNN as the representation of the source word. You could find it on arxiv (https://arxiv.org/pdf/1608.04738v2.pdf).
Thanks!
* Summary: This paper proposes a neural machine translation model that translates the source and the target texts in an end to end manner from characters to characters. The model can learn morphology in the encoder and in the decoder the authors use a hierarchical decoder. Authors provide very compelling results on various bilingual corpora for different language pairs. The paper is well-written, the results are competitive compared to other baselines in the literature.
* Review:
- I think the paper is very well written, I like the analysis presented in this paper. It is clean and precise.
- The idea of using hierarchical decoders have been explored before, e.g. [1]. Can you cite those papers?
- This paper is mainly an application paper and it is mainly the application of several existing components on the character-level NMT tasks. In this sense, it is good that authors made their codes available online.
However, the contributions from the general ML point of view is still limited.
* Some Requests:
-Can you add the size of the models to the Table 1?
- Can you add some of the failure cases of your model, where the model failed to translate correctly?
* An Overview of the Review:
Pros:
- The paper is well written
- Extensive analysis of the model on various language pairs
- Convincing experimental results.
Cons:
- The model is complicated.
- Mainly an architecture engineering/application paper(bringing together various well-known techniques), not much novelty.
- The proposed model is potentially slower than the regular models since it needs to operate over the characters instead of the words and uses several RNNs.
[1] Serban IV, Sordoni A, Bengio Y, Courville A, Pineau J. Hierarchical neural network generative models for movie dialogues. arXiv preprint arXiv:1507.04808. 2015 Jul 17.
Thank you for your insightful comments.
We have added the size of models in Table 1. We could find that our model is the smallest one to achieve the comparable performance. We also added more translation samples in the appendix. Thanks for your suggestion.
Below are the one-to-one responses to the comments.
Comment 1: Mainly an architecture engineering/application paper, not much novelty.
Response: We combine 6 RNNs to build this network. We think it is important to build from the basic components for new functionality in deep learning. The famous networks such as ResNet and GoogLeNet are also built from several existing components.
Comment 2: The idea of using hierarchical decoders.
Response:  It's true that the idea of using hierarchical decoders have been explored before and we have cited the paper [4] in the revision.  Various hierarchical decoders operate on different level.
For example, the hierarchical decoder in [4] decodes from sentence level to word level.
However, none of the prior hierarchical decoders are applicable to NMT because of the training inefficiency.
For example, Ling et al, [2] and Luong et al, [3] have tried the hierarchical decoder, but both are not efficient. The model of Luong et al, [3] needs to take 3 months to train the purely character models (see, Table 1). To our knowledge, our method is the first to make the hierarchical decoder work on NMT.
Comment 3: Learning words representation.
Response: We would like to highlight that another major novelty of our work is learning words representation that an ordinary RNN failed to learn (as shown in Figure 3 and Figure 5). It is not only useful in NMT, but also potentially useful in other NLP areas like language models.
Comment 4: The model is complicated.
Response: We admit that it is a little complicated to implement this network using existent deep learning frameworks, thus we make our codes available online.
However, in terms of the model size, our model is the simplest and smallest (we have added the size of models in the revision, thanks for your suggestion.). More importantly, we think our system is less complicated compared to [1] which uses thousands of filters (their model is much larger than ours). It's more natural and straightforward to using RNNs in this context (we describe the detailed architecture in Appendix). Because of using RNNs, our model is less redundant.
Comment 5: The proposed model is potentially slower.
Response: Our model is slower than the word-base models.
However, the training efficiency of our model is comparable to Lee et al, [1] which is also a character-based model (see, Table 1).
Moreover, the character-based models avoid the large vocabulary and the large softmax function at the cost of the training speed.
If you have any further comments, please let us know. Thanks.
[1] Lee, Jason, Kyunghyun Cho, and Thomas Hofmann. "Fully Character-Level Neural Machine Translation without Explicit Segmentation."
[2] Ling, Wang, et al. "Character-based Neural Machine Translation."
[3] Minh-Thang Luong and Christopher D. Manning. 2016. "Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models."
[4] Serban IV, Sordoni A, Bengio Y, Courville A, Pineau J. Hierarchical neural network generative models for movie dialogues. arXiv preprint arXiv:1507.04808. 2015 Jul 17.
Dear reviewers,
We uploaded an updated version with an appendix.  In the appendix, we describe our model in detail and add more translation samples.
We have added the size of models in Table 1. Table 1 becomes more comprehensive, thanks for your insightful suggestions.
Thanks.

Meta-review:
role: PC
data: comment: This paper is concurrently one of the first successful attempts to do machine translation using a character-character MT modeling, and generally the authors liked the approach. However, the reviewers raised several issues with the novelty and experimental setup of the work. 
 
 Pros:
 - The analysis of the work was strong. This illustrated the underlying property, and all reviewers praised these figures.
 
 Mixed: 
 - Some found the paper clear, praising it as a "well-written paper", however other found that important details were lacking and the notation was improperly overloaded. As the reviewers were generally experts in the area, this should be improved
 - Reviewers were also split on results. Some found the results quite "compelling" and comprehensive, but others thought there should be more comparison to BPE and other morphogically based work
 - Modeling novelty was also questionable. Reviewers like the partial novelty of the character based approach, but felt like the ML contributions were too shallow for ICLR
 
 Cons:
 - Reviewers generally found the model itself to be overly complicated and the paper to focus too much on engineering. 
 - There were questions about experimental setup. In particular, a call for more speed numbers and broader comparison.

Decision: Reject