Venue: ICLR 2017 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper presents one of the first neural translation systems that operates purely at the character-level, another one being https://arxiv.org/abs/1610.03017 , which can be considered a concurrent work. The system is rather complicated and consists of a lot of recurrent networks.  The quantitative results are quite good and the qualitative results are quite encouraging.

First, a few words about the quality of presentation. Despite being an expert in the area, it is hard for me to be sure that I exactly understood what is being done. The Subsections 3.1 and 3.2 sketch two main features of the architecture at a rather high-level. For example, does the RNN sentence encoder receive one vector per word as input or more? Figure 2 suggests that it’s just one. The notation h_t is overloaded, used in both Subsection 3.1 and 3.2 with clearly different meaning. An Appendix that explains unambiguously how the model works would be in order. Also, the approach appears to be limited by its reliance on the availability of blanks between words, a trait which not all languages possess.

Second, the results seem to be quite good. However, no significant improvement over bpe2char systems is reported. Also, I would be curious to know how long it takes to train such a model,  because from the description it seems like the model would be very slow to train (400 steps of BiNNN). On a related note, normally an ablation test is a must for such papers, to show that the architectural enhancements applied were actually necessary. I can imagine that this would take a lot of GPU time for such a complex model.

On the bright side, Figure 3 presents some really interesting properties that of the embeddings that the model learnt. Likewise interesting is Figure 5.

To conclude, I think that this an interesting application paper, but the execution quality could be improved. I am ready to increase my score if an ablation test confirms that the considered encoder is better than a trivial baseline, that e.g. takes the last hidden state for each RNN. 
review_title: Good paper, accept

final_score: rating: 5: Marginally below acceptance threshold
review_content: Update after reading the authors' responses & the paper revision dated Dec 21:
I have removed the comment "insufficient comparison to past work" in the title & update the score from 3 -> 5.
The main reason for the score is on novelty. The proposal of HGRU & the use of the R matrix are basically just to achieve the effect of "whether to continue from character-level states or using word-level states". It seems that these solutions are specific to symbolic frameworks like Theano (which the authors used) and TensorFlow. This, however, is not a problem for languages like Matlab (which Luong & Manning used) or Torch.

-----

This is a well-written paper with good analysis in which I especially like Figure 5. However I think there is little novelty in this work. The title is about learning morphology but there is nothing specifically enforced in the model to learn morphemes or subword units. For example, maybe some constraints can be put on the weights in w_i in Figure 1 to detect morpheme boundaries or some additional objective like MDL can be used (though it's not clear how these constraints can be incorporated cleanly). 

Moreover, I'm very surprised that litte comparison (only a brief mention) was given to the work of (Luong & Manning, 2016) [1], which trains deep 8-layer word-character models and achieves much better results on English-Czech, e.g., 19.6 BLEU compared to 17.0 BLEU achieved in the paper. I think the HGRU thing is over-complicated in terms of presentation. If I read correctly, what HGRU does is basically either continue the character decoder or reset using word-level states at boundaries, which is what was done in [1]. Luong & Manning (2016) even make it more efficient by not having to decode all target words at the morpheme level & it would be good to know the speed of the model proposed in this ICLR submission. What end up new in this paper are perhaps different analyses on what a character-based model learns & adding an additional RNN layer in the encoder.

One minor comment: annotate h_t in Figure 1.

[1] Minh-Thang Luong and Christopher D. Manning. 2016. Achieving Open Vocabulary Neural Machine Translation
with Hybrid Word-Character Models. ACL. https://arxiv.org/pdf/1604.00788v2.pdf
review_title: Well-executed paper with good analysis but little novelty

final_score: rating: null
review_content: Sorry for the delay in the pre-review questions, overall the paper is well-written. 

- One of the first question, popped up in my mind as the other reviewer raised was to compare against to the previously published results. However, in the revised version of the paper those results seemed to be added. That is very encouraging.
- Have you tried using hierarchical attention on the decoder[1]? Do you think it makes sense for this task? 


[1]Nallapati, Ramesh, Bowen Zhou, Çaglar Gulçehre, and Bing Xiang. "Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond."


review_title: Re: Deep Character-Level Neural Machine Translation By Learning Morphology

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
* Summary: This paper proposes a neural machine translation model that translates the source and the target texts in an end to end manner from characters to characters. The model can learn morphology in the encoder and in the decoder the authors use a hierarchical decoder. Authors provide very compelling results on various bilingual corpora for different language pairs. The paper is well-written, the results are competitive compared to other baselines in the literature.


* Review:
     - I think the paper is very well written, I like the analysis presented in this paper. It is clean and precise. 
     - The idea of using hierarchical decoders have been explored before, e.g. [1]. Can you cite those papers?
     - This paper is mainly an application paper and it is mainly the application of several existing components on the character-level NMT tasks. In this sense, it is good that authors made their codes available online. However, the contributions from the general ML point of view is still limited.
                   
* Some Requests:
 -Can you add the size of the models to the Table 1? 
- Can you add some of the failure cases of your model, where the model failed to translate correctly?

* An Overview of the Review:

Pros:
    - The paper is well written
    - Extensive analysis of the model on various language pairs
    - Convincing experimental results.    
    
Cons:
    - The model is complicated.
    - Mainly an architecture engineering/application paper(bringing together various well-known techniques), not much novelty.
    - The proposed model is potentially slower than the regular models since it needs to operate over the characters instead of the words and uses several RNNs.

[1] Serban IV, Sordoni A, Bengio Y, Courville A, Pineau J. Hierarchical neural network generative models for movie dialogues. arXiv preprint arXiv:1507.04808. 2015 Jul 17.

review_title: A well written paper

final_score: rating: null
review_content: Hi, interesting paper!

I have two questions. First, are you still planning to compare your results to the ones of Chung et al, 2016 and Lee et al, 2016? These seem to be absolutely necessary comparisons for your paper. Second, the first paragraph of Subsection 5.3 where you say that "learning morphology could speed up learning" leaves me confused. You would have to compare to the performance of the bpe baseline after one epoch, and I don't think you have this number in Table 1. Can you please clarify?
review_title: A few questions

Meta-review: This paper is concurrently one of the first successful attempts to do machine translation using a character-character MT modeling, and generally the authors liked the approach. However, the reviewers raised several issues with the novelty and experimental setup of the work. 
 
 Pros:
 - The analysis of the work was strong. This illustrated the underlying property, and all reviewers praised these figures.
 
 Mixed: 
 - Some found the paper clear, praising it as a "well-written paper", however other found that important details were lacking and the notation was improperly overloaded. As the reviewers were generally experts in the area, this should be improved
 - Reviewers were also split on results. Some found the results quite "compelling" and comprehensive, but others thought there should be more comparison to BPE and other morphogically based work
 - Modeling novelty was also questionable. Reviewers like the partial novelty of the character based approach, but felt like the ML contributions were too shallow for ICLR
 
 Cons:
 - Reviewers generally found the model itself to be overly complicated and the paper to focus too much on engineering. 
 - There were questions about experimental setup. In particular, a call for more speed numbers and broader comparison.
Decision: Reject