Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper presents a machine translation system based on a combination of a neural machine translation system (NMT) and a context-based machine translation (CBMT). The method is evaluated on a small parallel corpus application of English-Amharic translation. The idea is that in the small corpus setting, the CBMT can leverage a manually built bilingual dictionary
to improve on the standard NMT.  

Clarity: The paper is reasonably clear, though there are numerous typos and minor language glitches (missing articles, etc.). Overall the quality of the writing is probably a bit below what's acceptable for publication at ICLR, but nothing that could not be fixed on subsequent revisions.

Novelty: The method proposed appears to be a fairly straightforward variant of one proposed in a previous paper, where an NMT system was combined with a phrase-based MT system (Zoph & Knight, 2016). There seems to be no novel machine learning contribution (nor is it claimed). This paper seems more appropriate for a venue more focused on machine translation rather than a machine learning venue such as ICLR.

Empirical evidence in support of the claims: The authors set out to demonstrate that by combining a CBMT output into an NMT approach, one can get the best of both approaches. Their results do not strongly support this claim. The results suggest that in the context of the small-scale experiments considered, the baseline CBMT model is actually overall the best performing model. It is therefore strange that, in their last sentence of the conclusion, the authors persist in claiming that their combination "outperforms a simple NMT and a basic CBMT system". That being said, the sub-claim that the
NMT/CBMT hybrid improves on the baseline NMT system is well established. 

In light of the relatively low novelty and the lack of compelling empirical performance for the proposed combined MT system, I do not feel that this paper is appropriate for ICLR at this time.

review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: This paper aims to combine a traditional CBMT system with an NMT system. The core idea of the paper is to use the output of the CBMT system as a second source to a multi-source NMT system. The first source of the system is CBMT, the second source is the original source and the output is the translation in the target language. All the experiments are conducted for English-Amharic with a small amount of parallel data from the Bible. A lot of details are provided about the generation of the CBMT system using Google Translate and details are provided about the approach to create such a system.

Pros:
- The idea of using additional outputs to the NMT system and outputs from a context-aware system is neat. This has been done by others who have included PBMT outputs in the past. However, this might be the first to include a CBMT result as an additional source.
- Focusing on a low-resource language like Amharic is good for the community and will encourage more research in these underrepresented languages.

Cons:
- A lot of the techniques described for building the traditional CBMT system are obsolete these days and people prefer neural methods. I worry if this is relevant in the current day.
- The authors could have compared against other ways of incorporating context as a strong baseline - like a n-to-n or n-to-1 NMT system.
- Most experiments in the paper are conducted on a small data set and this is a big downside of the paper.
- More detailed analyses of where the contextual phenomena was incorporated might have helped the paper.


review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: For the low-resource pair English-Amharic, the authors propose to combine context-based machine translation (CNMT), which is built by using a bilingual dictionary and then connecting the resulting n-grams, with a neural MT system. The source sentence, as well as the CNMT output, are used as inputs to the RNN-based translation system.

I vote for rejection because the paper makes some unfounded claims, misses important related work, has some methodological issues and presents unconvincing results.

The paper claims that CBMT is a new approach, but it dates from 2006. The authors say that it outperforms other MT approaches, but a more recent reference would be needed. While neural machine translation may sometimes struggle with rare words, using sub-word units may help alleviate the issue (Sennrich et al. Neural machine translation of rare words with subword units). The claim that RNNs learn from their previous mistakes is also unclear. It's true in the sense that backpropagation is learning from errors, but using previous reference outputs can cause exposure bias (Ranzato et al. Sequence Level Training with Recurrent Neural Networks).

The paper fails to cite related work, in particular on low-resource NMT (e.g. Gu et al. Universal Neural Machine Translation for Extremely Low Resource Languages) and unsupervised translation (e.g. Artetxe et al. Unsupervised Neural Machine Translation).

The CBMT system is built on top of the Google Translate English-Amharic system. However, that model may have seen the test data during training.

By combining CBMT with NMT, the authors obtain better results than NMT alone, but worse than with CBMT only. As such, the usefulness of the approach in a very low-resource scenario is unclear.

Minor points:

Some typos could be corrected: BLUE -> BLEU, Loung -> Luong, weather -> whether
review_title: Official Blind Review #2

Meta-review: The authors propose a model which combines a neural machine translation system and a context-based machine translation model, which combines some aspects of rule and example based MT.  This paper presents work based on obsolete techniques, has relatively low novelty, has problematic experimental design and lacks compelling performance improvements. The authors rebutted some of the reviewers claims, but did not convince them to change their scores. 
Decision: Reject