Task: Language Identification
Input (Spanish, English): This paper proposes a novel strategy for zero-resource translation where (source, pivot) and (pivot, target) parallel corpora are available. A teacher model for p(target|pivot) is first trained on the (pivot, target) corpus, then a student model for p(target|source) is trained to minimize relative entropy with respect to the teacher on the (source, pivot) corpus. When using word-level relative entropy over samples from the teacher, this approach is shown to outperform previous variants on standard pivoting, as well as other zero-resource strategies.  This is a good contribution: a novel idea, clearly explained, and with convincing empirical support. Unlike some previous work, it makes fairly minimal assumptions about the nature of the NMT systems involved, and hence should be widely applicable.  I have only a few suggestions for further experiments. First, it would be interesting to see how robust this approach is to more dissimilar source and pivot languages, where intuitively the true p(target|source) and p(target|pivot) will be further apart. Second, given the success of introducing word-based diversity, it was surprising not to see a sentence n-best or sentence-sampling experiment. This would be more costly, but not much more so since you’re already doing beam search with the teacher. Finally, related to the previous, it might be interesting to explore transition from word-based diversity to sentence-based as the student converges and no longer needs the signal from low-probability words.  Some further comments:  line 241: Despite its simplicity -> Due to its simplicity  277: target sentence y -> target word y  442: I assume that K=1 and K=5 mean that you compare probabilities of the most probable and 5 most probable words in the current context. If so, how is the current context determined - greedily or with a beam?  Section 4.2. The comparison with an essentially uniform distribution doesn’t seem very informative here: it would be extremely surprising if p(y|z) were not significantly closer to p(y|x) than to uniform. It would be more interesting to know to what extent p(y|z) still provides a useful signal as p(y|x) gets better. This would be easy to measure by comparing p(y|z) to models for p(y|x) trained on different amounts of data or for different numbers of iterations. Another useful thing to explore in this section would be the effect of the mode approximation compared to n-best for sentence-level scores.  555: It’s odd that word beam does worse than word greedy, since word beam should be closer to word sampling. Do you have an explanation for this?  582: The claimed advantage of sent-beam here looks like it may just be noise, given the high variance of these curves.
Output (English): en