Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes the neural noisy channel model, P(x|y), where (x, y) is a input-to-out sequence pair,  based on the authors' previous work on segment to segment neural transduction (SSNT) model. For the noisy channel model, the key difference from sequence-to-sequence is that the complete sequence y is not observed beforehand. SSNT handles this problem elegantly by performing incremental alignment and prediction. However, this paper does not present anything that is particular novel on top of the SSNT. The SSNT model is still applicable by reverting the input and output sequences. The authors said that an unidirectional LSTM has to be used as an encoder instead of the bidirectional LSTM, but I think the difference is minor. The decoding algorithm presented in the appendix is relatively new. 

The experimental study is very comprehensive and strong, however, there is one important baseline number that is missing for all the experiments. Can you give the number that uses direct + LM + bias, and if you can give direct + bias number would be even better. Although using a LM for the direct model does not make a lot of sense mathematically, however, it works pretty well in practice, and the LM can rescore and smooth your predictions, see 

Deep Speech 2: End-to-End Speech Recognition in English and Mandarin

from Baidu for example. I think the LM may be also the key to explain why noisy channel is much better than direct model in Table 3. A couple minor questions are

1. it is not very clear to me is your direct model in the experiments SSNT or sequence-to-sequence model?

2. O(|x|^2*|y|) training complexity is OK, but it would be great to further cut down the computational cost, as it is still very expensive for long input sequences, for example, for paragraph or document level modeling, or speech sequences. 

The paper is well written, and overall, it is still an interesting paper, as the channel model is always of great interest to the general public.

review_title: Review

final_score: rating: 7: Good paper, accept
review_content: This paper proposes to use an SSNT model of p(x|y) to allow for a noisy channel model of conditional generation that (still) allows for incremental generation of y. The authors also propose an approximate search strategy for decoding, and do an extensive empirical evaluation.

PROs: This paper is generally well written, and the SSNT model is quite interesting and its application here well motivated. Furthermore, the empirical evaluation is very well done, and the authors obtain good results.

CONs: One might be concerned about whether the additional training and decoding complexity is warranted. For instance, one might plausibly obtain the benefits of the proposed approach by reranking (full) outputs from a standard seq2seq model with a score combining p(y|x), p(x|y), and p(y). (It's worth noting that Li et al. (NAACL 2016) do something similar for conversation modeling). At the same time, being able to rerank during search may be helpful, and so it might be nice to see some experiments addressing this.

Other Comments: 
 - Given that the main thrust of the paper is to provide a model for p(x|y), the paper might be slightly clearer if Section 2 were presented from the perspective of modeling p(x|y) instead of switching back to p(y|x) as in the original Yu et al. paper. 

 - It initially seems strange to suggest a noisy-channel model as a way of addressing the "explaining away" problem, since now you have an explicit, uncalibrated p(y) term. However, since seq2seq models appear to naturally do a lot of target-side language modeling, incorporating an explicit p(x|y) term seems quite clever.

review_title: Review

final_score: rating: null
review_content: Please can you explain how much it is different from your previous work in terms of the model and training algorithm, i.e. Yu et al 2016 Online segment to segment neural transduction, apart from the experiments?
review_title: Difference from previous work

final_score: rating: 7: Good paper, accept
review_content: The paper proposes an online variant of segment to segment transducers, which allows to circumvent the necessity of observing whole sentence, before making target predictions. Authors mostly build on their previous work, allowing additionally to leverage independent priors on the target hypotheses, like the language grammar or sentence length.

Strong points:
- well written, interesting idea of combining various sources of information in a Bayesian framework for seq2seq models
Handling something in an online manner typically makes things more difficult, and this is what the authors are trying to do here - which is definitely of interest to the community
- strong experimental section, with some strong results (though not complete: see weak points)

Weak points:
- Authors do not improve on computational complexity (w.r.t Tillmann proposal), hence the algorithms may be found difficult to apply in scenarios where inputs may be long (this already takes into account a rather constrained model of alignment latent variables)
- What about the baseline where you only combine direct, LM and bias contributions (no channel)? Was there any (non-obvious) algorithmic constraint why - this has not been included?

Some other (minor) comments:

- Related to the first weak point: can you elaborate more on how the clue of your work is conceptually different from the work of Tillmann et al. (1997) (except, of course, the fact you use connectionist discriminative models to derive particular conditional probabilities). 
- How sensitive is the model to different choices of hyper-parameters in eq (3). Do you naively search through the search space of those, or do something more clever?
- Some more comments on details of the auxiliary direct model would be definitely of interest.
- How crucial is the correct choice of the pruning variables (K1 and K2)? 
- Sec. 2: makes no Markovian assumptions -> no first-order Markovian assumption?

Typos:
Table 1: chanel -> channel (one before last row)

Apologies for late review.
review_title: hideously late review

Meta-review: This paper adapts NMT to a noisy channel formulation utilizing the recently developed SSNT framework. The paper is well-written and has solid experimental results. Howver, the paper can be improved with a bit more originality and impact. In, short the pros and cons of the paper are:
 
 Pro: 
 - Clarity: All agree paper was very "well written" 
 - Quality: Reviewers note the "strong experimental section". Comprehensive results. 
 
 Cons: 
 - Originality: There were concerns about technical novelty: (a) "this paper does not present anything that is particular novel on top of the SSNT" (b) not that "conceptually different from the work of Tillmann et al". 
 - Impact: Reviewers were not completely convinced that this method could not work with simpler means. For instance by using clever reranking or utilizing the deep speech style unprincipled combination. However, this paper does produce a better approach for this problem.
Decision: Accept (Poster)