Venue: ICLR 2017 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a sequence transduction model that first uses a traditional statistical alignment methods to provide alignments for an encoder-decoder type model. The paper provides experiments on a number of morphological inflection generation datasets. They shows an improvement over other models, although they have much smaller improvements over a soft attention model on some of their tasks. 

I found this paper to be well-written and to have very thorough experiments/analysis, but I have concerns that this work isn't particularly different from previous approaches and thus has a more focused contribution that is limited to its application on this type of shorter input (the authors "suggest" that their approach is sufficient for shorter sequences, but don't compare against the approach of Chorowski et al. 2015 or Jailty el at 2016).

In summary, I found this paper to be well-executed/well-written, but it's novelty and scope too small. That said, I feel this work would make a very good short paper. 
review_title: Review

final_score: rating: null
review_content: Did you try to pretrain the soft-attention model to predict the same fixed alignments that were used to train the hard attention? If not, then I am not sure that the comparison between the soft and hard attention was fair, especially when a very small dataset was used.
review_title: Attention pretraining

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes an approach to sequence transduction for the case when a monotonic alignment between the input and the output is plausible. It is assumed that the alignment can be provided as a part of training data, with Chinese Restaurant process being used in the actual experiments. 

The idea makes sense, although its applicability is limited to the domains where a monotonic alignment is available. But as discussed during the pre-review period, there has been a lot of strongly overlapping related work, such as probabilistic models with hard-alignment (Sequence Transduction With Recurrent Neural Network, Graves et al, 2012) and also attempts to use external alignments in end-to-end models (A Neural Transducer, Jaitly et al, 2015). That said, I do not think the approach is sufficiently novel. 

I also have a concern regarding the evaluation. I do not think it is fair to compare the proposed model that depends on external alignment with the vanilla soft-attention model that learns alignments from scratch. In a control experiment soft-attention could be trained to match the external alignment. Such a pretraining could reduce overfitting on the small dataset, the one on which the proposed approach brings the most improvement. On a larger dataset, especially SIGMORPHON, the improvements are not very big and are only obtained for a certain class of languages.

To sum up, two main issues are (a) lack of novelty (b) the comparison of a model trained with external alignment and one without it. 

review_title: Not novel enough

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper describes a recurrent transducer that uses hard monotonic alignments: at each step a discrete decision is taken either to emit the next symbol or to consume the next input token. 

The model is moderately novel - similar architecture was proposed for speech recognition (https://arxiv.org/pdf/1608.01281v1.pdf). Soft monotonic alignemts are also enforced by A. Graves in https://arxiv.org/abs/1308.0850.

The difficult part in training the proposed model is backpropagation through the discrete decisions. Typically, reinforcement learning techniques are used. In this contribution, the authors side-step the issue by using a problem-dependent aligner to generate optimal decisions for which they train the model.

The results indicate that such specially supervised model is better than the generic soft-attention model that doesn't require any problem-dependent external supervision. However the authors did not attempt to work on regularizing the soft-attention model, which is not fair - the extra supervision by using the ground-truth alignment is a form of regularization and it could be used as e.g. an extra signal to the soft-attention model for a better comparison. That being said the authors reash state-of-the-art results against other domain specific methods.

I believe the paper would more suit a NLP venue - it sound and properly written, but its applicability is limited to the considered NLP problem.


review_title: Nice idea, but limited applicability (need an auxiliary solver for alignments)

final_score: rating: null
review_content: In your appendix, you write that you report all results with a 5 model ensemble? Was there a specific reason for doing this?

Also you write that you did not do beamsearch with your model because it was slower and did not improve performance which seems sensible. Did you also not do beamsearch with your soft-attention model? 
review_title: Clarifications for choice to ensemble + not do beamsearch

final_score: rating: null
review_content: The proposed alignment model is virtually the same as https://arxiv.org/abs/1211.3711, which additionally can be trained without prior alignments by using a forward-backward like algorithm to marginalize  over all alignments. The idea was later extended to support hard alignments in https://arxiv.org/abs/1609.08194. Please clarify the relation. 

Very similar models were alos explored for speech recognition, please see: https://arxiv.org/abs/1511.04868 https://arxiv.org/abs/1603.00223

Also, it is not true that the soft-alignment model of Bahadanu et al must attend at each step to all of input sequence. Please see https://arxiv.org/abs/1508.04395 and https://arxiv.org/abs/1506.07503 for the use of location-based priors in a soft-attention model and the use of a windowing that constrains the attention to the vicinity of the previous selection.
review_title: Missing prior work

Meta-review: While this area chair disagrees with some reviewers about (1) the narrowness of the approach's applicability and hence lack of relevance to ICLR, and also (2) the fairness of the methodology, it is nonetheless clear that a stronger case needs to be made for novelty and applicability.
Decision: Reject