Venue: ICLR 2017 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This submission proposes to learn the word decomposition, or word to sub-word sequence mapping jointly with the attention based sequence-to-sequence model. A particular feature of this approach is that the decomposition is not static, instead, it also conditions on the acoustic input, and the mapping is probabilistic, i.e., one word may map to multiple sub-word sequences. The authors argue that the dynamic decomposition approach can more naturally reflect the acoustic pattern. Interestingly, the motivation behind this approach is analogous to learning the pronunciation mixture model for HMM based speech recognition, where the probabilistic mapping from a word to its pronunciations also conditions on the acoustic input, e.g.,

I. McGraw, I. Badr, and J. Glass, "Learning lexicons form speech using a pronunciation mixture model," in IEEE Transactions on Audio, Speech, and Language Processing, 2013

L. Lu, A. Ghoshal, S. Renals, "Acoustic data-driven pronunciation lexicon for large vocabulary speech recognition", in Proc. ASRU 

R. Singh, B. Raj, and R. Stern, "Automatic generation of subword units for speech recognition systems,"  in IEEE Transactions on Speech and Audio Processing, 2002

It would be interesting to put this work in the context by linking it to some previous works in the HMM framework.

Overall, the paper is well written, and it is theoretically convincing. The experimental study could be more solid, e.g., it is reasonable to have a word-level baseline, as the proposed approach lies in between the character-level and word-level systems. the vocabulary size of the WSJ si284 dataset is 20K at maximum, which is not very large for the softmax layer, and it is a closed vocabulary task. I guess the word-level system may be also competitive to the numbers reported in this paper. Furthermore, can you explain what is the computational bottleneck of the proposed approach? You downsampled the data by the factor of 4 using an RNN, and it still took around 5 days to converge. To me, it is a bit expensive, especially given that you only take one sample when computing the gradient. Table 2 is a little bit misleading, as CTC with language model and seq2seq with a language model model from Bahdanau et al. is much closer to the best number reported in this Table 2, while you may only get a very small improvement using a language model. Finally, "O(5) days to converge" sounds a bit odd to me. 
review_title: Review

final_score: rating: null
review_content: Is it correct to understand the collapse function as a word level mapping from a word to a valid subword sequence (or the other way around). If this is correct, it works like a dictionary, and do you use it to constrain your search of \hat{z} as eq(18)? Otherwise, if you do best path search or beam search of \hat{z} without this constraint, you may end up with a sequence of \hat{z} that does not correspond to any valid word sequence. 

It is still straightforward to incorporate a language model into the decoder, do you have any result of that other than Table 2 only?

The first term in the right side of eq (4) should be \frac{1}{p(y|x; \theta} instead of \frac{1}{p(y|x, z;\theta}?
review_title: The collapse function and language model

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper proposes to learn decomposition of sequences (such as words) for speech recognition. It addresses an important issue and I forsee it being useful for other applications such as machine translation. While the approach is novel and well-motivated, I would very much like to see a comparison against byte pair encoding (BPE). BPE is a very natural (and important) baseline (i.e. dynamic vs fixed decomposition). The BPE performance should be obtained for various BPE vocab sizes. 

Minor points
- Did the learned decompositions correspond to phonetically meaningful units? From the example in the appendix it's hard to tell if the model is learning phonemes or just most frequent character n-grams.
- Any thoughts on applications outside of speech recognition? If this is shown to be effective in other domains it would be a really strong contribution (but this is probably outside the scope for now).

review_title: null

final_score: rating: null
review_content: Could the authors give more motivation for input (acoustic) dependent segmentation of words?
Why have the authors not compared their method with any publicly available segmenter to get the sub-word units (assuming the segmentation should not depend on acoustic), e.g. morfessor?
During estimation of the gradient and sampling z with (approximated) p(z|y,x;\theta), how do you obtain the posteriors of p(z_t|y,x,z<t;\theta) exactly, when not the uniform distribution is used?
How many sample of z is used to estimate the gradient? If only one, why? Have you tried more?
There is some mismatch between training (sum over many z) and decoding (most probable z). Could you comment on that?

review_title: questions

final_score: rating: 7: Good paper, accept
review_content: Interesting paper which proposes jointly learning automatic segmentation of words to sub words and their acoustic models.

Although the training handles the word segmentation as hidden variable which depends also on the acoustic representations, during the decoding only maximum approximation is used.

The authors present nice improvements over character based results, however they did not compare results with word segmentation which does not assume the dependency on acoustic.

Obviously, only text based segmentation would result in two (but simpler) independent tasks. In order to extract such segmentation several publicly open tools are available and should be cited. Some of those tools can also exploit the unigram probabilities of the words to perform their segmentations.

It looks that the improvements come from the longer acoustical units - longer acoustical constraints which could lead to less confused search -, pointing towards full word models. In another way, less tokens are more probable due to less multiplication of probabilities. As a thought experiment for an extreme case: if all the possible segmentations would be possible (mixture of all word fragments, characters, and full-words), would the proposed model use word fragments at all? (WSJ is a closed vocabulary task). It would be good to show that the sub word model could outperform even a full-word model (no segmentation).
Your model estimates p(z_t|x,z<t;\theta), but during training both p(z_t|x,y,z<t;\theta) and p(z_t|x,z<t;\theta) are needed. Their connections and calculations, in general Section 2 should be more detailed. The experimental setup should be presented in a replicable way.

The authors use one sample of Z during the gradient calculation. Would not it be more consistent then to use the maximum instead of sampling? Or is even getting the maximum is computationally too expensive? Considering the left-to-right approximation, this might be possible and more consistent with your decoding, please address this issue in details.

review_title: An interesting paper, but raising a few questions.

Meta-review: This work proposes a method for segmenting target generation sequence that is learned as part of the model. Generally all reviewers found this paper novel and interesting.
 
 Pros:
 - Quality: The paper is both containing "with solid theoretically justified" and "present(s) nice improvements over character based result". One reviewer asked for the "the experimental study could be more solid."
 - Impact: methods like "BPE" are now somewhat standard hacks in seq2seq modeling. This type of model could be potentially impactful at disrupting. 
 - Clarity: Reviewers found the work to be a "clearly written paper" 
 
 Cons:
 - Some of the reviewers were not as enthuthiastic about this work compared to other papers. There were several comments asking for further experimental results. However I found that the author's responses clearly explained these away and provided clear justificition for why they were not necessary, already included, or explained by previous results. I feel that this takes care of the major issues, and warrants a small raise in score.
Decision: Accept (Poster)