Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: When dropping the normalization of acoustic model scores, the range of scores obtained might vary and would have an effect on beam pruning and on its relation to the normalized LM scores. Did you analyse this?
review_title: Normalization in beam search

final_score: rating: 7: Good paper, accept
review_content: This submission proposes a letter-level decoder with a variation of the CTC approach they call ASG, where the blank symbol is dropped and replaced by letter repetition symbols, and where explicit normalization is dropped. Both the description of a letter-level model (though not novel), as well as the CTC-variant are interesting. 

The approach is evaluated on the LibriSpeech task. The authors claim that their approach is competitive. They compare their modelling variant ASG to CTC, but a comparison of the letter-level approach to available word-level results are missing. Compared to the results obtained in Panayotov et al. 2015, the performance obtained here seems only comparable to word-level GMM/HMM models, but worse than word-level hybrid DNN/HMM models, though Panayotov et al. also appled speaker adaptation, which was not done, as far as I can see. I suggest to add a comparison to Panyotov's results (in addition to mentioning Baidu's results on Librispeech, which are not comparable due to much larger amounts of training data), to allow readers to get a quantitative idea. As pointed out by the authors in the text, Baidu's GPU implementation for CTC is more aimed at larger vocabularies, therefore the comparison to GPU in Tables 1a-c do not seem to be helpful for this work, without further discussing the implementations.

You are using quite a huge analysis window (nearly 2s). Even though other authors also use windows up to 0.5s to 1s (e.g. MRASTA features), some comments on how you arrive at such a large window, and what advantages you observe for it, would be interesting.

The submission is well written, though more details on the experiences with using non-normalized (transition) scores and beam pruning would be desirable. Table 1 would be better readable if the units of the numbers shown in a/b/c would be shown within the tables, and not only in the caption.

Prior (partial) publications of this work (your NIPS end-to-end workshop paper) should clearly be mentioned/referenced.

What do you mean by transition "scalars"?

I do not repeat further comments here, which were already given in the pre-review period.

Minor comments:
 - Sec. 2.3, end of 2nd sentence: train properly the model -> train the model properly
   End of same paragraph: boostrap -> bootstrap (such errors should be avoided by performing an automatic spell check)
 - Sec. 2.3: Bayse -> Bayes
 - definition of logadd is wrong (see comment) - (applies also for your NIPS end-to-end workshop paper).
 - line before Eq. (3): all possible sequence of letters -> all possible sequences of letters (plural)
 - Sec. 2.4, first line: threholding -> thresholding (spell check..)
 - Figure 4: mention the corpus used here - dev?

review_title: Review

final_score: rating: null
review_content: It seems that you use inconsistent notation - the variable 't' is used for different time scales: in Eq. (1) t represents strided time frames, whereas in x_t above it enumerates frames directly.
review_title: Notation

final_score: rating: 6: Marginally above acceptance threshold
review_content: ​There have been numerous works ​on learning from raw waveforms and training letter-based CTC networks for speech recognition, however, there are very few works on combining both of them with purely ConvNet as it is done in this paper. It is interesting to see results on a large scale corpus such as Librispeech that is used in this paper, though some baseline results from hybrid NN/HMM systems should be provided. To readers, it is unclear how this system is close to state-of-the-art only from Table 2.

The key contribution of this paper may be the end-to-end sequence training criterion for their CTC variant (where the blank symbol is dropped), which may be viewed as sequence training of CTC as H. Sak, et al. "Learning acoustic frame labeling for speech recognition with recurrent neural networks", 2015. However, instead of generating the denominator lattices using a frame-level trained CTC model first, this paper directly compute the sequence-level loss by considering all the competing hypothesis in the normalizer. Therefore, the model is trained end-to-end. From this perspective, it is closely related to D. Povey's LF-MMI for sequence-training of HMMs. As another reviewer has pointed out, references and discussions on that should be provided. 

This approach should be more expensive than frame-level training of CTCs, however, from Table 1, the authors' implementation is much faster. Did the systems there use the same sampling rate? You said at the end of 2.2 that the step size for your model is 20ms. Is it also the same for Baidu's CTC system. Also, have you tried increasing the step size, e.g. to 30ms or 40ms, as people have found that it may work (equally) better, while significantly cut down the computational cost.
review_title: Review

final_score: rating: null
review_content: Sec. 2.3: you use digits to label character repetitions. How do you handle numbers?
review_title: Modified CTC/blank replacement

final_score: rating: null
review_content: A slightly more compressed version of this submission will be presented at the NIPS end-to-end workshop on Dec. 10, 2016. The NIPS submission seems to be a clear subset of this submission and should at least be mentioned in this paper.
review_title: Publication at NIPS/end-to-end workshop

final_score: rating: null
review_content: Fig 3(a) looks like a monophone HMM with self-loop and using a character-level lexicon, and eq(3) is very similar to sequence-training of HMMs. So the question is how it is different from the HMM based system that people are using now?

d_y in section 2.2 is reused, see eq(1) and last paragraph of section 2.2.
review_title: How is this model different from conventional HMM based model?

final_score: rating: null
review_content: I suppose that your definition of the logadd function is wrong, correct would be log(e^a+e^b) if a and b are log-probabilities (with your definition you would end up with a times b). Please confirm.
review_title: logadd?

Meta-review: Without revisions to this paper or a rebuttal from the authors, it is hard to accept this paper. The main contribution of the paper is removing the blank from CTC to create a somewhat different criterion, but this is not particularly novel (see, for example, http://www.isca-speech.org/archive/Interspeech_2016/pdfs/1446.PDF). None of the reviewers were willing to argue for acceptance in the deliberation phase, so unfortunately the recommendation must be to reject this paper.
Decision: Reject