Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: I have not yet read the paper completely, but just a quick note about references to prior work on lipreading.  The paper claims that "All existing works, however, perform only word classification, not sentence-level sequence
prediction."  This is not true -- prior work on audio-visual speech recognition often also reports video-only (i.e. lipreading) results, and does include sentence prediction.  For example the Neti et al. 2000 paper that you cite does so.  Can you clarify this claim, or remove it?

In general I am a bit taken aback by the lack of mention of the wealth of prior papers on audio-visual speech recognition which include also lipreading results.  The paper may leave readers with the impression that no such work exists, or that the authors don't know about it.  I hope that you can rectify this in a revision.

review_title: prior work on lipreading

final_score: rating: 4: Ok but not good enough - rejection
review_content: - Proven again that end to end training with deep networks gives large
gains over traditional hybrid systems with hand crafted features. The results 
are very nice for the small vocabulary grammar task defined by the GRID corpus. The engineering here is clearly very good, will be interesting to see the performance on large vocabulary LM tasks. Comparison to human lip reading performance for conversational speech will be very interesting here.

- Traditional AV-ASR systems which apply weighted audio/visual posterior fusion reduce to pure lip reading when all the weight is on the visual, there are many curves showing performance of this channel in low audio SNR conditions for both grammar and LM tasks.

- Traditional hybrid approaches to AV-ASR are also sentence level sequence trained with fMPE/MPE/MMI etc. objectives (see old references), so we cannot say here that this is the first sentence-level objective for lipreading model (analogous to saying there was no sequence training in hybrid LVCSR ASR systems before CTC). 


review_title: Nice application of end to end training on the visual pipeline of traditional AV-ASR systems. Sentence level sequence objectives (MPE/MMI/fMPE) have been applied to this problem in the past

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors present a well thought out and constructed system for performing lipreading. The primary novelty is the end-to-end nature of the system for lipreading, with the sentence-level prediction also differentiating this with prior work. The described neural network architecture contains convolutional and recurrent layers with a CTC sequence loss at the end, and beam search decoding with an LM is done to obtain best results. Performance is evaluated on the GRID dataset, with some saliency map and confusion matrix analysis provided as well.

Overall, the work seems of high quality and clearly written with detailed explanations. The final results and analysis appear good as well. One gripe is that that the novelty lies in the choice of application domain as opposed to the methods. Lack of word-level comparisons also makes it difficult to determine the importance of using sentence-level information vs. choices in model architecture/decoding, and finally, the GRID dataset itself appears limited with the grammar and use of a n-gram dictionary. Clearly the system is well engineered and final results impress, though it's unclear how much broader insight the results yield.
review_title: Impressive result on lip reading, interesting analysis, some flawed comparison

final_score: rating: 4: Ok but not good enough - rejection
review_content: UPDATE:  I have read the authors' responses.  I did not read the social media comments about this paper prior to reviewing it.  

I appreciate the authors' updates in response to the reviewer comments.  Overall, however, my review stands.  The authors have taken a task that had not yet been addressed with a straightforward modern deep learning approach, and addressed it with such an approach.  I assume that if we pick up any task that hasn't been worked on for a while, and give it a solid deep learning treatment, we will do well.  I do not see such papers as a contribution to ICLR, unless they also provide new insights, analysis, or surprising results (which, to my mind, this paper does not).  This is a general point and the program chairs may disagree with it, of course.

I have removed my recommendation that this be accepted as a workshop paper, as I have since noticed that the workshop track this year has a different focus.  

************************

ORIGINAL REVIEW:

The authors show that an appropriately engineered LSTM+CNN+CTC network does an excellent job of lipreading on the GRID corpus.  This is a nice result to know about--yet another example of a really nice result that one can get the first time one applies such methods to an old task--and all of the work that went into getting it looks solid (and likely involved some significant engineering effort).  However, this in itself is not sufficiently novel for publication at ICLR.  The paper also needs to be revised to better represent prior work, and ideally remove some of the vague motivational language.  Some specifics on what I think needs to be revised:

- First, the claim of being the first to do sentence-level lipreading.  As mentioned in a pre-review comment, this is not true.  The paper should be revised to discuss the prior work on this task (even though much of it used data that is not public).  Ideally the title should also be changed in light of this.

- The comparison with human lipreaders needs to be qualified a bit.  This task is presumably very unnatural for humans because of the unusual grammar, so perhaps what you are showing is that a machine can better take into account the strong contraints.  This is great, but not a general statement about LipNet vs. humans.

- The paper contains some unnecessary motivational platitudes.  We do not need to invoke Easton and Basala 1982 to motivate modeling context in a linguistic sequence prediction task, and prior work using older sequence models (e.g. HMMs) for lipreading has modeled context as well.  The McGurk effect does not show that lipreading plays a crucial role in human communication.

- It is worth noting that even without the spatial convolution, your Baseline-2D already does extremely well.  So I am not sure about the "importance of spatiotemporal feature extraction" as stated in the conclusion.

Some more minor comments, typos, etc.:

- citations for LSTMs, CTC, etc. should be provided the first time they are mentioned.
- I did not quite follow the justification for upsampling.
- what is meant by "lip-rounding vowels"?  They seem to include almost all English vowels.
- Did you consider keeping the vowel visemes V1-V4 separate rather than collapsing them into one?  Since you list Neti et al.'s full viseme set, it is worth mentioning why you modified it.
- "Given that the speakers are British, the confusion between /aa/ and /ay/..." -- I am not sure what this has to do with British speakers, as the relationship between these vowels exists in other English dialects as well (e.g. American).
- The discussion about confusions within bilabial stops and within alveolar stops is a bit mismatched with the actual confusion data in Fig. 3(b,c).  For example, there does not seem to be any confusion between /m/ and /b/ or between /m/ and /p/.
- "lipreading actuations":  I am not sure what "actuations" means in this context
- "palato-alvealoar" --> "palato-alveolar"
- "Articulatorily alveolar" --> "Alveolar"?
review_title: A nice result, but with limited novelty

final_score: rating: null
review_content: 1. Could you clarify what is meant by "binarised" language model?

2. To see how much context helps, similar to language model vs. no language model results reported below, do the authors have results just training with words instead of sentences?

3. Curious if there's a way to measure the saliency/weight assigned to each time step to see if certain characters associated with those time steps are assigned more/less weight?
review_title: Pre-review questions

Meta-review: Let me start by saying that your area chair does not read Twitter, Reddit/ML, etc. The metareview below is, therefore, based purely on the manuscript and the reviews and rebuttal on OpenReview.
 
 The goal of the ICLR review process is to establish a constructive discussion between the authors of a paper on one side and reviewers and the broader machine-learning community on the other side. The goal of this discussion is to help the authors leverage the community for improving their manuscript.
 
 Whilst one may argue that some of the initial reviews could have provided a more detailed motivation for their rating, there is no evidence that the reviewers were influenced (or even aware of) discussions about this paper on social or other media --- in fact, none of the reviews refers to claims made in those media. Suggestions by the authors that the reviewers are biased by (social) media are, therefore, unfounded: there can be many valid reasons for the differences in opinion between reviewers and authors on the novelty, originality, or importance of this work. The authors are free to debate the opinion of the reviewers, but referring to the reviews as "absolute nonsense", "unreasonable", "condescending", and "disrespectful" is not helping the constructive scientific discussion that ICLR envisions and, frankly, is very offensive to reviewers who voluntarily spend their time in order to improve the quality of scientific research in our field.
 
 Two area chairs have read the paper. They independently reached the conclusion that (1) the reviewers raise valid concerns with respect to the novelty and importance of this work and (2) that the paper is, indeed, borderline for ICLR. The paper is an application paper, in which the authors propose the firstÊend-to-end sentence level lip reading using deep learning. Positive aspects of the paper include:
 
 - A comprehensive and organized review about previous work.
 - Clear description of the model and experimental methods.
 - Careful reporting of the results, with attention to detail.
 - Proposed method appears to perform better than the prior state-of-the-art, and generalizes across speakers.
 
 However, the paper has several prominent negative aspects as well:
 
 - The GRID corpus that is used for experimentation has very substantial (known) limitations. In particular, it is constructed in a way that leads to a very limited (non-natural) set of sentences.Ê(For every word, there is an average of just 8.5 possible options the model has to choose from.)
 - The paper overstates some of its claims. In particular, the claim that the model is "outperforming experienced human lipreaders" is questionable: it is not unlikely that model achieves its performance by exploiting unrealistic statistical biases in the corpus that humans cannot / do not exploit. Similarly, the claims about the "sentence-level" nature of the model are not substantiated: it remains unclear what aspects of the model make this a sentence-level model, nor is there much empirical evidence that the sentence-level treatment of video data is helping much (the NoLM baseline is almost as good as LipNet, despite the strong biases in the GRID corpus).
 - The paper makes several other statements that are not well-founded. As one of the reviewers correctly remarks, the McGurk effect does not show that lipreading plays a crucial role in human communication (it merely shows that vision can influence speech recognition). Similarly, the claim that "Bi-GRUs are crucial for efficient further aggregation" is not supported by empirical evidence.
 
 A high-level downside of this paper is that, while studying a relevant application of deep learning, it presents no technical contributions or novel insights that have impact beyond the application studied in the paper.
Decision: Reject