Title: LipNet: End-to-End Sentence-level Lipreading
Authors: Yannis M. Assael
Brendan Shillingford
Shimon Whiteson
Nando de Freitas
Venue: ICLR 2017
Keywords: Computer vision
Deep learning

Abstract: Lipreading is the task of decoding text from the movement of a speaker's mouth. Traditional approaches separated the problem into two stages: designing or learning visual features, and prediction. More recent deep lipreading approaches are end-to-end trainable (Wand et al., 2016; Chung & Zisserman, 2016a). However, existing work on models trained end-to-end perform only word classification, rather than sentence-level sequence prediction. Studies have shown that human lipreading performance increases for longer words (Easton & Basala, 1982), indicating the importance of features capturing temporal context in an ambiguous communication channel. Motivated by this observation, we present LipNet, a model that maps a variable-length sequence of video frames to text, making use of spatiotemporal convolutions, a recurrent network, and the connectionist temporal classification loss, trained entirely end-to-end. To the best of our knowledge, LipNet is the first end-to-end sentence-level lipreading model that simultaneously learns spatiotemporal visual features and a sequence model. On the GRID corpus, LipNet achieves 95.2% accuracy in sentence-level, overlapped speaker split task, outperforming experienced human lipreaders and the previous 86.4% word-level state-of-the-art accuracy (Gergen et al., 2016).

TL;DR: LipNet is the first end-to-end sentence-level lipreading model to simultaneously learn spatiotemporal visual features and a sequence model.

Review discussion:
I have not yet read the paper completely, but just a quick note about references to prior work on lipreading.  The paper claims that "All existing works, however, perform only word classification, not sentence-level sequence
prediction."  This is not true -- prior work on audio-visual speech recognition often also reports video-only (i.e. lipreading) results, and does include sentence prediction.
For example the Neti et al. 2000 paper that you cite does so.  Can you clarify this claim, or remove it?
In general I am a bit taken aback by the lack of mention of the wealth of prior papers on audio-visual speech recognition which include also lipreading results.  The paper may leave readers with the impression that no such work exists, or that the authors don't know about it.  I hope that you can rectify this in a revision.
After examining the long report of Neti et al, which mostly focuses on both audio and video as inputs, we have realized that indeed their pipeline applies to sentence-level predictions for video only. Thank you for bringing this to our attention. Their pipeline represents the old way of doing speech with many hand-engineered components. As stated, their visual-only results cannot be interpreted as visual-only recognition, as they are used as rescoring of the noisy audio-only lattices. Potamianos et al. (2003), report visual-only results on the same dataset which we have included in our references, as well as the work of Goldschen et al. 1997.
We believe our end-to-end pipeline, which is similar to modern ASR pipelines, is a better solution. In our revision, we are being more precise with our claim to novelty, namely end-to-end sentence-level prediction.
Related to this, and following the appearance of our paper,  Thanda et al. 2016 (Audio Visual Speech Recognition using Deep Recurrent Neural Networks) and much later Sanabria et al. 2016 (Robust end-to-end deep audiovisual speech recognition) have proposed similar RNN and CTC architectures.
However, the feature extraction in those publications is not fully end-to-end. The first paper does not achieve performance that is as good as LipNet. Interestingly, Sanabria et al present a new state-of-the-art in IBM ViaVoice (Neti et al 2000), using CTC and RNNs. The authors use SIFT features and PCA to reduce the dimensionality of the visual modality. The video-only performance reported is 65.7% accuracy in predicting 12 viseme categories as targets. We have no access to this dataset.
Thank you for pointing this out. We originally intended to refer the reader to the review of Zhou et al. 2014 for details on prior work due to the wealth of such papers which exist.
However, this statement appears to have been accidentally removed since. We have fixed this in our revision.
Our citation list is over two pages long, but we are happy to continue adding references to earlier work if reviewers think they improve the presentation and claims.
1. Could you clarify what is meant by "binarised" language model?
2. To see how much context helps, similar to language model vs. no language model results reported below, do the authors have results just training with words instead of sentences?
3. Curious if there's a way to measure the saliency/weight assigned to each time step to see if certain characters associated with those time steps are assigned more/less weight?
1) We refer to a language model that assigns binarised probabilities depending on the existence of an n-gram on the training set. Equivalently, this is interpretable as a uniform distribution over n-grams that occur at least once in the training set. This setup is also suggested in the decoding section of Graves and Jaitly 2014.
2) To illustrate the effect of the language model, we have introduced a new baseline (Baseline-NoLM), which exhibits a minor performance drop. A word-only model was never attempted, as it was not the goal of the present work. We think that sentence-level predictions are far more challenging, interesting, and useful.
3) Thank you. This would indeed be a nice thing to visualize.
UPDATE:  I have read the authors' responses.  I did not read the social media comments about this paper prior to reviewing it.
I appreciate the authors' updates in response to the reviewer comments.
Overall, however, my review stands.  The authors have taken a task that had not yet been addressed with a straightforward modern deep learning approach, and addressed it with such an approach.  I assume that if we pick up any task that hasn't been worked on for a while, and give it a solid deep learning treatment, we will do well.  I do not see such papers as a contribution to ICLR, unless they also provide new insights, analysis, or surprising results (which, to my mind, this paper does not).  This is a general point and the program chairs may disagree with it, of course.
I have removed my recommendation that this be accepted as a workshop paper, as I have since noticed that the workshop track this year has a different focus.
ORIGINAL REVIEW:
The authors show that an appropriately engineered LSTM+CNN+CTC network does an excellent job of lipreading on the GRID corpus.  This is a nice result to know about--yet another example of a really nice result that one can get the first time one applies such methods to an old task--and all of the work that went into getting it looks solid (and likely involved some significant engineering effort).
However, this in itself is not sufficiently novel for publication at ICLR.  The paper also needs to be revised to better represent prior work, and ideally remove some of the vague motivational language.  Some specifics on what I think needs to be revised:
- First, the claim of being the first to do sentence-level lipreading.  As mentioned in a pre-review comment, this is not true.  The paper should be revised to discuss the prior work on this task (even though much of it used data that is not public).  Ideally the title should also be changed in light of this.
- The comparison with human lipreaders needs to be qualified a bit.  This task is presumably very unnatural for humans because of the unusual grammar, so perhaps what you are showing is that a machine can better take into account the strong contraints.  This is great, but not a general statement about LipNet vs. humans.
- The paper contains some unnecessary motivational platitudes.  We do not need to invoke Easton and Basala 1982 to motivate modeling context in a linguistic sequence prediction task, and prior work using older sequence models (e.g. HMMs) for lipreading has modeled context as well.  The McGurk effect does not show that lipreading plays a crucial role in human communication.
- It is worth noting that even without the spatial convolution, your Baseline-2D already does extremely well.
So I am not sure about the "importance of spatiotemporal feature extraction" as stated in the conclusion.
Some more minor comments, typos, etc.:
- citations for LSTMs, CTC, etc. should be provided the first time they are mentioned.
- I did not quite follow the justification for upsampling.
- what is meant by "lip-rounding vowels"?  They seem to include almost all English vowels.
- Did you consider keeping the vowel visemes V1-V4 separate rather than collapsing them into one?  Since you list Neti et al.'s full viseme set, it is worth mentioning why you modified it.
- "Given that the speakers are British, the confusion between /aa/ and /ay/..." -- I am not sure what this has to do with British speakers, as the relationship between these vowels exists in other English dialects as well (e.g. American).
- The discussion about confusions within bilabial stops and within alveolar stops is a bit mismatched with the actual confusion data in Fig. 3(b,c).
For example, there does not seem to be any confusion between /m/ and /b/ or between /m/ and /p/.
- "lipreading actuations":  I am not sure what "actuations" means in this context
- "palato-alvealoar" --> "palato-alveolar"
- "Articulatorily alveolar" --> "Alveolar"?
1) We have innovated an end-to-end approach to sentence-level lipreading that outperforms the state-of-the-art on GRID.
Despite the vast literature on the topic, this had not been done properly before.
We respectfully, but emphatically, disagree with the subjective opinion that this work only qualifies as a workshop paper. Our line of reasoning is based on the following two facts.
First, the many other papers on lip-reading, which presented hand-engineered or insufficiently general models/architectures and far worse results in comparison to ours, were worthy of publication on venues other than workshops. Ours being a simpler, more powerful, and better-performance model should therefore be held to the same standard.
Second, much of the progress in speech recognition and computer vision mirrors what we have done here: the introduction of a simple, accessible, scalable, accurate, end-to-end trainable pipeline that yields better results on existing public datasets.
2) LipNet is the first end-to-end model that performs sentence-level sequence prediction. Previous work in the field requires either heavy preprocessing of frames to extract image features, temporal preprocessing of frames to extract video features (e.g., optical flow or movement detection), or other types of handcrafted vision pipelines.
However, we agree that to reflect this we should change the title to: LipNet: End-to-End Sentence-Level Lipreading. Please see our revised paper and reply to comments concerning Neti et al 2000 and claims of prior work on sentence level prediction.
3) The grammar of GRID and the setup are indeed unnatural.
However, the people doing the test had the grammar in front of them. Our results with people are specific to the GRID dataset and should be treated as a baseline, and no more than this. We chose to create as many baselines as possible because of the scarcity of prior work at the sentence level on GRID, and our desire to be thorough.
Until we build large datasets for lipreading and real-time apps, it is not wise to make wild claims of superhuman performance. The media recently made such claims about our work, and we spent a lot of time and energy correcting and appropriately qualifying such claims. We certainly make no such claims.
4) The McGurk effect is frequently used to motivate audio-visual interfaces. Some citations are perhaps unnecessary, but since these are common citations in this research area, we chose to include them.
5) Spatiotemporal convolutions result in 2.3x lower WER, compared to Baseline-2D. This is very significant and not to be dismissed. Again, we chose to be thorough and in the new version add further baselines.
6) We have addressed these concerns in the revised version of our paper.
7) We follow the phoneme-to-viseme categorisation provided by Neti et al. 2000. V1-V4 provided no interesting additional information in terms of plotting.
8) We agree with the reviewer that this exists in other dialects; our analysis is based on this dataset.
9) This is most likely a consequence of the grammar of the dataset. /p/ and /b/ appear in several words, while /m/ appears as a letter.
10) Thank you for the suggestions.
The update to the review is unreasonable and purely subjective. It is clear that this paper is not only a straightforward application of deep learning to lip-reading. There have been many such papers by Andrew Zisserman, Juergen Schmidhuber, etc. We are not the first to apply deep learning to lip-reading. We were however the first to do it well and get very good results. The success of the method is the result of careful design of how to use video as input (the baselines clearly show that many other representations do worse), and how to map from this to a sequence.
While the component modules (such as LSTMs, 3D convolutions, dynamic programming, and other layers) are known in deep learning, it is true most deep learning papers use these same modules. The challenge is how to put them together in an intelligent way to solve problems.
Engineering matters when building representations!
If we were to apply your standard for acceptance to all papers that use deep learning to solve real problems of interest, then no papers on applications of deep learning would ever be accepted. Your argument is absolute nonsense.
If you believe papers about applications of deep learning don't belong in ICLR, please speak to your area chair and ask if this is the case. It is not the case to the best of my understanding. Well executed solutions to important problems matter.
I really fail to understand your intentions. We addressed all your concerns in the rebuttal in a positive way. Now, you've come back and offered to reject the paper, and not even recommend it as a workshop, for a completely different subjective reason that most researchers on deep learning would disagree with.
All the analysis and insights you are asking for are in the experimental section of the paper. Please read the paper carefully and comment on actual content. It is clear from your comment about 2D convolution baselines that you don't understand the experiments. We spent a great deal of work creating many baselines and ablations specially to provide insights on how to design video to text models for lip reading, but more widely applicably. We would love objective feedback on the experiments and results.
Finally, your recommendations about how to write the paper are utterly condescending, disrespectful, and unnecessary.
Regardless of the relative technical merit of one side versus another, I think that as a community we should strive to avoid ad hominem attacks like this. Characterizing reviews that disagree with you as "utterly condescending, disrespectful, and unnecessary", "unreasonable and purely subjective", "absolute nonsense", something "that most researchers on deep learning would disagree with", "that you don't understand the experiments", is not conducive to an open and honest exchange of ideas, and will hurt our field in the long term.
I sympathize with your comment. I'm not enjoying this at all.
Our first reply addressed any concerns reviewers might have had in the appropriate way.
However, when a reviewer fails to provide a factual review based on evidence, concluding with the comment "I have removed my recommendation that this be accepted as a workshop paper, as I have since noticed that the workshop track this year has a different focus" despite the fact that the ICLR Workshops website --- http://www.iclr.cc/doku.php?id=iclr2017:workshopcfp --- states clearly that the topics covered may include "Applications in vision, audio, speech, natural language processing, robotics, neuroscience, or any other field", then we know something has gone very wrong and that the reviewer is being unreasonably adversarial.
The reviewer has done its best to reject this paper. We accept the reject decision, but we do not accept the reasons for rejection. One thing is to reject the work, another one is to trash it without proper objective reason as the reviewer has done.
As a community we should indeed be thinking of how to avoid this. Two recommendations:
(i) Anonymous commenting should not be allowed unless you're a reviewer. The tone is far more respectful when one knows who one is addressing.
(ii) Comments like the first comment below should have been moderated and deleted. They are about bigger problems that are orthogonal to this paper.
Otherwise, there is little difference between reviewing for ICLR and commenting on videos in youtube.
Unfortunately my review seems to have rubbed the author(s) the wrong way, and I do not want to add to the discussion on my or their reasonableness.
But I do want to clarify an apparent misunderstanding:
I'd very much like to see this paper in the workshop track.  The reason I removed that recommendation is that I noticed that the workshop CFP calls for "late-breaking developments, very novel ideas and position papers", which does not seem to me to fit this paper.  Personally I prefer a broader view of the workshop track, and in that broader view would see this as an excellent workshop contribution;
but that is not up to me.
Glad that we seem to agree in spirit, although I still think that your disagreement could have been expressed more respectfully. It's important to avoid the possible appearance of a senior figure from a top lab "bullying" a (possibly junior) reviewer.
In response to your recommendations,
(i) Can you point out anonymous comments from non-reviewers in this thread which haven't been respectful? I only see one other, besides mine, and they both seem pretty reasonable. I'd argue anonymity is important for the same reason as it is with reviewers. Frankly, without the assurance of anonymity, more junior people in the field (such as myself) would surely be more hesitant about questioning the work of established researchers like yourself.
(ii) I presume you're referring to Neil Lawrence's comment? He makes it pretty clear that he's "not making a comment about whether the paper should be accepted or not", although I appreciate where you're coming from.
I certainly don't mean to bully junior people.
In fact, there is no way of me knowing that the reviewer is a junior person and I never made that assumption. The only junior people I'm aware of here are my two PhD students who spent their first year of PhD building LipNet, and ended up with this reviewing process.
While I agree that as the discussion progressed I became rude, and apologize to the reviewer and everyone else for this, it is also the case that this reviewing process has the danger of wrong statements being associated with one's paper unless one public objects.
That is, in ICML and NIPS reviewing for instance, only the authors and area chairs get to see reviews that may be making incorrect statements about the paper.
However, in this ICLR public setup, everyone gets to see those statements even if they are incorrect. The authors therefore have the duty to clarify such statements. In the case of LipNet, this has been a long ongoing battle (see my next post).
I did not mean to say that anonymous folks commenting here have been particularly disrespectful.
However, often what they contribute is arguments that are completely orthogonal to the paper. I am personally of the view that I prefer to engage with people by name and have a polite conversation about science as we always have had. The reviewers choosing to stay anonymous is however a good thing. The problem with everyone commenting anonymously and about random topics is that soon a discussion that was meant to be about the paper becomes a discussion about social media, reviewing, etc.
I would like this to be an objective scientific discussion. I would be disappointed if this ends up being no more than another site like youtube. This I see as a threat to this reviewing process, and this particular paper already highlights this threat.
For example, while Neil is a wonderful colleague and had good intentions, by inappropriately posting a rant about the dangers of media in ML and emphasizing our paper here and in the media, our paper became associated with something it simply wasn't doing - even if he didn't mean it that way. There was a clear backlash, eg google LipNet.
Also in this reviewing setup, the reviewers can see the comments and previous reviews before entering their review. This tends to cause biases. We are of the opinion that the review process in this paper was subject to many biases, even if the reviewers had good intentions to simply write a good review as part of their contribution to the community.
All I can suggest is for reviewers to follow guidelines such as https://nips.cc/Conferences/2013/PaperInformation/ReviewerInstructions when entering the reviews. I often find it a good idea to read this website before I start writing my reviews. I would still like to see one review following these criteria for our paper.
Ultimately, even if rejecting a paper, one has to provide guidance to the authors on how to improve the paper or point out serious flaws. We are all in this together, and the progress in the field is what matters.
I apologize for being rude. I however strongly disagree with your opinions.
Let me expand so that perhaps our perspective can become a bit clear.
We released this paper on Friday and by Saturday it was all over reddit. The titles of the articles on reddit suggested we had achieved super-human performance for lip reading in general. Neil pointed this out to me on Twitter, and we immediately started correcting these statements on Sunday. By Monday the story had exploded fueled, ironically, by Neil's comments that culminated on a BBC article alluding to the dangers of lip reading, when ironically too they have been working on lip reading too.
So here is how the sequential adversarial game played out for us:
(i) First, we were accused of over-claiming. We made it clear that the paper made no general claims of super-human performance. We only claimed to do better than a human's baseline with crucial details stated clearly in the paper on the GRID dataset. This is no different than claiming superhuman performance on say MNIST. The claim was of course specific to that dataset and the paper was very clear about this. There is in addition a good record of us clarifying this in the media.
(ii) We were then accused of using too simple a dataset. We argued that this was the largest public dataset that has been used by many other labs including Juergen Schmidhuber's to benchmark lip reading. It is about 28 hours of video (more than 2,000,000 frames), and even if the grammar is reduced, the sequence to sequence treatment makes this a problem of complexity similar to ImageNet, and not MNIST or CIFAR as some might have thought. This is a large and difficult task, and many previous hand-engineered as well as deep learning approaches had done much worse.
Moreover, to further make this point we started working on building new datasets with NVIDIA. We continued having performance in the 90s with the new dataset, which is not finished because building such datasets is hard work.
(iii) We were told our approach is a toy-problem setting and doesn't scale. We argued that our approach is similar to what was used in automatic speech recognition (ASR) and that provided we could collect more data we saw not impediment to scaling.
(iv) Our previous comment relating to ASR then led to the accusation that there is nothing novel here as this is just a standard ASR pipeline. I hope that at this point the reader starts seeing where our frustration begun. We then argued that while there is a similarity to ASR, we are now doing video to speech and that many baselines (advanced in the paper) or other alternative deep learning approaches (including prior work which we cite as well as several papers that appeared after ours) have done worse than LipNet on GRID.
That is, one can come up with many deep learning solutions, but one in particular works much better: LipNet. The paper presents many ablations and comparisons to make this point clear.
(v) We were accused of being unfair because of using a sequence (video) to sequence (imperative sentences with restricted grammar) model and comparing it to what everyone was doing recently: classification. We noted that we were solving a much harder problem (the real one) and that even by solving a harder problem we still do better. We have now added experiments showing that our gains for the classification setup are of course even higher.
(vi) We were asked to compare against approaches that appeared after our paper on arxiv. Weeks after releasing our paper a few papers appeared - all of a sudden this became very popular:
- Thanda et al., 2016, “Audio Visual Speech Recognition using Deep Recurrent Neural Networks”, (9 Nov. 2016), present a multi-modal architecture using CTC and RNNs and evaluate on the GRID corpus. The authors train an audio model to obtain frame-level labels, and then train a feed-forward network with a bottleneck to predict those frame-level labels. This bottleneck is concatenated with its Δ and ΔΔ features and is used as the input to the RNN.
However, this pipeline is not end-to-end from pixels to characters. The performance is lower.
- Chung et al., 2016, “Lipreading Sentences in the Wild”, (16 Nov. 2016), present a multi-modal sequence-to-sequence model with attention. The model is pre-trained on a much larger dataset (BBC Sentences), which is yet not publicly available, and claim 3.3 WER on Speaker Independent and 5.8 WER on Speaker Dependent on the GRID corpus. The only way we can fairly compare to the new results is by also pre-training on the BBC dataset. Our performance is close, but without the need for pre-training. We think this is an excellent competitor to our approach. We have requested the data, but have been told to wait till mid-2007.
- Sanabria et al., 2016, "Robust end-to-end deep audiovisual speech recognition", (21 Nov. 2016), present a new state-of-the-art in IBM ViaVoice (Neti et al. 2000), using CTC and RNNs. The training dataset has 17111 utterances of 261 speakers for training (about 34.9 hours), and 1893 of 26 speakers utterance (4.6 hours) for testing, and is not publicly available. The authors use SIFT features and PCA to reduce the dimensionality of the visual modality. The video-only performance reported is 65.7% accuracy in predicting 12 viseme categories as targets.
- Ephrat and Peleg 2016, “VID2SPEECH: Speech Reconstruction from Silent Video”, (2 Jan. 2017, ICASSP 2017), use a CNN, which is similar to LipNet without the RNN temporal feature aggregation, for generating an intelligible acoustic speech signal from silent video frames of a speaking person from the GRID corpus.
Right now most of these papers cite our arxiv paper. Two have been accepted.
So it seems that the idea of applying deep learning to solve lip reading is considered to be valuable in the speech and vision communities at least.
(vii) We were accused of over claiming regarding sentence level. We hope this has been made more clear in our current paper version. LipNet is to the best of our knowledge the first "end-to-end sentence level" lip reading method. There were prior approaches as pointed out by the reviewers, but the results for those hand-engineered approaches were very poor in comparison. As shown by the references above, the end-to-end approach is now feasible.
In any case, this is a minor claim of the paper, the leap in state-of-the-art performance with a scalable architecture is the most important claim.
(viii) LipNet, in the end, has been accused of merely being an engineering contribution. We disagree with this as our paper contains many ablations, experimental variations, analyses using saliency etc.
However, ultimately, it is true that this paper is the result of a large engineering effort of two PhD students who in the process contributed largely to Torch and public domain speech code over a period longer than a year. Some might thing that engineering doesn't matter, others might see it as one of the most important things behind the deep learning revolution. I am of the latter view. See also http://blog.dennybritz.com/
What makes this frustrating is that we answered all the questions the reviewers asked us. We added the extra experiments they requested, thereby addressing their concerns. We extended the ablations. We felt we had answered all the questions, only to be told at the end the paper was being rejected because: "The authors have taken a task that had not yet been addressed with a straightforward modern deep learning approach, and addressed it with such an approach.  I assume that if we pick up any task that hasn't been worked on for a while, and give it a solid deep learning treatment, we will do well."
The thing is, none of the other deep learning attempts (and there are many) had obtained the same performance as LipNet.
So surely, this cannot be a straightforward solution.
Moreover, there is significant innovation in terms of baselines, analysis and ablations in this paper.
The other two reviewers haven't yet answered. We would be thankful if they could look at the extended paper and rebuttal and comment.
What makes this very perplexing too is that we have had and incredibly positive reception for this idea by most tech companies who are interested in the technology, this work was chosen to be show-cased at a plenary talk at CES (one of the largest conferences in the world), we have received numerous emails for folks dealing with hearing-impairment requesting that we continue this research toward hearing aids (Incidentally, in case you think this application is not relevant enough: "Over 5% of the world’s population – 360 million people – has disabling hearing loss", http://www.who.int/mediacentre/factsheets/fs300/en/ ).
So to be rejected with two straight 4's (what often we attributed to papers that are deeply flawed) is rather disheartening.
On a positive note, we continue building larger datasets and extending LipNet. We hope we will eventually use it to help folks with hearing problems and create silent interfaces. We also plant to release some of the new datasets to advance LipReading as well as what we think would be the first public good implementation of CTC.
We think in the future it will also be very interesting comparing our approach with latent variables and analytical integration via dynamic programming to the sequence-to-sequence approach using attention. There are pros and cons to both approaches. We believe, the dynamic programming approach (LipNet) requires less data, but this needs to be validated with experiments. Some argue that CTC is harder to understand than attention.
Yet we point out that the idea of having models with discrete latent variables and using exact algorithms to marginalize them (eg forward-backward, junction tree and belief propagation) is well established in classical machine learning. Many recent papers on networks with discrete latent variables end up using reinforce, instead of just doing everything analytically, which is often possible in those papers.
So at least, we hope ideas like CTC will become better understood in the literature. We also desperately need good public implementations of this idea.
Without wading into the larger discussion here, I'm just curious what you find lacking about the current implementations of CTC?
Google has one as part of Tensorflow.  Baidu has one that is written in C and CUDA C, is very fast and has torch and tensorflow bindings.
- Proven again that end to end training with deep networks gives large
gains over traditional hybrid systems with hand crafted features. The results
are very nice for the small vocabulary grammar task defined by the GRID corpus. The engineering here is clearly very good, will be interesting to see the performance on large vocabulary LM tasks. Comparison to human lip reading performance for conversational speech will be very interesting here.
- Traditional AV-ASR systems which apply weighted audio/visual posterior fusion reduce to pure lip reading when all the weight is on the visual, there are many curves showing performance of this channel in low audio SNR conditions for both grammar and LM tasks.
- Traditional hybrid approaches to AV-ASR are also sentence level sequence trained with fMPE/MPE/MMI etc. objectives (see old references), so we cannot say here that this is the first sentence-level objective for lipreading model (analogous to saying there was no sequence training in hybrid LVCSR ASR systems before CTC).
Please see our revised document and the responses to the questions below. We believe most of your concerns have been addressed in the latest version of our document.
Thanks to the reviewing process, we have learned that the claim of sentence-level lipreading was too broad. We have therefore fixed this. In our defense, the vast literature on lipreading from video only focuses on word-level predictions.
Moreover, in works doing both audio and video, video only results take a backstage role and performance tends to be poor.  Given this, it is not surprising that we and many other researchers (we are not alone in this) had arrived at this conclusion.
We are now making what we think is an accurate but still very important claim: end-to-end sentence level lipreading. We have an extensive list of citations in our paper. We have not found any reference in the literature that does end-to-end sentence-level lipreading, despite a thorough search and given the wide discussion we have had on this paper. Certainly, we have not found any works that match our performance on the popular GRID dataset.
Several works have appeared after this work using similar pipelines of RNNs and CTC, but those works have failed to achieve the same performance when tested on GRID. Here, and as in most of deep learning, good engineering matters a lot as you kindly acknowledge.
We only know of one work can can match the performance of our method (https://arxiv.org/abs/1611.05358), but that work appeared two weeks after we made our paper available. They use a dataset that is not publically available for pretraining, and hence we have no means of comparing to them directly yet.
The authors present a well thought out and constructed system for performing lipreading. The primary novelty is the end-to-end nature of the system for lipreading, with the sentence-level prediction also differentiating this with prior work. The described neural network architecture contains convolutional and recurrent layers with a CTC sequence loss at the end, and beam search decoding with an LM is done to obtain best results. Performance is evaluated on the GRID dataset, with some saliency map and confusion matrix analysis provided as well.
Overall, the work seems of high quality and clearly written with detailed explanations. The final results and analysis appear good as well. One gripe is that that the novelty lies in the choice of application domain as opposed to the methods. Lack of word-level comparisons also makes it difficult to determine the importance of using sentence-level information vs. choices in model architecture/decoding, and finally, the GRID dataset itself appears limited with the grammar and use of a n-gram dictionary. Clearly the system is well engineered and final results impress, though it's unclear how much broader insight the results yield.
Thank you for your feedback. Our novelty lies both in the domain as well as the architecture.
In contrast to the previous state-of-the-art, which reported a “14% performance drop” using spatiotemporal convolutions instead of spatial convolution (Chung et al. 2016), we present a spatiotemporal convolution architecture, coupled with a recurrent neural network, that substantially improves performance, resulting in ~2.3x lower WER.
Furthermore, for a fair comparison in sentence-level predictions, our Baseline-LSTM model replicates the architecture of Wand et al. 2016, which is the previous deep learning state-of-the-art in the GRID corpus. LipNet achieves ~5x lower WER than Baseline-LSTM.
To illustrate the effect of the language model, in our latest revision, we have introduced a new baseline (Baseline-NoLM), which exhibits only a minor performance drop. A word-only model was never attempted, as it was not the goal of the present work. We think that sentence-level predictions are far more challenging, interesting, and useful.
GRID is the largest publicly available audio-visual sentence-level dataset. Our results are a major step in end-to-end sentence-level training. Evaluating LipNet and building a larger dataset is one of our current efforts.
We strongly request that our paper be reviewed for its contents, contributions, significance, originality, and impact. We also would like to highlight the following three points to reviewers and readers.
(i) Claim of "sentence-level" lipreading: This was the main concern of reviewers 1 and 2, and likely why they gave us the surprisingly low scores of 4 (7 being borderline for acceptance). Our claim is that we proposed the first "end-to-end sentence-level" lipreading approach, and we have edited the paper to make this precise. The phrase “end-to-end” is very important. The previous few attempts at sentence-level lipreading used heuristic pipelines and obtained poor results. LipNet, on the other hand, is fully end-to-end and achieves state-of-the-art results by a significant margin. on the largest available public dataset  (with over 24 hours of video, making it far from trivial). Given the removal of the phrase sentence-level, the reviews of reviewers 1 and 2 regarding prior work are not longer applicable.
(ii) Is the dataset too simple? GRID despite all its shortcomings, is the largest available public dataset with over 24 hours of video (more than 2,000,000 frames and 64,000 possible sentences), making it more comparable to ImageNet than to say MNIST or CIFAR-10. May deep learning papers have used simplified versions of this dataset (eg restrictions to speaker-dependent recognition, or classification only) and even then obtained worse results.
Moreover, we have embarked on more commercial datasets with success as reported in this keynote by NVIDIA's CEO at CES (177,393 attendees). See this video (https://www.youtube.com/watch?v=YTkqA189pzQ) for an example.
As we have stated several times, our solution is scalable and the only key challenge is getting more data.
(iii) Claim of state-of-the-art: At the time of submission to ICLR, we had broken the state-of-the-art on GRID by a very large margin. This alone is a significant achievement and we are rather puzzled by the negative reviews. Two weeks later, another team at Oxford slightly outperformed our results using sequence to sequence attention models, but they pre-train on a much larger BBC dataset, so their results are not comparable to ours. We requested the BBC dataset but is not available for legal reasons.
Hence we cannot compare performance at this stage. We hope to do so in the future as there are pros and cons to the two approaches that we need to understand better as a research community. Weeks after our paper, several industrial and academic labs put forward solutions to lip-reading that are similar to our approach but they achieved lower performance on GRID, highlighting the importance of the design choices we made in this paper.
Some of those papers have been accepted at prestigious conferences and they cite us.
LipNet has done for lip-reading what deep learning did before for speech and image recognition. The architecture does however require careful thought and engineering as pointed out above. We came up with many ablations that demonstrate this in the paper.
LipNet continues being one of the best solutions for end-to-end sequence lip-reading, with no other method having been able to match its performance (unless pre-training on much larger, unavailable datasets), and it has had large impact already. It will be terribly unfair if this paper gets rejected on the grounds of subjective opinion with a discussion of the quality of the architecture, experiment details, and quality of the results.
We have addressed all the reviewer questions in detail below and in the current draft of our paper.

Meta-review:
role: PC
data: comment: Let me start by saying that your area chair does not read Twitter, Reddit/ML, etc. The metareview below is, therefore, based purely on the manuscript and the reviews and rebuttal on OpenReview.
 
 The goal of the ICLR review process is to establish a constructive discussion between the authors of a paper on one side and reviewers and the broader machine-learning community on the other side. The goal of this discussion is to help the authors leverage the community for improving their manuscript.
 
 Whilst one may argue that some of the initial reviews could have provided a more detailed motivation for their rating, there is no evidence that the reviewers were influenced (or even aware of) discussions about this paper on social or other media --- in fact, none of the reviews refers to claims made in those media. Suggestions by the authors that the reviewers are biased by (social) media are, therefore, unfounded: there can be many valid reasons for the differences in opinion between reviewers and authors on the novelty, originality, or importance of this work. The authors are free to debate the opinion of the reviewers, but referring to the reviews as "absolute nonsense", "unreasonable", "condescending", and "disrespectful" is not helping the constructive scientific discussion that ICLR envisions and, frankly, is very offensive to reviewers who voluntarily spend their time in order to improve the quality of scientific research in our field.
 
 Two area chairs have read the paper. They independently reached the conclusion that (1) the reviewers raise valid concerns with respect to the novelty and importance of this work and (2) that the paper is, indeed, borderline for ICLR. The paper is an application paper, in which the authors propose the firstÊend-to-end sentence level lip reading using deep learning. Positive aspects of the paper include:
 
 - A comprehensive and organized review about previous work.
 - Clear description of the model and experimental methods.
 - Careful reporting of the results, with attention to detail.
 - Proposed method appears to perform better than the prior state-of-the-art, and generalizes across speakers.
 
 However, the paper has several prominent negative aspects as well:
 
 - The GRID corpus that is used for experimentation has very substantial (known) limitations. In particular, it is constructed in a way that leads to a very limited (non-natural) set of sentences.Ê(For every word, there is an average of just 8.5 possible options the model has to choose from.)
 - The paper overstates some of its claims. In particular, the claim that the model is "outperforming experienced human lipreaders" is questionable: it is not unlikely that model achieves its performance by exploiting unrealistic statistical biases in the corpus that humans cannot / do not exploit. Similarly, the claims about the "sentence-level" nature of the model are not substantiated: it remains unclear what aspects of the model make this a sentence-level model, nor is there much empirical evidence that the sentence-level treatment of video data is helping much (the NoLM baseline is almost as good as LipNet, despite the strong biases in the GRID corpus).
 - The paper makes several other statements that are not well-founded. As one of the reviewers correctly remarks, the McGurk effect does not show that lipreading plays a crucial role in human communication (it merely shows that vision can influence speech recognition). Similarly, the claim that "Bi-GRUs are crucial for efficient further aggregation" is not supported by empirical evidence.
 
 A high-level downside of this paper is that, while studying a relevant application of deep learning, it presents no technical contributions or novel insights that have impact beyond the application studied in the paper.

Decision: Reject