Title: Under review as a

Abstract: Lipreading is the task of decoding text from the movement of a speaker's mouth. Traditional approaches separated the problem into two stages: designing or learning visual features, and prediction. More recent deep lipreading approaches are end-to-end trainable (Wand et al., 2016; Chung & Zisserman, 2016a). However, existing work on models trained end-to-end perform only word classification, rather than sentence-level sequence prediction. Studies have shown that human lipreading performance increases for longer words (Easton & Basala, 1982) , indicating the importance of features capturing temporal context in an ambiguous communication channel. Motivated by this observation, we present LipNet, a model that maps a variable-length sequence of video frames to text, making use of spatiotemporal convolutions, a recurrent network, and the connectionist temporal classification loss, trained entirely end-to-end. To the best of our knowledge, LipNet is the first end-to-end sentence-level lipreading model that simultaneously learns spatiotemporal visual features and a sequence model. On the GRID corpus, LipNet achieves 95.2% accuracy in sentence-level, overlapped speaker split task, outperforming experienced human lipreaders and the previous 86.4% word-level state-of-the-art accuracy (Gergen et al., 2016) .

1. INTRODUCTION:

Lipreading plays a crucial role in human communication and speech understanding, as highlighted by the McGurk effect (McGurk & MacDonald, 1976) , where one phoneme's audio dubbed on top of a video of someone speaking a different phoneme results in a third phoneme being perceived.

Lipreading is a notoriously difficult task for humans, specially in the absence of context 1 . Most lipreading actuations, besides the lips and sometimes tongue and teeth, are latent and difficult to disambiguate without context (Fisher, 1968; Woodward & Barber, 1960) . For example, Fisher (1968) gives 5 categories of visual phonemes (called visemes), out of a list of 23 initial consonant phonemes, that are commonly confused by people when viewing a speaker's mouth. Many of these were asymmetrically confused, and observations were similar for final consonant phonemes.

Consequently, human lipreading performance is poor. Hearing-impaired people achieve an accuracy of only 17±12% even for a limited subset of 30 monosyllabic words and 21±11% for 30 compound words (Easton & Basala, 1982) . An important goal, therefore, is to automate lipreading. Machine lipreaders have enormous practical potential, with applications in improved hearing aids, silent dictation in public spaces, security, speech recognition in noisy environments, biometric identification, and silent-movie processing.

Machine lipreading is difficult because it requires extracting spatiotemporal features from the video (since both position and motion are important). Recent deep learning approaches attempt to extract those features end-to-end. Most existing work, however, performs only word classification, not sentence-level sequence prediction.

In this paper, we present LipNet, which is to the best of our knowledge, the first end-to-end sentence-level lipreading model. As with modern deep learning based automatic speech recognition (ASR), LipNet is trained end-to-end to make sentence-level predictions. Our model operates at the character-level, using spatiotemporal convolutional neural networks (STCNNs), recurrent neural networks (RNNs), and the connectionist temporal classification loss (CTC) Graves et al. (2006) .

Our empirical results on the GRID corpus (Cooke et al., 2006) , one of the few public sentence-level datasets, show that LipNet attains a 95.2% sentence-level word accuracy, in a overlapped speakers split that is popular for benchmarking lipreading methods. The previous best accuracy reported on an aligned word classification version of this task was 86.4% (Gergen et al., 2016) . Furthermore, LipNet can generalise across unseen speakers in the GRID corpus with an accuracy of 88.6%.

We also compare the performance of LipNet with that of hearing-impaired people who can lipread on the GRID corpus task. On average, they achieve an accuracy of 52.3%, in contrast to LipNet's 1.69× higher accuracy in the same sentences.

Finally, by applying saliency visualisation techniques (Zeiler & Fergus, 2014; Simonyan et al., 2013) , we interpret LipNet's learned behaviour, showing that the model attends to phonologically important regions in the video. Furthermore, by computing intra-viseme and inter-viseme confusion matrices at the phoneme level, we show that almost all of LipNet's few erroneous predictions occur within visemes, since context is sometimes insufficient for disambiguation.

2. RELATED WORK:

In this section, we outline various existing approaches to automated lipreading.

Automated lipreading: Most existing work on lipreading does not employ deep learning. Such work requires either heavy preprocessing of frames to extract image features, temporal preprocessing of frames to extract video features (e.g., optical flow or movement detection), or other types of handcrafted vision pipelines (Matthews et al., 2002; Zhao et al., 2009; Gurban & Thiran, 2009; Papandreou et al., 2007; 2009; Pitsikalis et al., 2006; Lucey & Sridharan, 2006; Papandreou et al., 2009) . The automated lipreading literature is too vast to adequately cover, so we refer the reader to Zhou et al. (2014) for an extensive review.

Notably, Goldschen et al. (1997) were the first to do visual-only sentence-level lipreading using hidden Markov models (HMMs) in a limited dataset, using hand-segmented phones. Later, Neti et al. (2000) were the first to do sentence-level audiovisual speech recognition using an HMM combined with hand-engineered features, on the IBM ViaVoice (Neti et al., 2000) dataset. The authors improve speech recognition performance in noisy environments by fusing visual features with audio ones. The dataset contains 17111 utterances of 261 speakers for training (about 34.9 hours) and is not publicly available. As stated, their visual-only results cannot be interpreted as visual-only recognition, as they are used as rescoring of the noisy audio-only lattices. Using a similar approach, Potamianos et al. (2003) report speaker independent and speaker adapted 91.62%, 82.31% WER in the same dataset respectively, and 38.53%, 16.77% WER in the connected DIGIT corpus, which contains sentences of digits. Furthermore, Gergen et al. (2016) use speaker-dependent training on an LDA-transformed version of the Discrete Cosine Transforms of the mouth regions in an HMM/GMM system. This work holds the previous state-of-the-art on the GRID corpus with a speaker-dependent accuracy of 86.4%. Generalisation across speakers and extraction of motion features is considered an open problem, as noted in (Zhou et al., 2014) . LipNet addresses both of these issues.

Classification with deep learning: In recent years, there have been several attempts to apply deep learning to lipreading. However, all of these approaches perform only word or phoneme classification, whereas LipNet performs full sentence sequence prediction. Approaches include learning multimodal audio-visual representations (Ngiam et al., 2011; Sui et al., 2015; Ninomiya et al., 2015; Petridis & Pantic, 2016) , learning visual features as part of a traditional speech-style processing pipeline (e.g. HMMs, GMM-HMMs, etc.) for classifying words and/or phonemes (Almajai et al., 2016; Takashima et al., 2016; Noda et al., 2014; Koller et al., 2015) , or combinations thereof (Takashima et al., 2016) . Many of these approaches mirror early progress in applying neural networks for acoustic processing in speech recognition (Hinton et al., 2012) . Chung & Zisserman (2016a) propose spatial and spatiotemporal convolutional neural networks, based on VGG, for word classification. The architectures are evaluated on a word-level dataset BBC TV (333 and 500 classes), but, as reported, their spatiotemporal models fall short of the spatial architectures by an average of around 14%. Additionally, their models cannot handle variable sequence lengths and they do not attempt sentence-level sequence prediction.

Chung & Zisserman (2016b) train an audio-visual max-margin matching model for learning pretrained mouth features, which they use as inputs to an LSTM for 10-phrase classification on the OuluVS2 dataset, as well as a non-lipreading task. Wand et al. (2016) introduce LSTM recurrent neural networks for lipreading but address neither sentence-level sequence prediction nor speaker independence. Garg et al. (2016) apply a VGG pre-trained on faces to classifying words and phrases from the MIRACL-VC1 dataset, which has only 10 words and 10 phrases. However, their best recurrent model is trained by freezing the VGGNet parameters and then training the RNN, rather than training them jointly. Their best model achieves only 56.0% word classification accuracy, and 44.5% phrase classification accuracy, despite both of these being 10-class classification tasks.

Sequence prediction in speech recognition: The field of automatic speech recognition (ASR) would not be in the state it is today without modern advances in deep learning, many of which have occurred in the context of ASR (Graves et al., 2006; Dahl et al., 2012; Hinton et al., 2012) . The connectionist temporal classification loss (CTC) of Graves et al. (2006) drove the movement from deep learning as a component of ASR, to deep ASR systems trained end-to-end (Graves & Jaitly, 2014; Maas et al., 2015; Amodei et al., 2015) . As mentioned earlier, much recent lipreading progress has mirrored early progress in ASR, but stopping short of sequence prediction.

LipNet is the first end-to-end model that performs sentence-level sequence prediction for visual speech recogntion. That is, we demonstrate the first work that takes as input as sequence of images and outputs a distribution over sequences of tokens; it is trained end-to-end using CTC and thus also does not require alignments.

Lipreading Datasets: Lipreading datasets (AVICar, AVLetters, AVLetters2, BBC TV, CUAVE, OuluVS1, OuluVS2) are plentiful (Zhou et al., 2014; Chung & Zisserman, 2016a) , but most only contain single words or are too small. One exception is the GRID corpus (Cooke et al., 2006) , which has audio and video recordings of 34 speakers who produced 1000 sentences each, for a total of 28 hours across 34000 sentences. Table 1 summarises state-of-the-art performance in each of the main lipreading datasets.

Table 1 : Existing lipreading datasets and the state-of-the-art accuracy reported on these. The size column represents the number of utterances used by the authors for training. Although the GRID corpus contains entire sentences, Gergen et al. (2016) consider only the simpler case of predicting isolated words. LipNet predicts sequences and hence can exploit temporal context to attain much higher accuracy. Phrase-level approaches were treated as plain classification. We use the GRID corpus to evaluate LipNet because it is sentence-level and has the most data. The sentences are drawn from the following simple grammar: command (4) + color (4) + preposition (4) + letter (25) + digit (10) + adverb (4) , where the number denotes how many word choices there are for each of the 6 word categories. The categories consist of, respectively, {bin, lay, place, set}, {blue, green, red, white}, {at, by, in, with}, {A, . . . , Z}\{W }, {zero, . . . , nine}, and {again, now, please, soon}, yielding 64000 possible sentences. For example, two sentences in the data are "set blue by A four please" and "place red at C zero again".

Method:



3. LIPNET:

LipNet is a neural network architecture for lipreading that maps variable-length sequences of video frames to text sequences, and is trained end-to-end. In this section, we describe LipNet's building blocks and architecture.

3.1. SPATIOTEMPORAL CONVOLUTIONS:

Convolutional neural networks (CNNs), containing stacked convolutions operating spatially over an image, have been instrumental in advancing performance in computer visions tasks such as object recognition that receive an image as input (Krizhevsky et al., 2012) . A basic 2D convolution layer from C channels to C channels (without a bias and with unit stride) computes

[conv(x, w)] c ij = C c=1 kw i =1 k h j =1 w c ci j x c,i+i ,j+j ,

for input x and weights w ∈ R C ×C×kw×k h where we define x cij = 0 for i, j out of bounds. Spatiotemporal convolutional neural networks (STCNNs) can process video data by convolving across time, as well as the spatial dimensions (Karpathy et al., 2014; Ji et al., 2013) . Hence similarly,

[stconv(x, w)] c tij = C c=1 kt t =1 kw i =1 k h j =1

w c ct i j x c,t+t ,i+i ,j+j .

3.2. GATED RECURRENT UNIT:

Gated Recurrent Unit (GRU) (Chung et al., 2014 ) is a type of recurrent neural network (RNN) that improves upon earlier RNNs by adding cells and gates for propagating information over more timesteps and learning to control this information flow. It is similar to the Long Short-Term Memory (LSTM) RNN (Hochreiter & Schmidhuber, 1997) . We use the standard formulation:

[u t , r t ] T = sigm(W z z t + W h h t-1 + b g ) ht = tanh(U z z t + U h (r t h t-1 ) + b h ) h t = (1 -u t ) h t-1 + u t ht

where z := {z 1 , . . . , z T } is the input sequence to the RNN, denotes element-wise multiplication, and sigm(r) = 1/(1 + exp(-r)). We use a bidirectional GRU (Bi-GRU) as introduced by Graves & Schmidhuber (2005) in the context of LSTMs: one RNN maps {z 1 , . . . ,

z T } → { -→ h 1 , . . . , -→ h T }, and another {z T , . . . , z 1 } → { ← - h 1 , . . . , ← - h T }, then h t := [ - → h t , ← - h t ].

The Bi-GRU ensures that h t depends on z t for all t . To parameterise a distribution over sequences, at time-step t let p(u t |z) = softmax(mlp(h t ; W mlp )), where mlp is a feed-forward network with weights W mlp . Then we can define the distribution over length-T sequences as p(u 1 , . . . , u T |z) = 1≤t≤T p(u t |z), where T is determined by z, the input to the GRU. In LipNet, z is the output of the STCNN.

3.3. CONNECTIONIST TEMPORAL CLASSIFICATION:

The connectionist temporal classification (CTC) loss (Graves et al., 2006) is widely used in modern speech recognition as it eliminates the need for training data that aligns inputs to target outputs (Amodei et al., 2015; Graves & Jaitly, 2014; Maas et al., 2015) . Given a model that outputs a sequence of discrete distributions over the token classes (vocabulary) augmented with a special "blank" token, CTC computes the probability of a sequence by marginalising over all sequences that are defined as equivalent to this sequence. This simultaneously removes the need for alignments and addresses variable-length sequences. Let V denote the set of tokens that the model classifies at a single time-step of its output (vocabulary), and the blank-augmented vocabulary Ṽ = V ∪ { } 

3.4. LIPNET ARCHITECTURE:

Figure 1 illustrates the LipNet architecture, which starts with 3×(spatiotemporal convolutions, channel-wise dropout, spatial max-pooling). Subsequently, the features extracted are followed by two Bi-GRUs. The Bi-GRUs are crucial for efficient further aggregation of the STCNN output. Finally, a linear transformation is applied at each time-step, followed by a softmax over the vocabulary augmented with the CTC blank, and then the CTC loss. All layers use rectified linear unit (ReLU) activation functions. More details including hyperparameters can be found in Table 3 of Appendix A.

4. LIPREADING EVALUATION:

In this section, we evaluate LipNet on the GRID corpus. The augmentation methods employed don't make use of external data and rely purely on the GRID corpus.

4.1. DATA AUGMENTATION:

Preprocessing: The GRID corpus consists of 34 subjects, each narrating 1000 sentences. The videos for speaker 21 are missing, and a few others are empty or corrupt, leaving 32746 usable videos. We employ a split (unseen speakers; not previously used in the literature) holding out the data of two male speakers (1 and 2) and two female speakers (20 and 22) for evaluation (3971 videos). The remainder is used for training (28775 videos). We also use a sentence-level variant of the split (overlapped speakers) similar to Wand et al. (2016) , where 255 random sentences from each speaker are used for evaluation. All remaining data from all speakers is pooled together for training. All videos are 3 seconds long with a frame rate of 25fps. The videos were processed with the DLib face detector, and the iBug face landmark predictor (Sagonas et al., 2013) with 68 landmarks coupled with an online Kalman Filter. Using these landmarks, we apply an affine transformation to extract a mouth-centred crop of size 100 × 50 pixels per frame. We standardise the RGB channels over the whole training set to have zero mean and unit variance.

Augmentation: We augment the dataset with simple transformations to reduce overfitting. First, we train on both the regular and the horizontally mirrored image sequence. Second, since the dataset provides word start and end timings for each sentence video, we augment the sentence-level training data with video clips of individual words as additional training instances. These instances have a decay rate of 0.925. Third, to encourage resilience to varying motion speeds by deletion and duplication of frames, this is performed with a per-frame probability of 0.05. The same augmentation methods were followed in all proposed baselines and models.

4.2. BASELINES:

To evaluate LipNet, we compare its performance to that of three hearing-impaired people who can lipread, as well as three ablation models inspired by recent state-of-the-art work (Chung & Zisserman, 2016a; Wand et al., 2016) .

Hearing-Impaired People: This baseline was performed by three members of the Oxford Students' Disability Community. After being introduced to the grammar of the GRID corpus, they observed 10 minutes of annotated videos from the training dataset, then annotated 300 random videos from the evaluation dataset. When uncertain, they were asked to pick the most probable answer.

Baseline-LSTM: Using the sentence-level training setup of LipNet, we replicate the model architecture of the previous deep learning GRID corpus state-of-the-art (Wand et al., 2016) . See Appendix A for more implementation details.

Baseline-2D: Based on the LipNet architecture, we replace the STCNN with spatial-only convolutions similar to those of Chung & Zisserman (2016a) . Notably, contrary to the results we observe with LipNet, Chung & Zisserman (2016a) report 14% and 31% poorer performance of their STC-NNs compared to the 2D architectures in their two datasets.

Baseline-NoLM: Identical to LipNet, but with the language model used in beam search disabled.

4.3. PERFORMANCE EVALUATION:

To measure the performance of LipNet and the baselines, we compute the word error rate (WER) and the character error rate (CER), standard metrics for the performance of ASR models. We produce approximate maximum-probability predictions from LipNet by performing CTC beam search. WER (or CER) is defined as the minimum number of word (or character) insertions, substitutions, and deletions required to transform the prediction into the ground truth, divided by the number of words (or characters) in the ground truth. Note that WER is usually equal to classification error when the predicted sentence has the same number of words as the ground truth, particularly in our case since almost all errors are substitution errors.

Table 2 summarises the performance of LipNet compared to the baselines. According to the literature, the accuracy of human lipreaders is around 20% (Easton & Basala, 1982; Hilder et al., 2009) .

As expected, the fixed sentence structure and the limited subset of words for each position in the GRID corpus facilitate the use of context, increasing performance. On the unseen speakers split, the three hearing-impaired people achieve 57.3%, 50.4%, and 35.5% WER respectively, yielding an average of 47.7% WER. For both unseen and overlapped speakers evaluation, the highest performance is achieved by the architectures enhanced with convolutional stacks. LipNet exhibits a 2.3× higher performance in the overlapped compared to the unseen speakers split. For unseen speakers, Baseline-2D and LipNet achieve 1.8× and 4.2× lower WER, respectively, than hearing-impaired people.

The WER for unseen speakers Baseline-2D is 26.7%, whereas for LipNet it is 2.3× lower, at 11.4%.

Similarly, the error rate for overlapped speakers was 2.4× lower for LipNet compared to Baseline-2D. Both results demonstrate the importance of combining STCNNs with RNNs. This performance difference confirms the intuition that extracting spatiotemporal features using a STCNN is better than aggregating spatial-only features. This observation contrasts with the empirical observations of Chung & Zisserman (2016a) . Furthermore, LipNet's use of STCNN, RNNs, and CTC cleanly allow processing both variable-length input and variable-length output sequences, whereas the architectures of Chung & Zisserman (2016a) and Chung & Zisserman (2016b) only handle the former.

Baseline-LSTM exhibits the lowest performance, in both unseen and overlapped speakers, with 52.8% and 26.3% WER, respectively. Interestingly, although Baseline-LSTM replicates the architecture of Wand et al. (2016) , and despite the numerous data augmentation methods, the model performs 1.3× lower than the reported 79.6% word-level accuracy illustrating the difficulty of a sentence-level task even in a restricted grammar.

Finally, by disabling the language model, the Baseline-NoLM exhibits approximately 1.2× higher WER than our proposed model.

4.4. LEARNED REPRESENTATIONS:

In this section, we analyse the learned representations of LipNet from a phonological perspective. First, we create saliency visualisations (Simonyan et al., 2013; Zeiler & Fergus, 2014) to illustrate where LipNet has learned to attend. In particular, we feed an input into the model and greedily decode an output sequence, yielding a CTC alignment û ∈ Ṽ * (following the notation of Sections 3.2 and 3.3). Then, we compute the gradient of t p(û t |x) with respect to the input video frame sequence, but unlike Simonyan et al. (2013) , we use guided backpropagation (Springenberg et al., 2014) . Second, we train LipNet to predict ARPAbet phonemes, instead of characters, to analyse visual phoneme similarities using intra-viseme and inter-viseme confusion matrices. The production of the word please requires a great deal of articulatory movement at the beginning: the lips are pressed firmly together for the bilabial plosive /p/ (frame 1). At the same time, the blade of the tongue comes in contact with the alveolar ridge in anticipation of the following lateral /l/. The lips then part, allowing the compressed air to escape between the lips (frame 2). The jaw and lips then open further, seen in the distance between the midpoints of the upper and lower lips, and the lips spread (increasing the distance between the corners of the mouth), for the close vowel /iy/ (frame 3-4). Since this is a relatively steady-state vowel, lip position remains unchanged for the rest of its duration (frames 4-8), where the attention level drops considerably. The jaw and the lips then close slightly, as the blade of the tongue needs to be brought close to the alveolar ridge, for /z/ (frames 9-10), where attention resumes.

Lay is interesting since the bulk of frontally visible articulatory movement involves the blade of the tongue coming into contact with the alveolar ridge for /l/ (frames 2-6), and then going down for the vowel /ey/ (frames 7-9). That is exactly where most of LipNet's attention is focused, as there is little change in lip position.

4.4.2. VISEMES:

According to DeLand (1931) and Fisher (1968) , Alexander Graham Bell first hypothesised that multiple phonemes may be visually identical on a given speaker. This was later verified, giving rise to the concept of a viseme, a visual equivalent of a phoneme (Woodward & Barber, 1960; Fisher, 1968) . For our analysis, we use the phoneme-to-viseme mapping of Neti et al. (2000) , clustering the phonemes into the following categories: Lip-rounding based vowels (V), Alveolar-semivowels (A), Alveolar-fricatives (B), Alveolar (C), Palato-alveolar (D), Bilabial (E), Dental (F), Labio-dental (G), and Velar (H). The full mapping can be found in Table 4 in Appendix A. The GRID corpus contain 31 out of the 39 phonemes in ARPAbet. We compute confusion matrices between phonemes and then group phonemes into viseme clusters, following Neti et al. (2000) . Figure 3 shows the confusion matrices of the 3 most confused viseme categories, as well as the confusions between the viseme categories. The full phoneme confusion matrix is in Figure 4 in Appendix B.

Given that the speakers are British, the confusion between /aa/ and /ay/ (Figure 3a ) is most probably due to the fact that the first element, and the greater part, of the diphthong /ay/ is articulatorily identical with /aa/: an open back unrounded vowel (Ferragne & Pellegrino, 2010) . The confusion of /ih/ (a rather close vowel) and /ae/ (a very open vowel) is at first glance surprising, but in fact in the sample /ae/ occurs only in the word at, which is a function word normally pronounced with a reduced, weak vowel /ah/. /ah/ and /ih/ are the most frequent unstressed vowels and there is a good deal of variation within and between them, e.g. private and watches (Cruttenden, 2014) .

The confusion within the categories of bilabial stops /p b m/ and alveolar stops /t d n/ (Figures 3b-c ) is unsurprising: complete closure at the same place of articulation makes them look practically identical. The differences of velum action and vocal fold vibration are unobservable from the front.

Finally, the quality of the viseme categorisation of Neti et al. (2000) is confirmed by the fact that the matrix in Figure 3d is diagonal, with only minor confusion between alveolar (C) and palatoalveolar (D) visemes. Articulatorily, alveolar /s z/ and palato-alveolar /sh zh/ fricatives are distinguished by only a small difference in tongue position: against the palate just behind the alveolar ridge, which is not easily observed from the front. The same can be said about dental /th/ and alveolar /t/.

5. CONCLUSIONS:

We proposed LipNet, the first model to apply deep learning to end-to-end learning of a model that maps sequences of image frames of a speaker's mouth to entire sentences. The end-to-end model eliminates the need to segment videos into words before predicting a sentence. LipNet requires neither hand-engineered spatiotemporal visual features nor a separately-trained sequence model.

Our empirical evaluation illustrates the importance of spatiotemporal feature extraction and efficient temporal aggregation, confirming the intuition of Easton & Basala (1982) . Furthermore, LipNet greatly outperforms a human lipreading baseline, exhibiting 4.1× better performance, and 4.8% WER which is 2.8× lower than the word-level state-of-the-art (Gergen et al., 2016) in the GRID corpus.

While LipNet is already an empirical success, the deep speech recognition literature (Amodei et al., 2015) suggests that performance will only improve with more data. In future work, we hope to demonstrate this by applying LipNet to larger datasets, such as a sentence-level variant of that collected by Chung & Zisserman (2016a) .

Some applications, such as silent dictation, demand the use of video only. However, to extend the range of potential applications of LipNet, we aim to apply this approach to a jointly trained audiovisual speech recognition model, where visual input assists with robustness in noisy environments.

B PHONEMES AND VISEMES:

Table 4 shows the phoneme to viseme clustering of Neti et al. (2000) and Figure 4 shows LipNet's full phoneme confusion matrix. 

:

† These authors contributed equally to this work. 1 LipNet video: https://youtube.com/playlist?list=PLXkuFIFnXUAPIrXKgtIpctv2NuSo7xw3k

Figures and Tables:

Figure 1: LipNet architecture. A sequence of T frames is used as input, and is processed by 3 layers of STCNN, each followed by a spatial max-pooling layer. The features extracted are processed by 2 Bi-GRUs; each time-step of the GRU output is processed by a linear layer and a softmax. This end-to-end model is trained with CTC.where denotes the CTC blank symbol. Define the function B : Ṽ * → V * that, given a string over Ṽ , deletes adjacent duplicate characters and removes blank tokens. For a label sequence y ∈ V * , CTC defines p(y|x) = u∈B -1 (y) s.t. |u|=T p(u 1 , . . . , u T |x), where T is the number of time-steps in the sequence model. For example, if T = 3, CTC defines the probability of a string "am" as p(aam) + p(amm) + p( am) + p(a m) + p(am ). This sum is computed efficiently by dynamic programming, allowing us to perform maximum likelihood.

Figure 2: Saliency maps for the words (a) please and (b) lay, produced by backpropagation to the input, showing the places where LipNet has learned to attend. The pictured transcription is given by greedy CTC decoding. CTC blanks are denoted by ' '.

Figure 3: Intra-viseme and inter-viseme confusion matrices, depicting the three categories with the most confusions, as well as the confusions between viseme clusters. Colours are row-normalised to emphasise the errors.

Figure 4: LipNet's full phoneme confusion matrix.

Performance of LipNet on the GRID dataset compared to the baselines, measured on two splits: (a) evaluating on only unseen speakers, and (b) evaluating on a 255 video subset of each speakers' sentences.

Table content: <table><tr><td/><td colspan="4">Unseen Speakers Overlapped Speakers</td></tr><tr><td>Method</td><td>CER</td><td>WER</td><td>CER</td><td>WER</td></tr><tr><td>Hearing-Impaired Person (avg)</td><td>-</td><td>47.7%</td><td>-</td><td>-</td></tr><tr><td>Baseline-LSTM</td><td>38.4%</td><td colspan="2">52.8% 15.2%</td><td>26.3%</td></tr><tr><td>Baseline-2D</td><td>16.2%</td><td>26.7%</td><td>4.3%</td><td>11.6%</td></tr><tr><td>Baseline-NoLM</td><td>6.7%</td><td>13.6%</td><td>2.0%</td><td>5.6%</td></tr><tr><td>LipNet</td><td>6.4%</td><td>11.4%</td><td>1.9%</td><td>4.8%</td></tr></table>

Phoneme to viseme clustering ofNeti et al. (2000).

Table content: <table><tr><td>Code</td><td>Viseme Class</td><td>Phonemes in Cluster</td></tr><tr><td>V1</td><td/><td>/ao/ /ah/ /aa/ /er/ /oy/ /aw/ /hh/</td></tr><tr><td>V2</td><td colspan="2">Lip-rounding based vowels /uw/ /uh/ /ow/</td></tr><tr><td>V3</td><td/><td>/ae/ /eh/ /ey/ /ay/</td></tr><tr><td>V4</td><td/><td>/ih/ /iy/ /ax/</td></tr><tr><td>A</td><td>Alveolar-semivowels</td><td>/l/ /el/ /r/ /y/</td></tr><tr><td>B</td><td>Alveolar-fricatives</td><td>/s/ /z/</td></tr><tr><td>C</td><td>Alveolar</td><td>/t/ /d/ /n/ /en/</td></tr><tr><td>D</td><td>Palato-alveolar</td><td>/sh/ /zh/ /ch/ /jh/</td></tr><tr><td>E</td><td>Bilabial</td><td>/p/ /b/ /m/</td></tr><tr><td>F</td><td>Dental</td><td>/th/ /dh/</td></tr><tr><td>G</td><td>Labio-dental</td><td>/f/ /v/</td></tr><tr><td>H</td><td>Velar</td><td>/ng/ /k/ /g/ /w/</td></tr><tr><td>S</td><td>Silence</td><td>/sil/ /sp/</td></tr></table>

Peer review (iclr2017):
UPDATE: I have read the authors* responses. I did not read the social media comments about this paper prior to reviewing it. I appreciate the authors* updates in response to the reviewer comments. Overall, however, my review stands. The authors have taken a task that had not yet been addressed with a straightforward modern deep learning approach, and addressed it with such an approach. I assume that if we pick up any task that hasn*t been worked on for a while, and give it a solid deep learning treatment, we will do well. I do not see such papers as a contribution to ICLR, unless they also provide new insights, analysis, or surprising results (which, to my mind, this paper does not). This is a general point and the program chairs may disagree with it, of course. I have removed my recommendation that this be accepted as a workshop paper, as I have since noticed that the workshop track this year has a different focus. ************************ ORIGINAL REVIEW: The authors show that an appropriately engineered LSTM+CNN+CTC network does an excellent job of lipreading on the GRID corpus. This is a nice result to know about--yet another example of a really nice result that one can get the first time one applies such methods to an old task--and all of the work that went into getting it looks solid (and likely involved some significant engineering effort). However, this in itself is not sufficiently novel for publication at ICLR. The paper also needs to be revised to better represent prior work, and ideally remove some of the vague motivational language. Some specifics on what I think needs to be revised: - First, the claim of being the first to do sentence-level lipreading. As mentioned in a pre-review comment, this is not true. The paper should be revised to discuss the prior work on this task (even though much of it used data that is not public). Ideally the title should also be changed in light of this. - The comparison with human lipreaders needs to be qualified a bit. This task is presumably very unnatural for humans because of the unusual grammar, so perhaps what you are showing is that a machine can better take into account the strong contraints. This is great, but not a general statement about LipNet vs. humans. - The paper contains some unnecessary motivational platitudes. We do not need to invoke Easton and Basala 1982 to motivate modeling context in a linguistic sequence prediction task, and prior work using older sequence models (e.g. HMMs) for lipreading has modeled context as well. The McGurk effect does not show that lipreading plays a crucial role in human communication. - It is worth noting that even without the spatial convolution, your Baseline-2D already does extremely well. So I am not sure about the *importance of spatiotemporal feature extraction* as stated in the conclusion. Some more minor comments, typos, etc.: - citations for LSTMs, CTC, etc. should be provided the first time they are mentioned. - I did not quite follow the justification for upsampling. - what is meant by *lip-rounding vowels*? They seem to include almost all English vowels. - Did you consider keeping the vowel visemes V1-V4 separate rather than collapsing them into one? Since you list Neti et al.*s full viseme set, it is worth mentioning why you modified it. - *Given that the speakers are British, the confusion between /aa/ and /ay/...* -- I am not sure what this has to do with British speakers, as the relationship between these vowels exists in other English dialects as well (e.g. American). - The discussion about confusions within bilabial stops and within alveolar stops is a bit mismatched with the actual confusion data in Fig. 3(b,c). For example, there does not seem to be any confusion between /m/ and /b/ or between /m/ and /p/. - *lipreading actuations*: I am not sure what *actuations* means in this context - *palato-alvealoar* --> *palato-alveolar* - *Articulatorily alveolar* --> *Alveolar*?