Patent Description:
Visual speech recognition refers to processing a sequence of video frames that depict the lips of a person to predict a sequence of words being spoken by the person based on the movement of the lips in the video frames.

Visual speech recognition can be performed by machine learning models, e.g., neural network models. <NPL> (which refers to <NPL>) describes a lip reading neural network that outputs at the character level, is able to learn a language model, and has a dual attention mechanism that can operate over visual input only, audio input only, or both. <NPL>, describes a lip reading neural network system that encodes input video frames using a stacked 3D convolutional neural network, highway network, and bidirectional GRU (Gated Recurrent Unit) network, to generate an output comprising a sequence of characters.

This specification describes a system implemented as computer programs on one or more computers in one or more locations that performs visual speech recognition.

The invention is set out in claim <NUM>; further aspects are defined in the dependent claims.

Implementations of the system/method can provide significantly improved accuracy of visual speech recognition, and are readily adaptable to different applications without retraining. Further advantages are described later. An output of the method/system may comprises data defining the sequence of words, e.g. text data, and/or the words may be converted to audible speech, e.g. using a text-to-speech system to provide speech data.

In some implementations the volumetric convolutional neural network layers include a plurality of three-dimensional (convolutional) filters i.e. filters with a kernels operating over two spatial dimensions and a time dimension. This can help to capture the spatiotemporal relationships present when lips form a phoneme. In some implementations the visual speech recognition neural network includes at least five volumetric convolutional neural network layers.

In some implementations the time-aggregation neural network layers comprise one or more recurrent neural network layers, in particular one or more long short-term memory neural network layers, more particularly bi-directional long short-term memory neural network layers. These can work with the 3D convolutional layers in aggregating over longer time periods.

In some implementations the visual speech recognition neural network comprises one or more group normalization layers, e.g. interleaved between the volumetric convolutional neural network layers and/or the time-aggregation neural network layers. Such a group normalization layer may normalize over a group of (filter) channels. This can facilitate distributing the computation across multiple processing units by reducing communication between the units compared with e.g. batch normalization, and can also help to provide more stable learning during training.

As previously described, in some implementations determining the utterance or sequence of words expressed by the pair of lips depicted in the video using the output scores comprises processing the outputs scores using a decoder to generate the sequence of words. The decoder may comprise a so-called finite state transducer (FST). In implementations the decoder may perform operations comprising removing duplicate phonemes and blanks and/or mapping phonemes to words, in particular using an n-gram language model e.g. with backoff such as Katz's backoff.

There is also described a method of training a visual speech recognition neural network as described above, i.e. comprising one or more volumetric convolutional neural network layers and one or more time-aggregation neural network layers. The method comprises generating training data comprising a plurality of training examples, each training example comprising: (i) a training video comprising a plurality of training video frames, and (ii) a sequence of phonemes from a vocabulary of possible phonemes. The generating comprises, for each training video: obtaining a raw video comprising a plurality of raw video frames and corresponding audio data, determining the sequence of phonemes from the vocabulary of possible phonemes using the audio data, and determining each training video frame based on (i.e. so that it represents) a face depicted in a respective raw video frame. The method also comprises training the visual speech recognition neural network on the generated training data by determining trained values of visual speech recognition neural network parameters from initial values of visual speech recognition neural network parameters.

Such a method can generate large quantities of training data by an automated process and then use the generated training data to improve the performance of the visual speech recognition neural network. For example the training data may be generated from publically available videos such as YouTube videos.

In implementations, determining the sequence of phonemes from the vocabulary of possible phonemes using the audio data may comprise obtaining a transcript of the raw video, determining an alignment of the transcript and the audio data using a trained automatic speech recognition algorithm, and determining the sequence of phonemes from the aligned transcript. The method may further comprise determining the transcript is expressed in a specific natural language and/or determining that a quality measure of the raw video exceeds a minimum threshold.

The method of training may employ video processing steps as described below; some or all of these steps may also be applied to the video received by the previously described method/system for visual speech recognition.

Thus in implementations determining a training video frame based on a face depicted in a respective raw video frame may comprises detecting the face in the raw video frame; determining a plurality of landmarks on the face; determining a canonical (i.e. standardized) view of the face using the landmarks; and cropping a region depicting a pair of lips from the canonical view of the face. The method may further comprise smoothing, e.g. filtering/averaging over time and/or space, positions of the plurality of landmarks on the face. This can help to achieve good performance.

The video processing may further comprise determining that an orientation of the face is within an acceptable range of possible orientations using the landmarks on the face and/or determining that the lips are expressing an utterance based on i.e. dependent upon a variation in i.e. dispersion of, a measure of mouth openness between e.g. raw video frames.

Implementations may further comprise determining that the raw video and the corresponding audio data are aligned, e.g. by generating an embedding of the audio data by processing the audio data using an audio data embedding neural network, generating an embedding of the raw video by processing the raw video using a video embedding neural network, and determining that a similarity measure between the embedding of the audio data and the embedding of the raw video exceeds a threshold. The video embedding neural network may include one or more volumetric convolutional neural network layers.

In implementations training the visual speech recognition neural network on the generated training data may comprise, for each of a plurality of training examples: processing the training video included in the training example using the visual speech recognition neural network in accordance with current values of visual speech recognition neural network parameters to generate, for each output position in an output sequence, a respective output score for each phonemes in the vocabulary of possible phonemes; determining a gradient of a connectionist temporal classification (CTC) loss function based on the output scores and the sequence of phonemes from the vocabulary of possible phonemes included in the training example; and adjusting the current values of the visual speech recognition neural network parameters using the gradient. The CTC loss function may measure the likelihood of the sequence of phonemes based on a sequence of phoneme distributions without requiring the alignment of the sequence of phonemes and the sequence of phoneme distributions.

The system described in this specification can perform visual speech recognition (VSR) more accurately than some professional lip readers and some conventional systems. For example, the system described in this specification can achieve a word error rate of <NUM>% when performing VSR on a set of video frames. In comparison, professional lip readers can achieve a word error rate of <NUM>% on the same dataset, and some conventional VSR systems achieve word error rates of greater than <NUM>%, on the same set of video frames.

The system described in this specification decouples phoneme prediction and word decoding by using a VSR neural network to predict phonemes, and then using word decoding to predict the words corresponding to the phonemes. In contrast, some conventional systems use a VSR neural network to directly predict characters or words (i.e., without the intermediate word decoding step). Decoupling phoneme prediction and word decoding enables the system described in this specification to realize a number of advantages.

For example, decoupling phoneme prediction and word decoding enables the vocabulary of words that can be predicted by the system to be arbitrarily extended (i.e., scaled) or reduced without retraining the VSR neural network. This may be achieved, for example, by changing a vocabulary or language model used by the decoder.

As another example, decoupling phoneme prediction and word decoding can reduce the uncertainty that the VSR neural network has to model, thereby enabling the VSR neural network to be trained more effectively, e.g., to achieve a higher prediction accuracy over fewer training iterations. More specifically, uncertainty in VSR can originate from two main sources: uncertainty in the sounds that correspond to lip movements (e.g., due to similar lip movements corresponding to different sounds), and uncertainty in the words that correspond to these sounds (e.g., due to the same sounds corresponding to different words). Uncertainty in the words that correspond to sounds can be illustrated by the words "fair" and "fare": these are different words that are pronounced in the same way. By decoupling phoneme prediction and word decoding, the VSR neural network only has to model the uncertainty in the sounds that correspond to lip movements, while the uncertainty in the words that correspond to the sounds is modeled by a separate word decoding procedure.

The system described in this specification can be used to assist people who are able to move their lips but have difficulty in clearly pronouncing words, e.g., following surgery or injury. In one example, the system can be used to process video frames that depict the lips of a person with a speech impairment to generate a sequence of words corresponding to the lip movements of the person. The sequence of words can thereafter be verbalized by another automated system. In this manner, the system described in this specification can be used to enable people having speech impairments to communicate or communicate more clearly. The system may also be used by people who are paralyzed but nonetheless able to move their lips, to enable them to speak.

The system described in this specification can also be used by a user device (e.g., a mobile phone, computer, or digital assistant) to enable a user to interact with the device by silently mouthing commands specifying operations to be performed by the user device. A user may wish to interact with a device by silently mouthing commands if the user is in a quiet environment, e.g., a library.

It will be appreciated that there are many other applications, e.g. for deaf people and noisy environments.

<FIG> shows an example visual speech recognition system <NUM>. The visual speech recognition system <NUM> is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

The visual speech recognition system <NUM> is configured to process a sequence of multiple "raw" video frames <NUM> that depict the lips of a person to generate data defining a sequence of words <NUM> being spoken by the person. The system <NUM> can be understood to perform lip reading by identifying words being spoken by the person based on the movement of the lips of the person in the raw video frames <NUM>.

In addition to depicting the lips of the person, each raw video frame <NUM> may additionally depict other content in a scene, e.g., the whole face of the person, some or all of the body of the person, the environment (e.g., room) where the person is located, and one or more other people.

The raw video frames <NUM> can be represented in any appropriate format. For example, each raw video frame <NUM> can be represented by a respective set of one or more "channels", where each channel is represented by a two-dimensional array of numerical values (e.g., floating point values). In one example, the raw video frames <NUM> may be black-and-white video frames, and each raw video frame <NUM> may be represented by a respective grayscale intensity channel. In another example, the raw video frames <NUM> may be color video frames, and each raw video frame <NUM> may be represented by a respective set of red, green, and blue (i.e., RGB) color channels.

The system <NUM> may include: (i) a video processing engine <NUM>, (ii) a visual speech recognition neural network <NUM> (referred to in this specification as a "VSR network"), and (iii) a decoding engine <NUM>.

The video processing engine <NUM> may be configured to process the raw video frames <NUM> to generate a corresponding sequence of "lip" video frames <NUM> that depict the lips of the person from a canonical (i.e., standardized) point of view.

To generate a lip video frame <NUM> from a raw video frame <NUM>, the video processing engine <NUM> determines the position of the face of the person in the raw video frame <NUM>. For example, the video processing engine <NUM> may determine the position of the face by processing the raw video frame <NUM> using a face detection neural network to generate data defining a bounding box enclosing the face of the person in the raw video frame <NUM>.

After determining the position of the face in the raw video frame <NUM>, the video processing engine <NUM> determines the positions of facial landmarks on the face. The facial landmarks may include one or more of, e.g., the leftmost- and rightmost- points of each of the eyes and mouth, the topmost- and bottommost- points of each of the eyes and the mouth, and the tip of the nose. The video processing engine <NUM> may determine the positions of the facial landmarks, e.g., by processing the portion of the raw video frame <NUM> depicting the face using a landmark detection model (e.g., neural network) to generate data defining the positions of the facial landmarks.

Optionally, the video processing engine <NUM> can smooth the positions of the facial landmarks in each raw video frame based on the positions of the corresponding facial landmarks in the other raw video frames, e.g., using a temporal Gaussian kernel.

After determining the positions of the facial landmarks, the video processing engine <NUM> can determine the parameter values of an "alignment" transformation that, when applied to the raw video frame, causes the face in the raw video frame to be depicted from a canonical point of view. More specifically, the video processing engine <NUM> can determine an alignment transformation that, when applied to the raw video frame, approximately (or exactly) aligns the positions of the facial landmarks in the raw video frame with a set of reference facial landmark positions. The alignment transformation may be, e.g., an affine transformation or an elastic transformation. The video processing engine <NUM> can determine the parameter values of the alignment transformation using any appropriate optimization procedure, for example, a gradient descent optimization procedure.

The video processing engine <NUM> can generate the lip video frame <NUM> corresponding to the raw video frame <NUM> by applying the alignment transformation to the raw video frame, and then cropping a portion of the raw video frame that depicts at least the lips of the person. The video processing engine <NUM> can identify the position of the lips of the person in the raw video frame, e.g., based on the positions of the facial landmarks corresponding to the lips. In addition to depicting the lips of the person, the lip video frame <NUM> may depict, e.g., a portion of the face of the person around the lips, or the entire face of the person.

The VSR network <NUM> is configured to process the sequence of lip video frames <NUM> to generate a sequence of "phoneme distributions" <NUM>. The sequence of phoneme distributions <NUM> includes a respective phoneme distribution corresponding to each lip video frame <NUM>. Each phoneme distribution defines a respective score for each phoneme in a vocabulary of possible phonemes. For example, each phoneme distribution may include a respective score corresponding to each phoneme in the vocabulary of possible phonemes.

The score for a given phoneme that is defined by a phoneme distribution corresponding to a lip video frame <NUM> characterizes a likelihood that the lip video frame <NUM> captures the person pronouncing the given phoneme. For example, the score may be a probability value (i.e., a numerical value between <NUM> and <NUM>) that defines a probability that the lip video frame <NUM> captures the person pronouncing the given phoneme.

The vocabulary of possible phonemes includes a predetermined number (e.g., <NUM>) of possible phonemes. The possible phonemes may include a "blank" symbol that corresponds to silence (e.g., in a pause in speech).

An example architecture of the VSR network <NUM> is described in more detail with reference to <FIG> and <FIG>.

The decoding engine <NUM> is configured to process the sequence of phoneme distributions <NUM> to generate data defining the sequence of words <NUM> corresponding to the movement of the lips depicted in the lip video frames <NUM>. The decoding engine <NUM> can use any of a variety of decoding techniques to generate the sequence of words <NUM> from the sequence of phoneme distributions <NUM>. An example decoding engine <NUM> is described in more detail with reference to <FIG>.

<FIG> is an illustration <NUM> of an example architecture of a VSR network and a decoding engine, e.g., the VSR network <NUM> and the decoding engine <NUM> described with reference to <FIG>.

The VSR network illustrated by <FIG> has a three-dimensional (3D) convolutional module <NUM> for extracting spatiotemporal features from the lip video frames <NUM> and a temporal module <NUM> for aggregating the spatiotemporal features over time to generate the sequence of phoneme distributions <NUM>. Spatiotemporal features refer to features that integrate "spatial information" from within lip video frames and "temporal" information from across lip video frames.

The 3D convolutional module <NUM> has a sequence of five volumetric convolutional neural network layers ("spatiotemporal convolution layers") that each generate respective spatiotemporal features. A volumetric convolutional layer refers to a convolutional layer having three-dimensional (3D) convolutional filters. The 3D convolutional filters cause the neurons of the volumetric convolutional neural network layers to have receptive fields that extend both within and between lip video frames. Therefore, 3D convolutional filters enable volumetric convolutional neural network layers to generate spatiotemporal features which can characterize both lip shape within video frames and lip movement across video frames. The receptive field of the volumetric convolutional neural network layers may more than the typical duration of a phoneme, e.g. around twice this.

In addition to the volumetric convolutional neural network layers, the 3D convolutional module <NUM> includes a respective pooling layer and group normalization layer after each of the volumetric convolutional layers. A group normalization layer divides its input into groups of channels and computes normalization statistics within these groups (i.e., rather than along the batch dimension, as in batch normalization). The normalization statistics may comprise a mean µ and/or standard deviation σ or variance σ<NUM>; normalization may comprise computing x̂ = x - µ or <MAT>.

The VSR network provides the output of the 3D convolutional module <NUM>, which includes a respective spatio-temporal feature vector corresponding to each of the lip video frames <NUM>, to the temporal module <NUM>. The temporal module <NUM> has a sequence of three bi-directional long short-term memory (LSTM) neural network layers and a multi-layer perceptron (MLP).

The first bi-directional LSTM layer processes the sequence of spatio-temporal feature vectors output by the 3D convolutional module <NUM> to generate a respective aggregated feature vector corresponding to each of the video frames <NUM>. The second and third bi-directional LSTM layers process the sequence of aggregated feature vectors generated by the first and second bi-directional LSTM layers respectively.

The MLP includes two fully-connected layers and a soft-max layer that are configured to process the aggregated feature vectors generated by the third bi-directional LSTM layer to generate an output that defines the sequence of phoneme distributions <NUM>. The sequence of phoneme distributions includes a respective phoneme distribution corresponding to each lip video frame <NUM>.

In addition to the bi-directional LSTM layers and the MLP, the temporal module <NUM> includes a respective group normalization layer after each of the bi-directional LSTM layers and the MLP.

The decoding engine <NUM> processes the sequence of phoneme distributions <NUM> to generate the sequence of words <NUM> corresponding to the movement of the lips in the lip video frames <NUM>. In the example illustrated by <FIG>, the decoding engine <NUM> includes a collapse engine <NUM> (sometimes called a collapse finite state transducer), a lexicon engine <NUM> (sometimes called a lexicon finite state transducer), and a language model engine <NUM> (sometimes called a language model finite state transducer).

The collapse engine <NUM> is configured to process a sequence of phonemes to remove duplicate phonemes and blank symbols.

The lexicon engine <NUM> is configured to map the processed sequence of phonemes generated by the collapse engine <NUM> to a corresponding sequence of words (e.g., using a predetermined mapping from phonemes to words).

The language model engine <NUM> is configured to process the sequence of words generated by the lexicon engine <NUM> to determine a likelihood that the sequence of words represents a valid phrase in a given natural language (e.g., English). In one example, the language model engine <NUM> may implement an n-gram language model with Katz backoff.

The decoding engine <NUM> uses the collapse engine <NUM>, the lexicon engine <NUM>, and the language model engine <NUM> to perform a search (e.g., a beam search) over possible sequences of words corresponding to the movement of the lips in the lip video frames <NUM>. The decoding engine <NUM> can perform the search by determining, for each of multiple possible sequences of words, a likelihood that the sequence of words corresponds to the movement of the lips in the video frames <NUM>. The likelihood of each sequence of words can be determined based on (e.g., the product of): (i) the likelihood of the sequence of phonemes corresponding to the sequence of words according to the sequence of phoneme distributions <NUM>, and (ii) the likelihood of the sequence of words according to the language model engine <NUM>.

An example of the operations that can be performed by the decoding engine <NUM> is described in more detail with reference to, e.g., <NPL>), or <NPL>.

As one of example of the performance gains that can be achieved by performing visual speech recognition using the described systems, the VSR network and decoding engine described with reference to <FIG> can achieve a phoneme error rate of <NUM> ± <NUM>, a character error rate of <NUM> ± <NUM>, and a word error rate of <NUM> ± <NUM> on a particular test set of video frames. In contrast, a professional lip reader achieved a word error rate of <NUM> ± <NUM> on the same set of video frames.

<FIG> shows an example of: (i) the filter sizes and strides corresponding to the volumetric convolutional layers ("conv1" to "conv5") and pooling layers ("pooll" to "pool5") of the example VSR network illustrated by <FIG>, and (ii) the dimensionality of the output channels and the input of the volumetric convolutional layers, the pooling layers, the bi-directional LSTM layers ("bilstm6" to "bilstm8"), and the fully-connected layers ("fc9" to "fc10") of the example VSR network illustrated by <FIG>.

<FIG> shows an example of a data flow <NUM> for training the VSR network <NUM> described with reference to <FIG>. The VSR network <NUM> is trained on a set of training data composed of multiple training examples <NUM>. Each training example <NUM> includes: (i) a sequence of lip video frames <NUM> depicting the lips of a person (i. e, a training video that includes multiple training frames), and (ii) a sequence of phonemes <NUM> from a vocabulary of possible phonemes that corresponds to the movement of the lips depicted in the lip video frames <NUM>. Some or all of the same video processing steps may be used to pre-process raw video for use with a trained model (omitting the dashed components whose primary use is producing paired training data).

To generate the training examples <NUM>, a set of raw videos <NUM> and corresponding speech segments <NUM> (audio data) is obtained, e.g., extracted from videos available on a video-sharing website.

The segment length filter <NUM> identifies and removes videos with a duration that falls outside of a predetermined range (e.g., <NUM> second - <NUM> seconds).

The English language filter <NUM> identifies and removes videos with speech that is not expressed in English. To identify the language of the speech expressed in a video, the filter <NUM> can process a transcript of the speech (e.g., generated using automatic speech recognition techniques) using a language classifier. To train the VSR network <NUM> to perform visual speech recognition in a given language other than English, the filter <NUM> could identify and remove videos with speech that is not expressed in the given language.

The shot boundary detection filter <NUM> identifies and removes videos that include a shot transition, e.g., using a thresholding color histogram classifier. A shot transition in a video refers to a frame where the viewpoint of the video abruptly changes (e.g., when the video jumps from depicting the face of one person to depicting the face of another person).

The face detector / tracker <NUM> detects and tracks faces depicted in each remaining video that was not filtered by the segment length filter <NUM>, the English language filter <NUM>, or the shot boundary detection filter <NUM> (e.g., using a FaceNet neural network).

The clip quality filter <NUM> identifies and removes videos that fail to satisfy one or more quality criteria (i.e., where a quality measure of the raw video is below a minimum threshold). For example, the clip quality filter <NUM> may identify and remove videos that are blurry or shaky, videos where an eye-to-eye width of the depicted face is less than a predetermined number of pixels (i.e., where the face is too small), or videos with a frame rate that is less than a predetermined minimum frame rate.

The face landmark smoothing engine <NUM> processes each video to identify multiple facial landmarks in the face depicted in each video frame of the video and smooths the resulting facial landmark positions using a temporal Gaussian kernel (e.g., as described with reference to <FIG>).

Variations in the orientation (e.g., yaw and pitch) of the face in each video can be determined from the facial landmarks, and videos where variations in the orientation of the face exceed an acceptable range (e.g., ±<NUM>°) may be identified and removed.

The view canonicalization engine <NUM> processes each video frame of each video to determine a canonical (i.e., standardized) view of the face depicted in the video frame using the facial landmarks. For example, the view canonicalization engine <NUM> may apply a respective affine transformation to each video frame that approximately (or exactly) aligns the positions of the facial landmarks in the video frame with a set of reference facial landmark positions (e.g., as described with reference to <FIG>). After determining the canonical view of the face depicted in each video frame (e.g., by applying the affine transformation to the video frame), the view canonicalization engine <NUM> crops a region depicting the lips from the video frame (i.e., to generate lip video frames).

The speaking filter <NUM> identifies and removes videos where the face depicted in the video is not speaking. To identify whether the face depicted in a video is speaking, the speaking filter <NUM> computes a measure of mouth openness in each video frame of the video and normalizes the mouth openness measure by the size of a bounding box around the face. The speaking filter then determines a measure of dispersion (e.g., standard deviation) of the mouth openness measures and identifies the face as speaking only if the measure of dispersion satisfies a predetermined threshold. The speaking filter <NUM> may determine the measure of mouth openness in a video frame to be the number of pixels separating the facial landmarks indicating the topmost- and bottommost- points of the mouth.

The speaking classifier engine <NUM> identifies and removes videos where the speech segment <NUM> is not aligned (i.e., synchronized) with the video frames. To identify videos where the speech segment is not aligned with the video frames, the speaking classifier engine <NUM> can generate an embedding of the video frames and an embedding of the corresponding speech segment. The speaking classifier engine <NUM> can generate the embedding of the video frames by processing the video frames using a video embedding neural network (e.g., that can include or more volumetric convolutional layers). The speaking classifier engine <NUM> can generate the embeddings of the speech segment by processing a log mel-spectrogram representation of the speech segment using an audio embedding neural network. After generating the video embedding and the audio embedding, the speaking classifier engine <NUM> may identify the video frames as being aligned with the speech segment only if a similarity measure between the respective embeddings meets a predetermined threshold value. The similarity measure may be, e.g., a Euclidean similarity measure or a cosine similarity measure.

The video embedding neural network and the audio embedding neural network used by the speaking classifier engine <NUM> may be jointly trained to generate similar (i.e., according to a similarity measure) embeddings of a video and a speech segment if and only if the video and the speech segment are synchronized.

An "embedding" of a video or an audio segment refers to an ordered collection of numerical values (e.g., a vector or matrix of numerical values) representing the video or audio segment.

The sequence of phonemes <NUM> corresponding to a sequence of lip frames <NUM> can be determined in any of a variety of ways. In one example, an automatic speech recognition process can be used to determine an approximate transcript of the speech segment <NUM> corresponding to the lip frames <NUM>. The approximate transcript generated by the automatic speech recognition process can be used to align an actual transcript of the speech segment (e.g., generated by a person) with the video and the corresponding speech segment (audio data). Thereafter, the actual transcript can be mapped to a sequence of phonemes <NUM> (e.g., in accordance with a predetermined mapping from words to phonemes) having the same alignment with the video as the actual transcript.

A training engine <NUM> can use the training examples <NUM> to train the VSR network <NUM>, that is, to determine trained values of the model parameters <NUM> of the VSR network <NUM> from initial values of the model parameters <NUM> of the VSR network <NUM>.

The training engine <NUM> trains the VSR network <NUM> over multiple training iterations. At each training iteration, the training engine <NUM> samples a current "batch" (i.e., set) of multiple training examples.

For each training example in the current batch, the training engine <NUM> uses the VSR network <NUM> to process the sequences of lip frames included in the training example to generate a corresponding sequence of phoneme distributions. The training engine <NUM> then determines gradients <NUM> of an objective function <NUM>, and uses the gradients <NUM> to adjust the current values of the model parameters <NUM>. The objective function <NUM> depends on: (i) the sequence of phoneme distributions generated by the VSR network <NUM> for the sequence of lip frames in the training example, and (ii) the sequence of phonemes included in the training example.

The training engine <NUM> can determine the gradients of the objective function <NUM> using, e.g., backpropagation techniques. The training engine <NUM> can use the update rule of any of a variety of gradient descent optimization procedures (e.g., an RMSprop or Adam optimization procedure) to update the current values of the model parameters <NUM> using the gradients <NUM>.

The objective function <NUM> may be, e.g., a connectionist temporal classification (CTC) objective function (e.g. <NPL>). The CTC objective function measures the likelihood of the sequence of phonemes included in a training example according to the sequence of phoneme distributions generated by the VSR network <NUM> for the training example, without requiring the alignment of the sequence of phonemes and the sequence of phoneme distributions.

The objective function <NUM> may also be, e.g., a neural transducer loss objective function (e.g., as described in section <NUM> of <NPL>)).

The training engine <NUM> can continue training the VSR network <NUM> until a training termination criterion is satisfied, e.g., until a predetermined number of training iterations have been performed, or until an accuracy of the VSR network <NUM> achieves a predetermined threshold.

<FIG> illustrates the results of processing an example sequence of lip frames <NUM> using a VSR network with the architecture described with reference to <FIG>. <NUM> illustrates the lip frames <NUM> being overlaid with a saliency map, where pixels with a lighter color are determined to be more important to the prediction generated by the VSR network than pixels with a color that is less light. For each lip frame, "top-<NUM>" <NUM>, "top-<NUM>" <NUM>, and "top-<NUM>" <NUM> indicate the three phonemes associated with the highest scores by the phoneme distribution corresponding to the lip frame, and "entropy" <NUM> indicates the entropy of the phoneme distribution corresponding to the lip frame (where the magnitude of the entropy is illustrated by the length of a corresponding bar).

<FIG> is a flow diagram of an example process <NUM> for performing visual speech recognition. For convenience, the process <NUM> will be described as being performed by a system of one or more computers located in one or more locations. For example, a visual speech recognition system, e.g., the visual speech recognition system <NUM> of <FIG>, appropriately programmed in accordance with this specification, can perform the process <NUM>.

The system receives a video having multiple video frames, where each video frame depicts a pair of lips of a person (<NUM>). The system may process each video frame by applying an alignment transformation to the video frame that causes the lips and face in the video frame to be depicted from a canonical (i.e., standardized) point of view, and then cropping a region around the lips from the video frame.

The system processes the video using the VSR network to generate, for each output position in an output sequence, a respective output score for each token in a vocabulary of possible tokens (<NUM>). Each position in the output sequence corresponds to a respective video frame. The output scores for each token in the vocabulary of possible tokens at an output position can be referred to as a "token distribution". The tokens may be, e.g., phonemes (as described earlier), characters, word pieces, or whole words. The score for a given token at a given output position in the output sequence characterizes a likelihood that the video frame corresponding to the given output position captures the person pronouncing the given token. The VSR network includes one or more volumetric convolutional neural network layers (i.e., that each have 3D convolutional filters) and one or more "time-aggregation" neural network layers (e.g., recurrent layers). In some implementations, the VSR network includes at least five volumetric convolutional layers.

The system determines a sequence of words spoken by the pair of lips depicted in the video using the scores (i.e., token distributions) generated by the VSR network (<NUM>). For example, the system can process the token distributions generated by the VSR network using a decoder, e.g., the decoding engine described with reference to <FIG>.

A device implementing a system as described herein, e.g. a mobile phone, may require biometric authentication to unlock the device before use. In some implementations, e.g. where the VSR network is running on a server, the system may apply face-based authentication to the received video for further access control.

Claim 1:
A method for visual speech recognition to predict a sequence of words being spoken by a person based on movement of a pair of lips of the person in a sequence of video frames, the method comprising:
receiving a video comprising the sequence of video frames (<NUM>), wherein each video frame depicts the pair of lips of the person;
processing the sequence of video frames using a trained visual speech recognition neural network (<NUM>) in accordance with current values of visual speech recognition neural network parameters to generate, for each output position in an output sequence, a phoneme distribution that defines a respective output score for each phoneme in a vocabulary of possible phonemes, to thereby generate a sequence of phoneme distributions that includes a respective phoneme distribution corresponding to each video frame
wherein the visual speech recognition neural network comprises one or more volumetric convolutional neural network layers (<NUM>) and one or more time-aggregation neural network layers (<NUM>), wherein the volumetric convolutional neural network layers include a plurality of three-dimensional filters, and wherein the time-aggregation neural network layers comprise one or more recurrent neural network layers; and
determining the sequence of words (<NUM>) expressed by the pair of lips depicted in the video using the output scores;
wherein determining the sequence of words comprises providing the sequence of phoneme distributions to a word decoder (<NUM>) to process the sequence of phoneme distributions to generate data defining the sequence of words depicted in the video frames.