Patent Description:
Automatic speech recognition (ASR) is a technology typically used in mobile devices and other devices. In general, automatic speech recognition attempts to provide accurate transcriptions of what a person has said. In noisy environments, or otherwise when audio quality of a recorded utterance is poor, obtaining accurate ASR results can be a difficult task. When video data of a speaker is available, the video data can be leveraged to help improve ASR results. For instance, the video data of the speaker may provide motion of the lips while the speaker is speaking an utterance, which in turn, can be combined with the audio data of the utterance to assist in processing an ASR result.

Patent documents <CIT> and <CIT>, and scientific publication by <NPL>, disclose prior art methods to improve ASR results.

One aspect of the disclosure provides a method for rescoring automatic speech recognition (ASR) hypotheses using audio-visual matching. The method includes receiving, at data processing hardware, audio data corresponding to an utterance spoken by a user and video data representing motion of lips of the user while the user was speaking the utterance. The method also includes obtaining, by the data processing hardware, multiple candidate transcriptions for the utterance based on the audio data. For each candidate transcription of the multiple candidate transcriptions for the utterance, the method includes generating, by the data processing hardware, a synthesized speech representation of the corresponding candidate transcription; and determining, by the data processing hardware, an agreement score indicating a likelihood that the synthesized speech representation of the corresponding candidate transcription matches the motion of the lips of the user while the user speaks the utterance. The method also includes selecting, by the data processing hardware, one of the multiple candidate transcriptions for the utterance as a speech recognition output based on the agreement scores determined for the multiple candidate transcriptions for the utterance.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, determining the agreement score includes providing, to an agreement score model, the synthesized speech representation of the corresponding candidate transcription and the video data representing the motion of the lips of the user as feature inputs and determining, from the agreement score model, as a feature output, the agreement score based on a degree that the synthesized speech representation of the corresponding candidate transcription matches with the motion of the lips of the user. In these implementations, the agreement score model is trained on a plurality of training examples including positive training examples and negative training examples. The positive training examples include audio data representing utterances of speech and video data representing motion of lips of speakers that match the utterances of speech, while the negative training examples include audio data representing utterances of speech and video data representing motion of lips of speakers that do not match the utterances of speech.

In some examples, selecting one of the multiple candidate transcriptions for the utterance as the speech recognition output includes selecting, from among the multiple candidate transcriptions for the utterance, the candidate transcription associated with the highest agreement score as the speech recognition output for the utterance.

In some implementations, obtaining the multiple candidate transcriptions for the utterance includes generating, using a speech recognizer module, an initial set of candidate transcriptions for the utterance based on the audio data, each candidate transcription in the initial set of candidate transcriptions associated with a corresponding likelihood score indicating a likelihood that the candidate transcription is correct. The implementation further includes ranking the candidate transcriptions in the initial set of candidate transcriptions based on the likelihood scores, and determining the multiple candidate transcriptions for the utterance as the N-candidate transcriptions in the initial set of candidate transcriptions associated with the highest likelihood scores, the identified multiple candidate ranked according to the associated likelihood scores. In these implementations, the method may further include, prior to selecting one of the multiple transcriptions for the utterance, re-ranking, by the data processing hardware, the multiple candidate transcriptions for the utterance based on the agreement scores determined for the multiple candidate transcriptions for the utterance.

In some examples, obtaining the multiple candidate transcriptions for the utterance includes generating, using a speech recognizer module, an initial set of candidate transcriptions for the utterance based on the audio data, each candidate transcription in the initial set of candidate transcriptions associated with a corresponding likelihood score indicating a likelihood that the candidate transcription is correct. In these examples, the method further includes identifying two or more candidate transcriptions in the initial set of candidate transcriptions that are associated with likelihood scores that satisfy a likelihood threshold, and determining the multiple candidate transcriptions for the utterance as the identified two or more candidate transcriptions in the initial set of candidate transcriptions that are associated with likelihood scores that satisfy the likelihood threshold.

In some implementations, the multiple candidate transcriptions for the utterance are associated with the same language. In other examples, at least one of the multiple candidate transcriptions for the utterance is associated with a different language than the other multiple candidate transcriptions.

In some examples, receiving the audio data corresponding to the utterance spoken by the user includes receiving the audio data from a client device associated with the user, the client device in communication with one or more audio capture devices configured to capture the audio data corresponding to the utterance. In these examples, the data processing hardware resides on the client device. In other examples, client device is remote from the data processing hardware and communicates with the data processing hardware via a network.

In some implementations, receiving the video data representing the motion of the lips of the user while the user was speaking the utterance includes receiving the video data from the client device associated with the user. In these implementations, the client device includes one or more video capture devices configured to record the video data representing the motion of the lips of the user while the user was speaking the utterance.

Another aspect of the disclosure provides a system for rescoring (ASR) hypotheses using audio-visual matching. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that when executed by the data processing hardware cause the data processing hardware to perform operations that include receiving audio data corresponding to an utterance spoken by a user and video data representing motion of lips of the user while the user was speaking the utterance. The operations further include obtaining multiple candidate transcriptions for the utterance based on the audio data. For each candidate transcription of the multiple candidate transcriptions for the utterance, the operations include generating a synthesized speech representation of the corresponding candidate transcription and determining an agreement score indicating a likelihood that the synthesized speech representation of the corresponding candidate transcription matches the motion of the lips of the user while the user speaks the utterance. The operations further include selecting one of the multiple candidate transcriptions for the utterance as a speech recognition output based on the agreement scores determined for the multiple candidate transcriptions for the utterance.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, determining the agreement score includes providing, to an agreement score model, the synthesized speech representation of the corresponding candidate transcription and the video data representing the motion of the lips of the user as feature inputs and determining, from the agreement score model, as a feature output, the agreement score based on a degree that the synthesized speech representation of the corresponding candidate transcription matches the motion of the lips of the user. In these examples, the agreement score model is trained on a plurality of training examples including positive training examples and negative training examples. The positive training examples include audio data representing utterances of speech and video data representing motion of lips of speakers that match the utterances of speech; and the negative training examples include audio data representing utterances of speech and video data representing motion of lips of speakers that do not match the utterances of speech.

In some implementations, obtaining the multiple candidate transcriptions for the utterance includes generating, using a speech recognizer module, an initial set of candidate transcriptions for the utterance based on the audio data, each candidate transcription in the initial set of candidate transcriptions associated with a corresponding likelihood score indicating a likelihood that the candidate transcription is correct. In these implementations, the operations further include ranking the candidate transcriptions in the initial set of candidate transcriptions based on the likelihood scores, and determining the multiple candidate transcriptions for the utterance as the N-candidate transcriptions in the initial set of candidate transcriptions associated with the highest likelihood scores, the identified multiple candidate ranked according to the associated likelihood scores. The operations may further include, prior to selecting one of the multiple transcriptions for the utterance, re-ranking, by the data processing hardware, the multiple candidate transcriptions for the utterance based on the agreement scores determined for the multiple candidate transcriptions for the utterance.

In some examples, obtaining the multiple candidate transcriptions for the utterance includes generating, using a speech recognizer module, an initial set of candidate transcriptions for the utterance based on the audio data, each candidate transcription in the initial set of candidate transcriptions associated with a corresponding likelihood score indicating a likelihood that the candidate transcription is correct. In these examples, the operations further include identifying two or more candidate transcriptions in the initial set of candidate transcriptions that are associated with likelihood scores that satisfy a likelihood threshold, and determining the multiple candidate transcriptions for the utterance as the identified two or more candidate transcriptions in the initial set of candidate transcriptions that are associated with likelihood scores that satisfy the likelihood threshold.

The present disclosure provides a computer-implemented method which improves automated speech recognition (ASR) in relation to an utterance spoken by a user. The utterance may, for example, relate to a user speaking to a digital assistant on a user device such as a smart phone, smart speaker, or smart display. Audio data of the utterance is used to generate multiple candidate transcriptions (e.g., also referred to as "transcription hypotheses" or "ASR result hypotheses) for the utterance, thereby enabling generation of synthesized speech representations of the multiple candidate transcriptions (e.g. using a text-to-speech system). Video data of the user's face and/or lips whilst speaking the utterance may then be used to score or rank each of the synthesized speech representations based on how well each synthesized speech representation matches the video data (i.e. based on how well each synthesized speech representation matches the motion/movement of the user's face and/or lips in the video data). In this way, a speech recognition output may be selected based on the candidate transcription corresponding to the synthesized speech representation which best matches the video data.

One technical effect of this method (as compared to a method which relies solely on audio data) is to improve selection of the speech recognition output. In other words, the present methodology makes it more likely that the correct speech recognition output (i.e. an accurate transcription of the user utterance) will be selected. Effectively, the video data is used as an additional source of data to validate/verify/enhance the output of an audio-based automated speech recognition system. Thus, when video data of the user speaking the utterance is available, this video data may be used to determine which of the multiple candidate transcriptions is most likely to be correct, thereby improving the accuracy of the speech recognition system. The present methodology solves the technical problem of how to improve an audio-based automated speech recognition system. This is achieved here by using the video data to score or rank options produced using audio data only.

Another technical effect of the present methodology is improved identification of the language of an utterance. In particular, if the language of the utterance is unknown, the multiple candidate transcriptions may be generated in multiple languages. In this case, the language of the utterance may be identified based on the selected speech recognition output. Since the video data has been used to determine the best-matching synthesized speech representation, it is more likely that the associated candidate transcription will be in the correct language.

In some cases, it is envisaged that the audio data analysis is performed in the cloud (i.e. remote from the user device), with the subsequent video data matching done on the user device itself. One technical effect of this arrangement is a reduced bandwidth requirement since the video data may be retained on the user device without the need to transmit it to the remote cloud server. If the video data were to be transmitted to the cloud, it is likely that it would first need to be compressed for transmission. Therefore, another technical effect of performing the video matching on the user device itself is that the video data matching may be performed using the uncompressed (highest quality) video data. The use of uncompressed video data makes it easier to recognize matches/mismatches between the synthesized speech representations and the video data. Thus, improved scores/ranks are to be expected, thereby making it even more likely that the correct speech recognition output will be selected.

In some examples, it is envisaged that the degree of match between the synthesized speech representations and the video data be measured using a system (e.g. a deep neural network) trained on a large collection of audio-visual samples. In one example, the training examples/samples include both positive training examples/samples including audio data representing utterances of speech and video data representing motion of lips of speakers that match the utterances of speech, and negative training examples/samples including audio data representing utterances of speech and video data representing motion of lips of speakers that do not match the utterances of speech. Such training data ensures that the system is trained to recognize both matches and mismatches between the synthesized speech representations and the video data, thereby improving the accuracy of the system.

<FIG> is a block diagram that illustrates an example of a system <NUM> for automatic speech recognition of an utterance <NUM> spoken by a user <NUM> using audio data <NUM> corresponding to the utterance <NUM> and video data <NUM> representing motion of lips of the user <NUM> while the user <NUM> was speaking the utterance <NUM>. The system <NUM> includes a client device <NUM>, a computing system <NUM>, and a network <NUM>. In the example, the computing system <NUM> receives audio data <NUM> and video data <NUM> from the client device <NUM>, and the computing system <NUM> obtains multiple candidate transcriptions <NUM>, 135a-n for the utterance <NUM> based on the audio data <NUM>. As used herein, the terms "candidate transcription" and "transcription hypothesis" may be used interchangeably. As described in greater detail below, for each candidate transcription <NUM>, the computing system <NUM> is configured to generate a synthesized speech representation <NUM> of the corresponding candidate transcription <NUM> and use the video data <NUM> to determine an agreement score <NUM> indicating a likelihood that the synthesized speech representation <NUM> of the corresponding candidate transcription <NUM> matches the motion of the lips of the user <NUM> while speaking the utterance <NUM>. Thereafter, the computing system <NUM> selects one of the multiple candidate transcriptions <NUM> for the utterance as a speech recognition output <NUM> based on the agreement scores <NUM> determined for the multiple candidate transcriptions <NUM> for the utterance <NUM>. <FIG> shows stages (A) to (H) which illustrate a flow of data.

The client device <NUM> can be, for example, a desktop computer, a laptop computer, a smart phone, a smart speaker, a smart display, a tablet computer, a music player, an e-book reader, or a navigation system. The client device <NUM> includes one or more audio capture devices (e.g., microphone(s)) <NUM> configured to record utterances <NUM> spoken by the user <NUM> and one or more video image/video capture devices (e.g., camera(s)) <NUM> configured to capture image/video data <NUM> representing motion of the lips of the user <NUM> while the user <NUM> speaks the utterance <NUM>. In some examples, the microphone <NUM> or the camera <NUM> is separate from the client device <NUM> and in communication with the client device <NUM> to provide the recorded utterance <NUM> or the captured image/video data <NUM> to the client device <NUM>. The functions performed by the computing system <NUM> can be performed by individual computer systems or can be distributed across multiple computer systems. The network <NUM> can be wired or wireless or a combination of both, and may include private networks and/or public networks, such as the Internet.

As will become apparent, video data <NUM> representing motion of the lips of the user <NUM> while the user <NUM> is speaking can be used to improve speech recognition accuracy by re-scoring and re-ranking the multiple candidate transcriptions <NUM> obtained for the utterance <NUM> based on the audio data <NUM> alone. For example, after a set of candidate transcriptions <NUM> are obtained for an utterance <NUM>, further processing is done for a set of the n-best candidate transcriptions <NUM>, 135a-n, where n is an integer (e.g., the <NUM>, <NUM>, or <NUM> most likely transcriptions). Thus, rather than accepting the candidate transcription <NUM> that a speech recognizer module <NUM> indicates is most likely based on audio data alone, the video data <NUM> can be leveraged to re-score and re-rank the set of n-best candidate transcriptions <NUM>.

For example, the speech recognizer module <NUM> may employ a language model that is broad enough to model naturally spoken utterances, but may not be able to disambiguate between acoustically confusable sentences such as "I say" and "Ice Age". However, by comparing the video data representing motion of the user's <NUM> lips to synthesized speech representations <NUM> of candidate transcriptions <NUM> to determine agreement scores <NUM> for the candidate transcriptions <NUM>, the sentence "Ice Age" including a higher agreement score <NUM> than the agreement score <NUM> for "I say" may indicate that it is more likely that the user <NUM> uttered "Ice Age" and not "I say".

Each of the n-best candidate transcriptions <NUM> is provided as a synthesized speech representation <NUM> of the corresponding candidate transcription <NUM>. Agreement scores <NUM> for each of the n-best candidate transcriptions <NUM> are analyzed, for example, to determine how well each synthesized speech representation <NUM> of the corresponding candidate transcription <NUM> matches the video data <NUM> representing the motion of the lips of the user <NUM> while speaking the utterance <NUM> for which the candidate transcriptions <NUM> are obtained. The agreement score <NUM> for each corresponding candidate transcription <NUM> indicates a degree of likelihood that each candidate transcription is correct, e.g., based on a likelihood that the synthesized speech representation <NUM> of the corresponding candidate transcription <NUM> matches the motion of the lips of the user <NUM> while the user <NUM> speaks the utterance <NUM>. If a candidate transcription <NUM> has a low agreement score <NUM> (e.g., the agreement score <NUM> is less than an agreement score threshold), the candidate transcription <NUM> is unlikely to be a correct transcription for the utterance <NUM>. On the other hand, if a candidate transcription <NUM> has a high agreement score <NUM> (e.g., the agreement score <NUM> is greater than or equal to the agreement score threshold), the candidate transcription <NUM> is more likely to be correct. Accordingly, the agreement scores <NUM> based on the video data <NUM> and the synthesized speech representations <NUM> of the candidate transcriptions <NUM> may be used to re-rank the n-best candidate transcriptions <NUM> obtained by the speech recognizer module <NUM> based on the audio data <NUM> alone.

In the example of <FIG>, during stage (A), the user <NUM> speaks an utterance <NUM>, and the microphone <NUM> of the client device <NUM> records the utterance <NUM>. For instance, the utterance <NUM> may include the user <NUM> speaking the term "kite". Simultaneously, the camera <NUM> captures the video data <NUM> representing the motion of the lips of the user <NUM> while the user <NUM> speaks the utterance <NUM>. Thereafter, the client device <NUM> transmits the audio data <NUM>, corresponding to the utterance <NUM> recorded by the microphone <NUM>, and the video data <NUM>, captured by the camera <NUM>, to the computing system <NUM> via the network <NUM>.

During stage (B), the computing system <NUM> receives the audio data <NUM> and obtains multiple candidate transcriptions <NUM> for the utterance <NUM> based on the audio data <NUM>. For example, the computing system <NUM> may include the speech recognizer module <NUM> (e.g., an automated speech recognition (ASR) module) for producing a word lattice <NUM> indicating the multiple candidate transcriptions <NUM> that may be possible for the utterance <NUM> based on the audio data <NUM>.

<FIG> is an example of a word lattice <NUM>, 200a that may be provided by the speech recognizer module <NUM> of <FIG>. The word lattice 200a represents multiple possible combinations of words that may form different candidate transcriptions <NUM> for an utterance <NUM>.

The word lattice 200a includes one or more nodes 202a-g that correspond to the possible boundaries between words. The word lattice 200a includes multiple edges 204a-l for the possible words in the transcription hypotheses (e.g., candidate transcription <NUM>) that result from the word lattice 200a. In addition, each of the edges 204a-<NUM> can have one or more weights or probabilities of that edge being the correct edge from the corresponding node. The weights are determined by the speech recognizer module <NUM> and can be based on, for example, a confidence in the match between the speech data and the word for that edge and how well the word fits grammatically and/or lexically with other words in the word lattice 200a.

For example, initially, the most probable path (e.g., most probable candidate transcription <NUM>) through the word lattice 200a may include the edges 204c, 204e, 204i, <NUM>, which have the text "we're coming about <NUM>:<NUM>. " A second best path (e.g., second best candidate transcription) may include the edges 204d, <NUM>, 204j, <NUM>, which have the text "deer hunting scouts <NUM>:<NUM>.

Each pair of nodes may have one or more paths corresponding to the alternate words in the various candidate transcriptions <NUM>. For example, the initial most probable path between the node pair beginning at the node 202a and ending at the node 202c is the edge 204c "we're. " This path has alternate paths that include the edges 24a-b "we are" and the edge 204d "deer.

<FIG> is an example of a hierarchical word lattice <NUM>, 200b that may be provided by the speech recognizer module <NUM> of <FIG>. The word lattice 200b includes nodes 252a-<NUM> that represent the words that make up the various candidate transcriptions <NUM> for an utterance <NUM>. The edges between the nodes 252a-<NUM> show that the possible candidate transcriptions include: (<NUM>) the nodes 252c, 252e, 252i, <NUM> "we're coming about <NUM>:<NUM>"; (<NUM>) the nodes 252a, 252b, 252e, 252i, <NUM> "we are coming about <NUM>:<NUM>"; (<NUM>) the nodes 252a, 252b, 252f, <NUM>, 252i, <NUM> "we are come at about <NUM>:<NUM>"; (<NUM>) the nodes 252d, 252f, <NUM>, 252i, <NUM> "deer come at about <NUM>:<NUM>"; (<NUM>) the nodes 252d, <NUM>, 252j, <NUM> "deer hunting scouts <NUM>:<NUM>"; and (<NUM>) the nodes 252d, <NUM>, 252j, <NUM> "deer hunting scouts <NUM>:<NUM>.

Again, the edges between the nodes 252a-l may have associated weights or probabilities based on the confidence in the speech recognition and the grammatical/lexical analysis of the resulting text. In this example, "we're coming about <NUM>:<NUM>" may currently be the best hypothesis and "deer hunting scouts <NUM>:<NUM>" may be the next best hypothesis. One or more divisions, 254a-d, can be made in the word lattice 200b that group a word and its alternates together. For example, the division 254a includes the word "we're" and the alternates "we are" and "deer". The division 254b includes the word "coming" and the alternates "come at" and "hunting". The division 254c includes the word "about" and the alternate "scouts" and the division 254d includes the word "<NUM>:<NUM>" and the alternate "<NUM>:<NUM>.

Referring back to <FIG>, the speech recognizer module <NUM> may use an acoustic model and language model to generate the word lattice <NUM> or otherwise identify the multiple candidate transcriptions <NUM> for the utterance <NUM> based on the audio data <NUM>. The speech recognizer module <NUM> may also indicate which of the candidate transcriptions <NUM> the speech recognizer module <NUM> considers most likely to be correct, for example, by providing likelihood scores and/or ranking for the candidate transcriptions <NUM>.

During stage (C), the computing system <NUM> identifies a set of highest-ranking candidate transcriptions <NUM> from within the set of candidate transcriptions received in the word lattice <NUM>. For example, using likelihood scores or ranking information from the speech recognizer module <NUM>, the computing system <NUM> may select n candidate transcriptions <NUM> with the highest likelihoods, where n is an integer. In the illustrated example, the top five candidate transcriptions (e.g., the five that are indicated as most likely to be correct) are selected as the set of highest-ranking candidate transcriptions <NUM>, 135a-n. In the example shown, the set of highest-ranking candidate transcriptions <NUM> include the words "write", "bite", "sight", "night", and "kite" ranked in order from highest to lowest. Notably, the candidate transcription <NUM> of "kite" is ranked last even though this is the word actually spoken by the user <NUM> in the recorded utterance <NUM>. Put another way, if the highest ranking candidate transcription <NUM> output from the speech recognizer module <NUM> were selected as a speech recognition result, the word "write" would be selected in error over the word "kite.

In some examples, the speech recognizer module <NUM> generates an initial set of candidate transcriptions <NUM> for the utterance <NUM> based on the audio data <NUM>, whereby each candidate transcription <NUM> in the initial set is associated with a corresponding likelihood score indicating a likelihood that the candidate transcription <NUM> is correct. Thereafter, the speech recognizer module <NUM> ranks the candidate transcriptions <NUM> in the initial set based on the likelihood scores (e.g., from most to least likely) and stage (C) determines the multiple candidate transcriptions <NUM> for the utterance as the N-candidate transcriptions <NUM> in the initial set of candidate transcriptions associated with the highest likelihood scores. Here, the identified multiple candidate transcriptions 135a-n are ranked according to the associated likelihood scores.

In additional examples, after the speech recognizer module <NUM> generates the initial set of candidate transcriptions <NUM>, the speech recognizer module <NUM> identifies two or more candidate transcriptions <NUM> in the initial set that are associated with likelihood scores that satisfy a likelihood threshold. Here, stage (C) determines the multiple candidate transcriptions 135a-n for the utterance as the identified two or more candidate transcriptions <NUM> in the initial set that are associated with likelihood scores that satisfy the likelihood threshold. In these examples, candidate transcriptions <NUM> associated with low likelihood scores are eliminated from consideration.

During stage (D), the computing system <NUM> provides each candidate transcription <NUM> to a text-to-speech (TTS) module <NUM> (e.g., a speech synthesizer or speech synthesis module). For each candidate transcription 135a-n identified at stage (C), the TTS module <NUM> is configured to generate a synthesized speech representation <NUM>, 145a-n of the corresponding candidate transcription 135a-n. For instance, the TTS module <NUM> may convert text from each candidate transcription <NUM> into the corresponding synthesized speech representation <NUM>.

At stage (E), the computing system <NUM> provides, as feature inputs to an agreement score determiner <NUM>, the video data <NUM> representing the motion of the lips of the user <NUM> and the synthesized speech representation <NUM> output from the TTS module <NUM> for each candidate transcription <NUM>. In turn, the agreement score determiner <NUM> is configured to determine, as feature outputs, the agreement scores <NUM>, 155a-n for the candidate transcriptions 135a-n. The agreement score determiner <NUM> may determine the agreement scores <NUM> in parallel, determine each agreement score <NUM> individually, or a combination thereof.

Prior to determining the agreement scores <NUM>, the agreement score determiner <NUM> may initially process each synthesized speech representation <NUM> and the video data <NUM> to time-align each synthesized speech representation <NUM> with the video data <NUM>. That is, the agreement score determiner <NUM> may apply any technique to identify and mark frames in the video data <NUM> that contains motion of the lips of the user <NUM> while speaking the utterance <NUM>, and time-align each synthesized speech representation <NUM> with the video data <NUM> using the identified and marked frames.

In the example shown, the agreement score determiner <NUM> includes an agreement score model <NUM> (<FIG>) trained to predict an agreement score <NUM> for a corresponding candidate transcription <NUM> based on a degree that a synthesized speech representation <NUM> of the corresponding candidate transcription <NUM> matches the motion of the lips of the user <NUM> speaking the utterance <NUM>. In essence, the agreement score model <NUM> is trained to discern between synthesized speech representations <NUM> that match video data <NUM> representing motion of lips and synthesized speech representations <NUM> that do not match video data <NUM> representing motion of lips.

In some examples, the agreement scores <NUM> output from the agreement score determiner <NUM> (i.e., using the agreement score model <NUM>) include binary values, where "<NUM>" denotes a synthesized speech representation <NUM> that matches motion of the lips of the user <NUM> represented by video data <NUM>, and "<NUM>" denotes a synthesized speech representation that does not match motion of the lips of the user <NUM> represented by video data <NUM>. In additional examples, the agreement scores <NUM> are numerical values, e.g., from zero to one, indicative of the degree that a synthesized speech representation <NUM> of the corresponding candidate transcription <NUM> matches the motion of the lips. For instance, agreement scores <NUM> with values closer to one are more indicative of synthesized speech representations <NUM> that match the motion of the lips than agreement scores <NUM> with values closer to zero. In some scenarios, an agreement score <NUM> having a value that satisfies (e.g., exceeds) an agreement score threshold is indicative of a synthesized speech representation <NUM> matching the motion of the lips of a user <NUM> speaking an utterance <NUM>. In these scenarios, binary values representative of the agreement scores <NUM> may be output from the agreement score determiner <NUM> based on whether or not initial numerical values satisfy the agreement score threshold.

By indicating how well a synthesized speech representation <NUM> for a given candidate transcription <NUM> matches the motion of the lips of the user <NUM> in the video data <NUM>, each agreement score <NUM> represents a re-scoring of the multiple candidate transcriptions <NUM> identified at stage (C). The degree that video data <NUM> representing the motion of the lips of the user <NUM> matches a synthesized speech representation <NUM> of the candidate transcription <NUM> may be based on, for example, whether a sequence of phonetic features in the synthesized speech representation <NUM> completely or partially matches a sequence of lip positions and shapes of the user <NUM> in the video data <NUM>. For instance, at a given time instance, the model <NUM> would identify a match when the lip position/shape of the user <NUM> is indicative of the user's mouth being open while the synthesized speech representation <NUM> is pronouncing a vowel. Similarly, if at another time instance, the agreement score model <NUM> would not identify a match when the lip position of the user <NUM> is indicative of the user's mouth being open while the synthesized speech representation <NUM> is pronouncing a "B" consonant.

<FIG> shows an example model trainer <NUM> for generating the agreement score model <NUM>. In the example shown, the model trainer <NUM> is trained on a plurality of training examples <NUM> that include positive training examples 302a and negative training examples 302b. Each positive training example 302a contains training audio data 112T representing an utterance of speech and training video data 114T representing motion of lips of a speaker that matches (e.g., is synchronized with) the utterance of speech. That is, the model trainer <NUM> feeds the agreement score model <NUM> positive training examples 302a to teach the agreement score determiner <NUM> examples where the agreement score determiner <NUM> should output agreement scores <NUM> indicating a match/synchronization between synthesized speech representations and motion/movement of lips in video data <NUM>.

By contrast, each negative training example 302b contains training audio data 112T representing an utterance of speech and training video data 114T representing motion of lips of a speaker that does not match (e.g., is not synchronized with) the utterance of speech. That is, the model trainer <NUM> feeds the agreement score model <NUM> negative training examples 302b to teach the agreement score determiner <NUM> examples where the agreement score determiner <NUM> should output agreement scores <NUM> indicating no match and no synchronization between synthesized speech representations and motion/movement of lips in video data <NUM>.

By training the model trainer <NUM> on the positive training examples 302a and the negative training examples 302b to generate the agreement score model <NUM>, the agreement score determiner <NUM> is taught to discern between synthesized speech representations that match/synchronize with motion of the lips represented by video data <NUM> and synthesized speech representations <NUM> that do not match or synchronize with the motion/movement of the lips represented by the video data <NUM>. Accordingly, the agreement score determiner <NUM> can use the trained agreement score model <NUM> to generate an agreement score <NUM> that indicates the degree that the synthesized speech representation <NUM> of a corresponding candidate transcription <NUM> matches the motion of the lips represented by the video data <NUM>.

In some examples, the training audio data 112T includes human-generated utterances of speech <NUM>. In other examples, the training audio data 112T includes synthesized utterances <NUM> (e.g., generated by the TTS module <NUM>). In yet other examples, the training audio data 112T includes both synthesized utterances <NUM> and human-generated utterances <NUM>.

In some configurations, the model trainer <NUM> is configured to segregate training examples <NUM> into training and evaluation sets (e.g., <NUM>% training and <NUM>% evaluation. ) With these sets, the model trainer <NUM> trains the agreement score model <NUM> with the training examples <NUM> until a performance of the agreement score model <NUM> on the evaluation set stops decreasing. Once the performance stops decreasing on the evaluation set, the agreement score model <NUM> is ready for modeling where the agreement score model <NUM> allows the agreement score determiner <NUM> output agreement scores <NUM> each indicating a likelihood that a synthesized speech representation <NUM> of a corresponding candidate transcription <NUM> matches the motion of lips of a user <NUM> while the user speaks an utterance <NUM>.

Referring back to <FIG>, the agreement scores <NUM>, 155a-n for the multiple candidate transcriptions <NUM> include <NUM> for "Write", <NUM> for "Bite", <NUM> for "Sight", <NUM> for "Night", and <NUM> for "Kite". Here, the candidate transcription <NUM> of "Kite" includes the highest agreement score <NUM>, and is in fact, the word actually spoken by the user <NUM> in the utterance <NUM>. At stage (F), the re-ranker <NUM> receives, from the agreement score determiner <NUM>, the agreement scores <NUM> for the multiple candidate transcriptions <NUM> and outputs a re-ranked result <NUM> of the multiple candidate transcriptions <NUM> based on the agreement scores <NUM>. In the example shown, the multiple candidate transcriptions <NUM> are re-ranked from highest agreement scores <NUM> to lowest agreement scores <NUM>. Thus, the computing system <NUM> (e.g., via the re-ranker <NUM>) produces the re-ranked result <NUM> associated with a new ranking that is different than the initial/original ranking indicated by the speech recognizer module <NUM> based on the audio data <NUM> alone.

In some examples, in the event of a tie between two or more candidate transcriptions <NUM> having a same agreement score <NUM>, the candidate transcription <NUM> associated with a higher ranking identified at stage (C) may be ranked higher by the re-ranker <NUM> in the re-ranked result <NUM>. That is, the re-ranker <NUM> may consider speech recognition features associated with the multiple candidate transcriptions <NUM> generated by the speech recognizer module <NUM>. Speech recognition features may include information produced by a language model at the speech recognizer <NUM> for a given candidate transcription, such as a language model probability, a position within a ranking, a number of tokens, or a confidence score from the speech recognizer module <NUM>. In the event of a tie, the re-ranker <NUM> may additionally take into consideration sematic features and/or speech recognition features. Semantic features may indicate information about of pattern matching analysis, for example, matching a candidate transcription to a grammar. For example, if a candidate transcription matches a popular voice action pattern, it has a better chance to be correct recognition. Many voice queries are commands, such as "show me movies by Jim Carey" or "open web site.

At stage (G), the computing system <NUM> receives the re-ranked result <NUM> and selects, from among the multiple candidate transcriptions <NUM> for the utterance <NUM>, the candidate transcription <NUM> associated with the highest agreement score <NUM> as the speech recognition output <NUM> for the utterance <NUM>. In the example shown, the candidate transcription <NUM> for the word "Kite" includes the highest agreement score equal to "<NUM>", and is therefore selected as the speech recognition output <NUM> for the utterance <NUM>.

During stage (H), the computing system <NUM> provides the speech recognition output <NUM> to the client device <NUM> over the network <NUM>. The client device <NUM> may then display the speech recognition output <NUM> on a screen of the client device <NUM> and/or use the speech recognition output <NUM> to perform an action/command. For example, the client device <NUM> may submit the speech recognition output <NUM> as a search query or use the output <NUM> in another manner. In additional examples, the computing system <NUM> provides the speech recognition output <NUM> directly to another system to perform an action/command related to the speech recognition output <NUM>. For instance, the computing system <NUM> may provide the speech recognition output <NUM> to a search engine to perform a search query using the speech recognition output <NUM>.

While the candidate transcriptions <NUM> illustrated in <FIG> are depicted as individual words for simplicity, it should be understood that the multiple candidate transcriptions <NUM> and synthesized speech representations <NUM> produced therefrom may include multiple words contained in an utterance <NUM> of one or more phrases, one or more sentences, or even in longer-form utterances recorded from a meeting or lecture. For example, the audio data <NUM> and video data <NUM> may represent an entire query spoken by the user, and each of the candidate transcriptions <NUM> may be a respective candidate transcription for the audio data <NUM> as a whole.

The computing system <NUM> may include data processing hardware (e.g., a processor) <NUM> (<FIG>) and memory hardware <NUM> (<FIG>) in communication with the data processing hardware <NUM> and storing instructions that when executed on the data processing hardware cause the data processing hardware <NUM> to perform operations. For instance, the data processing hardware <NUM> may execute the speech recognizer module <NUM>, the TTS module <NUM>, the agreement score determiner <NUM>, and the re-ranker <NUM>. In some implementations, the entire functionality of the computing system <NUM> resides on-device on the client device <NUM>. Advantageously, latency may be improved since the client device <NUM> does not have to transmit the audio and video data <NUM>, <NUM> over a network <NUM> and wait to receive the resulting speech recognition output <NUM>.

In some implementations, the functionality of the computing system <NUM> described in <FIG> is partitioned among the client device <NUM> and the computing system <NUM>, whereby some operations are performed on the client device <NUM> while other operations are performed remotely on the computing system <NUM>. For example, audio data analysis may be performed on the computing system <NUM> (e.g., cloud computing environment) such that the client device <NUM> provides the audio data <NUM> to the speech recognizer module <NUM> to obtain the multiple candidate transcription candidates <NUM> for the utterance <NUM> and the TTS module <NUM> may generate the corresponding synthesized speech representation <NUM> for each candidate transcription <NUM> of the multiple candidate transcription candidates <NUM> for the utterance <NUM>. Instead of providing the video data <NUM> representing the motion of the lips of the user <NUM> as the user <NUM> speaks the utterance <NUM>, the client device <NUM> may execute the agreement score determiner <NUM> on-device. Thus, the computing system <NUM> may transmit the synthesized speech representations <NUM> for the multiple candidate transcriptions <NUM> to the client device <NUM> via the network <NUM>, whereby the client device <NUM> is configured to obtain the video data <NUM> and determine the agreement score <NUM> for each candidate transcription <NUM> of the multiple candidate transcriptions <NUM> received from the computing system. As described in detail above, each agreement score <NUM> indicates the likelihood that the synthesized speech representation <NUM> of the corresponding candidate transcription <NUM> matches the motion of the lips of the user <NUM> while the user <NUM> speaks the utterance <NUM>. The client device <NUM> may then select one of the multiple candidate transcriptions <NUM> (e.g., the one associated with the highest agreement score <NUM>) as the speech recognition output <NUM> based on the agreement scores determined on-device.

With this configuration, bandwidth requirements are reduced since the video data <NUM> is retained on the client device <NUM> without the need to transmit the video data <NUM> over the network <NUM> to the remote computing system <NUM>. Further, if the video data <NUM> were transmitted over the network <NUM>, the video data <NUM> would likely need to be compressed by the client device <NUM> prior to transmission, thereby reducing the quality of the video data <NUM>. Therefore, another advantage of retaining the video data <NUM> and performing the video data matching on-device is that the video data matching may use uncompressed (highest quality) video data <NUM>. That is, the use of uncompressed video data makes it easier to recognize matches/mismatches between the synthesized speech representations <NUM> and the video data <NUM>.

Each of the multiple candidate transcriptions <NUM> output by the speech recognizer module <NUM> for the utterance <NUM> may be associated with a same language. In some examples, at least one of the multiple candidate transcriptions <NUM> for the utterance <NUM> is associated with a different language than the other candidate transcriptions <NUM>. For instance, the computing system <NUM> may not know the language of the utterance <NUM> a priori, and may rely on the speech recognizer module <NUM> to use different language models to output multiple candidate transcriptions <NUM> divided between two or more different languages. In this scenario, the candidate transcription <NUM> associated with the correct language of the utterance <NUM> is identified/selected by comparing the corresponding synthesized speech representations <NUM> to the video data <NUM> representing the motion of the lips of the user <NUM> while speaking the utterance <NUM>. That is, the language of the candidate transcription <NUM> associated with a highest agreement score <NUM> may be selected as the speech recognition output <NUM> to identify the correct language. Since the video data <NUM> has been used to determine the best-matching synthesized speech representation <NUM>, it is more likely that the associated candidate transcription <NUM> will be in the correct language.

For example, a user is provided with control over whether programs or features collect user information about that particular user or other users relevant to the program or feature. Each user for which personal information is to be collected is presented with one or more options to allow control over the information collection relevant to that user, to provide permission or authorization as to whether the information is collected and as to which portions of the information are to be collected. For example, users can be provided with one or more such control options over a communication network. In addition, certain data may be treated in one or more ways before it is stored or used so that personally identifiable information is removed. As one example, a user's identity may be treated so that no personally identifiable information can be determined. As another example, a user's geographic location may be generalized to a larger region so that the user's particular location cannot be determined.

<FIG> is a flowchart of an example arrangement of operations for a method <NUM> of rescoring candidate transcriptions using audio-visual matching. At operation <NUM>, the method <NUM> includes receiving, at data processing hardware <NUM>, audio data <NUM> corresponding to an utterance <NUM> spoken by a user <NUM> and video data <NUM> representing motion of lips of the user <NUM> while the user <NUM> was speaking the utterance <NUM>. At operation <NUM>, the method <NUM> includes obtaining, by the data processing hardware <NUM>, multiple candidate transcriptions <NUM> for the utterance <NUM> based on the audio data <NUM>. At operation <NUM>, for each candidate transcription <NUM>, the method <NUM> also includes generating, by the data processing hardware <NUM>, a synthesized speech representation <NUM> of the corresponding candidate transcription <NUM>. At operation <NUM>, for each candidate transcription <NUM>, the method <NUM> also includes determining, by the data processing hardware, an agreement score <NUM> indicating a likelihood that the synthesized speech representation <NUM> of the corresponding candidate transcription <NUM> matches the motion of the lips of the user <NUM> while the user <NUM> speaks the utterance <NUM>. At operation <NUM>, the method <NUM> includes selecting, by the data processing hardware <NUM>, one of the multiple candidate transcriptions <NUM> for the utterance <NUM> as a speech recognition output <NUM> based on the agreement scores <NUM> determined for the multiple candidate transcriptions <NUM> for the utterance <NUM>.

The storage device <NUM> is a computer-readable medium.

Claim 1:
A method (<NUM>) comprising:
receiving, at data processing hardware (<NUM>), audio data (<NUM>) corresponding to an utterance (<NUM>) spoken by a user (<NUM>);
receiving, at the data processing hardware (<NUM>), video data (<NUM>) representing motion of lips of the user while the user (<NUM>) was speaking the utterance (<NUM>);
obtaining, by the data processing hardware (<NUM>), multiple candidate transcriptions (<NUM>) for the utterance (<NUM>) based on the audio data (<NUM>);
characterized by:
for each candidate transcription (<NUM>) of the multiple candidate transcriptions (<NUM>) for the utterance (<NUM>):
generating, by the data processing hardware (<NUM>), a synthesized speech representation (<NUM>) of the corresponding candidate transcription (<NUM>); and
determining, by the data processing hardware (<NUM>), an agreement score (<NUM>) indicating a likelihood that the synthesized speech representation (<NUM>) of the corresponding candidate transcription (<NUM>) matches the motion of the lips of the user while the user (<NUM>) speaks the utterance (<NUM>), the agreement score (<NUM>) being determined based on the synthesized speech representation (<NUM>) and the video data (<NUM>); and
selecting, by the data processing hardware (<NUM>), one of the multiple candidate transcriptions (<NUM>) for the utterance (<NUM>) as a speech recognition output (<NUM>) based on the agreement scores (<NUM>) determined for the multiple candidate transcriptions (<NUM>) for the utterance (<NUM>).