Patent Description:
Automatic speech recognition (ASR) is an important technology that is used in mobile devices and other devices. In general, automatic speech recognition attempts to provide accurate transcriptions of what a person has said. In noisy environments, or otherwise when audio quality of a recorded utterance is poor, obtaining accurate ASR results can be a difficult task. When video data of a speaker is available, the video data can be leveraged to help improve ASR results. For instance, the video data of the speaker may provide motion of the lips while the speaker is speaking an utterance, which in turn, can be combined with the audio data of the utterance to assist in processing an ASR result. <CIT> describes an information processing device includes an audio-based speech recognition processing unit which is input with audio information as observation information of a real space, executes an audio-based speech recognition process, thereby generating word information that is determined to have a high probability of being spoken, an image-based speech recognition processing unit which is input with image information as observation information of the real space, analyzes mouth movements of each user included in the input image, thereby generating mouth movement information, an audio-image-combined speech recognition score calculating unit which is input with the word information and the mouth movement information, executes a score setting process in which a mouth movement close to the word information is set with a high score, thereby executing a score setting process, and an information integration processing unit which is input with the score and executes a speaker specification process. In "<NPL>, et al. describe a fully differentiable A/V ASR model that is able to handle multiple face tracks in a video. Instead of relying on two separate models for speaker face selection and audiovisual ASR on a single face track, an attention layer is introduced to the ASR encoder that is able to soft-select the appropriate face video track.

One aspect of the disclosure provides a computer-readable medium comprising instructions which, when executed, cause a single audio-visual speech recognition (AV-ASR) model to transcribe speech from audio-visual data. The AV-ASR model includes an encoder frontend having an attention mechanism that is configured to receive an audio track of the audio-visual data and a video portion of the audio-visual data. The video portion of the audio-visual data includes a plurality of video face tracks. Each video face track of the plurality of video face tracks is associated with a face of a respective person. For each video face track of the plurality of video face tracks, the attention mechanism is further configured to determine a confidence score indicating a likelihood that the face of the respective person associated with the video face track includes a speaking face of the audio track. The AV-ASR model further includes a decoder configured to process the audio track and the video face track of the plurality of video face tracks associated with the highest confidence score to determine a speech recognition result of the audio track.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, the single AV-ASR model includes a sequence-to-sequence model. For instance, the AV-ASR model may include an Audio-Visual Recurrent Neural Network Transducer (RNN-T) model. The decoder may be configured to emit the speech recognition result of the audio track in real time to provide a streaming transcription of the audio track. In some examples, the single AV-ASR model does not include a separate face selection system for hard-selecting which video face track of the plurality of video face tracks comprises the speaking face of the audio track.

The attention mechanism may be configured to generate as output an attention-weighted visual feature vector for the plurality of video face tracks. Here, the attention-weighted visual feature vector represents a soft-selection of the video face track of the plurality of video face tracks that includes the face of the respective person with the highest likelihood of including the speaking face of the audio track. Additionally or alternatively, the attention mechanism may include a softmax layer having an inverse temperature parameter configured to cause the attention mechanism to converge to a hard-decision rule of selecting the video face track of the plurality of video face tracks associated with the highest confidence score as the speaking face of the audio track.

In some examples, the encoder frontend is trained on a training data set that includes a training audio track, a first training video face track, and one or more second video face tracks. The training audio track includes one or more spoken utterances and the first training video track includes a correct speaking face of the one or more spoken utterances of the training audio track. Each second training video face track includes an incorrect speaking face of the one or more spoken utterances of the training audio track. In these examples, during training, the attention mechanism is configured to learn how to gate the first training video face track as the correct speaking face of the one or more spoken utterances of the training audio track. Here, the attention mechanism may be trained with cross entropy loss.

Another aspect of the disclosure provides a method for transcribing speech from audio-visual data. The method includes receiving, at an attention mechanism of an encoder frontend of a single audio-visual automated speech recognition (AV-ASR) model, an audio track of the audio-visual data and a video portion of the audio-visual data. The video portion of the audio-visual data includes a plurality of video face tracks and each video face track of the plurality of video face tracks is associated with a face of a respective person. For each video face track of the plurality of video face tracks, the method also includes determining, by the attention mechanism, a confidence score indicating a likelihood that the face of the respective person associated with the video face track comprises a speaking face of the audio track. The method also includes processing, by a decoder of the single AV-ASR model, the audio track and the video face track of the plurality of video face tracks associated with the highest confidence score to determine a speech recognition result of the audio track.

This aspect may include one or more of the following optional features. In some implementations, the single AV-ASR model includes a sequence-to-sequence model. For instance, the AV-ASR model may include an Audio-Visual Recurrent Neural Network Transducer (RNN-T) model. The decoder may be configured to emit the speech recognition result of the audio track in real time to provide a streaming transcription of the audio track. In some examples, the single AV-ASR model does not include a separate face selection system for hard-selecting which video face track of the plurality of video face tracks comprises the speaking face of the audio track.

In some examples, determining the confidence score for each video face track of the plurality of video face tracks includes generating an attention-weighted visual feature vector for the plurality of video face tracks. Here, the attention-weighted visual feature vector represents a soft-selection of the video face track of the plurality of video face tracks that includes the face of the respective person with the highest likelihood of comprising the speaking face of the audio track. Additionally or alternatively, the attention mechanism may include a softmax layer having an inverse temperature parameter configured to cause the attention mechanism to converge to a hard-decision rule of selecting the video face track of the plurality of video face tracks associated with the highest confidence score as the speaking face of the audio track.

In some implementations, the method also includes training the encoder frontend on a training data set that includes a training audio track, a first training video face track, and one or more second video face tracks. The training audio track includes one or more spoken utterances and the first training video track includes a correct speaking face of the one or more spoken utterances of the training audio track. Each second training video face track includes an incorrect speaking face of the one or more spoken utterances of the training audio track. In these examples, training the encoder front end includes training the attention mechanism to learn how to gate the first training video face track as the correct speaking face of the one or more spoken utterances of the training audio track. Here, the attention mechanism may be trained with cross entropy loss.

Audio-visual (A/V) automated speech recognition (ASR) is able to make conventional ASR more robust by leveraging video data of a face of a speaker in addition to audio data of a spoken from the speaker. In a realistic setting, one has to decide at each point in time which face to designate as a speaking face to an audio track when there are multiple faces in an image. A conventional pipeline for A/V ASR includes a sequence of systems/models that include a face tracking module, an active speaker selection model, and an A/V ASR model. The face tracking module detects and tracks faces in audio-video data and the active speaker selection model selects a speaking face for each portion of audio and passes a face track of the selected speaking face and corresponding audio track to the A/V ASR model. The A/V ASR model uses the audio track and face track selected by the active speaker selection model to output speech recognition hypotheses (e.g., predicted transcriptions) for segments of the audio-video data.

Conventionally, the active speaker selection model and the A/V ASR model are separate models trained separately and independently from one another. That is, the A/V ASR model is traditionally trained with a single face track that is assumed to be the speaking face selected for a given portion of an audio track. With this traditional approach, performance (e.g., accuracy of transcriptions) of the A/V ASR model hinges on the ability of the separate active speaker selection model to accurately select the correct speaking face in the audio-video data. Otherwise, selection of the wrong speaking face by the active speaker selection model will result in a degradation of performance by the A/V ASR model.

Implementations herein are directed toward training a single A/V ASR model end-to-end (E2E) on video data with multiple face tracks and an audio track simultaneously so that the A/V ASR model learns how to gate a correct face track for each segment of the audio track to aid in speech recognition. Accordingly, by training the single A/V ASR model to operate on multiple face tracks, implementations herein discard the need for a separate active speaker selection model that is tasked with tracking multiple faces and detecting the correct speaking face passed as a single face track to the A/V ASR model. Simply put, for multi-speaker A/V ASR tasks, the single A/V ASR model is configured to receive audio-visual inputs with multiple face tracks and an audio track, employ an attention layer at an encoder front end to soft-select an appropriate face track as a speaking face for each portion of the audio track to assist a decoder portion in determining a speech recognition result for each portion of the audio track.

As opposed to relying on separately trained active speaker selection and A/V ASR models that each rely on separate visual frontends that potentially perform similar roles, training the single A/V ASR model to handle multiple face video tracks increases computational performance by eliminating the redundancy associated with the similar tasks performed by separate visual frontends. Moreover, the E2E nature of the single A/V ASR model simplifies coordination between subsystems since the only input to the A/V ASR model is an output from a face tracker module which is a common component in standard computer vision. As will become apparent, the single A/V ASR model also provides more robust speech recognition on multi-speaker ASR tasks since an early hard decision is not required for selecting an active face track as is the case for the conventional techniques using the separate active face selection model. Instead, the single A/V ASR model uses an attention mechanism to soft-select the active face track (i.e., the face track associational with an active speaking face), thereby permitting remaining portions of the A/V ASR model to naturally adapt even when a high probability is assigned to the wrong face track. The use of the separate active speaker selection model to select the correct active face track is also sensitive to dynamics of discrete speaker changes over time which are difficult to emulate during training.

Referring to <FIG>, in some implementations, an environment <NUM> includes a plurality of participants <NUM>, 10a-j attending a meeting (e.g., a video conference). Here, the environment <NUM> is a host meeting room with six participants 10a-f attending the meeting (e.g., a video conference) in the host meeting room. The environment <NUM> includes a user device <NUM> that receives one or more content feeds <NUM> (also referred to as a multi-media feed, a content stream, or a feed) via a network <NUM> from a remote system <NUM>. In the example shown, the user device <NUM> receives two feeds 12a, 12a each corresponding to a different remote meeting room. Here, the first feed 12a includes three participants <NUM>, <NUM>-i participating in the meeting from a remote New York office and the second feed includes a single participant <NUM>, 10j participating from a remotely located residence of the participant 10j. Each content feed <NUM> may correspond to audio-visual data <NUM> including an audio portion <NUM> corresponding to an audio track and a video portion <NUM> including one or more video face tracks <NUM> (<FIG>). As used herein, the terms "audio track" and "audio portion" may be used interchangeably. The video portion <NUM> may be associated with image data such as video content, video signal, or video stream. The user device <NUM> includes, or is in communication with, a display <NUM> configured to display the video portion <NUM> of the audio-visual data <NUM>. The user device <NUM> also includes, or is in communication with, an audio speaker <NUM> configured to audibly output the audio portion <NUM> of the audio-visual data <NUM>.

In addition to receiving audio-visual data <NUM> from the remote meeting rooms via respective content feeds <NUM>, the user device <NUM> includes, or is in communication with, one or more peripherals <NUM> for capturing audio-visual data <NUM> from the host meeting room. For instance, an audio capture device <NUM>, 116a (e.g., an array of one or more microphones) is configured to capture utterances <NUM> spoken by the participants 10a-g and convert the captured utterances <NUM> into audio data that corresponds to the audio portion <NUM> of the audio-visual data <NUM>. On the other hand, an image capture device <NUM>, 116b (e.g., one or more cameras) is configured to capture image data that corresponds to the video portion <NUM> of the audio-visual data <NUM>. Here, the video portion <NUM> includes video face tracks <NUM> each associated with a face of a respective one of the participants 10a-g. In some configurations, the image capturing device 116b is configured to capture <NUM>-degrees about the user device <NUM> to capture a full view of the environment <NUM>. For instance, the image capturing device 116b includes an array of cameras configured to capture the <NUM>-degree view.

The remote system <NUM> may be a distributed system (e.g., cloud computing environment or storage abstraction) having scalable/elastic resources <NUM>. The resources <NUM> include computing resources <NUM> (e.g., data processing hardware) and/or storage resources <NUM> (e.g. memory hardware). In some implementations, the remote system <NUM> hosts software that coordinates the environment <NUM> (e.g., on the computing resources <NUM>). For instance, the computing resources <NUM> of the remote system <NUM> execute software, such as a real-time communication application or a specialty meeting platform. In some examples, a face tracker module executes on the data processing hardware <NUM> to detect the video face tracks <NUM> in the video portion <NUM> of the audio-visual data <NUM>.

In the example shown, the user device <NUM> includes data processing hardware <NUM> and memory hardware <NUM> in communication with the data processing hardware <NUM> and storing instructions that when executed on the data processing hardware <NUM> cause the data processing hardware <NUM> to perform operations. In some examples, a face tracker module executes on the data processing hardware <NUM> to detect the video face tracks <NUM> in the video portion <NUM> of the audio-visual data <NUM>. Some examples of a user device <NUM> include a video conference computing device, a computer, a laptop, a mobile computing device, a television, a monitor, a smart device (e.g., smart speaker, smart display, smart appliance), a wearable device, etc..

With continued reference to <FIG>, an audio-visual automated speech recognition (AV-ASR) model <NUM> processes the audio-visual data <NUM> to generate a transcription <NUM> from the audio track <NUM> of the audio-visual data <NUM>. Notably, and described in greater detail below with reference to <FIG>, the AV-ASR model <NUM> includes a single end-to-end model that receives both the audio track <NUM> and a plurality of video face tracks <NUM> detected in the video portion <NUM> of the audio-visual data <NUM>, and determines which one of the video face tracks <NUM> includes a highest likelihood of including an active speaking face of the audio track <NUM>. The AV-ASR model <NUM> than uses the video face track <NUM> that is most likely to include the active speaking face of the audio track <NUM> to aid in transcribing speech from the audio track <NUM>. As such, the use of the video portion <NUM> increases the accuracy of the transcription <NUM> of the audio track <NUM> since the video face track <NUM> provides the AV-ASR model <NUM> with visual features (e.g., facial features/ lips). In some particular examples, using only audio for speech recognition is difficult when the audio is associated speakers with speech disabilities. The video portion may improve the accuracy of speech recognition using techniques of correlating lip motion from a user with a particular speech disorder in unison with the audio data.

The display <NUM> associated with the user device <NUM> may display the transcription <NUM> generated by the AV-ASR model <NUM>. The AV-ASR model <NUM> may stream the transcription <NUM> in real time for output on the display <NUM> and/or on displays associated with remotely located participants <NUM>-j, <NUM>. Additionally or alternatively, the transcription <NUM> may be saved on memory hardware <NUM>, <NUM> and retrieved at a later time for viewing. The AV-ASR model <NUM> may execute on the data processing hardware <NUM> of the user device <NUM>, thereby enabling the user device <NUM> to perform on-device speech recognition without the need to perform speech recognition on a server (e.g., remote system <NUM>). On-device speech recognition alleviates the requirement of establishing a network connection with a server, incurring latency due to bandwidth constraints, and also preserve data that a user may not want to share with the server. Moreover, executing the AV-ASR model <NUM> on the user device <NUM> may permit the use of higher fidelity audio-visual data <NUM> since neither one of the audio portion <NUM> or the video portion <NUM> would need be compressed to satisfy network bandwidth constraints, as may be required if the data <NUM> were sent to a server for processing.

The AV-ASR model <NUM> may also execute on the data processing hardware <NUM> of the remote system <NUM>. For instance, the data processing hardware <NUM> of the remote system <NUM> may execute instructions stored on the memory hardware <NUM> of the remote system <NUM> for executing the AV-ASR model <NUM>. Here, the AV-ASR model <NUM> may process the multi-speaker audio-visual data <NUM> to generate the transcription <NUM> as discussed above. The remote system <NUM> may transmit the transcription <NUM> over the network <NUM> to the user device <NUM> for display on the display <NUM>. The remote system <NUM> may similarly transmit the transcription <NUM> to computing devices/ display devices associated with the participants <NUM>-i corresponding to the first feed 12a and/or the participant 10j corresponding to the second feed 12b.

The data processing hardware <NUM> of the remote system <NUM> may provide increased processing capabilities not achievable on client devices and is not limited to memory constraints, thereby enabling the use of larger models with more parameters for increased accuracy. In some examples, some portions of the AV-ASR model <NUM> execute on the user device <NUM> while other portions of the AV-ASR model <NUM> execute on the remote system (e.g., server) <NUM>.

<FIG> provides an example of the end-to-end, single AV-ASR model <NUM> of <FIG> that is configured to receive audio-visual data <NUM> including an audio track <NUM> and a video portion <NUM> with multiple face tracks <NUM>, 230a-c to determine a speech recognition result <NUM> of the audio track <NUM>. While the example environment <NUM> of <FIG> depicts the audio-visual data <NUM> originating from a video conference scenerio, a single feed of audio-visual data <NUM> may arrive from any source. For instance, the AV-ASR <NUM> may receive a single feed of audio-visual data <NUM> from media content such as a movie or a live television broadcast. In this scenario, the AV-ASR model <NUM> may similarly use the video portion <NUM> of the audio-visual data <NUM> to aid in determining speech recognition results <NUM> of the audio track <NUM>, and thus provide a transcription <NUM> of speech in the audio track <NUM> that may be provided as close captioning on a display (e.g., a television screen).

Each video face track <NUM> is associated with a face of a respective person <NUM>. While the AV-ASR model <NUM> is shown in the example as receiving three video face tracks 230a-c, the number of video face tracks <NUM> the AV-ASR model <NUM> receives and subsequently processes is non-limiting. Thus, the AV-ASR model <NUM> may receive less than three video face tracks <NUM> or more than three video face tracks in other examples. Notably, the single AV-ASR model <NUM> does not include a separate face selection system for hard-selecting which video face track <NUM> of the plurality of video face tracks includes a speaking face of the audio track.

The AV-ASR model <NUM> includes an encoder portion ("encoder") <NUM> and a decoder portion ("decoder") <NUM>. The AV-ASR model <NUM> may include a sequence-to-sequence model. In some examples, the AV-ASR model <NUM> includes an Audio-Visual Recurrent Neural Network-Transducer (RNN-T) model. The Audio-Visual RNN-T may include a stack of five bidirectional long short-term memory (BiLSTM) layersof <NUM> units using layer normalization for the encoder <NUM> and two LSTM layers of <NUM>,<NUM> units with character tokens for the decoder <NUM>.

The encoder <NUM> is associated with an encoder frontend that includes an attention mechanism <NUM>. The attention mechanism <NUM> may be associated with an attention layer in the encoder portion <NUM> of the neural network model <NUM>. The encoder is configured to receive the audio track <NUM> of the audio-visual data <NUM> and the video portion <NUM> of the audio-visual data <NUM> that includes the plurality of video face tracks <NUM>, 230a-c. The audio track <NUM> may be segmented into <NUM> millisecond (ms) audio frames with steps of <NUM> between consecutive audio frames. Mel-spectral energies, such as <NUM> mel filter bank channels, may be computed for each audio frame to compress its range with a log function, and thereafter, folding every three consecutive feature vectors together to yield <NUM>-dimensional acoustic feature vectors 210a-n every <NUM>. Accordingly, the encoder portion receives and processes the acoustic feature vectors 210a-n derived from the audio track <NUM>.

For each video face track <NUM>, the attention mechanism <NUM> determines a corresponding confidence score indicating a likelihood that the face of the respective person associated with the corresponding video face track <NUM> includes a speaking face of the audio track <NUM>. In some implementations, the attention mechanism <NUM> includes a softmax layer having an inverse temperature parameter configured to cause the attention mechanism <NUM> to converge to a hard-decision rule of selecting the video face track <NUM> of the plurality of video face tracks 230a-c associated with the highest confidence score as the speaking face of the audio track <NUM>. The decoder portion <NUM> of the AV-ASR model <NUM> is configured to process the audio track <NUM> and the video track <NUM> of the plurality of video face tracks 230a-c with the highest confidence score to determine the speech recognition result <NUM> of the audio track <NUM>.

In some examples, the attention mechanism <NUM> represents the confidence associated with each video face track <NUM> as a corresponding attention weight applied to a visual feature vector associated with the corresponding video face track <NUM>. As such, the attention mechanism <NUM> may output an attention-weighted visual feature vector <NUM> for the plurality of video face tracks <NUM> that soft-selects the video face track <NUM> that is most likely to include the active speaking face of a corresponding synchronized segment (e.g., acoustic feature vector) of the audio track <NUM>.

In some implementations, the encoder <NUM> concatenates the attention-weighted visual feature vector <NUM> that soft-selects the video face track <NUM> associated with the active speaking face with the acoustic feature vector to provide a corresponding combined feature vector at each time step. The combined feature vector at each time step indicates an encoding of the audio track <NUM> and the video face track <NUM> among the plurality of video face tracks that is associated with the highest confidence score. Accordingly, at each time step, the decoder portion <NUM> is configured to decode the combined feature vector to determine a corresponding speech recognition result <NUM> of the audio track <NUM>. The speech recognition result <NUM> at each time step may include a probability distribution over possible recognition results. In examples when the AV-ASR model <NUM> is the Audio-Visual RNN-T model, the model <NUM> may emit the speech recognition result <NUM> at each time step in a streaming fashion. A speech recognition result may include a character, a space, a word-piece, or a word. The multiple speech recognition results <NUM> may combine to provide the transcription <NUM> of the audio track <NUM>. Thus, the Audio-Visual RNN-T model is capable of streaming a transcription <NUM> of the audio track <NUM> in real time. In some examples, the audio track <NUM> includes speech spoken in a first language and the decoder <NUM> is configured to determine a corresponding speech recognition <NUM> in a second language as a translation of the speech spoken in the first language.

In some examples, the AV-ASR <NUM> model is further configured provide speaker labels <NUM> to the transcription <NUM> to identify a source of the transcribed content. For instance, labeling a speaker of the transcribed content may be referred to as speaker diarization to answer both "who spoke what" and "who spoke when". Accordingly, by leveraging the video portion <NUM> of the audio-visual data <NUM>, the AV-ASR model <NUM> may provide diarization results that include a corresponding speaker label <NUM> assigned to each segment of the transcription <NUM> to identify "who spoke what" and "who spoke when".

<FIG> shows an example training process <NUM> for training the encoder portion <NUM> of the AV-ASR model <NUM> to learn how to gate a correct video face track <NUM> for each segment of the audio track to aid in speech recognition. The encoder portion <NUM> is trained on a training data set <NUM> that includes a training audio track 210T, a first training video face track 230Ta, and one or more second training video face tracks 230Tb. The training audio track <NUM> includes one or more spoken utterances. The first training video face track 230Ta includes a correct speaking face of the one or more spoken utterances of the training audio track 210T. The first raining video face track 230Ta is paired with a ground-truth correct face label 232C. Each second training video face track 230Tb includes an incorrect speaking face of the one or more spoken utterances of the audio track <NUM>. Each second training video face track 230Tb is paired with a ground-truth incorrect face label 232I.

At each of a plurality of time steps during the training process <NUM>, the encoder portion <NUM> receives, as input, the training audio track 210T, the first training video face track 230Ta, and the one or more second training video face tracks 230Tb, and generates/predicts as output via the attention mechanism <NUM>, an attention-weighted visual feature vector <NUM> that corresponds to a soft-selection of the video face track 230Ta, 230Tb that is most likely to include the active speaking face of the audio track <NUM> at the time step. In lieu computing the attention-weighted visual feature vector <NUM>, the encoder portion <NUM> may output a predicted probability distribution over possible training video face tracks 230T that include the correct speaking face of the audio track <NUM>.

The attention-weighted visual feature vector <NUM> (or probability distribution) is fed to a loss module <NUM> for determining a loss term <NUM> (i.e., a loss function) indicating the accuracy of the attention mechanism <NUM> in soft-selecting the first training video face track 230Ta as including the correct speaking face of the audio track <NUM>. Accordingly, the loss module <NUM> is a supervised loss term module that receives the correct speaking face label 232C paired with the first training video face track 210Ta and the incorrect speaking label 232I paired with each second training video face track 210Tb as ground-truth. The loss term <NUM> indicates cross entropy loss of the attention mechanism and is fed back to the attention mechanism <NUM> for teaching the attention mechanism <NUM> to learn how to gate the first training video face track 230Ta as the correct speaking face of the one or more spoken utterances of the training audio track <NUM>. Thus, the loss term <NUM> trains the attention mechanism <NUM> with gradient-decent cross entropy loss by updating parameters of the attention mechanism <NUM>.

<FIG> provides a flowchart of an example arrangement of operations for a method <NUM> of using a single audio-visual automates speech recognition (AV-ASR) model <NUM> to transcribe speech <NUM> from audio-visual data <NUM>. The single AV-ASR model <NUM> and the operations for the method <NUM> may execute on the data processing hardware <NUM> of the user device <NUM> of <FIG>, the data processing hardware <NUM> of the remote system (e.g., distributed system) <NUM> of <FIG>, or a combination thereof.

At operation <NUM>, the method <NUM> includes receiving, at an attention mechanism <NUM> of an encoder frontend <NUM> of the single AV-ASR model <NUM>, an audio track <NUM> of the audio-visual data <NUM> and a video portion <NUM> of the audio-visual data <NUM>. The video portion <NUM> includes a plurality of video face tracks <NUM>. Each video face track <NUM> of the plurality of video face tracks <NUM> is associated with a face of a respective person.

At operation <NUM>, for each video face track <NUM> of the plurality of video face tracks <NUM>, the method <NUM> also includes determining, by the attention mechanism <NUM>, a confidence score indicating a likelihood that the face of the respective person associated with the video face track <NUM> includes a speaking face of the audio track <NUM>. Here, determining the confidence score for each video face track <NUM> of the plurality of video face tracks <NUM> may include the attention mechanism <NUM> generating an attention-weighted visual feature vector <NUM> for the plurality of video face tracks <NUM>. The attention-weighted visual feature vector <NUM> may represent a soft-selection of the video face track <NUM> of the plurality of video face tracks <NUM> that includes the face of the respective person with the highest likelihood of including the speaking face of the audio track <NUM>. In some examples, the attention mechanism <NUM> includes a softmax layer having an inverse temperature parameter configured to cause the attention mechanism <NUM> to converge to a hard-decision rule of selecting the video face track <NUM> of the plurality of video face tracks <NUM> associated with the highest confidence score as the speaking face of the audio track <NUM>.

At operation <NUM>, the method <NUM> includes processing, by a decoder <NUM> of the single AV-ASR model <NUM>, the audio track <NUM> and the video face track <NUM> of the plurality of video face tracks <NUM> associated with the highest confidence score to determine a speech recognition result <NUM> of the audio track <NUM>. In some examples, the decoder <NUM> is configured to emit the speech recognition result <NUM> of the audio track <NUM> in real time to provide a streaming transcription <NUM> of the audio track <NUM>.

The components shown here, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the claimed invention.

Claim 1:
A computer-readable medium (<NUM>) comprising instructions which, when executed, cause a single audio-visual automated speech recognition, AV-ASR, model (<NUM>) to transcribe speech from audio-visual data (<NUM>), the single AV-ASR model (<NUM>) comprising:
an encoder frontend (<NUM>) comprising an attention mechanism (<NUM>), the attention mechanism (<NUM>) being configured to
receive an audio track (<NUM>) of the audio-visual data (<NUM>) and a video portion (<NUM>) of the audio-visual data (<NUM>), the video portion (<NUM>) of the audio-visual data (<NUM>) comprising a plurality of video face tracks (<NUM>), each video face track (<NUM>) of the plurality of video face tracks (<NUM>) is associated with a face of a respective person; and
for each video face track (<NUM>) of the plurality of video face tracks (<NUM>), determine a confidence score indicating a likelihood that the face of the respective person associated with the video face track (<NUM>) comprises a speaking face of the audio track (<NUM>); and
a decoder (<NUM>) configured to process the audio track (<NUM>) and the video face track (<NUM>) of the plurality of video face tracks (<NUM>) associated with the highest confidence score to determine a speech recognition result (<NUM>) of the audio track (<NUM>).