Patent ID: 11900919
Assignee: GOOGLE LLC
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 0:
1. An audio-visual automated speech recognition (AV-ASR) model for transcribing speech from audio-visual data, the AV-ASR model comprising:
an encoder configured to:
receive a sequence of acoustic feature vectors for an audio track and a plurality of video face tracks, each video face track of the plurality of video face tracks is associated with a face of a respective person; and
for each corresponding acoustic feature vector in the sequence of acoustic feature vectors:
determine a respective attention-weighted visual feature vector that soft-selects a particular one of the video face tracks that is most likely to include the face of the respective person as a speaking face of the corresponding acoustic feature vector; and
generate a combined feature vector by concatenating the respective attention-weighted visual feature vector with the corresponding acoustic feature vector; and

a decoder configured to process the combined feature vector generated for each corresponding acoustic feature vector in the sequence of acoustic feature vectors to determine a speech recognition result of the audio track.