Patent ID: 11894014
Assignee: GOOGLE LLC
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 17:
18. One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers, cause the one or more computers to perform operations comprising:
obtaining, for each frame in a stream of frames from a video in which faces of one or more speakers have been detected, a respective per-frame face embedding of a face of each speaker;
processing, for each speaker, the per-frame face embeddings of the face of the speaker using a video processing neural network to generate visual features for the face of the speaker;
obtaining a spectrogram of an audio soundtrack for the video;
processing the spectrogram using an audio processing neural network to generate an audio embedding for the audio soundtrack;
combining the visual features for the faces of the one or more speakers and the audio embedding for the audio soundtrack to generate an audio-visual embedding for the video, wherein the audio-visual embedding represents both audio features of the audio soundtrack and visual features of the respective faces of the one or more speakers;
processing, using a masking neural network, the audio-visual embedding for the video that represents both the audio features of the audio soundtrack and the visual features for the faces of the one or more speakers to generate a respective spectrogram mask for each of the one or more speakers, wherein each spectrogram mask describes time-frequency relationships between clean speech for a respective speaker and background interference in a spectrogram of the audio soundtrack; and
determining, using the respective spectrogram masks and the corresponding audio soundtrack, a respective isolated speech spectrogram for each speaker that isolates the speech of the speaker in the video.