A method implemented by one or more processors, the method comprising: receiving audio data that captures an utterance of a human speaker and that also captures one or more additional sounds that are not from the human speaker; processing the audio data with a frequency transformation portion of an automatic speech recognition, "ASR", engine to generate a frequency representation of the audio data; generating a revised version of the frequency representation that includes one or more portions of the frequency representation corresponding to the utterance and that excludes one or more other portions of the frequency representation that capture the one or more additional sounds, and wherein generating the revised version of the frequency representation comprises: processing, using a voice filter model, the frequency representation and a speaker embedding to generate a predicted mask that is conditioned on both the frequency representation and the speaker embedding, wherein the speaker embedding corresponds to the human speaker; generating the revised version of the frequency representation by processing the frequency representation using the predicted mask, wherein the processing comprises convolving the frequency representation with the predicted mask; and processing the revised version of the frequency representation using a speech recognition portion of the ASR engine to generate a text representation of the utterance of the human speaker. receiving audio data that captures an utterance of a human speaker and that also captures one or more additional sounds that are not from the human speaker; processing the audio data with a frequency transformation portion of an automatic speech recognition, "ASR", engine to generate a frequency representation of the audio data; generating a revised version of the frequency representation that includes one or more portions of the frequency representation corresponding to the utterance and that excludes one or more other portions of the frequency representation that capture the one or more additional sounds, and wherein generating the revised version of the frequency representation comprises: processing, using a voice filter model, the frequency representation and a speaker embedding to generate a predicted mask that is conditioned on both the frequency representation and the speaker embedding, wherein the speaker embedding corresponds to the human speaker; generating the revised version of the frequency representation by processing the frequency representation using the predicted mask, wherein the processing comprises convolving the frequency representation with the predicted mask; and processing, using a voice filter model, the frequency representation and a speaker embedding to generate a predicted mask that is conditioned on both the frequency representation and the speaker embedding, wherein the speaker embedding corresponds to the human speaker; generating the revised version of the frequency representation by processing the frequency representation using the predicted mask, wherein the processing comprises convolving the frequency representation with the predicted mask; and processing the revised version of the frequency representation using a speech recognition portion of the ASR engine to generate a text representation of the utterance of the human speaker.