A method implemented by one or more processors, the method comprising: generating a speaker embedding for a human speaker, wherein generating the speaker embedding for the human speaker comprises: processing one or more instances of speaker audio data, corresponding to the human speaker, using a trained speaker embedding model, and generating the speaker embedding based on one or more instances of output each generated based on processing a respective of the one or more instances of speaker audio data using the trained speaker embedding model; receiving audio data that captures one or more utterances of the human speaker and that also captures one or more additional sounds that are not from the human speaker; generating a refined version of the audio data, wherein the refined version of the audio data isolates the one or more utterances of the human speaker from the one or more additional sounds that are not from the human speaker, and wherein generating the refined version of the audio data comprises: sequentially processing the audio data using a trained generative model and using the speaker embedding in determining activations for layers of the trained generative model during the sequentially processing, and sequentially generating, based on the sequentially processing and as direct output from the trained generative model, the refined version of the audio data; and performing further processing on the refined version of the audio data, performing the further processing comprising one or both of: performing voice-to-text processing on the refined version of the audio data to generate predicted text for the one or more utterances of the human speaker, and assigning a single given speaker label to one or more temporal portions of the audio data based on the one or more temporal portions corresponding to at least a threshold level of audio in the refined version of the audio data. generating a speaker embedding for a human speaker, wherein generating the speaker embedding for the human speaker comprises: processing one or more instances of speaker audio data, corresponding to the human speaker, using a trained speaker embedding model, and generating the speaker embedding based on one or more instances of output each generated based on processing a respective of the one or more instances of speaker audio data using the trained speaker embedding model; processing one or more instances of speaker audio data, corresponding to the human speaker, using a trained speaker embedding model, and generating the speaker embedding based on one or more instances of output each generated based on processing a respective of the one or more instances of speaker audio data using the trained speaker embedding model; receiving audio data that captures one or more utterances of the human speaker and that also captures one or more additional sounds that are not from the human speaker; generating a refined version of the audio data, wherein the refined version of the audio data isolates the one or more utterances of the human speaker from the one or more additional sounds that are not from the human speaker, and wherein generating the refined version of the audio data comprises: sequentially processing the audio data using a trained generative model and using the speaker embedding in determining activations for layers of the trained generative model during the sequentially processing, and sequentially generating, based on the sequentially processing and as direct output from the trained generative model, the refined version of the audio data; and sequentially processing the audio data using a trained generative model and using the speaker embedding in determining activations for layers of the trained generative model during the sequentially processing, and sequentially generating, based on the sequentially processing and as direct output from the trained generative model, the refined version of the audio data; and performing further processing on the refined version of the audio data, performing the further processing comprising one or both of: performing voice-to-text processing on the refined version of the audio data to generate predicted text for the one or more utterances of the human speaker, and assigning a single given speaker label to one or more temporal portions of the audio data based on the one or more temporal portions corresponding to at least a threshold level of audio in the refined version of the audio data. performing voice-to-text processing on the refined version of the audio data to generate predicted text for the one or more utterances of the human speaker, and assigning a single given speaker label to one or more temporal portions of the audio data based on the one or more temporal portions corresponding to at least a threshold level of audio in the refined version of the audio data.