A method comprising: receiving, at data processing hardware, a first instance of raw audio data corresponding to a voice command for an assistant-enabled device to facilitate an audio-based communication between a user of the assistant-enabled device and a recipient, the voice command spoken by the user and captured by the assistant-enabled device; receiving, at the data processing hardware, a second instance of the raw audio data corresponding to an utterance of audible contents (126, 178) for the audio-based communication spoken by the user and captured by the assistant-enabled device, the second instance of the raw audio data capturing one or more additional sounds that are not spoken by the user; executing, by the data processing hardware, a voice filtering recognition routine to determine whether to activate voice filtering for at least a voice of the user in the audio-based communication based on the first instance of the raw audio data; when the voice filtering recognition routine determines to activate voice filtering for at least the voice of the user: obtaining, by the data processing hardware, a respective speaker embedding of the user that represents voice characteristics for the user; and processing, by the data processing hardware, using the respective speaker embedding of the user, the second instance of the raw audio data to generate enhanced audio data for the audio-based communication that isolates the utterance of the audible contents (126, 178) spoken by the user and excludes at least a portion of the one or more additional sounds that are not spoken by the user; and transmitting, by the data processing hardware, the enhanced audio data to a recipient device associated with the recipient, the enhanced audio data when received by the recipient device, causing the recipient device to audibly output the utterance of the audible contents (126, 178) spoken by the user. receiving, at data processing hardware, a first instance of raw audio data corresponding to a voice command for an assistant-enabled device to facilitate an audio-based communication between a user of the assistant-enabled device and a recipient, the voice command spoken by the user and captured by the assistant-enabled device; receiving, at the data processing hardware, a second instance of the raw audio data corresponding to an utterance of audible contents (126, 178) for the audio-based communication spoken by the user and captured by the assistant-enabled device, the second instance of the raw audio data capturing one or more additional sounds that are not spoken by the user; executing, by the data processing hardware, a voice filtering recognition routine to determine whether to activate voice filtering for at least a voice of the user in the audio-based communication based on the first instance of the raw audio data; when the voice filtering recognition routine determines to activate voice filtering for at least the voice of the user: obtaining, by the data processing hardware, a respective speaker embedding of the user that represents voice characteristics for the user; and processing, by the data processing hardware, using the respective speaker embedding of the user, the second instance of the raw audio data to generate enhanced audio data for the audio-based communication that isolates the utterance of the audible contents (126, 178) spoken by the user and excludes at least a portion of the one or more additional sounds that are not spoken by the user; and obtaining, by the data processing hardware, a respective speaker embedding of the user that represents voice characteristics for the user; and processing, by the data processing hardware, using the respective speaker embedding of the user, the second instance of the raw audio data to generate enhanced audio data for the audio-based communication that isolates the utterance of the audible contents (126, 178) spoken by the user and excludes at least a portion of the one or more additional sounds that are not spoken by the user; and transmitting, by the data processing hardware, the enhanced audio data to a recipient device associated with the recipient, the enhanced audio data when received by the recipient device, causing the recipient device to audibly output the utterance of the audible contents (126, 178) spoken by the user.