A method implemented by one or more processors, the method comprising: receiving, from a client device and via a network, an automated assistant request that includes: audio data that captures spoken input of a user, wherein the audio data is captured at one or more microphones of the client device, and a text dependent user measure generated locally at the client device using a TD speaker recognition model stored locally at the client device and using a TD speaker embedding stored locally at the client device, the TD speaker embedding being for a particular user; processing at least a portion of the audio data using a text independent (Tl) speaker recognition model to generate Tl output; determining a Tl user measure by comparing the Tl output with a Tl speaker embedding that is associated with the automated assistant request, and that is for the particular user; determining whether the particular user spoke the spoken input using both the TD user measure and the Tl user measure; in response to determining the spoken input is spoken by the particular user: generating responsive content that is responsive to the spoken input and that is customized for the particular user; and transmitting the responsive content to the client device to cause the client device to render output based on the responsive content. receiving, from a client device and via a network, an automated assistant request that includes: audio data that captures spoken input of a user, wherein the audio data is captured at one or more microphones of the client device, and a text dependent user measure generated locally at the client device using a TD speaker recognition model stored locally at the client device and using a TD speaker embedding stored locally at the client device, the TD speaker embedding being for a particular user; audio data that captures spoken input of a user, wherein the audio data is captured at one or more microphones of the client device, and a text dependent user measure generated locally at the client device using a TD speaker recognition model stored locally at the client device and using a TD speaker embedding stored locally at the client device, the TD speaker embedding being for a particular user; processing at least a portion of the audio data using a text independent (Tl) speaker recognition model to generate Tl output; determining a Tl user measure by comparing the Tl output with a Tl speaker embedding that is associated with the automated assistant request, and that is for the particular user; determining whether the particular user spoke the spoken input using both the TD user measure and the Tl user measure; in response to determining the spoken input is spoken by the particular user: generating responsive content that is responsive to the spoken input and that is customized for the particular user; and transmitting the responsive content to the client device to cause the client device to render output based on the responsive content. generating responsive content that is responsive to the spoken input and that is customized for the particular user; and transmitting the responsive content to the client device to cause the client device to render output based on the responsive content.