A method implemented by one or more processors of a client device, the method comprising: capturing, via at least one microphone of the client device, audio data that captures a spoken utterance of a user; processing the audio data to generate current text that corresponds to the spoken utterance, wherein processing the audio data to generate the current text utilizes a voice-to-text model stored locally on the client device; accessing a text-response map stored locally on the client device, wherein the text-response map includes a plurality of mappings, each of the mappings defining a corresponding direct relationship between corresponding text and a corresponding response based on the corresponding text being previously generated from previous audio data captured by the client device and based on the corresponding response being previously received from a remote system in response to transmitting, to the remote system, at least one of the previous audio data and the corresponding text; determining whether any of the corresponding texts of the text-response map matches the current text; and in response to determining that a given text, of the corresponding texts of the text-response map, matches the current text: selecting a given response of the corresponding responses of the text-response map, where selecting the given response is based on the text-response map including a mapping, of the mappings, that defines the given response as having a direct relationship with the given text; and causing the given response to be rendered via one or more user interface output devices associated with the client device; wherein the method further comprises: transmitting the audio data or the current text to the remote system; receiving, from the remote system in response to transmitting the audio data or the current text, a server response that is responsive to the spoken utterance; comparing the server response to the given response; and updating the text-response map based on the comparison. capturing, via at least one microphone of the client device, audio data that captures a spoken utterance of a user; processing the audio data to generate current text that corresponds to the spoken utterance, wherein processing the audio data to generate the current text utilizes a voice-to-text model stored locally on the client device; accessing a text-response map stored locally on the client device, wherein the text-response map includes a plurality of mappings, each of the mappings defining a corresponding direct relationship between corresponding text and a corresponding response based on the corresponding text being previously generated from previous audio data captured by the client device and based on the corresponding response being previously received from a remote system in response to transmitting, to the remote system, at least one of the previous audio data and the corresponding text; determining whether any of the corresponding texts of the text-response map matches the current text; and in response to determining that a given text, of the corresponding texts of the text-response map, matches the current text: selecting a given response of the corresponding responses of the text-response map, where selecting the given response is based on the text-response map including a mapping, of the mappings, that defines the given response as having a direct relationship with the given text; and causing the given response to be rendered via one or more user interface output devices associated with the client device; selecting a given response of the corresponding responses of the text-response map, where selecting the given response is based on the text-response map including a mapping, of the mappings, that defines the given response as having a direct relationship with the given text; and causing the given response to be rendered via one or more user interface output devices associated with the client device; wherein the method further comprises: transmitting the audio data or the current text to the remote system; receiving, from the remote system in response to transmitting the audio data or the current text, a server response that is responsive to the spoken utterance; comparing the server response to the given response; and updating the text-response map based on the comparison. transmitting the audio data or the current text to the remote system; receiving, from the remote system in response to transmitting the audio data or the current text, a server response that is responsive to the spoken utterance; comparing the server response to the given response; and updating the text-response map based on the comparison.