A method implemented by one or more processors, the method comprising: receiving an instance of a spoken utterance at a client device that is operating in an environment with one or more additional client devices that also received the instance of the spoken utterance, wherein each of the client device and the one or more additional client devices is connected to a local network and includes an assistant application that is responsive to the spoken utterance; providing, by the client device and based on receiving the instance of the spoken utterance at the client device, user interface output that provides a prompt, to a user, related to whether the client device is to be responsive to invocations of the assistant application, wherein each of the one or more additional client devices provides a respective prompt in response to receiving the instance of the spoken utterance, and wherein the prompt provided at the client device is unique relative to each respective prompt provided at each of the one or more additional client devices; determining, based on a response of the user to the prompt, that the client device is not to be responsive to invocations of the assistant application, and that a device of the one or more additional devices is to be isolated for being responsive to assistant invocations; and in response to determining that the client device is not to be responsive to invocations of the assistant application: causing the assistant application of the client device to temporarily limit responsiveness to one or more spoken invocations of the assistant application. receiving an instance of a spoken utterance at a client device that is operating in an environment with one or more additional client devices that also received the instance of the spoken utterance, wherein each of the client device and the one or more additional client devices is connected to a local network and includes an assistant application that is responsive to the spoken utterance; environment with one or more additional client devices that also received the instance of the spoken utterance, wherein each of the client device and the one or more additional client devices is connected to a local network and includes an assistant application that is responsive to the spoken utterance; providing, by the client device and based on receiving the instance of the spoken utterance at the client device, user interface output that provides a prompt, to a user, related to whether the client device is to be responsive to invocations of the assistant application, wherein each of the one or more additional client devices provides a respective prompt in response to receiving the instance of the spoken utterance, and wherein the prompt provided at the client device is unique relative to each respective prompt provided at each of the one or more additional client devices; utterance at the client device, user interface output that provides a prompt, to a user, related to whether the client device is to be responsive to invocations of the assistant application, wherein each of the one or more additional client devices provides a respective prompt in response to receiving the instance of the spoken utterance, and wherein the prompt provided at the client device is unique relative to each respective prompt provided at each of the one or more additional client devices; determining, based on a response of the user to the prompt, that the client device is not to be responsive to invocations of the assistant application, and that a device of the one or more additional devices is to be isolated for being responsive to assistant invocations; and in response to determining that the client device is not to be responsive to invocations of the assistant application: causing the assistant application of the client device to temporarily limit responsiveness to one or more spoken invocations of the assistant application.