A method implemented by one or more processors, the method comprising: receiving data packets at one or more server devices , wherein the data packets are received at the one or more server devices in response to transmission of the data packets by client devices in multiple environments; identifying a group of the data packets, wherein identifying the group of the data packets comprises identifying the group of the data packets based on: time stamps of the data packets of the group, and the data packets of the group each indicating receipt of a single spoken utterance of a user; selecting a subset of two or more of the client devices based on the data packets of the group of the data packets being received from the client devices of the subset, wherein the subset of the client devices are each located in an environment of the multiple environments; in response to selecting the subset of the client devices: transmitting one or more commands to at least one of the client devices of the subset to cause the at least one of the client devices to provide user interface output related to a device isolation process; based on user interface input received in response to providing the user interface output, identifying a single client device, from the subset of the client devices, for isolation; and based on identifying the single client device for isolation: causing automated assistant clients of each of the client devices of the subset that are in addition to the single client device to be unresponsive to one or more spoken invocation phrases. receiving data packets at one or more server devices , wherein the data packets are received at the one or more server devices in response to transmission of the data packets by client devices in multiple environments; identifying a group of the data packets, wherein identifying the group of the data packets comprises identifying the group of the data packets based on: time stamps of the data packets of the group, and the data packets of the group each indicating receipt of a single spoken utterance of a user; time stamps of the data packets of the group, and the data packets of the group each indicating receipt of a single spoken utterance of a user; selecting a subset of two or more of the client devices based on the data packets of the group of the data packets being received from the client devices of the subset, wherein the subset of the client devices are each located in an environment of the multiple environments; in response to selecting the subset of the client devices: transmitting one or more commands to at least one of the client devices of the subset to cause the at least one of the client devices to provide user interface output related to a device isolation process; based on user interface input received in response to providing the user interface output, identifying a single client device, from the subset of the client devices, for isolation; and based on identifying the single client device for isolation: causing automated assistant clients of each of the client devices of the subset that are in addition to the single client device to be unresponsive to one or more spoken invocation phrases. <EOS>