A method implemented by one or more processors, the method comprising: determining that warm word arbitration is to be initiated between a first assistant device and one or more additional assistant devices, the one or more additional assistant devices including a second assistant device, and the first assistant device and the one or more additional assistant devices being included in a group of assistant devices; in response to determining that warm word arbitration is to be initiated, performing warm word arbitration, the warm word arbitration comprising: broadcasting, by the first assistant device, to the one or more additional assistant devices in the group of assistant devices, an active set of warm words for the first assistant device; for each of the one or more additional assistant devices in the group of assistant devices, receiving, from the additional assistant device, an active set of warm words for the additional assistant device; identifying a matching warm word based on the matching warm word being included in the active set of warm words for the first assistant device and being included in the active set of warm words for the second assistant device; and enabling or disabling detection of the matching warm word by the first assistant device, in response to identifying the matching warm word. determining that warm word arbitration is to be initiated between a first assistant device and one or more additional assistant devices, the one or more additional assistant devices including a second assistant device, and the first assistant device and the one or more additional assistant devices being included in a group of assistant devices; in response to determining that warm word arbitration is to be initiated, performing warm word arbitration, the warm word arbitration comprising: broadcasting, by the first assistant device, to the one or more additional assistant devices in the group of assistant devices, an active set of warm words for the first assistant device; for each of the one or more additional assistant devices in the group of assistant devices, receiving, from the additional assistant device, an active set of warm words for the additional assistant device; identifying a matching warm word based on the matching warm word being included in the active set of warm words for the first assistant device and being included in the active set of warm words for the second assistant device; and enabling or disabling detection of the matching warm word by the first assistant device, in response to identifying the matching warm word. broadcasting, by the first assistant device, to the one or more additional assistant devices in the group of assistant devices, an active set of warm words for the first assistant device; for each of the one or more additional assistant devices in the group of assistant devices, receiving, from the additional assistant device, an active set of warm words for the additional assistant device; identifying a matching warm word based on the matching warm word being included in the active set of warm words for the first assistant device and being included in the active set of warm words for the second assistant device; and enabling or disabling detection of the matching warm word by the first assistant device, in response to identifying the matching warm word.