You could disable digital assistants through the speaker’s companion app if that’s supported. Many allow for the microphone to be turned off to disable the assistant.