A method comprising: receiving, at data processing hardware of a voice-enabled device, an indication of a microphone trigger event indicating a possible user interaction with the voice-enabled device through speech, the voice-enabled device having a microphone that, when open, is configured to capture speech for recognition by an automated speech recognition, ASR, system; in response to receiving the indication of the microphone trigger event: instructing, by the data processing hardware, the microphone to open or remain open for an open microphone duration window to capture an audio stream in an environment of the voice-enabled device; and providing, by the data processing hardware, the audio stream captured by the open microphone to the ASR system to perform ASR processing over the audio stream; and while the ASR system is performing the ASR processing over the audio stream captured by the open microphone: decaying, by the data processing hardware, a level of the ASR processing that the ASR system performs over the audio stream based on a function of the open microphone duration window; and instructing, by the data processing hardware, the ASR system to use the decayed level (204, 222) of the ASR processing over the audio stream captured by the open microphone. receiving, at data processing hardware of a voice-enabled device, an indication of a microphone trigger event indicating a possible user interaction with the voice-enabled device through speech, the voice-enabled device having a microphone that, when open, is configured to capture speech for recognition by an automated speech recognition, ASR, system; in response to receiving the indication of the microphone trigger event: instructing, by the data processing hardware, the microphone to open or remain open for an open microphone duration window to capture an audio stream in an environment of the voice-enabled device; and providing, by the data processing hardware, the audio stream captured by the open microphone to the ASR system to perform ASR processing over the audio stream; and instructing, by the data processing hardware, the microphone to open or remain open for an open microphone duration window to capture an audio stream in an environment of the voice-enabled device; and providing, by the data processing hardware, the audio stream captured by the open microphone to the ASR system to perform ASR processing over the audio stream; and while the ASR system is performing the ASR processing over the audio stream captured by the open microphone: decaying, by the data processing hardware, a level of the ASR processing that the ASR system performs over the audio stream based on a function of the open microphone duration window; and instructing, by the data processing hardware, the ASR system to use the decayed level (204, 222) of the ASR processing over the audio stream captured by the open microphone. decaying, by the data processing hardware, a level of the ASR processing that the ASR system performs over the audio stream based on a function of the open microphone duration window; and instructing, by the data processing hardware, the ASR system to use the decayed level (204, 222) of the ASR processing over the audio stream captured by the open microphone.