A method comprising: receiving, at data processing hardware of a voice-enabled device, a fixed set of training utterances, each training utterance in the fixed set of training utterances comprising a corresponding transcription paired with a corresponding speech representation of the corresponding training utterance; sampling, by the data processing hardware, noisy audio data from an environment of the voice-enabled device; for each training utterance in the fixed set of training utterances: augmenting, by the data processing hardware, using the noisy audio data sampled from the environment of the voice-enabled device, the corresponding speech representation of the corresponding training utterance to generate one or more corresponding noisy audio samples; and pairing, by the data processing hardware, each of the one or more corresponding noisy audio samples with the corresponding transcription of the corresponding training utterance; and training, by the data processing hardware, a speech model on the one or more corresponding noisy audio samples generated for each speech representation in the fixed set of training utterances, wherein sampling the noisy audio data from the environment of the voice-enabled device comprises: randomly sampling noisy audio data from the environment of the voice-enabled device throughout a day; and applying weights to any of the noisy audio data sampled from the environment during contexts and/or time windows when a user of the voice-enabled device frequently interacts with the voice-enabled device. receiving, at data processing hardware of a voice-enabled device, a fixed set of training utterances, each training utterance in the fixed set of training utterances comprising a corresponding transcription paired with a corresponding speech representation of the corresponding training utterance; sampling, by the data processing hardware, noisy audio data from an environment of the voice-enabled device; for each training utterance in the fixed set of training utterances: augmenting, by the data processing hardware, using the noisy audio data sampled from the environment of the voice-enabled device, the corresponding speech representation of the corresponding training utterance to generate one or more corresponding noisy audio samples; and pairing, by the data processing hardware, each of the one or more corresponding noisy audio samples with the corresponding transcription of the corresponding training utterance; and training, by the data processing hardware, a speech model on the one or more corresponding noisy audio samples generated for each speech representation in the fixed set of training utterances, augmenting, by the data processing hardware, using the noisy audio data sampled from the environment of the voice-enabled device, the corresponding speech representation of the corresponding training utterance to generate one or more corresponding noisy audio samples; and pairing, by the data processing hardware, each of the one or more corresponding noisy audio samples with the corresponding transcription of the corresponding training utterance; and training, by the data processing hardware, a speech model on the one or more corresponding noisy audio samples generated for each speech representation in the fixed set of training utterances, wherein sampling the noisy audio data from the environment of the voice-enabled device comprises: randomly sampling noisy audio data from the environment of the voice-enabled device throughout a day; and applying weights to any of the noisy audio data sampled from the environment during contexts and/or time windows when a user of the voice-enabled device frequently interacts with the voice-enabled device. randomly sampling noisy audio data from the environment of the voice-enabled device throughout a day; and applying weights to any of the noisy audio data sampled from the environment during contexts and/or time windows when a user of the voice-enabled device frequently interacts with the voice-enabled device.