A method for training a generative adversarial network, GAN, -based text-to-speech, TTS, model and a speech recognition model in unison, the method comprising: obtaining, by data processing hardware, a plurality of training text utterances, wherein: a first portion of the plurality of training text utterances comprises unspoken text utterances, each unspoken text utterance not paired with any corresponding spoken utterance; and a remaining second portion of the plurality of training text utterances comprises transcriptions in a set of spoken training utterances, each spoken training utterance comprising a corresponding transcription paired with a corresponding non-synthetic speech representation of the corresponding spoken training utterance; at each of a plurality of output steps for each training text utterance of the plurality of training text utterances: generating, by the data processing hardware, for output by the GAN-based TTS model, a synthetic speech representation of the corresponding training text utterance; and determining, by the data processing hardware, using an adversarial discriminator of the GAN, an adversarial loss term indicative of an amount of acoustic noise disparity in one of the non-synthetic speech representations selected from the set of spoken training utterances relative to the corresponding synthetic speech representation of the corresponding training text utterance; updating, by the data processing hardware, parameters of the GAN-based TTS model based on the adversarial loss term determined at each of the plurality of output steps for each training text utterance of the plurality of training text utterances; and training, by the data processing hardware, the speech recognition model on the synthetic speech representation generated at each of the plurality of output steps for each training text utterance of the plurality of training text utterances. obtaining, by data processing hardware, a plurality of training text utterances, wherein: a first portion of the plurality of training text utterances comprises unspoken text utterances, each unspoken text utterance not paired with any corresponding spoken utterance; and a remaining second portion of the plurality of training text utterances comprises transcriptions in a set of spoken training utterances, each spoken training utterance comprising a corresponding transcription paired with a corresponding non-synthetic speech representation of the corresponding spoken training utterance; a first portion of the plurality of training text utterances comprises unspoken text utterances, each unspoken text utterance not paired with any corresponding spoken utterance; and a remaining second portion of the plurality of training text utterances comprises transcriptions in a set of spoken training utterances, each spoken training utterance comprising a corresponding transcription paired with a corresponding non-synthetic speech representation of the corresponding spoken training utterance; at each of a plurality of output steps for each training text utterance of the plurality of training text utterances: generating, by the data processing hardware, for output by the GAN-based TTS model, a synthetic speech representation of the corresponding training text utterance; and determining, by the data processing hardware, using an adversarial discriminator of the GAN, an adversarial loss term indicative of an amount of acoustic noise disparity in one of the non-synthetic speech representations selected from the set of spoken training utterances relative to the corresponding synthetic speech representation of the corresponding training text utterance; generating, by the data processing hardware, for output by the GAN-based TTS model, a synthetic speech representation of the corresponding training text utterance; and determining, by the data processing hardware, using an adversarial discriminator of the GAN, an adversarial loss term indicative of an amount of acoustic noise disparity in one of the non-synthetic speech representations selected from the set of spoken training utterances relative to the corresponding synthetic speech representation of the corresponding training text utterance; updating, by the data processing hardware, parameters of the GAN-based TTS model based on the adversarial loss term determined at each of the plurality of output steps for each training text utterance of the plurality of training text utterances; and training, by the data processing hardware, the speech recognition model on the synthetic speech representation generated at each of the plurality of output steps for each training text utterance of the plurality of training text utterances.