A method comprising: receiving, at data processing hardware , a text utterance having one or more words , each word having one or more syllables , each syllable having one or more phonemes ; receiving, at the data processing hardware , as input to a vocoder model : prosodic features output from a prosody model that represent an intended prosody for the text utterance , the prosodic features comprising a duration, pitch contour, and energy contour for the text utterance ; and a linguistic specification of the text utterance , the linguistic specification comprising sentence-level linguistic features for the text utterance , word-level linguistic features for each word of the text utterance , syllable-level linguistic features for each syllable of the text utterance , and phoneme-level linguistic features for each phoneme of the text utterance ; predicting, by the data processing hardware , as output from the vocoder model , vocoder parameters based on the prosodic features output from the prosody model and the linguistic specification of the text utterance ; and providing, by the data processing hardware , the predicted vocoder parameters output from the vocoder model and the prosodic features output from the prosody model to a parametric vocoder , the parametric vocoder configured to generate a synthesized speech representation of the text utterance and having the intended prosody. receiving, at data processing hardware , a text utterance having one or more words , each word having one or more syllables , each syllable having one or more phonemes ; receiving, at the data processing hardware , as input to a vocoder model : prosodic features output from a prosody model that represent an intended prosody for the text utterance , the prosodic features comprising a duration, pitch contour, and energy contour for the text utterance ; and a linguistic specification of the text utterance , the linguistic specification comprising sentence-level linguistic features for the text utterance , word-level linguistic features for each word of the text utterance , syllable-level linguistic features for each syllable of the text utterance , and phoneme-level linguistic features for each phoneme of the text utterance ; prosodic features output from a prosody model that represent an intended prosody for the text utterance , the prosodic features comprising a duration, pitch contour, and energy contour for the text utterance ; and a linguistic specification of the text utterance , the linguistic specification comprising sentence-level linguistic features for the text utterance , word-level linguistic features for each word of the text utterance , syllable-level linguistic features for each syllable of the text utterance , and phoneme-level linguistic features for each phoneme of the text utterance ; predicting, by the data processing hardware , as output from the vocoder model , vocoder parameters based on the prosodic features output from the prosody model and the linguistic specification of the text utterance ; and providing, by the data processing hardware , the predicted vocoder parameters output from the vocoder model and the prosodic features output from the prosody model to a parametric vocoder , the parametric vocoder configured to generate a synthesized speech representation of the text utterance and having the intended prosody. <EOS>