A method comprising: receiving, at data processing hardware, a text utterance having one or more words, each word having one or more syllables, each syllable having one or more phonemes; receiving, at the data processing hardware, as input to a vocoder model: prosodic features output from a prosody model that represent an intended prosody for the text utterance, the prosodic features comprising a duration, pitch contour, and energy contour for the text utterance; and a linguistic specification of the text utterance, the linguistic specification comprising sentence-level linguistic features for the text utterance, word-level linguistic features for each word of the text utterance, syllable-level linguistic features for each syllable of the text utterance, and phoneme-level linguistic features for each phoneme of the text utterance; predicting, by the data processing hardware, as output from the vocoder model, vocoder parameters based on the prosodic features output from the prosody model and the linguistic specification of the text utterance; and providing, by the data processing hardware, the predicted vocoder parameters output from the vocoder model and the prosodic features output from the prosody model to a parametric vocoder, the parametric vocoder configured to generate a synthesized speech representation of the text utterance and having the intended prosody. receiving, at data processing hardware, a text utterance having one or more words, each word having one or more syllables, each syllable having one or more phonemes; receiving, at the data processing hardware, as input to a vocoder model: prosodic features output from a prosody model that represent an intended prosody for the text utterance, the prosodic features comprising a duration, pitch contour, and energy contour for the text utterance; and a linguistic specification of the text utterance, the linguistic specification comprising sentence-level linguistic features for the text utterance, word-level linguistic features for each word of the text utterance, syllable-level linguistic features for each syllable of the text utterance, and phoneme-level linguistic features for each phoneme of the text utterance; prosodic features output from a prosody model that represent an intended prosody for the text utterance, the prosodic features comprising a duration, pitch contour, and energy contour for the text utterance; and a linguistic specification of the text utterance, the linguistic specification comprising sentence-level linguistic features for the text utterance, word-level linguistic features for each word of the text utterance, syllable-level linguistic features for each syllable of the text utterance, and phoneme-level linguistic features for each phoneme of the text utterance; predicting, by the data processing hardware, as output from the vocoder model, vocoder parameters based on the prosodic features output from the prosody model and the linguistic specification of the text utterance; and providing, by the data processing hardware, the predicted vocoder parameters output from the vocoder model and the prosodic features output from the prosody model to a parametric vocoder, the parametric vocoder configured to generate a synthesized speech representation of the text utterance and having the intended prosody.