A computer-implemented method when executed by data processing hardware causes the data processing hardware to perform operations comprising: pre-training an encoder of a speech synthesis model by: feeding the encoder a plurality of training examples, each training example represented as a sequence of training grapheme token embeddings corresponding to a training sequence of words and a sequence of training phoneme tokens corresponding to the same training sequence of words; masking a training phoneme token from the sequence of training phoneme tokens for a respective word from the training sequence of words; and masking a training grapheme token embedding from the sequence of training phoneme tokens for the respective word from the training sequence of words; receiving, at the pre-trained encoder of the speech synthesis model, a text input comprising a sequence of words represented as an input encoder embedding, the input encoder embedding comprising a plurality of tokens, the plurality of tokens comprising a first set of grapheme token embeddings representing the text input as respective graphemes and a second set of phoneme tokens representing the text input as respective phonemes; for each respective phoneme token of the second set of phoneme tokens: identifying, by a transformer of the pre-trained encoder, a respective word of the sequence of words corresponding to the respective phoneme token; and determining, by the transformer of the pre-trained encoder, a respective grapheme token embedding representing the respective word of the sequence of words corresponding to the respective phoneme token; and generating, by the transformer of the pre-trained encoder, an output encoder embedding (Vc) based on the correspondence between each respective phoneme token and the respective grapheme token embedding determined to represent a same respective word as the respective phoneme token; decoding, by a decoder, the output encoder embedding (Vc) into synthesized speech. pre-training an encoder of a speech synthesis model by: feeding the encoder a plurality of training examples, each training example represented as a sequence of training grapheme token embeddings corresponding to a training sequence of words and a sequence of training phoneme tokens corresponding to the same training sequence of words; masking a training phoneme token from the sequence of training phoneme tokens for a respective word from the training sequence of words; and masking a training grapheme token embedding from the sequence of training phoneme tokens for the respective word from the training sequence of words; feeding the encoder a plurality of training examples, each training example represented as a sequence of training grapheme token embeddings corresponding to a training sequence of words and a sequence of training phoneme tokens corresponding to the same training sequence of words; masking a training phoneme token from the sequence of training phoneme tokens for a respective word from the training sequence of words; and masking a training grapheme token embedding from the sequence of training phoneme tokens for the respective word from the training sequence of words; receiving, at the pre-trained encoder of the speech synthesis model, a text input comprising a sequence of words represented as an input encoder embedding, the input encoder embedding comprising a plurality of tokens, the plurality of tokens comprising a first set of grapheme token embeddings representing the text input as respective graphemes and a second set of phoneme tokens representing the text input as respective phonemes; for each respective phoneme token of the second set of phoneme tokens: identifying, by a transformer of the pre-trained encoder, a respective word of the sequence of words corresponding to the respective phoneme token; and determining, by the transformer of the pre-trained encoder, a respective grapheme token embedding representing the respective word of the sequence of words corresponding to the respective phoneme token; and identifying, by a transformer of the pre-trained encoder, a respective word of the sequence of words corresponding to the respective phoneme token; and determining, by the transformer of the pre-trained encoder, a respective grapheme token embedding representing the respective word of the sequence of words corresponding to the respective phoneme token; and generating, by the transformer of the pre-trained encoder, an output encoder embedding (Vc) based on the correspondence between each respective phoneme token and the respective grapheme token embedding determined to represent a same respective word as the respective phoneme token; decoding, by a decoder, the output encoder embedding (Vc) into synthesized speech.