A method performed by one or more computers of a text-to-speech system, the method comprising: obtaining, by the one or more computers, data indicating a text for text-to-speech synthesis; providing, by the one or more computers, data indicating a linguistic unit of the text as input to an encoder, the encoder outputting speech unit representations indicative of acoustic characteristics of a corresponding speech unit based on linguistic information, the encoder providing speech unit representations learned through machine learning training; receiving, by the one or more computers, a speech unit representation that the encoder outputs in response to receiving the data indicating the linguistic unit as input to the encoder; selecting, by the one or more computers, a speech unit to represent the linguistic unit, the speech unit being selected from among a collection of speech units based on the speech unit representation output by the encoder; and providing, by the one or more computers and as output of the text-to-speech system, audio data for a synthesized utterance of the text that includes the selected speech unit; wherein the encoder comprises a neural network that was trained as part of an autoencoder network that includes the encoder, a second encoder, and a decoder, by, in a phase of the machine learning training: the encoder producing speech unit representations in response to receiving data indicating linguistic units; the second encoder producing speech unit representations in response to receiving data indicating acoustic features of speech units; and the decoder generating output indicating acoustic features of speech units in response to receiving speech unit representations for the speech units from either of the encoder and the second encoder. obtaining, by the one or more computers, data indicating a text for text-to-speech synthesis; providing, by the one or more computers, data indicating a linguistic unit of the text as input to an encoder, the encoder outputting speech unit representations indicative of acoustic characteristics of a corresponding speech unit based on linguistic information, the encoder providing speech unit representations learned through machine learning training; receiving, by the one or more computers, a speech unit representation that the encoder outputs in response to receiving the data indicating the linguistic unit as input to the encoder; selecting, by the one or more computers, a speech unit to represent the linguistic unit, the speech unit being selected from among a collection of speech units based on the speech unit representation output by the encoder; and providing, by the one or more computers and as output of the text-to-speech system, audio data for a synthesized utterance of the text that includes the selected speech unit; wherein the encoder comprises a neural network that was trained as part of an autoencoder network that includes the encoder, a second encoder, and a decoder, by, in a phase of the machine learning training: the encoder producing speech unit representations in response to receiving data indicating linguistic units; the second encoder producing speech unit representations in response to receiving data indicating acoustic features of speech units; and the decoder generating output indicating acoustic features of speech units in response to receiving speech unit representations for the speech units from either of the encoder and the second encoder. the encoder producing speech unit representations in response to receiving data indicating linguistic units; the second encoder producing speech unit representations in response to receiving data indicating acoustic features of speech units; and the decoder generating output indicating acoustic features of speech units in response to receiving speech unit representations for the speech units from either of the encoder and the second encoder.