A computer-implemented method for training a neural network system having a plurality of parameters, the method comprising: obtaining a training sequence of audio data that comprises a respective audio sample at each of a plurality of time steps; processing the training sequence of audio data using a convolutional subnetwork of the neural network system comprising one or more audio-processing convolutional neural network layers, wherein the audio-processing convolutional neural network layers are causal convolutional neural network layers, and wherein the convolutional subnetwork is configured to, for each of the plurality of time steps: receive a current sequence of audio data that comprises the respective audio sample at each time step that precedes the time step in the training sequence of audio data, and process the current sequence of audio data to generate an alternative representation for the time step; processing the alternative representations for the plurality of time steps using an output layer of the neural network system, wherein the output layer is configured to, for each of the plurality of time steps: receive the alternative representation for the time step, and process the alternative representation for the time step to generate an output that defines a score distribution over a plurality of possible audio samples for the time step; and training the plurality of parameters of the neural network system using supervised learning based on, for each of the plurality of time steps: (i) the score distribution over the plurality of possible audio samples for the time step, and (ii) the audio sample at the time step in the training sequence of audio data. obtaining a training sequence of audio data that comprises a respective audio sample at each of a plurality of time steps; processing the training sequence of audio data using a convolutional subnetwork of the neural network system comprising one or more audio-processing convolutional neural network layers, wherein the audio-processing convolutional neural network layers are causal convolutional neural network layers, and wherein the convolutional subnetwork is configured to, for each of the plurality of time steps: receive a current sequence of audio data that comprises the respective audio sample at each time step that precedes the time step in the training sequence of audio data, and process the current sequence of audio data to generate an alternative representation for the time step; receive a current sequence of audio data that comprises the respective audio sample at each time step that precedes the time step in the training sequence of audio data, and process the current sequence of audio data to generate an alternative representation for the time step; processing the alternative representations for the plurality of time steps using an output layer of the neural network system, wherein the output layer is configured to, for each of the plurality of time steps: receive the alternative representation for the time step, and process the alternative representation for the time step to generate an output that defines a score distribution over a plurality of possible audio samples for the time step; and receive the alternative representation for the time step, and process the alternative representation for the time step to generate an output that defines a score distribution over a plurality of possible audio samples for the time step; and training the plurality of parameters of the neural network system using supervised learning based on, for each of the plurality of time steps: (i) the score distribution over the plurality of possible audio samples for the time step, and (ii) the audio sample at the time step in the training sequence of audio data. <EOS>