A neural network system implemented by one or more computers, wherein the neural network system is configured to receive an input sequence comprising a plurality of inputs and to generate a neural network output from the input sequence, and wherein the neural network system comprises: a convolutional subnetwork comprising a plurality of dilated convolutional neural network layers, wherein the convolutional subnetwork is configured to, for each of the plurality of inputs: receive a current input sequence comprising the input and the inputs that precede the input in the input sequence, and process the current input sequence to generate an alternative representation for the input; and an output subnetwork, wherein the output subnetwork is configured to receive the alternative representations of the inputs and to process the alternative representations to generate the neural network output; wherein the neural network output comprises a plurality of sets of scores, wherein each set of scores includes a respective score for each of a plurality of possible outputs, and wherein the neural network system further comprises: a subsystem configured to select outputs in accordance with the sets of scores to generate an output sequence comprising a plurality of outputs; and wherein the dilated convolutional neural network layers are causal convolutional neural network layers; and wherein i) the input sequence is a sequence of audio data and the output sequence is a sequence of graphemes that is a grapheme representation of the input sequence, or ii) the input sequence is a sequence of audio data and the output sequence is a sequence of phonemes that is a phoneme representation of the input sequence, or iii) the input sequence is a sequence of phonemes or graphemes and the output sequence is a sequence of words in a first natural language that represents the input sequence, or iv) wherein the input sequence is a sequence of words in a first natural language and the output sequence is a sequence of words in a second natural language that is a translation of the input sequence into the second natural language. a convolutional subnetwork comprising a plurality of dilated convolutional neural network layers, wherein the convolutional subnetwork is configured to, for each of the plurality of inputs: receive a current input sequence comprising the input and the inputs that precede the input in the input sequence, and process the current input sequence to generate an alternative representation for the input; and receive a current input sequence comprising the input and the inputs that precede the input in the input sequence, and process the current input sequence to generate an alternative representation for the input; and an output subnetwork, wherein the output subnetwork is configured to receive the alternative representations of the inputs and to process the alternative representations to generate the neural network output; i) the input sequence is a sequence of audio data and the output sequence is a sequence of graphemes that is a grapheme representation of the input sequence, or ii) the input sequence is a sequence of audio data and the output sequence is a sequence of phonemes that is a phoneme representation of the input sequence, or iii) the input sequence is a sequence of phonemes or graphemes and the output sequence is a sequence of words in a first natural language that represents the input sequence, or iv) wherein the input sequence is a sequence of words in a first natural language and the output sequence is a sequence of words in a second natural language that is a translation of the input sequence into the second natural language.