A speech recognition neural network system implemented by one or more computers, wherein the neural network system is configured to receive an input sequence comprising a plurality of inputs and to generate a neural network output from the input sequence, and wherein the neural network system comprises: a convolutional subnetwork comprising a plurality of dilated convolutional neural network layers, wherein the convolutional subnetwork is configured to, for each of the plurality of inputs: receive a current input sequence comprising the input and the inputs that precede the input in the input sequence, and process the current input sequence to generate an alternative representation for the input; and an output subnetwork, wherein the output subnetwork is configured to receive the alternative representations of the inputs and to process the alternative representations to generate the neural network output; wherein the neural network output comprises a plurality of sets of scores, wherein each set of scores includes a respective score for each of a plurality of possible outputs, and wherein the neural network system further comprises: a subsystem configured to select outputs in accordance with the sets of scores to generate an output sequence comprising a plurality of outputs; and wherein generating the output sequence by processing the input sequence performs speech recognition. wherein the neural network system is configured to receive an input sequence comprising a plurality of inputs and to generate a neural network output from the input sequence, and wherein the neural network system comprises: a convolutional subnetwork comprising a plurality of dilated convolutional neural network layers, wherein the convolutional subnetwork is configured to, for each of the plurality of inputs: receive a current input sequence comprising the input and the inputs that precede the input in the input sequence, and process the current input sequence to generate an alternative representation for the input; and an output subnetwork, wherein the output subnetwork is configured to receive the alternative representations of the inputs and to process the alternative representations to generate the neural network output; a convolutional subnetwork comprising a plurality of dilated convolutional neural network layers, wherein the convolutional subnetwork is configured to, for each of the plurality of inputs: receive a current input sequence comprising the input and the inputs that precede the input in the input sequence, and process the current input sequence to generate an alternative representation for the input; and receive a current input sequence comprising the input and the inputs that precede the input in the input sequence, and process the current input sequence to generate an alternative representation for the input; and an output subnetwork, wherein the output subnetwork is configured to receive the alternative representations of the inputs and to process the alternative representations to generate the neural network output; wherein the neural network output comprises a plurality of sets of scores, wherein each set of scores includes a respective score for each of a plurality of possible outputs, and wherein the neural network system further comprises: a subsystem configured to select outputs in accordance with the sets of scores to generate an output sequence comprising a plurality of outputs; and wherein generating the output sequence by processing the input sequence performs speech recognition.