A neural network system implemented by one or more computers and configured to receive a network training input and to process the network training input through each of a sequence of neural network layers to generate a network output from the network training input, wherein: the network training input is an image or features that have been extracted from the image and the network output are scores for each of a set of object categories, each score representing an estimated likelihood that the image contains an image of an object belonging to the category; or the network training input is text in a first language and the network output is a score for each of a set of pieces of text in another language, each score representing an estimated likelihood that the piece of text in the other language is a proper translation of the input text into the other language; or the network training input are features derived from spoken utterances or sequences of spoken utterances and the network output is a score for each of a set of pieces of text, each score representing an estimated likelihood that the piece of text is the correct transcript based on the features; the sequence of neural network layers comprising: an input layer configured to receive the network training input; a whitened neural network layer, wherein the whitened neural network layer is configured to perform operations comprising: receiving an input activation generated by a layer before the whitened neural network layer in the sequence; wherein the layer before the whitened neural network layer is a convolutional layer, fully-connected layer or another whitened neural network layer; processing the received activation in accordance with a set of whitening parameters to generate a whitened activation; processing the whitened activation in accordance with a set of layer parameters to generate an output activation; and providing the output activation as input to a neural network layer after the whitened neural network layer in the sequence. the network training input is an image or features that have been extracted from the image and the network output are scores for each of a set of object categories, each score representing an estimated likelihood that the image contains an image of an object belonging to the category; or the network training input is text in a first language and the network output is a score for each of a set of pieces of text in another language, each score representing an estimated likelihood that the piece of text in the other language is a proper translation of the input text into the other language; or the network training input are features derived from spoken utterances or sequences of spoken utterances and the network output is a score for each of a set of pieces of text, each score representing an estimated likelihood that the piece of text is the correct transcript based on the features; an input layer configured to receive the network training input; a whitened neural network layer, wherein the whitened neural network layer is configured to perform operations comprising: receiving an input activation generated by a layer before the whitened neural network layer in the sequence; wherein the layer before the whitened neural network layer is a convolutional layer, fully-connected layer or another whitened neural network layer; processing the received activation in accordance with a set of whitening parameters to generate a whitened activation; processing the whitened activation in accordance with a set of layer parameters to generate an output activation; and providing the output activation as input to a neural network layer after the whitened neural network layer in the sequence. receiving an input activation generated by a layer before the whitened neural network layer in the sequence; wherein the layer before the whitened neural network layer is a convolutional layer, fully-connected layer or another whitened neural network layer; processing the received activation in accordance with a set of whitening parameters to generate a whitened activation; processing the whitened activation in accordance with a set of layer parameters to generate an output activation; and providing the output activation as input to a neural network layer after the whitened neural network layer in the sequence.