A neural network system implemented by one or more computers, the neural network system comprising: a batch normalization layer between a first neural network layer and a second neural network layer, wherein the first neural network layer generates first layer outputs having a plurality of components, and wherein the batch normalization layer is configured to, during training of the neural network system on a batch of training examples: receive a respective first layer output for each training example in the batch; compute a plurality of normalization statistics for the batch from the first layer outputs; normalize each component of each first layer output using the normalization statistics to generate a respective normalized layer output for each training example in the batch; generate a respective batch normalization layer output for each of the training examples from the normalized layer outputs; and provide the batch normalization layer output as an input to the second neural network layer; wherein the plurality of components of the first layer output are indexed by dimension, and wherein computing a plurality of normalization statistics for the first layer outputs comprises: computing, for each of the dimensions, a mean of the components of the first layer outputs in the dimension; and computing, for each of the dimensions, a standard deviation of the components of the first layer outputs in the dimension. a batch normalization layer between a first neural network layer and a second neural network layer, wherein the first neural network layer generates first layer outputs having a plurality of components, and wherein the batch normalization layer is configured to, during training of the neural network system on a batch of training examples: receive a respective first layer output for each training example in the batch; compute a plurality of normalization statistics for the batch from the first layer outputs; normalize each component of each first layer output using the normalization statistics to generate a respective normalized layer output for each training example in the batch; generate a respective batch normalization layer output for each of the training examples from the normalized layer outputs; and provide the batch normalization layer output as an input to the second neural network layer; receive a respective first layer output for each training example in the batch; compute a plurality of normalization statistics for the batch from the first layer outputs; normalize each component of each first layer output using the normalization statistics to generate a respective normalized layer output for each training example in the batch; generate a respective batch normalization layer output for each of the training examples from the normalized layer outputs; and provide the batch normalization layer output as an input to the second neural network layer; wherein the plurality of components of the first layer output are indexed by dimension, and wherein computing a plurality of normalization statistics for the first layer outputs comprises: computing, for each of the dimensions, a mean of the components of the first layer outputs in the dimension; and computing, for each of the dimensions, a standard deviation of the components of the first layer outputs in the dimension. computing, for each of the dimensions, a mean of the components of the first layer outputs in the dimension; and computing, for each of the dimensions, a standard deviation of the components of the first layer outputs in the dimension.