An neural network system implemented by one or more computers, the neural network system comprising: an input to receive an image or image features of training examples; an output to generate scores for a set of object categories each representing a likelihood that that the image or image features of the training example input contain an image of an object belonging to the category; and at least one batch normalization layer between a first neural network layer and a second neural network layer, wherein the first neural network layer generates first layer outputs having a plurality of components, and wherein the batch normalization layer is configured to, during training of the neural network system on a batch of the training examples: receive a respective first layer output for each training example, i, in the batch; compute a plurality of normalization statistics for the batch from the first layer outputs; normalize each component of each first layer output using the normalization statistics to generate a respective normalized layer output x̂ k,i for each training example in the batch and transform each component of each respective normalized layer output according to y k,i = γ k x̂ k,i + A k, where γ k and A k are parameters for the k -th component adjusted during the training, to generate a respective batch normalization layer output y k,i for each of the training examples from the normalized layer outputs; and provide the batch normalization layer output as an input to the second neural network layer. an input to receive an image or image features of training examples; an output to generate scores for a set of object categories each representing a likelihood that that the image or image features of the training example input contain an image of an object belonging to the category; and at least one batch normalization layer between a first neural network layer and a second neural network layer, wherein the first neural network layer generates first layer outputs having a plurality of components, and wherein the batch normalization layer is configured to, during training of the neural network system on a batch of the training examples: receive a respective first layer output for each training example, i, in the batch; compute a plurality of normalization statistics for the batch from the first layer outputs; normalize each component of each first layer output using the normalization statistics to generate a respective normalized layer output x̂ k,i for each training example in the batch and transform each component of each respective normalized layer output according to y k,i = γ k x̂ k,i + A k, where γ k and A k are parameters for the k -th component adjusted during the training, to generate a respective batch normalization layer output y k,i for each of the training examples from the normalized layer outputs; and provide the batch normalization layer output as an input to the second neural network layer. receive a respective first layer output for each training example, i, in the batch; compute a plurality of normalization statistics for the batch from the first layer outputs; normalize each component of each first layer output using the normalization statistics to generate a respective normalized layer output x̂ k,i for each training example in the batch and transform each component of each respective normalized layer output according to y k,i = γ k x̂ k,i + A k, where γ k and A k are parameters for the k -th component adjusted during the training, to generate a respective batch normalization layer output y k,i for each of the training examples from the normalized layer outputs; and provide the batch normalization layer output as an input to the second neural network layer.