Patent ID: 11887004
Assignee: GOOGLE LLC
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 24:
25. A method performed by one or more data processing apparatus, the method comprising:
training a neural network on a current batch of training examples, wherein the neural network comprises a batch renormalization layer between a first neural network layer and a second neural network layer, wherein the first neural network layer generates first layer outputs having a plurality of components, and wherein the batch renormalization layer is configured to, during training of the neural network on the current batch of training examples:
obtain respective current moving normalization statistics for each of the plurality of components that are based on previous first layer outputs generated by the first neural network layer during training of the neural network on previous batches of training examples;
receive a respective first layer output for each training example in the current batch;
compute respective current batch normalization statistics for each of the plurality of components from the first layer outputs for the training examples in the current batch;
determine respective transform function parameters for a transform function for each of the plurality of components from the current moving normalization statistics and the current batch normalization statistics; and

for each of the first layer outputs for each of the training examples in the current batch:
normalize each component of the first layer output using the current batch normalization statistics for the component to generate a normalized layer output for the training example,
apply the transform function to each component of the normalized layer output in accordance with the transform function parameters for the component to generate a renormalized layer output for the training example,
generate a batch renormalization layer output for the training example from the renormalized layer output, and
provide the batch renormalization layer output as an input to the second neural network layer.