A circuit for performing neural network computations for a neural network comprising a plurality of layers, the circuit comprising: activation circuitry configured to receive a vector of accumulated values and configured to apply a function to each accumulated value to generate a vector of activation values; and normalization circuitry coupled to the activation circuitry and configured to receive activation values of the vector of activation values and to generate a respective normalized value for each activation value, the normalization circuitry comprising a plurality of normalization register columns (604, 606. 608), each normalization register column comprising a plurality of normalization registers (616, 632, 640), and wherein at least one normalization register in each normalization register column is configured to receive a respective activation value for generating at least the respective normalized value for the activation value, wherein the plurality of normalization registers (616, 632, 640) in each normalization register column (604, 606, 608) are connected in series, each normalization register column is configured to receive the respective activation value, wherein the normalization circuitry is configured to form groups (624, 628) around one or more normalization registers, each group corresponding to a normalization unit, and each normalization unit configured to calculate a respective normalized value for the respective activation value; wherein the normalization circuitry is adapted to shift the activated value to a subsequent normalization register (632, 634, 640) in the normalization register column (604, 606) until all activated values of the respective group necessary to calculate the respective normalized value for the activation value are loaded. activation circuitry configured to receive a vector of accumulated values and configured to apply a function to each accumulated value to generate a vector of activation values; and normalization circuitry coupled to the activation circuitry and configured to receive activation values of the vector of activation values and to generate a respective normalized value for each activation value, the normalization circuitry comprising a plurality of normalization register columns (604, 606. 608), each normalization register column comprising a plurality of normalization registers (616, 632, 640), and wherein at least one normalization register in each normalization register column is configured to receive a respective activation value for generating at least the respective normalized value for the activation value, wherein the plurality of normalization registers (616, 632, 640) in each normalization register column (604, 606, 608) are connected in series, each normalization register column is configured to receive the respective activation value, wherein the normalization circuitry is configured to form groups (624, 628) around one or more normalization registers, each group corresponding to a normalization unit, and each normalization unit configured to calculate a respective normalized value for the respective activation value; wherein the normalization circuitry is adapted to shift the activated value to a subsequent normalization register (632, 634, 640) in the normalization register column (604, 606) until all activated values of the respective group necessary to calculate the respective normalized value for the activation value are loaded. wherein the plurality of normalization registers (616, 632, 640) in each normalization register column (604, 606, 608) are connected in series, each normalization register column is configured to receive the respective activation value, wherein the normalization circuitry is configured to form groups (624, 628) around one or more normalization registers, each group corresponding to a normalization unit, and each normalization unit configured to calculate a respective normalized value for the respective activation value; wherein the normalization circuitry is adapted to shift the activated value to a subsequent normalization register (632, 634, 640) in the normalization register column (604, 606) until all activated values of the respective group necessary to calculate the respective normalized value for the activation value are loaded.