A method for generating a respective neural network output for each of a plurality of activation inputs, wherein the generating comprises processing each activation input through each of a plurality of neural network layers in accordance with a plurality of weight inputs to generate the respective neural network output for the activation input, wherein the neural network layers are arranged in a directed graph structure, and wherein each neural network layer has a respective batch size associated therewith, the method comprising, for each of the neural network layers: receiving, by a special purpose hardware circuit, a plurality of activation inputs to be processed at the neural network layer; forming, by the special purpose hardware circuit, one or more batches of activation inputs from the plurality of activation inputs, each batch having a number of activation inputs equal to the respective batch size associated with the neural network layer; selecting, by the special purpose hardware circuit, a number of the one or more batches of activation inputs to process such that a count of the activation inputs in the number of the one or more batches is greater than or equal to the respective associated batch size of a subsequent layer in the directed graph structure, where the respective associated batch size is based at least on a weight reuse value, the weight reuse value representing a number of times that weight inputs need to be reused, so that a total compute time of the neural network output values for the respective activation input while reusing the weight inputs at a systolic array in the special purpose hardware circuit becomes greater than or equal to a fetch time of accessing a set of weight inputs for the subsequent layer; and processing, by the special purpose hardware circuit, the number of the one or more batches of activation inputs, using the systolic array, to generate the respective neural network layer output, where processing the number of the one or more batches of activation inputs comprises reusing the weight inputs with multiple activation inputs in the number of the one or more batches of activation inputs by computing accumulated values using the systolic array. receiving, by a special purpose hardware circuit, a plurality of activation inputs to be processed at the neural network layer; forming, by the special purpose hardware circuit, one or more batches of activation inputs from the plurality of activation inputs, each batch having a number of activation inputs equal to the respective batch size associated with the neural network layer; selecting, by the special purpose hardware circuit, a number of the one or more batches of activation inputs to process such that a count of the activation inputs in the number of the one or more batches is greater than or equal to the respective associated batch size of a subsequent layer in the directed graph structure, where the respective associated batch size is based at least on a weight reuse value, the weight reuse value representing a number of times that weight inputs need to be reused, so that a total compute time of the neural network output values for the respective activation input while reusing the weight inputs at a systolic array in the special purpose hardware circuit becomes greater than or equal to a fetch time of accessing a set of weight inputs for the subsequent layer; and processing, by the special purpose hardware circuit, the number of the one or more batches of activation inputs, using the systolic array, to generate the respective neural network layer output, where processing the number of the one or more batches of activation inputs comprises reusing the weight inputs with multiple activation inputs in the number of the one or more batches of activation inputs by computing accumulated values using the systolic array.