A device for performing training of a neural network via backward propagation, the device comprising: a memory configured to store input values and corresponding expected output values; at least one additional memory configured to store weights corresponding to connections between nodes of the neural network; and an array of systolic processing units that perform computations according to particular nodes of the neural network; wherein the array of systolic processing units are configured to perform computations for forward propagation of the input values through layers of the neural network to generate predicted output values by at least, for each particular processing unit of the array of systolic processing units : receiving input data; processing the input data using one or more of the weights to generate an activation output; and generating output data including the activation output and a tag identifying the particular processing unit ; wherein the array of systolic processing units are configured to perform computations for backward propagation of differences between the expected output values and corresponding ones of the predicted outputs at least partly based on the tags of the particular processing units , wherein the computations for backward propagation use the tags to retrieve respective weights corresponding to the particular processing unit from which the predicted outputs originated in forward propagation and wherein the backward propagation updates the weights; wherein particular arrangements of the array of systolic processing units are assigned to particular layers of the neural network, and wherein a first arrangement of first processing units of the array is configured to systolically pulse output data generated by the first processing units to a second arrangement of the systolic processing units in the array during the forward propagation. a memory configured to store input values and corresponding expected output values; at least one additional memory configured to store weights corresponding to connections between nodes of the neural network; and an array of systolic processing units that perform computations according to particular nodes of the neural network; wherein the array of systolic processing units are configured to perform computations for forward propagation of the input values through layers of the neural network to generate predicted output values by at least, for each particular processing unit of the array of systolic processing units : receiving input data; processing the input data using one or more of the weights to generate an activation output; and generating output data including the activation output and a tag identifying the particular processing unit ; receiving input data; processing the input data using one or more of the weights to generate an activation output; and generating output data including the activation output and a tag identifying the particular processing unit ; wherein the array of systolic processing units are configured to perform computations for backward propagation of differences between the expected output values and corresponding ones of the predicted outputs at least partly based on the tags of the particular processing units , wherein the computations for backward propagation use the tags to retrieve respective weights corresponding to the particular processing unit from which the predicted outputs originated in forward propagation and wherein the backward propagation updates the weights; wherein particular arrangements of the array of systolic processing units are assigned to particular layers of the neural network, and wherein a first arrangement of first processing units of the array is configured to systolically pulse output data generated by the first processing units to a second arrangement of the systolic processing units in the array during the forward propagation. <EOS>