Processing circuitry for a deep neural network (195, 200, 300, 400, 500), wherein the processing circuitry includes a low size, weight and power, system on chip, SoC, and a dynamic random access memory, DRAM, external to the SoC; the SoC comprising: a central processing unit, CPU,; a field programmable gate array, FPGA, with an on-chip static random access memory, SRAM; input ports and output ports; and a plurality of neural network processing elements, PEs organized into layers, each layer including at least one processing element, PE, the layers organized from a first layer to a second, hidden layer, to a third, last layer, at least one PE from the first layer is coupled to one or more of the input ports and to at least one PE of at least one higher numbered layer and the output ports, at least one PE from the second layer is coupled to at least one of the input ports and the at least one PE of the first layer and the at least one PE of at least one higher numbered layer and the output ports, and at least one PE of the third layer is coupled to the input ports and at least one PE of at least one lower-numbered layer and to the output ports, each of the plurality of neural network PEs including a weighted computational unit having the processing circuitry to interleave forward propagation of computational unit input values from the first layer to the last layer and backward propagation of output error values from the last layer to the first layer, wherein interleaving forward propagation and backward propagation includes, retrieving for a forward propagation and from the DRAM, one or more weight values associated with at least one PE of the last layer and storing the retrieved one or more weight values in the SRAM, and while a weight value associated with the last layer is still in the SRAM from the forward propagation, backward propagating an output error value from an output of the last layer to an input of the last layer using the weight value, providing a result of the backward propagating to the input/output ports; and updating the one or more weight values stored in the SRAM associated with the at least one PE based on the backward propagation. a central processing unit, CPU,; a field programmable gate array, FPGA, with an on-chip static random access memory, SRAM; input ports and output ports; and a plurality of neural network processing elements, PEs organized into layers, each layer including at least one processing element, PE, the layers organized from a first layer to a second, hidden layer, to a third, last layer, at least one PE from the first layer is coupled to one or more of the input ports and to at least one PE of at least one higher numbered layer and the output ports, at least one PE from the second layer is coupled to at least one of the input ports and the at least one PE of the first layer and the at least one PE of at least one higher numbered layer and the output ports, and at least one PE of the third layer is coupled to the input ports and at least one PE of at least one lower-numbered layer and to the output ports, each of the plurality of neural network PEs including a weighted computational unit having the processing circuitry to interleave forward propagation of computational unit input values from the first layer to the last layer and backward propagation of output error values from the last layer to the first layer, wherein interleaving forward propagation and backward propagation includes, retrieving for a forward propagation and from the DRAM, one or more weight values associated with at least one PE of the last layer and storing the retrieved one or more weight values in the SRAM, and while a weight value associated with the last layer is still in the SRAM from the forward propagation, backward propagating an output error value from an output of the last layer to an input of the last layer using the weight value, providing a result of the backward propagating to the input/output ports; and updating the one or more weight values stored in the SRAM associated with the at least one PE based on the backward propagation.