A neural network processor circuit for performing neural network calculations for an artificial neural network having one or more network layers, the neural network processor circuit comprising: - a hierarchical compute fabric including a plurality of computation units organized into multiple hierarchical levels, wherein a lowest hierarchical level is a processing element for performing multiply and accumulate operations, wherein each hierarchical level contains its own dedicated local memory, each processing element operative to multiply an input data stream with weights stored in a shared memory portion of a hierarchical memory fabric; - a hierarchical memory fabric structured to address needs of multiple memory consumers, said memory consumers including storing inter-layer data, intra-layer information, and weights, wherein said hierarchical memory fabric comprises the plurality of dedicated local memories contained in the multiple hierarchical levels of said hierarchical compute fabric; - wherein second and higher hierarchical levels of said hierarchical compute fabric comprise a plurality of the lowest hierarchical level of said hierarchical compute fabric; wherein the weights are stored either in the dedicated local memory of the second hierarchical level of said hierarchical compute fabric or in the dedicated local memory of a third hierarchical level of said hierarchical compute fabric, wherein the dedicated local memories of said second and third hierarchical levels are dynamically allocated for weights and intermediate results; - a control fabric that is programmable and operative to configure and control said plurality of computation units, where control signaling is performed out-of-band without requiring data driven memory access; and wherein higher levels in said hierarchical compute fabric are more complex and include a smaller number of instantiations than lower levels that are less complex and include a large number of instantiations. - a hierarchical compute fabric including a plurality of computation units organized into multiple hierarchical levels, wherein a lowest hierarchical level is a processing element for performing multiply and accumulate operations, wherein each hierarchical level contains its own dedicated local memory, each processing element operative to multiply an input data stream with weights stored in a shared memory portion of a hierarchical memory fabric; - a hierarchical memory fabric structured to address needs of multiple memory consumers, said memory consumers including storing inter-layer data, intra-layer information, and weights, wherein said hierarchical memory fabric comprises the plurality of dedicated local memories contained in the multiple hierarchical levels of said hierarchical compute fabric; - wherein second and higher hierarchical levels of said hierarchical compute fabric comprise a plurality of the lowest hierarchical level of said hierarchical compute fabric; wherein the weights are stored either in the dedicated local memory of the second hierarchical level of said hierarchical compute fabric or in the dedicated local memory of a third hierarchical level of said hierarchical compute fabric, wherein the dedicated local memories of said second and third hierarchical levels are dynamically allocated for weights and intermediate results; - a control fabric that is programmable and operative to configure and control said plurality of computation units, where control signaling is performed out-of-band without requiring data driven memory access; and wherein higher levels in said hierarchical compute fabric are more complex and include a smaller number of instantiations than lower levels that are less complex and include a large number of instantiations.