A method of exploiting activation sparsity in deep neural networks, comprising: retrieving an activation tensor and a weight tensor where the activation tensor is a sparse activation tensor comprising a number of non-zero activations and a number of zero value activations; generating a compressed activation tensor comprising the non-zero activations of the activation tensor, where the compressed activation tensor has fewer columns than the activation tensor based on a source-column constraint, wherein generating the compressed activation tensor comprises: moving only the non-zero activations from first memory locations of source-columns of the sparse activation tensor to second memory locations of compressed columns of a first-in-first-out, FIFO, buffer to meet, but not exceed, the source-column constraint, wherein the second locations in the FIFO buffer are mapped to empty vector lanes of a multiply-accumulate, MAC, hardware during a clock cycle; and processing the compressed activation tensor and the weight tensor using the MAC hardware to generate an output tensor, wherein the processing comprises: moving the non-zero activations from the second memory locations of the compressed columns of the FIFO buffer to third memory locations in tightly coupled memory for processing on said empty vector lanes of the MAC hardware during the clock cycle; redistributing at least one non-zero activation from one of the third memory locations to a fourth memory location in the tightly coupled memory for processing during a subsequent clock cycle when the source-column constraint will be exceeded; and inserting an artificial zero activation in the memory location of the at least one redistributed non-zero activation. retrieving an activation tensor and a weight tensor where the activation tensor is a sparse activation tensor comprising a number of non-zero activations and a number of zero value activations; generating a compressed activation tensor comprising the non-zero activations of the activation tensor, where the compressed activation tensor has fewer columns than the activation tensor based on a source-column constraint, wherein generating the compressed activation tensor comprises: moving only the non-zero activations from first memory locations of source-columns of the sparse activation tensor to second memory locations of compressed columns of a first-in-first-out, FIFO, buffer to meet, but not exceed, the source-column constraint, wherein the second locations in the FIFO buffer are mapped to empty vector lanes of a multiply-accumulate, MAC, hardware during a clock cycle; and moving only the non-zero activations from first memory locations of source-columns of the sparse activation tensor to second memory locations of compressed columns of a first-in-first-out, FIFO, buffer to meet, but not exceed, the source-column constraint, wherein the second locations in the FIFO buffer are mapped to empty vector lanes of a multiply-accumulate, MAC, hardware during a clock cycle; and processing the compressed activation tensor and the weight tensor using the MAC hardware to generate an output tensor, wherein the processing comprises: moving the non-zero activations from the second memory locations of the compressed columns of the FIFO buffer to third memory locations in tightly coupled memory for processing on said empty vector lanes of the MAC hardware during the clock cycle; redistributing at least one non-zero activation from one of the third memory locations to a fourth memory location in the tightly coupled memory for processing during a subsequent clock cycle when the source-column constraint will be exceeded; and inserting an artificial zero activation in the memory location of the at least one redistributed non-zero activation. moving the non-zero activations from the second memory locations of the compressed columns of the FIFO buffer to third memory locations in tightly coupled memory for processing on said empty vector lanes of the MAC hardware during the clock cycle; redistributing at least one non-zero activation from one of the third memory locations to a fourth memory location in the tightly coupled memory for processing during a subsequent clock cycle when the source-column constraint will be exceeded; and inserting an artificial zero activation in the memory location of the at least one redistributed non-zero activation.