A neural network processor, comprising: one or more neurons; and a compression unit configured to receive an uncompressed chunk of data generated by at least one of the neurons in the neural network processor, the uncompressed chunk of data comprising a fixed number of bytes; generate a mask portion of a compressed output chunk, the mask portion comprising a number of bits equivalent to the fixed number of bytes in the uncompressed chunk of data, each bit in the mask portion corresponding to a byte in the uncompressed chunk of data, and wherein each bit in the mask portion is set to a logical zero where a corresponding byte in the uncompressed chunk of data is zero and is set to a logical one where a corresponding byte in the uncompressed chunk of data is non-zero; generate a data portion of the compressed output chunk by determining a number of non-zero bytes in the uncompressed chunk of data, determining, based on the number of non-zero bytes in the uncompressed chunk of data, a number of bits in the data portion of the compressed output chunk available to store truncated non-zero bytes of the uncompressed chunk of data, truncating the non-zero bytes in the uncompressed chunk of data to the determined number of bits, and storing the truncated non-zero bytes in the data portion of the compressed output chunk; and output the compressed output chunk, the compressed output chunk comprising the mask portion and the data portion. one or more neurons; and a compression unit configured to receive an uncompressed chunk of data generated by at least one of the neurons in the neural network processor, the uncompressed chunk of data comprising a fixed number of bytes; generate a mask portion of a compressed output chunk, the mask portion comprising a number of bits equivalent to the fixed number of bytes in the uncompressed chunk of data, each bit in the mask portion corresponding to a byte in the uncompressed chunk of data, and wherein each bit in the mask portion is set to a logical zero where a corresponding byte in the uncompressed chunk of data is zero and is set to a logical one where a corresponding byte in the uncompressed chunk of data is non-zero; generate a data portion of the compressed output chunk by determining a number of non-zero bytes in the uncompressed chunk of data, determining, based on the number of non-zero bytes in the uncompressed chunk of data, a number of bits in the data portion of the compressed output chunk available to store truncated non-zero bytes of the uncompressed chunk of data, truncating the non-zero bytes in the uncompressed chunk of data to the determined number of bits, and storing the truncated non-zero bytes in the data portion of the compressed output chunk; and output the compressed output chunk, the compressed output chunk comprising the mask portion and the data portion. receive an uncompressed chunk of data generated by at least one of the neurons in the neural network processor, the uncompressed chunk of data comprising a fixed number of bytes; generate a mask portion of a compressed output chunk, the mask portion comprising a number of bits equivalent to the fixed number of bytes in the uncompressed chunk of data, each bit in the mask portion corresponding to a byte in the uncompressed chunk of data, and wherein each bit in the mask portion is set to a logical zero where a corresponding byte in the uncompressed chunk of data is zero and is set to a logical one where a corresponding byte in the uncompressed chunk of data is non-zero; generate a data portion of the compressed output chunk by determining a number of non-zero bytes in the uncompressed chunk of data, determining, based on the number of non-zero bytes in the uncompressed chunk of data, a number of bits in the data portion of the compressed output chunk available to store truncated non-zero bytes of the uncompressed chunk of data, truncating the non-zero bytes in the uncompressed chunk of data to the determined number of bits, and storing the truncated non-zero bytes in the data portion of the compressed output chunk; and determining a number of non-zero bytes in the uncompressed chunk of data, determining, based on the number of non-zero bytes in the uncompressed chunk of data, a number of bits in the data portion of the compressed output chunk available to store truncated non-zero bytes of the uncompressed chunk of data, truncating the non-zero bytes in the uncompressed chunk of data to the determined number of bits, and storing the truncated non-zero bytes in the data portion of the compressed output chunk; and output the compressed output chunk, the compressed output chunk comprising the mask portion and the data portion.