NeuZip: Memory-Efficient Training and Inference with Dynamic Compression
  of Neural Networks
Authors: name: Yongchang Hao; name: Yanshuai Cao; name: Lili Mou
Published: 2024-10-28 01:12:20

Abstract: The performance of neural networks improves when more parameters are used.
However, the model sizes are constrained by the available on-device memory
during training and inference. Although applying techniques like quantization
can alleviate the constraint, they suffer from performance degradation. In this
work, we introduce NeuZip, a new weight compression scheme based on the entropy
of floating-point numbers in neural networks. With NeuZip, we are able to
achieve memory-efficient training and inference without sacrificing
performance. Notably, we significantly reduce the memory footprint of training
a Llama-3 8B model from 31GB to less than 16GB, while keeping the training
dynamics fully unchanged. In inference, our method can reduce memory usage by
more than half while maintaining near-lossless performance. Our code is
publicly available.