any4: Learned 4-bit Numeric Representation for LLMs
Authors: name: Mostafa Elhoushi, name: Jeff Johnson

We present any4, a learned 4-bit weight quantization solution for large
language models (LLMs) providing arbitrary numeric representations without
requiring pre-processing of weights or activations. any4 yields higher accuracy
compared to other related 4-bit numeric representation types: int4, fp4 and
nf4, as evaluated on a range of model sizes, generations and families (Llama 2,
Llama 3, Mistral and Mixtral). While any4 does not require preprocessing of
weights or activations, it is also competitive with orthogonal techniques that
require such preprocessing (e.g., AWQ and GPTQ). We also experiment with any3
and any2 and show competitiveness at lower bits. Additionally, we show that we
can calibrate using a single curated diverse sample rather than hundreds of
samples from a dataset as done in most quantization approaches. We also open
source tinygemm, a latency optimized GPU matrix multiplication library for
LLMs, that implements any4 using a GPU-efficient lookup table strategy along
with other common quantization methods. We open source our code at
https://github.com/facebookresearch/any4 .