Model-Preserving Adaptive Rounding
Authors: name: Albert Tseng, name: Zhaofeng Sun, name: Christopher De Sa

The main goal of post-training quantization (PTQ) is to produced a compressed
model whose output distribution is as close to the original model's as
possible. To do this tractably, almost all LLM PTQ algorithms quantize linear
layers by independently minimizing the immediate activation error. However,
this localized objective ignores the effect of subsequent layers, so reducing
it does not necessarily give a closer model. In this work, we introduce Yet
Another Quantization Algorithm (YAQA), an adaptive rounding algorithm that uses
Kronecker-factored approximations of each linear layer's Hessian with respect
to the full model KL divergence. YAQA consists of two components:
Kronecker-factored sketches of the full layerwise Hessian that can be tractably
computed for hundred-billion parameter LLMs, and a quantizer-independent
rounding algorithm that uses these sketches and comes with theoretical
guarantees. Across a wide range of models and quantizers, YAQA empirically
reduces the KL divergence to the original model by approx 30% while
achieving state of the art performance on downstream tasks.