Title: GlowQ: Group-Shared LOw-Rank Approximation for Quantized LLMs
Primary area: optimization
Keywords: Quantized large language models, low-rank error correction, group-shared factorization, randomized SVD, selective restoration, low-latency inference

Abstract: Low-bit quantization (e.g., weight-only 4-bit with BitsAndBytes/AWQ/GPTQ is now standard for deploying large language models but often degrades accuracy. Low-rank correction methods (e.g., LQER, QERA, ASER) mitigate this; however, they restore all layers and insert error-correction modules into every decoder block, which increases latency and memory overhead. We address this limitation by proposing GlowQ, a group-shared low-rank approximation for quantized LLMs that caches a single right-side projection per input-sharing group and selects only high-importance units for restoration. This caching-and-selection design amortizes computation and reduces parameter overhead while preserving expressivity; we also instantiate a selective variant, GlowQ-S, that applies the cached shared module only where it provides the largest benefit. Compared with strong baselines, our approach reduces TTFB by 5.6\% and increases throughput by 9.6\% on average, while reducing perplexity on WikiText-2 by 0.17\% and increasing downstream accuracy by 0.42 percentage points. The selective model GlowQ-S further reduces latency, cutting TTFB by 23.4\% and increasing throughput by 37.4\%, while maintaining accuracy within 0.2 percentage points on average.