Title: Reliability Scaling Laws for Quantized Large Language Models
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: LLM Evaluation, Quantization, Scaling Laws, Reliability

Abstract: Quantization is a powerful strategy to build capable and resource-efficient large language models (LLMs) by reducing the bitwidth of the parameters. While quantized LLMs achieve state-of-the-art performance on unperturbed inputs using standard predictive metrics, their performance on perturbed inputs, measured using subsidiary reliability metrics, remains underexplored, despite its importance for safe and reliable deployment. To address this gap, we conduct a comprehensive reliability evaluation of quantized LLMs consisting of three key components: (1) Uncertainty: We assess the trustworthiness of LLMs quantized to 2, 3, 4, and 8 bits using six different quantization methods, employing established uncertainty metrics operating at both token and sequence levels. 
(2) Robustness: We design character-level and word-level input perturbations to evaluate the reliability of quantized models under semantically-preserving variations in the inputs that commonly arise in real-world applications.
(3) Reliability scaling trends: We investigate how the reliability scales with the total number of model bits. Interestingly, our study reveals that while the performance scales monotonically with the total number of bits, the reliability scalings show nonlinear trends. Specifically, a reliability peak occurs for 4-bit quantized models, indicating that quantizing moderately sized base models offers the best reliability-efficiency trade-off. Additionally, our empirical findings reveal that quantization can enhance the robustness of LLMs to natural input perturbations.

TL;DR: We study how reliability scales with the total number of bits in quantized LLMs.