Title: When LLMs get significantly worse: A statistical approach to detect model degradations
Primary area: generative models
Keywords: LLM, Benchmarking, Statistics, Accuracy

Abstract: Minimizing the inference cost and latency of foundation models has become a
crucial area of research. Optimization approaches include theoretically lossless
methods and such without accuracy guarantees like quantization. In all of these
cases it is crucial to ensure that the model quality did not degrade. However, even
at temperature zero, model generations are not necessarily robust even to theo-
retically lossless model optimizations due to numerical errors. We thus require
statistical tools to decide whether a finite-sample accuracy deviation is an evi-
dence of a model’s degradation or whether it can be attributed to (harmless) noise
in the evaluation. We propose a statistically sound hypothesis testing framework
based on McNemar’s test allowing to efficiently detect model degradations, whilst
guaranteeing a controlled rate of false positives. The crucial insight is that we have
to confront the model scores on each sample, rather than aggregated on the task
level. Furthermore, we propose three approaches to aggregate accuracy estimates
across multiple benchmarks into a single decision. We provide an implementation
on top of the largely adopted open-source LM Evaluation Harness and provide
a case study illustrating that the method correctly flags degraded models, whilst
not flagging model optimizations that are provably lossless. We find that with our
tests even empirical accuracy degradations of 0.3% can be confidently attributed
to actual degradations rather than noise.