Large Language Model Evaluation via Matrix Nuclear-Norm
Authors: name: Yahan Li; name: Tingyu Xia; name: Yi Chang; name: Yuan Wu
Published: 2024-10-14 16:15:57

Abstract: As large language models (LLMs) continue to evolve, efficient evaluation
metrics are vital for assessing their ability to compress information and
reduce redundancy. While traditional metrics like Matrix Entropy offer valuable
insights, they are computationally intensive for large-scale models due to
their \( O(n^3) \) time complexity with Singular Value Decomposition (SVD). To
mitigate this issue, we introduce the Matrix Nuclear-Norm, which not only
serves as a metric to quantify the data compression proficiency of LLM but also
provides a convex approximation of matrix rank to capture both predictive
discriminability and diversity. By employing the \( L_{1,2}-norm \) to
further approximate the nuclear norm, we can effectively assess the model's
information compression capabilities. This approach reduces the time complexity
to \( O(n^2) \) and eliminates the need for SVD computation. Consequently, the
Matrix Nuclear-Norm achieves speeds 8 to 24 times faster than Matrix Entropy
for the CEREBRAS-GPT model as sizes increase from 111M to 6.7B. This
performance gap becomes more pronounced with larger models, as validated in
tests with other models like Pythia. Additionally, evaluations on benchmarks
and model responses confirm that our proposed Matrix Nuclear-Norm is a
reliable, scalable, and efficient tool for assessing LLMs' performance,
striking a balance between accuracy and computational efficiency. The code is
available at https://github.com/MLGroupJLU/MatrixNuclearNorm.