Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Models

Authors: lai  wei
zhiquan  tan
chenghai  li
jindong  wang
weiran  huang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 16:03:08

Abstract: Large Language Models (LLMs) have transformed natural language processing and extended their powerful capabilities to multi-modal domains. As LLMs continue to advance, it is crucial to develop diverse and appropriate metrics for their evaluation. In this paper, we introduce a novel rank-based metric, Diff-eRank, grounded in information theory and geometry principles. Diff-eRank assesses LLMs by analyzing their hidden representations, providing a quantitative measure of how efficiently they eliminate redundant information during training. We demonstrate the applicability of Diff-eRank in both single-modal (e.g., language) and multi-modal settings. For language models, our results show that Diff-eRank increases with model size and correlates well with conventional metrics such as loss and accuracy. In the multi-modal context, we propose an alignment evaluation method based on the eRank, and verify that contemporary multi-modal LLMs exhibit strong alignment performance based on our method. Our code is publicly available at https://github.com/waltonfuture/Diff-eRank.

Main paper: 
 introduction: 
 Large Language Models (LLMs) such as GPT [4, 23] , Chinchilla [17] , and PaLM [7] , have gained considerable attention for their outstanding performance in various natural language processing tasks. LLMs have expanded from single-modal models to multi-modal models, including MiniGPT-4 [49] and LLaVA [20] , which have achieved remarkable results in various application scenarios. Pre-trained LLMs rely on large networks, computational power, and massive amounts of data, aiming for greater generalization capabilities.LLMs understand the world knowledge through training on huge amounts of data. One famous belief [34] of how LLMs work is that larger models can find more shared hidden structures in data samples by eliminating redundant information through training. In particular, in the early phase of training, following random initialization, the representations derived from the training data tend to be somewhat chaotic. As training progresses, these representations become increasingly structured, and the model discards extraneous information from the training data, which resembles a process similar to "noise reduction". This perspective motivates us that LLM could be evaluated by characterizing the "noise reduction" process.However, defining and quantifying the degree of "noise reduction" remains a significant challenge. To address this, we hypothesize that a reasonable metric should 1) reflect the geometric characteristics of the data such as the dimensionality of its representations, and 2) be rooted in information theory.In this paper, we introduce Diff-eRank (difference between effective ranks), an information-theoretic metric that fulfills both criteria, providing a measure for quantifying "noise reduction" in LLMs. In particular, we consider the effective rank (eRank) of the representations extracted by an LLM from a dataset to measure the uncertainty, based on concepts from (quantum) information theory [30] . Through the removal of redundant information, eRank decreases, indicating the representations become more structured and compact. Thus, the reduction of representations' eRank can signify the degree of "noise reduction". Therefore, we can evaluate a well-trained LLM via the eRank reduction of the model representations from its untrained status. We remark that different from conventional metrics like loss, which are derived from the predictions of LLMs, the proposed Diff-eRank focuses on the model representations. Our approach offers a novel perspective of model assessment, independent of prediction-based metrics, and can provide new insights into the understanding of LLM's behavior.To verify the effectiveness of our approach, we conduct experiments on the contexts of both uni-modal LLMs and multi-modal LLMs. In particular, for uni-modal LLMs, we compute Diff-eRanks for models within the OPT family [45] across various datasets. Intriguingly, we observe that Diff-eRank increases as the model scales, suggesting that larger models exhibit a stronger noise reduction ability. Moreover, Diff-eRank has a consistent trend when compared with (reduced) cross-entropy loss and benchmark accuracy, highlighting its potential as an effective and easy-to-use evaluation metric. For multi-modal (vision-language) LLMs, visual and language information is usually encoded separately by two independent encoders and aligned through a connecting layer. Therefore, evaluating the quality of modality alignment in multi-modal LLMs is crucial. Building on insights from uni-modal LLMs, we can assess modality alignment by examining the matching degree of eRanks between representations from different modalities. Additionally, this approach yields interesting observations within the context of such multi-modal architectures.Our contribution can be summarized as follows:• We propose a rank-based metric, Diff-eRank, for evaluating LLMs, where Diff-eRank reflects the "noise reduction" ability of pre-trained language models. Diff-eRank focuses on the model representations, different from conventional metrics such as loss and benchmark accuracy.• We validate the effectiveness of Diff-eRank by observing its correlation with the trends in loss and downstream task accuracy as the model scales up.• We also propose eRank-based modality alignment metrics for multi-modal LLMs, and verify that contemporary multi-modal LLMs exhibit strong alignment performance via our metrics. 
related works: 
 Evaluation of Large Language Models. Evaluation of LLMs is a fast-evolving field across various tasks, datasets, and benchmarks [5, 33, 36, 48] . Precise evaluations are important for the enhancement of language models' performance and reliability. Conventional metrics such as accuracy, F1 [29] , BLEU [24] and ROUGE [18] estimate between the annotated label and the prediction generated by the language model in different downstream tasks. Other metrics like perplexity and crossentropy loss are independent of annotated labels and can be computed solely based on the input texts. However, these metrics focus on "extrinsic" evaluation, assessing performance based on the predictions of LLMs. We propose Diff-eRank for "intrinsic" evaluation based on the input data's hidden representations of LLMs, concentrating on their "noise reduction" capabilities.Information Theory for Understanding Deep Learning. Information theory has been used to gain significant insights into understanding neural networks. For example, the information bottleneck [37, 38] is instrumental in explaining supervised learning. Recently, researchers have also utilized information theory to understand and improve (vision) semi and self-supervised learning [32, 35, 46, 47] . Notably, Zhang et al. [46] find the closed-form connection of matrix entropy and effective rank when the matrix is positive semi-definite. As for language models, prior works [16, 25, 41] also used information theory to analyze hidden representations by training probes on specific downstream tasks to estimate the information contained in the pre-trained language model. Several other works explore the lossless compression of LLMs with arithmetic coding [10, 40] based on information theory. In this paper, we take a further step toward evaluating LLMs through the proposed Diff-eRank rooted in information theory, which represents a complementary perspective to these prior studies. 
the proposed metric for evaluating llms: 
 In this section, we will introduce a rank-based metric called Diff-eRank for evaluating LLMs. The proposed metric is based on the representations obtained by an LLM, fundamentally diverging from conventional metrics like loss, which are based on the model's predictions.When processing a sequence of tokens, an LLM will generate a representation (i.e., the hidden states before the last classification head) for each token within the sequence. These high-dimensional representations are usually used to capture the semantic and syntactic information of the sentences. This inspires us to consider evaluating LLMs by analyzing these representations. In particular, we study the characteristics of these representations by examining their ranks through both the geometric and information-theoretic perspective. On the one hand, studying the rank of these representations allows us to measure the extent of linear independence among them, which corresponds to the effective dimensions in the representation space (i.e., the geometric structure). On the other hand, the rank is also related to the amount of information contained in these representations, while a lower rank indicates that the information has been structured or compressed. Therefore, we consider to leverage the rank of data representations encoded by LLMs for model evaluation.However, the size of data representation matrix varies with the sample size, making it less suitable for consistent analysis. Therefore, instead of directly computing the rank of the data representations, we use the rank of their covariance matrix, which has a fixed size and also contains all the essential information. To see this, let S = {z 1 , z 2 , . . . , z N } denote the set of data representations, and z be the mean representation. The rank of data representation matrix can be re-formulated asrank([z 1 -z, • • • , z N -z]) = rank 1 N [z 1 -z, • • • , z N -z][z 1 -z, • • • , z N -z] ⊤ = rank 1 N N i=1 (z i -z)(z i -z) ⊤ ,where the last term is exactly the rank of covariance matrix. We remark that the above rank also equals to the dimension of the affine subspace spanned by S ∪ {z}.The formal construction of covariance matrix is shown as follows. For ease of analysis, each z i -z is being normalized. Definition 3.1 (Construction of Covariance Matrix). Given a set of representations S = {z i ∈ R d | i = 1, 2, . . . , N }, the covariance matrix Σ S is constructed asΣ S = 1 N N i=1 z i - z ∥z i -z∥ z i - z ∥z i -z∥ ⊤ , where z = N i=1 z i /N is the mean representation and notation ∥ • ∥ represents ℓ 2 norm.Since rank is highly sensitive to outliers [27] , we instead use its "continuous" counterpart, the effective rank (eRank), when applied to the covariance matrix, defined as below. Definition 3.2 (eRank [27] ). The effective rank of any non-zero matrix A ∈ R d×N is defined aseRank(A) = exp - Q i=1 σ i Q i=1 σ i log σ i Q i=1 σ i, where Q = min{N, d} and σ 1 , σ 2 , . . . , σ Q are the singular values of matrix A.We remark that the above eRank is closely related to the matrix entropy (i.e., Von Neumann entropy for matrices [42] ), which is defined in Definition 3.3. In fact, Zhang et al. [46] point out that, for a covariance matrix of normalized vectors, eRank(Σ S ) is the same as exp(H(Σ S )).). Given a positive semi-definite matrix K ∈ R d×d satisfying tr (K) = 1, the matrix entropy of matrix K is defined asH(K) = -tr (K log K) .It is equivalent to the Shannon entropy [31] over the spectrum, i.e.,H(K) = - d i=1 λ i log λ i ,where λ 1 , λ 2 , . . . , λ d are the eigenvalues of matrix K.Note that eRank of the covariance matrix is commonly interpreted as a measure of the "degree of freedom" that the sentence contains in a geometric sense, one may wonder whether there is a more "information-theoretic" explanation for it. Interestingly, under the terminology of quantum information theory [44] , if we regard the representation of each token as a state in a quantum system, the construction given by Definition 3.1 is a standard process of constructing a density matrix. From the quantum noiseless coding theorem [30] , the entropy of a density matrix H(Σ S ) represents the average number of qubits required to encode the states. Therefore, exp (H(Σ S )) can be viewed as a measure of randomness for a sentence through the quantum information theory.As eRank measures the amount of uncertainty in a system, we can now define Diff-eRank to measure the degree of "noise reduction" for an LLM. Definition 3.4 (Diff-eRank). Given a sentence x, an untrained language model M 0 , and a computeoptimal [17] trained language model M 1 , we obtain two sets of representations, M 0 (x) and M 1 (x), by processing each token of x through the respective models. Then the rank difference (i.e., Diff-eRank) between these two models based on sentence x is defined as follows:∆ eRank(x, M 0 , M 1 ) = eRank Σ M0(x) -eRank Σ M1(x) ,where Σ Mi(x) is the covariance matrix of model M i 's representations on sentence x for i ∈ {0, 1}.Upon completing training, the model's data representations shift from being random to more structured, enabling it to effectively capture patterns and structures from the data. In the above definition, the effective ranks eRank(Σ M0(x) ) and eRank(Σ M1(x) ) quantify the uncertainty in the representations of the untrained and trained models, respectively. Thus, Diff-eRank ∆ eRank(x, M 0 , M 1 ) measures how much uncertainty the model has reduced as a result of training.The above definition applies to a single sentence but can be extended to a dataset consisting of multiple sentences. Specifically, Diff-eRank for the entire dataset can be defined as the average Diff-eRank across all sentences, formulated as follows. Definition 3.5 (Diff-eRank of a Dataset). Given a dataset D consisting of sentences x 1 , . . . , x n , an untrained language model M 0 , and a compute-optimal [17] trained language model M 1 , Diff-eRank of dataset D is defined as∆ eRank(D, M 0 , M 1 ) = exp n i=1 H Σ M0(xi) n -exp n i=1 H Σ M1(xi) n .In summary, Diff-eRank reflects the dimension reduction of the space spanned by data representations. It can be viewed as a measure of removing redundant information in the data for a compute-optimal language model. A higher Diff-eRank indicates more organized and structured internal representations of the model, therefore reflecting the model's increasing effectiveness in capturing patterns and regularities in the data.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes]• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
evaluations of large language models: 
 We start with evaluating different sizes of language models via Diff-eRank in Section 4.2. We find that Diff-eRank increases as the model scales up on various datasets. Additionally, we extend the application of eRank to multi-modalities beyond the language domain in Section 5.We experiment by using popular transformer-based language models from OPT [45] family, ranging from 125 million to 13 billion parameters. Such diversity in OPT's model size allows for a comprehensive analysis across different scales of pre-trained language models in our experimental setting. We refer the reader to Appendix A for additional implementation details about the selection of language datasets.Given a text sequence U = [u 1 , . . . , u T ], the cross-entropy loss of a language model M can be defined asL(U, M ) = - 1 T T i=1 log P (u i |u 1 , . . . , u i-1 ).The cross-entropy loss is a canonical quantity in Shannon information theory, based on the model's predictions. As we study the rank difference between untrained model M 0 and compute-optimal trained model M 1 based on representation, we adopt the difference in loss for comparison, correspondingly. Therefore, we can similarly define reduced (cross-entropy) loss as∆L(U, M 0 , M 1 ) = L(U, M 0 ) -L(U, M 1 ).As the training progresses, the LLM gets better predictions on the input data, leading to an increase in reduced loss. Therefore, reduced loss can also be seen as a useful evaluation metric for LLMs, and we use it for comparison with Diff-eRank in our following experiments.To substantiate Diff-eRank as a viable metric for evaluation, we evaluate the series of OPT [45] models over different and diverse datasets using Diff-eRank and (reduced) loss for comparison. Specifically, we consider including pre-training datasets such as Wikipedia [14] and openwebtext2 [15] , instruction dataset dolly-15k [8] , and preference dataset hh-rlhf [2] for the diversity of their usage.Figure 1 presents that Diff-eRank and reduced loss both increase progressively as the model scales up. The increase in reduced loss (equals to a decrease in cross-entropy loss) can be interpreted as larger models providing closer predictions to the actual values so that they can better capture the underlying patterns and relationships within the data. As for the increase in Diff-eRank based on hidden representations, it suggests that the redundant dimensions of the data can be effectively reduced in the larger models, thereby resulting in stronger "noise reduction" abilities and larger Diff-eRanks. Overall, the strong correlation between Diff-eRank and (reduced) loss indicates that Diff-eRank can provide a novel and inspirational evaluation for LLMs through the lens of "noise reduction" in dimension spaces. We summarize detailed results tables in Appendix B.Based on the exploration in the earlier section, a natural question arises: does Diff-eRank relate to the downstream task accuracy of large language models? To address this question, we integrate accuracy as a comparative metric in addition to Diff-eRank and reduced loss in our evaluations on benchmark datasets. We use the evaluation set of openbookqa [22] and piqa [3] by combining the question and correct answer of each piece of data as inputs.The results presented in Table 1 illustrate a similar relationship among Diff-eRank, reduced loss, and downstream task accuracy. All of these three metrics increase when model size increases. Although occasional outliers are observed in the upward trends of these indicators, we think this is normal and does not affect the overall trend. Therefore, it can be concluded that Diff-eRank generally correlates with the trend of loss and accuracy, particularly as the model size scales within the same model family.An increase in Diff-eRank (i.e., a higher denoising ability of the model) corresponds to enhanced model performance (i.e., higher reduced loss and higher accuracy), which shows great potential in the evaluation of LLMs. 
evaluations of multi-modal large language models: 
 After verifying that Diff-eRank can indeed reflect the LLMs' intrinsic ability in the previous sections, our study extends to the evaluation of Multi-modal Large Language Models (MLLMs) [1, 20, 43, 49] .We define new metrics based on the eRank to evaluate the modality alignment.For our multi-modal experiments, we select two advanced and open-sourced MLLMs as shown in Table 5 in the appendix: LLaVA-1.5 [19] and MiniGPT-v2 [6] . Both the two MLLMs utilize a simple connector for aligning the vision encoder with the LLM, providing a streamlined approach to multi-modal learning. We conduct the experiments on two high-quality multi-modal instruction datasets: detail_23k [20] and cc_sbu_align [49] . Each piece of data in these datasets contains a triplet of image, instruction, and response. We concatenate the instruction and response of each triplet as the textual input in our experiments.Most of the MLLMs typically employ a projector mechanism (usually linear layer or MLP), which aligns image representations from a vision encoder (usually ViT [12] ) with LLM's language representations. Our experiments include analyzing the effective rank of representation of images post vision encoder (eRank 1 ) and post connector (eRank 2 ), as well as the representation output by the LLM for individual images (eRank 3 ), text (eRank 4 ), and image-text pairs (eRank 5 ), as shown in Figure 2 .To measure the "modality alignment" of MLLMs, we introduce two distinct metrics based on eRank:Image Reduction Ratio = eRank 1 -eRank 2 eRank 1 ,and Image-Text Alignment = avg(eRank 3 , eRank 4 , eRank 5 ) max(eRank 3 , eRank 4 , eRank 5 ) . On the one hand, the "Image Reduction Ratio" metric is formulated to quantify the reduction in effective rank from the vision encoder output (eRank 1 ) to the post-connector stage (eRank 2 ). Note that normalization is necessary here for a fair comparison because the vision encoder and connector are entirely different networks. This metric evaluates the connector network's efficiency in condensing and refining visual information during image-text alignment training. On the other hand, the "Image-Text Alignment" metric is designed to evaluate the closeness among the effective rank of representations post LLM processing, considering individual images (eRank 3 ), text (eRank 4 ), and image-text pairs (eRank 5 ) as inputs. In particular, the absolute eRank can be seen as the amount of absolute uncertainty or randomness. The mentioned three eRanks show how much the model integrates and represents each modality. If these three eRanks from different modalities are close to each other, it means that they align well from the perspective of information theory. Thus, this metric reflects the degree of closeness (i.e., alignment) among different modalities. A higher alignment score indicates a more proficient alignment between image and text modalities for MLLMs.Results in Table 2 exhibit the performance of two MLLMs, LLaVA-1.5 [19] and MiniGPT-v2 [6] , across different datasets (detail_23k [20] and cc_sbu_align [49] ). Both models align well as they all have a relatively high alignment score.In particular, comparing the two models, LLaVA-1.5 and MiniGPT-v2 both exhibit similar "Image Reduction Ratio" scores, indicating efficient condensation of visual information. Additionally, LLaVA-1.5 outperforms MiniGPT-v2 in "Image-Text Alignment", suggesting a closer integration between visual and textual modalities. This finding is also consistent with their performance, as LLaVA-1.5 surpasses MiniGPT-v2 in most of benchmarks [9] . We leave exploring a more comprehensive evaluation for multi-modal models via effective rank as future work.To further investigate the role of each component in MLLM, we conduct additional experiments to calculate the eRank after rotating the images clockwise. We summarize the results in Table 3 .As the rotation of images introduces new semantic information into the model, by noticing all the image-related quantities (eRank i (i ̸ = 4)) all increase from the base model when performing rotation, this semantic influence can propagate through the model. Therefore, we suggest that the multimodal model (including the connector and the language model) can indeed perceive subtle semantic variations in images, especially the position information. In addition, the "Image Reduction Ratio" score and "Image-Text Alignment" score both decrease after conducting image rotation, suggesting that the connector performs less effectively in condensing visual information, and the rotated images , Cerebras-GPT [11] , and OpenELM [21] .are less well-aligned with the corresponding text. This is primarily because the rotation alters the spatial relationships within the image, possibly making it more challenging for the model to maintain the coherence between visual and textual information. Overall, this experiment indicates that subtle changes in the vision encoder's understanding of images can be effectively conveyed to the LLM part and affect the MLLM's modality alignment. It demonstrates the validity of such a popular multi-modal architecture.In conclusion, these rank-based approaches enable a thorough understanding of how well the multimodal models align different modalities of data and how the models process and integrate different forms of input data. 
ablation study: 
 To better confirm the rationality of our algorithm and experimental design, we further conduct a series of ablation studies.Besides observing Diff-eRank on the OPT family, we also conduct experiments on Cerebras-GPT [11] family and OpenELM [21] family. LLMs in these three families are all pre-trained well on public data and range in various sizes. To demonstrate that Diff-eRank is not dependent on specific datasets, we choose not to use benchmark datasets but instead select a general dataset. In particular, we adopt the dolly-15k [8] dataset to compute Diff-eRank along with reduced loss, and we calculate the average benchmark accuracy of winogrande [28] and piqa [3] for these three LLM families. The empirical findings in Figure 3 substantiate the increase of Diff-eRank within these LLM families as the models scale up, which correlates with the trend of reduced loss and benchmark accuracy. This observation shows the potential of Diff-eRank as an insightful metric for the evaluation of different model families.In this section, we choose other types of algorithms for designing Diff-eRank between untrained model M 0 and trained model M 1 . The goal is to validate that the increasing type relation is robust to the algorithm we used.We denote our standard computation of effective rank on a dataset D (Definition 3.5) as "Algorithm (a)", which calculates the effective rank based on the average matrix entropy. In addition, we also consider the operation of calculating the average effective rank on a dataset D, denoted by "Algorithm (b)". Specifically, for an LLM M , the effective rank on a dataset D of Algorithm (b) is defined as Therefore, Diff-eRank between untrained model M 0 and trained model M 1 of Algorithm (b) can be formulated aseRank (b) (D, M ) = x∈D exp(H(Σ M (x) )) |D| = x∈D eRank(Σ M (x) ) |D| .∆ eRank (b) (D, M 0 , M 1 ) = eRank (b) (D, M 0 ) -eRank (b) (D, M 1 ).To compare these two ways for defining Diff-eRank, we conduct experiments using OPT models on dolly-15k dataset. The experimental results in Figure 4 demonstrate that Diff-eRank consistently increases across model sizes, irrespective of whether Algorithm (a) or Algorithm (b) is used. This observation verifies that the increasing trend for Diff-eRank is robust across different algorithms of effective rank defined on a dataset.In our research, we predominantly concentrate on the last layer of LLMs, as it usually represents the most comprehensive information encoded by the model. This layer may offer the most indicative measure of Diff-eRank. Besides, we also extend our experiments to encompass additional layers within the models. Specifically, our investigations include analyses of the first layer, the middle layer, and the last layer for language models in the OPT [45] family on dolly-15k [8] dataset. Our findings in Table 4 reveal that only the Diff-eRank on the last layer reveals an increasing trend across model sizes, which indicates that it's reasonable to analyze data representation through the last layer that encodes the most comprehensive information of the model. This may be interpreted that LLM is an integrated system where information processing occurs across the entire architecture. If we rely on early layers for analyzing Diff-eRank, this could lead to a loss of important information and we may miss crucial information processing that occurs in subsequent layers. The last layer, on the other hand, integrates this information, providing a more complete representation of the input data. The observation in our experiments reveals that early layers do not exhibit clear patterns in terms of Diff-eRank. This underscores the importance of considering the model as a whole when analyzing the representation. 
conclusion and discussion: 
 We introduce Diff-eRank, a new metric that can measure the "noise reduction" ability of LLM based on data representation and reflects the extent to which a pre-trained LLM eliminates the redundant dimension in the information-theoretic sense. Our method reveals the geometric characteristics of the data and is grounded in information theory. The empirical investigations show that the Diff-eRank increases when the model scales and correlates with the trend of loss and downstream task accuracy. Moreover, we use this metric to define the alignment metrics for multi-modal LLMs and find contemporary models align very well. In addition, some useful techniques like pruning, quantization, and distillation may benefit from such metrics that reveal internal redundancies. The Diff-eRank metric may aid in identifying which parts of the model can be compressed without significant loss of information. We hope that Diff-eRank will open up avenues for future studies to explore how such internal representation metrics can be integrated into different potential cases.  
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We follow the license and terms of use in our experiments.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: To further investigate how "Diff-eRank" changes during training, we conduct additional experiments to observe the behavior of Diff-eRank across different training stages for a fixed model size. In particular, we fix the model size by using the pre-trained OPT-1.3B [45] model and continually train it on a cleaned Wikipedia [14] dataset.According to the additional experimental results in Table 10 , we observe that the trend of Diff-eRank, first increasing before fully trained and then slightly decreasing when overfitting, aligns well with the trend of benchmark accuracy and the opposite trend of loss. This suggests that Diff-eRank may serve as a complementary metric that helps understand the LLM's "noise reduction" behavior during training, and monitor the training progress.As mentioned in Appendix A.1, random sampling is employed to extract subsets from the whole datasets of Wikipedia [14] and openwebtext2 [15] , each subset comprising 10,000 data entries, as these pre-training datasets are too large for computation. To assess the robustness of Diff-eRank in random selection, we incorporate variations in the sample sizes of the Wikipedia dataset in this ablation study. Table 11 illustrates that fluctuations in the sample size bring insignificant influence on the Diff-eRank, which affirms the stability of Diff-eRank in random sampling. Thus, this ablation study indicates the rationality of the random sampling process when dealing with large pre-training datasets in our experiments. Justification: We have mentioned limitations along with the conclusion in the last part of our paper. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Justification: There is no societal impact of the work performed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: This paper proposes a new rank-based evaluation metric for LLMs which quantifies the amount of redundant information in the hidden representations of a model and applies it to both text-only and multi-modal models. The idea is to compute the "rank difference" between trained and untrained models and use it as a measure of model performance. Experiments show the potential of rank difference in assessing model capabilities within certain model series and measuring the cross-modal alignment of MLLMs.

Reviewers had mixed feelings about this paper. They pointed out as strengths the originality of the proposed metric and its theoretical soundness, as well as their empirical alignment with other metrics and the potential for rank difference to guide future work on model compression. They also pointed out the comprehensive experiments and the clarity of the writing. The main weaknesses are the lack of clarity about the additional information that the proposed metrics bring to the table (particularly for multi-modal models), as well as a a concern that rank difference may not accurately measure the extent of noise reduction, due to the correlation between the dimensionality of the hidden states (and model size) with performance, which makes larger models to have a higher rank difference but less "noise reduction" than smaller models. 

Overall, I am leaning towards acceptance. The proposed "rank difference" is original and it seems useful not only for evaluation but potentially for model development and compression. The authors addressed in detail all the questions asked by the reviewers and in my opinion they alleviated some of their concerns. They reported additional results which seem to address the main weaknesses. I believe the missing details can be incorporated easily in the final version and enhance the paper.

Reviews: review: paper_summary: paper_summary: The paper proposes a rank-based evaluation metric that quantifies the amount of redundant information in the hidden representations of a model and applies it to both text-only and multi-modal models. The effective rank is obtained by the rank of its covariance matrix and is interpreted using information theory. This rank represents the degree of variations among the principal components in data.  The rank difference between two models is used to measure how much redundant information is reduced in a model relative to another. The proposed metric aligns well with commonly used evaluation metrics (loss and accuracy).
strength_weakness: strength_weakness: 1. It is unclear what additional information the absolute rank or the rank difference brings to the table apart from a new interpretation. The rank differences are hard to interpret (L261: both models align well with a high alignment score) given a lack of detail on how they are computed for models of varying sizes.
2. For the multi-modal models, the authors again propose two metrics: image reduction ratio and image-text alignment and while the reduction ratio is a bit intuitive, it is unclear why image-text alignment is defined as is and why we need the different ranks (erank 3, erank 4, and erank5). In Table 2, the reduction and alignment follow opposite trends, is it something informative?
questions: questions: 1. L224-225: can you provide any explanation on why these occasional outliers appear? This is much vivid in Table 4 where any other layer does not follow this upward trend.
2. Was there any experiment also on how the rank difference involves over the training of a model and whether that signal is informative on how long a model can be trained?
3. Is there any explanation on why the different layers are not informative of the representation redundancy (table 4)?
limitations: limitations: yes
paper_summary: paper_summary: The article presents a measure known as "rank difference" to assess the effectiveness of Language Models (LLMs) by analyzing their internal representations. This metric is based on information theory and geometric principles aiming to quantify how LLMs eliminate unnecessary information post training. The authors illustrate the utility of rank difference, in both modal (language) and multi modal contexts.

In terms of language models the study reveals that the rank difference grows with model size indicating noise reduction capability. This pattern aligns with metrics such as entropy loss and accuracy. Regarding modal models the authors suggest an assessment approach utilizing rank difference to evaluate alignment quality across modalities showing that contemporary multi modal LLMs demonstrate strong alignment performance.

Key contributions of the paper include;

Introducing rank difference as a metric for gauging the "noise reduction" capability of trained language models.
Demonstrating the correlation between rank difference, loss metrics and downstream task accuracy underscoring its potential as an evaluation criterion.
Defining alignment measures, between linguistic modalities showcasing that modern multi modal LLMs excel in achieving alignment.

The study provides real world data that backs the idea of utilizing rank variance, in datasets and model scales indicating its reliability and effectiveness in assessing language learning models. In general the research introduces a viewpoint on evaluating language learning models moving away from metrics based on predictions, to focusing on model representations. This shift brings forth perspectives on comprehending how language learning models behave.
strength_weakness: strength_weakness: The experiments have a scope;
Regarding Training Dynamics; The paper fails to delve into how the rank difference evolves throughout training missing insights, into its behavior as the model progresses.
Considering Model Diversity; Broadening the evaluation to include a range of model families and architectures would enhance the validation of the rank difference metric.

In terms of Presentation;
Clarity; Some sections require explanations to improve readability.
Visual Support; Incorporating diagrams or visual aids could boost understanding and reader engagement.

Delving Into Analysis Depth;
Insights by Layer; Analyzing rank differences across model layers would offer a holistic view.
Comparative Examination; Providing in depth comparisons with established metrics and recent studies would spotlight both strengths and limitations of the metric.

Application, in Real Scenarios;
Real world Contexts; Integrating real world application scenarios or case studies would enhance the relevance of the paper.

Computational Considerations;
Efficiency Concerns; Addressing the efficiency of calculating rank difference for large scale models and suggesting optimizations could be advantageous.
questions: questions: Do you have any insights or early findings regarding the variations, in the rank difference metric as LLMs undergo training? Examining this aspect could offer insights into how the behaves and its usefulness throughout the model development process.

Have you thought about assessing the rank difference metric across model families or architectures aside from the OPT family? Incorporating a range of models could help validate how broadly applicable the metric is.

Could you delve deeper into comparing the rank difference metric with established evaluation metrics and recent research findings? This, in depth analysis would shed light on both the strengths and potential limitations of using the rank difference metric.
limitations: limitations: The authors recognize the limitations in conducting experiments to observe changes, in rank difference during training due to constraints. They emphasize the importance of conducting experiments to assess how applicable the metric is across model layers and its efficiency in computations.

Suggestions for Enhancements;
Training Progress; Conduct a small scale study or simulation to observe how rank difference evolves during training with models or subsets of data.
Model Variation; Assess the effectiveness of the rank difference metric across a range of model families and structures. Provide initial findings if a comprehensive evaluation is not feasible.
Computational Efficacy; Explore optimizations for calculating rank difference, such, as enhancements or parallelization methods and briefly touch upon any societal implications and ethical considerations to demonstrate awareness of broader impacts.
paper_summary: paper_summary: This paper introduces “rank difference” that measures the reduction in the rank of LLM’s representations. It evaluates the quality of LLMs, which could be used in addition to the reduction in the cross-entropy loss. The idea is based on the assumption that LLM’s representations (e.g., the hidden states of each token before the classification head) encodes the semantic and syntactic information about the input sentence. Before training, these representations are expected to be chaotic (hence high rank), but after training and alignment, the representations are expected to be more structured. 

Based on this idea, “rank difference” is defined as the difference between $erank_{m0}(\Sigma_S)$ and $erank_{m1}(\Sigma_S)$ where $\Sigma_S$ is the covariance matrix of representations and $erank$ is the effective rank. The difference is computed between the representation before and after training. This measures the extend to which the model can compress the information, while other metrics such as cross-entropy measure the quality of model’s prediction.

Experiments are conducted on OPT models of sizes from 125m to 13B parameters. The results show that there is an upward trend between “rank difference” and model size (which reflects model performance) on a range of datasets, similar to the reduction in cross-entropy. In addition, rank difference is applied to measure the alignment between visual modality and text modality in Llava-1.5 and MiniGPT-v2. An ablation study on different model families is shown.
strength_weakness: strength_weakness: 1. Although the idea is novel, its practical usability is limited. To evaluate the quality of systems (e.g., LLM, ASR, etc), downstream metrics such as accuracy, ROUGE, BLEU, WER, etc. are used as they better align with real use cases. While, a metric like cross-entropy is used as it is differentiable, allowing it to be a training loss. I’m not sure if “rank difference” will be adopted as it’s neither related to real use case nor applicable as a training objective.

2. Regarding experiments, current results (both text-only and visual-text) only show the trend post-training, so it would be more indicative of demonstrating how “rank difference” changes during training (e.g., similar to cross-entropy loss which usually decreases monotonically during training).
questions: questions: 1. How do you think this method could be adopted?
2. What is the computational cost of computing rank difference?
3. Would it be comparable across models of different representation sizes?
4. Does/how rank difference during the alignment stage (e.g., SFT or PPO/DPO) ?
limitations: limitations: Limitations section is provided, and it has covered the main points
paper_summary: paper_summary: This paper explores the potential of using the rank of model hidden states for evaluating model capabilities. Specifically, the authors propose calculating the rank difference between trained and untrained models as a measure of model performance. The core idea is that the rank difference can reflect the model's extent of "noise reduction". The authors compared the consistency of rank difference with other metrics (loss, accuracy) under multiple model series settings. The results showed the potential of rank difference in assessing model capabilities within certain model series and measuring the cross-modal alignment of MLLMs.
strength_weakness: strength_weakness: - My main concern lies in the fact that for models within a series, the dimensionality of their hidden states is often positively correlated with size, which also implies a positive correlation with performance. A larger dimensionality of hidden states usually means a larger rank difference when the effective rank proportion remains the same. In other words, although larger models have a higher rank difference, the proportion of noise reduction might be smaller compared to smaller models. Does this suggest that rank difference may not accurately measure the extent of noise reduction? Could rank ratio be a more reasonable metric (erank_M1/erank_M0 in eq.3)?

- Although the authors observed a positive correlation between rank difference and model size within a single model series, this conclusion no longer holds when comparing models from different series simultaneously, as shown in Figure 3. Is this due to the varying dimensionality of hidden states across different models (where rank ratio might be effective)? Or is it caused by the different training methods employed for various models? If rank difference cannot be used to compare models from different series, what advantages does it offer compared to traditional metrics such as accuracy?

- When the number of test set samples lies between the dimensionality of hidden states for the large and small models being evaluated, Q in Equation 1 will take values from N and d, respectively. Could this lead to non-robust results? I believe experiments should be conducted to investigate this scenario.
questions: questions: See Weaknesses.

I am looking forward to authors' responses and open to rise my score.
limitations: limitations: The main limitation of this paper is that it fails to specifically discuss the advantages of rank difference compared to other metrics, especially considering that the rank differences of models from different series are not on the same scale.

Comments: title: Thanks for raising your score.
Response to Reviewer LQXh
Thanks for raising your score.
comment: Thank you for taking the time to reassess our paper and raising the score from 5 to 6. We are grateful for your thoughtful and constructive suggestions. Your insightful feedback, along with that of the other reviewers, will be incorporated into our revised revision.

Thank you again for your time and effort.
We thank the authors for there efforts to address my concerns and my concerns have mostly been solved. I update my rating accordingly.
Thank you for your detailed and insightful feedback. We believe that the rank difference metric provides valuable insights into the internal representations and redundancy within models, which are not directly captured by traditional metrics like accuracy or loss. This makes rank difference a useful complementary metric for understanding model behavior.

We also understand your concern regarding the natural correlation between model size and rank difference, and we agree that your suggested experiment, comparing rank differences across different training checkpoints for a fixed model size, would provide a more robust validation of our metric.

Following your suggestions, **we have conducted additional experiments to observe the behavior of rank difference across different training stages for a fixed model size**. In particular, we fix the model size by using the pre-trained OPT-1.3B model and continually train it on a cleaned Wikipedia dataset. We evaluate checkpoints at various stages of training, including random initialized (untrained), initialized from OPT-1.3B (pre-trained OPT-1.3B), fully trained (achieving the best accuracy on Openbookqa benchmark), and overfitting. The results of rank difference, loss, and accuracy are presented in the table below.


| Metrics/Training Stages |Random Initialized|Initialized from OPT-1.3B|Fully Trained|Overfitting
| ------- | ---- | ---- | ---- | ---- | 
| Rank Difference| 0| 2.140| 2.161 | 2.156 
| Loss| 10.830| 4.692| 4.654 | 4.663 
| Accuracy|0.250| 0.332 |0.340| 0.336


According to the experimental results, **we observe that the trend of rank difference, first increasing before fully trained and then slightly decreasing when overfitting, aligns well with the trend of benchmark accuracy and the opposite trend of loss**. This suggests that rank difference can serve as a complementary metric that helps understand the model's behavior, and monitor the training progress. 

We hope this additional analysis addresses your concerns. Thank you again for your constructive and thoughtful suggestion. We believe our findings strengthen the reliability of our conclusions.
Sorry for the late. Thank you very much for the detailed responses for addressing my concerns. However, I still have some questions:

Considering that rank difference = erank_M1 - erank_M0, both erank_M1 and erank_M0 will increase as the hidden state dimensions grow. Rank difference can only remain unchanged if erank_M1 and erank_M0 increase by the same scale, which implies that erank_M0 needs to increase by a larger proportion (given erank_M0 < erank_M1), which is evidently challenging. Therefore, an increase in rank difference with larger hidden state dimensions is a natural outcome. Given that model size is positively correlated with the model's hidden state dimensions, it is also quite natural for there to be a positive correlation between model size and rank difference.

I believe a potential way to address my concerns is to fix the model size and compare the rank differences of different checkpoints. Ideally, the checkpoints should include cases of under-training, fully training (achieving the best accuracy), and overfitting. If a trend of rank difference first increasing and then decreasing is observed (with the best performance at the fully trained checkpoint), then the conclusions of the paper would be sufficiently reliable. Otherwise, if accuracy shows a trend of first increasing and then decreasing, while rank difference does not, then rank difference may not be a suitable substitute for accuracy.
Thank you for your reconsideration of our paper and the adjustment of the score. We are very grateful for your acknowledgment of the empirical contribution our work provides to the field. We assure you that the valuable suggestions and insights from you and other reviewers, as well as our explanations, will certainly be integrated into our revised version.

We sincerely appreciate the time and effort you've dedicated to this. Thanks again for your review and comments.
Thank you for the response. I have read other reviews, the author's rebuttal, and updated my scores accordingly. It would be very useful to integrate the explanations (including the outliers, the training evolution and the explanation for over layers) in the next version of the paper.