Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Models
Authors: Lai Wei
Zhiquan Tan
Chenghai Li
Jindong Wang
Weiran Huang
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Model
Multi-modal Large Language Model
Evaluation Metric
Primary area: evaluation
TL;DR: We propose a novel rank-based metric, Diff-eRank, to evaluate LLMs based on representations.

Abstract: Large Language Models (LLMs) have transformed natural language processing and extended their powerful capabilities to multi-modal domains. As LLMs continue to advance, it is crucial to develop diverse and appropriate metrics for their evaluation. In this paper, we introduce a novel rank-based metric, Diff-eRank, grounded in information theory and geometry principles. Diff-eRank assesses LLMs by analyzing their hidden representations, providing a quantitative measure of how efficiently they eliminate redundant information during training. We demonstrate the applicability of Diff-eRank in both single-modal (e.g., language) and multi-modal settings. For language models, our results show that Diff-eRank increases with model size and correlates well with conventional metrics such as loss and accuracy. In the multi-modal context, we propose an alignment evaluation method based on the eRank, and verify that contemporary multi-modal LLMs exhibit strong alignment performance based on our method. Our code is publicly available at https://github.com/waltonfuture/Diff-eRank.

Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Models
Lai Wei
Zhiquan T
Chenghai Li*
Jindong Wang?
Weiran Huang
MIFA Lab, Qing Yuan Research Institute, SEIEE, Shanghai Jiao Tong University
Department of Mathematical Sciences, Tsinghua University
3 William & Mary 4 Independent
Abstract
Large Language Models (LLMs) have transformed natural language processing and extended their powerful capabilities to multi-modal domains. As LLMs continue advance, it is crucial to develop diverse and appropriate metrics for their evaluation. In this paper, introduce a novel rank-based metric, Diff-eRank, grounded in information theory and geometry principles. Diff-eRank assesses LLMs by analyzing their hidden representations, providing a quantitative measure of how efficiently they eliminate redundant information during training. We demonstrate the applicability of Diff-eRank in both singlemmdal (c.g.. language) and multi- modal settings. For language models, our results show that Diff-eRank increases with model size and correlates well with conventional metrics such as loss and accuracy. In the multi-modal context, we propose alignment evaluation method based on the eRank, and verify that contemporary multi-modal LLMs exhibit strong alignment performance based our method. Our code is publicly available at https: /ttp:/giitttt thub. com/wa wal wonfttorruturffftooo. eRank.
Introduction
Large Language Models (LLMs)) such as GPT [4, 23], Chinchilla [17], and PaLM [7], have gained considerable attention for their outstanding performance in various natural language processing tasks. LLMs have expanded from single-modal models to multi-modal models, including MiniGPT-4 [49] and LLaVA [20], which have achieved remarkable results in various application scenarios. Pre-trained LLMs rely on large networks, computational power, and massive amounts of data, aiming for greater generalization capabilities.
LLMs understand the world knowledge through training on huge amounts of data. One famous belief [34] of how LLMs work is that larger models can find more shared hidden structures in data samples by eliminating redundant information through training. In particular, in the early phase of training. following random initialization, the representations derived from the training data tend to be somewhat chaotic. As training progresses, these representations become increasingly structured, and the model discards extraneous information from the training data, which resembles process similar to "noise reduction"". This perspective motivates us that LLM could evaluated by characterizing the "noise reduction" process.
However, defining and quantifying the degree of "noise reduction" remains significant challenge. To address this, we hypothesize that a reasonable metric should 1) reflect the geometric characteristics of the data such as the dimensionality its representations, and be rooted in information theory.
"Lai Wei (waltonfuture @ sjtu.edu edu.cn) and Zhiquan Tan (lanzq21 @ mails. .singgua.eduu.cn) contributed equally. I Correspondence Weiran Huang Ceairan.haang@@ee @outlook.com).
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

In this paper, we introduce Diff-eRank (difference between effective ranks), an cobormation-thhoreeee metric that fulfills both criteria, providing a measure for quantifying "noise reduction" in LLMs. In particular, we consider the effective rank (eRank) of the representations extracted by an L.L. from a dataset to measure the uncertainty, based on concepts from (quantum) information theory [30]. Through the removal of redundant information, eRank decreases, indicating the representations become more structured and compact. Thus, the reduction of representations' eRank can signify the degree of "noise reduction". Therefore, we can evaluate a well-trained LLM via the eRank reduction of the model representations, from its untrained status. We remark that different from conventional metrics like loss, which are derived from the predictions of LLMs, the proposed Diff-eRank focuses on the model representations. Our approach offers a novel perspective of model assessment, independent of prediction-baan based metrics, and can provide new insights into the understanding of LLM's behavior.
To verify the effectiveness of our approach, we conduct experiments on the contexts of both uni-modal LLMs and multi- L.Mss. In particular, for uni-modal I.I.ss, we compute Diff-eRanks for models within the OPT family [45] across various datasets. Intriguingly, we observe that Diff-eRan increases as the model scales, suggesting that larger models exhibit stronger noise reduction ability. Moreover, Diff-eRank has a consistent trend when compared with (reduced) cross-entropy loss and benchmark accuracy, highlighting its potential as effective and easy-to-use evaluation metric. For multi-modal (isionn-languae) LLMs, visual and language information is usually encoded separately by two independent encoders and aligned through a connecting layer. Therefore, evaluating the quality of modality alignment in multi- LLMs is crucial. Building on insights from uni mnill LLMs,
assess modality aligmment by examining the matching degree of eRanks between representations from different modalities. Additionally, this approach yields interesting observations within the context of such multi-modal architectures.
Our contribution can be summarized as follows:
We propose a rank-based metric, Diff-eRank, for evaluating LLMs, where Diff-eRank reflects the "noise reduction" ability of pre-trained language models. Diff-eRank focuses on the model representations, different from conventional metrics such as loss and benchmark accuracy.
We validate the effectiveness of Diff-eRank by observing its correlation with the trends in loss and downstream task accuracy the model scales up.
We also propose eRank-based modality alignment metrics for multi-modal LLMs, and verify that contemporary multi-modal LLMs exhibit strong alignment performance via our metrics.
2
Related Works
Evaluation of Large Language Models. Evaluation of LLMs is a fast-evolving field across various tasks, datasets, and benchmarks [5, 33, 36, 48]. Precise evaluations are important for the enhancement of language models' performance and reliability. Conventional metries such as accuracy. Fl [29], BLEU [24] and ROUGE [18] estimate between the annotated label and the prediction generated by the language model in different downstream tasks. Other metrics like perplexity and cross- entropy loss are independent of annotated labels and can be computed solely based on the input texts. However, these metries focus on "extrinsic" evaluation, assessing performance based on the predictions of LLMs. We propose Diff-eRank for "intrinsic" evaluation based on the input data's ' hidden representations of LLMs, concentrating on their "noise reduction" capabilities.
Information Theory for Understanding Deep Learning. Information theory has been used to gain significant insights into understanding neural networks. For example, the information bottleneck [37. is instrumental in explaining supervised learning. Recently, researchers have also utilized information theory to understand and improve (vision) semi and self-supervised learning [32. 35, 46, 47]. Notably, Zhang et al. [46] find the closed-fomm connection of matrix entropy and effective rank when the matrix is positive semi-definite. As for language models, prior works [16, 25, 41] also used information theory to analyze hidden representations by training probes on specific downstream tasks to estimate the information contained in the pre-trained language model. Several other works explore the lossless compression of L.LMs with arithmetic coding [10, 40] based on information theory. In this paper, we take further step toward evaluating I.LMs through the proposed Diff-eRank rooted in information theory, which represents complementary perspective to these prior studies.

---

The Proposed Metric for Evaluating LLMs
In this section, we will introduce a rank-based metric called Diff-eRank for evaluating LLMs. The proposed metric is based on the representations obtained by an LLM, fundamentally diverging from conventional metrics like loss, which are based on the model's predictions.
When processing a sequence of tokens, an LLM will generate representation (i.e., the hidden states before the last classification head) for each token within the sequence. These nigh-dimensional representations are usually used to capture the semantic and syntactic information of the sentences. This inspires us to consider evaluating L.Mss by analyzing these representations. study the characteristics of these representations by examining their ranks through both the geometric and information-theoree
particular, we
perspective. On the one hand, studying the rank of these representations allows to measure the extent of linear independence among them, which corresponds to the effective dimensions the representation space (i.e., the geometric structure). On the other hand, rank is also related to the amount of information contained in these representations, while lower rank indicates that the information has been structured or compressed. Therefore, we consider to leverage the rank of data representations encoded by LLMs for model evaluation.
However, the size of data representation matrix varies with the sample size, making it less suitable for consistent analysis. Therefore, instead of directly computing the rank of the data representations, we use the rank of their covariance matrix, which has a fixed size and also contains all the essential information. To see this, let S = [Z1, ..,..... 2N denote the set of data representations, and be the mean representation. The rank of data representation matrix can be re-formulated as
rank([2] - 2, NN - 2)) = rank
[1
2, 2.... ,2N
21/21 - 2. ... NN 2]
N
N
rank
2;
2) 2)
N
where the last term is exactly the rank of covariance matrix. We remark that the above rank also equals to the dimension of the affine subspace spanned by S [z].
The formal construction of covariance matrix is shown as follows. For case of analysis, each z; being normalized.
Definition 3.1 (Construction Covariance Matrix). Given set of representations S {zz E IRd i i= .,,,...,A the covariance matrix Eg is constructed as
2,
2:
Es
2
22
where E 2:/N is the mean representation and notation
represents lo norm.
Since rank is highly sensitive to outliers [27], instead use its "continuous" counterpart, the effective rank (eRank), when applied to the covariance matrix, defined as below.
Definition 3.2 (cRank [27]). The effective rank of any non-zero matrix E gaxN is defined as
Σ
eRank(A) exp
log
where Q min( (, d) and σı, 02...., are the singular values of matrix A.
We remark that the above eRank is closely related the matrix entropy (i.e., Von Neumann entropy for matrices [42]), which is defined in Definition 3.3. In fact, Zhang et al. [46] point out that, for covariance matrix of normalized vectors, cRank Es) is the same exp(H(E) (Es)).
Definition 3.3 (Matrix Entropy). Given positive semi- matrix K Rdxd satisfying
ir (K) = 1, the matrix entropy of matrix K is defined as
H(K) = tr (Klog K).
3

---

It is equivalent to the Shannon entropy [31] over the spectrum, i.e.
H(K) Σ A log A,
where A1, Ag,..., bag....,Aa Aa the eigenvalues of matrix K.
Note that eRank of the covariance matrix is commonly interpreted as measure of the "degree of freedom" that the sentence contains in geometric sense, one may wonder whether there is more "information-theoretee
explanation for it. Interestingly, under the terminology of quantum information theory [44], if we regard the representation of each token state in quantum system, the construction given by Definition 3.1 is a standard process constructing a density matrix. From the quantum noiseless coding theorem [30], the entropy of density matrix H(Eg) represents the average number of qubits required to encode the states. Therefore, exp (H(Es)) can be viewed as a measure of randomness for sentence through the quantum information theory.
As eRank measures the amount of uncertainty system, we can now define Diff-eRank to measure the degree of "noise reduction" for an LLM.
Definition 3.4 (Diff-eRank). Given a sentence z, an untrained language model Mo. and compute- optimal [17] trained language model Mr we obtain two sets of representations, Mo(x) and (x), by processing each token through the respective models. Then the rank difference (i.e., Diff-eRank) between these two models based on sentence x is defined as follows:
A eRank(æ, Mo, M1) eRank [ (z)) - eRank (E),
where (.) is the covariance matrix of model M, 's representations sentence for E {0,
Upon completing training, the model's data representations shift from being random to struc- tured, enabling it to effectively capture patterns and structures from the data. In the above definition, the effective ranks eRank (Mp(=)) and eRank eRank(EM, (x)) quantify the uncertainty the represen- tations of the untrained and trained models, respectively. Thus, Diff-eRank A cRank(æ, Ma. M,) measures how much uncertainty the model has reduced as result of training.
The above definition applies to single sentence but can be extended to dataset consisting of multiple sentences. Specifically, Diff-eRank for the entire dataset can be defined as the average Diff-eRank across all sentences, formulated as follows.
Definition 3.5 (Diff-eRank of Dataset). Given dataset D consisting sentences F1,... . Any an untrained language model Mo, and compute-optimal [17] trained language model M1, Diff-eRank of dataset D is defined as
D H (EMo(as))
Σ H (EMi(.))
A cRank(D, Mo. M1) = cxp
exp
In summary, Diff-eRank reflects the dimension reduction of the space spanned by data representations. can be viewed measure of removing redundant information in the data for a compute-opting language model. A higher Diff-eRank indicates more organized and structured internal representations of the model, therefore reflecting the model's increasing effectiveness in capturing patterns and regularities in the data.
Evaluations of Large Language Models
We start with evaluating different sizes of language models via Diff-eRank in Section 4.2. We find that Diff-eRank increases as the model scales up on various datasets. Additionally, we extend the application of eRank multi-modalities beyond the language domain in Section 5.
4.1 Experimental Settings
4.1.1 Model Choice
We experiment by using popular transformer-hased language models from OPT [45] family, rang- ing from 125 million to 13 billion parameters. Such diversity in OPT's model size allows for

---

dclly-15k
wikipedia
spanwelken22
N--tif
Diff- Offf-eRank
-..
Dift- Diff-eRank
Diff- eRank
Diff- BiffeRank
107
Iwa
307
30
13
10
20
Model Size
Model Size
Model Size
Model Size
11.00
delly-15k
wikipedia
opanmetteet2
Ni-thff
Reduced Loss
Reduced Loss
Reduceed Loss
Reduced Loss
13
Model Size
Model Size
Model Size
Model Size
Figure 1: Comparison of Diff-eRank and reduced loss when model scales up across various datasets. Both Dift-eRank and reduced loss show an upward trend when the model scales up.
comprehensive analysis across different scales of pre-trained language models in our experimental setting. We refer the reader to Appendix A for additional implementation details about the selection of language datasets.
4.1.2 Metric for Comparison
Given a text sequence U = [uig..., WT], the cross-entropy loss of a language model M can be defined as
L(U, = -) - Σ log Plu u;-1).
The cross-entropy loss is a canonical quantity in Shannon information theory, based on the model's predictions. As we study the rank difference between untrained model Mo and compute-optimal trained model M2 based on representation, spondingly. Therefore, we can similarly define reduced (cross-cntropy) loss as
adopt the difference in loss for comparison, corre-
AL Mo, M1) = L(U,
L(U,,,)
As the training progresses, the LLM gets better predictions on the input data, leading to increase reduced loss. Therefore, reduced loss can also be seen as useful evaluation metric for LLMs, and we use it for comparison with Diff-eRank in our following experiments.
4.2 The Trend of Diff-eRank with Model Size
To substantiate Diff-eRank as viable metric for evaluation, evaluate the series of OPT [45] models over different and diverse datasets using Diff-eRank and (reduced) loss for comparison. Specifically, consider including pre-training datasets such Wikipedia [14] and openwebtext2 [15], instruction dataset dolly-15k [8]. and preference dataset hh-rlhf [2] for the diversity of their usage.
Figure presents that Diff-eRank and reduced loss both increase progressively the model scales up. The increase reduced loss (equals to decrease in cross-entropy loss) can be interpreted larger models providing closer predictions to the actual values that they can better capture the underlying patterns and relationships within the data. As for the increase in Diff-eRank based on hidden representations, it suggests that the redundant dimensions of the data be effectively reduced in the larger models, thereby resulting in stronger "noise reduction" abilities and larger DankssRanks. Overall, the strong correlation between Diff-eRank and (reduced) loss indicates that Diff-eRank can provide a novel and inspirational evaluation for LLMs through the lens of "noise reduction" in dimension spaces. We summarize detailed results tables in Appendix B.
4.3 Relationship among Benchmark Metries
Based on the exploration in the earlier section, natural question arises: does Diff-eRank relate the downstream task accuracy of large language models? To address this question, we integrate accuracy
5

---

Table I: Comparison of benchmark metrics on openbookqa [22] and piqa [3]. AcC denotes benchmark
accuracy and AL indicates reduced loss. The results indicate that larger Diff-eRank values generally correspond to higher model performance.
OPT MODELS
SiZE
BENCHMARKS
INDICATORS
125M
1.3B
2.7B
6.7B
13B
Acc
0.276
0.332
0.370
0.360
0.366
OPENBOORQA
AL
5.734
6.138
6.204
6.258
6,236
DIFN-ERANK
1,410
2.140
2.338
2.280
3.032
AcC
0.619
0.714
0.733
0.756
0.767
PIQA
AT.
6.472
6.928
6.999
7.077
7.068
DIFF-ERANK
4.647
6.294
6.774
6.950
7.267
as comparative metric in addition to Diff-eRank and reduced loss in our evaluations on benchmark datasets. We use the evaluation set of openbookqa [22] and piqa [3] by combining the question and correct answer of each piece of data as inputs.
The results presented in Table illustrate a similar relationship among Diff-eRank, reduced loss, and downstream task accuracy. All of these three metrics increase when model size increases. Although occasional outliers are observed in the upward trends of these indicators, we think this is normal and does not affect the overall trend. Therefore, it can be concluded that Diff-eRank generally correlates with the trend of loss and accuracy, particularly as the model size scales within the same model family. An increase in Diff-eRank (i.e., higher denoising ability of the model) corresponds to enhanced model performance (i.e., higher reduced loss and higher accuracy), which shows great potential in the evaluation of LLMs.
Evaluations of Multi-Modal Large Language Models
After verifying that Diff-eRank can indeed reflect the LLMs' intrinsic ability in the previous sections, our study extends to the evaluation of Multi-modal Large Language Models (MLLMs) [1, 20, 43, 49]. We define metries based on the eRank evaluate the modality alignment.
5.1 Experimental Settings
For our multi-modal experiments, select two advanced and open-sourced MLLMs shown Table in the appendix: LLa [19] and MiniGPT-v2 [6]. Both the two MLLMS utilize simple connector for aligning the vision encoder with the LLM, providing a streamlined approach to multi-modal learning. conduct the experiments on two high-quality multi-modal instruction datasets: detail 23k [20] and sbu align [49]. Each piece of data in these datasets contains triplet of image, instruction, and response. We concatenate the instruction and response of each triplet as the textual input in experiments.
5.2 Empirical Observations
Most of the MILLMs typically employ projector mechanism (usually linear layer or MLP), which aligns image representations from a vision encoder (usually ViT [12]) with L.M''s language represen- tations. Our experiments include analyzing the effective rank of representation of images post vision encoder (eRank)) and post connector (eRankg), as well as the representation output by the LLM for individual images (eRanka),, text (eRanka), and image-text pairs (eRanks),, as shown in Figure 2. To measure the "modality alignment" of MLLMs, we introduce two distinct metrics based on eRank:
eRank; eRanke
Image Reduction Ratio
eRanki
and
avg wvg(eRankg, eRank4, eRanks) max(eRank3, eRank, eRank;)
Image-Teas Texte-Tettaoe Alignment

---

Output
Table 2: Multi-modal LLMs' results. "Image Reduc- tion Ratio" and "Image-Tex: Alignment" measure the degree of "modality alignment" based on eRank.
eRank
Immage Teat)
(Image only)
Rank
eRank
only)
LLM
EXPACTIVE RANK
LLAVA-1
MINICIFTE07
DETAIL_22%
CC_SEU_AUICN
DETAIL__266
CC_SRC_AL.ON
Rank2
<Bask
18.34
9:00
90.59
74.79
elborky
11:76
5:30
55.70
Connector
(Bask
> eanky
cltskk
cBusk
16.34
47.07
Vision Encoder
Deans REDECTIONN RAtio
0.3550
0.3568
0.7618
0.1331
0.7084
0.3329
0.4955
Image: x
Text: t
Table 3: Results of the image operation by clockwise rotating.
Figure 2: Illustration of the eRank mea- surement in the MLLM framework. The evaluation encompasses the effective rank of image representations after the vision en- coder (eRanki)), post-connector representa- tions (eRanky), as well as the output rep- resentations generated by the LLM includ- ing individual images (eRanks), textual data (eRank4), and the combined image-text pairs (eRanks).
LLAVA 1.5 ON DETAII_
DIPECTIVE RANK
ROTATE
cluunay
18.34
19.20 (7)
cRanka
claank,
45.62
46.54 (7)
cRaak,
74.21
74:21
<Ranky
76.34
77.69 (1)
REDECTION RamrO
0.3850
0.3588
INAAE--TEX ALIONMENT
0.8566
0.8514 (.)
On the one hand, the "Image Reduction Ratio" metric is formulated to quantify the reduction in effective rank from the vision encoder output (eRankı) to the post-connector stage (eRank2). Note that normalization is necessary here for fair comparison because the vision encoder and connector are entirely different networks. This metric evaluates the connector network's 's efficiency in condensing and refining visual information during image-text alignment training. On the other hand, the "Image-Tex Alignment" metric is designed to evaluate the closeness among the effective rank of representations post LLM processing, considering individual images (cRanks), text (cRank4), and image-text pairs (cRanks) as inputs. In particular, the absolute eRank can be seen as the amount of absolute uncertainty or randomness. The mentioned three eRanks show how much the model integrates and represents each modality. If these three eRanks from different modalities are close to each other, it means that they align well from the perspective of information theory. Thus, this metric reflects the degree of closeness (i.e., alignment) among different modalities. A higher alignment score indicates a more proficient alignment between image and text modalities for MILMs.
Results in Table 2 exhibit the performance of two MLLMs, LLaVA-1.5 [19] and MiniGPT-v2 [6], across different datasets (detail 23k [20] and cc_sbu s___sb__alig align [49]). Both models align well as they all have a relatively high alignment score.
In particular, comparing the two models. LLa and MiniGPT-v2 both exhibit similar "Image Reduction Ratio" scores, indicating efficient condensation of visual information. Additionally, LLaVA-1.5 outperforms MiniGPT-v2 in "Image-Texx Alignment", suggesting a closer integration between visual and textual modalities. This finding also consistent with their performance, as 1.5 surpasses MiniGPT-v2 in most of benchmarks [9]. We leave exploring more comprehensive evaluation for multi-modal models via effective rank as future work.
To further investigate the role of each component in MLLM, we conduct additional experiments to caleulate the eRank after rotating the images clockwise. We summarize the results in Table 3. As the rotation of images introduces new semantic information into the model, by noticing all the image-related quantities (cRank, (i # 4)) all increase from the base model when performing rotation, this semantic influence can propagate through the model. Therefore, we suggest that the multi- modal model (including the connector and the language model) can indeed perceive subtle semantic variations in images, especially the position information. In addition, the "Image Reduction Ratio" score and "Image-Tez Alignment" score both decrease after conducting image rotation, suggesting that the connector performs less effectively in condensing visual information, and the rotated images
7

---

101
13377
Corebras-Che
Cembbres OpentLn
Consoras-OPf
Benchmark. Accuracy
120
Reduced Loss
Diff-eRank
190
10P
103h
1014
107
2014
2014
10P
1097
10
Model Size
Model Size
Model Size
Figure 3: Comparing Diff-eRank with reduced loss and benchmark accuracy across different model families, including OPT [45], Cerebras-GPP [11], and OpenELM [21].
are less well-aligned with the corresponding text. This is primarily because the rotation alters the spatial relationships within the image, possibly making more challenging for the model maintain the coherence between visual and textual information. Overall, this experiment indicates that subtle changes in the vision encoder's 's understanding of images can be effectively conveyed the LLM part and affect the MLLM's modality alignment. It demonstrates the validity of such popular multi-modal architecture.
In conclusion, these rank-based approaches enable a thorough understanding how well the multi- modal models align different modalities of data and how the models process and integrate different forms of input data.
6 Ablation Study
To better confirm the rationality of our algorithm and experimental design, we further conduct a series of ablation studies.
6.1 Different Model Families
Besides observing Diff-eRank on the OPT family, we also conduct experiments on Cerebras-Gie [11] family and OpenELM [21] family. LLMs in these three families are all pre-trained well on public data and range in various sizes. To demonstrate that Diff-eRank is not dependent on specific datasets, choose not to use benchmark datasets but instead select a general dataset. In particular, we adopt the dolly-15t [8] dataset to compute Diff-eRank along with reduced loss, and we calculate average benchmark accuracy of winogrande [28] and piqa [3] for these three LLM families. The empirical findings in Figure 3 substantiate the increase of Diff-eRank within these LLM families as the models scale up. which correlates with the trend of reduced loss and benchmark accuracy. This observation shows the potential of Diff-eRank as an insightful metric for the evaluation of different model families.
6.2 Algorithm Design
In this section, we choose other types of algorithms for designing Diff-eRank between untrained model Mo and trained model M1. The goal is to validate that the increasing type relation is robust to the algorithm we used.
We denote our standard computation of effective rank on a dataset D (Definition 3.5) as "Algorithm (a)", which calculates the effective rank based on the average matrix entropy. In addition, also consider the operation of calculating the average effective rank on dataset DD denoted by "Algorithm (b)". Specifically, for an LLM M, the effective rank dataset D of Algorithm (b) is defined
eRank ("), ExeD exp(H(EMM(s))
Exep eRank Σ EM(s)
[D]
D
8

---

Different Algorithms
Algorithm (b)
Table 4: Diff-eRank on different layers of OPT mod- els. Only the Diff-eRank on the last layer indicates an increasing trend.
Algerithm (a)
Dift eRank
OPT MODELS
125M
1.3B
2.7B
6.7B
13B
6.39
FIRST LAYER
73.07
73.03
66.93
49.24
41.83
107
1gth
MIDDLE LAYER
87.75
51.98
56.16
66.63
73.88
Model Size
LAST LAYER (?)
54.35
76.39
83.02
89.60
89.81
Figure 4: Different designs for Diff-eRank.
Therefore, Diff-eRank between untrained model Mo and trained model M1 of Algorithm (b) can be formulated as
A AeRank (D, (D, Mo, M1) eRank (b) Mo) eRank (6) (D, (D,M,) M1).
To compare these two ways for defining Diff-eRank, we conduct experiments using OPT models on dolly-15k dataset. The experimental results in Figure demonstrate that Diff-eRank consistently increases across model sizes, irrespective of whether Algorithm (a) Algorithm (b) is used. This observation verifies that the increasing trend for Diff-eRank robust across different algorithms of effective rank defined on dataset.
6.3 Measure Diff-eRank on Different Layers
In our research, we predominantly concentrate on the last layer of LLMs, as it usually represents the most comprehensive information encoded by the model. This layer may offer the most indicative measure of Diff-eRank. Besides, we also extend our experiments to encompass additional layers within the models. Specifically, our investigations include analyses of the first layer, the middle layer, and the last layer for language models in the OPT [45] family on d5lly-15k [8] dataset. Our findings in Table reveal that only the Diff-eRank on the last layer reveals an increasing trend across model sizes, which indicates that it's reasonable analyze data representation through the last layer that encodes the most comprehensive information of the model. This may be interpreted that L.L. is an integrated system where information processing occurs across the entire architecture. If we rely on early layers for analyzing Diff-eRank, this could lead to loss of important information and we may miss crucial information processing that occurs in subsequent layers. The last layer, on the other hand, integrates this information, providing a more complete representation of the input data. The observation in our experiments reveals that early layers do not exhibit clear patterns terms of Diff-eRank. This underscores the importance of considering the model as whole when analyzing the representation.
Conclusion and Discussion
We introduce Diff-eRank, a new metric that can measure the "noise reduction" ability of LLM based on data representation and reflects the extent which a pre-trained LLM eliminates the redundant dimension in the information-theored the data and is grounded information theory. The empirical investigations show that the Diff- eRank increases when the model scales and correlates with the trend of loss and downstream task accuracy. Moreover, we use this metric to define the alignment metrics for multi-n multi-modal LLMs and find contemporary models align very well.
sense. Our method reveals the geometric characteristics of
However, we haven't conducted experiments to observe the change of Diff-eRank during the LLMs' whole pre-training and post training stages due to the limited computation resources. Future research may broaden the investigative scope by introducing the Diff-eRank in LLMs' complete training stages, In addition, some useful techniques like pruning. quantization, and distillation may benefit from such metrics that reveal internal redundancies. The Diff-eRank metric may aid identifying which parts of the model can be compressed without significant loss of information. We hope that Diff-eRank will open up avenues for future studies to explore how such internal representation metrics can be integrated into different potential cases.

---

Acknowledgement
This project was funded by National Natural Science Foundation of China (62406192) and MSR Asia StarTrack Scholars Program. The authors also thank Kai Chen (Beijing Academy of Artificial Intelligence) for the support of computation resources.
References
[1] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang. Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 12966. 2023.
[2] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 2x1::2204.05866 2022.
[3] Yonatan Bisk. Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. Advancey neural information processing systems, 33:1877-1901, 2020.
[5] Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. Evaluation of text generation: A survey. arXiv preprint arXiv:2006. 14799, 2020.
[6] Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krish- namoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision- vision-language multi-task learning. arXiv preprint Xxxx:23100094455 2023.
[7] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Revearch, 24(240):1-113,, 2023.
[8] Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open insrucction-unee 1lm. Dolly, 2023.
[9] OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. https://gi thub. . com/
www.coonm/c/ooccccooooce.o http//ppe-ccomppae cowpass/ conpass/opencomppass opencompass, 2023.
[10] Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, et al. Language modeling compression. arXiv preprint arXiv:2309. 10668, 2023.
[11] Nolan Dey, Gurpreet Gosal, Hemant Khachane, William Marshall, Ribhu Pathria, Marvin Tom, Joel Hestness, Cerebras-gpe gpt: Open compute-optimal language models trained on the cerebras scaer cluster. arXiv preprint arXiv:2304.032000 2023.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale, arXiv preprint arXiv:2010. 11999, 2020.
[13] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang. and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pages 19358-19369, 2023.
[14] Wikimedia Foundation. Wikimedia downloads. https: /wwpss.umpppp.aa/// wwkimedia..comm org..
[15] Aaron Go Vanya Cohen* Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus. http: 2019.
/ //Skylion007 wa:.:07.gitthae git.u.. http://www.ggg.a.aaa.//////aerer
[16] John Hewitt, Kawin Ethayarajh, Percy Liang, and Christopher D Manning. Conditional probing: measuring usable information beyond baseline. arXiv preprint arXiv:2109.09234, 2021.
10

---

[17] Jordan Hoffimann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
[18] Chin- Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarizaaion branches out, pages 14-88,200 2004.
[19] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023.
[20] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08483 2023.
[21] Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, and Mohammad Rastegari. OpenELM: An Efficient Language Model Family with Open Training and Inference Framework. arXiv.org, April 2024.
URL. https: //axivv..ii///////a org/ abs/2404. 14619v1.
[22] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. In EMNLP, 2018.
[23] OpenAL. Gpt-4 technical report. arXiv preprint arXiv: arXiv:2303.08774 2023.
[24] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 31 311-18 2002.
[25] Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. Information-theerrticc probing for linguistic structure. arXiv preprint arXiv:2004.3361,, 2020.
[26] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. International conference on machine learning, pages 8748-8763, PMLR, 2021.
[27] Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference, pages 606-610. IEEE, 2007.
[28] Keisuke Sakaguchi, Ronan Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
[29] Yutaka Sasaki et al. The truth of the f-measure. Teach tutor mater, 2007.
[30] Benjamin Schumacher. Quantum coding. Physical Review A, 51(4))2738, 1995.
[31] Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27 (3):379-423, 1948.
[32] Oscar Skean, Jhoan Keider Hoyos Osorio, Austin Brockmeier, and Luis Gonzalo Sanchez Giraldo. Dime: Maximizing mutual information by a difference of matrix- entropics. arXiv preprint arXiv:2301.08166 2023.
[33] Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, et al. Trustlim: Trustworthiness in large language models. arXiv preprint arXiv:2401. 055661, 2024.
[34] Ilya Sutskever. Stronger compressors find more shared structure. The Ilya's Talk, 2023.
[35] Zhiquan Tan, Jinggin Yang, Weiran Huang, Yang Yuan, and Yifan Zhang. Information flow in self- supervised learning. arXiv preprint arXiv:2309. 172881 2023.
[36] Zhiquan Tan, Lai Wei, Jindong Wang, Xing Xie, and Weiran Huang. Can i understand what i create? self-knowledge evaluation of large language models. arXiv preprint 4rXiv:2406.06144 2024.
[37] Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (inw), pages 1-5. IEEE, 2015.
[38] Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method, arXiv preprint physic///0000007 2000.

---

[39] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint 09280880,,, 2023.
[40] Chandra Shekhara Kaushik Valmeekam, Krishna Narayanan. Dileep Kalathil, Jean- Man-Fraacoois Chamberland, and Srinivas Shakkottai. Llmzip: Lossless text compression using large language models. arXiv preprint arXiv:2306.04000 2023.
[41] Elena Voita Ivan Titov, Information thhorretic probing with minimum description length. arXiv preprint arXiv:2003. 12298,, 2020.
[42] John Von Neumann. Mathematicche grundlagen der quantenmechamik, volume 38. Springer-Verlag, Verlag, 2013.
[43] Lai Wei, Zihao Jiang, Weiran Huang, and Lichao Sun. Instructiongpt-4; A 200-instruction paradigm for line-tuning minigpt-4, arXiv preprint arXiv:2308. 12067, 2023.
[44] Mark M Wilde. Quantum information theory. Cambridge university press, 2013.
[45] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diah, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0 00000.. 2022.
[46] Yifan Zhang, Zhiquan Tan, Jinggin Yang, Weiran Huang, and Yang Yuan. Matrix information theory for self-supervised learning. arXiv preprint arXiv:2305. 17326, 2023.
[47] Yifan Zhang, Jingqin Yang, Zhiquan Tan, and Yang Yuan. Relationmatch: Matching in-batch relationships sumi-super learning, arXiv preprint wwXiv:2305.10397 2023.
[48] Lianmin Zheng, Wei-Lin Chiang. Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chathot arena. arXiv preprint Xxxv:2306.05683 2023.
[49] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint XXx::2304.1059. 2023.
12

---

Appendix
A Implementation Details
A.1 Language Datasets
Pre-training Datasets. All sizes of OPT models are pre-trained on various datasets, including Wikipidea [14] and openwebtext2 [15]. Due to resouree constraints, we select the subset of these datasets by random sampling 10 thousand pieces of data ((which is further discussed in Section D)) for the Diff-eRank observation. In addition to the datasets utilized for pre-training the models, we also incorporate supplementary datasets that were not directly involved in the OPT model's pre-training process as follows.
Instruction- Instruction-Tuning Datasets, For the Diff-eRank observation, we choose dolly-15k [8], which is generated by human employees, as one the instruction datasets. Specifically, we select the "context" part of this dataset as input because it contains more informative text.
RLHF Dataset. We utilize hh-rlhf [2] that consists of human preference data about helpfulness and harmlessness as the RLHF dataset. Each piece of this dataset contains pair of texts, one "chosen" and one "rejected". We feed the "chosen" part of the dataset into models and calculate the performance indicators.
Benchmark Datasets. For the observation of benchmark indicators, select openbookqa [22], winogrande [28] and piqa [3] for evaluation. These benchmarks are structured multiple-choice format. We combine the question and correct answer of each piece of data as inputs.
A.2 Multi-modal Model Architecture
Recent Multi-modal Large Language Models (MLLMs) utilize similar model architecture by con- structing simple connector network to align the vision encoder with the LLM. This architecture simple and efficient in aligning the vision and language information, utilizing strong LLM as the "CPU" of the multi-moda, model. We showcase the architecture of LLaVA-1.5 and MiniGPT-v2 used in our experiments Table 5.
Table 5: The model architecture comparison between LLaVA-1.5 and MiniGPT-v2.
ARCHITETTRRE
LLAVA-1.5
MINIGPT-V2
VISION ENCODER
CLIP-VIT [26]
EVA-VIT [13]
CONNECTOR
LLM
VICUNA-VI.5 [48]
LLAMA-2-CHAT [39]
A.3 Compute Resources
We conduct our experiments using NVIDIA A800-80G GPUs. The experimental time using a single A800 for calculating the Diff-eRank for 1.3B LLM on the dolly [8] dataset is around hour.
B Complete Experimental Results
Table 6 contains the complete results for the comparison of Diff-eRank and reduced loss based on OPT [45] family considered in Figure 1. Table and Table illustrate the numerical results of different model families when using Diff-eRank and reduced loss for evaluation. Table 9 showcases the whole ablation results discussed in Section 6.2.
13

---

Table 6: Language modeling indicators on dolly-15k, 15k, Wikipedia, openwebtex12 and hh-rlhf.
OPT MODELS SIZE
DATASETS
INDICATORS
125M
1.3B
2.7B
6.7B
13B
DIAE-ERANK (t)
54.35
76.39
83.02
89.60
89.81
DOLLY-15K
(t)
7.6838
8.322
8.4471
8.5961
8.6505
DIAF- (t)
83.55
136.20
148.59
161.09
162.88
WIKIPEDIA
(t)
7.8671
8.4575
8.5746
8.7009
8.7581
DIRE-ERANK (t)
103.23
184.76
205.48
254.30
262.70
OPENWEBTEXT2
(t)
7,8090
8.3601
8.4697
8.5915
8.6396
DIFF-ERANK (1)
53.02
76.44
82.82
90.41
93.30
HI--LHH
AL
8.1041
8.4800
8.5242
8.5914
8.6928
Table 7: Comparison of Diff-eRank, reduced cross-entropy loss, and benchmark accuracy for models OpenELM [21] family.
MODEL SIZE
270M
450M
1.1B
3B
DINK-ERANK (1)
114.76
115.69
117.53
119.40
(t)
8.5164
8.6417
8.8210
9.0060
ACC (t)
0.6183
0.6516
0.6865
0.7188
Table 8: Comparison of Diff-eRank, reduced cross-entropy loss, and benchmark accuracy for models Cerebras- GPT [11] family.
MODEL SIZE
111M
256M
590M
1.3B
2.7B
6.7B
13B
DIFF-ERANK (f)
42.48
42.68
45.90
46.00
53.90
67.13
77.78
(T
7.1540
7.5343
7.7891
8.0733
8.3235
8.5339
8.5152
ACC (t)
0.5410
0.5620
0.5625
0.5925
0.6300
0.6705
0.7060
Table 9: Comparison of Algorithm (a) and Algorithm (b) for models in OPT [45] family.
MODEL SIZE
125M
1.3B
2.7B
6.7B
13B
ALGORITHM (B)
66.81
103.78
114.60
128.99
131.42
ALGORITHM (A)
54.35
76.39
83.02
89.60
89.81
Table 10: Comparison of metrics across different training stages.
METRICS/TRAINING STAGES
RANDOM INITIALIZED
INITIALIZED FROM OPT-1.3B
FULLY TRAINED
OVERFITTING
DIFF-ERANK
0.000
2.140
2.161
2.156
Loss
10.830
4.692
4.663
ACCURACY
0.250
0.332
0.340
0.336
14

---

C Additional Experiments
To further investigate how "Diff-cRank" changes during training, we conduct additional experiments to observe the behavior of Diff-eRank across different training stages for fixed model size. In partieular, we fix the model size by using the pre-trained OPT-1.3B [45] model and continually train it on a cleaned Wikipedia [14] dataset.
According the additional experimental results in Table 10, observe that the trend of Diff-eRank, first increasing before fully trained and then slightly decreasing when overfitting, aligns well with the trend of benchmark accuracy and the opposite trend of loss. This suggests that Diff-eRank may serve as a complementary metric that helps understand the LLM's "noise reduction" behavior during training, and monitor the training progress.
D Additional Ablation Study
As mentioned in Appendix A. random sampling is employed to extract subsets from the whole datasets of Wikipedia [14] and openwebtext2 [15], each subset comprising 10,000 data entries, as these pre-training datasets are too large for computation. To assess the robustness of Diff-eRank in random selection, we incorporate variations in the sample sizes of the Wikipedia dataset this ablation study. Table 11 illustrates that fluctuations in the sample size bring insignificant influence on the Diff-eRank, which affirms the stability of Diff-eRank in random sampling. Thus, this ablation study indicates the rationality of the random sampling process when dealing with large pre-training datasets in our experiments.
Table 11: Ablation study of different sampling strategies on the Wikipedia [14] dataset.
SAMPLING STRATEGY
MODEL
STANDARD DEVIATION
10000
5000
1000
OPT-1.3B
136.20
132.39
136.14
1.782
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have mentioned in the abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
We have mentioned limitations along with the conclusion in the last part of our paper.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

Justification:
We provide them in our methodology part.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes)
Justification: We open-sourced models, which are easy reproduce.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
We provide codes in our supplemental material. The data we used is all open-sourced.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. ec/
Stwlless/Cooeuuuissiiiiiii Polocyy)))) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips... cc/ mublic/guides: /CodeSobmissssssseeeeee bolbmmsssionnl sesiisssisiiyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide in experimental settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We provide in our appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide it in our appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /tttar///////e/./eeeeee/ee cc htte/publcceEetee
h cttphiiccccclleeieccccc ines?
Answer: [Yes]
Justification:
We follow NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We follow the license and terms of use in experiments.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com /atasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "IslQgJKtoy", "reviewer": "Reviewer_y598", "summary": "The paper proposes a rank-based evaluation metric that quantifies the amount of redundant information in the hidden representations of a model and applies it to both text-only and multi-modal models. The effective rank is obtained by the rank of its covariance matrix and is interpreted using information theory. This rank represents the degree of variations among the principal components in data.  The rank difference between two models is used to measure how much redundant information is reduced in a model relative to another. The proposed metric aligns well with commonly used evaluation metrics (loss and accuracy).", "questions": "1. L224-225: can you provide any explanation on why these occasional outliers appear? This is much vivid in Table 4 where any other layer does not follow this upward trend.\n2. Was there any experiment also on how the rank difference involves over the training of a model and whether that signal is informative on how long a model can be trained?\n3. Is there any explanation on why the different layers are not informative of the representation redundancy (table 4)?", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper presents a new evaluation metric through the view of the geometric property of representation matrices and information theory and shows that it aligns well with commonly used metrics like loss and accuracy for LLMs.\n2. Experiments show how the rank difference varies by length, choice of the layer from which representations are extracted, type of models (text vs multi-modal), and algorithm design and can potentially guide future work on model compression.", "weaknesses": "1. It is unclear what additional information the absolute rank or the rank difference brings to the table apart from a new interpretation. The rank differences are hard to interpret (L261: both models align well with a high alignment score) given a lack of detail on how they are computed for models of varying sizes.\n2. For the multi-modal models, the authors again propose two metrics: image reduction ratio and image-text alignment and while the reduction ratio is a bit intuitive, it is unclear why image-text alignment is defined as is and why we need the different ranks (erank 3, erank 4, and erank5). In Table 2, the reduction and alignment follow opposite trends, is it something informative?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: It is unclear what additional information the absolute rank or the rank difference brings to the table apart from a new interpretation. The rank differences are hard to interpret given a lack of detail on how they are computed for models of varying sizes.\n\nA1: \nThanks for your comments. We would like to provide further explanations as follows.\n\nFirstly, as we have discussed in Section 1, **the effective rank of the hidden representations extracted by a model from a dataset can be considered as the uncertainty or randomness from (quantum) information theory. Meanwhile, the rank difference quantifies the reduction in uncertainty or noise in the representations before and after training for the model.** \n\nSecondly, we interpret that **as training progresses, a model's representations transition from being random and high-dimensional to more structured and lower-dimensional. The rank difference reflects this transition by measuring the extent to which redundant information has been reduced**, which is also discussed in Section 1 and 3. \n\nThirdly, the rank difference is computed based on the effective rank of the covariance matrix of the data representations (shown in Section 3), and it does not rely on model sizes. **Rank difference can be directly calculated and compared among models of varying sizes within the same family.**\n\nQ2: For the multi-modal models, it is unclear why image-text alignment is defined as is and why we need the different ranks. The reduction and alignment follow opposite trends. Is it something informative? \n\nA2: \nThanks for your questions. We would like to highlight these key aspects:\n\nFirstly, we want to design a metric to evaluate the degree of alignment between the image and text modalities, ranging from 0 to 1. When the MLLM reaches the perfect alignment, the metric is close to 1. As the alignment worsens, the metric will decrease. **In particular, the absolute rank can be seen as the amount of absolute uncertainty or randomness.** The ranks for individual images (erank3), text (erank4), and image-text pairs (erank5) show how much the model integrates and represents each modality. **If these three ranks from different modalities are close to each other, it means that they align well from the perspective of information theory.** Thus, we design this metric to reflect the degree of closeness (i.e., alignment) among different modalities.\n\nSecondly, regarding the observation where Image Reduction Ratio and Image-Text Alignment follow opposite trends, **it is important to note that these two metrics measure different aspects of the MLLMs and should be considered independently.** In particular, Image Reduction Ratio focuses on the model’s ability to condense visual information, while Image-Text Alignment measures the quality of alignment between different modalities. **The trends observed in Table 2 do not imply a direct relationship between the two metrics. Instead, they highlight different dimensions of MLLM's performance.**\n\nQ3: Can you provide any explanation on why these occasional outliers appear?\n\nA3: Thanks for your question. Actually, the observed deviations are within an acceptable range and do not affect the robustness of our findings. In particular, several factors may contribute to the presence of these outliers:\n\nTraining randomness: The training process of large language models inherently involves randomness. **Even within the same model family, slight differences in training dynamics can affect the performance in our evaluations.**\n\nOutliers are not unexpected: It is worth noting that outliers are not unique to the rank difference. **Other commonly used metrics, such as loss and accuracy, also exhibit occasional outliers in their trends across different model sizes. For instance, OPT-6.7B gains lower performance on openbookqa than OPT-2.7B in Table 1.** Thus, outliers are not unexpected in the evaluation of LLMs with a single metric.\n\nDespite these occasional outliers, the overall trend remains consistent and supports our conclusion that rank difference generally correlates with loss reduction and accuracy improvement as model size increases. \n\nQ4: Was there any experiment also on how the rank difference involves over the training of a model and whether that signal is informative on how long a model can be trained?\n\nA4: \nThanks for your question. To address your concern and further investigate how \"rank difference\" changes during training, we continually train OPT-1.3B on cleaned wikipedia dataset. We present the results in the table below.\n\n|Metrics/Training Stages|1|2|3|4|5|6\n|-|-|-|-|-|-|-|\n|rank difference ($\\uparrow$)|2.148|2.154|2.158|2.160|**2.161**|2.156\n|loss ($\\downarrow$)|4.655|4.654|4.653|**4.653**|4.654|4.663\n|acc ($\\uparrow$)|0.332|0.334|0.336|0.332|**0.340**|0.336\n\nThe additional results indicate that the rank difference shows a gradual increase during the model's training. The rank difference converges together with accuracy, which is later than the fluctuation of loss. It suggests that rank difference may be a useful metric for monitoring the training progress as it better correlates with the trend of accuracy.\n\nQ5: Is there any explanation on why the different layers are not informative of the representation redundancy?\n\nA5:  Thanks for your comment. We would like to offer several potential explanations: LLM is an integrated system where information processing occurs across the entire architecture. **If we rely on early layers for analyzing the rank difference, this could lead to a loss of important information and we may miss crucial information processing that occurs in subsequent layers.** The last layer, on the other hand, integrates this information, providing a more complete representation of the input data. We indeed observe that early layers do not exhibit clear patterns in terms of the rank difference in our experiments. This underscores the importance of considering the model as a whole when analyzing the representation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5aYKHQRJzt", "reviewer": "Reviewer_YACX", "summary": "The article presents a measure known as \"rank difference\" to assess the effectiveness of Language Models (LLMs) by analyzing their internal representations. This metric is based on information theory and geometric principles aiming to quantify how LLMs eliminate unnecessary information post training. The authors illustrate the utility of rank difference, in both modal (language) and multi modal contexts.\n\nIn terms of language models the study reveals that the rank difference grows with model size indicating noise reduction capability. This pattern aligns with metrics such as entropy loss and accuracy. Regarding modal models the authors suggest an assessment approach utilizing rank difference to evaluate alignment quality across modalities showing that contemporary multi modal LLMs demonstrate strong alignment performance.\n\nKey contributions of the paper include;\n\nIntroducing rank difference as a metric for gauging the \"noise reduction\" capability of trained language models.\nDemonstrating the correlation between rank difference, loss metrics and downstream task accuracy underscoring its potential as an evaluation criterion.\nDefining alignment measures, between linguistic modalities showcasing that modern multi modal LLMs excel in achieving alignment.\n\nThe study provides real world data that backs the idea of utilizing rank variance, in datasets and model scales indicating its reliability and effectiveness in assessing language learning models. In general the research introduces a viewpoint on evaluating language learning models moving away from metrics based on predictions, to focusing on model representations. This shift brings forth perspectives on comprehending how language learning models behave.", "questions": "Do you have any insights or early findings regarding the variations, in the rank difference metric as LLMs undergo training? Examining this aspect could offer insights into how the behaves and its usefulness throughout the model development process.\n\nHave you thought about assessing the rank difference metric across model families or architectures aside from the OPT family? Incorporating a range of models could help validate how broadly applicable the metric is.\n\nCould you delve deeper into comparing the rank difference metric with established evaluation metrics and recent research findings? This, in depth analysis would shed light on both the strengths and potential limitations of using the rank difference metric.", "limitations": "The authors recognize the limitations in conducting experiments to observe changes, in rank difference during training due to constraints. They emphasize the importance of conducting experiments to assess how applicable the metric is across model layers and its efficiency in computations.\n\nSuggestions for Enhancements;\nTraining Progress; Conduct a small scale study or simulation to observe how rank difference evolves during training with models or subsets of data.\nModel Variation; Assess the effectiveness of the rank difference metric across a range of model families and structures. Provide initial findings if a comprehensive evaluation is not feasible.\nComputational Efficacy; Explore optimizations for calculating rank difference, such, as enhancements or parallelization methods and briefly touch upon any societal implications and ethical considerations to demonstrate awareness of broader impacts.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Novelty:\nThe introduction of the rank difference metric offers an approach, to assessing LLMs by focusing on their representations rather than just their outputs.\nRooted in information theory and geometric principles providing a perspective on comprehending LLM behavior.\nExpanding the metrics scope to modal setups and evaluating alignment coherence across different modalities.\n\nQuality:\nAssessments spanning datasets and model dimensions.\nConvincing data showcasing the relationship between rank difference and traditional metrics like entropy loss and accuracy.\nAnalysis elucidating the mathematical underpinnings and implications of the rank difference metric.\n\nClarity:\nOrganized and easily understandable elucidation of ideas.\nThorough explanation of the proposed metric, its basis and practical implications.\nFollows a sequence from problem statement to methodology and outcomes.\n\nSignificance:\nTackles queries regarding LLM evaluation underscoring the necessity for metrics that go beyond mere model results assessment.\nPotential to shape how researchers and professionals evaluate and interpret LLMs offering insights, into model behavior and effectiveness.\nThis can be used in both mode and mode models making it versatile, for different situations.", "weaknesses": "The experiments have a scope;\nRegarding Training Dynamics; The paper fails to delve into how the rank difference evolves throughout training missing insights, into its behavior as the model progresses.\nConsidering Model Diversity; Broadening the evaluation to include a range of model families and architectures would enhance the validation of the rank difference metric.\n\nIn terms of Presentation;\nClarity; Some sections require explanations to improve readability.\nVisual Support; Incorporating diagrams or visual aids could boost understanding and reader engagement.\n\nDelving Into Analysis Depth;\nInsights by Layer; Analyzing rank differences across model layers would offer a holistic view.\nComparative Examination; Providing in depth comparisons with established metrics and recent studies would spotlight both strengths and limitations of the metric.\n\nApplication, in Real Scenarios;\nReal world Contexts; Integrating real world application scenarios or case studies would enhance the relevance of the paper.\n\nComputational Considerations;\nEfficiency Concerns; Addressing the efficiency of calculating rank difference for large scale models and suggesting optimizations could be advantageous.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: The paper fails to delve into how the rank difference evolves throughout training missing insights, into its behavior as the model progresses. \n\nA1: Thanks for your question. To address your concern and further investigate how \"rank difference\" changes during training, we continually train OPT-1.3B on cleaned wikipedia dataset. We present the results in the table below.\n\n|Metrics/Training Stages|1|2|3|4|5|6\n|-|-|-|-|-|-|-|\n|rank difference ($\\uparrow$)|2.148|2.154|2.158|2.160|**2.161**|2.156\n|loss ($\\downarrow$)|4.655|4.654|4.653|**4.653**|4.654|4.663\n|acc ($\\uparrow$)|0.332|0.334|0.336|0.332|**0.340**|0.336\n\nThe additional results indicate that the rank difference shows a gradual increase during the model's training. The rank difference converges together with accuracy, which is later than the fluctuation of loss. It suggests that rank difference may be a useful metric for monitoring the training progress as it better correlates with the trend of accuracy.\n\n---\n\nQ2: Broadening the evaluation to include a range of model families and architectures would enhance the validation of the rank difference metric.\n\nA2: Thank you for your comments. **We would like to clarify that we have indeed addressed this aspect in our study, specifically in Section 6.1 of our paper.** \n\n**In addition to the OPT family, we have extended our experiments to include two other model families: Cerebras-GPT and OpenELM.** These families represent a diverse range of well-trained LLMs of various sizes. **By including these different model families, we inherently incorporated a variety of architectures into our evaluation.** Each family has its unique architecture, allowing us to test the robustness of our rank difference metric across different model designs.  **As illustrated in Figure 5 of our paper, we observe a consistent increase in rank difference as models scale up across all three families.** This trend correlates with reduced loss and improved benchmark accuracy, supporting the validity of rank difference as an evaluation metric across diverse model architectures.\n\n---\n\nQ3: Some sections require explanations to improve readability. Visual Support; Incorporating diagrams or visual aids could boost understanding and reader engagement.\n\nA3: Thanks for your suggestion. We will add more discussions shown in our rebuttal to our paper.\n\n---\n\nQ4: Analyzing rank differences across model layers would offer a holistic view. Comparative Examination; Providing in depth comparisons with established metrics and recent studies would spotlight both strengths and limitations of the metric.\n\nA4: Thanks for your suggestions. We would like to clarify some aspects of our study to address your points:\n\nInsights by Layer: Our findings, as shown in Table 4, reveal that only the last layer demonstrates a consistent increasing trend in rank difference across model sizes. This may be interpreted as LLM is an integrated system where information processing occurs across the entire architecture. **If we rely on early layers for analyzing the rank difference, this could lead to a loss of important information and we may miss crucial information processing that occurs in subsequent layers. The last layer, on the other hand, integrates this information, providing a more complete representation of the input data.** Our experimental results indeed show that early layers do not exhibit clear patterns in terms of the rank difference. This observation supports our focus on the last layer and provides insights into how information is processed through the model.\n   \nComparative Examination: **In our study, we have indeed conducted comprehensive comparisons with the primary metrics used in LLM's evaluation, including loss and downstream task performance.** In particular, we demonstrate the correlation among our rank difference, the reduced loss and performance on downstream tasks in Section 4.2 and 4.3. This comparison shows how our metric aligns with the primary metrics. **To the best of our knowledge, loss and downstream task performance are the primary quantitative measures used for assessing LLM.** If you are aware of other specific metrics for comparison, we would greatly appreciate if you could provide references.\n\n---\n\nQ5: Integrating real world application scenarios or case studies would enhance the relevance of the paper.\n\nA5: Thanks for your suggestion. We aim to integrate real world applications into our future work. Rank difference may open up avenues for future research to explore how internal representation metrics can be integrated into different cases. Techniques such as pruning, quantization, and distillation may benefit from this metric that reveals internal redundancies. **The rank difference metric may also aid in monitoring the LLM's training process as discussed in A2.**\n\n---\n\nQ6: Addressing the efficiency of calculating rank difference for large scale models and suggesting optimizations could be advantageous.\n\nA6: Thank you for your suggestion. Our rank difference calculation method is designed with efficiency in mind, particularly for large-scale models. Here are some key points regarding the efficiency of our approach:\n1. The efficiency of calculating rank difference for large scale models:\n   - We use PyTorch's efficient tensor operations, which are optimized for GPU computation. **The core operations (normalization, covariance calculation, and effective rank computation) are matrix operations** that scale well with increasing model sizes.\n   - **The rank difference can be calculated using a relatively small dataset**, further reducing computational requirements.\n2. Potential Optimizations:\n    For extremely large models, we could implement batch processing of the representations to reduce memory requirements.\n\nWe are actively exploring additional techniques to enhance the efficiency of our method in the future. We appreciate your attention to this important aspect of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mK8ycAg6Jh", "reviewer": "Reviewer_c8rT", "summary": "This paper introduces “rank difference” that measures the reduction in the rank of LLM’s representations. It evaluates the quality of LLMs, which could be used in addition to the reduction in the cross-entropy loss. The idea is based on the assumption that LLM’s representations (e.g., the hidden states of each token before the classification head) encodes the semantic and syntactic information about the input sentence. Before training, these representations are expected to be chaotic (hence high rank), but after training and alignment, the representations are expected to be more structured. \n\nBased on this idea, “rank difference” is defined as the difference between $erank_{m0}(\\Sigma_S)$ and $erank_{m1}(\\Sigma_S)$ where $\\Sigma_S$ is the covariance matrix of representations and $erank$ is the effective rank. The difference is computed between the representation before and after training. This measures the extend to which the model can compress the information, while other metrics such as cross-entropy measure the quality of model’s prediction.\n\nExperiments are conducted on OPT models of sizes from 125m to 13B parameters. The results show that there is an upward trend between “rank difference” and model size (which reflects model performance) on a range of datasets, similar to the reduction in cross-entropy. In addition, rank difference is applied to measure the alignment between visual modality and text modality in Llava-1.5 and MiniGPT-v2. An ablation study on different model families is shown.", "questions": "1. How do you think this method could be adopted?\n2. What is the computational cost of computing rank difference?\n3. Would it be comparable across models of different representation sizes?\n4. Does/how rank difference during the alignment stage (e.g., SFT or PPO/DPO) ?", "limitations": "Limitations section is provided, and it has covered the main points", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper presents a new evaluation metric, grounded on information theory, and demonstrates that it correlates well with model performance (as measured by model size and downstream task performance). It also allows measuring the alignment between different modalities.", "weaknesses": "1. Although the idea is novel, its practical usability is limited. To evaluate the quality of systems (e.g., LLM, ASR, etc), downstream metrics such as accuracy, ROUGE, BLEU, WER, etc. are used as they better align with real use cases. While, a metric like cross-entropy is used as it is differentiable, allowing it to be a training loss. I’m not sure if “rank difference” will be adopted as it’s neither related to real use case nor applicable as a training objective.\n\n2. Regarding experiments, current results (both text-only and visual-text) only show the trend post-training, so it would be more indicative of demonstrating how “rank difference” changes during training (e.g., similar to cross-entropy loss which usually decreases monotonically during training).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: Although the idea is novel, its practical usability is limited. To evaluate the quality of systems (e.g., LLM, ASR, etc), downstream metrics such as accuracy, ROUGE, BLEU, WER, etc. are used as they better align with real use cases. While, a metric like cross-entropy is used as it is differentiable, allowing it to be a training loss. I’m not sure if “rank difference” will be adopted as it’s neither related to real use case nor applicable as a training objective. And, How do you think this method could be adopted?\n\nA1: Thanks for your thoughtful feedback regarding the practical usability of our proposed \"rank difference\" metric. Your points about the importance of downstream metrics and differentiable training losses are well-taken. We would like to address your concerns and highlight the potential value of our approach:\n\nComplementary to Existing Metrics: We agree that downstream metrics like accuracy, ROUGE, BLEU, and WER are crucial for evaluating system performance in real-world scenarios. **Our rank difference metric is not intended to replace these measures but to complement them by providing additional insights into model behavior during training and evaluation.**\n\nInsights into Model Representation: Rank difference can measure the \"noise reduction\" ability of LLM based on data representation and reflects the extent to which a pre-trained LLM eliminates the redundant dimension in the information-theoretic sense. **The rank difference metric provides unique insights into the internal representations of language models, which may not be directly captured by task-specific metrics.** This can be valuable for understanding model behavior from a theoretical perspective.\n\nFuture Research Directions: Rank difference may open up avenues for future research to explore how internal representation metrics can be integrated into different cases. Techniques such as pruning, quantization, and distillation may benefit from metrics that reveal internal redundancies. The rank difference metric may aid in identifying which parts of the model can be compressed without significant loss of information.\n\nThe adoption of new metrics in the field may take time and require extensive validation. Our work aims to contribute to the ongoing discussion about how to evaluate and understand LLMs from a new perspective. **We believe that a diverse set of evaluation tools, including both task-specific metrics and intrinsic measures like rank difference, can provide a more comprehensive view of model quality and behavior.**\n\n---\n\nQ2: Regarding experiments, current results (both text-only and visual-text) only show the trend post-training, so it would be more indicative of demonstrating how “rank difference” changes during training (e.g., similar to cross-entropy loss which usually decreases monotonically during training). Does/how rank difference during the alignment stage (e.g., SFT or PPO/DPO)?\n\nA2: \nThanks for your suggestion. In our paper, we haven’t conducted experiments to observe the change of rank difference during training due to the limited computation resources at first. To address your concern and further investigate how \"rank difference\" changes during training, we conduct additional experiments by continually training OPT-1.3B on cleaned wikipedia dataset. We present the results in the table below.\n\n|Metrics/Training Stages|1|2|3|4|5|6\n|-|-|-|-|-|-|-|\n|rank difference ($\\uparrow$)|2.148|2.154|2.158|2.160|**2.161**|2.156\n|loss ($\\downarrow$)|4.655|4.654|4.653|**4.653**|4.654|4.663\n|acc ($\\uparrow$)|0.332|0.334|0.336|0.332|**0.340**|0.336\n\nThe additional results indicate that the rank difference shows a gradual increase during the model's training. The rank difference converges together with accuracy, which is later than the fluctuation of loss. It suggests that rank difference may be a useful metric for monitoring the training progress as it better correlates with the trend of accuracy.\n\n---\n\nQ3: What is the computational cost of computing rank difference?\n\nA3: Thank you for your question. Below, we provide a detailed breakdown of the steps to reveal the computational cost.\n\nNormalization of Representations: For a set of representations, the complexity is $O(Nd)$ for mean subtraction and $O(Nd)$ for normalization, resulting in a total complexity of $O(Nd)$.\n\nConstruction of Covariance Matrix: The covariance matrix computation involves matrix multiplication, which has a complexity of $O(Nd^2)$.\n\nObtaining Singular Values: For a $d \\times d$ matrix, the complexity of SVD is $O(d^3)$ to obtain singular values.\n\nCalculation Effective Rank: The effective rank calculation involves summing over $d$ singular values, which has a complexity of $O(d)$.\n\nCombining the complexities of the individual steps, the total computational complexity for computing the rank difference is: $O(Nd) + O(Nd^2) + O(d^3) + O(d) = O(Nd^2 + d^3)$. The computational cost of computing the rank difference is primarily influenced by the dimensionality of the representations ($d$) and the number of tokens ($N$) in the dataset. Note that the relationship between $N$ and $d$ can vary depending on the model size. In smaller models, $N$ may be larger than $d$. In larger models, $N$ may be smaller than $d$.\n\n---\n\nQ4: Would it be comparable across models of different representation sizes?\n\nA4: Thanks for your question. We'd like to clarify that our experimental design specifically addresses this concern. **Our study is actually designed to measure models within the same family of different (representation) sizes in Section 4.** By focusing on the same model family, we ensure that the fundamental architecture and training paradigm remain consistent, while the model sizes and representation sizes vary. We believe this approach provides a fair and meaningful comparison, as it isolates the effect of model size and representation dimensionality while controlling for other variables.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UnrVi51TOC", "reviewer": "Reviewer_LQXh", "summary": "This paper explores the potential of using the rank of model hidden states for evaluating model capabilities. Specifically, the authors propose calculating the rank difference between trained and untrained models as a measure of model performance. The core idea is that the rank difference can reflect the model's extent of \"noise reduction\". The authors compared the consistency of rank difference with other metrics (loss, accuracy) under multiple model series settings. The results showed the potential of rank difference in assessing model capabilities within certain model series and measuring the cross-modal alignment of MLLMs.", "questions": "See Weaknesses.\n\nI am looking forward to authors' responses and open to rise my score.", "limitations": "The main limitation of this paper is that it fails to specifically discuss the advantages of rank difference compared to other metrics, especially considering that the rank differences of models from different series are not on the same scale.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- To the best of my knowledge, this paper is the first to explore and discuss the idea of using the rank of hidden states to evaluate model capabilities, which sounds promising.\n\n- The paper provides an intuitive understanding and theoretical significance of using rank difference as an evaluation metric.\n\n- The experiments in this paper are relatively comprehensive, and the writing is quite clear.", "weaknesses": "- My main concern lies in the fact that for models within a series, the dimensionality of their hidden states is often positively correlated with size, which also implies a positive correlation with performance. A larger dimensionality of hidden states usually means a larger rank difference when the effective rank proportion remains the same. In other words, although larger models have a higher rank difference, the proportion of noise reduction might be smaller compared to smaller models. Does this suggest that rank difference may not accurately measure the extent of noise reduction? Could rank ratio be a more reasonable metric (erank_M1/erank_M0 in eq.3)?\n\n- Although the authors observed a positive correlation between rank difference and model size within a single model series, this conclusion no longer holds when comparing models from different series simultaneously, as shown in Figure 3. Is this due to the varying dimensionality of hidden states across different models (where rank ratio might be effective)? Or is it caused by the different training methods employed for various models? If rank difference cannot be used to compare models from different series, what advantages does it offer compared to traditional metrics such as accuracy?\n\n- When the number of test set samples lies between the dimensionality of hidden states for the large and small models being evaluated, Q in Equation 1 will take values from N and d, respectively. Could this lead to non-robust results? I believe experiments should be conducted to investigate this scenario.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: A larger dimensionality of hidden states usually means a larger rank difference when the effective rank proportion remains the same. In other words, although larger models have a higher rank difference, the proportion of noise reduction might be smaller compared to smaller models. Does this suggest that rank difference may not accurately measure the extent of noise reduction? Could rank ratio be a more reasonable metric (erank_M1/erank_M0 in eq.3)? \n\nA1: Thank you for your thoughtful feedback and for suggesting the rank ratio as an alternative metric.  Below, we'd like to address this concern and explain our rationale for using rank difference over rank ratio.\n\nFirstly, we have also considered this metric initially and we agree that both rank difference and rank ratio can provide valuable insights. Each metric captures a different aspect of the change in effective rank between untrained and trained models. **The rank difference metric is designed to capture the absolute reduction in uncertainty or noise, which reflects the model's ability to compress and structure information.** This can be interpreted as a greater capacity for the model to distill relevant information and discard noise, even if the proportional change (as measured by rank ratio) might be smaller.\n\nSecondly, a decrease in rank ratio ($\\frac{erank_{M_1}}{erank_{M_0}} = 1 - \\frac{erank_{M_0} - erank_{M_1}}{erank_{M_0}}$) as model size grows ($erank_{M_0}$ will also grow) typically corresponds to an increase in rank difference ($erank_{M_0} - erank_{M_1}$). However, the converse is not necessarily true. Our additional experiments in Table A below reveal that the rank ratio exhibits an oscillating downward trend as model size increases, indicating a degree of instability. In contrast, we found that rank difference demonstrates a more consistent and stable pattern of increase with model size. **This inconsistency suggests that the rank ratio may not reliably capture the extent of noise reduction or the model's ability to structure information.**\n\n| Datasets/Models | OPT-125m | OPT-1.3B | OPT-2.7B | OPT-6.7B | OPT-13B |\n| --------------- | -------- | -------- | -------- | -------- | ------- |\n| Dolly  | 0.5398| 0.4719| 0.4548 | 0.4485 | 0.4593 |\n| Wiki |0.4761 | 0.3870| 0.3717| 0.3667|0.3822|\n| Openwenbtext2 | 0.4984|0.3936 |0.3761 | 0.3443|  0.3555 |\n| hh-rlhf  |0.5180 |  0.4396| 0.4218|0.4062|  0.4242|\n\nTable A: Using rank ratio as a metric for evaluation\n\nIn summary, while the rank ratio provides an interesting perspective, our empirical experiments support the use of rank difference as a more reliable metric for evaluating noise reduction in LLMs.\n\n---\n\nQ2: Although the authors observed a positive correlation between rank difference and model size within a single model series, this conclusion no longer holds when comparing models from different series simultaneously, as shown in Figure 3. Is this due to the varying dimensionality of hidden states across different models (where rank ratio might be effective)? Or is it caused by the different training methods employed for various models? If rank difference cannot be used to compare models from different series, what advantages does it offer compared to traditional metrics such as accuracy?\n\nA2: Thank you for your question. We would like to clarify this point. It is indeed normal that rank difference cannot be directly compared across different model families. **This is primarily due to the varying architectures, training methodologies, and objectives used in different model series.**\n\n**Besides, we would like to emphasize that during the LLM's pretraining phase, the focus is primarily on the validation loss rather than accuracy.** In addition, we have conducted further experiments that demonstrate that lower loss across different model families does not necessarily correlate with better performance. For instance, in our tests on the Wiki dataset, we observe that the test loss of LLaMA-7b is 1.72, while the LLaMA2-7b is 1.66 and the LLaMA3-8b is 1.85. While LLaMA3-8b has the highest loss, it actually demonstrates the best overall performance. **This example clearly illustrates that loss values cannot be reliably used to compare performance across different model families.**\n\nGiven these observations, the proposed rank difference, while not suitable for cross-family comparisons, still offers valuable insights within a single model series. **It shifts the evaluation towards model representations and provides new insights into the theoretical understanding of LLM's behavior, which traditional metrics like accuracy or loss alone may not capture.**\n\n---\n\nQ3: When the number of test set samples lies between the dimensionality of hidden states for the large and small models being evaluated, Q in Equation 1 will take values from N and d, respectively. Could this lead to non-robust results? I believe experiments should be conducted to investigate this scenario.\n\nA3: Thanks for your question. In fact, this situation won't lead to non-robust results in our evaluations. Specifically, in our experiments, we did not encounter a scenario where d is consistently smaller than N, primarily because N would exceed the maximum input context length of the models. However, we have conducted experiments where d is consistently larger than N. Specifically, in Section 4.3, we select datasets where the sequence length N for each sample is less than 100, which is smaller than the dimensionality d of the models, as even the smallest model has a dimensionality of 768. Our findings in Section 4.3 indicate that the rank difference still increases as the model scales up. Thus, the results are robust even when d is consistently larger than N. **This consistency suggests that though d is sometimes larger than N and N is sometimes larger than d, it does not lead to non-robust results.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a new rank-based evaluation metric for LLMs which quantifies the amount of redundant information in the hidden representations of a model and applies it to both text-only and multi-modal models. The idea is to compute the "rank difference" between trained and untrained models and use it as a measure of model performance. Experiments show the potential of rank difference in assessing model capabilities within certain model series and measuring the cross-modal alignment of MLLMs.

Reviewers had mixed feelings about this paper. They pointed out as strengths the originality of the proposed metric and its theoretical soundness, as well as their empirical alignment with other metrics and the potential for rank difference to guide future work on model compression. They also pointed out the comprehensive experiments and the clarity of the writing. The main weaknesses are the lack of clarity about the additional information that the proposed metrics bring to the table (particularly for multi-modal models), as well as a a concern that rank difference may not accurately measure the extent of noise reduction, due to the correlation between the dimensionality of the hidden states (and model size) with performance, which makes larger models to have a higher rank difference but less "noise reduction" than smaller models. 

Overall, I am leaning towards acceptance. The proposed "rank difference" is original and it seems useful not only for evaluation but potentially for model development and compression. The authors addressed in detail all the questions asked by the reviewers and in my opinion they alleviated some of their concerns. They reported additional results which seem to address the main weaknesses. I believe the missing details can be incorporated easily in the final version and enhance the paper.
Decision: Accept (poster)