DOCS: Quantifying Weight Similarity for Deeper Insights into Large Language Models

Authors: Zeping Min, Xinshang Wang
Primary area: foundation or frontier models, including LLMs
Keywords: Weight Similarity, Large Language Models, Distribution of Cosine Similarity, Cluster Analysis

Abstract:
We introduce a novel index, the Distribution of Cosine Similarity (DOCS), for quantitatively assessing the similarity between weight matrices in Large Language Models (LLMs),  aiming to facilitate the analysis of their complex architectures. Leveraging DOCS, our analysis uncovers intriguing patterns in the latest open-source LLMs: adjacent layers frequently exhibit high weight similarity and tend to form clusters, suggesting depth-wise functional specialization. Additionally, we prove that DOCS is theoretically effective in quantifying similarity for orthogonal matrices, a crucial aspect given the prevalence of orthogonal initializations in LLMs. This research contributes to a deeper understanding of LLM architecture and behavior, offering tools with potential implications for developing more efficient and interpretable models.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 5, 8

Reviews and discussion:
<thread>
creation date:
1729104813213

thread_type:
official_review

<thread_object>
creator: reviewer_37uy

<content>
summary:
This paper proposes a novel similarity index, DOCS, designed to measure the similarity between two matrices, with a focus on its application to neural network matrices. A key feature of DOCS is its ability to distinguish between orthogonal matrices, which the authors claim is crucial for training neural networks. In addition, DOCS possesses desirable properties such as symmetry and reflexivity.

The computation of DOCS is both simple and elegant, making it easy to understand and computationally efficient. It calculates the maximum cosine similarity between the rows and columns of two matrices, then fits a Gumbel distribution to each of these values. Finally, it averages the two scores to produce the final similarity measure.

The authors present a series of experiments demonstrating the results of applying DOCS to several large language models, comparing the results to other similarity indices such as CCA and SVCCA.

Overall, the paper introduces a promising new similarity index, DOCS, with several interesting properties for comparing neural network weight matrices, particularly in the context of language models. The authors provide an extensive set of experiments with recent open-weight models, comparing DOCS with other indices. However, while the contributions are valid, there are several issues that need to be addressed in the current version of the paper:

Evaluation
------

It is unclear how to evaluate DOCS or whether it captures a truly desirable property. The experiments in the paper show results, but it is not clear why DOCS should be considered superior to other similarity indices. How should we interpret these results? Are the similarities computed by DOCS more meaningful than those computed by other indices? The paper does not provide sufficient justification for why the "elegant" similarity matrices obtained by DOCS are more informative or useful than those obtained by other methods. The authors also suggest that DOCS could help in developing more efficient models (L.124) or provide deeper insights into model behavior (L.134), but there is no empirical evidence to support these claims.
Providing some empirical evidence that using DOCS can lead to better empirical results, or new insights about models are crucial as part of the evaluation of this method.

The authors attempt to justify DOCS by presenting heatmaps of weight matrices, suggesting that these show clearer patterns with less noise. However, this is a highly subjective measure and not convincing. They also compute the Gini coefficient (which should be explained more thoroughly) and claim that its higher value supports DOCS’s superiority. I find this reasoning unconvincing.


Overclaiming
--------
Several claims made in the paper are either overstated or lack sufficient evidence.

- The assertion that neural network matrices require a similarity index that discriminates between orthogonal matrices is one such claim. The authors should (1) empirically demonstrate that the matrices are orthogonal, which should be straightforward, and (2) show that this property is important in practice. Currently, these claims are made without adequate justification.
- L93-98: The authors claim that the heatmaps demonstrate the effectiveness of DOCS. Why do these heatmaps prove effectiveness? Why should we expect the DOCS index to be more informative than other similarity indices?
- L.113: The paper claims that DOCS shows high similarity, but the values are around 0.12, which seems quite low to me.
- L.124: The statement that DOCS "could lead to more efficient models" feels like an overreach.
- L.420-421: The claim that large language models (LLMs) might consist of multiple reasoning phases based on the similarity clusters is speculative and not well-substantiated.
- L.424: The argument that multiple clusters imply multiple training strategies is unconvincing. This could easily be tested by comparing models at different training stages (e.g., base model, fine-tuned model, RLHF model) and examining whether the number of clusters changes.
- L.477: The trends in similarity scores across different layers are minimal (e.g., differences of <0.001 in Llama-3.1-8B), so I question how meaningful these differences really are.


Missing Details
---------
(Note: these are smaller issues, but still important to address in the paper.)

- When computing DOCS, part 3 of the algorithm involves fitting a Gumbel distribution to the obtained values. However, it is unclear how the final single numeric value is derived. Is it the mean, median, or another statistic from the distribution?
- The claim that DOCS can distinguish between orthogonal matrices (L.267) is not explicitly demonstrated in the paper. More details should be provided.
- There is no information on how the baselines for comparison were computed. These details should be included for transparency.


Need for Elaboration
-------------
The paper discusses at length DOCS’s ability to discriminate between orthogonal matrices, but it is unclear why this is a valuable property. Expanding on why this distinction matters in practice would strengthen the paper.

The theoretical proof in Section 3.1 is also not entirely clear. It would be helpful if the authors could expand on the practical implications of this proof.


-------
Rebuttal summary:

After the authors' response I decided to increase my scores due to the satisfying responses that satisfied most of my concerns.
I still believe it's not completely clear how the metric can be used, but I believe some may find it useful, and I do not object of having another similarity metric, especially since it operates on matrices, unlike most previous work that operates on hidden representations.

strengths:
- Introduction of a new similarity index, DOCS, which possesses intriguing properties that make it well-suited for comparing neural network similarities.
- Extensive experimentation on recent language models, with comparisons to established similarity indices.

weaknesses:
- The evaluation of DOCS is insufficient and unconvincing, as discussed in detail in the review.
- Several claims made in the paper are overstated or lack adequate justification (see the summary section for further elaboration).
- Certain important details are missing or insufficiently explained (further discussed in the summary section).

questions:
- It could be valuable to experiment with an initialized model and compute similarity indices, including DOCS. Given the presence of residual connections, the results may exhibit non-random patterns.
- L.115: This sentence is unclear. Are you suggesting that the layer configurations remain uniform during the initialization of the model?
- Suggestion: The font sizes in Figure 3 panels (1-3) and (d-f) are inconsistent. Ensure uniform font sizes across the figures for clarity and visual consistency.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730417405852

thread_type:
official_review

<thread_object>
creator: reviewer_fe2j

<content>
summary:
This paper presents a new metric for measuring the similarity between the weights of transformer models with the goal of gaining insights into LM learning. It is pointed out that existing methods like Canonical Correlation Analysis (CCA) and Centered Kernel Alignment (CKA) do not discriminate between orthogonal matrices well enough, which is a requirement for handling transformer weights. Moreover many existing metrics do not have certain desirable mathematical properties such as transformation and scaling invariance, symmetry, and reflexivity. The proposed metric, DOCS has all these properties.

Empirical justification for the suitability of DOCS for measuring similarity among transformer weights is provided by visualizing the similarity between pairs of MLP layers in a pretrained model.

The main findings using DOCS are that
1. neighboring layers in pretrained LMs are very similar to each other, and there exist clusters of similar layers, both indicating that the transformer architecture can be made more efficient by minimizing such redundancy.
2. the weights of base and instruction tuned models are generally highly similar to each other, and no clear trends exist at which layers weights are most dissimilar (i.e., impacted by instruction tuning).
3. some experts in MoE models are different from others.

strengths:
The approach, mathematical notation, and the visualizations presented in the paper are easy to follow. The comparison of mathematical properties of similarity functions is informative.

weaknesses:
**Motivation**: The main motivation behind studying the similarity between weights instead of representations is unclear. The arguments about discriminating between orthogonal matrices, and the mathematical properties of scaling invariance etc. are built on top of the assumption that studying similarity between the weight space is more insightful than in the representation space, but it is not clear to me why that is the case. After all, is it not the representations that directly affect the outputs and the behavior of a model? Moreover, representational similarity is arguably agnostic to the depth of a network, which is not the case with weight-space similarity. It would be great if the authors could clarify their motivation in the response.

**Empirical justification**: The empirical justification provided for DOCS being superior to other similarity measures is a) that the visualization of the similarity between layers lack a clear structure and appear noisy for CCA and Linear Regression in contrast to DOCS (Section 4.1), and b) the Gini coefficient of DOCS is higher than that of other metrics.
- (a) is not a convincing justification. A metric showing clearer patterns is not necessarily better.This could also mean that DOCS is missing out on useful information other metrics are capturing. It would be more meaningful to compare metrics against a ground truth, e.g., are models that have similar output distributions, like different training checkpoints of the same model, more similar according to one measure than the other?
- (b) requires elaboration. How is the Gini coefficient computed? Why is t relevant? How should one interpret these results?

**Impact of the findings**:
- That there is potential redundancy of knowledge in the weights of neural networks is an interesting finding, but it is not new. This has been empirically studied in many works that are well cited in this paper.
- The experiments comparing base and finetuned models and MoE do not show clear trends, and it is unclear what can be inferred from them.

questions:
- It makes sense that the mathematical properties listed in Section 2 are desirable in a similarity measure of transformer weights. But are these properties *sufficient* for such a metric?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730632336004

thread_type:
official_review

<thread_object>
creator: reviewer_ucwi

<content>
summary:
This paper introduces DOCS, a novel matrix-similarity index that measures weight similarity of LLMs. It provides a more reliable and accurate measure of similarity between LLM weight matrices.

strengths:
1.The approach is novel. It is a discriminative index which captures meaningful differences between weight matrices than other methods.

2.This paper encompasses detailed theoretical derivation on discriminative property and comprehensive experiments, which demonstrate discriminative nature of DOCS and provides some insights into the structure of LLMs.

weaknesses:
1.Some of findings are not straightforward in this paper. For example, the authors associate multiple reasoning phases with clusters and attribute different numbers of clusters across LLMs to different training strategies. However, they do not clarify the rationale behind these conclusions.

2.Although the authors claim that DOCS satisfies all desired mathematical properties, they do not provide corresponding proofs in the paper.

questions:
1.Could you provide a detailed explanation on the relation between clusters of similar phases and multiple reasoning phases?

2.Proofs of DOCS satisfying all desired mathematical properties could be added to improve the rigorousness of the claim.

3.I am curious about other similarity indices could achieve similar findings on LLMs. I believe this is important to further demonstrate the superiority of DOCS over other methods.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730707569738

thread_type:
official_review

<thread_object>
creator: reviewer_rk55

<content>
summary:
The paper introduces the Distribution of Cosine Similarity (DOCS), a novel index designed to quantitatively assess the similarity between weight matrices in large language models (LLMs). This tool aims to enhance the analysis of LLM architectures by revealing significant patterns within them. Using DOCS, the authors identify that adjacent layers in contemporary open-source LLMs often show high weight similarity and tend to cluster, indicating a trend toward depth-wise functional specialization. Furthermore, the paper demonstrates that DOCS effectively quantifies similarity for orthogonal matrices, an important consideration given the common use of orthogonal initializations in LLMs.

strengths:
1.	This paper is overall well-written.

2.	This paper proposes to better calculate the parameter similarity between different layers by computing the cosine similarity between corresponding vectors and analyzing their distribution.

3.	Using the technique they propose, the authors observed that: 1) neighboring transformer layers exhibit similar weights, 2) clusters of similar transformer layers exist, 3) the difference of parameters among different models.

weaknesses:
1.	Some experimental designs could be more detailed and justified, such as the use of the gumbel distribution.

2.	No direct application could be derived from this analysis paper, which could be a weakness of the paper. The authors could add a discussion section to further point out the practical value of the findings in this paper.

questions:
1.	Why did you choose to use Gumbel distribution to summarize each distribution to a parameter? Is there any specific reason or proof for that?

2.	Could you suggest some other potential applications besides what people have already done (e.g., leveraging layer clusters to reduce computational requirements) that could benefit from the findings in this paper?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730710975279

thread_type:
official_review

<thread_object>
creator: reviewer_6myg

<content>
summary:
The paper introduces Distribution of Cosine Similarity (DOCS), a new method for measuring similarity between LLM weights.  Unlike previous methods that focused on analyzing representations, DOCS directly examines weight matrices, providing a more precise tool for understanding the internal structure of these complex models. DOCS reveales two important insights, (i) adjacent layers in LLMs often share similar weights and (ii) tend to form functional clusters suggesting depth-wise functional specialization. The findings show that the conventional use of uniform layer configurations in large language models may not be optimal, suggesting hints for more efficient architectures design.

strengths:
- The method provides a quantitatively similarity score for LLM weights (not representation) which may be helpful for model merging and pruning research.
- The method is proposed with a good level of interpretability and visualization with impressive analysis.

weaknesses:
While the paper showed an impressive set of analysis (with page limitations), I would very much like to see an analysis on what makes DOCS work. Especially in Algorithm 1, which step is the most crucial one. A small set of experiments may help.

questions:
- Please follow weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1732366544465

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Reference

comment:
[1] Morcos, A., Raghu, M., & Bengio, S. (2018). Insights on representational similarity in neural networks with canonical correlation. Advances in neural information processing systems, 31.  

[2] Raghu, M., Gilmer, J., Yosinski, J., & Sohl-Dickstein, J. (2017). Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30.  

[3] Kornblith, S., Norouzi, M., Lee, H., & Hinton, G. (2019, May). Similarity of neural network representations revisited. In International conference on machine learning (pp. 3519-3529). PMLR.  

[4] Tian, Y., Wang, Y., Zhang, Z., Chen, B., & Du, S. (2023). Joma: Demystifying multilayer transformers via joint dynamics of mlp and attention. arXiv preprint arXiv:2310.00535.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732366681653

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Reply (3): Advantages of DOCS Compared to Other Similarity Indices

comment:
In addition to the discussions on the advantages of DOCS compared to other similarity indices presented in Section 4.1 of the paper, we have included *additional experimental results* in Appendix E.3 of the updated manuscript.

In Section 4.1, Figure 2 showcases the evaluation outcomes of eight different similarity measures on the $MLP\text{-}Up$ layers from the Meta-Llama-3.1-8B-Instruct model. The resulting heatmaps highlight that metrics like Linear Regression, Canonical Correlation Analysis (CCA), and CCA (Nuclear) fail to display discernible structural patterns. On the other hand, methods such as Singular Vector CCA (SVCCA), SVCCA (Nuclear), and Linear Centered Kernel Alignment (Linear CKA) exhibit faint block-like or striped formations in the off-diagonal regions, which might be attributed to either noise or inherent limitations in the measures themselves. These anomalies could arise due to the reduced sensitivity of these indices in differentiating between orthogonal matrices, as elaborated in Section 2.

In Appendix E.3 of the updated manuscript, to better illustrate the superiority of the DOCS method compared to other approaches, we conducted *further* experiments using three models:

- (A) `meta-llama/Meta-Llama-3.1-8B` (the base model)
- (B) `meta-llama/Meta-Llama-3.1-8B-Instruct` (an instruction-tuned version of the base model)
- (C) A version of `meta-llama/Meta-Llama-3.1-8B` with randomly initialized weights

Intuitively, the weight matrices of corresponding layers in models (A) and (B) should exhibit significantly higher similarity compared to those between models (A) and (C), since model (C) contains random weights and lacks the learned structure present in models (A) and (B).

To quantify this, we define the *similarity ratio* as the ratio of the similarity scores between models (A) and (B) to those between models (A) and (C). Specifically, for each similarity index, we compute the similarity of the corresponding $MLP\text{-}Up$ and $MLP\text{-}Down$ weight matrices between models (A) and (B), and between models (A) and (C). A higher ratio indicates that the similarity index is better at distinguishing between meaningful relationships in model weights (as seen in models (A) and (B)) and unrelated weights (as seen in models (A) and (C)). Thus, a higher ratio reflects the ability of the index to highlight structural patterns specific to related models while minimizing noise from uncorrelated data.

As shown in Figure 10, the experimental results reveal that the similarity ratio computed using the DOCS method is much higher—approximately 10 times greater—than those obtained with the other indices. This demonstrates the effectiveness of DOCS in capturing meaningful similarity patterns in model weights.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732366825434

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Reply (2) : Motivation of DOCS

comment:
### Non-Discriminative for Orthogonal Matrices
In Section 1, we emphasized that many existing similarity indices, such as Canonical Correlation Analysis (CCA) [1], Singular Vector Canonical Correlation Analysis (SVCCA) [2], and linear Centered Kernel Alignment (linear CKA) [3], are *non-discriminative for orthogonal matrices*. An *orthogonal matrix* $Q$ is defined by the property $Q^\top Q = Q Q^\top = I$, where $I$ is the identity matrix. This non-discriminative nature means that these indices can yield the same score when assessing the similarity between *any two* orthogonal matrices, regardless of their actual differences. These conclusions are uniformly shown in Table 2, with proofs provided in Appendix B. This issue is *particularly relevant* in the context of LLMs, where orthogonal matrices commonly occur throughout the training process [4].

In fact, in Figure 2, we observe that when CKA is directly used to measure weight similarity, a large number of values concentrate in the high range of 0.78–0.80. This is likely due to the non-discriminative property for orthogonal matrices.

In Appendix E.2 of the updated manuscript, we provide a *further result* indicating a high degree of orthogonality among the weight matrices in LLMs. 
To quantify this orthogonality, we design an index named the Off-diagonal Average Cosine Similarity metric. This metric provides a systematic way to measure the extent of orthogonality between columns of a given weight matrix. The formulation of the metric is as follows:

$$
\text{Off-Diagonal Average Cosine Similarity} (\mathbf{X}) = \frac{1}{n(n-1)} \sum_{i=1}^{n} \sum_{\substack{j=1 \\ j \neq i}}^{n} \left| \frac{\mathbf{x}_i \cdot \mathbf{x}_j}{\|\mathbf{x}_i\| \, \|\mathbf{x}_j\|} \right|,
$$

where $\mathbf{X}$ is an $n$-column matrix, and $\mathbf{x}_i$ represents the $i$-th column of $\mathbf{X}$. From this formulation, it can be observed that the larger the Off-Diagonal Average Cosine Similarity value, the smaller the orthogonality of the matrix.

We constructed a family of approximately orthogonal matrices, denoted as $\mathcal{M}\_\theta$, which are defined as follows:

$$
\mathcal{M}\_\theta = I\_{n} + \theta  v  \mathbf{1}^\top
$$

where:
- $I_{n}$ represents the identity matrix of size $n \times n$.
- $\theta \in \mathbb{R}$ is a scalar that controls the magnitude of orthogonality; smaller values of $\theta$ correspond to matrices that are closer to orthogonal.
- $v \in \mathbb{R}^{n}$ is a random vector sampled from a normal distribution, specifically $v \sim \mathcal{N}(0, 1)$.
- $\mathbf{1} \in \mathbb{R}^{n}$ is a vector of ones.

We utilized the Q matrices and O matrices from each layer of the Meta-Llama-3.1-8B-Instruct model. Additionally, we generated four sets of approximately orthogonal matrices by sampling $\mathcal{M}\_\theta$ with $\theta$ values of 0.005, 0.003, 0.002, and 0.001, ensuring that their shapes matched those of the Q and O matrices. We then computed the Off-Diagonal Average Cosine Similarity for these matrices.

The experimental results, presented in Figure 9, reveal that the majority of the Q and O matrices from each layer of the Meta-Llama-3.1-8B-Instruct model exhibit stronger orthogonality compared to $\mathcal{M}\_{0.003}$. According to the definition of $\mathcal{M}\_\theta$, $\mathcal{M}\_{0.003}$ only has a very small perturbation added to the identity matrix. This observation suggests that the Q and O matrices in the Meta-Llama-3.1-8B-Instruct model are highly orthogonal.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732369501172

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Reply (1): Motivation of DOCS

comment:
As introduced in Section 1, while prior research has explored many methods for characterizing the similarity of neural networks --- such as Canonical Correlation Analysis (CCA) [1], Singular Vector Canonical Correlation Analysis (SVCCA) [2], and linear Centered Kernel Alignment (linear CKA) [3] --- applying these methods to weight matrices presents challenges due to two key factors: **Focus on Representation, Not Weights** and **Non-Discriminative for Orthogonal Matrices**. In addition to the discussions in Section 1 of the manuscript, we have added additional experimental results in Appendix E.1 and E.2 of the updated manuscript.

### Focus on Representation, Not Weights
In Section 1, we emphasized that representation and weight are two aspects of the model. Similar representations across layers do not necessarily imply similar weight matrices. This discrepancy arises from the use of *residual connections* in transformer architectures. This is evidenced by Figures 1a and 1b, which show that the input and output of the feedforward network have similar patterns of representation similarity.

In Appendix E.1 of the updated manuscript, we provide a *further result* showing that weight similarity helps discover intriguing patterns in open-source LLMs that would otherwise be overlooked. We conducted experiments on the `01-ai/Yi-1.5-9B-Chat` model. On one hand, we used the Linear CKA method to calculate the similarity of the $MLP\text{-}Up$ layer outputs in each block of the model. On the other hand, we used the DOCS method to calculate the similarity of the $MLP\text{-}Up$ parameters in each block. The experimental results are shown in Figure 8.

The results in Figure 8 underscore the strength of weight similarity compared to representational similarity methods in revealing structural patterns within the model. Specifically, the weight similarity shown in Subfigure 8a highlights a notable correspondence between layers 9–24 and layers 25–40. This suggests that the model may have employed a training strategy that duplicates a section of layers (9–24) to increase the model's size while saving on training costs. 

Our analysis provides tools that can help reveal such structures in open-source LLMs. In contrast, Subfigures 8b, 8c, and 8d present the representational similarity of the $MLP\text{-}Up$ outputs. Despite using multiple input sentences for the analysis, these results primarily provide a limited understanding of weight structure. They exhibit relatively homogeneous patterns, lacking the fine-grained layer correspondence seen with weight similarity analysis. Therefore, in this example, weight similarity helps uncover patterns that would otherwise remain hidden, deepening our understanding of the model.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces DOCS (Distribution of Cosine Similarity), a novel similarity index for measuring weight similarity in LLMs. It directly analyzes weight matrices for discovering of structural patterns in LLMs,  such as layer redundancy, functional specialization, and weight preservation across fine-tuning. DOCS demonstrates strong performance over established similarity indices (e.g., CCA, SVCCA) and satisfies desirable mathematical properties like scaling invariance, symmetry, and orthogonal matrix discrimination.

Strengths:
- The proposed approach is a novel and computationally efficient, in addition to provides insights into LLMs' internal structures.
- It is well-grounded in theory, addressing key issues like orthogonal matrix discrimination.

Weakness: 
- Reviewers raised concerns about the evaluation of DOCS, particularly its superiority over other indices without clear ground truth. While the authors introduced metrics like the Gini coefficient to quantify performance, its interpretability remains subjective.
- In addition, after rebuttal, some claims still remain speculative and lacked empirical support, such as the relationship between layer clusters and reasoning phases.