Layers at Similar Depths Generate Similar Activations Across LLM Architectures
Authors: Christopher Wolfram
Aaron Schein
Venue: COLM 2025 (2025)
Keywords: universality
representational similarity
TL;DR: Independently-trained large language models generate activations that are similar between models, but not within models.

Abstract: How do the latent spaces used by independently-trained LLMs relate to one another? We study the nearest neighbor relationships induced by activations at different layers of 24 open-weight LLMs, and find that they 1) tend to vary from layer to layer within a model, and 2) are approximately shared between corresponding layers of different models. Claim 2 shows that these nearest neighbor relationships are not arbitrary, as they are shared across models, but Claim 1 shows that they are not "obvious" either, as there is no single set of nearest neighbor relationships that is universally shared. Together, these suggest that LLMs generate a progression of activation geometries from layer to layer, but that this entire progression is largely shared between models, stretched and squeezed to fit into different architectures.

Published as a conference paper at COLM 2025
Layers at Similar Depths Generate Similar Activations Across LLM Architectures
Christopher Wolfram
Aaron Schein Department of Statistics University of Chicago Chicago, IL 60637, USA schein@uchicago.c edu
Department of Computer Science
University of Chicago
Chicago, IL 60637, USA
fraissollaammeuhccaag edu
Abstract
How do the latent spaces used by independently-trainn LLMs relate to one another? We study the nearest neighbor relationships induced by activa- tions at different layers of 24 open-weight LLMs, and find that they 1) tend vary from layer to layer within a model, and 2) are approximately shared between corresponding layers of different models. Claim 2 shows that these nearest neighbor relationships are not arbitrary, as they are shared across models, but Claim shows that they not "obvious" either, there is no single set nearest neighbor relationships that universally shared. Together, these suggest that LLMs generate progression of activation ge- ometries from layer to layer, but that this entire progression largely shared between models, stretched and squeezed to fit into different architectures.
1 Introduction
Instead of trying to understand the internal mechanisms of specific language models, we study structural similarities in the activations generated by models different architectures. In particular, how do the latent spaces used by different models relate to one another, and do they have any universal properties?
Consider the following experiment. Let D be set of prompts. We pass each prompt in D to language model M and collect the activations generated at particular layer? L. For any prompt E D, we can then ask: which are the k = 10 other prompts that yielded activations most similar to this one? This gives set of the k nearest neighbors of among the other elements of DD Repeating this for all prompts in D gives a collection of nearest neighbor relationships that encode which prompts are nearest to which others according to layer l in the model M.
By comparing nearest neighbor relationships across wide range of models, we make two findings that seem unremarkable on their own, but together are quite surprising:
Claim 1. Activations collected at different depths within the same model tend to have different nearest neighbor relationships.
Claim 2. Activations collected at corresponding depths of different models tend to have similar nearest neighbor relationships.
Claim shows that nearest neighbor relationships are not arbitrary, as they are shared across independently-trriied models. Claim however, shows that they not "obvious" either, as nearest neighbor relationships vary with depth. Put another way, Claim shows that near- est neighbor relationships are sensitive to differences between layers within single model, while Claim 2 shows that activations at corresponding depths of different models are similar.
"ooee available https://gg www.tttt..cccoeeeeeooooooootoo swol wwwlfram/uniren
We use the term "layer" to refer to an entire decoder module. See Section 3.2 for details on how extract activations from transformer models.

---

Published as a conference paper at COLM 2025
Llama-2 1-8B
Gemma-2-90
Layer
different
similar
Layer 20
different
Layer
Similar
Layer 40
Figure 1: Layers at different depths generate activations with different nearest neighbor relationships (Claim 1), but layers at corresponding depths of different models generate activations with similar nearest neighbor relationships (Claim 2). The particular example used in Sections 1 and is shown here.
Such nearest neighbor relationships reflect the local geometry of the embeddings generated at particular layer, and are closely related to operations used in transformer models. To- gether, Claims and 2 suggest that transformer-based language models use a progression of distinct local geometries from layer to layer, and that this entire progression of local geometries is (largely) shared across models regardless of architecture. In other words, -raependentlyytrrineddd language models tend to go through similar sequence of intermedi- ate representations when given the same prompt.
To give concrete example, let ł be this text from Open WebText (Gokaslan et al., 2019), dataset mostly containing internet news articles:
Training For Ice and Mixed Climbing Series brought you Furnace Industries continues...
According to layer 10 of 3. 1-88, these are the 3 nearest neighbors to that text among sample of 2047 other texts from Web
There have been lot crazy reactions from the democrats now that Donald Trum... Prosecutor Christina Weilander filed the charges withh Södertörn district court o... 7 questions on the Great British Bake Off Great British Bake Off Quiz Then there...
At first glance, these nearest neighbors look arbitrary, and it is not clear what relationship they have with the original input or with one another. That makes it all the more surprising that layer 20 of Gemma-2-9B gives exactly the same nearest neighbors (in line with Claim 2). Moreover, layer 30 of L1ama-3. 1-8B gives another completely different set of nearest neigh- bors (in line with Claim 1), but many of them the same the nearest neighbors given by layer 40 of Gemma 2amma-2-990 2-98 (Figure 1).
Llama-3. 1-8B and Gemma-2-9h are architecturally different models independently trained by different companies, so it is surprising that we not only find similarities in their latent representations, but that these similarities only appear corresponding depths. (See Section 2 for more investigation of this specific example.)
We study this phenomenon systematically by generating the 11 x n2 affinity matrix, which stores the similarity of nearest neighbor relationships implied by each pair of layers (i,j) with layer i E [n,] from model M1 and layer j E [nz] from model M2. The resulting affinity matrices exhibit diagonal structure, with higher similarity among layers at corresponding depths fimplying Claim 2), and lower similarity among pairs of layers from different depths (implying Claim 1).
This paper studies the emergence of diagonal structure in affinity matrices across 24 open- weight language models, with sizes spanning from 1B to 70B parameters. We find that diago- nal structure is present when comparing a wide range of models (Section 4), and that this di- agonal structure is statistically significant for every pair of models consider (Section 4.1).
2

---

Published as a conference paper at COLM 2025
1.1 Related work
There is an extensive literature on representational similarity in neural networks (Kornblith et al., 2019; Klabunde et al., 2024), and with rising interest in universality LLMs (Olah et al,, 2020; Rai et al., 2025) more recent work has begun to apply representational similarity methods to large transformer models (Huh et al., 2024; Klabunde et al., 2023). However, none have systematically studied affinity matrices comparing layers of architecturally differ- ent LLMs or the resulting diagonal structure. Most either 1) compared layers within single model (Vulié et al., 2020), 2) compared layers between two copies of a model trained with different random seeds (Nguyen et al., 2022; Gurnee et al., 2024), 3) focused on CNNs or very small language models (Raghu et al., 2017), 4) used a representational similarity measure that did not reveal diagonal structure (Lan et al., 2024; Baek et al., 2024; Brown et al., 2023), or some combination thereof (Korbblth et al., 2019; Nguyen et al., 2021). Of particular note is the work of Wang et al. (2025), who used sparse autoencoders to study representational sim- ilarity of Pythia- 160M and Mamba-1 -30M and noted that layers at similar depths tended to be more similar to one another. Also notable is the work of Wu et al. (2020), who measured the representational similarity of layers from a range of early LLMs. They found that early layers tended to be more similar between models than later layers, and observed diagonal structure in some model comparisons (notably in a comparison of BERT-base and BERT-large).
2 Motivating experiment
Let 'D be a random sample of 2048 texts from Webneee (Gokaslan et al., 2019), a dataset of internet text dominated by news articles. As before, let t be the following text from OpenWebText (which has been truncated for space):
Training For Ice and Mixed Climbing Series brought to you by Furnace Industries continues...
We feed all of the texts in D to Llama-3. 1-8B and collect the activations generated at layer 10. We then compute the 10 elements of DD that generated activations most similar (by cosine distance) to those generated when f was inputted. In other words, we compute the 10 nearest neighbors of according to layer 10 of Llama-3. 1-8B.
We then repeat this process for layer 20 of Gemma-2 2-98, and make a Venn diagram showing which elements of 'D were considered nearest neighbors according to the two layers (Figure 2).
Llama-3. 3. 3.
1BB (layer 10)
New Jersey man is in critical condition following a struggle with Fayettevi...
Image copyright Thinkstock UK teenagers might have reputation for binge dri...
Viz Media Celebrates 'My Neighbor Totoro' 25th Anniversary With Novel, Pictur...
questions on the Great British Bake Off Great British Bake Off Quiz Then th...
DENVER Nearly 30 Air Force Academy cadets required medical care, with six
In move sure to inflame the left, Republican congressional candidate in L...
Prosecutor Christina Weilander filed the charges with Södertörn district cour
There have been lot of crazy reactions from the democrats now that Donald T... The richest person in Venezuela isn't billionaire industrialist, but the da... push mode: 'thumbnails-e' I'm trying to find out more about the Census night disaster. Remember the Cen...
window._ taboola = handow__tabool. II taboola. tushooll...ushh(
On Monday, November 25, UrtheCast will present live the launch of its two cam...
Wrinkle In Time" Director Ava DuVernay Says DC's Big Barda Is Her... Wri...
Gemma-2 2--B (layer 20)
Figure 2: Venn diagram showing the 10 nearest neighbors to the text among the 2048 elements of 'D according to layer 10 of Llama-3. 1-8B (in the blue box) and layer 20 of Gemma-2-98 (in the green box). Entries that both layers agreed were nearest neighbors are marked with O while others marked with X
3

---

Published as a conference paper at COLM 2025
In this example, these two layers from completely different models agreed on 7 out of the 10 nearest neighbors. This is all the more surprising because, at first glance, these texts seem to have little to do with the text f to which they are all nearest neighbors, much with one another. They appear arbitrary, but are evidently not arbitrary because they are independently identified by different layers of different models. On closer inspection, these texts do share a common feature in that many end with a variant of "Follow on Twitter." " It could be that the activations generated at layer 10 of Llama-3. 1-8B and layer 20 of Gemma-2-88 are both dominated by mentions of Twitter.
do they share
The previous example could give the impression that nearest neighbor relations in activa- tions are dominated by superficial but obvious features, and that agreement between models trivial consequence of both models identifying the same obvious features. However, this idea is dispelled by looking at different layers.
We now compute the nearest neighbors according to layer 30 of Llama-3. 1-8B and layer 40 of Gemma-2-98 (Figure 3).
-lama-3.. 1 1-BB (layer 30)
Long Forgotten London Old House on Tower Hill There is the London we know and... you trust? Subscribe to free newsletters. Texas alre... Sure, I love tea. And admire those who go caffeine-free. But I have become Wrightwood. Cal. 21 October, 1949 Dear Mr. Orwell, It very kind of you to. About DONALD TRUMP and HILLARY CLINTON, Skewered by Their Own Words with Funn... In sign of the Queen's determination to master new technology, Buckingham P... In case you were living under a rock, today Manchester City announced the sig...
Looking for
Low-Riders at the Museum El Paso Museum of Art Saturday, May 10, 2014 from 10. his party...
Mitt Romney's hurdle in winning the Love/respect/admlratioooo/
We all love selfie, regardless of the occasion. But where are the most popu...
On Monday, November 25, UrtheCast will present live the launch of its two cam...
Review: "Sight Unseen" @ NJT The art world is full of beautifully complex, cr...
Since launching The Athletic NHL and The Athletic Toronto more than a year ag...
= window._ tabbolla
I| []; _tbboola, pusho( mode: 'thumbnails-c'..
Gemma-2-99 (layer 40)
Figure 3: Venn diagram showing the 10 nearest neighbors to the text f according to layer 30 of Llama-3. 1-8 (in the blue box) and layer 40 Gemma-2 2-9B (in the green box). The two layers agree on out of the top 10 nearest neighbors.
Notably, the set of nearest neighbors according to layer 30 of Llama-3. 1-8B is completely disjoint from the set of nearest neighbors identified by layer 10 of the same model. If there was some obvious feature of that was identified by the earlier layer (such as mentions of Twitter), it is not being identified by this later layer of the same model. In addition, this new list of nearest neighbors not arbitrary either, as it shares most of its elements with the nearest neighbors identified by later layers of Gemma-2-99
That the nearest neighbors according to layer 10 of Llama-3. 1-8B and layer 20 of Gemma-2-9B are so similar (as well as layer 30 of Llama-3, 3. 1-8B and layer 40 of Gemma-2-998 is in line with Claim (that activations from corresponding layers of different models induce similar nearest neighbor relationships). In contrast, that the nearest neighbors according to layer 10 and layer 30 of 3.ama-3. 1-8B are so different is in line with Claim (Figure 1).
For layer 10 of Llama-3. 1-BB and layer 20 of Gemma-2-9B, we identified superficial property of the inputs that could be dominating the geometry of the latent spaces (i.e., that they mention Twitter). We found no such superficial feature relating nearest neighbors according to layer 30 of Llama-3. 1-8B and layer 40 of Gemma-2 2-9B, but it is possible that there is one. However, neither of these findings would explain away the main observation: Why should independently-trriin trained models identify the same (perhaps superficial) properties to focus on, especially if different layers in the same models focus on different properties?
4

---

Published as a conference paper at COLM 2025
We do not purport to know why the activations at these layers of these models induced these particular nearest neighbors, only that some these sets are remarkably similar and others are disjoint. These patterns of similarity and dissimilarity are the object of our study,
3 Representational similarity
In order to study this systematically, first define some notation for embeddings, nearest neighbors, and representational similarity.
Definition 1 (Embedding functions) An embedding function is map from input space X to R for some embedding dimension d.
The embedding functions we use extract activations from transformer models.
X Definition 2 (Representational similarity measures) Let f - Rd and g ::X embedding functions. A representational similarity measure a function
X - Rd be
s: (X' - R4)) (X' R +
that maps a pair of embedding functions to IR.
Note that do not restrict ourselves to cases where d, and d2 equal, models of different architectures often have different embedding dimensions.
We focus primarily measuring similarity between nearest neighbor relationships among activations, but our framework could be applied to any measure of representational sim- ilarity satisfying Definition 2, such as CKA (Kornblith et al., 2019) or one of many others d (Klabunde et al., 2024).
The central problem of measuring representational similarity that neural networks can generate activations that differ in trivial ways. For example, one neural network might generate activations identical to another, except that the axes are permuted, or the signs are flipped, etc. The challenge of measuring representational similarity is in choosing what properties of representations are arbitrary (e.g,, the sign or order of dimensions), and what properties are fundamental (e.g,, the nearest neighbor relationships).
In this work, we study patterns of similarity: that layers at corresponding depths in different models tend to generate activations that share properties that layers at non-corresponding depths do not. As such, the particular choice of representational similarity measure is not our focus. We do not claim that the nearest neighbor-based measure we use is the "correct" way to measure representational similarity (whatever that would mean), but that shows that there exist properties of activations that shared according to these patterns.
3.1 Nearest neighbor relationships
We measure the extent to which nearest neighbor sets agree using the mutual k-nearest neighbors (mutual k-NN) representational similarity measure described by Huh et. (2024).
Definition 3 (k-nearest neighbors) Let u (P) be the k-nearest neighbor function
u (x,
argmin U-ugD)(x) [ y'll
_a(f(t).((())
(1)
for distance function d. Unless otherwisse stated, the cosine distance.
Intuitively, (P (x, gives the elements of dataset D that are closest to x, after they have been mapped to vectors by f.

---

Published as a conference paper at COLM 2025
Definition 4 (Mutual k-nearest neighbors) n similarity measure M is defined by
The mutual k-nearest neighbors representational
A
(x k||
Σ XCD
for an input dataset DD
Mutual k-NN measures the similarity of the nearest neighbor relationships induced by to those induced by g. For a set inputs, it gives the expected fraction the k-nearest neighbors that both embedding functions agree on. (Note that M takes two embedding functions and returns a scalar, and therefore satisfies Definition 2.)
In general, when we refer to layers being "similar" to one another, we mean that they achieve a high mutual k-NN similarity score.
3.2 Activations in transformer models
Transformer models generate many activations during their operation, and can be probed at several different points. We restrict ourselves to the activations generated at the end of each decoder module (which are sometimes referred to as "layers").
Thus, we can think of transformer model M with n layers as providing the embedding functions fM f....., pln) :T + R2 which map from texts T to vectors. Each embedding function gives the activations the end of layer in M on the last token of the input.
3.3 Affinity matrices
Definition An affinity matrix for models M1 and M2 a matrix such that
Aij s sfM, MM)
(2)
given some representational similarity measure s. Unless otherwise noted, we use (D) 10 for s, which measures similarity with mutual k-NN with k 10. The dataset D must also be specified.
If Mj and M2 have лı and n2 layers respectively, the resulting matrix has dimensions N1 n2, with each entry Aij storing the representational similarity of the layer of M1 and layer of Mz. Because we use mutual k-NN as our representational similarity measure, this means that AI, stores the similarity of the nearest neighbor relationships generated by those two layers.
It is worth noting that mutual k-NN is always measured with respect to a dataset of inputs D. This extends to affinity matrices generated with mutual k-NN as well.
We are not the first to generate such matrices (Korbbith et al., 2019; Nguyen et al., 2021; 2022; Gurnee et al., 2024; Raghu et al., 2017; Wang et al., 2025; Abnar et al., 2019; Brown et al., 2023; Wu et al., 2020; Vulié et al., 2020), though they have most often been used to compare layers either from the same model or from multiple instances of the same architecture trained with different random seeds.
Affinity matrices reveal diagonal structure
In Section 2, we saw a single example where activations collected at different depths within the same model had different nearest neighbor relationships (Claim 1), and activations collected at corresponding depths of different models had similar nearest neighbor relation- ships (Claim 2). Do these results generalize to other inputs, layers, models, etc.?
The mutual K-NN representational similarity measure captures the extent to which two layers generate activations with similar nearest neighbor relationships. By generating

---

Published as a conference paper at COLM 2025
33000
6iilarlyy
Kile (194
Telov 2422
299]
sim mmmtrty
Utme 32-07)
2000
(c)
(b)
Figure 4: Layers at corresponding depths tend to generate similar activations on internet text, as represented by diagonal structure affinity matrices. 4a The affinity matrix of Gemma-2-9B and Llama-3. 1-8B on a sample of texts from WpenWeb'Text. Each cell corresponds to a pair of layers, with one from Gemma-2-9 and the other from Llama-3. 1-8B. The value that cell corresponds to the similarity of nearest neighbor relations induced by the activations at those layers. 4b The same as 4a, but for Llama-3. 1-70B and Mistral-7B 4c An array of affinity matrices comparing activations on web text across wide range of models, all of which show diagonal structure. (See Figure 17 for an extended version and Figure 25 for version that resizes all affinity matrices to squares for comparability.)
affinity matrices that show the mutual k-NN similarity of every pair layers across a range of models, we can see whether and how the observations of Section generalize.
We consider 24 open-weight models developed by Google (Gemma Team et al., 2024), Meta (Grattafiori et al., 2024), Mistral (Jiang al., 2023), Microsoft (Abdin et al., 2024), and TII (Almazrouei et al., 2023). (See Appendix F for the full list of models.) For every pair of models, we generate an affinity matrix (Definition 5) which shows how similar the nearest neighbor relationships are between every pair of layers in those models. We measure similarity using mutual k-NN (Definition with k 10, and using a dataset of 2048 texts randomly sampled from WpenWe (Gokaslan et al., 2019).
We visualize selection of the resulting affinity matrices Figure The most striking feature of these affinity matrices is their diagonal structure, which is present across wide range of model comparisons (Figure 4c). This diagonal structure implies that layers at proportionally similar depths generate more similar activations one another than other pairs of layers, just as seen in Section 2. This indicates that models generate distinct geometries of activations at different depths, but that activations at given depth are similar across diverse models. Put another way, language models generate a progression of distinct activation geometries (Claim 1), and this entire progression is largely shared between models (Claim 2).
Models generate similar progressions of embeddings, but stretched squeezed The most similar layers between models cannot align one-to-one because the models being compared have different numbers of layers. In the case of Llama-3, 1 1-8B and Gemma-2-98
7

---

Published as a conference paper at COLM 2025
1.0
1.0
model2
0.0
0.8
0.8
0.
0.4
0.4
04
0.2
0.0
0.2
0.4
0.5
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
depth model
model
(a)
(b)
(c)
Figure 5: 5a Maximally similar layers appear at proportional depths. Each line represents pair of models M1 and M2, and for each layer in M1 it shows the depth of the most similar layer in M2- 5b Layers at all depths have similar counterparts in other models. Each line represents a pair of models Mj and M2, and shows how similar the most similar layer of M2 is each layer of Mj. 5c The mean affinity matrix when all affinity matrices have been resized to fit into a square.
(Figure 4a), the most similar layers not at the same absolute depths (e.g., layer 10 of each model), but instead at similar depths relative to the total number of layers in the model (e-g., 30% of the way through). This pattern persists across a range of model pairs, and the correspondence between maximally similar layers does not deviate far from proportionality in any of the model pairs we consider (Figures 5a and 5c).
Layer compression When comparing models of vastly different sizes, there is yet more flexibility in the correspondence between layers. For example, consider the affinity matrix comparing Llama-3. 1-70B and Mistral-78 which have 80 and 32 layers respectively (Fig- ure 4b). The early layers appear to correspond almost one-to-one, but the middle layers of Llama-3. 1-70 appear highly compressed in Mistral-7B. The later layers are all relatively similar to one another, and so show less diagonal structure. This general pattern is typical, though often stretched and squeezed to fit into models with different numbers of layers (Figures 4c and 17).
Existing work has argued that layers at different depths in transformer language models perform different functions, for example, with early layers dedicated to parsing the input and resolving basic grammar, and later layers dedicated to finding the next token of the response etc. (Lad et al., 2024; Tenney et al., 2019; Durrani et al., 2020; Kaplan et al., 2025; Cheng et al., 2025). It is possible that these functions are reflected in the correspondence between layers of large and small models. Language models need to be able to parse their input, regardless of size, and so the first 15 to 20 layers generally correspond one-to-one with little compression. Middle layers, however, could be responsible for more abstract and optional work that can be highly compressed smaller models, as reflected the affinity matrices.
There is similarity at every depth Intuitively, one might expect there to be higher similarity between models when comparing the earliest and latest layers, as these are the most closely tied to model behavior and the training process. Indeed, in their experiments, Wu et al. (2020) found that earlier layers were more similar across models than later layers. However, we find that the absolute similarity between layers does not vary much with depth, with middle layers being just as similar to one another as early and late layers (Figure 5b). Even in architecturally identical models trained with identical data (but with different random initialization), there is no intrinsic guarantee that their latent activations be similar, but here we find that similar latent activations are generated at every depth.
Sensitivity to the experimental setup The experiments above all look at activations generated on inputs sampled from OpenWebText. In Appendix G, we generate affinity matrices for range of input datasets, including IMDB movie reviews (Maas et al., 2011), text
8

---

Published as a conference paper at COLM 2025
30
off-diegona mean mutual K-NN
0.4
loyer (88)
20
0.2
0.1
10
20
30
40
0.3
0.4
0.5
0.6
0.7
0.8
löyor of Gemma 2 (98)
on-diagonal mean mutual K-NN
(a)
(b)
Figure 6: 6a An affinity matrix (from Figure 4a) overlaid with the generalized diagonal region (red). 6b Layers pairs that lie on the generalized diagonal have higher mean similarity than those that do not. Each point represents a pair of models, with its position determined by the mean similarity of its layer pairs that lie on and off the generalized diagonal.
from books in English and German (Tiedemann, 2012), and various LLM evaluations (Zhou et al., 2023; Hendryeks et al., 2021). The diagonal structure observed with Open Web is generally present for other distributions of web text (e.g,, IMDB movie reviews). Behavior on other input distributions is investigated Appendix A.
We also perform experiments with other values of and find that the choice of k shifts the measured mutual k-NN similarity, but does not affect diagonal structure, even in extreme cases like k = (Appendix D).
4.1 Statistical test of diagonal structure
For each model pair, we perform test of diagonal structure based on comparing the mean similarity of layer pairs near the diagonal to the mean similarity of those away from the diagonal. We first have define what we mean by "near" the diagonal.
Generalized diagonal For a square matrix A, the main diagonal contains those elements Aij where = j. However, many of the affinity matrices we generate are not square. For rectangular matrix A with dimensions
m, define the generalized dingonal to be those elements A satisfying j I i j+n m (where n 2 m without loss generality). the case where A is square (i.e,, n m), the generalized diagonal is exactly the usual diagonal. An example of the generalized diagonal is shown in Figure 6a.
For each affinity matrix (generated with inputs on WebTWeeText) we compute the mean similarity of those pairs of layers that are included in the generalized diagonal, and compare them to the mean similarity of all other pairs (Figure 6b). For every pair of models, we find that the on-diagonal mean similarity is higher than the off-diagonal mean similarity.
Naive 1-test For each model pair (M,, M,) we then have the null hypothesis H : Mup S My, where Ha, denotes the mean similarity of on-diagonal cells in the affinity matrix comparing M,, and M,, and pup denotes the mean similarity of off-diagonal cells. A naive approach to generate p-value for H which assumes that the affinity matrix's entries are independent, is to perform a one-sided 1-test comparing on- and off-diagonal cells the affinity matrix for M,, and Mp. Performing such t-test for each model pair, we find that the resulting p-values astronomically small, with maximum of x 10 Moreover, the p-value for the union null hypothesis
Hg
H Eup Mup I plyp

---

Published as a conference paper at COLM 2025
is given by the maximum of all of the constituent p-values (Berger, 1982; Kim & Allison, 2006). Therefore, according to the naive I-tests, the p-value for Ho is 8 10 7, indicating that every pair of models exhibit statistically higher similarity on the diagonal than off of it.
Block bootstrap The f-test approach assumes independence of the cells in each affinity matrix. However, this assumption is clearly violated, as any pair of cells in a row i-e.. 11 and 4 A / the affinity matrix comparing models M, and Mp are both functions of the same embedding function M.. The same is true for pairs of cells in any column j, both of which are functions of p(t In addition, the embedding functions at nearby layers within M..
model-e.g, M, and M, -may be dependent due to residual connections and other aspects of the transformer architecture.
p(++1)
To account for these local dependencies, we use the moving block bootstrap (Lahiri, 1999; Davison & Hinkley, 1999) with a block size of 5 x 5 to sample from null distribution which preserves local dependencies but scrambles diagonal structure (see Appendix B for more details). By comparing samples from this null distribution to the actual affinity matrix, we can estimate the probability that the difference of on- and off-diagonal means would be as great under the null model as those of the observed affinity matrix. Repeating this process for every affinity matrix, find maximum p-value of 4.84 10-3 again indicating that the on-diagonal mean is greater than the off-diagonal mean for every pair of models.
5 Conclusion
We compare the activations of wide range of modern LLMs and find that they generate different activation geometries at different depths (Claim 1), but that layers at corresponding depths from different models tend to generate similar activation geometries (Claim 2). We find that early layers from different models are just as similar to one another as middle and late layers, and validate our findings by developing statistical test of diagonal structure.
Acknowledgments
We would like to thank Claire Donnat and Nikos Ignatiadis for their advice on statistical tests of diagonal structure.
References
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah,
Hany Hassan Awadalla, Nguyen Bach, Amit Bahree, Árash Bakhtiari, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio
César Teodoro Mendes, Weizhu Chen, et al. Phi-3 Technical Report: Highly Capable Language Model Locally on Your Phone. Technical Report MSR-TR-2024-1,, Mi- //ww:. /www. mi micoosott...mmmmmmoomommoocooc e- wttp.///.//oe//eeeeeeac////allaa
crosoft, August 2024. URL https:
Samira Abnar, Lisa Beinborn, Rochelle Choenni, and Willem Zuidema. Blackbox Meets Blackbox: Representational Similarity & Stability Analysis of Neural Language Models and Brains. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networrk for NLP, 2019. URL https://aclannhologg.ooo wrg/w1 9-4820/.
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxan- dra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guil- herme Penedo. The Falcon Series of Open Language Models, 2023. URL https: darxxiv.org/abs/2230 16867.
10

---

Published as a conference paper at COLM 2025
David D. Baek, Yuxiao Li, and Max Tegmark. Generalization from Starvation: Hints of Universality in LLM Knowledge Graph Learning, 2024. URL https://axxxi..orrrr 2410. 08255.
Roger L. Berger. Multiparameter Hypothesis Testing and Acceptance Sampling. Technomet- rics, 24(4))295-300, 1982. ISSN 00401706. URL http:///twit h/tp:///wwwww// jstor cettioo.rrg/stblllee httg//tabbe//267823..
http://www.jstor
Davis Brown, Charles Godfrey, Nicholas Konz, Jonathan Tu, and Henry Kvinge. Un- derstanding the lnner-workings of Language Models Through Representation Dissim- ilarity. In Conference on Empirical Methods in Natural Language Processing, 2023. URL https://openreview.
Emily Cheng, Diego Doimo, Corentin Kervadec, Iuri Macocco, Lei Yu, Alessandro Laio, and Marco Baroni. Emergence of High-Dimensional Abstraction Phase in Language Transformers. In International Conference on Learning Representations, 2025. URL hps //arxiv.org//bb//2////.. 15471.
Anthony Christopher Davison and David Victor Hinkley. Bootstrap Methods and Their Appli- cation, volume Reprinted with corrections of Cambridge Series on Statistical and Probabilistic Mathematics. Cambridge University Press, 1999, ISBN 9780521574716.
Nadir Durrani, Hassan Sajjad, Fahim Dalvi, and Yonatan Belinkov. Analyzing Individual Neurons in Pre-trained Language Models. In Conference on Empirical Methods in Natural Language Processing, 2020. URL https: htttp:://acaanthologg.oo wwwptthooogooooo/22222222 org/2020. umnlp-main. 395/.
Gemma Team, Morgane Riviere Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, et al. Gemma 2: Improving Open Language Models at Practical Size, 2024. URL httttt:ts//s/ttte /ttrtp:://arii.oor//ab////////// /etaaa////...e// 00468.000111 18.
Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. OpenWebText Corpus. http://Skylion000, github. how/peeneebbeextorppuuu 2019.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, et al. The Llama 3 Herd of Models, 2024. URL 21783.
Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaei Kheirkhah, Qinyi Sun, Will Hathaway, Neel Nanda, and Dimitris Bertsimas. Universal Neurons in GPT2 Lan- guage Models. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openrevieev hwt..nee///eee.... www.nt//oormm1dd-e111000000
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. URL. J7KBBmI3GGmQ
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. Position: The Platonic Representation Hypothesis. In International Conference on Machine Learning, 2024. URL https://arxiv. wttp://w.g..b//.....s///// 07987.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7B, 2023. URL http://axxiv.oogabb////
06825.
Guy Kaplan, Matanel Oren, Yuval Reif, and Roy Schwartz. From Tokens to Words: On the Inner Lexicon LLMs. In International Conference on Learning Representations, 2025, URL https://arxiv.org//
05864.
11

---

Published as a conference paper at COLM 2025
Kyoungmi Kim and David B Allison. Composite Hypothesis Testing from Multiple Experi- ments: An Approach Built on Intersection- Union Tests and Bayesian Posterior Probabili- ties. Guerra and Allison, eds. Meta-analysis and Combining Information in Genetics, 2006.
Max Klabunde, Mehdi Ben Amor, Michael Granitzer, and Florian Lemmerich. Towards Measuring Representational Similarity of Large Language Models. In UniReps: the First Workshop on Unifying Representations in Neural Models, 2023. URL https://arxiv.cr 022/2312.02730.
Max Klabunde, Tobias Schumacher, Markus Strohmaier, and Florian Lemmerich. Similarity of Neural Network Models: A Survey of Functional and Representational Measures, 2024.
URL http::://arxvv..rg/aaaas//////
06329.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of Neural Network Revrrssskeepessnttatttonnss Revisited. In International Conference on Machine Learning, 2019. URL https:
wrg.obs/////////o/oooooooos//s/b https://arxiv.org/bbb////
00414.
Vedang Lad, Wes Gurnee, and Max Tegmark. The Remarkable Robustness of LLMss: Stages of Inference?, 2024. URL https://arxiv.oor//bbs 2406. 19384.
Soumendra Nath Lahiri. Theoretical Comparisons of Block Bootstrap Methods. The Annals of Statistics, 27(1):386-41) 1999. ISSN 00905364, 21688966. URL httpp://ww.jjsto// org/ stable/120134
Michael Lan, Philip Torr, Austin Meek, Ashkan Khakzar, David Krueger, and Fazl Barez. Sparse Autoencoders Reveal Universal Feature Spaces Across Large Language Models, 2024. URL httpss://arxtte http:/wwwww//b///////ea/
0rg//b5//211..0691111
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning Word Vectors for Sentiment Analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 2011. URL http::/www aclweb. org/anthologg/PP111100
Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth. In International Conference on Learning Representations, 2021. URL https: //openreview.net/eeeeeeeeeee wwt..e/oorrounnnddnckkk.......
Thao Nguyen, Maithra Raghu, and Simon Kornblith. On the Origins of the Block Structure Phenomenon in Neural Network Representations. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom In: An Introduction to Circuits. Distill, 2020. URL https://distiin pubt 0t20//irccits/zoooe
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability. In Advances in Neural Information Processing Systems, 2017. URL https: //ttx//aaxvvv///a///aa wwwaarii.oog//bba...ee org/abs/
1706. 05806.
Daking Rai, Yilun Zhou, Shi Feng, Abulhair Saparov, and Ziyu Yao. A Practical Review of Mechanistic Interpretability for Hansformer-Based Language Models, 2025. URL atgp/bb////0..00/0 02646.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT Rediscovers the Classical NLP Pipeline. In Association for Computationnl Linguistics, 2019. URL https://acllantholog org/
P19-1452/.
Jörg Tiedemann. Parallel Data, Tools and Interfaces in OPUS. In International Conference on Language Resources and Evaluation, 2012. URL htttttoo////caaattooooogyy httphhhloooo.oor/1111225255
12

---

Published as a conference paper at COLM 2025
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems, 2017. URL httpp:://axxvv.or//bbat
.37622.
Ivan Vulić, Edoardo Maria Ponti, Robert Litschko, Goran Glavaš, and Anna Korhonen. Probing Pretrained Language Models for Lexical Semantics. In Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2020.
URL http:///allatthooooy org/ 200//2000.aaa wttp:////22...eem-mmmm. www.lpmmann..-m--emmmmmmmm 586...
Junxuan Wang, Xuyang Ge, Wentao Shu, Qiong Tang, Yunhua Zhou, Zhengfu He, and Xipeng Qiu. Towards Universality: Studying Mechanistic Similarity Across Language Model Architectures. In International Conference on Learning Representations, 2025. URL
http://ppnweeviwe.eee/fooor////e 21181100
John Wu, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. Similarity Analysis of Contextual Word Representation Models. In Association for Compu- fational Linguistics, 2020. URL https: / https//////aaaolooo///oo org/2020/ acl20.acl--00 acl-main. 422/.
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, and Guoyin Wang. Instruction Tuning for Large 10792.
Language Models: A Survey, 2024. URL http://///axxio.o/aabb/////s/
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. InstructionnFoolooiing Evaluation for Large Language Models, 2023.
URL https: ptaps:///arivve http://arvivvoog/a////....o
13

---

Published as a conference paper at COLM 2025
A Diagonal structure is sensitive to the input distribution
The experiments above all focused on the similarity of nearest neighbor relationships on dataset of inputs from OpenWebTexx In Appendix G, we generate affinity matrices for range of input datasets, including IMDB movie reviews (Maas et al., 2011), text from books in English and German (Tiedemann, 2012), and various LLM evaluations (Zhou et al., 2023; Hendrycks et al., 2021). The diagonal structure observed in Section 4 is generally present for other distributions of web text (e.g., IMDB movie reviews), but different patterns emerge when looking at other input distributions, which we investigate here.
A.1 Nearest neighbor relationships are (weakly) preserved between languages
Many of the models we use in our experiments have been trained on multilingual data, and can write and understand multiple languages. In the experiments above, we feed the same dataset D to two models M1 and M2 and collect the resulting activations. Here, using parallel corpus of English and German book translations (Tiedemann, 2012), we instead feed corpus of English texts to M1 and a parallel corpus of translated German texts to M2. The resulting nearest neighbor sets exhibit lower similarity than when using only single language, but we still observe weak diagonal structure, with middle layers being more similar to one another than to early or late layers (Figure 7).
siniartyy
Figure 7: Weak diagonal structure is present when comparing embeddings of different models on parallel texts in different languages. Note that the color scale is zoomed to show lower similarity cells better.
A.2 Instruction tuning changes activation structure in late layers
Instruction tuning is process in which base model (which has generally been pretrained on natural language) is fine-tuned to be better at following instructions (Zhang et al., 2024). We investigate how instruction tuning affects the activations generated by language models.
We compare the base version of Gemma-2-98 to its instruction-tuned counterpart. When given inputs from OpenWeb we observe strong diagonal structure across all depths (Figure 8a). However, when given prompts from a dataset of inputs designed to test model's ability to follow instructions (Zhou et al., 2023) there is much lower similarity at later layers (Figure 8b). This finding is in line with previous work that observed later layers being more strongly affected by fine-tum tuning (Wu et al., 2020).
We hypothesize that later layers could be dedicated to functions that are strongly targeted by instruction tuning, while early layers could be dedicated to functions like input parsing and basic grammar that are needed with or without instruction tuning. Alternatively, the instruction tuning process used could have been designed to only train later layers.
Regardless of the mechanism, this shows that 1) the distribution from which input texts are drawn can affect diagonal structure, and 2) that we can study which parts of a model have been affected by fine-tuning with affinity matrices.
14

---

Published as a conference paper at COLM 2025
40
0.8
layer of Genma (00)
kayer of Genma 2
0.8
20
0.4
10
10
0.2
20
10
40
layor Gemma 2/T (98)
layer of Germa 2:17 (90)
(a)
(b)
Figure 8: Instruction tuning changes the activations generated by models in later layers, but only when given inputs that are targeted by instruction tuning. 8a The affinity matrix of the base version of Gemma-2-98 and its instruction tuned counterparts, given dataset of inputs from OpenWebText. 8b The same as Sa, but using a dataset of inputs from IFEval.
A.3 Random alphanumeric strings
We additionally test sensitivity to the input distribution by repeating the experiments above using a da dataset of of length 100, each consisting of letters, numbers, and punctuation sampled uniformly at random.
random alphanumeric strings. Specifically, we generate 2048 strings
ER
Oceent
Upp Of Linee 1340
Usma
simiarity
1200
2
(T0)
(c)
(b)
Figure 9: Affinity matrices generated with a catasett random strings reveal similarity in nearest neighbor relationships, but not diagonal structure.
Figure 9 reveals that there is similarity between the nearest neighbor relationships of random strings, but without diagonal structure. Just like with web text in Section layers from different models often agree on the nearest neighbor relationships among the random strings. However, unlike with web text, the most similar layers do not necessarily come
15

---

Published as a conference paper at COLM 2025
from corresponding depths. Instead, all early and late layers show similarity with one another, while middle layers are dissimilar between models.
Further investigation of the nearest neighbor sets among random strings reveals that early and late layers tend to assign nearest neighbors based on agreement in the last few characters. For example, if string ends with "Uc", then early and late layers will generally consider any other strings ending with "Uc" to be nearest neighbor. This leads to high agreement, as all of these layers are focused the superficial feature of the inputs. That middle layers do not follow this pattern could suggest that they are not as susceptible to focusing on superficial features of the input, and so generate less structured activations when given random strings.
This experiment shows that, at least in extreme cases, the diagonal structure observed above contingent on the input distribution. It also shows that the mere presence of similarity between two models does not tell the full story, and that we learn more from looking at the pattern of similarity between different pairs of layers.
B Block bootstrap test of diagonal structure
Given an affinity matrix, we want to test whether the on-diagonal cells show higher sim- ilarity scores than the off-diagonal cells. Naively, could perform one-sided f-test comparing the scores in on- and off-diagonal cells, but this would assume that each cell is independent, which need not be true because of residual connections in transformer models (Vaswani et al., 2017).
(a) An affinity matrix A
(b) Bootstrap samples where has been scrambled with block size of 10 x 10.
Figure 10
We use the block bootstrap to generate samples from a null distribution that preserves local dependence structure, but scrambles large-scale structure (Lahiri, 1999; Davison & Hinkley, 1999). The block bootstrap works by breaking an affinity matrix into blocks of given size and then resampling those blocks (with replacement) to create a new matrix that inherits the local dependence of the original, but does not inherit its global structure (Figure 10). We use overlapping blocks with cyclic boundary condition so that every cell has an equal probability of appearing in the resampled matrix. By comparing the difference of on- and off-diagonal means in the observed affinity matrices with their corresponding resampled matrices, test whether the observed diagonal structure is stronger than would be expected as a consequence the local dependence structure and chance.
For each affinity matrix, we generate 10t scrambled bootstrap samples and compute the difference of the and off-diagonal means. The fraction of bootstrap samples that yields a difference at least as large as that observed from the original affinity matrix gives us p-value (Figure 11).
16

---

Published as a conference paper at COLM 2025
7000
5000
0.8
4000
0.4
2000
0.0
diegrnne regonnal
(a)
(b)
Figure 11: Distribution of differences between on- and off-diagonal means for bootstrap samples for particular affinity matrix. The red line shows the difference between on- and off-diagonal means observed with the unscrambled affinity matrix. In Figure 11b, the blue line shows 0.95.
We used this process to compute p-value for every affinity matrix (generated on Open- WebText data), using block sizes of x 5 and 10 x 10. For most model pairs, there was not single bootstrap sample with an on- and off-diagonal difference as large as the observed affinity matrix (Figure 12). No model pair yielded p-value greater than 0.05 with either block size. The maximum p-value with block size 5 x is 0.00484, and the maximum p-value with block size 10 10 is 0.0154.
100
200
500
400
3C0
100
200
500
400
500
Indow
(a)
(b)
Figure 12: Every test yields p-value below 0.05. 12a Manhattan plot generated with block size of 5 x 5. Tests where no bootstrap sample had a difference as great as the observed dif- ference clipped to one the number of bootstrap samples for visualization purposes. 12b Manhattan plot generated with a block size of 10 x 10.
We can combine these tests into single intersection-union test of whether the on-diagonal similarity is higher than the off-diagonal similarity for every pair of models simultaneously. Because we can reject the null-hypothesis for each of the individual tests (as the maximum p-value is below 0.05), we can reject for the intersection-union test, giving us the strong result that every pair of models shows significant diagonal structure (Berger, 1982; Kim & Allison, 2006).
17

---

Published as a conference paper at COLM 2025
C Null model for mutual k-nearest neighbors
How can we interpret the absolute values of mutual k-NN similarity scores, and how much similarity would be expected by random chance?
Deriving the null model Suppose that the nearest neighbor function u returned every size subset of DD with equal probability, i.e.
u (x, f) Uniform ((X X D, ||| =k})
(3)
for any x and f. We then have that
|4 (x, (x, 8) ~ Hypergeometric [||| -
(4)
where Hypergeometric(kkk |P]| 1) is the distribution of successes in draws from population of size |D] contains & successes. By the central limit theorem, we then have that
M (f.g)
(x, ( (x,8)
(5)
D|
XED
k
(k - D || + 1)2
Normal
(6)
|D -1-1' 1'
D ( 2) 1)2
which gives the distribution of M (f,g) under the null assumption made above.
5000000
-10x
-15
-20-10
Figure 13: Null distribution of M with k 10 and || 2048. The log pdf is astronomically small for values larger than approximately 0.02.
Applying the null model The null distribution derived above is extremely concentrated, with the null probability of M (f,g) 2 0.4 being on the order of 1441444 (when & 10 and | 2048). Thus, the mutual k-NN similarity scores appearing in the affinity matrices above are far higher than would be expected under this (very strong) null model.
At the same time, we emphasize the pattern of similarity more so than its absolute value. Even if the level of similarity were much lower, it would still be remarkable that there is more similarity among layers similar depths when compared with pairs of layers with different depths.
18

---

Published as a conference paper at COLM 2025
D Other values of k
In the experiments above, we measure similarity with mutual k-NN with k = 10. Here, we generate versions Figure 4a using various other values of k (Figure 14). The results show similar diagonal structure, even with k = 1 (which corresponds to the probability that the layers being compared agree on the single nearest neighbor).
k=
90
30
lyper of Librra 9.1 (88)
0.8
20
Lter
20
0.4
0.2
10
20
30
40
of Gemmaa (9R)
Ciemer 2
(a)
(c)
= 50
= 100
30
30
layer of Lair (68)
0.6
20
of 3a
20
00
10
10
20
30
40
10
20
30
Inyer of Gemma 2 (9R)
bayer Diemmro 2 (p6))
(b)
(d)
Figure 14: Affinity matrices comparing activations Wpen from layers of Llama-3. 1-8B and demma-2-99 with different values of k. Diagonal structure is present throughout.
19

---

Published as a conference paper at COLM 2025
E Decay curves
The following plots visualize how the relative depth of layers relates to their similarity. In particular, they show how similarity decays as distance increases.
Gamma (88) va Gamma 2
Faizon (40B) Falcon (408)
Liama 3.1 (708) Liame 3:1 (708)
1.0
1.0
1.0
0.0
011375
D.C
0.0
0.4
04
02
02
0.0
D02
0.0
deriaity
McCEMNT frmm smilantt
CESIROSTTTE hoom ARr Barty
20
depth
depth
deept
(a)
(b)
(c)
Figure 15: Nearby layers within a model tend be more similar than distant ones, For given model, each plot shows the similarity between layers as function of their distance. Each line corresponds relative depth of another layer in the same model, and the y-axis shows their similarity. Each curve can be thought of as showing slice of the affinity matrices above. All plots use mutual k-nearest neighbors with TppeWeeTeext as the dataset.
layer with the color indicating its depth. The x-axis shows the
Liama 3.1 (BB) va Gemma 2 (BB)
Liame 3:1 (706) va Mistral vo.3 (7B)
Mistral Nemo va Phi Medium
1.5
0.1
0.0
0.4
044
04
20
10
10
50
0.0
-30
-20
-10
10
20
30
0.0
200
20
-0
10
20
30
armirry
from
dapth
desth
depth
(a)
(b)
(c)
Figure 16: Corresponding layers from different models tend to be more similar than non- corresponding ones. For a given pair of models, each plot shows the similarity between layers in one model to layers in another model those layers get further away from the path of maximum similarity. Each line corresponds to a layer in one model with the color indicating its depth. The x-axis shows the depth of the layer in the other model, shifted so that the highest-similand
layer is at O. The y-axis shows the similarity of the two layers. Each curve can be thought of showing slice of the affinity matrices above. All plots use mutual k-nearest neighbors with OpenWebTer Text as the dataset.
20

---

Published as a conference paper at COLM 2025
F
Model list
We use the following models in our experiments:
Name
Layers
IT
HuggingFace ID
Gemma (2B)
18
No
google/gemma.
Gemma (7B)
28
No
google/ geaele//mmmmaab
Gemma 2 (2B)
26
No
google/ Baaleeggmma-2--1
Gemma 2 (9B)
42
No progle/gemma-2
Gemma 2 IT (9B)
42
Yes
Gemma 2 (27B)
46
No gemma--222
Llama 3.1 (8B)
32
No netellamaa/MeaaLLaaa-1111111
Llama 3.1 IT (8B)
32
Yes nnta:llamnn/MMeae-looass111-----
Llama 3.2 (1B)
16
No meta-laama/llama--221BB
Llama 3.2 (3B)
28
No meta-llamaa/Llama--22--88
Llama 3.2 IT (3B)
28
Yes nete-laamm/Lammm222---nnttntt
Llama 3.2 Vision (11B)
40
No meta-llamma//aamms:::::11-----
Llama 3.1 (70B)
80
No netaallamma/LLamm---1-7088
Llama 3.1 IT (70B)
80
Yes
Llama 3.3 IT (70B)
80
Yes pedeellama//lamma3333--------
Mistral v0.3 (7B)
32
No
mistralai/Misttra/7B--133.
Mistral Nemo
40
No
mistrla//Misttal-emmo-Bsss-22
Mixtral v0.1 (8x7B)
32
No
mistralii/Miittra--577--001
Phi 3 Mini IT
32
Yes
microsott/Phis--min----nsnstt
Phi 3 Medium
40
Yes microsot/hhi--mmciunnkkkiiiniiie
Phi 3.5 Mini IT
32
Yes microsottPPhi33-mninnnnnnuuut
Falcon (40B)
60
No tiluae/falcon--/
Falcon (11B)
60
No tiiuae/facoon-118
Falcon Mamba (7B)
64
No
tilue/falcoonmmmbb27b
Table 1: Models used in experiments, their layer counts, and whether they were instruction
tuned (IT).
21

---

Published as a conference paper at COLM 2025
G Extended affinity matrix arrays
We computed affinity matrices comparing all pairs of all layers in 24 open-weight LLMs. For the sake of brevity, we only included subset in Figure 4c. This appendix contains affinity matrices for all models on all datasets that we tested.
G.1 Rectangular affinity matrices
These affinity matrices have not been resized to fit into squares, and preserve the original dimensions of the affinity matrices, with deeper models occupying more space.
Genre 20 990
200
24
oeme 2000
900
63700
Moon
Saton
Falson Mamba (7%)
Palcon (113)
Falcso (483)
the 3.35M
PhiS Medumit
Matur 1750
Uhemu
Lewa 1 (78R)
Wilio (11R)
133)
Utra (13)
Ukru 3717 195)
Genna (2%)
Chemme 125)
Figure 17: Affinity matrices for embeddings of OpenWebText (Gokaslan et al., 2019).
22

---

Published as a conference paper at COLM 2025
U 32 N00OO (710)
40 (WKWW)
122
Oonee 960
Oenree 9090
Uene 200
11
Unme33 35 200
USWa 3 ToRe
Maer (00)
Maan home
Faboe
Fairw Mannaa (73)
Ralcow (113)
Falcon (483)
the W111
PhiS MedamiT
PeloM
(73)
Uiara (710785))
Cheru (
Lioaa 11 (188)
(118)
Liea 3: (13)
180)
1bboa (8R)
Germa (03)
Genmas (2%)
Okeana (75)
123)
Figure 18: Affinity matrices for embeddings of random alphanumeric strings.
23

---

Published as a conference paper at COLM 2025
Unna 32 N00OO (710
w0 (WKWW)
Genten 122
Oonee 960
Osnee 1199
Oennee g090
Unne 200
11
Unmu 35 200
Usea 3 ToRe
Meee us (09)
Maan home
Oennea
Faboo
Rabce
Fairw Mannaa (2
Ralsow (115)
Falcon (488)
the WiT1
PhiS MedamiT
(73)
Uiara (710788))
Chara 1.1 (185)
Lioaa 1 (188)
(118)
Liera 3.2 (13)
100)
1bbaa (BR)
Germa (02)
Genmas (2%)
Okemna (75)
123)
Figure 19: Affinity matrices for embeddings of IMDB movie reviews (Maas et al., 2011).
24

---

Published as a conference paper at COLM 2025
Gente 030
Oonee 2,00
300
Lams 13t
Uame 300
os (PP))
9490
Faloo Mannaa (7%)
(113)
Ralcow (483)
the Mil1
Phis Medumi
(bo7)
Mattal Kame
Matull (75)
Uaaaa 3317 (790)
Uiera 1.17
Limaa 11 1788
Deme 92 Welio (115)
Lbra 3 (13)
Uaru 1717 185)
Gonena (1775)
Gernaa IT (95)
Gemma 125)
Figure 20: Affinity matrices for embeddings of lead paragraphs from featured articles on Wikipedia.
25

---

Published as a conference paper at COLM 2025
Un 22 NO0OO (710)
w0 (WKWW)
Gentee 122
Oenne D98
Oonee 96)
Oonee 1109
Oennee g090
11
Useee 35 200
Meme ui (00)
Maan home
Faboo
Rabce
Fairw Mannta (2
Ralsow (113)
Falcon (483)
the WIT
PhiS MedamiT
PeloM MI
Matull (73)
Uiara (71078)
Lioaa 1 (788)
(118)
Liera 3:2 (19)
100)
Libaa (BR)
Germa (02)
Genmas (2%)
Ckemna 175)
123)
Figure 21: Affinity matrices for embeddings [FEval (Zhou al., 2023).
26

---

Published as a conference paper at COLM 2025
Unna 32 N00OO (710)
Genten 122
Oonee 360
Oonee 1199
Oennee g090
Uene 200
11
Unmu 35 200
Usea 3 ToRe
Meer us (R0)
Maand home
w0 (WKWW)
Oennea
Faboo
Fabce
Fairw Mannaa (2
Ralcow (113)
Falcon (483)
the WT
PhiS MedamiT
(73)
Uiara (710778)
3.1 3.11 (185)
Lioaa 1 (188)
(118)
Liera 3:2 (13)
100)
Libaa (BR)
Germa (53)
Genmas (2%)
Okemna
123)
Figure 22: Affinity matrices for embeddings of MMLU (Hendrycks et al., 2021).
27

---

Published as a conference paper at COLM 2025
Unna 22 N00OO (710
Genten 122
Oonee 360
Ooneer 1199
Oennee g090
Uene 200
11
U. 35 200
Usea 2 TrRe
Meee us (00)
Maand home
w0 (WKWW)
Oennea
Faboo
Rabce
Fairw Mannaa (2
Ralsow (113)
Falcon (483)
the W1
PhiS MedamiT
(73)
Uheru 3.11 (185)
Lioaa 1 (788)
(118)
Liera 3:2 (13)
100)
1bbaa (8R)
Germa (53)
Genmas (2%)
Ckerma (75)
123)
Figure 23: Affinity matrices for embeddings of OPUS Books (English) (Tiedemann, 2012).
28

---

Published as a conference paper at COLM 2025
32 N00OO (710)
w0 (WKWW)
Genten 122
Oonee 360
Ooneer 1199
Oennee g090
11
U. 35 200
Usea 3 ToRe
Meme us (00)
Maand home
Oennea
Faboo
Rabce
Fairw Mannaa (2
Ralsow (113)
Falcon (483)
the WI
PhiS MedamiT
(73)
Uhenu 3.11 (185)
Lioaa 1 (188)
(118)
Liena 3:2 (13)
180)
1bbua (8R)
Germa (02)
Genmas (2%)
Ckerna (75)
Genna 123)
Figure 24: Affinity matrices for embeddings of OPUS Books (German) (Tiedemann, 2012).
29

---

Published as a conference paper at COLM 2025
G.2 Square affinity matrices
These affinity matrices have been stretched and squished to fit into squares to make them more comparable, in Figure 4c.
Genea 217 (00
Moos (6000
Genew
UnI 00
Mare
Feon (00)
Telun 110
Marde (73)
Malcon (113)
the Wi
Maraal Namo
Mitall v0.3 (73)
Uiema (790)
(78))
Lier IT 133)
Libra 3: (33)
Lier (19)
Larw (85))
Goneaa (177))
Germus (92)
Genna (23)
(75)
Figure 25: Affinity matrices for embeddings of OpenWebText (Gokaslan et al., 2019).
30

---

Published as a conference paper at COLM 2025
Livee 32 ibcon 11t0
Genea 1TRE
g10)
Unmu 11000
Unma 55 (1
Mabu 913 1750
Oonee 125
Oonee 1P8
PAAc 1400
Ulno 32
Palton Manba (73)
Pilloon (422)
the WTT
Munia oot (6ur))
(7%)
Uaaaa 3317
Lanaa (7)0)
Dena 3 (a))
Benui (3A)
Lina 1: (85)
Liera 3:1
21273)
Genmas 2 (85)
Genma 125)
175)
Genera (23)
Figure 26: Affinity matrices for embeddings of random alphanumeric strings.
31

---

Published as a conference paper at COLM 2025
Live 32 ibcon 11t0
Genee 1IR
Gentre g10)
Unmu 11
Unme 55 (1
Mabu M131
(20)
Oonee 128
Ooneea P9g
- 1400
Unno 32
Palton Manba (73)
Pillzon (422)
the IITT
Munna oot (ours)
Kama
(7
Uaaaa 3317 (70)
Lanaa (788)
Dena 3 IT (a))
Benuo 3 (3A)
Lina 17 (85)
Liera 3:1
21273)
Genmas (8%)
Genma 125)
173)
Genera (23)
Figure 27: Affinity matrices for embeddings of IMDB movie reviews (Maas et al., 2011).
32

---

Published as a conference paper at COLM 2025
Lomee 2.7 P0000 m
Oonte (3)
Oonee (PS)
Gonnea 2,00
Oenna pi0
Gerte 23TD
PHO Wedumi
9000
Pallue (150)
Lovoo
Facow Manba (73)
Faloon (113)
PhL We
Midra oon
Mona Kemo
Matull (73)
Liaaa (10)
Uieaa (155))
Ukma (11))
pa)
1 3 (38)
Lier (13)
Libna 3:1 (82)
21275) (175)
Genmaa (25)
(75)
Figure 28: Affinity matrices for embeddings of lead paragraphs from featured articles on Wikipedia.
33

---

Published as a conference paper at COLM 2025
LIve 32 iboo 11t0
g10)
Unmu 11
Unma 55 (11900
Mabu 9131
Oonee 123
Ooneea P9g
-AAc 1400
Ulno 32
Palton Manba (73)
Pillzon (422)
the WTTT
Munna oot (eur))
(7
3317 (700)
Laaaa (780)
Dena 3 IT (a))
Lina I7 (80)
Liera 3:1 (B8)
21273)
Genmaa (85)
Genmaa 125)
1730
Genera (23)
Figure 29: Affinity matrices for embeddings [FEval (Zhou al., 2023).
34

---

Published as a conference paper at COLM 2025
Live 32 ibco (1t0)
Unmu 11
Unma 55 (1190))
Mabu 1131
(20)
Oonee 128
Ooneea P9g
- (400
Unno 32
Palton Manba (73)
Pillzon (402)
the IIT
Munia oot (bur))
(7
3317 (70)
Lanaa (788)
Dena 3 IT (a))
Aenuu 3 (3A)
Lina 17 (85)
Liera (88)
21273)
Genmas (85)
Genmaa 125)
173)
Genera (23)
Figure 30: Affinity matrices for embeddings of MMLU (Hendrycks et al., 2021).
35

---

Published as a conference paper at COLM 2025
Live 32 iboo (1t0
Gentre g10)
Unmu 11
Unma 55 (11000)
Mabu 9131
(20)
Oonte 128
Ooneea P9g
- 1400
Ulno 32
Palton Manba (73)
Pillzon (422)
the IIT
Munia oot (burs)
(7
Uaraa 3317 (70)
Lanaa (750)
Dena 3 IT (a))
Benuo 3 (3A)
Lina 17 (85)
Liera (88)
21273)
Genmaa (85)
Genma 125)
173)
Genera (33)
Figure 31: Affinity matrices for embeddings of OPUS Books (English) (Tiedemann, 2012).
36

---

Published as a conference paper at COLM 2025
Live 32 ibcon (110)
Gentre g10)
Unmu 11
Unma 55 (1
Mabu 1131
(20)
Oonee 128
Ooneea P9g
- 1400
Ulno 32
Palton Manba (73)
Pillzon (402)
the WTT
Munia oot (bur))
(7%)
3317 (70)
Lanaa (758)
Dena 3 IT (a))
Aenuu 3 (3A)
Lina 17 (85)
Liera (88)
21273)
Gennaa (85)
Genmaa 125)
173)
Genera (23)
Figure 32: Affinity matrices for embeddings of OPUS Books (German) (Tiedemann, 2012).
37


Reviews:
[{"review_id": "P4NqXfxyd4", "reviewer": "Reviewer_j38m", "summary": "This paper study the nearest neighbor relationships in hidden layers across different LLM models. The paper makes two observations: 1. Activations collected at different depths within the same model tend to have different nearest neighbor relationships. 2. Activations collected at corresponding depths of different models tend to have similar nearest neighbor relationships. Importantly, the similarity respects the relative depths (e.g., lower layers are similar to lower layers) suggesting representations evolve in similar pattern across LLMs.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The diagonal structure is an interesting finding.\n2. Substantial experimental efforts of many models and queries.", "reasons_to_reject": "1. Claim 1 and 2 are a bit vague, but given claim 2 being true, one can argue claim 1 might be false [if you are happy with the affinity across models, you should probabily be happy with the affinity of two random layers within the same model]. In appendix E1, one can see that the big diagonal shines the most, and **consecutive layers** in one model have high affinity. In paritucular, different layers in Llama models appear to have higher affinity than their affinity against a similar layer in another model.  Clearly, we should get more detailed numerical results to support claim 1.", "questions_to_authors": "I have some suggestions:\n\n1. Decay Rate, as mentioned earlier, we can see consecutive layers in one model have high affinity. We can measure the relationship between affinity and distance (maybe normalize total depth) in layers (within the same model and across models）. For cross models study, maybe we need to pick the maximum affinity layer first, then do the decay rate computation.\n\n2. I would put the diagonal structure （within the same model and across models）as the main message instead of claim 1 and 2\n\nI really like this paper and its' finding, but feels the presentation needs more work.", "extra_scores": {}}, {"review_id": "yDsHFO0FrD", "reviewer": "Reviewer_Dzwk", "summary": "This paper is about representational similarity of LLMs at different depths. The paper defines a nearest neighbor structure for every input and compares similarity of nearest neighbors across different models at different depths.\n\nThe main finding is that there is a diagonal structure, i.e., roughly, models have similarly nearest neighbors at similar depths.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Well-written paper\n- on an interesting research question\n- with really interesting results\n\nOverall, I think, I find this paper strong enough to appear at the main conference, because of its interesting results, even despite some weaknesses, as discussed below.", "reasons_to_reject": "- There is no clear explanation why different models may behave similar at similar depths\n\n- I would find that some more hard numbers would be helpful, i.e., how much more similar are diagonal elements than non-diagonal entries, besides the visual impression you give in the figures and the significance tests in Section 4.1.", "questions_to_authors": "- Why is Fig. 5 not discussed in the text?\n\n- In Def. 3, what is x? Should it be t? There is also no t in this definition, at odds with line 164. (In general, please check your math formula, as NLP reviewers and authors often do not)", "extra_scores": {}}, {"review_id": "EG0KJN84HW", "reviewer": "Reviewer_VG7E", "summary": "The authors show that the activation latent spaces between different LLMs have a significant overlap, sometimes higher than inter-layer overlap between the same LLMs. To measure this overlap the authors use nearest neighbour analysis of the respresentations over sets of prompts and measure how much they overlap. They observe a diagonal structure that emerges, both when comparing the affinity between layers of the same model but also across models.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors explore these results in a large number of different models and datasets, clarifying a result that had been observed - in multiple different ways - in the literature. Their two claims: that LLMs have different activation geometries at different depths and that they share geometries at similar depths even between different models, seem to be well justified.", "reasons_to_reject": "It is possible that cosine similarity is not the best metric to measure the similarity between the internal representations, and that the k-nn overlap results don't generalize to more complex metrics of similarity. \nThe correlation between the activations might not mean that the models use similar representations in similar ways, and no causal experiments were made.", "questions_to_authors": "How does the overlap metric change if examples from multiple datasets are used together? What if the number of examples is different (more than 2048)? Are embeddings at different token positions also similar (eg embedding at the penultimate token position with the last token positon)? \n\nWhere is the blue line is figure 6a?\n\nMight be interesting to cite https://arxiv.org/pdf/2503.04429, which found that one can steer models using activations of different models, even though they had to train \"translators\"", "extra_scores": {}}, {"review_id": "6VIcVwQKCf", "reviewer": "Reviewer_BgXt", "summary": "This paper makes two main claims: 1) that nearest-neighbor relationships induced by model activations vary highly layer to layer and 2) that these vary less across different models at similar / parallel layers. It shows this by computing the nearest neighbor for each datapoint \n\nBy and large, I think that this paper is correct and easy to understand. However, its results (in particular regarding Claim 1) are rather unsurprising, as similar claims have already been discussed at length in existing literature. Much ink has been spilled discussing what LLMs do at each of their layers: detokenization in the lower layers, the beginnings of syntax in the middle layers, and more complex syntax / semantics / retokenization in the upper layers. This discussion has been going on in earnest since at least 4-5 years ago, and I don't think that the consensus has ever shifted away from \"models do different things at different layers\". This paper carefully notes that it studies representations, not mechanisms, but I'm not sure this is a meaningful distinction in this case.\n\nClaim 2 is somewhat more novel - I am personally unsurprised by it, but I think that less existing work has studied this topic. It is interesting that different models have similar k-nearest-neighbors at similar layers. However, I do think some larger takeaway is missing here: what are the factors underlying the similarities between models? Can we tell which things are being encoded across models? There are techniques - in particular cross-coders - that I think could have answered these more interesting questions within the span of a COLM paper.\n\nOverall, my main concerns for this paper regard novelty - is the content novel, and is there enough novel content? So far, I feel that Claim 1 is not very novel, and Claim 2 could be better explored, given the methods available. While this paper is 9 pages long, the amount of novel content is somewhat smaller than I would expect.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper is well-written and easy to follow. Its methods are simple (a good thing!), and for that reason quite clear.\n- In general, I feel that this paper's results are correct. Although they have chosen only one similarity metric (and I would have liked more discussion of / experimentation with alternatives), I personally believe these results are most likely robust to the particular metric chosen.", "reasons_to_reject": "**Low novelty / Lack of engagement with prior work**: In general, this paper is not very novel; its Claim 1 is one that has already been well-addressed (if indirectly) in past work. This comes in part from a lack of references to the interpretability literature. I think that a thorough read of interpretability review papers, from older ones like [Bertology](https://aclanthology.org/2020.tacl-1.54/), to newer ones like [this one (section 4 in particular)](https://arxiv.org/abs/2405.00208) would be helpful for contextualizing this work within the interpretability literature.\n- Claim 1: This flavor of claim has been studied (indirectly) quite a lot. [Tenney et al. (2019)](https://aclanthology.org/P19-1452/) studies what model representations contain at each layer, and finds that each contains different things. While Tenney et al. is a bit outdated at this point, more recent work has spoken to this issue as well. [Lad et al. (2024)](https://arxiv.org/abs/2406.19384?) for example, review some relevant literature; see also work like [this](https://arxiv.org/abs/2401.06102). In general, the review papers I pointed to earlier will show quite a bit of interest in what models do / represent at each layer, and a pretty clear consensus (as I see it) that models have different stages of processing, wherein different (groups of) layers do / represent different (specific) things.\n- Claim 2: I think that this claim is somewhat more novel, but could have better been answered in a more satisfying way. Right now, the only conclusion that can be drawn is that models are cross-layer similar according to this one metric / dataset. By expanding this to other metrics / datasets, these results would become a bit more robust. Moreover, other techniques for cross-model comparison, in particular crosscoders ([Lindsey et al., 2024](https://transformer-circuits.pub/2024/crosscoders/index.html)) would provide a little more insight into *what* is driving this similarity. In general, some discussion of SAEs seems appropriate; SAE work has also looked at feature evolution across layers, and found similar features across models. More broadly, follow-ups / ablations / etc. would have strengthened this claim/paper, which right now feels a bit spare. \n\n**Methodological weaknesses**: although I believe that this paper's conclusions are correct, I don't always agree with the experimental setup / interpretation of results.\n- The authors always take representations from the last position of an input. This means that their representation comparison is performed only over a limited subset of the dataset - always the last token (it's unclear to me if this is always e.g. a period, which would be quite bad, or just the end of a sentence). There's not a good reason to do this: people often do this in order to extract \"sentence representations\" from autoregressive LMs, but this is (in my opinion) poor practice. Although the last token of a passage is the only one that can contain information about all prior tokens, autoregressive LMs have no reason to generate a \"sentence representation\" per se, and such final tokens cannot be treated as one.\n- The authors study only 2048 texts. I imagine that this is for computational reasons, but I worry that using such a small number of texts limits the robustness of these results. Results could be somewhat different for other datasets; in general, representational analysis on a small set of data is dangerous; [see this paper](https://arxiv.org/abs/2104.07143) for an example.\n- I know that \"As such, the particular choice of representational similarity measure is not [y]our focus.\" but have you experimented with any others to show that your results are robust to this choice? Given the small number of datapoints studied, this would be a rather inexpensive experiment to run.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Reviewers overall liked the paper. Authors properly responded to comments. Perhaps reorient some of the claims around the diagonal structure or emphasise it a bit more.
Decision: Accept