The Representation Landscape of Few-Shot Learning and Fine-Tuning in Large Language Models
Authors: Diego Doimo
Alessandro Pietro Serra
Alessio ansuini
Alberto Cazzaniga
Venue: NeurIPS 2024 poster (2024)
Keywords: LLMs
geometry hidden representations
clustering
Primary area: interpretability_and_explainability

Abstract: In-context learning (ICL) and supervised fine-tuning (SFT) are two common strategies for improving the performance of modern large language models (LLMs) on specific tasks. Despite their different natures, these strategies often lead to comparable performance gains. 
However, little is known about whether they induce similar representations inside LLMs.  We approach this problem by analyzing the probability landscape of their hidden representations in the two cases. More specifically, we compare how LLMs solve the same question-answering task, finding that ICL and SFT create very different internal structures, in both cases undergoing a sharp transition in the middle of the network. In the first half of the network, ICL shapes interpretable representations hierarchically organized according to their semantic content. In contrast, the probability landscape obtained with SFT is fuzzier and semantically mixed. In the second half of the model, the fine-tuned representations develop probability modes that better encode the identity of answers, while less-defined peaks characterize the landscape of ICL representations. Our approach reveals the diverse computational strategies developed inside LLMs to solve the same task across different conditions, allowing us to make a step towards designing optimal methods to extract information from language models.

The Representation Landscape of Few-Shot Learning and Fine- Tunnin in Large Language Models
Diego Doimo
Alessandro Serra
Alessio Ansuini
Alberto Cazzaniga
Area Science Park, Trieste, Italy
{diego. doimo, alessandro serra, serra, alessio. ansuini, alberto........ aalberto.cazzaanggll @areasciencepark.
Abstract
In-context learning (ICL) and supervised fine-tuning (SFT) are two common strate- gies for improving the performance of modern large language models (LLMs) specific tasks. Despite their different natures, these strategies often lead to compara- ble performance gains. However, little is known about whether they induce similar representations inside L.MMs. We approach this problem by analyzing the probabil- ity landscape of their hidden representations in the cases. More specifically, we compare how LLMs solve the same question-ans wering task, finding that ICL and SFT create very different internal structures, in both cases undergoing sharp transition in the middle of the network. In the first half of the network, ICL shapes interpretable representations hierarchically organized according their semantic content. In contrast, the probability landscape obtained with SFT is fuzzier and semantically mixed. In the second half of the model, the fine-tuned representa- tions develop probability modes that better encode the identity of answers, while the landscape of ICI. representations is characterized by less defined peaks. Our approach reveals the diverse computational strategies developed inside LLMs to solve the same task across different conditions, allowing us to make step towards designing optimal methods to extract information from language models.
Introduction
With the rise of pre-trained large language models (LLMs), supervised fine-tuning (SFT) and in- context learning (ICL) have become the central paradigms for solving domain-specifie language tasks [1]. Fine-tuning requires set of labeled examples to adapt the pre-trained LLM to the target task by modifying all part of the model parameters. ICL, on the other hand. preferred when little or no supervised data is available. In ICL, the model receives an input request with a task description followed by few examples. then generates a response based this context without updating its parameters.
While operationally, the differences between SFT and ICL are clear in how they handle model param- eters, it is less clear how they affect the model's 's representation space. Although both methods can achieve similar performance, it is unknown whether they also structure their internal representations in the same way. Differently from recent contributions which compared ICL and SFT in terms of generalization performance (2-4] and efficiency [5], in this work analyze how these two learning paradigms affect the geometry of the representations.
Previous studies described the geometry of the hidden layers using distances and angles [6, 7]. often relying on low-dimensional projections of the data [8-10]. In contrast, we take density- based approach [11] that leverages the diwddimensionality of the hidden layers [12-14] 1-11] and finds
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

zero-shot
few-shot
fine-tuned
machine statiistics
philesepy
machine learning
machine statistice
phllesory
early layers
statistica
philosapy
physica
mnath
math
mAtE
chemistry
point
(a) Consistency between the peak composition and the MMLU subjects. A schematic view of the density
peaks in the early. The coloring reflects the presence of the subjects.
zero-shot
few-shot
fine-tuned
late layers
(b) Consistency between the peak composition and the MMLU answers. A schematic view of the density peaks in the late layers. The coloring reflects the presence of answers with the same letter: A (blue), B (okra), C (pink), and D (light bluc).
Figure 1: The LLMs representation landscape of few-shot learning and fine-tuning. This figure illustrates the distribution of probability modes in large language models (LLMs) during question- answering task (MMLU). The top row shows representations from layers near the input, while the bottom shows those near the output. We compare three scenarios: zero-shot (O-shot, left), in-context learning (5-shot, center), and fine-tuning (right). In the 5-shot scenario, early layers (top, center) develop better representations of the dataset's subjects. Conversely, in the fine-tuned model, the late layers (bottom, right) more accurately reflect better the letter answers.
the probability modes directly on the data manifold without performing an explicit dimensional reduction.
For ICL and fine-tuning, we track how the multimodal structure of the data evolves across different layers as LLMS solve semantically rich multiple-choice question task and measure how the geomet- rical properties intertwine with the emergence of high-level, abstract concepts. We study ICL in few-shot prompting setup and find that despite ICI. and SFT reach the same performance, they affect the geometry of the representations differently. Few-shot learning creates more interpretable representations in early layers of the network, organized according to the underlying semantics of data (Fig. 1, top-center). On the other hand, fine-tuning induces multimodal structure coherent with the answer identity in the later layers of the network (Fig. bottom-right).
The key findings of our study are:
1. Both few-shot learning and fine-tuning show clear division between model layers, marked peak the intrinsic dimension of the dataset, and sudden change in the geometrical structure of the representations (Secion 4.1);
2. Few-shot learning leads to semantically meaningful clustering of the representations in early layers, organized hierarchically by subject (Section 4.2);
3. Fine-tuning enhances the sharp emergence of clustered representations according to answers in the second half of the network (Section 4.3).
summary, our geometric analysis reveals a transition between layers that encode high-level semantic information and those involved in generating answers. By studying the probability landscape on either side of this transition, we uncover how fine-tuning and few-shot learning take different approaches to extract information from LLMs, ultimately solving the same problem in distinct ways.

---

2 Related work
Probing the geometry of the embeddings. A classical approach to understanding the linguistic content encoded in token representations is probing [15]. Inspecting the embeddings with linear [16, 17] or nonlinear [18, 19] classifier probes allowed to extract morphological [20] syntactic [6] and semantic [17, 21-24] information. Probing has also been used to explore the geometrical properties of hidden layers. LLMs can represent linearly in hidden layers concepts such as space and time [25], the structure of the Othello game board [26], the truth or falsehood of factual statements [27]. The linear representation hypotheses [28] can be used to show that LLMs encode hyponym relations as simplices in their last hidden layer [29]. However, another line of work suggests that L.MSs represent temporal concepts like days, months, and years, as well as arithmetic operations in a nonlinear way using circular and cylindric features [30, 31].
Describing the geometry of the embeddings directly. Directly analyzing the geometrical distribu- tion of embedding vectors and their clustering can also provide insights into how LLMs organize their internal knowledge. Early studies on BERT, GPT2, and ELMo found that token embeddings are distributed anisotropically, forming narrow cones [32] and isolated clusters [7. 13]. This phenomenon
has also been observed in CLIP embeddings [33], where image and text tokens occupy different cones, separated by a gap. A similar gap also exists between the subspace of different languages in multilingual LLMs [10], which enables approximating language translation with geometrie translation between these subspaces. Recent work has shown that changes in the intrinsic dimension of the representations is related to different stages of information processing in LLMs, linking the rise of abstract semantic content to layers characterized by geometric compression [34] and the transition from surface-level to syntactic and semantic processing to layers of high dimension [35].
Comparing in-context learning and fine-tuning in LLMs. Recent studies have compared ICL to fine-tuning in LLMs by analyzing their ability to generalize, their efficiency, and how well they handle changes in the training data. Several factors can influence the outcomes when comparing ICL and fine-tuning. These include the format of the prompts in ICL [36, 37]. the amount of training data used for fine-tuning [38, 39], and the size of models being compared [40]. In large models, ICL can be more robust to domain shifts and text perturbations than it is fine-tuning smaller-scale ones [2, 3]. However, when ICL and fine-tuning are compared in models of the same size fine-tuned on sufficient data, SFT can be more robust aattof-diitribution especially for medium-sized models [4]. Additionally, SFT achieves higher accuracy with lower inference costs [41].
3 Methods
3.1 Models and Dataset.
MMLU Dataset. We analyze the Massive Multitask Language Understanding question answering dataset, MMLU [42], taking the implementation of cais_mmlu from Huggingface. The MMLU test set is one of the most widely used benchmarks for testing factual knowledge in state-of-the-at LLMS [43-46]. The dataset consists of multiple-choice question-answer pairs divided into 57 subjects All the questions have four possible options labeled with the letters "A," "B," "C," and "D." When prompted with question and set of options, the LLMS must output the letter of the right answer. In this work, we will analyze the MMLU test set, where each subject contains at least 100 samples, with a median population of 152 samples and the most populated class containing about 1534 examples. To reduce the class imbalance without excessively reducing the dataset size, we randomly choose up to 200 examples from each subject. The final size of our dataset is 9181.
Language models and token representations analyzed. We study the models of Llama3 [45], Llama2 [47] families, and Mistral [43]. We choose these LLMs because, as of May 2024, they are among the most competitive open-source models on the MMLU benchmark, with an accuracy significantly higher than the baseline of random guessing (25%, see Table A1). All the models we analyze are decoder-only, with layer normalization at the beginning of each attention and MLP block. Llama2-7b, Llamma-8b, and Mistral-7b have 32 hidden representations, Alama2-13b 40, 1lama2-70 and 1lama3-70 80. all cases, we analyze the representation of the last token of the prompt after the normalization layer at the beginning of each transformer block. For transformers
3

---

trained to predict the next token, the last token is the only one that can attend to all the sequence. and in the output layer, encodes the answer to the question.
Few-shot and fine-tuning details. We sample the shots from the MMLU dev set, which has five examples per subject. This choice differs from the standard practice, where the shots are always given in the same order for every input question. Table Al shows that the final accuracies are consistent with the values reported in the models' technical reports [43, 45, 47]. We fine-tune the models with LoRA [48] on a data set formed by the union of the dev set and some question-answer pairs of the validation set to reach an accuracy comparable the 5-shot one. The specific training details are in Sec. A.
3.2 Density peaks clustering
We study the structure of the probability density of data representations with the Advanced Density Peaks algorithm (ADP) presented in D'Errico et al. [11] and implemented in the DADApy package [49]. ADP is mode-seeking, density-based clustering algorithm that finds the modes of the probability density by harnessing the cow-dimensional structure of the data without performing any explicit dimensional reduction. ADP also estimates the density of the saddle points between pairs of clusters, which measures their similarity and provides information on their hierarchical arrangement. At high level, the ADP algorithm can be divided into three steps: the estimation of the data's intrinsic dimension (ID), the estimation of the density around each point, and a final density-based clustering of the data.
Intrinsic dimension estimation. We measure the ID of the token embeddings with Gride [50]. Gride estimates the ID of data points embedded in RD, using the distances between a token and its
d(u d
nearest neighbors. This is done by maximizing the likelihood function L((xs) = B(k 4(24 17F1
where μl is the ratio of the Euclidean distances between a point and its nearest neighbors of rank kg 2k and ki k, is the ID, and (kk, k) normalizing Beta function. By increasing the value of k. the ID is measured on nearest neighbors of increasing distance. The ID estimate is chosen as the value where the ID is less dependent on the hyperparameter & and the graph d((k) exhibits plateau [50, 51]. On this basis, we choose k= = 16.
Density estimation. We measure the local density Pa. with ANN estimate: PLA = R/NV... Here, N the number of data points and Vi, the volume of the ball, which has a radius equal to the distance between the point i and its 4th nearest neighbor. Crucially, in this step, we compute the volume using the intrinsic dimension, setting k = 16, the value used to estimate the ID.
Density-based clustering. With the knowledge of the Pt, we find collection of density maxima C= (c assign the data points around them, and find the saddle point density pa.33 between a pair of clusters Ca Cø with the procedure described in Sec. B. We can not regard all the local density maxima as genuine probability modes due to random density fluctuations arising from finite sampling. ADP assesses the statistical reliability of the density maxima with t-test on log p - log pa.3. where p" is the maximum density in c.. Once the confidence level Z is fixed, all the clusters that do not pass the t-test are merged since the value of their density peaks is compatible with the density of the saddle point. The process is repeated until all the peaks satisfy the t-test and are statistically robust with a confidence Z. We set Z = 1.6, the default value of the DADAPY package.
In the following sections, we will also use the notion of core cluster points defined as the set of points with density higher than the lowest saddle point density. These are the points whose assignation to a cluster is considered reliable [11, 52]. With a slight abuse of terminology, we will use the terms "clusters", "density peaks", and "probability modes" interchangeably.
Measuring the cluster similarity. The ADP algorithm considers two clusters similar if connected through a high-density saddle point. This is done defining the dissimilarity So,e between pair of clusters and c3 as S.,3 = log maz - log pr..3 n being the density of the highest peak. With Sa,9, we perform hierarchical clustering of the peaks. We link the peaks starting from the pair with the lowest dissimilarity according to Sa,8 and update the saddle point density between their union c, and

---

Uama 3
Liama 3 8b
Liama 3 Bb
224
310
0
0.65
0.68
Number Clunters
Core Foint Folnt raacion fraction
344
232
22
0.39
Layer
Layer
Layer
Figure 2: Intrinsic dimension, number of density peaks, and fraction of core points. Figure shows the ID (left), the number of density peaks (center), and the fraction of core points (right) for the last-token representation of Llama3-8b for increasing number of few-shots and fine-tuned models. The three quantities change in the proximity of layer 17 in a two-phased fashion.
the rest of the peaks cs with the WPGMA linkage strategy [53]: log p7: log pa. log 8
After the update of S, repeat the procedure until merge all the clusters. In this way, display the topography [11] of the representation landscape of layer, namely the relations between the density peaks, as dendrogram.
Reproducibility. We the experiments single Nvidia A100 GPU with VRAM of 40GB. Extracting the hidden representation of 70 billion parameter models requires 5 A100, and their fine-tuning requires 8 A100. We provide code to reproduce our experiments at https: https://githbb com/diegodoimoo geometry.icl..ccc....
4
Results
4.1
The geometry of LLMs' representations shows two-phased behavior.
We start by exploring the geometric properties of the representation landscape of LLMs. Our analysis proceeds from broad description of the manifold geometry its finer details. First, we measure the intrinsic dimension (ID) to understand the global structure of the data manifold. Next, we will describe the intermediate-scut behavior, counting the number of probability modes on it. Finally, we analyze the density distribution at the level of individual data points within the clusters. These three quantities consistently show a two-phased behavior across the hidden layers of the LLMS we analyzed. All profiles of this and the following sections are smoothed using a moving average over two consecutive layers. We report the original profiles in the Appendix from Sec. D.2 to D.4 and in Sec. D.6.
Abrupt changes in intrinsic dimension and probability landscape in middle layers. We measure the ID of the hidden representations with the Gride algorithm (see Sec. 3.2). Figure shows the results for Llama3-80 -bb; the analysis on the other models can be found in Sec. D.2 to D.4 of the Appendix. The left panel shows that the ID changes through the layers with two phases, increasing during the first half of Llama3-8b and decreasing towards the output layers. Specifically, the ID rises from 2.5 after the first attention block and peaks around layer 16. The value with the number of shots, from 14 in the base model to 16.5 when 5-shot context is added. The fine-tuned model (0-shot) reaches a maximum ID of 21 at this layer. In the second half of the network, the IDs sharply decrease over the next three layers. For the few-shot representations, the ID profiles gradually decay in the final part of the network, while for the O-shot models, the ID increases again.
peak increases
The same two-phased behavior appears in the evolution of the number of clusters on the hidden manifold (center panel). In the first half of the network, the probability landscape has higher number of modes, ranging between 60 and 70, when the model is given shots, roughly matching the number subjects. In the 0-shot and fine-tuned cases, it remains below 40. After layer 16, number of peaks decreases significantly, remaining between 10 and 20. The right panel describes the representation landscape within individual clusters, measuring the fraction of core cluster points those with a density higher than the lowest saddle point, indicating reliable cluster assignment [11].
5

---

Liama 3 8b
Liama 3 70c
Mistral 1 76
0.4
0.8
0.60
0
pt
0.72
0.
8.3
0.63
sho:
g00
0:44
127
022
0.2
030
Layer
Layer
Layer
Figure 3: Adjusted Rand Index (ART) between clusters and subjects. ARI between clusters and the subjects for Llama-3-8b (left), Llama-3-700 (center), and Mistral-7b (right) for an increasing number of few-shots and fine-tuned representations. In all cases, the match between cluster and subjects partition is highest at the beginning of the network and for an increasing number of shots.
Before layer 17, the core point fraction is higher, indicating a better separation between probability modes. Notably, the few-shot setting shows a core-point fraction of about 0.6, much higher than the O-shot and fine-tuned case, which remains around 0.2.
The evolution of ID, number of clusters, and core cluster points is qualitatively consistent among the models we analyzed. In the Llama2 family, the ID peak is less evident (Fig. A3-bottom). In particular, it is absent for the less accurate model, Llama2-7b. Nonetheless, the number of clusters, core points, and the remaining quantities, presented in the following sections, change according to the same woo-pasedd trend as the other models.
4.2
The probability landscape before the transition.
We now describe the data distribution by focusing the semantic content of the last token, specifically analyzing whether the cluster composition is consistent with the prompt's 's topic. Using the 57 MMLU subjects as reference, compare the differences in the early layers of the LLMs between ICL and fine-tuning.
Few-shot learning forms clusters grouped by subject. To evaluate how well the clusters align with the subjects, we use the Adjusted Rand Index (ARI) [54]. An ARI of zero indicates that the density peaks do not correspond to subjects, while an ARI of one means a perfect match (see Appendix C for a detailed presentation). Figure 3 shows that as the number of few-shots increases, the ARI rises from below 0.27 the O-shot context to 0.82 in Llama3-8h (left), 0.72 in Llama3-70f (center) and 0.63 in Mistral (right) in the 5-shot settings. These ARI values correspond to remarkable degree of purity of the clusters with respect to the subject composition. When the ARI is at its highest, 75 out of 77 clusters in Llama3-8b (layer 4), 53 out of 69 in Llama3-70b (layer 7), and 43 out of 53 in Mistral (layer 5) contain more than 80% of tokens from the same subject. In the next section, leveraging this high homogeneity of the clusters, we will connect the cluster similarity to the similarity between the subjects of the points they contain.
Additionally, when the number of shots grows, the ARI peak shifts to earlier layers, and the peak becomes narrower. For example, Llama3-8b, the 0-shot profile (blue) has a broad plateau extending until layer 13. With one and two-shot settings (orange and green), the profiles show a couple of peaks between layers and 9, and with 5-shots (red), a single large maximum at layer 3. The trend is consistent across other models, especially those with 70 billion parameters (see Fig, 3-center for Llama3-70b, and Fig. A9 in the Appendix for Llama2-70b), In all cases, providing a longer, contextually relevant prompt enables models to identify high-level semantic features (i.c., the subjects) more accurately and earlier in their hidden representations.
Few-shot prompting is not the only factor that increases the ARI with the subject. In the 0-shot setup, as the performance improves, LLMs organize their hidden representations more coherently with respect to the subject. In Llama3-8b and 70b models, where the 0-shot accuracy is above 62% (see Table A AI), the O-shot ART is around 0.25. For the rest of the models with lower accuracy (below 56%), the ARI is below 0818 (see blue profiles in Figs. 3 and A7).

---

divea kroskigg
preh 500Y
hunna sbxvaityy no ao disu
High stheed pirit Sighonen
mathee mathemalt
mal santttattto
Virlloge
10g aenaa
accounting. nictory
medicie
martering parchatus
palicy
geagrapph
econamica
phytecoshy
Jurisgh Serusencc
physiis cheresrr
lagic
madiine learning splence eccnannattio
absstr abvtrrcc aigetora
5 shot
nelx 11
mix 7
max 26
mix
mik
mix 12
mix 19
kgg dondny
log density
shot
fine-tuned
Figure 4: Density peaks in the layers that best encode the subjects in Llama3-8b. The dendrograms show the organization of the density peaks Llama3-8b the layers where the ARI with the subjects is highest for the 5-shot setup (top) and O-shot set-up (bottom left) and fine-tuned model (bottom- right). In the 5-shot setup, the clusters are populated by examples from one or two related subjects, and their similarity reflects the semantic relationships between the subjects. In O-shot and fine-tuned representations (bottom panels), some large clusters contain many subjects.
The distribution of the density peaks mirrors the subject similarity. When the models learn "in context", not only do the number and composition of density peaks become consistent with the subjects, but they also organize hierarchically to reflect their semantic relationships. In this section, describe only the cluster distribution in Llama3-8b in the layers where the ARI is highest and report the other models the Appendix, Sec. D.7.
Figure 4 shows the probability landscapes of the Llama3-8b in O-shot (bottom left), 5-shot (top), and fine-tuned model (bottom-right) as dendrograms. Dendrograms are helpful visual descriptions of hierarchical clustering algorithms [55]. We perform hierarchical clustering of the density peaks using agglomerative procedure described in Sec. 3.2. the layers where the ARI is highest, the density peaks are homogeneous, and one-to-one mapping between clusters and subjects allows us to estimate subject similarity based on the dendrogram obtained from the (density-based) hierarchical clustering of the peaks.
assign single subject to each Ieaf of the dendrogram. This
In all cases (0-shot, 5-shot, and fine-tuned model), clusters of subjects from the same broader field (STEM, medicine/biology, humanities, etc.) tend to be close together. However, in 0-shot and fine-tuned settings, the probability landscape has fewer and less pure density peaks at the subject level. In contrast, in the 5-shot setting, the number of clusters and their purity increase, and the 77 peaks are organized according to their high-level semantic relationships. For example, in the top panel of Fig. 4, four major groups of similar subjects can be identified: medicine and biology (orange). philosophy, jurisprudence, and moral disputes (bluc), math, physics, and chemistry (red) and machine learning and computer science (green). In addition, these groups and hierarchical structures consistent across different models (see All to A12). For instance, clusters related to statistics, machine learning, and computer science often grouped together, as are those of chemistry, physics, and electrical engineering, or economy, geography, and global facts.
The structure we described is also robust to changes in the confidence level Z. In the Appendix, from Sec. E.2. to Sec. E.2.8, we report the dendrograms obtained with Z and Z = Importantly, even with Z = 0, where all local density maxima are considered as probability modes, the probability landscape of the 0-shot and fine-tuned models remains largely mixed. In contrast, the probability landscape of 5-shot representation is more stable to variations of Z.
7

---

Lama 3 Bb
Liama 3 70c
Mistral 1 7b
0.5
035
0.50
1:40
040
0.15
s
033
1:25
0023
0.23
133
038
0.31
655
0.5
100
Layer
Layer
Layer
Figure 5: Adjusted Rand Index between clusters and final answers. Adjusted Rand Index (ARI) between clusters and the MMLU answers (test set) for Llama3-8b (left), Llama3-70b (center), and Mistral-7b (right). In the second part of the network, the purity of the clusters w.r.t the answer partition is highest for fine-tuned models.
4.3 The probability landscape after the transition.
In Sec. 4.1, we observed that the number of density peaks decreases in the middle layers of the network. This reduction happens because the model needs to identify the answer from four options the output, causing points related to the same answer to cluster around the same unembedding vector. In addition, when the model is uncertain about its predictions, some output embeddings tend to lie close to the decision boundaries of the last hidden representation, resulting in a flatter density distribution with fewer peaks. Even when the model is highly accurate, the linear separability of the answers does not guarantee distinct density peaks because the embeddings may still be near the decision boundary as long as they on the correct side. However, more pronounced density peaks emerge as the model confidence grows and the data moves away from the decision boundaries. This section shows that SFT sharpens these density peaks in the later layers more than ICL. However, as model size and accuracy increase, the representation landscapes of ICL and SFT become more similar.
Fine-tuned density peaks better encode the answers better than few-shot ones. We evaluate how well the clusters match the answer partition (i.e., "A," "B," " "C," "D") using the ARI (see See. 4.2). When the models are fine-tuned, four to five large clusters emerge in the second part of the network, grouping answers with the same label. These clusters collect more than 70% of the data between layers and 30 Llama3-8b, more than 90% between layers 45 and 75 of Llama3-70b, and more than 65% between layers 21 and 30 in Mistral. In these clusters, the most common letter represents over 90% of the point Llama3-8b, over 70% in Llama3-70b, and over 90% in Mistral. As a result, the ARI (see purple profiles in Fig. 5) rises sharply in the middle of the network, reaching approximately 0.25 in Llama3-8b. 0.45 in Llamaa-70b, and 0.2 in Mistral. These ARI are related to the MMLU test accuracies of 65%, 78.5%, and 62%, respectively (see Table A2).
In contrast, in ICL, the clusters are more mixed, and their number is smaller. Llama3-80 in the 5-shot setup, one cluster contains 70% of the points in the last layers. In the 0-shot case, four clusters with a roughly equal distribution of letters contain the same amount of data (blue profile). Similar trends appear in Mistral's 's late layers. In both models, the ARI values for few-shot context stay below 0.05 (Fig. 5). Interestingly, in Llama3-70b (and a lesser extent Llama2-70b, see Fig. A8-right), the representation landscape of ICI. starts to resemble that of the fine-tuned models. Between layers 40 and 77, about 80% of the dataset forms five large peaks, and in four of them, the fraction of the most common letter is above 0.9, similar to fine-tuned models. Consequently, in these layers, the ARI for few-shot contexts (see orange, green, and red profiles in Fig. 5-center) oscillates between 0.35 and 0.40, except for the 0-shot profile, which decays from 0.3 to 0.05 (blue profile).
The different ways in which tne-tuning and ICL shape the representations of the network in the second half depend on the learning protocol, model size, and performance. In smaller models with moderate accuracy (below 65% / 70%), SFT and ICL can perform similarly as the 5-shot setup (see Table AI) but they alter the geometry of the layers in different ways. However, with higher accuracy models like Llama3-70k (accuracy above 75%), both the performance of the model and the topography of the hidden representation tend to converge.
8

---

Liama 80
Uame 3 706
Mistral 1 7b
24
1.6
similarity
Sayeer similiart
Layer similanty
275
1ayer
layer
1ayer
Figure 6: Evolution of similarity between O-shot and fine-tuned models during training. Panels show the dynamics of the representations' similarity with the base models for Llama3-8b (left) and Llama3-70b (center), and Mistral (right). Late representations change the most during fine-tuning.
Fine-tuning primarily alters the representations after the transition. In fine-tuned models, training leads the emergence of structured representations that align with the labels. Figure 6 shows where, during the training, layers change the most in Llama3-8b (left), Llama3-706 (center), and Mistral (right). We compare the fine-tuned checkpoints with the O-shot representations before training begins. To measure the similarity between representations, we use the neighborhood overlap metric [34, 56] that calculates the fraction of the first nearest neighbors of each point shared between pairs of representations, averaged over the dataset. Figure 6 shows that the similarity between representations is lower in the second part of the networks, decaying more sharply to its final value in the middle of the network from 0.5 to roughly 0.3 between layers 13 and 17 in Llama3- 8b and Mistral, from 0.6 0.4 after layer 33 in Llama3-70b (see dark blue profiles).
This picture is consistent with what is shown in the previous sections. In the first half of the network, the representation landscapes of 0-shot and fine-tuned models are similar both geometrically (Fig. 2) and semantically (Figs. 3 and 4). In the second half of the network, where the representations are modified more during the training, fine-tuned models develop fewer peaks, more consistent with the label distribution than those of the other models (Fig. 5).
5 Discussion and conclusion
This study described how the probability landscape within the hidden layers of language models changes as they solve a question-answering task. comparing the differences between in-context learning and fine-tuning. We identified two phases in the model's internal processing, which are separated by significant changes in the geometry of the middle layers. The transition is marked by peak in the ID and a sharp decrease in the number and separation of the probability modes. Notably, few-shot learning and fine-tuning display complementary behavior with respect to this transition. When examples are included in the prompt, the early layers of LLMS exhibit well- defined hierarchical organization of the density peaks that recovers semantic relationships among questions' subjects. Conversely, fine-tuning primarily modifies the representations to encode the answers after the transition in the middle of the network.
Advantages of the density-based clustering approach. Our research highlights how variations in density within the hidden layers relate to the emergence of different levels of semantic abstraction, concept previously explored by Doimo et al. [56] in convolutional neural networks (CNNs) trained for classification. In CNNs, the probability landscape remains unimodal until the last handful of layers, where multiple probability modes emerge according to a hicrarchical structure that mirrors the similarity of the categories. In decoder-only LLMs solving semantically rich question- question-answeriine answering task, these hicrarchically organized density peaks appear the early layers of the models, especially when they learn in context. Our methodology also extends the work of Park et al. [29], enabling the discovery of meaningful hierarchies of concepts beyond the final hidden layer of LLMs, where the data representation can be non-linear [30].
Moreover, unlike previous studies that utilized k-means to identify clusters within hidden representa-
tions [9, 13, 57], the density peak method does not assume a convex cluster shape or impose a priori the number of clusters. Instead, clusters emerge naturally once specifie value is set (see Sec. 3.2). This allows for the automatic discovery of potentially meaningful data categorizations based on the

---

structure of the representation landscape without specifying external linguistic labels and without introducing additional probing parameters. In this respect, an approach is similar that of Michael et al. [58], who used a weakly supervised method relying on pairs of positive/negative samples to uncover latent ontologies within representations.
Intrinsic dimension and information processing. As discussed in section 4.1, peak in intrinsic dimension separates two groups of layers serving different functions and being distinctly influenced by in-context learning and supervised fine-tuning. Other studies have also highlighted the crucial role of peaks in marking blocks of layers dedicated to different stages of information processing within deep neural networks. For example, Ansuini et al. [12] observed peak of the ID in the intermediate layers of CNNs, separating layers that remove low-level image features like brightness from those that focus extracting abstract concepts necessary for classification. In transformers trained to generate images, Valeriani et al. [34] identified two intermediate peaks delimiting layers rich in semantic features of the data characterized by geometric compression. In LLMs, Cheng et al. [35] showed that an ID peak marks a transition from representation that encodes surface-leve linguistic properties to one rich in syntactic and semantic information. These studies suggest that ID peaks consistently indicate transitions between different stages of information processing within the hidden layers.
Application to adaptive low-rank fine-tuning. Our findings could improve strategies for adaptive low-rank tne-tuning. Several studies [59-61] tried adjust the ranks of the LoRA matrices based on various criteria of 'importance" or relevance to downstream tasks. Our analysis of the similarity between fine-tuned and pre-trained layers (see Fig. 6) reveals that later layers are most impacted by fine-tuning, indicating that these layers should be assigned ranks. This approach would naturally prevent unnecessary modifications to the early layers during fine-tuning.
Limitations. Estimating the density reliably requires a good sampling of the probability landscape. This can be delicate condition if the intrinsic dimension is high, often happens in neural network hidden layers. The ID values we report in this work lie between 4 and 22, with most of the layers having an ID below 16. Rodriguez al. [62] showed that the density can be estimated reliably up to 15-20 dimensional spaces. Still, the upper bounds are problem-specife and depend on the density estimator used, the nature of the data, and the dataset size. In this work, we analyzed MMLU, which has semantically rich set of topics characterized by good enough sampling. In the Appendix, in Sec. D. we show that our results extend another dataset mixture with subject partition similar to MMLU. However, the analysis of other QA datasets and generic textual sources would make our observations more general. The prompt structure can also be made more general. In the current study, we studied ICL framed as few-shot learning but further investigations would strengthen our findings. Finally, the description of the transition observed the proximity of half of the network can be analyzed more in detail, for instance, by providing an interpretation of the mechanism [63] underlying the information flow from the context to the last token position [64].
more differentiated contexts
Acknowledgements and disclosure of funding
We thank Alessandro Laio for many helpful discussions and valuable suggestions. We also thank the technical support of the Laboratory of Data Engineering staff and acknowledge the AREA Science Park supercomputing platform ORFEO.
A.A., A.C., and D. D. were supported by the project "Supporto alla diagnosi di malattic rare tramite l'intelligenza artificiale" CUP: F53C22001 770002. A.A., A. C. were supported by the European Union NextGenerationEU within the project PNRR "PRP@ CERIC" IR0000028 Mission Component 2 Investment 3.1 Action 1. A.S. was supported by the project PON "BIO Open Lab (BOL)- Raforzamento del capitale umano"- CUP: J72F200009400077.
10

---

References
[1] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhenghao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing, 2021.
[2] Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Lee Boyd- Graber, and Lijuan Wang. Prompting GPT-3 to be reliable. In The Eleventh International Conference on Learning Representations, 2023.
[3] Anas Awadalla, Mitchell Wortsman, Gabriel Ilbarco, Sewon Min, Ian Magnusson, Hannaneh Hajishirzi, and Ludwig Schmidt. Exploring the landscape of distributional robustness for question answering models. In Yoav Goldberg. Zoritssa Kozareva, and Yue Zhang. editors. Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5971-5987, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguisties.
[4] Marius Mosbach, Tiago Pimentel, Shauli Ravfogel, Dietrich Klakow, and Yanai Elazar. Few-shot fine-tuning vs. in-context learning: A fair comparison and evaluation. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Compulational Linguistics: ACL 2023, pages 12284-12314, Toronto, Canada, July 2023. Association for Computational Linguisties.
[5] Haokun Liu, Derek Tam, Muqeeth Mohammed, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficien finettuning is better and cheaper than in-context learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[6] John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 4129-4138, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
[7] William Timkey and Marten van Schijndel. All bark and no bite: Rogue dimensions in trans- former language models obscure representational quality. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4527-4546, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[8] Daniel Loureiro, Kiamehr Rezaee, Mohammad Taher Pilehvar, and José Camacho-Colladee Analysis and evaluation of language models for word sense disambiguation. Computational Linguistics, 47:387-443, 2021.
[9] Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda Viegas, Andy Coenen, Adam Pearce, and Been Kim. Visualizing and measuring the geometry of bert. In H. Wallach. H. Larochelle, A. Beygelzimer, d' d'Alché-Bue, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[10] Tyler Chang, Zhuowen Tu, and Benjamin Bergen. The geometry of multilingual language model representations. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 119-136, 136, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
[11] Maria d' Errrico, Elena Facco, Alessandro Laio, and Alex Rodriguez. Automatic topography of high-dimensional data sets by non parameetit density peak clustering. Information Sciences, 560:476-492, 2021.
[12] Alessio Ansuini, Alessandro Laio, Jakob H Macke, and Davide Zoccolan. Intrinsic dimension of data representations in deep neural networks. Advances in Neural Information Processing Systems, 32, 2019.
[13] Xingyu Cai, Jiaji Huang, Yuchen Bian, and Kenneth Church. Isotropy in the contextual embed- ding space: Clusters and manifolds. In International Conference on Learning Representations, 2.21.

---

[14] Emily Cheng, Corentin Kervadec, and Marco Baroni. Bridging information-theowe and geometric compression in language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pages 12397-12420, Singapore, December 2023. Association for Computational Linguistics.
[15] Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207-219, March 2022.
[16] Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes, 2017.
[17] Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 1073- 1094, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics.
[18] Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang. Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference Learning Representations, 2019.
[19] Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can cram into a single ! vector: Probing sentence embeddings for linguistic properties. In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2126-2136, Melbourne, Australia, July 2018. Association for Computational Linguisties.
[20] Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. What do neural machine translation models learn about morphology? In Regina Barzilay and Min-Yen Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Pupers), pages 861-872, Vancouver, Canada, July 2017. Association for Computational Linguistics.
[21] Yonatan Belinkov, Lluis Mårquez, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. Evaluating layers of representation in neural machine translation on part-of-speech and semantic tagging tasks. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1-10, Taipei, Taiwan, November 2017. Asian Federation of Natural Language Processing.
[22] T. Blevins, O. Levy, and L. Zettlemoyer. Deep rnns encode soft hierarchical syntax. Proceedings of the 56th Association for Computational Linguistics (ACL), 2018
[23] Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593 4601, Florence, Italy, July 2019. Association for Computational Linguistics.
[24] Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842-866, 2020.
[25] Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint 22Xiv:2310.06822. 2023.
[26] Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. In Yonatan Belinkov, Sophie Hao, Jaap Jumelet, Na- joung Kim, Arya McCarthy, and Hosein Mohebbi, editors. Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 16-30, Singapore, December 2023. Association for Computational Linguistics.
[27] Wes Gurnee and Max Tegmark. Language models represent space and time, In The Twelfth International Conference on Learning Representations, 2024.
12

---

[28] Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41 International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 39643-39666. PMLR, 21-27 Jul 2024.
[29] Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. The geometry of categorical and hierarchical concepts in large language models, 2024.
[30] Joshua Engels, Isaac Liao, Eric J. Michaud, Wes Gurnee, and Max Tegmark. Not all language model features are linear, 2024.
[31] Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gen- eralization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02111 2022.
[32] Kawin Ethayarajh. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Kentaro Inui, Jing Jiang. Vincent Ng. and Xiaojun Wan, editors, Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing TEMNLP-IJCNLLS pages 55-65, Hong Kong, China, November 2019. Association for Computational Linguistics.
[33] Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 17625. Curran Associates, Inc., 2022.
[34] Lucrezia Valeriani, Diego Doimo, Francesca Cuturello, Alessandro Laio, Alessio Ansuini, and Alberto Cazzaniga. The geometry of hidden representations of large transformer models. Advances in Neural Information Processing Systems, 36, 2024.
[35] Emily Cheng, Diego Doimo, Corentin Kervadec, luri Macocco, Jade Yu, Alessandro Laio, and Marco Baroni. Emergence high-dimensional abstraction phase in language transformers. arXiv preprini arXiv:2405. 15471, 2024.
[36] Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Reseurch, pages 12697-12706. PMLR, 18-24 Jul 2021.
[37] Albert Webson and Ellie Pavlick. Do prompt-based models really understand the meaning of de Marneffe, and Ivan Vladimir Meza Ruiz,
their prompts? In Marine Carpuat, editors, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2300-2344, Seattle, United States, July 2022. Association for Computational Linguistics.
[38]
Teven Le Scao and Alexander Rush. How many data points is prompt worth? In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tammoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Confer- ence of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 2627-2636, Online, June 2021. Association for Computational Linguistics.
[39] Branislav Pecher, Ivan Srba, and Maria Bielikova. Comparing specialised small and general large language models on text classification: 100 labelled samples to achieve break-even performance, 2024.
[40] Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 8086-8099 Dublin, Ireland, May 2022. Association for Computational Linguisties.
13

---

[41] Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-effiieett fine-tuning better and cheaper than in-context learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 1950-1965. Curran Associates, Inc., 2022.
[42] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[43] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023,
[44] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 2xXiv:3303.08774 2023.
[45] Meta.
Introducing meta llama 3: The most capable openly available 11m to date, 2024.
[46] Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, arXiv preprint arXiv:2403.055330 2024.
[47] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint Siti::2307.092886 2023.
[48] Edward Hu, yelong shen, Phillip Wallis, Zeyuan Alu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[49] Aldo Glielmo, luri Macocco, Diego Doimo, Matteo Carli, Claudio Zeni, Romina Wild, Maria d' l'Errico, Alex Rodriguez, and Alessandro Laio. Dadapy: Distance-based analysis of data- manifolds python. Putterns, page 100589, 2022.
[50] Francesco Denti, Diego Doimo, Alessandro Laio, and Antonictta Mira. The generalized ratios intrinsic dimension estimator. Scientific Reports, 12(1):20005, 2022.
[51] Elena Facco, Andrea Pagnani, Elena Tea Russo, and Alessandro Laio. The intrinsic dimension of protein sequence evolution. PLoS computational biology, 15(4)=e1006767, 2019.
[52] Alex Rodriguez and Alessandro Laio. Clustering by fast search and find of density peaks. Science, (14(6119):14422114999 2014.
[53] Robert R Sokal. A statistical method for evaluating systematic relationships. Univ. Kansas, Sci. Bull., 38: 1409-143488 1958.
[54] Lawrence Hubert and Phipps Arabie. Comparing partitions. J. of Classification, 2(1):193-118 1985.
[55] R. Tibshirani, T. Hastie, and J.H. Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction: : with 200 Full-color Illustrations. Springer series in statistics. Springer, 2001.
[56] Diego Doimo, Aldo Glielmo, Alessio Ansuini, and Alessandro Laio. Hierarchical nucleation in deep neural networks. Advances in Neural Information Processing Systems, 33:7526-7536. 2020.
[57] Sara Rajace and Mohammad Taher Pilehvar. A cluster-basee based approach for improving isotropy in contextual embedding space. Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, edi- tors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 575-584, Online, August 2021. Association for Computational Linguistics.
14

---

[58] Julian Michael, Jan A. Botha, and Ian Temney, Asking without telling: Exploring latent ontologies in contextual representations. In Bonnie Webber, Trevor Cohn. Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNI.P), pages 6792-6812, Online, November 2020. Association for Computational Linguistics.
pages
[59] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-effun efffcccett fine-tuning. The Eleventh International Conference Learning Representations, 2023.
[60] Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. DyLoRA: Parameter- efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. In Andreas Vlachos and Isabelle Augenstein, editors, Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3274-3287, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics.
[61] Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. Sparse low-rank adaptation of pre-trained language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4145,, Singapore, December 2023. Association for Computational Linguistics.
[62] Maria d' Errico, Elena Facco, Alessandro Laio, and Alex Rodriguez. Automatic topography of high-dimensional data sets by non-parametric density peak clustering. Information Sciences, 560:476-492 2021.
[63] Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds Danny Hernandez, Andy Jones, Jackson Kermion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. pttps//transfomme--crrrttt pitcuits w/wra//wwrr//nde.....ee
[64] Tom Lieberum, Matthew Rahtz, János Kramár, Neel Nanda, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla, 2023.
[65] Glenn W. Milligan and Martha C. Cooper. A study of the comparability of external criteria for hierarchical cluster analysis. Multivariate Behavioral Research, 21(4):441 1986.
[66] William M. Rand. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical Association, 66(33)):8468500, 1971.
[67] Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. TheoremQA: theorem-driven question answering dataset. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7889-7901, Singapore, December 2023. Association for Computational Linguistics.
[68] Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R. Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level scientific groblem-solvingg abilities of large language models, 2024.
[69] Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale ReAding comprehension dataset from examinations. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785-794, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.
15

---

Appendix
A Fine-tuning setup.
The dataset on which we fine-tune the models is the union of the MMLU dev set (all five examples per subject are selected) and subset of the MMLU validation set. We select up 20 examples per subject for Llama2-7b and Llama2-133 13bb and up to 40 examples per subject for the rest of the models. In the first case, the dataset size 1065, and the second is 1439. We train Llama3-8 and Mistral-7b for 6 epochs and the remaining models for 4.
We fine-tune the models with LoRA. The LoRA rank is 64, a is 16, and dropout 0. For the 70 billion models, we choose a rank of 128, and a is 32. This is an empirically reasonable choice since the embedding dimension of these models learning rate = 2 10 10-4; for the 70 billion models we decrease it to 1 10 10-4. For all the models, we apply a cosine annealing scheduler and a linear warm-up for 5% of the total iterations. We fine-tune all the models with batch size 16 using the Adam optimizer without weight decay.
times larger than the billion ones. We use
Below, report the performance of the MMLU test and tine-tuned models. To compute the macro average, we first measure the accuracy for each subject. Then, compute the arithmetic mean of the subject accuracies. The micro average is the fraction of correct answers taken over the dataset. The two quantities differ since the dataset unbalanced.
all the 14042 samples, for O-shot, 5-shot,
Table A1: Zero and 5-shot accuracies. We report micro and macro averages on the MMLU test set.
model
num shots accuracy % (macro/micro)
Llama3-88
0 shot
63.7 62.1 66.5 65.1
Llama3-8b
5 shot
Llama3-70b
0 shot
76.7 / 75.0 79.2 78.5
Llama3-700
5 shot
Mistral-7b Mistral-7b
0 shot
57.8 / 55.9 63.7 / 62.1
5 shot
Llama2-7b Llama2-7b
0 shot
38.9 37.5 46.6 45.9
5 shot
Llama2-133
0 shot 5 shot
52.9 / 52.0
Llama2-13b
55.4 54.8
Llama2- 1loma2-70bb
0 shot
66.3 / 65.5 69.5 68.8
Llama2-70b
5 shot
Table A2: Fine-tuning accuracies. We report the micro and macro averages on the MMLU test set. We only report the micro average the train sets.
model
epochs
accuracy test (macro/micro)
accuracy % train (micro)
Llama3-8b
65.6 64.8 78.5 / 79.0 61.4 59.9
94.8
Llama3-70b
93.8
Mistral-7b
96.4
Llama2-7b
51.9 / 50.7
73.8
Llama2- 133b
56.1 55.55
79.7
Llama2-70b
69.9 71.1
92.4
16

---

B Iterative search of density peaks and saddle points.
Let Nig1) be the of points nearest to x; in Euclidean distance a given layer
The first step of the density-peant clustering is finding the point of maximum density pi (namely the probability peaks). Data point i is maximum if the following two properties hold: (1) P: Pj for all the points j belonging to Na. (f): (II) i does not belong to the neighborhood (i) of any other point of higher density [11]. (I) and (II) must be jointly verified the neighborhood ranks symmetric between pairs of points. A different integer label C = fc'....c") is then assigned to each of the maxima. The data points that
maxima iteratively linked to one of these labels by assigning the same label as its nearest neighbor of higher density to each point. The set of points with the same label corresponds a cluster.
The saddle points between two clusters are identified as the points of maximum density between those lying on the borders between the clusters. A point X; c is assumed belong to the border de-,eh with a different peak c exists point x, E (1) n c whose distance from is smaller than the distance from any other point belonging to c². The saddle point between c and e is the point of maximum density 0ge .0.
C The Adjusted Rand Index.
To determine to which degree the density peaks are consistent with the abstractions of the data, we will compare the clustering induced by the density peaks algorithm with the partition given by the 300 classes and that given by the high-level subdivision in animals and objects. Among the many possible scores, we choose the Adjusted Rand Index (ARI) [54], one of the best clusters of external evaluation measures according to [65]. The Rand Index (RI) [66] measures the consistency between a cluster partition to reference partition R counting how many times a pair points:
are placed in the same group in C and R; b placed in different groups in and R;
e are placed in different groups but in the same group in R:
d are placed in the same group in C but in different groups in R:
and measures the consistency with RI (a+b)/(a+b+c--+ee d). The Rand Index is corrected for
chance, meaning it does not give a constant value (e.g., zero) when the two assignments are random.
[54] proposed to adjust RI, taking into account the expected value of the Rand Index, Ne. under suitable null model for chance:
a + b- 1he
ARI = a a+b+c+d-
(1)
ARI is equal when the two partitions are consistent and when the assignments are random and can take negative values. A large value of ARI not only implies that instances of the same class are put the same cluster (homogeneity) but also that the data points of a class are assigned single cluster (completeness).
17

---

D Additional experiments
D.1 Analysis an additional dataset mixture.
In this section, we validate the findings shown in Section 4.1 on a dataset constructed from Theo- remQA [67], ScibenchQA [68], Stemez [3], and RACE [69]. This dataset contains roughly 6700 examples not included in MMLU, ten subjects from STEM topics, and a middle school reading comprehension task (RACE), with least 300 examples per subject. We keep four choices for each answer. The 0-shot, 5-shot, and fine-tuned accuracies in Llama3-8b are 55%, 57%, and 58%, respectively.
In Fig. Al-left, we see that the intrinsic dimension profiles have peak around layers 15/16, the same layers in MMLU (see Fig. 2-1eft). This peak in ID signals the transition between the phases described in the paper. Before layer 17, few-shot models encode better information about the subjects (ARI with subjects above 0.8). Between layers 3 and 7, the peaks in 5-shot layers reflect the semantic similarity of the subjects (see the dendrograms for layer five reported in Fig. A2).
Fine-tuning instead changes the representations after layer 17, where ARI with the answers for the ~ 0.15, higher than that of the 5-shot and O-shot models. The absolute value is lower than that reported in the main paper (Fig. 5-left) because the fine-tuned accuracy reached the STEM subjects in this dataset is lower. Overall, the results are consistent with those shown in the paper for MMLU.
375
225
ANl Subjerrt
Lotter
0.
shtt
shoth
0.80
Layer
Figure A1: Intrinsic dimension (left), ARI with the subjects (center) and with the answers (right) in Llama 3 8b. The dataset is constructed from Scihench, which has 541 QA pairs about chemistry, math, physics), TheoremQA with 598 QA pairs about business, computer science, math, and physics, Stemez with 4083 QA pairs about biology, business, chemistry, computer science, economics, engineering, physics, psychology, and the test set of RACE with 1436 QA pairs about middle school tests. We finetuned Llama 8b, keeping 50 examples per topic.
computer science
computer science
engineering
physics
physics
biology
chemistry
chemistry
math
math
physics
psychology
business
economics
RACE
Figure A2: Organization of the density peaks of layer 5 of Llama 3 8b in the 5-shot setup.
18

---

D.2 Intrinsic dimension profiles.
Liama 8b
Liama 70b
Mistral
110
31.0
13.4
132
111
44
22
27-
Layer
Layer
Layer
Uame 7b
Liema 2 13b
Liama 70b
225
225
225
pt
354
15.4
15.4
22
22
22
Layer
Layer
Layer
Figure A3: Intrinsic dimension (ID) of the data representations in hidden layers of large transformers for increasing number of shots in the prompt and fine-tuned representations. Llama 8b (upper left), Llama 3 70b (upper center), Mistral 7b (upper right), Llama 2 7b (lower left), Llama 13b (lower center), and Llama 70b (lower right). For each layer, we calculated the intrinsic dimension of the hidden representations produced the last token of the prompt, which provides the answer. The algorithm used is Gride with 16 nearest neighbors considered. We observe that all models exhibit similar behavior in the early layers regardless of the number of shots in the prompt. However, from the middle of the network onwards, the profiles diverge. Configurations that achieve higher agreement between clusters and the letter given as the answer (as shown in A0)) in the last layers will have a lower intrinsic dimension. These results support the intuitive observation that fewer features are needed to represent the information as the model starts to encode the letter will provide as the answer.
D.2.1 Scale analysis of the intrinsic dimension.
Figure A4: Seale analysis the intrinsic dimension for Llama 3 8b. The plot presented herein depicts the intrinsic dimension (ID) at layers 5, 10, 15, 20, 25, and 30, varying the number of nearest neighbors & utilized by the Gride algorithm. Following the methodology delineated [50. 51]: we observe the evolution of the ID function of & to identify plateau the estimates. After initial growth, from & = 8 to = 16, the ID stabilizes and then decreases. Consequently, we select & 16 it represents the scale appropriate for examining the topology of the data being.
19

---

D.3 Number of Clusters
Liama
Uama 70b
Mistral 7b
83.
012
E 14
223
Number of Clusters
57.3
Number
23.3
22.5
15.
Liama
Uama 133
Liame
Eilee 14
U.nee pt
Number Clusters
15.3
Figure A5: Number of clusters across layers large transformers. The plots above show number of clusters identified by the Adaptive Density Peak (ADP) clustering algorithm at each model layer. In configurations where multiple examples are provided in the context, we can observe that the number of clusters closely matches the 57 subject categories present in the MMLUU dataset. This phenomenon aligns with the observation that the agreement between the cluster partition and the subject-induced partition from MMLU is primarily influenced by the number of examples provided, as illustrated in Figure 3.
D.4
Core points
Lisma
Ulma
7b
Dahst mve 2uhs
s
2
Point Frccioon
4.49
1.54
0335
Layer
Layer
Liama
264
Ealek
thtt
3.5
3454
2
2 2t.
Layer
Layer
Figure A6: Number of core points across layers in large transformers. The figure shows the fraction of points classified by ADP "core." indicating they were assigned a cluster with higher confidence. Conversely, the algorithm designates points with lower densities than the highest border density between clusters as "halo" and thus, these points are discarded.
20

---

D.5
Experiments on Llama2-7b, Llama2-13b and Llama2-70b.
Liama 7b
Liama 2 13b
Liama 70b
1.5
0.50
672
0.12
0.12
8.4
h
0.54
127
0:27
0.2
040
Layer
Layer
Layer
Figure A7: Adjusted Rand Index (ARI) between clusters and MMLU subjects (test set). The figure shows the ARI with the subjects for Llama2-7b (left), Llama2-13b (center), and Llama2-70b (right) for increasing the number of shots in the prompt and fine-tuned representations. The plot illustrates the agreement between the partition generated by the clusters and the partition induced by the subject labels from the MMLU dataset. The clustering was performed using Density Peak Clustering with Z= 1.6, the default value of the DADAPY package. As depicted in 3, the agreement is higher the initial layers of the network and increases with growing number of examples provided in the context. In other words, the clustering aligns more closely with the subject label in the early layers of the network in the few-shot setting.
shat
0
shat
6.0
2 2
2:5h:
Schor
shot
130
shot
0.22
0 sho:
0.2
0 ft
022
0.33
0.20
035
0:31
028
0.31
0.05
100
0.04
21
23 35
37
17 Layer
75
Layer
Layer
Figure A8: Adjusted Rand Index between clusters and final answers. Llama2-7b (left), Llama2- 13b (center), and Ilama2-70b (right) for increasing number of shots in the prompt and fine-tuned representations. The plot depicts the correspondence between the partitions generated by the clusters and the partitions induced by the model's predicted answer letters. The clustering was performed using Density Peak Clustering with Z = 1.6, the default value of the DADAPY package. As shown in 3, we can observe trend that is opposite to the one seen with the subject label: the purity of the clusters with respect to the answer partition is higher in the final layers of the model, and it does not depend on the number of examples provided in the prompt. In this case, the fine- fine-tuned model with zero examples in the context (0-shot) achieves the highest Adjusted Rand Index (ARI), indicating the highest agreement between the clustering and the model's predicted answer partitions.
21

---

D.6 Profiles without moving average
Liama 8b
Liama 70b
Mistral
0.50
8.0
0.2
1.3
h
0.4
0.54
8.43
0.45
g 0.45
0.7
0.2
0.0
Layer
Layer
Layer
Ujama 7b
Liama 2 13b
Liama 2.7
0.50
0
o pt
0.72
sho.h
3.10
0.2
0.7
319
0.00
Layer
Layer
Layer
Figure A9: Adjusted Rand Index (ARI) between clusters and MMLU subjects (test set) without moving average. Llama 3 8b (upper left), Llama 3 70b (upper center), Mistral 7b (upper right), Llama 7b (lower left), Llama 13b (lower center), and Llama 70b (lower right). The plots produced on the same data as the one in A7, but in this case, we displayed the result without averaging over range of successive layers
0.49
sbbu
shot
0.55
stst
2 <her
0.4
5 -het
shot
330
0.2
0
2.0
0.2
0.20
0.3
6.0
0.65
0.00
22
43
33
21
Layer
Layer
Ilama
[lama
70b
139
sbat
1.4
0.48
Scher
0.5
0 0
0 o-hor
0.5
0:3
0.20
635
0.3
100
0104
25
Layer
Figure A10: Adjusted Rand Index between clusters and final answers without moving average. Llama 3 8b (upper left), Llama 3 70b (upper center), Mistral 7b (upper right), Llama 7b (lower left), Llama 2 13b (lower center), and Llama 2 70b (lower right). The plots are produced on the same data the one in but in this case, we displayed the result without averaging over a range of successive layers
22

---

D.7 Dendrograms (5-shot) in layers where the subject ARI is highest
D.7.1 Llama3-8b
abstract algebra
dastract_algebrra
abstract_aigebra
astronomy
electrical
econometrics
high_school, sumputer_science high school statistics
high school
college www.p/ommppeeeeeeeee serenter_science
machine_learmng
Computer_securfe
siamentarymmathemaa
farmal logic
formal logic
highh_school_hhee chemistry. college_chenistty
college, chemistryy
callege, physics
conceptual physics
conceptual physics
high_school mat.emattthssss
college_mete satg___mttemaaics
highoschool physks, high_schooll_heeee chemistry
high gigh_school_pyyee physicss
moral disputes
moral disputes
sexuality
prahistory
prehistory
philosophy
jurispuudence
philosophy
high_school micrrocconomics
school
high _aamroncooomiiss
hiph_school gww.gggmeeetee.ommmmm politics
human aging
global_fact
miscellaanouus
high_school gengraphyy
high school geography
public_ revii_eeeaiios,, jurisprudence
hct.sscchoooerrrrree poriicliiicc
seciology
us fo_foriign_po international
securittystudee studies
marketing
high_school_psyye pssycolooyyyyyyyy
professional, peychology
business ethics
diinical knowledge, college_ medicine
porkcal_kowweedd
virology
medical medical_genetics gegeeeeeetss
high school biology. callege_biel blology
schoos biclogy, college_ medicine
college biology
anatomy
anatomy
professional, medicine
nutrition
high history
high school european Nistory, sighoschool us Nistory
high school world histary
professional accounting
prafessional accounting
professional law
logical fallacies
logicai,fallacieee
world religions world_religiann
Figure All: Llama3-8b, 5-shot, Z=1.6, ARI 0.82, 77 clusters
23

---

D.7.2 Llama3-70b
public_
management
security
high_school microeonnomiss
high,
high, school govemment, and politics
high_ goveroingvemmmmeni,.oom and poltics
miscelleneous
human, aging
high. school paych psychhlogyyy.... slogy, professional protessiona__pycchllngy
college biology, college, medicine, college,
college_physics physics
virology
medical gennicclgeeetics. high_
mix of 7 subjects
knowledge, anatomy
anatomy
moral, disputes
moral,
business_etie ethics
business, ethics
excurit_studiis
high, school chemistry, college, chemistry
vollege_physic.
pectrical_engineere
high_schie school statistics, high_school science, callege_ computermmcccccccee science
high_ school physics, tomia_Jogic
college computer,
computer,sslencee science, college college_mtthemeeeeeeee mathematics, computer,
conceptual
conceptuel_physics
professional aaccsutnnnlaccuuuiing
ecomometrics
Ifglcal_sallacies
Ingical_sallacies
philosaphy
tormal
prehistery
prehistory
astronomy
high. school world history
high school, eurapean high schosscholl.oo us, _iistoryy
high, school us history, high sigh__shhoo__uuer europeean_hitteeee
Figure A12: Llama3-70b, 5-shot, Z=1.6, ARI 0.65, 69 clusters
24

---

D.7.3 Mistral-7b
world history
High pistery
High school european history, high school us Fistory
Hightschool wtwpseccooouushrrrr.o.o righ wttpssccoooooppppeeee puroopurrrrrrreee Mistory
ecomometrics
moral scerarios
protessional
concoptual physics
concaptuall
concsptui_physiis physics, medical_genctis
globai_farts
callege, chemstryy high school chemistry, High school physics
college_shysics
prenistory
astronomy
mix of Bsubjects
rutrition, cinirai
recuriy_th.dde
security_sluee sludies
virekey.
morai_dispune
philosophy
raligiors
miscellar
management
us_toregr_policy
jurkpruderce, publi jelations
govemment polities
high/schol_geeemment.
markeling, public relations, Jurisprudence
righ_scheolpoyyhhllyy
professional poychelogg
high school miccoeccnnomcss formal logk
peccecooocccccoooooooo aging, high_school.googrphhy hiph_school, microcconomics
high_school_geeraphhy
sociclogy
business ettics
logical tallacies
logical_sallacies
farmal logic
formal Jagic
abstract
abstroct aigebra
high_school_tatstiic sagh__shool__ommee comprtrreetttteeeeeeec science, machine Jearning
callege, computer hightschool computer
mix of subjects
machine Jeaming
mathomattcss
High_cchol_mathheee
slemettarymmtthhmaaii
mathamatics
protessional sccounting
profisssional accounting
professional_bn
Figure A A13: Mistral, 5-shot, 7=1.6, ARI 0.53, 57 clusters
25

---

D.7.4 Llama2-7b
logical_falie fallacies
logical
professional accounting
professional accounting
professional_be low
high, school world histery
per_sshhol_eurappe history, high_school__s, history
professional medicine
us_foreign_polig
college, high biology, clinical knwwleoge college, biology
jinnial_knneledde
nutrition
viralogy
medical genetics, anatomy
anatomy
college_biology
public
markztting
seciology
professional pey.hcloyhhyy... high schoolchheeeee peychology
philosophy
human human sexuality
mospl_cisputes
jurtsprudence
prehistory
high sighoschool_ag mmarrocconmmics
high school geography
high sphhscchol_gg govemment and politics
miscellaneous
glcbalfarts
managemeent
ethics
security studies
abstract_algebra
Dostract_algebrrr
high mathematics
college mathematics, high school mathematics
formal_logic
formal, logic
high geph_schoolsstaee statistics callege_campiine
wwwper/omppteeteeeeee science
high, school statistics, econometrics, high, school computer science, college, computer science college_computeer science, high, school computeer_sceecc
machine learning
econometrics
high_schoolphysic
college_physics
school
astronemy
electrical_eggineering
electrical, engineering
conceptual_physic
comeentary_mahhmmaiin
world, ruligioa
Figure A14: Llama2-7b, 5-shot, Z=1.6, ARI 0.76, clusters
26

---

D.7.5 Llama2-13b
highsschoo__ hatery
curhopee history, high_school history
high matthemtstt
professeral_medicn
security_stun studies
professioral_ie
professieral accouuttingg
professioral accounding
college_come car_pptmppp____e science, college_ methematics
abstract algabra
abstract, aigebra
high_schoo_ statistics high school computersstteee science
high computer science, school collage computer science
slementary, mathemattics
mix of subjects
medical genetics, human, sexuality, anatomy
college, biology, high, school chemissry
mix
electrical, engineeriig
econametrics
pomal_Jogic
formal logic
econometrics
Iearning
security
conceptual, physics
virology
management
nuginn
glebal, facts
world_reigions
miscellaneous
socielogy
us_foreigr_palicyy sociology
intemationa_lev
marketing, public relations
professioralpsyyteee pppphhserrppyyyyooyyyy.. high_scheo__syyhhlgg
us_forein__eeicyy
philosophy
jurisprudence
high
gahhschholl_gaae
geagrrahyy
high Btthee__gvemmmet_nnng
momi__issputes
moral, disputes
prehistory
iellccl_s
astronamy
ausiness_etiics
clinical krowledge
anatomy
Figure A15: Llama2-13b, 5-shot, Z=1.6, ARI 0.64, 59 clusters
27

---

D.7.6 Llama2-70b
Termallogii
Lllgeba
hight higtt_xchooL_orke Lorhstrroridhhhsttrrrry
high, saaht_shhooleerooo Leropeen, history. sighoschool,,ao histoor
professional accountinng
professional_mee mediciee
"igh_xxhoo__pyyeee
high school biology, college_ biology, haaph_shhol_chhiiaee
college, chemistry, high htap_sccoooochhhe chemistry
elementary, mathematicss
prehistory
astronomy
electrical Lnggineriig
college, phhyiics
marketting
rorld_religions
moral_disputes
jurisprudence
philosophy
high geogrepyyy
us_foreign_polte
international Law
business_eeie ethics
nutrition
clinical knowledge, college, medicine, college, biology
professional_ posycoologpppoog
high Bagh__choo__pseee psychology
socislogy
genetics,
sexuality
human_agine
high nigh_school mecrooconnomics, high sihhoo_miireecooooiic
managementt
computer securtty
conceptuel_physic
logical_fellacie
logical.fellecce
security: studies
professional law
high, schhol school_statsttes statistics
econometrics
machine_
college, mathematics
college_mathhmattic
college computer science, high school, computer science
high school mathematiccs
abstract algebbn
econometrics
Figure A16: Llama2-70b, 5-shot, Z=1.6, ARI 0.80, 56 clusters
28

---

E Robustness with respect to Z
E.1 ARI profiles with letters
Liama
Ulema
2
2 0.5
2 0.5
Setata
3
03
24
2
04
2
06
Daret
05
0.
64
0.0
Figure A17: The Adjusted Rand Index (ART) between clusters and the MMLU answer letters (test set) varying free parameter Z. Llama 8b (left, center) and Llama 3 8b fine-tuned (right). We replicated the experiment depicted in 5, monitoring the changes in ARI as a function of Z. The results indicate that the metric is robust to small variations in the free parameter, with all three configurations showing that a change of Z 2.1 corresponds to an approximate change of 0.3 the resulting ARI.
Llama 8b
Liama 3 8b
Laama 3 8b
1.50
0.50
0
0 o
0.11
2
0.45
3.30
031
0.3
010
Layer
Layer
Layer
Figure A18: The Adjusted Rand Index (ARI) between clusters and the MMLU subjects (test set) varying free parameter Z. Llama 8b (left, center) and Llama 8b fine-tuned (right). We replicated the experiment depicted in 5, monitoring the changes in ARI function of Z.
29

---

E.2 Dendrograms robustness with respect to Z in Llama3-8b (0-shot, 5-shot, fine-tuned).
The probability landscape of the O-shot and fine-tuned models remains largely mixed even when Z equals zero, and we consider all the local density maxima genuine probability modes. For O-shot, the number of clusters increases from 43 70, but we still have two large clusters containing mixture of 22 and 10 subjects (see Fig. A A19). Similarly, the fine-tuned model has several clusters with more than 7 subjects (see Fig. A24). We also notice that the probability landscape of 5-shot representation is more robust to variations of Z. Indeed, by increasing Z from 1.6 to 4, the number of density peaks slightly decreases from 77 to 57, and the ARI with the subjects remains around 0.8 (see Fig. A23). In contrast, for the 0-shot and fine-tuned representations, it drops to 0 as most of the probability peaks are merged into large-scale structures (Figs. A21, A26).
E.2.1 Llama3-8h Oshot 7=0
hin_schooLmathhmic.
asstroct_aggee
electfica Lenginzenngg
word, relgions
misca aneous
giabalfacts, Hih_schoo__commre
oncepuunsayyscss por_schodd_eemmttt astrnommy, collge_phyyits
conceptual prysics, computer securiy
high_sshhooochmmey shhnmssyy
collegee phys es, achol phyacc callage chembuzy
school chemitry, collega, chamstry
higtsshhoo_pyyscc
callege
socolegr: protess psychology. tuman secuaity
high govemmett and pollics
hamen, aging
school psychdoor, collage tedoggy arafessional paychulagy, high school blalogy mactine pretiscrry sipht geograpny high biskgy
mix el susjects
anatomy
veology
dinical
mix subjects
callage bintogy, high school biotogy
allej__camputerrddee business othics, college_ mwt.emattsssss.... high_schoo cop_schho_omuuuerecccccao sdeene
busdness ethics
professionel medicine
compurer_sccure business marketing
marcoting paydhhor
mix of susjects
management, publkrreations jar jarsprudence, mechine_eemmig
marcating
us_foreinn.ollic
wew.seeyyyyueeeeeee public_resttons
prilsseshy. jertssrudencs, moral disputes
moral_diguuts phicooophy, foral logic
Lngh_sdhooLmmeeeeee Liteneeemmncs sigh_schedl necrocconomiss
school macreeconomiss school crseconanics, moril disputes
secur compuier
computer warning
economeiz
high_school statistes
high.xchos_.om computer, sclence, lege, computer cerpppte_scceese science
professional lew
asychology
professional accouning
prolassional accountnn
lngical (allsces
logical (ellcce
secorty_stadin
high hietery hight school eurppen hatery, high, sthool world histery
Liohrdd Merrt_h ust Nitooy, Miphuschool Jisttry
"gghsschoo_eirppees Nistory
forme
foma Jogic
machine_Jeaming
morsi snenvrtes
asstroct_algean
Figure A19: Llama3-8b, 0-shot, Z=0, ARI 0.28, 70 clusters
30

---

E.2.2 Llama3-8b Oshot Z=1.6
High achod styass ege physts calage. dwnirry fin screet cxratty
tuiresseethirs cel egn, remputer orienss. tusness ethirs of egr, matheredsss hap, shovl computer
mic segeese
economatrcs
High achod micrssonsm tight school macrasammmia, ecoromeint
36 siveett
mil of sungeet
mik soteect
pii beashy Lateprusenee, fomal
logical fullac
pryenelyyy
opgees
mix vl 23 sult
socidlogy:
schhs govenmert _aiiii poliies
concustuz wwccocc alty,
astronomy
ieess onai, mediciee
fomall Juy
protess enal Jos, prrkeeccal paychelogy
Nigh schoo hetery, right schod, surppan, Nettry, righ schos world Nitrey
te enal porcunt
securty atacha
femal
machine Jraming
Figure A20: Llama3-8b, 0-shot, Z=1.6, ARI = 0.15, 34 clusters
31

---

E.2.3 Llama3-8b Oshot Z-4
highschhrd
nix of suljects
Hot_ocheoLphyyic collgg_physkcs collot_cheméétry. High cheniisy
fscst httpycccho_mmmmeeerroooo
snjeett
formal agie
scmnometrics
nofessoonsmmmedd
professional
formal Igic
machine Jearning
acanarics
Figure A21: Llama3-8b, 0-shot, Z=4, ARI 0.02, 13 clusters
32

---

E.2.4 Llama3-8b 5shot Z=0
astronomy
aztronomy
abstract algebra
algebra
abstract"algeber
eloctrical ehginnerring
electrical angineering
markeling
business ethics
profossional psychology
proressional Btcpssorn__pyycchoee poychology
high school scchooo_aycchhcopp
schhoschool psychology
college biology
college biology
college blology
clinical knowledge
clinicalknowwldde
anatontyy
anatomy
high_schaol biology
high-school biology
college medicine
college medičine
callege mediaine, clinical knowlcdge
medi genetics
nutrition
nutrition
professional medicine
professional
philosophy
jurisprudence
philosoply
high Righ_school_ggwee
wtw.bollggemmmett..oom politics
prehistory
moral disputes
maral disputes
human sexuality
school microcconomics
high school microeconomiss
high school macroeconamics
high Schhos macrooccoommss
geography
sigh"schhooooeegna
ancous
global facts
ğlobaifactts
foreign wef.eeignpollte
public relations
sociology
intematiooal
Govemmentt
politics
high Schhol gevemment and_poltics
high school european history
high School european history
high
school
histery
high school world history
high School world worldhhssorr
securfty
ctudies
management
formal logic
Sormalloggr
high chemistry, callege. chemistry
school chohiiiccccmmtrrry. calloge_cheeeeeeeee
college, chemistry
college_physks physks
conceptual physks
conceptuul physks
con.ceptual
high school mathematics
high School mathematicc
callege mathematics
School
high.school.cccooooooocccocoooco chemiityy
high school physics
high school statisicss
high School statstics
school
calloge computor Science
high School
ecorämetriis
econometrics
computer security
professional accounting
proressional accounting
professional
mathematirs
elementary mathamatics
logiral fallacies
logical fallacies
religions
Figure A22: Llama3-8b, 5-shot, Z-0, ARI 0.72, 110 clusters
33

---

E.2.5 Llama3-8b 5shot Z-4
aestractaageerra
abstrect
actronomy
unginwering
ecorometrics
ngh_schoossiattttis htw.schooooomgeeeer.oooo computer_seenne
collge pow__ooerueerseeeee.
elementary, mathematics
formal logi
protessional
concephunl_hyyicc
collage physics
scgho phrykis
college: chemistry, high school chenistry
ngn_schossmatremetes
college mathomatics
Nigh_schoo_ collgr_biitegy. college_mediciee
clnical knowieoge, college, medicine
arotomy
protessional, medicine
nutrition
Nigh_schol Nistory, High, Luugeenn history
sociology
public_reatter, wttp:schholppveemeee.ccmmm polttics, jurispnddenct
Internationa
sigh_schoo__ govermeett politics
humar humar_saruatt sorr_sttttttyy
shilesophy
norst_disputes, jurisprudence
philosophy
high school geography
high school geography
human, Leging
globel_fects
miscelanneus
marketiing
lotessional_psychoooe
viriloy
managemment
machinc_Jeaming
computer security
securty_stutles
logica]
logical fallaciss
religooss
Figure A23: Llama3-8h 5-shot, Z=0, ARI 0.81, 57 clusters
34

---

E.2.6 Llama3-8b ft Z=0
subjects
mix of7 subjects
collega_@ology. hgg_scchol__ioogeoooooo medical _aneetcss virology
mix subjects
mix of5 subyects
globa
humon_aging
ot5
vircloy
hog_sch-ool_Deelgy college_biooe biology, global facts, prehistory
clinical knowledge
cliiicl trowledge, anatomy, college medicin, medical medical_genttic genatits
pretistory
High_schoolggegpee geography. school sehdssch__bbltogy biology, prehistory
professional mediciee
figh school govemment, and politic, businss ethics
socialogy, sexualty
foreign policy
us.trrelo_pooliy
intarnational law
worid_relglons, jurppuudere
worid_relgions
#ogh_cthool.micecccomniic micrreeconomics, high school macrocconomiss
Noghschool
High, sogh_school_ao
merromeocneessss,, ecanometrirs, high schel mirroecoromics
htt__choooLmmaceec hot. macreccoommss marceting, public_ rebltiose maragement, infeessioaaaerrouiingg maragement, public relations, business, attics ma rheling markzting,
alposchology
professioral peofesinaal_accounnng accrounting
profassioral accounting
professioral_jes
logical [alllciss juisprudercce
righ school, goverment politics, righ school, geograshy
philosophy, moral, disputes
jurisorudence, security
moral, disiues
logical fallacies
sohocchoooo us history. school histery. high curspear history
usshholl__s_hssttnn high, scchooo workks history
studics
secunty_studice
ptt_echhol_chemisry college_chereee chemistry
chemistry, high chamistry, astronomy
mix ef subjects
dlcctrcal_erginnerrig
vecricalleeginnerri
concestual_phytzz
computerr security
High_school_sie statsttss
calegp_compptte_eeerrcee
htw..cchrooooommoereee.eooooe computer_returty wttpeeegmtttteetettttt..
econometrics
leaming
slementar_mmee mathematics, college_mahemmmlleee mathematiss, high sichoon mathenatics, electrical er ergneering
school mathematics, college mathemaiks
amatics, college,
abstract, algebra
abstradi alpebra
logic
tormalJogic
Figure A24: Llama3-8b, finetuned, 7=0, ARI = 0.35, 69 clusters
35

---

E.2.7 Llama3-8b ft Z=1.6
oul, emngreen
sudjects
no 10 sudlects
commmbryyyy.. of chombity,
subjects
Merwer plontrr
reittlinn jeann
temel
arofiiadarnt low
figh hichoo in ursconem sconematiio, High uchool Mtatica maodsorrreoeeeeeee
sinting porlesieaa ao cottinn perfeciiontt coathol
eumplyy mone pdpputer
ctudien, Autegralince
ofl Srllades, umporuteone
Siladia
uichot pww.gmoeonnnnnoo.o politp
ontoratioaa Jou, foiieen palley
din ot browlorge, enstorty collepe wwc.rreeteeeee eeckee
andessionnd anoiltin
Figure A25: Llama3-8b, finetuned, Z=1.6, ARI 0.23, 34 clusters
36

---

E.2.8 Llama3-8b ft Z=4
subjezt
subjects
oncostooLalyiic
mixe subjett
wbstract slgerra
profesiional
machine
sigebra
dlggy, sumannseuuul
Jooweegee Maatom,, wwl.gge mediche, goneto
professional chn
forma lgk:
econometrics
Figure A26: Llama3-8b, finetuned, Z=4, ARI = 0.05, clusters
37

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification:
The abstract summarizes the main results of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes].
Justification: The main limitations are described in a dedicated paragraph of the Section: Discussion and conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA].
38

---

Justification:
The paper does not present new theoretical contributions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced
statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes].
Justification: A general description the experiments are included in the main text, detailed explanations are then provided in the appendices as needed. For the computation of ID, clustering and the production of the dendograms code and we specify the package used for the main methods.
provide link to github repo for our
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
39

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes].
Justification: For all experiments, we utilized publicly available datasets and open source package for our primary methods. The chosen hyperparameters are detailed in the appendices plot captions. Additionally, we provide link to an anonymized GitHub repository for further transparency.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
healbmmmssinoooicyyy...... for more details. BaggliccguiiessCdeSSubmii
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https: SCt//eeueeeeieaeeeeeee //nips. ctips.cc/public/ wttpc//ppwpppguueeeeeeeeeeeeeeee / www.uubmmsssinpe.. issoiicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes].
Justification: We provide high-level overview of our experiments in the main text, with detailed explanations included the appendices as needed. Our ID and clustering com- putation data and code are available, accompanied by readme file that documents their usage.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes].
Justification: In the appendix we analyzed how the computed metrics varied slightly with small change the free parameters, demonstrating the robustness of our results.
Guidelines:
The answer NA means that the paper does not include experiments.
40

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eπror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes].
Justification: Details of the computational resources used are provided in the reproducibility paragraph at the end of method section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: :
Answer: [Yes]
Justification: We have reviewed the Code of Ethics, and we confirm that our paper conforms it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
41

---

The current work interprets the geometry of hidden representation in LLMs. The authors do not foresee any substantial societal impact.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (c.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not pose any such risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We credit the creators or original owners of assets (e.g., code, data, models) citing their work.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
42

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
we do not introduce any new asset.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No research with human subjects has been involved in the creation of this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Same as above: no research with human subjects has been involved in the creation of this paper.
43

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
44


Reviews:
[{"review_id": "nNJ0EkszMu", "reviewer": "Reviewer_P3MS", "summary": "In this paper, the authors analyze the probability landscapes of the hidden representations of LLMs when they perform in-context learning (ICL) and supervised fine-tuning (SFT) using the MMLU question answering benchmark, and the Llama-3, Llama-2, and Mistral-7B LLMs are used for this purpose. The Advanced Density Peaks (ADP) algorithm is used for this analysis. The experiments conducted recover fingerprints that showcase the distinctions between ICL and SFT, specifically that representations in the earlier layers of the network are aligned with the subjects (topic of the question) during ICL and that later layers are better aligned with the final answers for SFT. The authors conduct a number of different experiments analyzing the probability landscape and draw interesting conclusions from the results.", "questions": "Please see the weaknesses listed above as each of those is also framed as a question to the authors.", "limitations": "The authors discuss limitations sufficiently. Note that owing to the strengths and weaknesses mentioned above, the paper's contributions currently constitute a technically solid, moderate-to-high impact work, which forms the basis for my score.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper has a number of strengths, namely the extensive set of experiments that lead to interesting findings. The findings are useful to the community-- with the main takeaway being the two-phased nature of the hidden layer representations' probability landscapes.\n- Additionally, I appreciate the findings obtained that demonstrate the fingerprints obtained for ICL and SFT in the early and later layers respectively. More specifically, the results imply that during ICL the representations in the earlier layers of the network are more aligned with the subject partitions and that during SFT representations of the later layers are better aligned with the answer/label distributions of questions. This finding showcases the distinctions between ICL and SFT from the perspective of the representation landscape.\n- The experiments on the hierarchical organization of the cluster density peaks via linkage clustering with respect to the subject relationships show how information in the internal model layers are organized.", "weaknesses": "- My main issue with this work is that the experimental analysis is only conducted on the MMLU benchmark. This detrimentally restricts the impact of the work, especially as the authors draw general conclusions. Is it possible to extend some of this experimental analysis to other benchmarks, perhaps just even to other question answering based benchmarks?\n- Can the authors provide more details on the average linkage experiments that demonstrate the hierarchical organization of the cluster density peaks? For instance, how do the authors map the subjects at a granular level to each of the dendograms/leaves? Overall, I believe the subsection \"Hierarchical organization of density peaks\" can be further improved in terms of writing and readability for future readers.\n- There are a number of typos throughout the paper that need to be corrected. For example, the authors write \"LLama\" in a number of places although it should read \"Llama\" (e.g. line 86), \"assignation\" -> \"assignment\" on line 140, among others. These can be fixed in the revision.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your constructive and thoughtful comments and for the time spent on our manuscript. We will address all the main concerns below:\n\n> *Is it possible to extend some of this experimental analysis to other benchmarks?*\n\nIn our work, we analyzed MMLU because it includes a wide variety of topics (57), with clear semantic relationships between them and a rich set of samples (more than 100) for each topic. These requirements are hard to find in other datasets.\n\nWe agree with you that an additional benchmark would strengthen our claims. Therefore following your suggestion, we performed a new experiment on a second dataset constructed from TheoremQA [1], ScibenchQA [2], Stemez [3], and RACE [4]. This dataset contains roughly 6700 examples, not included in MMLU, 10 subjects from STEM topics [1-3], and middle school reading comprehension task [4], with at least 300 examples per subject. We keep 4 choices for each answer. The 0 shot, 5 shot, and fine-tuned accuracies in Llama 3 8b are 55%, 57%, and 58%, respectively. \nWe report the analysis in Fig. 1 and 2 of the attached PDF. In Fig. 1-left, we see that the intrinsic dimension profiles have a peak around layers 15/16, the *same layers as in MMLU* (Fig. 2-left main paper). This peak in ID signals the transition between the two phases described in the paper. \nBefore layer 17,  few-shot models encode better information about the subjects (ARI with subjects above 0.8). Between layers 3 and 7, the peaks in 5 shot layers reflect the semantic similarity of the subjects (see the dendrograms for layer 5 in Fig. 2 of the attached PDF).\n\nFine-tuning instead changes the representations after layer 17, where ARI with the answers for the is $\\sim 0.15$, higher than that of the 5 shot and 0 shot models. The absolute value is lower than that reported in the main paper (Fig. 5-left) because the fine-tuned accuracy reached on the STEM subjects in this dataset is lower.\n\nOverall, the results are consistent with those shown in the paper for MMLU. \n\n> *Can the authors provide more details on the average linkage experiments [...]? How do the authors map the subjects [...] to each of the dendograms/leaves?*\n\nThe two key points to understanding our procedure are:\n1. Each peak is associated with a single subject; \n2. The saddle points between the density peaks (subjects) allow us to cluster them hierarchically using Weighted Pair Group Method with Arithmetic Mean (WPGMA)..\n\nWe elaborate on these points below. \n\n1. The clusters are homogeneous in the layers where ARI is highest (layers 4 to 6, Fig. 3-left of the main manuscript). In these layers in Llama 3 8b, 51 out of 77 clusters are pure, and in 70 out of 77, 90% of the points belong to the same subject. This high purity allows us to map one or two subjects to one density peak, which becomes a leaf of the dendrogram (see also Fig. A9).\n\n2. Average linkage is a strategy to do hierarchical clustering on the density peaks. Hierarchical clustering requires a notion of **dissimilarity** between the peaks/subjects [5].  The Advanced Density Peaks (ADP) defines the dissimilarity between a pair of peaks $\\alpha$ and $\\beta$ as [6]: \\\n$S_{\\alpha, \\beta} = \\log \\rho_{max} - \\log \\rho_{\\alpha,\\beta}$ \\\nwhere $\\rho_{max}$ is the density of the highest peak, and $\\rho^{\\alpha, \\beta}$ is  saddle point density between $\\alpha$ and $\\beta$. \nIntuitively, the higher is $\\log \\rho^{\\alpha, \\beta}$, the smaller is $S_{\\alpha, \\beta}$. The ADP algorithm considers two peaks similar if they are connected through a high-density saddle point. \\\nThe peaks are then linked hierarchically, starting from the pair with the lowest dissimilarity according to $S_{\\alpha, \\beta}$.  Once two peaks $\\alpha$, $\\beta$ have been merged into a peak \\gamma, the saddle point density between the peak representing their union and the rest of the peaks ${\\delta_i}$ is determined according to the average linkage strategy (WPGMA) [7] (see line. 222) namely: \\\n$ \\log \\rho^{\\gamma, \\delta_i}= \\dfrac{\\log \\rho^{\\alpha, \\delta_i}+\\log \\rho^{\\beta, \\delta_i}}{2} $ \\\nAfter the update of $S$ we repeat the procedure until we reach a single cluster which is the root of the dendrogram.\n\nWe will include the description of the WPGMA methodology in the method section at the end of the section devoted to the ADP algorithm.\n\n> *There are a number of typos.*\n\nThank you. we will fix them in the camera-ready version. \n\nWe hope that our additional experiment addresses your main request and that we have satisfactorily clarified your second concern. If so, we hope you can consider raising your score. \nWe'll be more than happy to give further replies to any remaining doubts.  \n\n\n[1] Chen et. al, Theoremqa: A theorem-driven question answering dataset \\\n[2] Wang et. al, Scibench: Evaluating college-level scientific problem-solving abilities of large language models. \\\n[3] https://stemez.com/subjects \\\n[4] Lai et al, RACE: Large-scale ReAding Comprehension Dataset From Examinations \\\n[5] Hastie et al., Elements of statistical learning. \\\n[6] D’Errico et al., Automatic topography of high-dimensional data sets by non-parametric density peak clustering. \\\n[7] Sokal, A statistical method for evaluation systematic relationship.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yis7KlKGms", "reviewer": "Reviewer_mxnJ", "summary": "The paper explores the internal dynamics of LLMs when subjected to ICL and SFT strategies. Despite achieving comparable outcomes, the paper reveals that these methods result in distinct representation landscapes within the model. ICL fosters a hierarchical organization of representations based on semantic content in the initial layers, whereas SFT yields a more diffuse and semantically mixed landscape. In the latter stages of the model, fine-tuning leads to representations with sharper probability modes that better capture answer identities compared to the less defined peaks produced by ICL.", "questions": "1. Do few-shot examples significantly impact the analysis results? If so, in what aspects are these impacts manifested?\n2. Is te method applicable to studying tasks that generate longer texts?\n3. What do you think the specific practical applications of the findings might be? For example, improving ICL or SFT methods, or perhaps combining the two in some way?", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper takes a novel approach by analyzing the geometric properties of the hidden representations in LLMs, offering new insights into the workings of ICL and SFT. The use of the Advanced Density Peaks algorithm and the systematic investigation of data transformations across hidden layers demonstrate a insightful methodological framework.", "weaknesses": "See Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time spent reading our manuscript, for your general appreciation of our work, and for your questions. We will answer point by point below.\n\n> *1. Do few-shot examples significantly impact the analysis results? If so, in what aspects are these impacts manifested?*\n\nThe order and identity of the shots can slightly affect few-shot accuracy, it has a smaller impact on the clusters found by the ADP algorithm and, more generally, on our results. In Table A1 in the appendix of the main paper, we show that the 5-shot accuracy can change from 66.7 to 66.2, changing the few-shot order in Llama 3 8b. \n\nWe performed an additional experiment to test the impact of the order and identity of the shots on the ARI with the subjects. Figure 3 in the attached pdf shows the results of this test. We measured the standard deviation of the ARI with 5 runs with different shot orders by shuffling the MMLU dev set (red profile) and the identity of the few shots sapling from the union of the dev and validation set. On average, the standard deviation is about $\\sim0.02$. This means that our findings are robust to changes in the prompt since the number of clusters and their internal composition remain consistent across different runs.\n\n> *2. Is the method applicable to studying tasks that generate longer texts?*\n\nThe method can be applied to sequences with more than one token. However, since the clustering and intrinsic dimension estimation rely on the Euclidean distances between samples, it is important to map the generated sequence to a common embedding. \nOne strategy is to average over the sequence tokens as done, for instance, by Valeriani et al. [1] who measure the intrinsic dimension of biological sequences of varying length by averaging over the sequence axis.\n\n> *3. What do you think the specific practical applications of the findings might be? For example, improving ICL or SFT methods, or perhaps combining the two in some way?*\n\nOur findings can help improve strategies for LoRA finetuning that adapt the rank for LoRA matrices. Several studies [2-4] attempt to adapt the ranks of the LoRA matrices based on various notions of matrix ‘importance’ or relevance to downstream tasks.\nOur analysis of the similarity between finetuned layers and the pre-trained layers (see Fig. 6 and lines 275 284) shows that late layers are most significant for finetuning. This indicates that the layers that change the most should have higher ranks. This approach would naturally prevent the early layers from being modified by fine-tuning and could combine the benefits of ICL and FT, as you correctly suggest. \nWe are actively working on these aspects in our lab and will include a brief discussion of the practical application of our study in the revised version of the manuscript. \n\n[1] Valeriani et al., The geometry of hidden representation in Large Transformer Models \\\n[2] Zhang et al., AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning.\\\n[3] Valipour et al., DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation.\\\n[4] Ding et al., Sparse Low-rank Adaptation of Pre-trained Language Models", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tlhpm1S2VJ", "reviewer": "Reviewer_UJWa", "summary": "This work focuses on understanding supervised finie-tuning (denoted SFT, where parameter weights are changed) and in-context learning (denoted ICL, where weights remain frozen and the adaptivity is done through a few shots in prompts), in the context of modern LLM. In doing so, the authors propose to apply the Advanced Density Peaks algorithm (ADP) to layers in the transformers that are used in a question answering task. The authors presented a wide range of plots and discussion, observing that both few-shot learning and supervised fine-tuning shows complementary behavior where earlier layers and late layers focus on different levels of information.", "questions": "N/A", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This manuscript could gain importance due to the recent advances in modern LLMs and the widely adapted practice of supervised fine-tuning and  in-context learning. The authors make great efforts showing a wide range of analysis (although in a less clearly organized manner).", "weaknesses": "I have a few major concerns regarding the manuscript in its current form.\n\nThis manuscript has several issues in clarity. First, the description of the ADA algorithm, which is the cornerstone of this work, remains largely hard to follow, and I have to refer to the original ADA paper to figure out what the authors propose. Also in the results section, a wide range of observations are discussed, but such discussions are largely scattered and I found it hard to gain insight from them.\n\nThere are also issues with the originality and significance of this manuscript. This is partially due to the issues in clarity mentioned above and also the lack of significant contributions. Technically, this manuscript adapts ADA algorithms in understanding the representation, in each layer of the transformer. This is technically not a significant contribution. This, alone, is not a concern, since many great analysis works use simple methods to reveal important insight. However, in this manuscript’s case, the insights are less clear and a reader may find it difficult to draw inspiration from the discussions.\n\nIn an ideal form, the manuscript could contain clear, organized discussion revealing interesting insights. A non-exhausting topic that can be studied includes semantics, fine-tuning dynamics, theoretical bounds, practical rule-of-thumb, or the lack thereof in modern models. I would encourage the authors to consider such aspects.\n\nNit-picketing:\nTable A1: last columns for Llama-3-70b have formatting issues (should uniformly use percentage or decimal).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time spent reading our manuscript and the concerns you raised, which will help improve our study and make its exposition clearer. We will address all your points below.\n\n> *This manuscript has several issues in clarity. [...]*\n\nWe agree with you that the description of the ADP algorithm can be improved, and we plan to do so in the revised version of the paper. We plan to modify the ADP exposition as follows:\n1. To facilitate understanding of the algorithm's logical flow, we will split the description of the ADP algorithm into four paragraphs: “intrinsic dimension estimation”, “density estimation,” “density-based clustering,” and “hierarchical clustering of the peaks with WPGMA.” \n2. We will focus on the algorithm's assumptions and give an intuitive picture, adding a figure to represent the geometrical intuition behind the ADP. \n3. We will include all the technical details in the appendix (we already did that in Sec. B) to avoid making the exposition in the main paper too long and difficult to follow.\n\n>*discussions are largely scattered [...]*\n\nThe main messages of our work, as explicitly stated in the bullet points of the introduction, are the following:\n1. Across the hidden layers of LLMs, the probability landscape changes in a two-phased manner, separated by a sharp transition in the middle of the network (1st bullet point, and Sec 3.1)\n2. When LLMs are adapted to solve a downstream task, ICL and SFT exhibit complementary behavior with respect to these two phases: \\\n2a. before the transition, ICL induces an interpretable organization of the density peaks, which is consistent with the semantic information of the data (2nd bullet point, and Sec 3.2) \\\n2b. SFT modified the geometry of the probability density after the transition where few (3rd bullet point, and Sec 3.3)\n\nIn the revised version of the manuscript, we will make this correspondence clearer by reducing the bullet points from 4 to 3 so that they can explicitly match each section of the results and assign more explicit titles to Sec. 3.2 and 3.3. \n\n> *There are also issues with the originality [...]*\n\nWe believe that our study has a number of methodological and substantial novel contributions: \\\nWhile most of the previous works compare ICL and finetuning in terms of performance on downstream tasks (see e.g. [1] and refs. therein), we propose a very different analysis of the changes in the geometry of the hidden layers induced by these two paradigms;\\\nWhile most previous studies trained linear probes to discover semantic information encoded in hidden layers (see [2-3] among others), we propose and use a fully unsupervised nonparametric approach to study the probability landscape in LLM, finding interpretable semantic relations between the modes of distribution;\\\nFinally, we validate our claims on state-of-the-art models like Llama-3, released only a few months ago (April 2024), including a study of the 70B parameter models, which are often overlooked. \n\n>*Topics that can be studied include semantics, fine-tuning dynamics, theoretical bounds, practical rule-of-thumb [...].*\n\nWe appreciate your perspective on this matter. However, we'd like to respectfully clarify that our work does indeed address the *semantics* and *fine-tuning dynamics* in modern models. In addition, these analyses allow us to give clear-cut practical rules-of-thumb.\n\n*Semantics*:\\\nIn Section 3.2, we explore the extent to which the contextual embeddings encode a global property of prompts, specifically the topic of the question. We show that in ICL, the nearby density peaks contain the embeddings of \nsentences denoting similar topics (e.g. math, physics, ...), and their global hierarchical organization closely mirrors the semantic similarity of the subjects (see lines 182-185, 212-214, 229-236, and Figure 4). \nIn line 44 and bullet points 1, 2, and 4, we explicitly state that our work also addresses semantic analysis.\n\n*Fine-tuning dynamics*:\\\nFigure 6 and lines 275-284 are dedicated to the analysis of the fine-tuning dynamics through the lens of the representation similarity between hidden layers before and after fine-tuning. \nThis analysis shows that most training steps are spent chaining the last layers rather than the early ones (see lines 282-284). \nIn Figure 3 of the accompanying PDF, we show that as the late layers become more dissimilar to their initial state, their ARI with the letters correspondingly increases. This change results from the onset of a few homogeneous density peaks on the answer label. \\\nThis analysis addresses another point raised by the reviewer, namely, the discussion of practical rules of thumb.\n\n*Practical rule-of-thumb*:\\\nSince fine-tuning affects the second half of the network, our analysis suggests a possible strategy to adjust the ranks of the LoRA matrices. Several studies [4-6] attempt to adapt the ranks of the LoRA matrices based on different notions of matrix ‘importance.’ Our study of the dynamics of similarity (Fig. 6) gives a principled measure of importance: the layers that change the most should have higher ranks. We are currently investigating these aspects in our lab and will include a brief discussion of the practical application of our study in the revised version of the manuscript.\n\nWe thank you again for pointing out clarity issues since these will allow us to improve our manuscript. We hope our responses can alleviate your concerns regarding the clarity of our results. If so, we hope you will consider raising your score. \nWe’ll be happy to reply to any further questions.\n\n[1] Mosbach et al., Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation\\\n[2] Conneau et al, What you can cram into a single vector: Probing sentence embeddings for linguistic properties\\\n[3] Hewitt and Liang, Designing and interpreting probes with control tasks.\\\n[4] Zhang et al., AdaLoRA\\\n[5] Valipour et al., DyLoRA\\\n[6] Ding et al., Sparse Low-rank Adaptation of Pre-trained Language Models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the chair and the senior area chair for their time spent reviewing our work.

*Summary of our contribution*:\
In this study, we compare fine-tuning (FT) and in-context learning (ICL) in LLMs by
analyzing the evolution of the probability density in the hidden layers of state-of-the-art LLMs (Llama-3 8B/70B, Llama-2 7B/13B/70B, Mistral 7B) as they solve a challenging question answering task (MMLU). \
We find that the evolution of the probability density in LLMs occurs in two phases, separated by a sharp modification of the representation landscape in the middle layers. In early layers, ICL promotes a remarkable organization of the probability modes that are close to each other according to the semantic similarity of the MMLU subjects. In contrast, FT  affects the probability landscape of late layers, leaving that of the early layers mostly unchanged. 

*Summary of reviewers’ praise and concerns*:\
Reviewer **mxnJ** appreciated the novelty of our approach, which analyzes *the geometric properties of the hidden representations in LLMs, offering new insights into the workings of ICL and SFT*. For Reviewer **P3MS** our analysis of the representation landscape *has a number of strengths, namely the extensive set of experiments that lead to interesting findings useful to the community*. **UJWa** acknowledges the *extensive set of experiments* carried out in our work.\
The reviewers' main concerns were:
1. Lack of clarity in some parts of our exposition (**UJWa** and **P3MS**).
2. The need for an additional dataset to strengthen our claims (**P3MS**).

*Summary of our reply to reviewers' concerns*:
1. We have addressed reviewer **P3MS**'s primary concern with a new experiment on a *second dataset*, which confirms and strengthens our findings (see attached pdf Figures 1 and 2). 
2. To improve the clarity of our exposition, we will: \
2a. Organize the description of the Advanced Density Peaks algorithm into four distinct paragraphs (see reply to **UJWa**), including an explicit description of the average linkage algorithm we use (see reply to **P3MS**).\
2b. We will make explicit the match between bullet points and the section of the Results (see reply to **UJWa**) and polish the paragraph about the Hierarchical organization of the density peaks, as suggested by Reviewer **P3MS**.

We hope that our additional experiment and our replies to the reviewers' comments address their concerns. We are open to clarifying any remaining questions.

Meta-review: This work studies the geometry of the representations of large language models, comparing the behaviors observed early and late in the models in two settings, for fine-tuned models and for in-context learning. The authors perform a solid empirical study and produce some quite interesting findings (e.g., ICL produces certain hierarchical behaviors that resemble the relationships between semantic concepts).

Reviewers are generally positive, and I agree with them. This is a nice work that tries to better understand LLM functionality for popular approaches rather than focusing on new approaches or techniques, something that is definitely needed in this space.
Decision: Accept (poster)