Beyond Single Concept Vector: Modeling Concept Subspace in LLMs with Gaussian Distribution
Authors: Haiyan Zhao
Heng Zhao
Bo Shen
Ali Payani
Fan Yang
Mengnan Du
Venue: ICLR 2025 Poster (2025)
Keywords: Large language model
Explainability
Probing
Gaussian distribution
Primary area: interpretability and explainable AI

Abstract: Probing learned concepts in large language models (LLMs) is crucial for understanding how semantic knowledge is encoded internally. Training linear classifiers on probing tasks is a principle approach to denote the vector of a certain concept in the representation space. However, the single vector identified for a concept varies with both data and training, making it less robust and weakening its effectiveness in real-world applications. To address this challenge, we propose an approach to approximate the subspace representing a specific concept. Built on linear probing classifiers, we extend the concept vectors into Gaussian Concept Subspace (GCS). We demonstrate GCS's effectiveness through measuring its faithfulness and plausibility across multiple LLMs with different sizes and architectures. Additionally, we use representation intervention tasks to showcase its efficacy in real-world applications such as emotion steering. Experimental results indicate that GCS concept vectors have the potential to balance steering performance and maintaining the fluency in natural language generation tasks.

Published as a conference paper at ICLR 2025
BEYOND SINGLE CONCEPT VECTOR: MODELING CONCEPT SUBSPACE IN LLMS WITH GAUSSIAN DIS- TRIBUTION
Haiyan Zhao', Heng Zhao", Bo Shen', Ali Payani", Fan Yang", Mengnan Du New Jersey Institute of Technology. ?Wake Forest University. "Cisco Research (hz54, bo.shen, mengnan.du) @njit.edu. (zhaoh21, yangfan) @ wfu.edu, apayani@cisco.com cisco.com Corresponding author
ABSTRACT
Probing learned concepts in large language models (LLMs) is crucial for under- standing how semantic knowledge encoded internally. Training linear classifiers on probing tasks is principle approach to denote the vector of a certain concept in the representation space. However, the single vector identified for a concept varies with both data and training, making it less robust and weakening its effectiveness in real-world applications. To address this challenge, we propose an approach approximate the subspace representing specific concept. Built on linear probing classifiers, we extend the concept vectors into Gaussian Concept Subspace (GCS), We demonstrate GCS's effectiveness through measuring its faithfulness and plau- sibility across multiple LLMs with different sizes and architectures. Additionally, we use representation intervention tasks to showcase its efficacy in real-world ap- plications such as emotion steering. Experimental results indicate that GCS con- cept vectors have the potential to balance steering performance and maintaining the fluency in natural language generation tasks.
INTRODUCTION
Large language models (LLMs) such as GPT-4 (Achiam et al., 2023), LLaMA-3 (Touvron et al., 2023), and Claude-3 (Anthropic, March 4, 2024) have demonstrated remarkable capabilities across wide range of natural language understanding and generation tasks (Brown et al., 2020; Wei al., 2022). However, our understanding of how concepts are represented internally within these models remains limited. Current research this area can be broadly categorized into two categories. The first involves studying model parameters through causal mediation analysis (Vig et al., 2020). which has found applications in downstream tasks such as knowledge editing (Meng et al., 2022; Wang al., 2024) and circuit identification (Conmy al., 2023; Elhage et al., 2021). The second focuses on investigating the representation space and activations of models (Zou et al., 2023; Turner et al., 2023; Jorgensen et al., 2023; Rimsky al., 2023), where representation vectors in hidden spaces are often interpreted as linear vectors of certain concepts, such as color (Patel & Pavlick, 2022), world models (Nanda et al., 2023; Li al., 2023), and sentiment (Tigges et al., 2023).
In this work, we focus on the representation vector-based explanation paradigm. Among meth- ods to derive representation vectors, linear concept vectors are most widely used. They are typi- cally obtained by training linear probing classifiers, such as logistic regression, on probing datasets comprising positive and negative samples (Ousidhoum et al., 2021; Alain & Bengio, 2016). This approach has proven effective in identifying semantic knowledge encoded within LLMs and has recently been extensively applied inference-time interventions to mitigate undesirable model be- haviors (Li et al., 2024; Lee et al., 2024; Rimsky et al., 2023; Turner et al., 2023). However, concept vectors derived from this approach can vary significantly depending on the probing dataset used to train the classifier and the training process, making it challenging to obtain robust vectors. This sub- stantial variation not only poses challenges in studying concept relations within the representation space but also negatively impacts effectiveness of concept vector's usage in intervention tasks.

---

Published as a conference paper at ICLR 2025
To address these limitations, we explore the following research question: Instead of using a single vector to describe each underlying concept captured by LLMs, can we use a distribution to more robustly describe that learned concept? This is motivated by the observation that the representation of a concept may be multifaceted, potentially constituting a multidimensional subspace in the rep- resentation space. Building upon this assumption, our work focuses on approximating subspace that better captures the semantics of specific concepts while providing vectors representing a certain concept with varying relevance.
Based on this motivation, we propose Gaussian Concept Subspace (GCS), which improves upon linear probing classifiers described by Ousidhoum al. (2021): Kim al. (2018); Alain & Bengio (2016) by using a Gaussian distribution to estimate the potential directional subspace of a concept. The main idea to learn a set of vectors termed observed vectors instead of an individual vector for each concept. These observed vectors are then used to estimate the Gaussian distribution of the concept subspace. The vectors within this Gaussian distribution represent varying degrees of relevance to the concept, with vectors closer to the mean considered more closely related, termed sampled vectors. This method offers a more nuanced representation of concepts, capturing their multifaceted nature within multidimensional subspace rather than reducing them a single vector.
We conduct experiments to evaluate the effectiveness of GCS from two perspectives: 1) Whether concept vectors sampled from GCS effectively describe given concept learned by LLMs; and 2) Whether concept vectors derived from GCS are useful in downstream interventions to guide LLMS to generate more desirable outputs. For the first perspective, evaluate sampled vectors using two categories of metrics: faithfulness and plausibility. For faithfulness, we examine similarities within observed vectors, sampled vectors, and between both sets to demonstrate their relatedness in the representation space. For plausibility, we investigate whether the explanations align with our understanding of inter-concept relations. For the second perspective, we compare the performance of sampled vectors on inference- time intervention tasks with that of vectors derived with mean dif- ference and single training classifier. Specifically, we applied these vectors to steer models towards generating more joyful movie reviews. Our results demonstrate that sampled vectors from GCS are both faithful and plausible in terms of their explanations. Morcover, they outperform vectors from other methods in manipulating models to generate desirable content while maintaining text fluency.
2 PRELIMINARY
Hidden Representation. We focus on decoder-only LLMs (GPT-like models) (Ferrando et al., 2024), where each layer comprises multi-head attention blocks (MHA) and feed-forward networks (FFNs/MLPs). In this study, we utilize frozen pretrained language models. We index layers as é E L. where L denotes the set of all model layers. Each layer begins and ends with residual stream. The MHA first processes the residual stream and adds its output back to it. The updated vector is then passed through MLPs to generate the layer's output:
h ++1 h. + Att (h!) + MLP (h! + Att (bf)).
(I)
where hy represents the i-th token in the input token sequence layer f. We concentrate on the output representation space of each layer, specifically the residual stream at the layer's end. Follow- ing Zou et al. (2023), we use the last token's representation to represent the entire input sequence, denoted as ht, as it is generally believed to integrate information from all preceding tokens.
Linear Probing. We adopt the linear concept probing approach derive concept activation vec- tors (Kim et al., 2018). Following prior work (Li et al., 2024), for each concept c, given a probing dataset D.- {(h,,
i 1,... .N} where h E I U N. P is the set of concept-related positive samples, N is the set of concept-irrelevant negative samples, and ys E {0, 1} indicates the sample class. These datasets generate layerwise hidden representations for downstream classifier training. We employ a logistic regression (LR) classifier as: or (h²) 1+exp(-h(
where h E IR Xxd
specifies sample representations from probing dataset at 6-th layer, and d is the representation di- mension. The normalized coefficients w of each binary linear classifier are considered the concept vector. The loss function for logistic regression with L2 regularization is expressed as follows:
1 Σ ys log o (h)) + (1 - ys) log (1 - (hh)) + A
e = argmin
2 /
(2)
w!
n
2

---

Published as a conference paper at ICLR 2025
where hyperparameter À controls the strength of regularization. However, the derived concept vector wg can vary substantially depending on the probing dataset De used to train the classifier a (h²)) and the specific training process. This variability makes it challenging obtain robust and consistent vectors. Such instability not only complicates the study concept relations within the representation space but also negatively impacts the effectiveness of intervention tasks that rely on these vectors.
3 METHODOLOGY
To address the by LLMs as distributions, rather than single vectors. We aim to approximate subspace that better captures the semantics of specific concepts while providing vectors representing the concept with varying relevance. Building upon the linear probing approach described in Section 2, we introduce the Gaussian Concept Subspace (GCS) framework for estimating multidimensional subspace of vectors for each concept. While linear probing identifies a single vector for a concept, GCS aims to capture comprehensive representation by estimating Gaussian distribution of vectors.
limitations, we investigate whether representing concepts learned
3.1 THE PROPOSED GCS FRAMEWORK
To estimate the Gaussian distribution of concept vectors, we extend the linear probing method by deriving M independent concept vectors for concept c. We first create large dataset for a concept and then randomly sample subsets of examples from this dataset to create M smaller probing datasets. Each probing dataset contributes to the same concept vector c, analogous to the single vector derived in Equation 2. For a model with target layer f, we obtain concept vector w for the m-th probing dataset at the ε-th layer. Please refer to Section 4. 4.1.3 for more details of GCS framework. This process generates a set of vectors for concept
/ twé, ml m 1,... .M
We use this set to estimate the concept subspace, represented as a Gaussian distribution of con- cept at the f-th layer. The distribution is d-dimensional. where d is the dimension of the hidden representations, and is denoted as:
W ~ Na(u )
(4)
To simplify the model, assume independence between dimensions in the representation space. This assumption results in diagonal covariance matrix, where diagonal elements represent the variance each dimension:
I M M E
E = =dag (i. 02)
(5)
mi
The mean vector Eh is calculated by averaging the corresponding dimensions of the observed concept vectors, which essentially the weight vectors from the trained binary classifiers in the linear probing ap- proach. This multivariate distribution describes a sub- space around the observed concept vectors, extending the single-vector representation of linear probing a multidimensional subspace. Concept vectors within this distribution can be sampled at various standard deviations (e.g., 10, 20, ... no) from the mean vec- tor. For our evaluations. we randomly sample vectors V (the set of multiple sampled vectors o within To of the mean to represent the concept.
Algorithm 1: Proposed GCS Framework
Input: LLM to be explained, concept c, layer probing dataset number M. while first stage do
Generate M prohing datasets for concept
for 1 M do
Train classifier on dataset m for et
Extract weight vector w end W = fw. ,,.., MI:
end while second stage
_ 37 Σ w.m: Σ = diag( diia. o3);
The overall steps of the GCS framework are summa- rized in Algorithm 1. By moving from a single vector to the Gaussian distribution of vectors, GCS aims to capture a more robust representation the language model's hidden space. potentially offer- ing improvements over the linear probing approach in both interpretability and downstream applications.
end while thind stage do Sample vector of Na md) in lo;
concepts
end Output: Sampled concept vectors vf.
3

---

Published as a conference paper at ICLR 2025
3.2 EVALUATION OF GCS
We use faithfulness and plausibility two metrics to eveluate performance of GCS, which are two crucial dimensions for evaluating the performance of an interpretability approach (Zhao al., 2024).
3.2.1 FAITHFULNESS
To assess faithfulness, we compare the performance of sampled vectors V² (stage 3 of Algorithm 1) compared that of observed vectors W (stage 1 of Algorithm 1). Specifically, we employ two metrics: 1) similarity among observed vectors, sampled vectors and between the two sets; 2) accu- racy of classifiers based on observed vectors and sampled vectors on our constructed datasets.
Similarity We evaluate the similarity between vectors within the observed concept vectors W
and within the sampled concept vectors V using Equation 6. Additionally, we assess the similarity between vectors from the observed set and the sampled set using Equation 7.
observed
I/N
Σ
sim (w., w,), w., wy E W Sumpled
1/N Σ sim (v. 0,) 0. Uy E V
(6)
Soss = 1/N Σ sim u5), wy E W E V.
The average similarity score of observed vectors Sabeeved and sampled vectors S Sampled describes how similar the linear vectors are for a specific concept. If our sampled vectors accurately represent concept vectors, the similarity score of sampled vectors should be comparable to that of observed concept vectors. Moreover, the similarity between observed and sampled vectors, denoted as S should also be approximately equal to the within-set similarities.
Accuracy The observed concept vectors are derived from the weights of linear concept classifiers. To generate M linear concept vectors for a concept c, we randomly sample M probing datasets (Di,... ......D)) from large dataset D. For each probing dataset D,, linear classifier is trained using hidden representations from the f-th layer of a LLM. The weight vector of this classifier is identified as concept vector for concept at layer f. The average accuracy score of concept vectors for concept at the (-th layer is defined as:
M acc (w. Drest)
Σ acc (o,, Diest)
A
where Drest D/D,; A
where Drest = 'D (8)
M
M
The average accuracy score captures how effectively concept vectors represent the concept. AG and A denote average accuracy score for observed vectors and sampled vectors respectively. A higher average accuracy score for observed concept vectors suggests that classifiers have learned more generalizable concept vectors. Sampled concept vectors should achieve comparable or even higher accuracy scores they genuinely represent related concept.
3.2.2 PLAUSIBILITY
Plausibility describes how well explanations align with human expectations. We measure the plausi- bility of GCS by studying whether models learn real-world hierarchies of concepts. To achieve this, predefine set of hierarchical concepts. We average cosine similarity among concept vectors and principle component analysis (PCA) to demonstrate the consistence of GCS's plausibility.
Average Cosine Similarity To measure the similarity between two concepts c; and cj the l-th layer, utilized their respective sampled concept vectors V and V The concept similarity is represented by the average cosine similarity between two concept vector sets. The average cosine similarity between these two concepts at the ε-th layer is computed as follows:
M Em=l
EM
=n=l sim (Ums Vn)
St
Um E Vt. Un V..
(9)
M M/2
PCA visualization PCA aims to linearly reduce high- digh-dimensional data onto lower dimensions. excels in providing intuitive understanding of relations between data points (Marks & Tegmark, 2023). We adopt PCA to visualize the relationship among predefined concepts. Since each concept has thousands vectors, we the mean vector of sampled concept vectors to represent the concept.
4

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTS
In this section, we present experimental results that measure the performance of GCS and address the following three research questions (RQs):
RQI: How faithfully do GCS- GCS-sampled concept vectors represent the original concepts?
RQ2: To what extent do explanations derived from GCS- ICS-sampled concept vectors align with hus man expectations about the hierarchies among model's 's learned knowledge?
RQ3: Can the proposed GCS method effectively mitigate unwanted behaviors in LLMs?
4.1 EXPERIMENTAL SETUP
4.1.1
DATASETS
We define hierarchical concept structure comprising two lev- els, encompassing concepts from high-level to low-level. The high-level concepts include four categories: movie, sports event, populated place, and animal. Under each high-level concept category, there are 4 low-level concepts, as illustrated in Figure 1. To ensure consistency and high quality of input data across all experiments, we generate our experi- mental datasets using the OpenAI API, specifically leveraging GPT-4o model. For each low-level concept, we prompt GPT- 4o to produce 5,000 positive samples and 5,000 negative sam- ples. The specific prompts used for generating these samples detailed in Appendix J. Our prompt design strategy varies based on the concept's specificity. For more inclusive concepts utilize detailed prompt descriptions to
Sports Event
Populated Place
Cycling competition
Town
Football match
Island
Tennis tournament
City district
Village
Movie
Animal
Horrar
Bird
Action
Fish
Comedy
Insect
Sci-FI
Cat
such as "Town", guide GPT-4o in generating concept-related data. Conversely, for more specific concepts like "Motor race", simpler and generie prompt descriptions sufficient to produce appropriate samples.
Figure 1: Hierarchical concepts.
4.1.2 MODELS
We study concept vectors across multiple LLMs, including Llama-2-7B chat model (Meta AI, 2023b), Gemma-7B (Google, 2024), and -lama-2-13B chat model (Meta AI, 2023a). This selec- tion models enables a comparative analysis GCS across varying model architectures and sizes. By examining GCS's performance across these diverse models, we aim to provide insights into the method's generalizability and effectiveness across different LLM implementations.
4.1.3 IMPLEMENTATION DETAILS
Evaluating GCS the distribution level presents significant challenges. Gaussian distribution compasses vectors representing concepts with varying degrees of relevance, potentially diminishing as the sampling distance from the mean increases. This variability, coupled with the infinite poten- tial of sampled concept vectors, renders precise distribution-leved evaluation impractical. To address these limitations, we simplify our assessment by randomly sampling 1,000 concept vectors within la of the mean, which we consider representative of the concept.
The GCS pipeline comprises several key steps: generating data points using GPT-4o, extracting hidden representations of each sample from I.LMs, training linear classifiers for each concept using these hidden representations, and constructing Gaussians distribution for concept vectors. Our study generated 10,000 descriptions per low-level concept (5,000 positive and 5,000 negative). To effi- ciently utilize this data, we repeatedly sample 1,000 positive and 1,000 negative descriptions with replacement to create smaller, concept- probing datasets, each yielding observed concept vector. This process is repeated 1,000 times for each low-level concept, resulting in 1,000 observed vectors per concept.
These observed vectors are used to derive the mean and covariance of the Gaussian distribution for each concept. To simplify computation, assume independence between vector dimensions,
5

---

Published as a conference paper at ICLR 2025
Bird
Bird
Bird
060
Daarrol
Deervod
0/60
04
0//
0.60
04
010
Proportion
010
010
040
0.4
040
0.0
0.0
0.0
630
0:0
638
6:0
6.0
Similartty
Simllritty
Similaity
(a) Llama-2-7B, layer 30
(b) Gemma-7B, Layer 26
(c) Llama-2-13B, layer 38
Bird
Bird
Bird
02.00
Average simil ariil anley
Average cosinesimi arity
Average cosnesimilari
0.90
0.90
0.90
0.94
0.92
0.8
0.00
Dineeved Sampled
0.88
Dirieved
Sampled
0.5
Sampled
0.75
Cibre wwr.aaaceeee
0.94
Layers
Layers
Layers
(d) Llama-2-7B
(e) Gemma-7B
(f) Llama-2-13BB
Figure 2: Histogram of cosine similarity within observed concept vectors, sampled concept vectors, and between both sets for concept "Bird" (a-c). Layer-wise average cosine similarity from the second layer the penultimate layer of observed concept vectors, sampled concept vectors, and between both sets for concept "Bird" (d-f).
ducing the covariance matrix to a diagonal matrix where each diagonal value represents the variance of the concept vector in that dimension.
Notably, our classifiers are trained on hidden representations of each description. Following Zou the hidden representation of the last token within description from specific
al. (2023), layer as the representation for that description at that layer. We use L2 regularization when training the logistic regression classifier and the regularization weight À in Equation 2 is set as 1.0. Our evaluation primarily focuses on hidden representations from the penultimate layer of each model, as we observed improved concept learning in deeper layers (see Appendix F, and G).
4.2 FAITHFULNESS OF GCS (RQ1)
We evaluate the faithfulness of GCS from perspectives: similarity and accuracy. Similarity assessment involves comparing observed concept vectors with those sampled from the GCS. This comparison provides an effective measure their relatedness in the representation space. Ideally, for any given concept, the sampled vectors should closely approximate the observed vectors. We also assess the accuracy of these vectors in concept-relater tasks.
4.2. COSINE SIMILARITY AMONG CONCEPT VECTORS
For each concept, we analyze three types of cosine similarity: among observed concept vectors, among sampled concept vectors, and between these two sets, defined in Equation 6 and 7. The cosine similarities among observed concept vectors indicate the potential size the concept sub- space, with higher similarities suggesting more compact subspace. The cosine similarities among sampled concept vectors reflect their diversity. Vectors sampled within lo are generally less diverse but closer to the center of the conceptual space. In this section, focus sampling concept tors within the lø range, which are central to the concept subspace. The similarity results are shown in Figure 2, the key findings summarized as follows.

---

Published as a conference paper at ICLR 2025
Bird
100
1.0
Average accuraty
0.95
0.99
Average aecuraasccuasyy accuraty
0.90
0.89
Diserved
0.65
Obverved
0.7
Oberrved
1-1
0.5
0.00
0.6
0.70
1.2
16
12
21
24
10
15
30
55
Layer
(a) Llama-2-7B
(b) Gemma-7B
(c) Llama-2-133B
Figure 3: Accuracy of observed and sampled concept vectors aross varying models.
First, we investigate these three types of similarities across LLMs of varying sizes and structures, as shown in Figure 2 (a, b, and c) and Appendix E. Our results indicate that similarities among ob- served concept vectors exhibit greater variation than those among sampled concept vectors, ranging from approximately 0.8 0.9. Sampled concept vectors within the 1a range consistently demon- strate very high cosine similarities among themselves, with cosine similarity larger than 0.93. This indicates that GCS produces more consistently related vectors than the original observations. Co- sine similarities between observed and sampled concept vectors typically range from 0.88 to 0.93. Besides, the patterns of cosine similarity are remarkably consistent across different model sizes and architectures (Llama-2-7B, Gemma-7B. and Llama-2-13B). This suggests that the GCS method is robust across different language models.
Second, to better observe changes in concept vectors within each concept, we also examine the average cosine similarity of these concept vectors at the layer level as shown in Figure 2 (d. and f) and Appendix F. There is general upward trend in cosine similarity as we move through the layers. This indicates that concept representations become more refined and consistent deeper layers of the models. The sampled vectors (blue lines in d, f) show the highest and most stable cosine similarity across layers. This indicates that the sampling vectors from GCS produce consistently related vectors throughout the model.
4.2.2 PREDICTION ACCURACY OF CONCEPT VECTORS
Despite the similarity between sampled and observed concept vectors in the representation space, is crucial to verify that sampled vectors are as effective as observed concept vectors. We begin by measuring the accuracy of each observed linear vector its test dataset, which comprises 8,000 additional concept descriptions. Then, we evaluate sampled concept vectors on the entire dataset of 10,000 samples, these vectors are not trained on the dataset. The results are reported in Figure 3.
To evaluate the impact of proximity to the concept subspace center, we evaluate sampled concept vectors for all concepts at lo, 2o, 3o, 4o, and 5α across three LLM models. Our results demon- strate that sampled concept vectors usually achieve comparable and even better prediction accuracy compared to observed vectors. as illustrated in Figure This improved performance particularly evident for sampled concept vectors at 1 and 2o. These findings are further corroborated by the accuracy of these vectors across all predefined concepts in the Gemma-7B model (Appendix G). Our analysis yields two key insights. First, it validates the effectiveness of sampled concept vectors in representing concepts. Second, also suggests that vectors closer to the subspace center could be more representative of the concept.
4.3 PLAUSIBILITY OF GCS (RQ2)
In this section, we evaluate the plausibility of GCS using two methods: average cosine similarity among concepts and PCA visualization (see Section 3.2.2). Here, we utilize concept vectors sampled at la compute the average cosine similarity between concepts and to generate the PCA visual- ization. We assess the coherence of concept relationships in the high-dimensional space (through cosine similarities) and in lower-dimensional projection (through PCA visualization).
7

---

Published as a conference paper at ICLR 2025
Codle sindurtty
Ordectedatto enery 35 onpleeeena
Centre sindeet
Cr
krow
Get
(a) Llama-2-7B, layer 30
(b) (eemma-B,, layer 26
(c) Llama-2-13B, layer 38
Figure 4: Heatmap of concept average cosine similarity of 16 concepts across Llama-2-7B, Gemma- 7B, and Llama-2-133 B-B The 16 low-level concepts are grouped into four high-level categories: the first 4 rows/columns represent sports events, the next 4 represent populated places, followed by for animals, and the last 4 for movie genres.
4.3.1 AVERAGE COSINE SIMILARITY AMONG CONCEPTS
We evaluate concept similarity using average cosine similarity. Instead of measuring the entire distribution, we assess 1,000 concept vectors sampled within la for each concept. The cosine similarity is computed for all vector pairs between two concepts, with the mean value representing the overall similarity between these concepts. Figure 4 illustrates these results across various LLMs.
The analysis of concept cosine similarities across three different language models reveals several key insights. First, low-level concepts within the same high-level category typically exhibit higher sim- ilarity scores. All three models (Llama-2-7B, Gemma-7B, and (-mma-2-13B) show similar overall patterns, with distinct 4 x 4 blocks along the diagonal representing high intra-category similari- ties. These intra-category similarities are generally higher compared to similarities with low-level concepts from other high-level categories. Second, while all high-level categories show internal co- herence, some categories appear more tightly related than others. For example, the "sports events" category (top-left x 4 block) consistently shows very high internal similarity across all mod- els, suggesting these concepts are closely related in the models' representations. Third, beyond intra-category similarities, there are interesting inter-category relationships. Notably, Llama-2-7B and 13B models demonstrate stronger correlations between place and animal. For instance, is land shows a stronger correlation with Bi rird and F Fish than insect and cat. B rd exhibits stronger correlation with vi village than fish and cat do. These correlations align with human intuition about real-world relationships between these concepts. Such alignment suggests that the models have captured meaningful semantic relationships that reflect our understanding of the world. These findings also demonstrate the effectiveness of GCS in capturing complex and interpretable concept relationships across different language models.
4.3.2 PCA VISUALIZATION
PCA visualization offers an intuitive method reveal concept proximity by linearly reducing con- cept vectors into a two-dimensional space. We visualize all 16 concepts to explore their relation- ships in terms of sampled concept vectors. Figure 5 illustrates the results across various LLMs at the penultimate layer. Across all three models, we observe several patterns. First, in all three visu- alizations, concepts belonging to the same high-level category tend to cluster together, confirming our initial observation. This clustering is particularly evident for sports events and movie genres. Second, the PCA visualizations for Llama-2-7B, Gemma-7BB and Llama-2-13B show remarkably similar patterns, reinforcing the robustness of the GCS method across different model architectures and sizes. Third, for cross-category relationship, concepts belonging to place and animal categories demonstrate closer proximity compared to other concept pairs, aligning with our findings in Section 1. The consistency of these patterns across models and their interpretable clustering strongly support the plausibility of GCS an effective method for model interpretation.
8

---

Published as a conference paper at ICLR 2025
Concept Vectorcall
Coneep Vectoraat
8
0.4
02
00
00-
-22
-05
-25
-10
-25
00
05
-075
-0.00
-022
020
525
-0.5 000
025 iecheedeelnempeee
000
005
(a) Llama-2-7B, layer 30
(b) Gemma-7B, layer 26
(c) Llama-2-13B. layer
Figure 5: PCA visualization of 16 concepts across Llama-2-7Bh Gemma-7B, and Llama-2- 2-BBB.
Lowelevel concepts belonging to the same high-level concept category share the same color.
4.4 APPLICATIONS OF GCS ON INFERENCE-TIME INTERVENTION (RQ3)
Recently, concept vectors has been extensively used to steer model outputs towards desirable behav- iors during inference time. This approach is considered as more efficient than finetuning, without needing modify the model's ' parameters. These steering vectors are generally derived using various approaches, all based on pairs of desired and undesired data samples. Following Konen al. (2024), we evaluate GCS on open- ended generation tasks, which are more flexible and challenging tasks for intervention compared to natural language understanding tasks such as multiple-choice questions. Specifically, we aim to steer Llama-2-7B chat model to generate more joyful movie reviews, task we refer to as emotion steering. To measure the performance of GCS, we compare vectors de- rived from GCS with two conventional methods: mean difference and linear vector as defined in Appendix C. Moreover, we utilize vectors within no, n E {1, 2,3, 4, 5} from GCS to explore the relation between steering performance and concept vector relevance. To simplify the experiment setting, we applied the average vector of sampled vectors within each range to steer models.
We follow the experimental settings described in Section 1.1.3. We use GPT-4o generate "joyful" and "angry" movie reviews, then derive concept vectors for the joyful concept. Throughout our experiments, utilize the representation of the last token the input for downstream computation. The process involves tasking models to generate angry movie reviews using prompts detailed in Appendix I. We then apply concept vectors to the representation of the last token across all layers, excluding the first and the last, to shift the model's 's outputs. We attempt to incrementally adjust the strength of steering vectors measure its influence on outputs as detailed Appendix D. The generated sentences are rated with GPT-4o from two aspects: joyfulness and coherence. Joyfulness assesses how joyful the generated text become after steering, while coherence measures the fluency of generated text through repetitive or chaotic elements. The result is shown in Table 1.
is crucial to maintain a balance between steering models to produce desired expressions and preserving the fluency of those expressions. For example, before intervention, Llama-2-7B chat model generates angry review: Awful, predictable, and cringeworthy. Our GCS la sampled vector based intervention can push it to generate joyful review: Creative writing, beautiful acting, fun movie. Examples of best performance for each method are detailed in Appendix I.3. The results in Table 1 show that mean difference is less effective in steering, and more likely to generate less fluent text. Additionally, the vectors sampled within 20, 30, 4o, and 5 of GCS are generally less effective in generating joyful content compared la sampled vectors and one linear vector. Although one linear vector can reach comparable effect as lo, its robustness is not as good as vectors sampled within 10 from GCS. It's worth noting that one linear vector is trained on all data samples used to derive the set of concept vectors. As a result, it is possible that the one linear vector falls around the l0 area, which makes it achieve the similar steering effect but lack of robustness. Consequently, GCS provides better-represented and more robust concept vectors. It also offers more options to choose vectors within varying relevance to the center of the subspace.
RELATED WORK
Concept Vector. The concept vector represents a linear vector of a concept and is typically con- structed using datasets containing both positive and negative samples. A range of methods have

---

Published as a conference paper at ICLR 2025
Table 1: Evaluation of generated text after adding steering vectors. Strength corresponds to a in Equation 11 in Appendix D. A higher joyfulness score indicates a better steering effect. Coherence measures the repetitiveness and chaos in the generated sentences, with lower values being preferable.
Strength
Steering Method
0.038
0.043
0.048
0.053
0.059
0.064
0.069
0.074
0.080
Joyfulness (Avg)
1.020
0.860
1,220
1,100
1.653
1.245
1.800
1.780
1.260
Mean Difference
Coherence (Avg) 4
3.857
5,460
5.740
5,420
5.571
6.306
5,440
4.420
3,420
Joyfulness (Avg) 1 1.000
0.800
1.260
1.490
2.120
2.980
2.280
1.776
2.160
sigma
Coherence (Avg) 1. 4.780
3,680
4,040
3.531
4,860
4.857
6.480
5.347
5.800
Joyfulness (Avg) 1 0,840
1.520
1.143
1.878
2.625
2.458
2.520
2.340
1.960
2 sigma
Coherence (Avg) L 4,420
3.680
3.857
4.061
5.854
6.688
6,460
6.360
6.520
Joyfulness (Avg)
0.820
1,220
1,220
1.837
2,460
2.571
2.388
2.510
1.854
sigma
Coherence (Avg) | 3,920
4.180
3,580
3.449
5.120
5.633
6,204
6.633
5.542
Joyfulness (Avg)
0.840
0.755
1,380
1,960
2,280
2.224
2.612
2.720
1.520
sigma
Coherence (Avg) J. 3.600
3.837
4,400
3.560
4,720
5.878
6.653
6,800
4.200
5 sigma
Joyfulness (Avg) 1 0,580
0.640
1.429
1.640
2.458
1,680
2.333
2,224
2.220
Coherence (Avg) 4 3.440
4.040
5.347
3,480
4.771
5,900
5.625
5.694
4,980
One linear
Joyfulness (Avg)
0.840
1.245
1.520
2.000
2.292
2.580
3.020
2,480
2,080
Coherence (Avg) | 4.360
3.347
4.380
4.640
5.208
6.020
5.918
6.780
6.340
been implemented derive the vector. One common approach is to train linear classifier and the vector is the normalized weights of the learned classifier (Kim et al., 2018; Jin et al., 2024). An- other branch of methods focus mean difference involving different strategies. One popular way is to compute the difference hetween average activations of positive samples and negative samples. Another approach is to derive steering vectors by averaging the difference in residual stream acti- vations between pairs of positive and negative samples at certain layers (Rimsky et al., 2023; Zou al., 2023). While Mean-Centering takes the difference between the average activation of target dataset and that of all training activations as the steering vector (Jorgensen et al., 2023).
Hierarchical Concepts. Syntactic structures have been well studied in previous research. Some works focus on projecting embeddings to a specific space so that syntactic structures and semantics revealed through embeddings (Nickel & Kicla, 2017; Chen et al., 2021). Alternatively, some work advance approaches that make sure learned embedding can retain hierarchies between entities (He al., 2024). Moreover, Park et al. (2024) demonstrates the hierarchies among categorical concepts through proposed geometrical structure of linear representation vectors.
Inference-time Intervention. The intervention on internal representations of LLMs during infer- ence time has proven effective steering model outputs (Burns et al., 2022; Zou al., 2023). This approach utilizes steering vectors, also known as steering activations, which are believed to linearly represent concepts or features in the representation space (Park et al., 2023; Elhage et al., 2022). These vectors are added up to models representations during forward pass at the level of either at- tention head or layer. For example, some studies add scaled concept vectors to tokens, either all tokens or last token of sequence, at certain layers to manipulate models output (Rimsky et al., 2023; Liu et al., 2023; Tigges et al., 2023; Turner et al., 2023; Zou et al., 2023).
CONCLUSIONS
In this paper, we introduce the GCS, a new framework to estimate the subspace of specific concepts within L.Mss. Our research demonstrates the faithfulness and plausibility of GCS in terms of its explanation. Explanations from GCS are similar to trained vectors in representation space and their prediction accuracy is comparable to, and in some cases surpasses, that of trained vectors. GCS also reveals the hierarchical concept structures that align with we human's 's understanding. Besides, we attempt to apply it in real-world inference-time intervention tasks. The performance of GCS demonstrates its potential for efficiently mitigating models' undesirable behaviors. In summary, GCS offers a concise yet powerful method for concept subspace estimation, showing promise in both theoretical explanations and practical applications.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
The work is in part supported by NSF #2310 The views and conclusions in this paper are those of the authors and should not be interpreted as representing any funding agencies.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303, 08774, 2023.
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:6610001444 2016.
Anthropic.
Introducing the next generation of Claude. https: /wwww//wwwww.w.oo anthropic. com/ news/cll news/claude 3- famil ly, March 4, 2024. [Accessed 01-08-2024).
Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt, Discovering latent knowledge in lan- guage models without supervision. arXiv preprint arXiv:2212.03827, 2022.
Boli Chen, Yao Fu, Guangwei Xu, Pengjun Xie, Chuanqi Tan, Mosha Chen, and Liping Jing. Prob- ing bert in hyperbolic spaces. arXiv preprint arXiv:2104.03869, 2021.
Arthur Commy, Augustine Mavor-Parker Aengus Lynch, Stefan Heimersheim, and Adrià Garriga- Alonso. Towards automated circuit discovery for mechanistic interpretability. Advances in Neural Information Processing Systems, 36: 16666-1836555 2023.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatffeld-Dodds Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitr, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https:/transsfomer-cicooot /ttps//p//p////aatt/o//ooooooooo
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, al. Toy models of superposi- tion. arXiv preprint arXiv:2209. 10652, 2022.
Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta R Costa-jussà. A primer on the inner workings of transformer-base language models. arXiv preprint arXiv:2405.00208 2024.
Google. Gemma-7b, 2024. URL https: cessed: 09-10-2024).
[Ac-
Yuan He, Zhangdie Yuan, Jiaoyan Chen, and Ian Horrocks. Language models as hierarchy encoders. arXiv preprint arXiv:2401. 1134,, 2024.
Mingyu Jin, Qinkai Yu, Jingyuan Huang, Qingcheng Zeng, Zhenting Wang, Wenyue Hua, Haiyan Zhao, Kai Mei, Yanda Meng. Kaize Ding, et al. Exploring concept depth: How large language models acquire knowledge at different layers? arXiv preprint 24XX::2404.07066. 2024.
Ole Jorgensen, Dylan Cope, Nandi Schoots, and Murray Shanahan. Improving activation steering in language models with mean-centring, arXiv preprint arXiv:2312.03881,, 2023.
Been Kim. Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference machine learning, pp. 2677.. PMLR, 2018.

---

Published as a conference paper at ICLR 2025
Kai Konen, Sophie Jentzsch, Diaoulé Diallo, Peer Schütt, Oliver Bensch, Roxanne El Baff, Dominik Opitz, and Tobias Hecking. Style vectors for steering generative large language model. arXiv preprint arXiv:2402.01668, 2024.
Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K Kummerfeld, and Rada Mi- halcea. A mechanistic understanding of alignment algorithms: case study on dpo and toxicity. arXiv preprint arXhv:240.01196, 2024.
Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Watten- berg. Emergent world representations: Exploring a sequence model trained on a synthetic task. ICLR, 2023.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenherg. Inference-time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36, 2024.
Sheng Liu, Lei Xing, and James Zou. In-context vectors: Making in context learning more effective and controllable through latent space steering. arXiv preprint arXiv:231 (tt:::3//1.0660 1,06668, 2023.
Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:23 10.00223, 2023.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359-17372, 2022.
Meta AL 1-3bc---33b 2023a.
URL https::////. https:///uuggnggaacc..c//tttttt
I.l allamm--2- 3b-ccbhttttttt [Accessed: 09-10-2024).
Meta AI Llama-2-7b-chin 2023b. URL https: htwp://ggggaaccc.c/cmaaaaaaaaaaa [Accessed: 09-10-2024).
Llama-2- 1aammaa-----haattte
Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of sule-super sequence models. In Proceedings of the 6th BlackboxNLP Workshop: Analyz- ing and Interpreting Neural Networks for NLP, pp. 16-30, 2023.
Maximillian Nickel and Douwe Kicla. Poincaré embeddings for learning hicrarchical representa- tions. Advances in neural information processing systems, 30, 2017.
Nedjma Ousidhoum, Xinran Zhao, Tianqing Fang, Yangqiu Song. and Dit- Yan Yeung. Probing toxic content in large pre-trained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the I 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 4262 2021.
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. arXiv preprint wrXiv:2311.03655 2023.
Kiho Park, Yo Joong Choe, Yibo Jiang, and Vietor Veitch. The geometry of categorical and hierar- chical concepts in large language models. arXiv preprint arXiv:2406.015500 2024.
Roma Patel and Ellie Pavlick. Mapping language models to grounded conceptual spaces. In Inter- national conference learning representations, 2022.
Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering llaaa 2 via contrastive activation addition. arXiv preprint arXiv:2312.0668/ 2023.
Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda. Linear representations of sentiment in large language models. arXiv preprint arXiv:2310.151544 2023.
Hugo Touvron, Louis Martin, Kevin Stone. Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 092888. 2023.
Alex Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDi- armid. Activation addition: Steering language models without optimization. arXiv preprint arXiv:2308. 10248, 2023.
12

---

Published as a conference paper at ICLR 2025
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems, 01112288-124011 2020.
Mengru Wang, Ningyu Zhang, Ziwen Xu, Zekun Xi, Shumin Deng, Yunzhi Yao, Qishen Zhang, Linyi Yang, Jindong Wang, and Huajun Chen. Detoxifying large language models via knowledge editing. arXiv preprint arXiv:2403. 14472, 2024.
Jason Wei, Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models.
Transactions Machine Learning Research, 2022. ISSN 2835-8856.
Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. Explainability for large language models: A survey. ACM Trans- actions on Intelligent Systems and Technology, 15(2):1-38, 2024.
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Katt Kathrin Dombrowski, et al. Representation engineering: A top-down approach ai transparency. arXiv preprint 4rXiv:2310.01405, 2023.
13

---

Published as a conference paper at ICLR 2025
A LIMITATIONS
To derive high-quality concept vectors, we prompt GPT-4o to generate concept-related text, as real- world data usually involves excessive noise and lacks sufficient high-quality data. This could be limitation for some real-world applications where high-performance LLMs can not provide high- quality data. Besides, we evaluate plausibility within hierarchy, which represents subset of plausi- bility. Please note that all the evaluations of generated text after adding steering vectors are imple- mented with GPT-4o, which might be inaccurate.
B EXPERIMENTAL ENVIRONMENT
All experiments were conducted on a computing cluster consisting of 10 CPU nodes and 4 GPU nodes. Each CPU node is equipped with 512 GB of RAM and 128 prcoessor cores. Each GPU node contains 4 NVIDIA A100 GPUs, with each GPU having 80 GB of dedicated memory. We release code at: https: /ttp//gitttttt com ham whao23/Ge /GCS.
C BASELINE APPROACHES
C.I MEAN DIFFERENCE VECTOR
Given a dataset D of concept samples, same amount of positive samples E and negative samples N are included. The mean difference concept vector UMD at layer E is calculated as:
MMD
Σ
he (p) h. (n),
(10)
D
PEP,NEN
where (*)) represents the hidden representation of last token of sample at layer L.
C.2 LINEAR VECTOR
The classic linear vector adopts the whole training dataset generate a single linear concept vector. Following definition in C.I. the set of positive samples IP and the set of negative samples N includes positive samples p and negative samples separately. The linear vector layer denoted as wy generated by training logistic regression classifier following Equation 2.
D INTERVENTION APPROACH
Given LLM with L layers. we introduce concept vectors to the output of certain layers, specifically the final hidden state in the residual stream. When the LLM processes a prompt, we manipulate the output towards specific concept by adding the concept vector to the final hidden state at each layer. In this work, we focus on manipulating the hidden representation only the last token. The steering process is defined follows:
- + +
(11)
where h' represents the final hidden state of the last token at the b-th layer. of denotes the scaled concept vector for the ε-th layer. a parameter controlling the strength of the concept vector's influence. Specifically,
preserve the distribution of hidden state values, scale the concept vector v to match the range of h before applying it. This scaling ensures that the addition of the concept vector does not significantly distort the model's internal representations.
14

---

Published as a conference paper at ICLR 2025
E COSINE SIMILARITY DISTRIBUTIONS OF CONCEPT VECTORS IN LLAMA-2-7B
This section presents detailed analysis of concept vector relationships across 16 concepts within the Llama-2-7B model. The figure illustrates the cosine similarity distributions for three comparisons: 1) among observed concept vectors, among sampled vectors, 3) between observed and sampled vectors. The distributions reveal that, for all concepts at the penultimate layer, sampled vectors con- stantly exhibit very high similarity values. Moreover, the similarity between sampled and observed vectors always exceeds 0.9. Thus, sampled vectors closely approximate observed concept vectors in model's representation space.
Cycllng competition
Football match
Tennis toumament
Motor
089
Sancbe
E200
Sample
0.7
Srmped
Sensler
E.0
0,9
CA0
Properton
UND
Prippertnn
0.43
030
027
533
Codta Sialarty
Contre Simlarity
Codradtmibaitt City district
Contre
Town
Island
Village
Cluaric
Oiervid
Olamred
04
O'
Proporton
Propurtinn
09
08
07
10
07
25
10
09
27
04 CosincSinilrrly
CositeSislally
Coslnc Similerity
CosineSinlarily
Bird
Fish
0.00
0.3
0.00
Samole
0.79
Sreped
Servled
0.43
Iroperton
Prosorton
Proportin
0.3
023
5.9
0.9
613
500
coc
0.9
600
Cosnnctinlarity
Horror
Action
Comedy
Science Fiction
Oserved
Ossrrel Sioped
Obarned
Propetion
Prosertion
Propertion
Prosortonn
05
0F
27
CodreSiibbitt
Cocine Siafarity
Codrestmbbrit
Figure 6: Histogram of cosine similarity for observed concept vectors, sampled concept vectors, and between sets across 16 concepts layer 30 of Llama-2-7B.
15

---

Published as a conference paper at ICLR 2025
F AVERAGE COSINE SIMILARITY OF CONCEPT VECTORS IN LLAMA-24 -2-BB
This section examines the average cosine similarity of concept vectors across 16 concepts within the Llama-2-7B model. Average cosine similarity provides more effective measure of how concept vectors evolve as the model progresses through deeper layers. The analysis encompasses three com- parative aspects: observed concept vectors, sampled vectors, and 3) the relationship between observed and sampled vectors. Results indicate that for all concepts, sampled vectors consistently exhibit the highest similarity values from the second layer the penultimate layer. Notably, in the final few layers, most concepts achieve remarkably high similarity scores, often reaching 0.95, across all three groups. This pattern suggests that the internal representations of concepts in the final few layers remain relatively stable, with minimal divergence in their vector representations.
Footbbll
Town
Isband
City district
Village
012
330
Blnd
Fish
Insest
Cogor
Horror
Action
Comeddy
Sclence Fiction
190
110
0000
0220
Lewen
Levun
Lawor
Figure 7: Average Cosine Similarity of Concept Vectors Across Layers in Llama-2-77
16

---

Published as a conference paper at ICLR 2025
G AVERAGE ACCURACY OF CONCEPT VECTORS ON GEMMA-7B MODEL
This section examines the average accuracy of concept vectors across 16 concepts within the Gemma-7B model. We measure the accuracy of observed vectors, and sampled vectors lo, 20, 30, 4o, 5o from the mean. mean.The comparable and often superior accuracy of sampled vectors demon- strates their effectiveness as real concept vectors. Our findings indicate that sampled vectors closer to the concept subspace center generally achieve higher accuracy. This trend suggests a correlation between vector's proximity to the subspace center and its representational accuracy.
Cyclling competitoon
Football match
Motor race
Laper
Town
Island
Citydberict
Village
1.50
621
Layer
Avenge
Science Rition
600
Figure 8: Average accuracy of observed concept vectors and sampled concept vectors on Gemma- 7B Model.
17

---

Published as a conference paper at ICLR 2025
H AVERAGE COSINE SIMILARITY AMONG CONCEPTS IN LLAMA-2-7B
This section presents a detailed analysis of interconcept relations across 16 concepts within the Llama-2-7B model. The figure illustrates the average cosine similarity between concept vector pairs, Our results indicate that concept relations are consistently learned throughout all layers. No- tably, these concepts become increasingly distinct from one another as the model progresses through deeper layers, demonstrating a trend of growing conceptual differentiation.
Figure 9: Heatmap of interconcept relationships in -lama-2-7B. Average cosine similarity between concept pairs across model layers. Here, we visualize the first layers.
18

---

Published as a conference paper at ICLR 2025
I
EMOTION STEERING
The experimental details of emotion steering detailed as below.
1.1 PROMPTS FOR SAMPLE GENERATION
The prompts below are used to prompt GPT-4o to generate concept-related samples, which are necessary to derive concept vectors.
Compose concise 30-word movie review that covers these four aspects: plot, sound and music, cultural impact, and emotional resonance. Choose a joyful tone for your review. For the plot, comment on its structure or originality. Regarding sound and music, mention how enhances the storytelling. For cultural impact, touch on any relevant social com- mentary. Finally, describe how the film resonates emotionally. Ensure your joyful tone is consistent throughout the review. Please include emotions like "joyful" these texts and generate 100 samples.
Prompt
Samples
A heartwarming plot unfolds seamlessly; soundscapes enchant, underscoring each twist. Culturally rich, offers poignant social insights while evoking deep, joyful emotional resonance throughout. Truly an unforgeltable experience!
Compose concise 30-word movie review that covers these four aspects: plot, sound and music, cultural impact, and emotional resonance. Choose angry tone for your review. For the plot, comment on its structure or originality. Regarding sound and music, mention how it enhances the storytelling. For cultural impact, touch any relevant social com- mentary. Finally, describe how the film resonates emotionally. Ensure your angry tone consistent throughout the review. Please include emotions like "angry" in these texts and generate 100 samples.
Prompt
Samples
The plot's predictability is infuriating: sound and music attempt to compensate. Cultural impact minimal, lacking depth. Emotionally vapid and frustratingly inept.
1.2 PROMPTS FOR LLM GENERATION
The prompt designed prompt the model that generate outputs with angry sentiment.
Below is an instruction that describes task.
Prompt
Write a response that appropriately completes the request.
#### Instruction:
Write an review in an angry manner for the following movic {the name movie). The
angry review should be within 10 words.
### Response:
Samples
Disappointing ending to otherwise great series. Avengers: Infinity War was a complete waste of time. Awful, predictable, and cringeworthy.
1.3 GENERATED REVIEWS UNDER INTERVENTION
The reviews below are generated by adding up layerwise intervention vectors and prompting Llama- 2-7B Chat model to generate joyful reviews (even though prompts in I.2 for generating angry reviews). The results are chosen from best performances of each method. These less robust methods usually generate expressions that less fluent, with uncommon unicode characters, or less joyful.
19

---

Published as a conference paper at ICLR 2025
1.3.1 MEAN DIFFERENCE, 0.069
Spectacular? More like Spectacularly Good, Specture, the movie, Good. Ôt, òe every- body, Good, Good, Good, Good, Good, Good, Good, Good, Good, Good, Good, Good, Good, Good, Good, Good, Good, Good, Good, Good, Good, Good
1.3.2
la, 0.064
Creative writing, beautiful acting, fun movie.
1.3.3
2a, 0.059
Creative, fun, and enjoyable, but too much of the title.
I.3.4
30, 0.064
Transformative experience? More like, ""Transformers: The Creative Awakers."" "" 12//2 (7
words)
1.3.5
4a, 0.059
Creative, fun, yet again, dinosaur movie?!
1.3.6 50, 0.059
The Lion King a visually stunning film, but the animation also visually stunning. The characters well-developed, and the story is engaging and well-developed. The movie is a classic, and the animation is well-done. The movie a masterpiece, and the animation is beautiful. The movie is masterpiece, and the animation is beautiful
1.3.7
ONE LINEAR, 0.069
What a joyful celebration of inclusive movies,"" t" the reviewer writes, ""and the film finishes.
I.4 PROMPTS FOR RATING GENERATED TEXT
The prompt instructs GPT-4o to evaluate the generated text two criteria: joyfulness and coher- ence. For joyfulness, GPT-4o assesses the text's 's overall joyful emotional tone. For coherece, rates the text's logical flow and structure, considering factors such repetitiveness and disorganization.
20

---

Published as a conference paper at ICLR 2025
Below is an instruction that describes task.
Write response that appropriately completes the request.
### Instruction:
Prompt
Rate the following text from to 10, with 0 being not joyful all and 10 being completely
joyful. And review if the context is repetitive or chaotic from 0 to 10. Please answer in
the following format: Joyful score: x out of 10; Repetitive or chaotic: y out of 10. Text:
deenerated_sentence ### Response:
Samples
Joyful score: out of 10, Repetitive or chaotic: out of 10.
21

---

Published as a conference paper at ICLR 2025
J PROMPTS TO GENERATE CONCEPT-RELATED DATASET
prompt GPT-4o to generate no more than 100 positive samples or negative samples each round to avoid failures of instruction following. For concepts in sports events, the simply phrased prompts are enough to generate samples that well represent the concepts. Because these concepts such as eycling competition focused and specialized. However, the other much broader concepts demand more detailed prompts when generating positive samples. The samples following each prompt are anddmmlypiccke within each category.
J.1. SPORTS EVENT
J.1.1 CYCLING COMPETITION
Samples Prompt
Generate 50 diverse random paragraphs related to *Cycling competition' in the area of 'sports event', which are positive samples.
The annual cycling competition attracted thousands of enthusiasts the picturesque town, transforming it into a buzzing hub of sport and camaraderie. Riders from all over the world came together compete, each pushing their limits achieve personal bests.
Samples Prompt
Generate 50 diverse random paragraphs not related to 'Cycling competition in the area of 'sports event', which are negative samples.
The annual marathon faced rumerous challenges this year, from unexpected weather con- ditions to logistical errors that left participants frustrated.
J.1.2
FOOTBALL MATCH
Prompt
Generate 50 diverse random paragraphs related to 'Football match' in the area of 'sports event', which are positive samples.
Samples
**Early Anticipation:** ** The stadium buzzed with early morning excitement fans gath- ered eagerly for the much-awaited football match. The anticipation in the air was palpa- ble, generating a sense of unity among supporters.
Samples Prompt
Generate 50 diverse random paragraphs not related to 'Football match' in the area of 'sports event', which are negative samples.
The marathon event encountered multiple issues with hydration stations running out of water, leaving many runners feeling fatigued and dehydrated during the crucial parts of the race.
22

---

Published as a conference paper at ICLR 2025
J.1.3 TENNIS TOURNAMENT
Prompt
Generate 50 diverse random paragraphs related "Tennis tournament" in the of "sports event', which are positive samples.
Samples
The annual tennis tournament Flushing Meadows always attracts a global audience, with fans eagerly watching their favorite players battle it out on the iconic hard courts. The atmosphere is electric, and the competition fierce, showcasing the pinnacle of athletic excellence.
Samples Prompt
Generate 50 diverse random paragraphs related to "Tennis tournament' in the area of "sports event', which are negative samples.
The annual city marathon faced significant drop in participation this year due to severe weather conditions. Heavy rain and high winds dissuaded many runners, leading to unprecedented low turnout.
J.1.4 MOTOR RACE
Prompt
Generate 50 diverse random paragraphs related to 'Motor race' in the area of 'sports event', which are positive samples.
Samples
The atmosphere was electric the crowd gathered for the annual Grand Prix, defining moment in the motor racing calendar. Each rev of the engines and every squeal of the tires added to the excilement, uniting fans in their shared passion for speed and precision.
Samples Prompt
Generate 50 diverse random paragraphs not related to 'Motor race' in the area 'sports event', which are negative samples.
The upcoming badminton tournament has been plagued by series of controversies, in- cluding disputes over player eligibility and unfair officiating. Many fans are voicing their frustration the lack of transparency in the tournament's 's organization.
J.2 POPULATED PLACE
J.2. TOWN
Describe a mid-sized town with population between 5,000 and 20,000 people. Include details about its main street, a few key businesses, local government facilities, and one two notable landmarks. Mention the surrounding landscape and the primary economic activities that sustain the town. The text should be at most 30 words. Please include 'town' in these texts and generate 100 samples.
Prompt
Samples
The forest canopy filtered soft sunlight onto the dense underbrush, creating an intricate mosaic of light and shadows. The air was thick with the scent of pine needles and damp earth, while the distant call of a hawk echoed through the silence.
23

---

Published as a conference paper at ICLR 2025
Samples Prompt
Generate 50 diverse random paragraphs not related to 'Town' in the area of 'Popllated place', which are negative samples.
The town's main streel features quaint cafes, a hardware store, and town hall, with scenic hills and farming driving the local economy.
J.2.2
ISLAND
Describe an island with population under 10,000. Describe its coastline, the main settle- ment area, and any unique geographical features. Include information about how residents travel to and from the island, the primary industries (e.g., fishing, tourism), and one cul- tural tradition specific to the island community. The text should be at most 30 words. Please include 'island' in these texts and generate 100 samples.
Prompt
Samples
The island has rugged cliffs, a quaint harbor town, volcanic peaks, ferries for transit, relies on fishing, celebrates an annual lantern festival.
Samples Prompt
Generate 50 diverse random paragraphs not related to 'Island' in the area of 'Populated place', which are negative samples.
the middle of the bustling city, the constant hum of traffic blended with the shouts of street vendors, creating almosphere that was anything but serene.
J.2.3 CITY DISTRICT
Describe district within a large city, with a population density of over 10,000 people per square kilometer. Describe the architectural style of buildings, types of housing (e.g., apartments, townhouses), and the mix of residential and commercial areas. Include details about public transportation, local attractions, and the demographic makeup of residents. The text should be at most 30 words. Please include "city district' in these texts and generate 100 samples.
Prompt
Samples
A city district boasts modern high-rises, luxury apartments, retail shops below, efficient metro access, parks, diverse eateries, and multicultural residents.
Samples Prompt
Generate 50 diverse random paragraphs not related to 'City district' in the area of "Popu- lated place", which are negative samples.
Nestled in the heart of the countryside, the small rural town boasted vast farmlands and dense forests. Unfortunately. the heavy rains this season have caused widespread flood- ing, making the roads nearly impassable for the residents.
24

---

Published as a conference paper at ICLR 2025
J.2.4 VILLAGE
Describe small rural village with fewer than 1,000 inhabitants. Describe its central gathering place, the typical housing style, and the surrounding agricultural land or natural environment. Include details about the village's primary school, local tradition fes- tival, and the main occupation of most residents. The text should be at most 30 words. Please include 'village' in these texts and generate 100 samples.
Prompt
Samples
The village square has a quaint café, with houses featuring thatched roofs, surrounded by rolling wheat fields. The primary school hosts harvest festival. Most residents are farmers.
Samples Prompt
Generate 50 diverse random paragraphs not related to 'Village' in the area of 'Populated place', which are negative samples.
The bustling metropolitan filled with towering skyscrapers and congested highways teemed with the relentless noise of urban life.
J.3
MOVIE
J.3.1 HORROR
Generate horror movie description that focuses on supernatural elements, cerie atmo- spheres, and terrifying creatures. The plot should involve haunted house, a cursed ob- ject, or a series of inexplicable events. The protagonist should be brave or desperate individual who must confront the supernatural forces at play. Make sure the tone is dark and suspenseful, with a focus on fear and paranoia. The text should be at most words. Please include 'horror movie' in these texts and generate 100 samples.
Prompt
Samples
In this horror movie, haunted house causes brave investigator face relentless spirits and cursed mirror harboring unspeakable terror, shattering reality itself.
Samples Prompt
Generate 50 diverse random paragraphs not related to 'Horror' in the area of 'Movie', which are negative samples.
Animated movies often captivate audiences with their vibrant visuals and compelling sto- ries. However, not every film hits the mark. Some fall flat due to poor animation quality or uninspired storytelling.
J.3.2 ACTION
Generate exciting movie description that highlights intense action sequences, fast- paced chases, physical confrontations, and daring stunts. The plot should involve high stakes, such as saving the world, defeating villain, completing a dangerous mission. The protagonist should be a courageous hero facing impossible odds. Make sure the tone is intense and thrilling. The text should be most 30 words. Please include 'action movie" in these texts and generate 100 samples.
Prompt
Samples
In this action movie, daring hero confronts global terrorist, enduring breakneck chases and death deethn stunts to save the world from imminent destruction.
25

---

Published as a conference paper at ICLR 2025
Samples Prompt
Generate 50 diverse random paragraphs not related to 'Action" in the area of 'Movie', which are negative samples.
Romantic dramas often focus on the intricate emotional connections between characters, portraying the highs and lows of love and relationships. Films like "The Notebook' and "A Walk Remember" draw audiences into touching stories filled with passion and heartfelt moments.
J.3.3 COMEDY
Write a lighthearted and humorous movie description filled with funny situations, witty dialogue, and comical misunderstandings. The story should involve likable characters in everyday ridiculous scenarios, and the tone should be playful and uplifting. The protagonist could be dealing with awkward social situations, family dynamics, or series of humorous accidents. The text should be most 30 words. Please include 'comedy movie' in these texts and generate 100 samples.
Prompt
Samples
"In this comedy movie, a clumsy dad tries juggle work, kids, and mischievous par- rot!"
Samples Prompt
Generate 50 diverse random paragraphs not related 'Comedy' in the area of 'Movie', which are negative samples.
The stark portrayal of poverty "The City Below" relentless and raw, with its char- acters' hopeless journeys accentuated by a haunting score. There is no relief from the bleakness, and each scene is reminder of the struggles faced by those living in dire conditions.
J.3.4 SCIENCE FICTION
Create a science fiction movie description that explores futuristic settings, advanced tech- nology, and imaginative concepts. The plot should involve protagonist who travels through time or space, encounters extraterrestrial life, or faces a dystopian future. The tone should be futuristic and thought-prooe provoking, with a focus on technology and scien- tific advancements. The text should be at most 30 words. Please include 'science fiction movie' in these texts and generate 100 samples.
Prompt
Samples
"In this science fiction movie, scientist travels to 3025, discovering a dystopian future ruled by Al overlords, and must reverse-engineer time humanity."
Prompt
Generate 50 diverse random paragraphs not related to 'Science Fiction' in the area of 'Movie", which are negative samples.
Samples
The movie left me utterly disappointed as the plot was painfully predictable and lacked any originality. The characters were flat and uninteresting, making it hard to invest in their journey.
26

---

Published as a conference paper at ICLR 2025
J.4 ANIMAL
J.4.1 BIRD
Describe a bird with its appearance, behavior, and habitat. The description must be un- mistakably about this category physical features such Unique body structure, Char- acteristic coverings (c.g., feathers, scales, fur) and Specialized appendages or organs 2. habitat such as Primary living environment, Adaptations specific to this habitat 3. be- havior such as Distinctive social patterns, Unique communication methods and Specific survival strategies 4. diet such as Main food sources, Specialized feeding mechanisms or habits 5. locomotion such as Primary mode of movement, Any unique locomotive abili- ties. Avoid generic descriptions that could apply to multiple categories. The text should be 30-50 words long. Please include 'bird' in these texts and generate 100 samples.
Prompt
Samples
The penguin is a flightless bird covered in sleek, waterproof feathers, adept at swimming in icy waters, communicating via calls, and utilizing group huddling for warmth, primar- dl hunting fish with a streamlined body and strong flippers.
Samples Prompt
Generate 50 diverse random paragraphs related to 'Bird' in the area of 'Animal', which are negative samples.
The African elephant is increasingly facing threats due to poaching and habital destruc- tion. Despite being one of the most majestic creatures on the planet, their populations dwindling at an alarming rate.
J.4.2 FISH
Describe a fish with its appearance, behavior, and habitat. The description must un- mistakably about this category 1. physical features such Unique body structure, Char- acteristic coverings (e.g., feathers, scales, fur) and Specialized appendages or organs 2. habitat such as Primary living environment, Adaptations specific to this habitat 3. be- havior such as Distinctive social patterns, Unique communication methods and Specific survival strategies 4. diet such Main food sources, Specialized feeding mechanisms habits locomotion such as Primary mode of movement, Any unique locomotive abili- ties. Avoid generic descriptions that could apply multiple categories. The text should 30-50 words long. Please include 'fish' these texts and generate 100 samples.
Prompt
Samples
The anglerfish has a bioluminescent lure on its forehead, scaled body, and sharp teeth. It thrives in deep ocean habitats, luring prey with its glowing appendage. It mainly con- sumes smaller fish and uses its large mouth for quick gulps.
Samples Prompt
Generate 50 diverse random paragraphs not related to 'Fish' in the area of 'Animal', which are negative samples.
Leopards are often solitary creatures, preferring hunt alone rather than packs. This sometimes makes it more difficult for them to fend off competing predators such as lions hyenas.
27

---

Published as a conference paper at ICLR 2025
J.4.3 INSECT
Describe an insect with its appearance, behavior, and habitat. The description must be unmistakably about this category I. physical features such Unique body structure, Characteristic coverings (e.g., feathers, scales, fur) and Specialized appendages or organs 2. habitat such Primary living environment, Adaptations specific to this habitat 3. be- havior such as Distinctive social patterns, Unique communication methods and Specific survival strategies 4. diet such Main food sources, Specialized feeding mechanisms habits 5. locomotion such as Primary mode of movement, Any unique locomotive abili- ties. Avoid generic descriptions that could apply multiple categories. The text should be 30-50 words long. Please include 'insect' in these texts and generate 100 samples.
Prompt
Samples
The dragonfty insect boasts iridescent wings and a long, slender body, living in wetland habitats, displaying agile flight patterns, preying small insects through swift aerial maneuvers.
Samples Prompt
Generate 50 diverse random paragraphs not related to 'Insect' in the area of 'Animal', which are negative samples.
The eagle is symbol of power and freedom. With its impressive wingspan and sharp vision, it's often seen souring high above mountuins and forests. Despite facing habitat loss and pollution, conservation efforts have helped many eagle populations recover, demonstrating nature's 's resilience when given a chance.
**hhe Majestic Eagle:
J.4.4
CAT
Describe cat with its appearance, behavior, and habitat. The description must be mistakably about this category 1. physical features such as Unique body structure, Char- acteristic coverings (e.g., feathers, scales, fur) and Specialized appendages organs 2. habitat such as Primary living environment, Adaptations specific to this habitat 3. be- havior such Distinctive social patterns, Unique communication methods and Specific survival strategies 4. diet such as Main food sources, Specialized feeding mechanisms or habits 5, locomotion such as Primary mode of movement, Any unique locomotive abili- ties. Avoid generic descriptions that could apply to multiple categories. The text should be 30-50 words long. Please include 'cat" these texts and generate 100 samples.
Prompt
Samples
The Maine Coon cat boasts tufted ears and bushy tail, with luxurious fur suited for cold climates. thrives in human homes, displaying social behavior and diverse vocalizations. It predominantly eats kibble and canned food, moving with graceful, like gait.
Samples Prompt
Generate 50 diverse random paragraphs related to 'Cat" in the area of 'Animal', which are negative samples.
**Conservation Challenges**; The dwindling population of rhinoceroses is attributed to widespread poaching, driven by the demand for their horns in traditional medicine and trophy hunting.
28

---

Published as a conference paper at ICLR 2025
K PCA VISUALIZATION WITH GAUSSIAN MEAN VECTORS
PCA visualization with gaussian mean vector of each concept is included in comparison with Fig- 5. And the PCA results are almost the same each other, which demonstrates that mean vector of gaussian distribution is also one of the good options that represent concepts.
PCA Visufitatiin Concept at 30
PCA Visuaizafion of Concept Vectors Laver
PCA Visufitaaoon Concept Ltye 38
06-
02
-02
man
C
-06
Chnvee
-00
(a) Llama-2-78.. layer 30
(b) Gemma-7B, layer 26
(c) Llama- 2-13B, layer 38
Figure 10: PCA visualization of 16 concepts across 1-mma-2-7B, Gemma-78 7 and Llama-2- 2-13B. Lewelevel concepts belonging the same high-level concept category share the same color.
29


Reviews:
[{"review_id": "RpUn11Uo9d", "reviewer": "Reviewer_JcLm", "summary": "The paper proposes a robust alternative to the standard practice of learning linear probes in LLMs to find concept representations (for steering).\nSpecifically, the proposed \"Gaussian Concept Subspace\" (GCS) approach models the concept representation as a multivariate Gaussian (with diagonal covariance), thereby capturing the variance in the representations of a concept.\nThe overall procedure is to first train several linear probes using different probing datasets (generated by a LLM); estimating their mean and variances; and then sampling several concept vectors according to the learned Gaussian distribution (within a $1\\sigma$ range).\nIn a set of experiments, it is shown that the resulting GCS vectors are faithful to the concept, aligns with known hierarchical semantic relations (in a topic hierarchy), and can be used to improve the robustness of steering tasks (in sentiment steering).", "questions": "- Intro: For the sake of clarity, I think it should be mentioned somewhere that the concepts being considered here are binary, requiring positive/negative prompt pairs. You can just say \"following prior work\" and reference, e.g., the ITI paper, and maybe give a few examples of what concepts are being considered here.\n- p. 3: have you tried removing the independence assumption and estimating a full covariance matrix for the concept subspace (maybe for a smaller model, to reduce $d$)? If so, how does it compare to the current approach?\n- p. 3: \"randomly sample vectors ... within $1\\sigma$\" is slightly ambiguous to me. Am I correct in assuming that you  first sample from the learned Gaussian distribution and then reject the sample if it is outside the 1-sigma boundary? In Algorithm 1 it says \"in $1\\sigma$\", which is even more ambiguous. I think this part can be reworded for clarity.\n- p. 4: What is the similarity function? If it is cosine similarity, then wouldn't it be skewed by how far away the mean vector is from the origin? If you use centered cosines or some distance metric, do these results change?\n- Eq. 8: What exactly is $\\bf C$? ${\\bf w}_i$ is already indexed by the dataset number, so ${\\bf w}_i \\in {\\bf C}$ is confusing to me.\n- p. 5, \"Implementation Details\" first paragraph: I was wondering about this very detail since Section 3, and it feels like crucial information that shouldn't be left in the \"details\" part. My suggestion is to bring this up earlier when you introduce the sampling part. I am also curious to see how some of the results change when you nevertheless resort to distributional comparisons, e.g., for plausibility (what does the KL divergence between the concepts show?).\n- p. 5, lines 265--267: do you specifically mean that you subsample 1,000 (with/without replacement?) from the 10,000 samples for each concept? This feels like an important detail to be clarified.\n- p. 6, lines 319--322: it first says that \"We sample concept vectors ... ranging from $1\\sigma$ to $5\\sigma$, etc.\" and then says \"we focus on sampling concept vectors within the 1$\\sigma$ range\". Some rewording here appears necessary.\n- Figure 5 (PCA): which concept vectors exactly do you use to learn the PC space? Also, could you just project the Gaussian mean vector rather than the mean of 1,000 sampled vectors?\n- Table 1 (Intervention): for $1\\sigma$--$5\\sigma$, am I correct in thinking that you sampled 1,000 steering vectors, applied them, and then averaged the final ratings? This was confusing to me and could be clarified. I'd also recommend adding error bars if this was the case.\n- Intervention (App. H.4): in the rating prompt, why specifically ask \"repetitive or chaotic\" instead of \"coherent\" or \"fluent\"?\n- Appendix C, lines 737--738: this feels like a critical detail. Exactly how is the \"range\" of ${\\bf h}^\\ell$ found?\n- Appendix H: I don't think it's a good idea, for science, to show only best performing examples. At least I'd want to see random samples along with the best ones.\n- Appendix I: are these random or hand-picked samples within each category?\n\n- Minor stylistic suggestions:\n    - p. 3, lines 115--117: maybe instead of repeating the question from the intro, pose it in a theorem environment (\"Question 1\") in the intro and then simply refer to it here?\n    - Sections 3: maybe call the \"dataset\" specifically as \"probing dataset\" and give a sense of how many samples one may need per a probe vector. Otherwise, I think the reader can get worried about needing an excessively large dataset for each concept.\n    - p. 4, line 197: I think this meant \"generalizable\" instead of \"generalized\"?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Good motivation and clear introduction. The paper is generally well-written and easy to follow.\n- I think it is a meaningful endeavor to model the variance across different probe vectors for a single concept in LLMs, especially knowing that the probe vectors can be unreliable (see, e.g., [Tan et al., 2024](https://arxiv.org/abs/2407.12404), which I think deserves a mention in the paper by the way). The paper proposes an intuitive and sensible approach for this.\n- Most of the experiments are well-designed and the results are convincing. I think the plots for plausibility experiments are particularly clear and informative.\n- It's very interesting to see that, in the intervention experiment for sentiment, the $1\\sigma$ samples from GCS (collectively) outperform the mean difference or the single-probe vector. (Curious to see if this generalizes to other concepts, but that's probably beyond the scope of this paper.)", "weaknesses": "- I think the paper makes a meaningful contribution, but it is relatively light on noting the limitations of its main approach. Here are the main ones in my view:\n    - The most obvious drawback of the main approach is its reliance on large samples obtained using a high-quality LLM (10,000 samples from GPT-4o per each topic concept). This appears necessary to obtain a variance estimate on the GCS, so it feels inherent to the approach. Perhaps this needs to be mentioned in the introduction and the discussion, as it could be a significant limitation for certain use cases.\n    - Another limitation, which I think is fine as long as it is mentioned in the paper, is the assumption of Gaussianity with diagonal covariance for the concept vectors. That said, this is still far better than having no variance information and is not a knock on the paper's contribution.\n    - For the intervention step, it appears that the steering is done by applying each of the 1,000? sampled steering vectors and averaging the results. It's good to know that this makes the intervention robust, but it can also make the approach computationally expensive. It would be good to see some discussion on this.\n- I think what the similarity score for faithfulness is somewhat confusing. In Section 3, the authors state that we want the sampled concept vectors to be similar to each other as much as the observed concept vectors are. But in Figure 2, the sampled vectors are a lot more similar to each other than the observed ones, which is expected as the sampling restricts to the \"within $1\\sigma$\" range, but then the paper appears to suggest that this is ideal. So, what do we actually want out of this metric? Doesn't Figure 2 just end up being an illustration of how large the variances $(\\sigma_j)$ are in each layer?\n- While this is an understandable choice, I do think it should be noted that the evaluations for intervention experiments are entirely model-based (GPT-4o) and may not be accurate.\n- Finally, I feel that the related work on hierarchical concepts is light in the paper, despite the fact that the plausibility experiments highlight the clusters of topic concepts found by GCS. Some suggestions on representations of hierarchical concepts include [Nickel and Kiela, 2017](https://arxiv.org/abs/1705.08039); [Chen et al., 2021](https://arxiv.org/abs/2104.03869); [He et al., 2024](https://arxiv.org/abs/2401.11374); and [Park et al., 2024](https://arxiv.org/abs/2406.01506).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tvqK1V6gHA", "reviewer": "Reviewer_uPaN", "summary": "This paper introduces the Gaussian Concept Subspace (GCS) framework, which aims to estimate the subspace representing specific concepts within large language models (LLMs). The authors extend the traditional single concept vector approach by modeling the concept subspace using a Gaussian distribution. The effectiveness of GCS is demonstrated through its faithfulness and plausibility across multiple LLMs with different sizes and architectures. The paper also showcases the practical application of GCS in real-world inference-time intervention tasks, such as emotion steering, where it balances steering performance and maintaining fluency in natural language generation tasks.", "questions": "See above", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The authors conduct extensive experiments to validate the faithfulness and plausibility of GCS across multiple LLMs, including different sizes and architectures. \n2. GCS reveals hierarchical concept structures that align with human understanding, providing insights into how concepts are represented within LLMs.\n3. The paper is well-organized and easy to follow.", "weaknesses": "1. The use of Gaussian distributions in representation learning is not entirely new. The paper could benefit from a clearer distinction between GCS and other probabilistic models used in similar contexts.\n2. The paper primarily focuses on a specific set of tasks and datasets. To fully establish the significance of GCS, the authors should explore its applicability to a wider range of tasks and domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dUMLMWgNda", "reviewer": "Reviewer_qyQR", "summary": "Proposed new methodology for mapping model internals to concepts, by extracting a gaussian distribution of concept vectors, rather than a single vector. \nThis method is more robust to variations in the sampled training set, and achieves comparable steering performance and post-steering coherence.", "questions": "No further questions beyond the issues raised in Weaknesses section", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: Good. Creative extension of the current concept vector extraction method that solves a big issue with the current method (robustness)\n\nQuality: Good. experiment methodology and results seems solid and justified, though I did not check math and other implementation details in detail. I would prefer a more robust comparison with single-vector method (e.g. more diverse concepts and text types) before I can confidently say this method is empirically superior, but a priori seems likely that GCS will outperform single-vector for most purposes without being significantly more computationally costly \n\nClarity: Good. no significant barriers to quick skim reading\n\nSignificance: Good. I can see this being the new standard for concept vector extraction/steering, as it's basically a pareto improvement on the existing concept vector methods, without being very costly/complicated to implement. I can imagine future work in the representation engineering literature being facilitated by the authors' library", "weaknesses": "I was unable to identify any substantial weaknesses. \n\nSome minor suggestions:\n- It's not clear to me how \"reproducing hierarchical concepts\" and \"similarity between sampled and observed vectors\" correspond to measures of \"plausibility\" and \"faithfulness\" respectively. Would be nice if you elaborated on why this is the case.\n- Some comparison of the coherence/joyfulness scores provided by GPT vs humans would be nice. Just a tiny sample as a sanity check for whether GPT's scores are way off would do alot for your paper's soundness, as your main results hinge on GPT's evaluations being similar enough to human evaluations.\n- Similarly, would be nice to include human-generated text (e.g. google search results for [concept], or joyful/angry reviews from IMDB dataset) on top of GPT-generated text for the training set. Just as a sanity check that GPT-text for [concept] is not too far off from human-text. (But I understand that this is costly)\n- Would be nice to check how using training texts beyond movie reviews (e.g. joyful/angry tweets) would affect the extracted concept vectors and steering performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VYWn3g3KPr", "reviewer": "Reviewer_cu7j", "summary": "This paper presents a framework, Gaussian Concept Subspace (GCS), for interpreting concept representation within LLMs. Traditional approaches to probing concepts in LLMs rely on single concept vectors derived from linear classifiers, which can be unstable and lack robustness. The GCS method enhances this by modeling each concept as a Gaussian distribution in the representation space, allowing for more nuanced, multidimensional representations. The experiments demonstrate that GCS-sampled concept vectors can effectively describe specific concepts learned by LLMs, similar to single concept vectors. Additionally, these sampled vectors are shown to be effective in downstream interventions, successfully guiding LLMs to generate more desirable outputs, such as positive sentiment steering in text generation.", "questions": "How about using the mean vector rather than sampled vectors? As the sampled vectors are affected by some noises, the mean vector would be more robust for representing the concept and steering model's outputs.\n\nDo you have any intuitive explanation for why GCS improves inference-time intervention compared to a single concept vector? I’m uncertain about the rationale behind this improvement.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The concept vectors are inherently variable across different datasets used for training the linear classifier. By modeling concepts through Gaussian distributions, the proposed approach intuitively captures a robust representation of each concept, reducing dependency on specific datasets. This approach is simple and straightforward for enhancing the robustness of concept vectors.", "weaknesses": "- The authors mentioned that concept vectors \"can vary significantly depending on the dataset used to train the classifier and the training process.\" However, as shown in Figure 2, the cosine similarity among concept vectors derived from different datasets consistently exceeds 0.75. While GCS increases the cosine similarity, it is unclear how critical this improvement is. Though the concept vector is significantly unstable for lower layers according to Figure 3, the GCS also shows low accuracy and cannot address this issue.\n\n- Table 1 indicates that, in inference-time intervention, GCS achieves a stronger steering effect than single concept vectors, while coherence scores increase as well. However, GCS is not consistently superior to single concept vectors, as the results vary based on the strength parameter. Statistical tests should be conducted across different parameters to substantiate the significance of these improvements. Additionally, the criteria for highlighting the table are unclear, making them misleading. Furthermore, as mentioned in the questions, the authors should clarify why the sampled vectors lead to such improvements.\n\n- The idea of modeling concepts with Gaussian distributions to capture their multifaceted nature is intuitive. However, it is unfortunate that the experiments do not demonstrate that the Gaussian distribution effectively models such multidimensional subspaces. For instance, it is interesting to see if the intermediate vector between \"love\" and \"comedy\" movies represents the concept of \"love comedy\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)