Beyond Single Concept Vector: Modeling Concept Subspace in LLMs with Gaussian Distribution

## Abstract

Abstract Probing learned concepts in large language models (LLMs) is crucial for understanding how semantic knowledge is encoded internally. Training linear classifiers on probing tasks is a principle approach to denote the vector of a certain concept in the representation space. However, the single vector identified for a concept varies with both data and training, making it less robust and weakening its effectiveness in real-world applications. To address this challenge, we propose an approach to approximate the subspace representing a specific concept. Built on linear probing classifiers, we extend the concept vectors into Gaussian Concept Subspace (GCS). We demonstrate GCS’s effectiveness through measuring its faithfulness and plausibility across multiple LLMs with different sizes and architectures. Additionally, we use representation intervention tasks to showcase its efficacy in real-world applications such as emotion steering. Experimental results indicate that GCS concept vectors have the potential to balance steering performance and maintaining the fluency in natural language generation tasks.

## 1 Introduction

Large language models (LLMs) such as GPT-4 (Achiam et al., ), LLaMA-3 (Touvron et al., ), and Claude-3 (Anthropic, ) have demonstrated remarkable capabilities across a wide range of natural language understanding and generation tasks (Brown et al., ; Wei et al., ). However, our understanding of how concepts are represented internally within these models remains limited. Current research in this area can be broadly categorized into two categories. The first involves studying model parameters through causal mediation analysis (Vig et al., ), which has found applications in downstream tasks such as knowledge editing (Meng et al., ; Wang et al., ) and circuit identification (Conmy et al., ; Elhage et al., ). The second focuses on investigating the representation space and activations of models (Zou et al., ; Turner et al., ; Jorgensen et al., ; Rimsky et al., ), where representation vectors in hidden spaces are often interpreted as linear vectors of certain concepts, such as color (Patel & Pavlick, ), world models (Nanda et al., ; Li et al., ), and sentiment (Tigges et al., ).

In this work, we focus on the representation vector-based explanation paradigm. Among methods to derive representation vectors, linear concept vectors are most widely used. They are typically obtained by training linear probing classifiers, such as logistic regression, on datasets comprising positive and negative samples (Ousidhoum et al., ; Alain & Bengio, ). This approach has proven effective in identifying semantic knowledge encoded within LLMs and has recently been extensively applied in inference-time interventions to mitigate undesirable model behaviors (Li et al., ; Lee et al., ; Rimsky et al., ; Turner et al., ). However, concept vectors derived from this approach can vary significantly depending on the dataset used to train the classifier and the training process, making it challenging to obtain robust vectors. This substantial variation not only poses challenges in studying concept relations within the representation space but also negatively impacts effectiveness of concept vector’s usage in intervention tasks.

To address these limitations, we explore the following research question: *Instead of using a single vector to describe each underlying concept captured by LLMs, can we use a distribution to more robustly describe that learned concept?* This is motivated by the observation that the representation of a concept may be multifaceted, potentially constituting a multidimensional subspace in the representation space rather than a single linear vector. Building upon this assumption, our work focuses on approximating a subspace that better captures the semantics of specific concepts while providing vectors representing a certain concept with varying relevance.

Based on this motivation, we propose Gaussian Concept Subspace (GCS), which improves upon linear probing classifiers described by Ousidhoum et al. (); Kim et al. (); Alain & Bengio () by using a Gaussian distribution to estimate the potential directional subspace of a concept. The main idea is to learn a set of vectors termed observed vectors instead of an individual vector for each concept. These observed vectors are then used to estimate the Gaussian distribution of the concept subspace. The vectors within this Gaussian distribution represent varying degrees of relevance to the concept, with vectors closer to the mean considered more closely related, termed sampled vectors. This method offers a more nuanced representation of concepts, capturing their multifaceted nature within a multidimensional subspace rather than reducing them to a single vector.

We conduct experiments to evaluate the effectiveness of GCS from two perspectives: 1) Whether concept vectors sampled from GCS effectively describe a given concept learned by LLMs; and 2) Whether concept vectors derived from GCS are useful in downstream interventions to guide LLMs to generate more desirable outputs. To address the first perspective, we evaluate sampled vectors using two categories of metrics: faithfulness and plausibility. For faithfulness, we examine similarities within observed vectors, sampled vectors, and between both sets to demonstrate their relatedness in the representation space. We also compare the performance of both observed vectors and sampled vectors in classifying concept-specific data.
For plausibility, we investigate whether the explanations align with our understanding of inter-concept relations. We evaluate concept hierarchies using average cosine similarity and PCA visualization.
To address the second perspective, we compare the performance of sampled vectors on inference-time intervention tasks with that of vectors derived with mean difference and single training classifier. Specifically, we applied these vectors to steer models towards generating more joyful movie reviews. Our results demonstrate that sampled vectors from GCS are both faithful and plausible in terms of their explanations. Moreover, they outperform vectors from other methods in manipulating models to generate desirable content while maintaining text fluency.

## 2 Preliminary

##### Hidden Representation.

We focus on decoder-only LLMs (GPT-like models) (Ferrando et al., ), where each layer comprises multi-head attention blocks (MHA) and feed-forward networks (FFNs/MLPs). In this study, we utilize frozen pretrained language models. We index layers as $\ell\in L$, where $L$ denotes the set of all model layers. Each layer begins and ends with a residual stream. The MHA first processes the residual stream and adds its output back to it. The updated vector is then passed through MLPs to generate the layer’s output:

$$ $\bm{h}_{{i}}^{\ell+1}=\bm{h}_{i}^{\ell}+\operatorname{MLP}^{\ell}\left(\bm{h}_ {i}^{\ell}+\operatorname{Att}^{\ell}\left(\bm{h}_{i}^{\ell}\right)\right),$ (1) $$

where $\bm{h}_{i}^{\ell}$ represents the $i$-th token in the input token sequence at layer $\ell$. We concentrate on the output representation space of each layer, specifically the residual stream at the layer’s end. Following Zou et al. (), we use the last token’s representation to represent the entire input sequence, denoted as $\bm{h}^{\ell}$, as it is generally believed to integrate information from all preceding tokens.

##### Linear Probing.

We adopt the classic linear concept probing approach to derive concept activation vectors (Kim et al., ). For each concept $c$, given a dataset $\mathcal{D}_{c}=\{(h_{i},y_{i})\mid i=1,\cdots,N\}$ where $h\in\mathbb{P}\cup\mathbb{N}$, $\mathbb{P}$ is the set of concept-related positive samples, $\mathbb{N}$ is the set of concept-irrelevant negative samples, and $y_{i}\ \in\{0,1\}$ indicates the sample class. These datasets generate layerwise hidden representations for downstream classifier training. We employ a logistic regression (LR) classifier as follows:
$\bm{\sigma}(\bm{h}_{c}^{\ell})=\frac{1}{1+\exp\left(-\bm{h}_{c}^{\ell}\cdot\bm
{w}_{c}^{\ell}\right)}$,
where $\bm{h}_{c}^{\ell}\in\mathbb{R}^{N\times d}$ specifies sample representations from a dataset at $\ell$-th layer, and $d$ is the representation dimension. The normalized coefficients $\bm{w}_{c}^{\ell}$
of each binary linear classifier
are considered the concept vector. The loss function for logistic regression with L2 regularization is expressed as follows:

$$ $\hat{\bm{w}_{c}^{\ell}}=\underset{\bm{w}_{c}^{\ell}}{\operatorname{argmin}}- \frac{1}{n}\sum_{i=1}^{n}y_{i}\log\bm{\sigma}(\bm{h}_{i}^{\ell})+(1-y_{i})\log \left(1-\bm{\sigma}(\bm{h}_{i}^{\ell})\right)+\frac{\lambda}{2}|\bm{w}_{c}^{ \ell}|_{2}^{2},$ (2) $$

where hyperparameter $\lambda$ controls the strength of regularization.
However, the derived concept vector $\hat{\bm{w}_{c}^{\ell}}$ can vary substantially depending on the dataset $\mathcal{D}_{c}$ used to train the classifier $\bm{\sigma}(\bm{h}_{c}^{\ell})$ and the specific training process. This variability makes it challenging to obtain robust and consistent vectors.
Such instability not only complicates the study of concept relations within the representation space but also negatively impacts the effectiveness of intervention tasks that rely on these vectors.

## 3 Methodology

To address the above-mentioned limitations, we explore the following research question: *Instead of using a single vector to describe the concept captured by LLMs, can we use a distribution to more robustly describe the learned concept?* We aim to approximate a subspace that better captures the semantics of specific concepts while providing vectors representing the concept with varying relevance.
Building upon the linear probing approach described in Section 2, we introduce the Gaussian Concept Subspace (GCS) framework for estimating a multidimensional subspace of vectors for each concept. While linear probing identifies a single vector for a concept, GCS aims to capture a more comprehensive representation by estimating a Gaussian distribution of vectors.

### 3.1 The Proposed GCS Framework

To estimate the Gaussian distribution of concept vectors, we extend the linear probing method by deriving $M$ independent concept vectors for a concept $c$. We first create a large dataset for a concept $c$ and then randomly sample subsets of examples from this dataset to create $M$ smaller datasets. Each dataset contributes to the same concept vector $c$, analogous to the single vector derived in Equation 2. For a model with target layer $\ell$, we obtain a concept vector $\bm{w}_{c,\ m}^{\ell}$ for the $m$-th dataset at the $\ell$-th layer. This process generates a set of concept vectors for concept $c$:

$$ $\bm{{W}}_{c}^{\ell}=\{\bm{w}_{c,\ m}^{\ell}|\ m=1,\cdots,M\}.$ (3) $$

We use this set to estimate the concept subspace, represented as a Gaussian distribution of concept $c$ at the $\ell$-th layer. The distribution is $d$-dimensional, where $d$ is the dimension of the hidden representations, and is denoted as:

$$ $\bm{W}_{c}^{\ell}\sim\mathcal{N}_{d}(\bm{\mu}_{c}^{\ell},\bm{\Sigma}_{c}^{\ell }).$ (4) $$

To simplify the model, we assume independence between dimensions in the representation space. This assumption results in a diagonal covariance matrix, where diagonal elements represent the variance of each dimension:

$$ $\bm{\mu}_{c}^{\ell}=\frac{1}{M}\sum w_{c,\ m}^{\ell};\quad\bm{\Sigma}_{c}^{ \ell}=\text{diag}(\sigma_{1}^{2},\cdots,\sigma_{d}^{2})$ (5) $$

Figure: Algorithm 1 Proposed GCS Framework

The mean vector $\bm{\mu}_{c}^{\ell}$ is calculated by averaging the corresponding dimensions of the $M$ observed concept vectors, which are essentially the weight vectors from the trained binary classifiers in the linear probing approach.
This multivariate distribution describes a subspace around the observed concept vectors, extending the single-vector representation of linear probing to a multidimensional subspace. Concept vectors within this distribution can be sampled at various standard deviations (e.g., $1\sigma$, $2\sigma$, …, $n\sigma$) from the mean vector. For our evaluations, we randomly sample vectors $\bm{V}_{c}^{\ell}$ (the set of multiple sampled vectors $\bm{v}_{c}^{\ell}$) within $1\sigma$ of the mean to represent the concept.

The overall steps of the GCS framework are summarized in Algorithm 1. By moving from a single vector to the Gaussian distribution of vectors, GCS aims to capture a more robust representation of concepts in the language model’s hidden space, potentially offering improvements over the linear probing approach in both interpretability and downstream applications.

### 3.2 Evaluation of GCS

We use faithfulness and plausibility two metrics to eveluate performance of GCS, which are two crucial dimensions for evaluating the performance of an interpretability approach (Zhao et al., ).

#### 3.2.1 Faithfulness

To assess faithfulness, we compare the performance of sampled vectors $\bm{V}_{c}^{\ell}$ (stage 3 of Algorithm 1) compared to that of observed vectors $\bm{W}_{c}^{\ell}$ (stage 1 of Algorithm 1). Specifically, we employ two metrics: 1) similarity among observed vectors, sampled vectors and between the two sets; 2) accuracy of classifiers based on observed vectors and sampled vectors on our constructed datasets.

##### Similarity

We evaluate the similarity between vectors within the observed concept vectors $\bm{W}^{\ell}_{c}$ and within the sampled concept vectors $\bm{V}^{\ell}_{c}$ using Equation 6. Additionally, we assess the similarity between vectors from the observed set and the sampled set using Equation 7.

$$ $\displaystyle S_{\text{observed}}^{\ell}$ $\displaystyle=1/{N}*{\sum_{i,j}\operatorname{sim}\left(\bm{w}_{i},\bm{w}_{j} \right)},\ \bm{w}_{i},\bm{w}_{j}\in\bm{W}_{c}^{\ell};\,S_{\text{sampled}}^{ \ell}=1/{N}*{\sum_{i,j}\operatorname{sim}\left(\bm{v}_{i},\bm{v}_{j}\right)}\ \bm{v}_{i},\bm{v}_{j}\in\bm{V}_{c}^{\ell}$ (6) $\displaystyle S_{\text{O-S}}^{\ell}$ $\displaystyle=1/{N}*{\sum_{i,j}\operatorname{sim}\left(\bm{w}_{i},\bm{v}_{j} \right)},\ \bm{w}_{i}\in\bm{W}^{\ell}_{c},\ \bm{v}_{j}\in\bm{V}^{\ell}_{c}$ (7) $$

The average similarity score of observed vectors $S_{\text{observed}}^{\ell}$ and sampled vectors $S_{\text{sampled}}^{\ell}$ describes how similar the linear vectors are for a specific concept. If our sampled vectors accurately represent concept vectors, the similarity score of sampled vectors should be comparable to that of observed concept vectors. Moreover, the similarity between observed and sampled vectors, denoted as $S_{\text{O-S}}^{\ell}$, should also be approximately equal to the within-set similarities.

##### Accuracy

The observed concept vectors are derived from the weights of linear concept classifiers. To generate $M$ linear concept vectors for a concept $c$, we randomly sample $M$ datasets $\{\mathcal{D}_{1},\cdots,\mathcal{D}_{M}\}$ from a large dataset $\mathcal{D}$. For each dataset $\mathcal{D}_{i}$, a linear classifier is trained using hidden representations from the $\ell$-th layer of a LLM. The weight vector of this classifier is identified as a concept vector for concept $c$ at layer $\ell$. The average accuracy score of concept vectors for concept $c$ at the $\ell$-th layer is defined as:

$$ $\displaystyle A^{\ell}$ $\displaystyle=\frac{\sum_{i=1}^{M}\operatorname{acc}\left(\bm{w}_{i},\mathcal{ D}_{test}\right)}{M},\bm{w}_{i}\in\bm{C},\ \text{where }\mathcal{D}_{test}= \left\{\begin{array}[]{ll}\mathcal{D}/\mathcal{D}_{i},&\text{if }\bm{C} \subseteq\bm{W}_{c}^{\ell}\\ \mathcal{D},&\text{if }\bm{C}\subseteq\bm{V}_{c}^{\ell}\end{array}\right.$ (8) $$

The average accuracy score captures how effectively concept vectors represent the concept. A higher average accuracy score for observed concept vectors suggests that classifiers have learned more generalized concept vectors.
Sampled concept vectors should achieve comparable or even higher accuracy scores if they genuinely represent related concept.

#### 3.2.2 Plausibility

Plausibility describes how well explanations align with human expectations. We measure the plausibility of GCS by studying whether models learn real-world hierarchies of concepts. To achieve this, we predefine a set of hierarchical concepts.
We use average cosine similarity among concept vectors and principle component analysis (PCA) to demonstrate the consistence of GCS’s plausibility.

##### Average Cosine Similarity

To measure the similarity between two concepts $c_{i}$ and $c_{j}$ at the $\ell$-th layer, we utilized their respective sampled concept vectors $\bm{V}_{c_{i}}^{\ell}$ and $\bm{V}_{c_{j}}^{\ell}$. The concept similarity is represented by the average cosine similarity between two concept vector sets.
The average cosine similarity between these two concepts at the $\ell$-th layer is computed as follows:

$$ $S_{{c_{i}},{c_{j}}}^{\ell}=\frac{\sum_{m=1}^{M}\sum_{n=1}^{M}\operatorname{sim }\left(\bm{v}_{m},\bm{v}_{n}\right)}{M\times M/2},\bm{v}_{m}\in\bm{V}_{{c_{i}} }^{\ell},\bm{v}_{n}\in\bm{V}_{{c_{j}}}^{\ell}.$ (9) $$

##### PCA visualization

PCA aims to linearly reduce high-dimensional data onto lower dimensions. It excels in providing an intuitive understanding of relations between data points (Marks & Tegmark, ). We adopt PCA to visualize the relationship among predefined concepts. Since each concept has thousands vectors, we use the mean vector of sampled concept vectors to represent the concept.

## 4 Experiments

In this section, we present experimental results that measure the performance of GCS and address the following three research questions (RQs):

- RQ1:
How faithfully do GCS-sampled concept vectors represent the original concepts?
- RQ2:
To what extent do explanations derived from GCS-sampled concept vectors align with human expectations about the hierarchies among model’s learned knowledge?
- RQ3:
Can the proposed GCS method effectively mitigate unwanted behaviors in LLMs?

### 4.1 Experimental Setup

#### 4.1.1 Datasets

Figure: Figure 1: Hierarchical concepts.
Refer to caption: x1.png

We define a hierarchical concept structure comprising two levels, encompassing concepts from high-level to low-level. The high-level concepts include four categories: moive, sports event, populated place, and animal. Under each high-level concept category, there are 4 low-level concepts, as illustrated in Figure 1. To ensure consistency and high quality of input data across all experiments, we generate our experimental dataset using the OpenAI API, specifically leveraging GPT-4o model. For each low-level concept, we prompt GPT-4o to produce 5,000 positive samples and 5,000 negative samples. The specific prompts used for generating these samples are detailed in Appendix I. Our prompt design strategy varies based on the concept’s specificity. For more inclusive concepts such as “Town”, we utilize detailed prompt descriptions to guide GPT-4o in generating concept-related data. Conversely, for more specific concepts like “Motor race”, simpler and more generic prompt descriptions are sufficient to produce appropriate samples.

#### 4.1.2 Models

We study concept vectors across multiple LLMs, including Llama-2-7B chat model (Meta AI, ), Gemma-7B (Google, ), and Llama-2-13B chat model (Meta AI, ). This selection of models enables a comparative analysis of GCS across varying model architectures and sizes. By examining GCS’s performance across these diverse models, we aim to provide insights into the method’s generalizability and effectiveness across different LLM implementations.

#### 4.1.3 Implemention details

Evaluating GCS at the distribution level presents significant challenges. Gaussian distribution encompasses vectors representing concepts with varying degrees of relevance, potentially diminishing as the sampling distance from the mean increases. This variability, coupled with the infinite potential of sampled concept vectors, renders precise distribution-level evaluation impractical. To address these limitations, we simplify our assessment by randomly sampling 1,000 concept vectors within $1\sigma$ of the mean, which we consider representative of the concept.

The GCS pipeline comprises several key steps: generating data points using GPT-4o, extracting hidden representations of each sample from LLMs, training linear classifiers for each concept using these hidden representations, and constructing Gaussians distribution for concept vectors. Our study generated 10,000 descriptions per low-level concept (5,000 positive and 5,000 negative). To efficiently utilize this data, we repeatedly sample 1,000 positive and 1,000 negative descriptions to create smaller, concept-specific datasets, each yielding an observed concept vector. This process is repeated 1,000 times for each low-level concept, resulting in 1,000 observed vectors per concept.

These observed vectors are used to derive the mean and covariance of the Gaussian distribution for each concept. To simplify computation, we assume independence between vector dimensions, reducing the covariance matrix to a diagonal matrix where each diagonal value represents the variance of the concept vector in that dimension.

Notably, our classifiers are trained on hidden representations of each description. Following Zou et al. (), we use the hidden representation of the last token within a description from a specific layer as the representation for that description at that layer.
We use L2 regularization when training the logistic regression classifier and the regularization weight $\lambda$ in Equation 2 is set as 1.0.
Our evaluation primarily focuses on hidden representations from the penultimate layer of each model, as we observed improved concept learning in deeper layers (see Appendix D, E, and F).

Figure: (a) Llama-2-7B, layer 30
Refer to caption: x2.png

### 4.2 Faithfulness of GCS ( RQ1 )

We evaluate the faithfulness of GCS from two perspectives: *similarity* and *accuracy*. Similarity assessment involves comparing observed concept vectors with those sampled from the GCS. This comparison provides an effective measure of their relatedness in the representation space. Ideally, for any given concept, the sampled vectors should closely approximate the observed vectors.
We also assess the accuracy of these vectors in concept-related tasks.

#### 4.2.1 Cosine Similarity among Concept vectors

For each concept, we analyze three types of cosine similarity: among observed concept vectors, among sampled concept vectors, and between these two sets, as defined in Equation 6 and 7. The cosine similarities among observed concept vectors indicate the potential size of the concept subspace, with higher similarities suggesting a more compact subspace. The cosine similarities among sampled concept vectors reflect their diversity.
We sample concept vectors according to standard deviations from the mean, such as ranging from $1\sigma$ to $5\sigma$, etc.
Vectors sampled within $1\sigma$ are generally less diverse but closer to the center of the conceptual space. In this work, we focus on sampling concept vectors within the $1\sigma$ range, which are central to the concept subspace. The similarity results are shown in Figure 2, the key findings are summarized as follows.

First,
we investigate these three types of similarities across LLMs of varying sizes and structures, as shown in Figure 2 (a, b, and c) and Appendix D. Our results indicate that similarities among observed concept vectors exhibit greater variation than those among sampled concept vectors, ranging from approximately 0.8 to 0.9. Sampled concept vectors within the $1\sigma$ range consistently demonstrate very high cosine similarities among themselves, with cosine similarity larger than 0.93. This indicates that GCS produces more consistently related vectors than the original observations. Cosine similarities between observed and sampled concept vectors typically range from 0.88 to 0.93. Besides, the patterns of cosine similarity are remarkably consistent across different model sizes and architectures (Llama-2-7B, Gemma-7B, and Llama-2-13B). This suggests that the GCS method is robust across different language models.

Second, to better observe changes in concept vectors within each concept, we also examine the average cosine similarity of these concept vectors at the layer level as shown in Figure 2 (d, e and f) and Appendix E. There is a general upward trend in cosine similarity as we move through the layers. This indicates that concept representations become more refined and consistent in deeper layers of the models. The sampled vectors (blue lines in d, e, f) show the highest and most stable cosine similarity across layers. This indicates that the sampling vectors from GCS produce consistently related vectors throughout the model.

Figure: (a) Llama-2-7B
Refer to caption: x8.png

#### 4.2.2 Prediction Accuracy of Concept Vectors

Despite the similarity between sampled and observed concept vectors in the representation space, it is crucial to verify that sampled vectors are as effective as observed concept vectors. We begin by measuring the accuracy of each observed linear vector on its test dataset, which comprises 8,000 additional concept descriptions. Then, we evaluate sampled concept vectors on the entire dataset of 10,000 samples, as these vectors are not trained on the dataset. The results are reported in Figure 3.

To evaluate the impact of proximity to the concept subspace center, we evaluate sampled concept vectors for all concepts at $1\sigma$, $2\sigma$, $3\sigma$, $4\sigma$, and $5\sigma$ across three LLM models. Our results demonstrate that sampled concept vectors usually achieve comparable and even better prediction accuracy compared to observed vectors, as illustrated in Figure 3. This improved performance is particularly evident for sampled concept vectors at $1\sigma$ and $2\sigma$. These findings are further corroborated by the accuracy of these vectors across all predefined concepts in the Gemma-7B model (Appendix F). Our analysis yields two key insights. First, it validates the effectiveness of sampled concept vectors in representing concepts. Second, it also suggests that vectors closer to the subspace center could be more representative of the concept.

### 4.3 Plausibility of GCS ( RQ2 )

In this section, we evaluate the plausibility of GCS using two methods: average cosine similarity among concepts and PCA visualization (see Section 3.2.2). Here, we utilize concept vectors sampled at $1\sigma$ to compute the average cosine similarity between concepts and to generate the PCA visualization.
We assess the coherence of concept relationships in the high-dimensional space (through cosine similarities) and in a lower-dimensional projection (through PCA visualization).

#### 4.3.1 Average Cosine Similarity among Concepts

We evaluate concept similarity using average cosine similarity. Instead of measuring the entire distribution, we assess 1,000 concept vectors sampled within $1\sigma$ for each concept. The cosine similarity is computed for all vector pairs between two concepts, with the mean value representing the overall similarity between these concepts.
Figure 4 illustrates these results across various LLMs.

Figure: (a) Llama-2-7B, layer 30
Refer to caption: x11.png

The analysis of concept cosine similarities across three different language models reveals several key insights. First, low-level concepts within the same high-level category typically exhibit higher similarity scores. All three models (Llama-2-7B, Gemma-7B, and Llama-2-13B) show similar overall patterns, with distinct $4\times 4$ blocks along the diagonal representing high intra-category similarities. These intra-category similarities are generally higher compared to similarities with low-level concepts from other high-level categories.
Second, while all high-level categories show internal coherence, some categories appear more tightly related than others. For example, the “sports events” category (top-left $4\times 4$ block) consistently shows very high internal similarity across all models, suggesting these concepts are closely related in the models’ representations. Third, beyond intra-category similarities, there are interesting inter-category relationships.
Notably, Llama-2-7B and 13B models demonstrate stronger correlations between place and animal. For instance, island shows a stronger correlation with Bird and Fish than insect and cat. Bird exhibits a stronger correlation with village than fish and cat do. These correlations align with human intuition about real-world relationships between these concepts. Such alignment suggests that the models have captured meaningful semantic relationships that reflect our understanding of the world.
These findings also demonstrate the effectiveness of GCS in capturing complex and interpretable concept relationships across different language models.

#### 4.3.2 PCA Visualization

PCA visualization offers an intuitive method to reveal concept proximity by linearly reducing concept vectors into a two-dimensional space. We visualize all 16 concepts to explore their relationships in terms of sampled concept vectors. Figure 5 illustrates the results across various LLMs at the penultimate layer. Across all three models, we observe several patterns. First, in all three visualizations, concepts belonging to the same high-level category tend to cluster together, confirming our initial observation. This clustering is particularly evident for sports events and movie genres.
Second, the PCA visualizations for Llama-2-7B, Gemma-7B, and Llama-2-13B show remarkably similar patterns, reinforcing the robustness of the GCS method across different model architectures and sizes.
Third, for cross-category relationship, concepts belonging to place and animal categories demonstrate closer proximity compared to other concept pairs, aligning with our findings in Section 4.3.1. The consistency of these patterns across models and their interpretable clustering strongly support the plausibility of GCS as an effective method for model interpretation.

Figure: (a) Llama-2-7B, layer 30
Refer to caption: x14.png

### 4.4 Applications of GCS on Inference-time Intervention ( RQ3 )

Recently, concept vectors has been extensively used to steer model outputs towards desirable behaviors during inference time. This approach is considered as more efficient than finetuning, without needing modify the model’s parameters. These steering vectors are generally derived using various approaches, all based on pairs of desired and undesired data samples. Following Konen et al. (), we evaluate GCS on open-ended generation tasks, which are more flexible and challenging tasks for intervention compared to natural language understanding tasks such as multiple-choice questions. Specifically, we aim to steer Llama-2-7B chat model to generate more joyful movie reviews, a task we refer to as emotion steering. To measure the performance of GCS, we compare vectors derived from GCS with two conventional methods: mean difference and linear vector as defined in Appendix B. Moreover, we utilize vectors within $n\sigma,n\in\{1,2,3,4,5\}$ from GCS to explore the relation between steering performance and concept vector relevance.

We follow the experimental settings described in Section 4.1.3. We use GPT-4o to generate “joyful” and “angry” movie reviews, then derive concept vectors for the *joyful* concept. Throughtout our experiments, we utilize the representation of the last token as the input for downstream computation. The process involves tasking models to generate angry movie reviews using prompts detailed in Appendix H. We then apply concept vectors to the representation of the last token across all layers, excluding the first and the last, to shift the model’s outputs. We attempt to incrementally adjust the strength of steering vectors to measure its influence on outputs as detailed in Appendix C. The generated sentences are rated with GPT-4o from two aspects: joyfulness and coherence. Joyfulness assesses how joyful the generated text become after steering, while coherence measures the fluency of generated text through repetitive or chaotic elements. The result is shown in Table 1.

**Table 1: Evaluation of generated text after adding steering vectors.**
| Steering Method | Strength |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0.038 | 0.043 | 0.048 | 0.053 | 0.059 | 0.064 | 0.069 | 0.074 | 0.080 |  |  |
| Mean Difference | Joyfulness (Avg) $\uparrow$ | 1.020 | 0.860 | 1.220 | 1.100 | 1.653 | 1.245 | 1.800 | 1.780 | 1.260 |
| Coherence (Avg) $\downarrow$ | 3.857 | 5.460 | 5.740 | 5.420 | 5.571 | 6.306 | 5.440 | 4.420 | 3.420 |  |
| 1 sigma | Joyfulness (Avg) $\uparrow$ | 1.000 | 0.800 | 1.260 | 1.490 | 2.120 | 2.980 | 2.280 | 1.776 | 2.160 |
| Coherence (Avg) $\downarrow$ | 4.780 | 3.680 | 4.040 | 3.531 | 4.860 | 4.857 | 6.480 | 5.347 | 5.800 |  |
| 2 sigma | Joyfulness (Avg) $\uparrow$ | 0.840 | 1.520 | 1.143 | 1.878 | 2.625 | 2.458 | 2.520 | 2.340 | 1.960 |
| Coherence (Avg) $\downarrow$ | 4.420 | 3.680 | 3.857 | 4.061 | 5.854 | 6.688 | 6.460 | 6.360 | 6.520 |  |
| 3 sigma | Joyfulness (Avg) $\uparrow$ | 0.820 | 1.220 | 1.220 | 1.837 | 2.460 | 2.571 | 2.388 | 2.510 | 1.854 |
| Coherence (Avg) $\downarrow$ | 3.920 | 4.180 | 3.580 | 3.449 | 5.120 | 5.633 | 6.204 | 6.633 | 5.542 |  |
| 4 sigma | Joyfulness (Avg) $\uparrow$ | 0.840 | 0.755 | 1.380 | 1.960 | 2.280 | 2.224 | 2.612 | 2.720 | 1.520 |
| Coherence (Avg) $\downarrow$ | 3.600 | 3.837 | 4.400 | 3.560 | 4.720 | 5.878 | 6.653 | 6.800 | 4.200 |  |
| 5 sigma | Joyfulness (Avg) $\uparrow$ | 0.580 | 0.640 | 1.429 | 1.640 | 2.458 | 1.680 | 2.333 | 2.224 | 2.220 |
| Coherence (Avg) $\downarrow$ | 3.440 | 4.040 | 5.347 | 3.480 | 4.771 | 5.900 | 5.625 | 5.694 | 4.980 |  |
| One linear | Joyfulness (Avg) $\uparrow$ | 0.840 | 1.245 | 1.520 | 2.000 | 2.292 | 2.580 | 3.020 | 2.480 | 2.080 |
| Coherence (Avg) $\downarrow$ | 4.360 | 3.347 | 4.380 | 4.640 | 5.208 | 6.020 | 5.918 | 6.780 | 6.340 |  |
| Note: A higher joyfulness score indicates a better steering effect. Coherence measures the repetitiveness and chaos in the generated sentences, with lower values being preferable. The best performance for each method is highlighted in bold within the table. |  |  |  |  |  |  |  |  |  |  |

It is crucial to maintain a balance between steering models to produce desired expressions and preserving the fluency of those expressions.
For example, before intervention, Llama-2-7B chat model generates angry review: *Awful, predictable, and cringeworthy.* Our GCS $1\sigma$ sampled vector based intervention can push it to generate joyful review: *Creative writing, beautiful acting, fun movie.*
Examples of best performance for each method are detailed in Appendix H.3.
The results in Table 1 show that mean difference is less effective in steering, and more likely to generate less fluent text. Additionally, the vectors sampled within $2\sigma$, $3\sigma$, $4\sigma$, and $5\sigma$ of GCS are generally less effective in generating joyful content compared to $1\sigma$ sampled vectors and one linear vector. Although one linear vector can reach comparable effect as $1\sigma$, its robustness is not as good as vectors sampled within $1\sigma$ from GCS. It’s worthnoting that one linear vector is trained on all data samples used to derive the set of concept vectors. As a result, it is possible that the one linear vector falls around the $1\sigma$ area, which makes it achieve the similar steering effect but lack of robustness. Consequently, GCS provides better-represented and more robust concept vectors. It also offers more options to choose vectors within varying relevance to the center of the subspace.

## 5 Related Work

##### Concept Vector.

The concept vector represents a linear vector of a concept and is typically constructed using datasets containing both positive and negative samples. A range of methods have been implemented to derive the vector. One common approach is to train a linear classifier and the vector is the normalized weights of the learned classifier (Kim et al., ).
Another branch of methods focus on mean difference involving different strategies. One popular way is to compute the difference between average activations of positive samples and negative samples. Another approach is to derive steering vectors by averaging the difference in residual stream activations between pairs of positive and negative samples at certain layers (Rimsky et al., ; Zou et al., ). While Mean-Centering takes the difference between the average activation of a target dataset and that of all training activations as the steering vector (Jorgensen et al., ). K-means is also proven feasible to compute vectors, where positive and negative clusters are clustered unsupervisedly and then their mean values are used (Tigges et al., ).
Besides, Principle Component Analysis (PCA) is another popular way to produce concept vectors through identifying specific dimensions of representations (Liu et al., ; Lee et al., ).

##### Inference-time Intervention.

The intervention on internal representations of LLMs during inference time has proven effective in steering model outputs (Burns et al., ; Zou et al., ). This approach utilizes steering vectors, also known as steering activations, which are believed to linearly represent concepts or features in the representation space (Park et al., ; Elhage et al., ). These vectors are added up to models representations during forward pass at the level of either attention head or layer.
For example, some studies add scaled concept vectors to tokens, either all tokens or last token of a sequence, at certain layers to manipulate models output (Rimsky et al., ; Liu et al., ; Tigges et al., ; Turner et al., ; Zou et al., ).
However, some work prefer intervening on a more delicate level, such as attention heads. They focuses on heads that are responsible for specific behaviors, then intervening selected heads to achieve comparable performance (Wang et al., ; Li et al., ).

## 6 Conclusions

In this paper, we introduce the Gaussian Concept Subspace (GCS), a new framework to estimate the subspace of specific concepts within LLMs. Our research demonstrates the faithfulness and plausibility of GCS in terms of its explanation. Explanations from GCS are similar to trained vectors in representation space and their prediction accuracy is comparable to, and in some cases surpasses, that of trained vectors. GCS also reveals the hierarchical concept structures that align with we human’s understanding. Besides, we attempt to apply it in real-world inference-time intervention tasks. The performance of GCS demonstrates its potential for efficiently mitigating models’ undesirable behaviors. In summary, GCS offers a concise yet powerful method for concept subspace estimation, showing promise in both theoretical explanations and practical applications.

## Acknowledgment

The work is in part supported by NSF #2310261. The views and conclusions in this paper are those of the authors and should not be interpreted as representing any funding agencies.

## Appendix A Experimental Environment

All experiments were conducted on a computing cluster consisting of 10 CPU nodes and 4 GPU nodes. Each CPU node is equipped with 512 GB of RAM and 128 prcoessor cores. Each GPU node contains 4 NVIDIA A100 GPUs, with each GPU having 80 GB of dedicated memory.

## Appendix B Baseline Approaches

### B.1 Mean Difference Vector

Given a dataset $\mathcal{D}$ of concept samples, same amount of positive samples $p\in\mathbb{P}$ and negative samples $n\in\mathbb{N}$ are included. The mean difference concept vector $\bm{v}_{MD}$ at layer $\ell$ is calculated as:

$$ $\bm{v}_{MD}^{\ell}=\frac{2}{|\mathcal{D}|}\sum_{{p\in\mathbb{P}},n\in\mathbb{N }}\bm{h}^{\ell}\left(p\right)-\bm{h}^{\ell}\left(n\right),$ (10) $$

where $\bm{h}^{\ell}(*)$ represents the hidden representation of last token of a sample at layer $\ell$.

### B.2 Linear vector

The classic linear vector adopts the whole training dataset to generate a single linear concept vector. Following definition in B.1, the set of positive samples $\mathbb{P}$ and the set of negative samples $\mathbb{N}$ includes positive samples $p$ and negative samples $n$ separately. The linear vector at layer $\ell$ denoted as $\bm{w}_{\ell}$ is generated by training a logistic regression classifier following Equation 2.

## Appendix C Intervention Approach

Given a LLM with $L$ layers, we introduce concept vectors to the output of certain layers, specifically to the final hidden state in the residual stream. When the LLM processes a prompt, we manipulate the output towards a specific concept by adding the concept vector to the final hidden state at each layer. In this work, we focus on manipulating the hidden representation of only the last token. The steering process is defined as follows:

$$ $\bm{h}^{\ell}=(1-a)\times\bm{h}^{\ell}+a\times\bm{v}_{c}^{\ell},$ (11) $$

where $\bm{h}^{\ell}$ represents the final hidden state of the last token at the $\ell$-th layer. $\bm{v}_{c}^{\ell}$ denotes the scaled concept vector for the $\ell$-th layer. $a$ is a parameter controlling the strength of the concept vector’s influence. Specifically, to preserve the distribution of hidden state values, we scale the concept vector $\bm{v}_{c}^{\ell}$ to match the range of $\bm{h}^{\ell}$ before applying it. This scaling ensures that the addition of the concept vector does not significantly distort the model’s internal representations.

## Appendix D Cosine Similarity Distributions of Concept Vectors in Llama-2-7B

This section presents a detailed analysis of concept vector relationships across 16 concepts within the Llama-2-7B model. The figure illustrates the cosine similarity distributions for three comparisons: 1) among observed concept vectors, 2) among sampled vectors, 3) between observed and sampled vectors. The distributions reveal that, for all concepts at the penultimate layer, sampled vectors constantly exhibit very high similarity values. Moreover, the similarity between sampled and observed vectors always exceeds 0.9. Thus, sampled vectors closely approximate observed concept vectors in the model’s representation space.

Figure: Figure 6: Histogram of cosine similarity for observed concept vectors, sampled concept vectors, and between two sets across 16 concepts at layer 30 of Llama-2-7B.
Refer to caption: x17.png

## Appendix E Average Cosine Similarity of Concept Vectors in Llama-2-7B

This section examines the average cosine similarity of concept vectors across 16 concepts within the Llama-2-7B model. Average cosine similarity provides a more effective measure of how concept vectors evolve as the model progresses through deeper layers. The analysis encompasses three comparative aspects: 1) observed concept vectors, 2) sampled vectors, and 3) the relationship between observed and sampled vectors. Results indicate that for all concepts, sampled vectors consistently exhibit the highest similarity values from the second layer to the penultimate layer. Notably, in the final few layers, most concepts achieve remarkably high similarity scores, often reaching 0.95, across all three groups. This pattern suggests that the internal representations of concepts in the final few layers remain relatively stable, with minimal divergence in their vector representations.

Figure: Figure 7: Average Cosine Similarity of Concept Vectors Across Layers in Llama-2-7B.
Refer to caption: x18.png

## Appendix F Average Accuracy of Concept Vectors on Gemma-7B Model

This section examines the average accuracy of concept vectors across 16 concepts within the Gemma-7B model. We measure the accuracy of observed vectors, and sampled vectors at $1\sigma$, $2\sigma$, $3\sigma$, $4\sigma$, $5\sigma$ from the mean.The comparable and often superior accuracy of sampled vectors demonstrates their effectiveness as real concept vectors. Our findings indicate that sampled vectors closer to the concept subspace center generally achieve higher accuracy. This trend suggests a correlation between a vector’s proximity to the subspace center and its representational accuracy.

Figure: Figure 8: Average accuracy of observed concept vectors and sampled concept vectors on Gemma-7B Model.
Refer to caption: x19.png

## Appendix G Average Cosine Similarity among Concepts in Llama-2-7B

This section presents a detailed analysis of interconcept relations across 16 concepts within the Llama-2-7B model. The figure illustrates the average cosine similarity between concept vector pairs. Our results indicate that concept relations are consistently learned throughout all layers. Notably, these concepts become increasingly distinct from one another as the model progresses through deeper layers, demonstrating a trend of growing conceptual differentiation.

Figure: Figure 9: Heatmap of interconcept relationships in Llama-2-7B. Average cosine similarity between concept pairs across model layers. Here, we visualize the first 30 layers.
Refer to caption: x20.png

## Appendix H Emotion Steering

The experimental details of emotion steering are detailed as below.

### H.1 Prompts for Sample Generation

The prompts below are used to prompt GPT-4o to generate concept-related samples, which are necessary to derive concept vectors.

### H.2 Prompts for LLM Generation

The prompt is designed to prompt the model that we steer to generate outputs with angry sentiment.

### H.3 Generated Reviews under Intervention

The reviews below are generated by adding up layerwise intervention vectors and prompting Llama-2-7B Chat model to generate joyful reviews (even though prompts in H.2 are for generating angry reviews). The results are chosen from best performances of each method. These less robust methods usually generate expressions that are less fluent, with uncommon unicode characters, or less joyful.

#### H.3.1 Mean Shift, 0.069

#### H.3.2 1 ⁢ σ 1 𝜎 1\sigma 1 italic_σ , 0.064

#### H.3.3 2 ⁢ σ 2 𝜎 2\sigma 2 italic_σ , 0.059

#### H.3.4 3 ⁢ σ 3 𝜎 3\sigma 3 italic_σ , 0.064

#### H.3.5 4 ⁢ σ 4 𝜎 4\sigma 4 italic_σ , 0.059

#### H.3.6 5 ⁢ σ 5 𝜎 5\sigma 5 italic_σ , 0.059

#### H.3.7 One linear, 0.069

### H.4 Prompts for Rating Generated Text

The prompt instructs GPT-4o to evaluate the generated text on two criteria: joyfulness and coherence. For joyfulness, GPT-4o assesses the text’s overall joyful emotional tone. For coherece, it rates the text’s logical flow and structure, considering factors such as repetitiveness and disorganization.

## Appendix I Prompts to Generate Concept-related Dataset

We prompt GPT-4o to generate no more than 100 positive samples or negative samples each round to avoid failures of instruction following. For concepts in sports events, the simply phrased prompts are enough to generate samples that well represent the concepts. Because these concepts such as cycling competition are focused and specialized. However, the other much broader concepts demand more detailed prompts when generating positive samples.

### I.1 Sports Event

#### I.1.1 Cycling competition

#### I.1.2 Football match

#### I.1.3 Tennis tournament

#### I.1.4 Motor race

### I.2 Populated Place

#### I.2.1 Town

#### I.2.2 Island

#### I.2.3 City district

#### I.2.4 Village

### I.3 Movie

#### I.3.1 Horror

#### I.3.2 Action

#### I.3.3 Comedy

#### I.3.4 Science Fiction

### I.4 Animal

#### I.4.1 Bird

#### I.4.2 Fish

#### I.4.3 Insect

#### I.4.4 Cat

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: - Intro: For the sake of clarity, I think it should be mentioned somewhere that the concepts being considered here are binary, requiring positive/negative prompt pairs. You can just say "following prior work" and reference, e.g., the ITI paper, and maybe give a few examples of what concepts are being considered here.
- p. 3: have you tried removing the independence assumption and estimating a full covariance matrix for the concept subspace (maybe for a smaller model, to reduce $d$)? If so, how does it compare to the current approach?
- p. 3: "randomly sample vectors ... within $1\sigma$" is slightly ambiguous to me. Am I correct in assuming that you  first sample from the learned Gaussian distribution and then reject the sample if it is outside the 1-sigma boundary? In Algorithm 1 it says "in $1\sigma$", which is even more ambiguous. I think this part can be reworded for clarity.
- p. 4: What is the similarity function? If it is cosine similarity, then wouldn't it be skewed by how far away the mean vector is from the origin? If you use centered cosines or some distance metric, do these results change?
- Eq. 8: What exactly is $\bf C$? ${\bf w}_i$ is already indexed by the dataset number, so ${\bf w}_i \in {\bf C}$ is confusing to me.
- p. 5, "Implementation Details" first paragraph: I was wondering about this very detail since Section 3, and it feels like crucial information that shouldn't be left in the "details" part. My suggestion is to bring this up earlier when you introduce the sampling part. I am also curious to see how some of the results change when you nevertheless resort to distributional comparisons, e.g., for plausibility (what does the KL divergence between the concepts show?).
- p. 5, lines 265--267: do you specifically mean that you subsample 1,000 (with/without replacement?) from the 10,000 samples for each concept? This feels like an important detail to be clarified.
- p. 6, lines 319--322: it first says that "We sample concept vectors ... ranging from $1\sigma$ to $5\sigma$, etc." and then says "we focus on sampling concept vectors within the 1$\sigma$ range". Some rewording here appears necessary.
- Figure 5 (PCA): which concept vectors exactly do you use to learn the PC space? Also, could you just project the Gaussian mean vector rather than the mean of 1,000 sampled vectors?
- Table 1 (Intervention): for $1\sigma$--$5\sigma$, am I correct in thinking that you sampled 1,000 steering vectors, applied them, and then averaged the final ratings? This was confusing to me and could be clarified. I'd also recommend adding error bars if this was the case.
- Intervention (App. H.4): in the rating prompt, why specifically ask "repetitive or chaotic" instead of "coherent" or "fluent"?
- Appendix C, lines 737--738: this feels like a critical detail. Exactly how is the "range" of ${\bf h}^\ell$ found?
- Appendix H: I don't think it's a good idea, for science, to show only best performing examples. At least I'd want to see random samples along with the best ones.
- Appendix I: are these random or hand-picked samples within each category?

- Minor stylistic suggestions:
    - p. 3, lines 115--117: maybe instead of repeating the question from the intro, pose it in a theorem environment ("Question 1") in the intro and then simply refer to it here?
    - Sections 3: maybe call the "dataset" specifically as "probing dataset" and give a sense of how many samples one may need per a probe vector. Otherwise, I think the reader can get worried about needing an excessively large dataset for each concept.
    - p. 4, line 197: I think this meant "generalizable" instead of "generalized"?
soundness: 3
strengths: - Good motivation and clear introduction. The paper is generally well-written and easy to follow.
- I think it is a meaningful endeavor to model the variance across different probe vectors for a single concept in LLMs, especially knowing that the probe vectors can be unreliable (see, e.g., [Tan et al., 2024](https://arxiv.org/abs/2407.12404), which I think deserves a mention in the paper by the way). The paper proposes an intuitive and sensible approach for this.
- Most of the experiments are well-designed and the results are convincing. I think the plots for plausibility experiments are particularly clear and informative.
- It's very interesting to see that, in the intervention experiment for sentiment, the $1\sigma$ samples from GCS (collectively) outperform the mean difference or the single-probe vector. (Curious to see if this generalizes to other concepts, but that's probably beyond the scope of this paper.)
summary: The paper proposes a robust alternative to the standard practice of learning linear probes in LLMs to find concept representations (for steering).
Specifically, the proposed "Gaussian Concept Subspace" (GCS) approach models the concept representation as a multivariate Gaussian (with diagonal covariance), thereby capturing the variance in the representations of a concept.
The overall procedure is to first train several linear probes using different probing datasets (generated by a LLM); estimating their mean and variances; and then sampling several concept vectors according to the learned Gaussian distribution (within a $1\sigma$ range).
In a set of experiments, it is shown that the resulting GCS vectors are faithful to the concept, aligns with known hierarchical semantic relations (in a topic hierarchy), and can be used to improve the robustness of steering tasks (in sentiment steering).
weaknesses: - I think the paper makes a meaningful contribution, but it is relatively light on noting the limitations of its main approach. Here are the main ones in my view:
    - The most obvious drawback of the main approach is its reliance on large samples obtained using a high-quality LLM (10,000 samples from GPT-4o per each topic concept). This appears necessary to obtain a variance estimate on the GCS, so it feels inherent to the approach. Perhaps this needs to be mentioned in the introduction and the discussion, as it could be a significant limitation for certain use cases.
    - Another limitation, which I think is fine as long as it is mentioned in the paper, is the assumption of Gaussianity with diagonal covariance for the concept vectors. That said, this is still far better than having no variance information and is not a knock on the paper's contribution.
    - For the intervention step, it appears that the steering is done by applying each of the 1,000? sampled steering vectors and averaging the results. It's good to know that this makes the intervention robust, but it can also make the approach computationally expensive. It would be good to see some discussion on this.
- I think what the similarity score for faithfulness is somewhat confusing. In Section 3, the authors state that we want the sampled concept vectors to be similar to each other as much as the observed concept vectors are. But in Figure 2, the sampled vectors are a lot more similar to each other than the observed ones, which is expected as the sampling restricts to the "within $1\sigma$" range, but then the paper appears to suggest that this is ideal. So, what do we actually want out of this metric? Doesn't Figure 2 just end up being an illustration of how large the variances $(\sigma_j)$ are in each layer?
- While this is an understandable choice, I do think it should be noted that the evaluations for intervention experiments are entirely model-based (GPT-4o) and may not be accurate.
- Finally, I feel that the related work on hierarchical concepts is light in the paper, despite the fact that the plausibility experiments highlight the clusters of topic concepts found by GCS. Some suggestions on representations of hierarchical concepts include [Nickel and Kiela, 2017](https://arxiv.org/abs/1705.08039); [Chen et al., 2021](https://arxiv.org/abs/2104.03869); [He et al., 2024](https://arxiv.org/abs/2401.11374); and [Park et al., 2024](https://arxiv.org/abs/2406.01506).

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: See above
soundness: 3
strengths: 1. The authors conduct extensive experiments to validate the faithfulness and plausibility of GCS across multiple LLMs, including different sizes and architectures. 
2. GCS reveals hierarchical concept structures that align with human understanding, providing insights into how concepts are represented within LLMs.
3. The paper is well-organized and easy to follow.
summary: This paper introduces the Gaussian Concept Subspace (GCS) framework, which aims to estimate the subspace representing specific concepts within large language models (LLMs). The authors extend the traditional single concept vector approach by modeling the concept subspace using a Gaussian distribution. The effectiveness of GCS is demonstrated through its faithfulness and plausibility across multiple LLMs with different sizes and architectures. The paper also showcases the practical application of GCS in real-world inference-time intervention tasks, such as emotion steering, where it balances steering performance and maintaining fluency in natural language generation tasks.
weaknesses: 1. The use of Gaussian distributions in representation learning is not entirely new. The paper could benefit from a clearer distinction between GCS and other probabilistic models used in similar contexts.
2. The paper primarily focuses on a specific set of tasks and datasets. To fully establish the significance of GCS, the authors should explore its applicability to a wider range of tasks and domains.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: No further questions beyond the issues raised in Weaknesses section
soundness: 3
strengths: Originality: Good. Creative extension of the current concept vector extraction method that solves a big issue with the current method (robustness)

Quality: Good. experiment methodology and results seems solid and justified, though I did not check math and other implementation details in detail. I would prefer a more robust comparison with single-vector method (e.g. more diverse concepts and text types) before I can confidently say this method is empirically superior, but a priori seems likely that GCS will outperform single-vector for most purposes without being significantly more computationally costly 

Clarity: Good. no significant barriers to quick skim reading

Significance: Good. I can see this being the new standard for concept vector extraction/steering, as it's basically a pareto improvement on the existing concept vector methods, without being very costly/complicated to implement. I can imagine future work in the representation engineering literature being facilitated by the authors' library
summary: Proposed new methodology for mapping model internals to concepts, by extracting a gaussian distribution of concept vectors, rather than a single vector. 
This method is more robust to variations in the sampled training set, and achieves comparable steering performance and post-steering coherence.
weaknesses: I was unable to identify any substantial weaknesses. 

Some minor suggestions:
- It's not clear to me how "reproducing hierarchical concepts" and "similarity between sampled and observed vectors" correspond to measures of "plausibility" and "faithfulness" respectively. Would be nice if you elaborated on why this is the case.
- Some comparison of the coherence/joyfulness scores provided by GPT vs humans would be nice. Just a tiny sample as a sanity check for whether GPT's scores are way off would do alot for your paper's soundness, as your main results hinge on GPT's evaluations being similar enough to human evaluations.
- Similarly, would be nice to include human-generated text (e.g. google search results for [concept], or joyful/angry reviews from IMDB dataset) on top of GPT-generated text for the training set. Just as a sanity check that GPT-text for [concept] is not too far off from human-text. (But I understand that this is costly)
- Would be nice to check how using training texts beyond movie reviews (e.g. joyful/angry tweets) would affect the extracted concept vectors and steering performance.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: How about using the mean vector rather than sampled vectors? As the sampled vectors are affected by some noises, the mean vector would be more robust for representing the concept and steering model's outputs.

Do you have any intuitive explanation for why GCS improves inference-time intervention compared to a single concept vector? I’m uncertain about the rationale behind this improvement.
soundness: 3
strengths: - The concept vectors are inherently variable across different datasets used for training the linear classifier. By modeling concepts through Gaussian distributions, the proposed approach intuitively captures a robust representation of each concept, reducing dependency on specific datasets. This approach is simple and straightforward for enhancing the robustness of concept vectors.
summary: This paper presents a framework, Gaussian Concept Subspace (GCS), for interpreting concept representation within LLMs. Traditional approaches to probing concepts in LLMs rely on single concept vectors derived from linear classifiers, which can be unstable and lack robustness. The GCS method enhances this by modeling each concept as a Gaussian distribution in the representation space, allowing for more nuanced, multidimensional representations. The experiments demonstrate that GCS-sampled concept vectors can effectively describe specific concepts learned by LLMs, similar to single concept vectors. Additionally, these sampled vectors are shown to be effective in downstream interventions, successfully guiding LLMs to generate more desirable outputs, such as positive sentiment steering in text generation.
weaknesses: - The authors mentioned that concept vectors "can vary significantly depending on the dataset used to train the classifier and the training process." However, as shown in Figure 2, the cosine similarity among concept vectors derived from different datasets consistently exceeds 0.75. While GCS increases the cosine similarity, it is unclear how critical this improvement is. Though the concept vector is significantly unstable for lower layers according to Figure 3, the GCS also shows low accuracy and cannot address this issue.

- Table 1 indicates that, in inference-time intervention, GCS achieves a stronger steering effect than single concept vectors, while coherence scores increase as well. However, GCS is not consistently superior to single concept vectors, as the results vary based on the strength parameter. Statistical tests should be conducted across different parameters to substantiate the significance of these improvements. Additionally, the criteria for highlighting the table are unclear, making them misleading. Furthermore, as mentioned in the questions, the authors should clarify why the sampled vectors lead to such improvements.

- The idea of modeling concepts with Gaussian distributions to capture their multifaceted nature is intuitive. However, it is unfortunate that the experiments do not demonstrate that the Gaussian distribution effectively models such multidimensional subspaces. For instance, it is interesting to see if the intermediate vector between "love" and "comedy" movies represents the concept of "love comedy".

Meta-review: This paper moves from modeling concept representations as directions in a representation space to modeling them as normal distributions (with diagonal covariance) in the representation space. Reviewers appreciated the novelty and clarity, and are optimistic that characterizing the variance will improve the usefulness of LLM representation approaches. The reviewers offered extensive (though relatively minor) feedback, which should be incorporated in the camera ready revision.

Decision: Accept (Poster)