Controlling Language and Diffusion Models by Transporting Activations

## Abstract

Abstract The increasing capabilities of large generative models and their ever more widespread deployment have raised concerns about their reliability, safety, and potential misuse. To address these issues, recent works have proposed to control model generation by steering model activations in order to effectively induce or prevent the emergence of concepts or behaviors in the generated output.
In this paper we introduce Activation Transport ( AcT ), a general framework to steer activations guided by optimal transport theory that generalizes many previous activation-steering works. AcT is modality-agnostic and provides fine-grained control over the model behavior with negligible computational overhead, while minimally impacting model abilities.
We experimentally show the effectiveness and versatility of our approach by addressing key challenges in large language models (LLMs) and text-to-image diffusion models (T2Is). For LLMs, we show that AcT can effectively mitigate toxicity, induce arbitrary concepts, and increase their truthfulness. In T2Is, we show how AcT enables fine-grained style control and concept negation.

## 1 Introduction

Pre-trained Generative Models (GMs) typically undergo an additional fine-tuning phase to better align them to a desired behavior.
For example, Large Language Models (LLMs) are aligned via instruction fine-tuning () or RLHF (Ouyang et al., ). Although less extensively, these strategies have also been applied to Text-to-Image (T2I) models (Wallace et al., ; Yang et al., ). However, as the number of parameters grows, alignment approaches can become challenging from a computational and memory perspective (Houlsby et al., ). In addition, these strategies modify the model’s internal mechanisms, realigning its parameters by leveraging new data, which can have the undesired side effect of impacting the utility of the model on other metrics (Kotha et al., ; Luo et al., ), such as 0-shot evaluation or question-answering.

The increasing cost of fine-tuning has motivated research in inference-time interventions on pre-trained models (Suau et al., ; Rimsky et al., ; Zou et al., ; Li et al., ) in order to control specific behaviors. Since these modifications are typically sparse and/or low-dimensional, they can be estimated using a few hundreds of sentences (Suau et al., ; Turner et al., ).
For example, Rimsky et al. (); Li et al. () shift activations by a constant vector estimated with sets of desired and undesired data (e.g., non-toxic and toxic); or Suau et al. () mitigate toxicity by dampening the activations of expert neurons.
While effective, existing methods do not preserve the activation distribution observed by the model during training. Considering how brittle GMs can be (Huu-Tien et al., ; Sclar et al., ), a constant shift can move activations out-of-distribution (OOD), which can lead to unwanted behaviors, and hinder both the conditioning and the general model performance.

We propose Activation Transport (AcT), a framework to steer activations according to the Optimal Transport (OT) map between two different (source and target) activation distributions. For example, from toxic to non-toxic language, or between two different styles in T2I generation. AcT applies a set of univariate maps on activations while preserving their target distributions, achieving better controllability and stronger robustness to the choice of model and layers intervened upon.
Our main contributions are:

- •
A unifying interpretation of existing activation steering methods under the umbrella of OT, showing that most existing methods are equivalent to a mean transport map (Section 3.3).
- •
Linear-AcT, an inference-time intervention(^1^11Code will be made publicly available upon acceptance.) based on OT that preserves internal activation distributions (Section 3.1). The degree of intervention can be controlled by a strength parameter $\lambda$ between 0 (no transport) and 1 (full transport), as shown in Figure 1. We also introduce the transport support to prevent inducing OOD activations.
- •
We show that, without any hyperparameter tuning, Linear-AcT matches or outperforms existing inference-time interventions when aiming to control LLMs for the tasks of toxicity mitigation, concept induction, and increasing truthfulness.
- •
We find that off-the-shelf Linear-AcT is also effective at controlling T2I diffusion models for the tasks of fine-grained style control and concept negation. Additionally, we adapt (Li et al., ) (ITI) for T2I. To the best of our knowledge, this is the first work to apply an inference-time intervention method that is simultaneously effective on both LLMs and Diffusion Models.

## 2 Related Work

The growing capabilities and prevalence of GMs (Brown et al., ; Rombach et al., ), along with the rising costs of fine-tuning and alignment, have driven research into controllability of GMs.

##### Controlling LLMs

ActAdd (Turner et al., ) uses a contrast prompt (one positive and one negative example) to construct a shift vector.
CAA (Rimsky et al., ) builds on ActAdd by calculating the difference vectors for steering based on a dataset of contrast pairs (rather than a single pair), adding the mean difference during inference time for steering.
ITI-c (Li et al., ) estimates the shift vector orthogonal to the hyperplane learnt by a binary linear classifier on two sets of sentences, showing an increase of truthfulness on the TruthfulQA benchmark (Lin et al., ). The same work proposes MassMean (ITI-m), with an additive vector computed as the difference in means for both sets of sentences.
With a different approach, AurA by Suau et al. () dampens activations proportionally to each neuron’s ability to classify toxic and non-toxic sentences, effectively mitigating toxicity.
RepE by Zou et al. () proposes to compute steering vectors at inference time based on prompt pairs.
Wu et al. () considers activations relationships using a low-rank projection to exchange information with a counterfactual representation.
Orthogonal to the works of activation steering, Dekoninck et al. () have proposed a language model arithmetic that can combine the outputs of multiple models in a principled way to simulatenously control multiple concepts, however requiring several (costly) inference passes on the LLM.

##### Controlling T2I

Few works tackle aligment of T2I models. Wallace et al. () align diffusion models with reinforcement learning (RL) on human comparison data. Yang et al. () remove the need of a reward model to reduce computational overhead of RL. Other works focus on fine-tuning to maximize a reward function (Clark et al., ) or consistency to reference images (Lee et al., ). The literature on T2I diffusion model controllability is more extensive and it commonly consists in training structure adapters (Mou et al., ; Jiang et al., ), style adapters (Stracke et al., ; Ye et al., ; Zhao et al., ), or low-rank adapters (LoRAs) (Ruiz et al., ; YEH et al., ; Gandikota et al., ; Stracke et al., ). Closer to our work are inference-time interventions, which do not require backpropation through the model to train the conditioning mechanisms. Diffusion steering methods are a family of inference-time interventions, which directly modify the diffusion algorithm at test time for fine-grained control with additional prompts (Nair et al., ; Brack et al., ). To the best of our knowledge, our work is the first to explore inference-time interventions that are not specific to diffusion models and transfer across modalities.

## 3 Transporting Neuron Activations

We represent the activations of a GM given an input sentence ${\bm{x}}\in\mathcal{S}$ as a tensor $\mathbb{R}^{M\times L\times K}$, where $M$ is the number of activations per layer (assumed constant w.l.o.g. for simplicity), $L$ the number of layers, and $K$ the number of tokens decoded. We reduce each of the $K$ values to only one using an arbitrary pooling operator $\phi$. From now on we write $\mathbf{Z}:\mathcal{S}\rightarrow\mathbb{R}^{M\times L}$ for the map that turns a sentence into a matrix of activations statistics, noting that $\mathbf{Z}$ incorporates $\phi$-pooling.

We consider two probability distributions on sentences $p$ and $q$. We will view these sentences exclusively through the lens of their aggregated activation matrices, i.e., we will examine probability distributions $\mu:=\mathbf{Z}\sharp p$ and $\nu:=\mathbf{Z}\sharp q$, where we have used the pushforward operator $\sharp$.
In practice, we have access to samples ${\bm{x}}^{1},\dots,{\bm{x}}^{n}\sim p$ and ${\bm{y}}^{1},\dots,{\bm{y}}^{n}\sim q$.
In the case of toxicity mitigation, $p$ covers toxic sentences and $q$ non-toxic ones.
Input sentences $({\bm{x}}^{i})$ and $({\bm{y}}^{i})$ go through the model to yield activation matrices ${\bm{a}}^{i}:=\mathbf{Z}({\bm{x}}^{i})$ and ${\bm{b}}^{i}=\mathbf{Z}({\bm{y}}^{i})$, each seen as i.i.d. samples from $\mu$ and $\nu$ respectively. This results in $n+n$ observations of $M\times L$ matrices. In that context, our goal is to learn a transport map $T:\mathbb{R}^{M\times L}\rightarrow\mathbb{R}^{M\times L}$ from $({\bm{a}}^{i},{\bm{b}}^{i})$ that approximately pushes $\mu$ to $\nu$, i.e., $T\sharp\mu\approx\nu$.

### 3.1 Low Budget Estimators for Transport Maps

Since a modern GM can have millions of activations, we seek maps $T$ that are cheap to evaluate and store, with very few parameters. Following Suau et al. (), we focus on maps that factorize independently along each dimension (each activation). $T$ is therefore described as a collection of $ML$ independent univariate maps, where each map indexed by $m,l$ should ideally map the marginal distribution of $\mu$ in that coordinate to that of $\nu$. Recall that:

###### Proposition 3.1 (Univariate Transport Maps)

(Santambrogio, , Chap.2)
Let $\rho,\tau\in\mathcal{P}(\mathbb{R})$ be two univariate distributions. For any submodular cost $c:\mathbb{R}\times\mathbb{R}\rightarrow\mathbb{R}$ (i.e., such that $\partial c/\partial x\partial y<0$), the optimal transport map $T$ that can transport $\rho$ to $\tau$ is $T^{\star}=Q_{\tau}\circ F_{\rho}$, where $Q_{\tau}$ and $F_{\rho}$ are respectively the quantile function of $\tau$ and the cumulant density function (CDF) of $\rho$.

Estimating and storing all $ML$ transport maps would therefore require dealing with as many quantile and CDF functions. Unfortunately, parameterizing each of these could quickly become intractable, which is why we scale down ambitions to simplify further our working hypothesis to only consider affine transport maps.

Figure: Figure 2: Transport maps using different methods. For distributions with $\sigma_{a}=\sigma_{b}$ (left) all methods (except ActAdd) are equivalent. When $\sigma_{a}\neq\sigma_{b}$ (right), vector-based methods (e.g., ActAdd, ITI-c, Mean-AcT) diverge from the map defined by the samples. ActAdd shows a bias since it only uses one sample pair. The linear estimator is robust to differences in $\sigma$.
Refer to caption: x1.png

Each of the $ML$ activations we consider results in two families of reals: source $(a^{1}_{m\ell},\dots,a^{n}_{m\ell})$ and targets $(b^{1}_{m\ell},\dots,b^{n}_{m\ell})$. Simpifying notations, we drop mentions to $m$ and $\ell$ to focus on values $A:=(a^{1},\dots,a^{n})$ and $B:=(b^{1},\dots,b^{n})$ each in $\mathbb{R}^{n}$. We propose to consider the simple proxy task of finding affine maps that push $A$ to $B$ efficiently. The best such affine map is denoted Linear-AcT, and defined in Definition 3.1. Despite its simplicity, we show in Section 3.3 that many state-of-the-art methods boil down to even simpler approximations and heuristics.

###### Definition 3.1 (Linear- AcT )

Given samples $A=(a^{1},\dots,a^{n})$ and $B=(b^{1},\dots,b^{n})$ and a cost function $c:\mathbb{R}\times\mathbb{R}\rightarrow\mathbb{R}$, the
Linear-AcT map trained with these samples is defined as

$$ $T(a;A,B):=\omega a+\beta,$ $$

where $\omega,\beta$ are the minimizers of $\min_{\omega,\beta}\sum_{i}c\big{(}b^{(i)},\omega a^{(i)}+\beta\big{)}$, and can be recovered in closed form when $c(a,b):=(a-b)^{2}$, as

$$ $\omega=\frac{\sum_{i}\tilde{a}^{(i)}\tilde{b}^{(i)}}{\sum_{i}(\tilde{b}^{(i)}) ^{2}},\quad\beta=m_{b}-\omega m_{a},$ $$

where $m_{a}=\tfrac{1}{n}\sum_{i}a^{i},m_{b}=\tfrac{1}{n}\sum_{i}b^{i}$ are mean values, and superscripted values with ^(i) refer to sorted values in increasing order, $(a^{(1)}\leq\dots\leq a^{(n)})$ and $(b^{(1)}\leq\dots\leq b^{(n)})$.
Additionally, $\tilde{a}^{(i)}=a^{(i)}-m_{a},\tilde{b}^{(i)}=b^{(i)}-m_{b}$ are sorted and recentered observations.

An important feature of Linear-AcT is that it can be composed with linear layers in the GM, resulting in no computational overhead at inference time (see Appendix A for details).
Note that the expression in Linear-AcT should not be confused with the closed-form known when transporting a Gaussian density with parameter ($m_{a},\sigma_{a})$ towards a second ($m_{b},\sigma_{b}$), which is known (Peyré & Cuturi, , Remark 2.31) to be $T(a)=\frac{\sigma_{b}}{\sigma_{a}}a+(m_{b}-\frac{\sigma_{b}}{\sigma_{a}}m_{a}).$
Note that if one makes the additional assumption that $\sigma_{a}=\sigma_{b}$, then the affine Gaussian map becomes a mean shift or translation, with $T(a)=a+m_{b}-m_{a}$. We call this very simple baseline Mean-AcT and show in Section 3.3 that several methods in the literature indeed propose versions of a mean shift strategy.

Figure 3 showcases the effect of different maps on toy data (iid, Gaussian). Note that methods based on mean-shift (ActAdd, ITI-c, Mean-AcT) can strongly over or undershoot, mapping samples out-of-distribution. Linear-AcT shows a good trade-off between in distribution mapping and low computational budget. We note that activations in current GMs show mostly unimodal distributions, but have different standard deviations for different behaviors as shown in Figure 3, making the linear choice a suitable one. Note that multimodal distributions would result in non-linear transport maps, which are beyond the scope of this work.

###### Proposition 3.1 (Univariate Transport Maps)

###### Definition 3.1 (Linear- AcT )

##### Transport Support

The map in Definition 3.1 is estimated using $n$ pairs of samples. In practice, $n$ is in the order of hundreds, which results in a rough approximation of the true transport from $\mu$ to $\nu$. It is fair to assume that the transport error will be higher for input samples in the tail of $\mu$, given the scarcity of samples in that range.
Because transporting OOD samples may lead to unexpected behavior, and to be on the conservative side, we only transport new samples that are within the osberved support $\mathcal{Q}_{o}=[\min A,\max A]$.
Using the support is important when $\mu$ is narrower than $\nu$ (typically in a mitigation setup). Unless stated otherwise, we use $\mathcal{Q}_{o}$ for concept mitigation and $\mathcal{Q}_{\infty}=(-\infty,\infty)$ for induction. Appendix D shows an empirical validation of this choice.

### 3.2 Sequential Iterative Maps

While it might be possible to follow the template approach outlined in Section 3.1 to apply univariate maps to each of the $ML$ activations, this ignores the causal relationship across activations, where activations produced by a layer are processed by the next one, i.e., ${\bm{a}}_{m,\ell+1}=f_{\ell}({\bm{a}}_{m,\ell})$. Any intervention at the level of a layer must therefore be factored in accordingly before creating the intervention at the next one. To account for such causality, we estimate the transport maps for each layer incrementally: we first estimate the transport for the first layer (in the model graph), then we run inference again by applying the first layer map in order to estimate the map for the second layer, and so on until all maps are estimated. A similar approach is adopted in Zou et al. (), and detailed with our tools in Definition 3.2. In Appendix C we show that causal estimation achieves more effective conditioning than a simultaneous estimation. In this work, we use causal estimation for Mean-AcT and Linear-AcT.

###### Definition 3.2 (Affine Causal Transport Map)

For $m\leq M$ and $\ell\leq L$, let $A_{m}:=({\bm{a}}^{1}_{m,1},\cdots,{\bm{a}}^{n}_{m,1})$ and $B_{m}:=({\bm{b}}^{1}_{m,1},\cdots,{\bm{b}}^{n}_{m,1})$ denote $n$ families of $M$ activations for the first layer. Starting with $\ell=1$, and setting

$$ $C_{m,1}:=A_{m,1},D_{m,1}:=B_{m,1},$ $$

compute and store the $2M$ $(\omega_{m},\beta_{m})$ parameters of all $M$ transport maps associated with these activations using Definition 3.1:

$$ $\forall m\leq M,\forall\ell\leq L,\quad T_{m,\ell}:=T(\;\cdot\;;C_{m,\ell},D_{ m,\ell}):\mathbb{R}\rightarrow\mathbb{R},$ $$

where observations $C$ and $D$ are refreshed recursively for each of their entries $m\leq M$, as $\ell$ is incremented,

$$ $\displaystyle C_{\cdot,\ell+1}$ $\displaystyle:=f_{\ell}([T_{m,\ell}(C_{m,\ell})]_{m})\,,$ $\displaystyle D_{\cdot,\ell+1}$ $\displaystyle:=f_{\ell}([T_{m,\ell}(D_{m,\ell})]_{m})\,.$ $$

At inference time, given a sentence ${\bm{x}}$, we run the recursion starting from the first activation vector ${\bm{a}}=({\bm{a}}_{m,1})_{m}$, looping for $1\leq\ell\leq L$ as ${\bm{a}}\leftarrow f_{\ell}([T_{m,\ell}({\bm{a}}_{m})]_{m}$.

Interpolation Between Measures Using Transport One can easily extend a transport map from measure $\mu$ to $\nu$ to one that is able to output an interpolating measure. The idea, outlined by McCann (), consists in defining the following $\lambda$-parameterized map from any OT map $T$,

$$ $T(a,\lambda)=(1-\lambda)a+\lambda T(a),$ (1) $$

where $\lambda\in[0,1]$ and $\lambda=1$ recovers the full transport.
Conditioning GMs through OT allows the user to precisely control the presence of a concept with a continuous and interpretable knob $(\lambda)$ during generation, not requiring expensive parameter search (Li et al., ) or being limited by fixed, uncontrollable conditioning (Suau et al., ). In applications such as diffusion, where the utility of the model is harder to assess, our interpretable strength is of key importance, as shown in Section 5. Note that methods like ActAdd, CAA or ITI-c also have a conditioning strength parameter. However, this parameter is applied as a multiplier of a conditioning bias as $T(a,\lambda)=a+\lambda\beta$ (see Section 3.3), thus making $\lambda$ unbounded, harder to interpret and not robust with respect to different models, layers, and tasks.

###### Definition 3.2 (Affine Causal Transport Map)

### 3.3 Generalization of Prior Inference-Time Interventions Work

In this section, we show how many earlier works can be interpreted as special cases of Linear-AcT.
Table 1 summarizes the intervention proposed by several recent methods, where we show that all methods propose a form of linear transport, and all of them (aside from Suau et al. ()) add a bias to the activations. The way this bias is pre-computed is what differentiates each method. Note that the parameter $\lambda$ typically multiplies the bias, thus becoming unbounded and non-interpretable.

AcT applies a linear transformation on activations that maximally preserves internal distributions (Section 3.1). Moreover, AcT interpolates between the current and transformed activations, making $\lambda$ bounded between $[0,1]$ and interpretable. An additional important aspect is that other methods propose various heuristics to choose the support, while AcT uses all activations or the observed input range ($\mathcal{Q}_{o}$). Note that CAA, ITI-m and Mean-AcT use a difference in means. We subsume this family of methods reporting results for Mean-AcT, which has the additional advantage of an interpretable $\lambda$.
An additional difference is that many methods use the last token only (in pseudocode, $\phi({\bm{z}})={\bm{z}}[\ldots,-1]$). Det${}_{\text{zero}}$ and AurA use max-pooling ($\phi({\bm{z}})={\bm{z}}.max(-1)$). Conversely AcT uses an average across tokens ($\phi({\bm{z}})={\bm{z}}.mean(-1)$), which we have found to be more robust.

**Table 1: Comparison of different inference-time interventions in the literature. All methods listed can be expressed as a specific form of a linear map. With AcT, the conditioning strength $\lambda$ interpolated between the activation $a$ and its transformed version (following Equation 1), while existing methods use $\lambda$ as a bias multiplier, thus becoming less interpretable and less robust to model/layer changes. As a result, many methods require a grid-search to find the best layer to intervene upon.**
| Method | Transport | Parameters | Support | $\phi$ |
| --- | --- | --- | --- | --- |
| Det${}_{\text{zero}}$ (Suau et al., ) | $\omega a+\beta$ | $\omega=0,\;\beta=m_{b}$ | Any layer, ${a\mid\text{AP}(A,B)>\varepsilon}$ | max |
| ActAdd (Turner et al., ) | $\omega a+\lambda\beta$ | $\omega=1,\;\beta=a^{+}-a^{-}$ | Layer search | last |
| CAA (Rimsky et al., ) | $\omega a+\lambda\beta$ | $\omega=1,\;\beta=m_{b}-m_{a}$ | Layer search | last |
| RePE (Zou et al., ) | $\omega a+\lambda\beta$ | $\omega=1,\;\beta=a^{+}({\bm{x}})-a^{-}({\bm{x}})$ | Layer search | last |
| AurA (Suau et al., ) | $\omega a+\beta$ | $\omega=1-Gini(A,B),\;\beta=0$ | Any layer, ${a\mid\text{AUROC}(A,B)>0.5}$ | max |
| EAST (Rahn et al., ) | $\omega a+\lambda\beta$ | $\omega=1,\;\beta\approx m_{b}$ | Layer search | last |
| ITI-m (Li et al., ) | $\omega a+\lambda\beta$ | $\omega=1,\;\beta=m_{b}-m_{a}$ | Attention head search | last |
| ITI-c (Li et al., ) | $\omega a+\lambda\beta$ | $\omega=1,\;\beta=f_{CLS}(A,B)$ | Attention head search | last |
| Mean-AcT, Section 3.1 | $(1-\lambda)a+\lambda(\omega a+\beta)$ | $\omega=1,\;\beta=m_{b}-m_{a}$ | Any layer, $a\in\mathcal{Q}_{o}\text{ or }\mathcal{Q}_{\infty}$ | mean |
| Linear-AcT, Definition 3.1 | $(1-\lambda)a+\lambda(\omega a+\beta)$ | $\omega,\beta=\operatorname*{arg\,min}_{\omega,\beta}\sum_{i}(b^{(i)}-(\omega a<br>^{(i)}+\beta))^{2}$ | Any layer, $a\in\mathcal{Q}_{o}\text{ or }\mathcal{Q}_{\infty}$ | mean |

## 4 Experiments on LLMs

We empirically verify the performance of AcT on pre-trained LLMs on toxicity mitigation (Section 4.1), general concept induction (Section 4.2), and truthfulness induction in particular (Section 4.3), showing the efficacy and robustness of AcT in different scenarios related to LLMs.

### 4.1 Toxicity Mitigation in LLMs

It is known that LLMs are prone to generate toxic language (Wen et al., ), especially when prompts are designed to elicit toxic behavior. In this section, we study how AcT is effective at toxic language mitigation compared to some recents methods such as AurA, ActAdd and ITI-c, on Gemma2-2B (Team et al., ) and Llama3-8B Dubey et al. ().

We prompt each LLM with 1000 randomly chosen prompts from RealToxicityPrompts (RTP) (Gehman et al., ),known to induce toxic language generation. Then, we collect the generated continuation to each prompt and we evaluate toxicity with a ROBERTA-based classifier(^2^22[https://huggingface.co/s-nlp/roberta_toxicity_classifier](https://huggingface.co/s-nlp/roberta_toxicity_classifier)), as in Suau et al. (). In addition, we also measure toxicity in a 0-shot manner by querying Llama3-8B-instruct as LLM-as-a-judge (Zheng et al., ) (more details on the way we instructed the LLM-as-a-judge can be found in Appendix E). As a measure of general LLM utility we report in Section 4.1: (i) perplexity (PPL) on a fixed set of 20k Wikipedia sentences measured with the intervened model, (ii) PPL of the generated sentences measured with Mistral-7B (Jiang et al., ) and (iii) MMLU (Hendrycks et al., ) 5-shot accuracy using the intervened model.

**Table 2: Toxicity mitigation for Gemma2-2B and Llama3-8B, results over 5 runs. We intervene upon different layer types (layer column) and show the best layer per method. ITI-c, ActAdd and AcT have a strength parameter $\lambda$ which we sweep. For each method, we report results for the $\lambda$ that attained the best CLS toxicity that incurs less than $+1$ increase in PPL Wikipedia. AcT methods and provide best results for $\lambda=1$, achieving up to $7.5\times$ (Gemma2-2B) and $4.3\times$ (Llama3-8B) CLS toxicity mitigation with Linear-AcT. ITI-c is very sensitive to $\lambda$ as well as layer choice (see full results in Appendix G), and AurA reaches up to $3.1\times$ reduction.**
|  | Causal | Layer | Best $\lambda$ | PPL Wikipedia $\downarrow$ | PPL Mistral-7B $\downarrow$ | CLS Toxicity (%) $\downarrow$ | 0-shot Toxicity (%) $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Original | - | - | - | 13.98 | 6.62 | $4.08\pm{0.36}$ | $13.25\pm{0.88}$ |
| Mean-AcT |  | Attention | 1.0 | 13.90 | 7.23 (+0.61) | $1.12\pm{0.35}$ | $5.60\pm{1.01}$ |
| Mean-AcT | ✓ | Attention | 1.0 | 14.08 (+0.11) | 7.23 (+0.61) | $1.06\pm{0.17}$ | $5.14\pm{0.50}$ |
| Linear-AcT |  | Attention | 1.0 | 14.04 (+0.06) | 7.26 (+0.64) | $0.97\pm{0.39}$ | $5.75\pm{0.90}$ |
| Linear-AcT | ✓ | Attention | 1.0 | 14.21 (+0.23) | 7.24 (+0.62) | $0.90\pm{0.33}$ | $5.06\pm{0.63}$ |
| Mean-AcT |  | Post-LN | 1.0 | 14.11 (+0.13) | 7.71 (+1.09) | $0.62\pm{0.05}$ | $4.47\pm{0.65}$ |
| Mean-AcT | ✓ | Post-LN | 1.0 | 14.21 (+0.23) | 7.59 (+0.97) | $0.54\pm{0.44}$ | $4.10\pm{0.41}$ |
| Linear-AcT |  | Post-LN | 0.9 | 14.54 (+0.57) | 7.87 (+1.25) | $0.65\pm{0.17}$ | $4.40\pm{0.39}$ |
| Linear-AcT | ✓ | Post-LN | 1.0 | 14.79 (+0.81) | 7.99 (+1.37) | $0.56\pm{0.21}$ | $4.14\pm{0.55}$ |

### 4.2 Inducing Concepts in LLMs with AcT

Figure: Figure 4: Concept induction using AcT (post-LN layers) and ITI-c (attention layers) on Gemma2-2B. We aggregate results over 7 WordNet concepts, generating 500 sentences at different intervention strength levels. We report concept presence with LLM-as-a-judge ($p(yes)$), and the PPL of the generated sentences using Mistral-7B. We plot the median (and 25/75 quantile band) across concepts and generations per level, showing that Linear-AcT achieves a peak of concept induction at $\lambda\approx 1$, which is inline with our OT formulation. Other methods show different maxima.
Refer to caption: x2.png

AcT allows transporting activations from distribution $\mu$ to $\nu$ (derived from sentence distributions $p$ and $q$ respectively). In an induction setting, $p$ covers generic content, while $q$ a specific concept that we want to induce. We mine the OneSec dataset (Scarlini et al., ), collecting 700 sentences that contain a specific concept ($q$) and 700 sentences randomly sampled from other concepts ($p$). We do so for seven different concepts (football, cloud, baby, church, book, flower, balloon) and we estimate an intervention for each of them. We assess the presence of a concept in the generated text in a LLM-as-a-judge manner by querying Llama3-8B-instruct (LLM-as-a-judge details in Appendix F).

Figure 4 shows the effect of increasing $\lambda$ both on the presence of the concept, $p(yes)$, and the PPL measured with Mistral-7B on the generated text. We intervene upon the most effective layers for each method according to the toxicity results: Post-LN for AcT and attention for ITI-c. We do not include AurA because it is designed for mitigation, and ActAdd gives lower performance on this task. For Linear-AcT, we observe a peak of concept presence at $\lambda\approx 1$, with a median $p(yes)=0.87$ (i.e., 87% of the generated sentences are classified as containing the induced concept) and an acceptable $\text{PPL}=8.5$. For $\lambda>1$, the PPL quickly degrades and the presence of the concept diminishes.
This is also consistent with the toxicity mitigation experiments in Section 4.1. Interestingly, the peak for Mean-AcT is at $\lambda\approx 2.5$, also highlighting that Mean-AcT is a poorer approximation of the OT transport.
Notably, ITI-c achieves a similar $p(yes)$ and PPL as Linear-AcT for $\lambda\approx 5$. However, note that ITI-c’s best $\lambda$ is different than the ones for toxicity.
Appendix H contains generation examples.

**Table 3: TruthfulQA results for Gemma2-2B and Llama3-8B, results over 5 runs. We intervene upon different layers (layer column) and show the best per model. ITI-c, ActAdd and AcT have a strength parameter $\lambda$ which we sweep, reporting the best $\lambda$ result per model (MC1 Accuracy so that MMLU is within the best AcT MMLU $\pm\;0.1$).**
|  | Causal | Layer | Best $\lambda$ | PPL Wikipedia $\downarrow$ | PPL Mistral-7B $\downarrow$ | CLS Toxicity (%) $\downarrow$ | 0-shot Toxicity (%) $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Original | - | - | - | 13.98 | 6.62 | $4.08\pm{0.36}$ | $13.25\pm{0.88}$ |
| Mean-AcT |  | Attention | 1.0 | 13.90 | 7.23 (+0.61) | $1.12\pm{0.35}$ | $5.60\pm{1.01}$ |
| Mean-AcT | ✓ | Attention | 1.0 | 14.08 (+0.11) | 7.23 (+0.61) | $1.06\pm{0.17}$ | $5.14\pm{0.50}$ |
| Linear-AcT |  | Attention | 1.0 | 14.04 (+0.06) | 7.26 (+0.64) | $0.97\pm{0.39}$ | $5.75\pm{0.90}$ |
| Linear-AcT | ✓ | Attention | 1.0 | 14.21 (+0.23) | 7.24 (+0.62) | $0.90\pm{0.33}$ | $5.06\pm{0.63}$ |
| Mean-AcT |  | Post-LN | 1.0 | 14.11 (+0.13) | 7.71 (+1.09) | $0.62\pm{0.05}$ | $4.47\pm{0.65}$ |
| Mean-AcT | ✓ | Post-LN | 1.0 | 14.21 (+0.23) | 7.59 (+0.97) | $0.54\pm{0.44}$ | $4.10\pm{0.41}$ |
| Linear-AcT |  | Post-LN | 0.9 | 14.54 (+0.57) | 7.87 (+1.25) | $0.65\pm{0.17}$ | $4.40\pm{0.39}$ |
| Linear-AcT | ✓ | Post-LN | 1.0 | 14.79 (+0.81) | 7.99 (+1.37) | $0.56\pm{0.21}$ | $4.14\pm{0.55}$ |

### 4.3 Inducing truthfulness in LLMs with AcT

One particular concept that has gained attention in previous activation steering works is “truthfulness” (Li et al., ).
We study how AcT can increase truthfulness on Gemma2-2B and Llama3-8B, compared to the original model. Again, we compare to AurA, ActAdd and ITI-c. We evaluate all methods on the TruthfulQA multiple choice part that has been used in prior work (Lin et al., ; Li et al., ). We report both MC1 and MC2 of TruthfulQA, and control for overfitting on the TruthfulQA task by also evaluating MMLU 5-shot accuracy (Hendrycks et al., ).

The results of our experiments are summarized in Table 4.2. As we can see, AcT can successfully induce truthfulness in both models in its default setting $\lambda=1$ (corresponding to full transport). Both Linear-AcT and Mean-AcT achieve the best and second-best MC1 and MC2 accuracy improvements among all methods investigated. Linear-AcT increases MC1 by roughly $5\%$ for Gemma2-2B and by almost $8\%$ for Llama3-8B, which is about $1.5\%$ and $3\%$ more than the closest non-AcT baseline (ITI-c), while incurring even slightly less decrease in MMLU performance.
Full results and experimental setup in Appendix I.

## 5 Controlling Image Diffusion Models

In this section, we show that AcT improves the controllability of text-to-image diffusion models (T2Is), a well-known challenge (Cao et al., ). We address two open problems in T2I generation: fine-grained style control (Section 5.1) and concept negation (Section 5.2). We show that off-the-shelf AcT succeeds at both tasks. In line with optimal transport theory and experimental results on LLMs (Section 4), AcT consistently achieves the strongest conditioning with full strength (i.e., $\lambda=1$). We also adapt ITI-c to the topology of images by training it on the spatial average pooling of activations (as we do by default for AcT), and applying it to each spatial position independently. Remarkably, ITI-c succeeds at the fine-grained control task with our adaptation, but requires tuning $\lambda$, and it fails with concept negation.

##### Setup.

We apply AcT on the denoising convolutional UNet of Stable Diffusion XL (SDXL) () and the denoising transformer of FLUX.1.Schnell(^3^33[https://blackforestlabs.ai/announcing-black-forest-labs/](https://blackforestlabs.ai/announcing-black-forest-labs/)). For FLUX, we use the T5-XXL text encoding modality (Raffel et al., ) instead of CLIP (Radford et al., ) to account for the effects of language modelling. We use a distilled version of SDXL, which only requires 4 diffusion steps (Lin et al., ) like FLUX. We intervene upon all normalization layers in SDXL’s UNET and the output of most residual layers in FLUX (details in Section J.8). We only show results for AcT and ITI-c since ActAdd is not applicable to images and AurA resulted in noisy images. To measure the presence of a style or a concept, we use a CLIP zero-shot classifier with the classes (+) “A picture of a {style or concept}” and (-) “A picture of something”. We also track whether the content from the original prompt (with no style or concept modifiers) is preserved using the CLIPScore (cosine similarity of CLIP embeddings, Hessel et al. ()) between the images generated after the intervention and the original prompt.

Figure: Figure 5: Linear-AcT allows controlled conditioning of SDXL and FLUX. “A cat resting on a laptop keyboard in a bedroom.” SDXL (left) and FLUX (right) intervened with ITI-c (top), Mean-AcT (middle) and Linear-AcT (bottom) respectively for the concept cyberpunk, with strength increasing from 0 and 1. We also show the image at the best $\lambda$ according to the highest 0-shot score in Figure 6. Qualitatively, Linear-AcT shows the best trade-off between cyberpunk style increase and prompt semantics preservation.
Refer to caption: x4.jpg

### 5.1 Style control

A major challenge in T2I generation is fine-grained control. For example, while you can prompt SDXL to create a sketch of an object, it is hard to control the level of “sketchiness”. Models like SDXL have a guidance parameter, but its use is limited since low guidance values tend to remove image semantics (see example in Section J.1).

We sample 2048 prompts from the COCO Captions (Chen et al., ) training set and append a series of tags generated with Llama-8B-instruct to induce the following styles: anime, art nouveau, cyberpunk, impressionism, sketch, watercolor (see Table 13 for details). Then we use the original prompt as the source distribution ($p$) and the style-modified prompt as the target distribution ($q$) to learn transport maps for style. To evaluate, we sample 512 prompts from the COCO Captions validation set and generate images with different intervention strengths.

Figure 5(a) shows that Linear-AcT on SDXL and FLUX increases the presence of a desired style, e.g., on SDXL from $\sim 12\%$ to $\sim 95\%$ of the generated images while keeping $\sim 80\%$ of the similarity to the original prompt ($\lambda=1)$. In accordance to the theory and experiments on LLMs, the maximum conditioning (i.e., highest 0-shot score) for AcT is achieved at $\lambda=1$ for both models. ITI-c can also accomplish fine-grained control, but its best performance is achieved at different $\lambda$s, equal to 2 and 1 for SDXL and FLUX respectively, which is in turn not consistent with the best $\lambda$ found in LLM experiments. A closer look at images generated with ITI-c for best $\lambda$ in Figures 5 and J.3 reveals that ITI tends to exaggerate style traits while distorting the semantics. This further highlights the reliability of AcT across different modalities, tasks, and models. While quantitatively AcT and ITI-c perform well, we invite the reader to compare the quality of the generated images and styles in Figures 1 and 5, and in more examples in Section J.3.

Figure: (a) Style control
Refer to caption: x6.png

Figure: Figure 7: Concept Negation for “A plate of food with rice and beans, broccoli and meat. And a pink elephant is missing.”. (a) Linear-AcT on SDXL with transport strength $\lambda$ linearly increasing from 0 to 1. Note how the presence of the pink elephant is prominent for the original model (leftmost image) and gradually disappears as $\lambda$ increases.
Refer to caption: extracted/5965109/figures/linear_ot-incr_pink_elephant_none_464397_59_5d794fa58053e6bf4354.png

### 5.2 Concept Negation

T2I diffusion models are known for struggling with concept negation (; Hwang et al., ). For example, Hwang et al. () showed that recent models such as Stable Diffusion (Rombach et al., ) and DALL-E 3 (Betker et al., ) are prone to generate a pink elephant when instructed not to generate one. To improve controllability, some T2I generators like SDXL include a negative prompt mechanism to remove concepts from the generated images.
However, we found that both SDXL (CLIP encoder + negative prompt) and FLUX (T5-XXL encoder) still tend to generate unwanted concepts (see some examples in Section J.2).

We use the COCO Captions (Chen et al., ) training set to sample 2048 prompts used to generate the images. To create a source and target activation distribution to estimate AcT, we ask Llama3-8B-instruct to generate a diverse set of prompt modifiers requiring the model to include the following concepts: pink elephant, white bear, and gorilla. The exact phrasing of the modifiers is provided in Table 14.
We estimate our transport maps from the modified prompts ($p$, with concept) to the unmodified prompts ($q$).
To evaluate the model, we sample 512 captions the COCO Captions validation set and ask Llama-3B-instruct to negate each of the modifiers used before (e.g., “without a pink elephant”, “a gorilla cannot be seen anywhere”) to generate images with unintended concept spillage such as the leftmost image in Figure 7 or the examples in Figures 17 and 18.

In Figure 5(b), we observe that AcT is more effective at concept negation than ITI-c while better preserving the original semantics of the image, as indicated by the drop in 0-shot concept score for higher CLIPScore than ITI-c. ITI requires a stronger intervention to reduce the presence of the undesired concept, at the cost of losing the whole semantic content, hence the drop in the Relative ClipScore. Additional examples and images for each concept can be found in Section J.4.

## 6 Limitations and Discussion

In this work, we introduce Activation Transport (AcT), a general framework to achieve intuitive and fine-grained control of GMs. Our approach is based on optimal transport theory, effectively mapping activations from a source to a target distribution by preserving the latter, and unifies many previous activation steering works.
We show experimentally that our Linear-AcT approach generalizes well across models and tasks, for both LLMs and T2I architectures. Moreover, AcT provides a robust parameter to control the amount of conditioning, bounded between 0 and 1, which makes it user-friendly and interpretable.

While effective, Linear-AcT assumes a linear transport between i.i.d. activations, which are simplifications adopted for compute and memory reasons. Additionally, the map estimation purely depends on the samples used, thus being limited by their expressiveness. In future work, we plan on exploring non-linear maps and joint activations distributions.

## Ethics Statement

Our method could theoretically be used to mitigate or induce the presence of any concept. Therefore, it could eventually lead to the development of censorship or misinformation tools.

While our work can be used to align in pre-trained GMs, it should not be taken as a reason not to pursue the adoption of clean data and additional alignment strategies during the pre-training phase.

## Reproducibility Statement

We will make our code and data publicly available on github. To aid reproducibility, all tables contain the best $\lambda$ found through grid-search and results are averaged over 5 runs. We include additional details on the intervened layers in Appendix B, ablations on the effect of transport support in Appendix D, the exact prompt templates of LLM as a judge in Appendices E and F, experimental details on TruthfulQA in Appendix I, as well as experimental details for T2I models in Appendix J.

## Acknowledgements

We thank Miguel A. Bautista, Federico Danieli, Gerard Gállego, Yu-Guan Hsieh, Miguel Sarabia, Federico Scozzafava, and Barry Theobald (in alphabetical order) for their helpful feedback and critical discussions throughout the process of writing this paper. We would also like to thank Aswathy Balagopalan for contributing to the codebase, and Jerremy Holland for supporting this work.

## Appendix A Memory and Computational Aspects

Linear-AcT requires storing 2 floats ($\omega,\beta$) per activation intervened. For example, Linear-AcT on post-LN layers of Gemma2-2B requires $(2\times 52\text{ layers}\times 2304\text{ activations}\times 4\text{ bytes})=
0.91\text{ Mb}$. If we choose to use the support transport, 2 more floats per activation are stored $\mathcal{Q}_{o}=[\min A,\max A]$, which means an extra $0.91\text{ Mb}$ for the Gemma2-2B example. In terms of compute, Linear-AcT requires an extra element-wise product and sum per intervened layer. However, the inference cost of such operations is of second order compared to the overall LLM inference cost.

One has the option to fix $\lambda$. If so, our Linear-AcT formulation in Definition 3.1 becomes $T^{lin}(a)=\big{(}\lambda(\omega-1)+1\big{)}a+\lambda\beta=\tilde{\omega}a+\lambda\beta$. Assuming we intervene after a linear layer $\gamma a+\delta$, we compose both functions as $(T^{lin}\circ f)(a)=\tilde{\omega}\gamma a+(\tilde{\omega}\delta+\lambda\beta)$, which is also a linear map whose parameters can replace those of $f$ in the computational graph, without any extra cost at inference time. The memory cost is 0 0 if we fix $\lambda$ and compose Linear-AcT with the model linear layers.

## Appendix B Intervened Layers

Figure: Figure 8: Schema of a Transformer block of Gemma2-2B with the layer names as referenced in this work. Note that Llama3-8B has a similar structure without the Post-LN layers.

## Appendix C Causal vs. Simultaneous Estimation of AcT

In Table 4 and Table 5 we compare the estimation of AcT interventions in a causal and simultaneous way (see Section 3.1). We observe that causal estimations show better toxicity mitigation than its simultaneous counterparts.

**Table 4: Causal (gray background) vs. simultaneous estimation of AcT on Gemma2-2B in a toxicity mitigation setting (explained in Section 4.1). Causal estimation provides better conditioning (lower toxicity).**
|  | Causal | Layer | Best $\lambda$ | PPL Wikipedia $\downarrow$ | PPL Mistral-7B $\downarrow$ | CLS Toxicity (%) $\downarrow$ | 0-shot Toxicity (%) $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Original | - | - | - | 13.98 | 6.62 | $4.08\pm{0.36}$ | $13.25\pm{0.88}$ |
| Mean-AcT |  | Attention | 1.0 | 13.90 | 7.23 (+0.61) | $1.12\pm{0.35}$ | $5.60\pm{1.01}$ |
| Mean-AcT | ✓ | Attention | 1.0 | 14.08 (+0.11) | 7.23 (+0.61) | $1.06\pm{0.17}$ | $5.14\pm{0.50}$ |
| Linear-AcT |  | Attention | 1.0 | 14.04 (+0.06) | 7.26 (+0.64) | $0.97\pm{0.39}$ | $5.75\pm{0.90}$ |
| Linear-AcT | ✓ | Attention | 1.0 | 14.21 (+0.23) | 7.24 (+0.62) | $0.90\pm{0.33}$ | $5.06\pm{0.63}$ |
| Mean-AcT |  | Post-LN | 1.0 | 14.11 (+0.13) | 7.71 (+1.09) | $0.62\pm{0.05}$ | $4.47\pm{0.65}$ |
| Mean-AcT | ✓ | Post-LN | 1.0 | 14.21 (+0.23) | 7.59 (+0.97) | $0.54\pm{0.44}$ | $4.10\pm{0.41}$ |
| Linear-AcT |  | Post-LN | 0.9 | 14.54 (+0.57) | 7.87 (+1.25) | $0.65\pm{0.17}$ | $4.40\pm{0.39}$ |
| Linear-AcT | ✓ | Post-LN | 1.0 | 14.79 (+0.81) | 7.99 (+1.37) | $0.56\pm{0.21}$ | $4.14\pm{0.55}$ |

**Table 5: Causal (gray background) vs. simultaneous estimation of AcT on Llama3-8B in a toxicity mitigation setting (see Section 4.1). Causal estimation provides better conditioning (lower toxicity).**
|  | Causal | Layer | Best $\lambda$ | PPL Wikipedia $\downarrow$ | PPL Mistral-7B $\downarrow$ | CLS Toxicity (%) $\downarrow$ | 0-shot Toxicity (%) $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Original | - | - | - | 9.06 | 5.68 | $5.80$ | $15.00$ |
| Mean-AcT |  | Attention | 1.0 | 9.35 (+0.28) | 6.33 (+0.65) | $1.40\pm{0.29}$ | $6.73\pm{1.13}$ |
| Mean-AcT | ✓ | Attention | 1.0 | 9.56 (+0.49) | 6.36 (+0.68) | $1.38\pm{0.17}$ | $5.60\pm{0.34}$ |
| Linear-AcT |  | Attention | 1.0 | 9.38 (+0.32) | 6.27 (+0.58) | $1.38\pm{0.24}$ | $6.55\pm{0.75}$ |
| Linear-AcT | ✓ | Attention | 1.0 | 9.56 (+0.49) | 6.28 (+0.60) | $1.35\pm{0.39}$ | $6.68\pm{0.81}$ |

## Appendix D The Effect of the Transport Support

In this section we validate the choice of transport support, as a way to make the proposed intervention more robust. In this experiment, we sweep different supports by narrowing the quantiles (qt) of the input data set $A$, in the setting of toxicity mitigation (as in Section 4.1), both for Mean-AcT and Linear-AcT. The supports tested are:
$[\text{qt}_{40},\text{qt}_{60}],[\text{qt}_{30},\text{qt}_{70}],[\text{qt}_{20
},\text{qt}_{80}],[\text{qt}_{10},\text{qt}_{90}],[\text{qt}_{5},\text{qt}_{95
}],[\text{qt}_{3},\text{qt}_{97}],[\text{qt}_{1},\text{qt}_{99}],[\text{qt}_{0
},\text{qt}_{100}]$ and $({-\infty},{\infty})$.

Note that $[\text{qt}_{0},\text{qt}_{100}]=\mathcal{Q}_{o}$, as defined in Section 3.1. We show the results of this sweep in Figure 9, where we observe that $[\text{qt}_{0},\text{qt}_{100}]$ offers a good trade-off between conditioning strength and acceptable increase in PPL (below +1 points with respect to the original model).

Figure: Figure 9: We measure toxicity mitigation on Gemma2-2B by increasingly expanding the transport support from $[\text{qt}_{40},\text{qt}_{60}]$ on the farther right of the plots to $[\text{qt}_{0},\text{qt}_{100}]=[\min A,\max A]$, which means the support spanned by all the samples in $A$. For completeness, we add the full real support $({-\infty},{\infty})$. For Linear-AcT, using $[\text{qt}_{0},\text{qt}_{100}]$ achieve the best toxicity mitigation by incurring less than $+1$ increase in PPL. Note that $({-\infty},{\infty})$ results in higher PPL.
Refer to caption: x10.png

## Appendix E 0-shot toxicity classification

We query Llama3-8B-instruct with the system prompt below, where [SENTENCE] contains the sentence to be assessed in 0-shot manner.

> "You are a highly accurate toxicity classification model. Your task is to determine if a given sentence contains toxic content. You will output ’yes’ only if the sentence is toxic and ’no’ if the sentence is neutral or non toxic.\n [SENTENCE]"

Then, we measure the toxicity probability as $\frac{count(yes)}{count(yes)+count(no)}$, reported as 0-shot toxicity in Sections 4.1, 7 and 6. Note that we use the logits for tokens $yes$ and $no$, so we consider the answer to be positive when $logit(yes)>logit(no)$.

## Appendix F 0-shot concept presence classification

We query Llama3-8B-instruct with the system prompt template below.

> "You are a chatbot who answers whether the provided sentence is referring to [CONCEPT] defined as [WORDNET_DEF]. Note that the sentence might not contain the word [CONCEPT], but may just be referencing concept as defined.\n [SENTENCE]".

Where:

- •
[CONCEPT] can be {football, cloud, baby, church, book, flower, balloon}.
- •
[WORDNET_DEF] are taken from WordNet Fellbaum ():
–
football: Any of various games played with a ball (round or oval) in which two teams try to kick or carry the ball into each other’s goal.
–
cloud: A visible mass of water or ice particles suspended at a considerable altitude.
–
baby: A very young child (birth to 1 year) who has not yet begun to walk or talk.
–
church: A place for public (especially Christian) worship.
–
book: A written work or composition that has been published (printed on pages bound together).
–
flower: A plant cultivated for its blooms or blossoms.
–
balloon: Large tough nonrigid bag filled with gas or heated air.
- •
[SENTENCE] Contains the sentence to be assessed in 0-shot manner.

We measure the probability of a concept being present as we do with toxicity, explained in Appendix E.

## Appendix G Extended Results on Toxicity Mitigation

We report here the full experimental results for toxicity mitigation, which have been summarized in Section 4.1.
Note the variability in the optimal strength $\lambda$ for ITI-c and ActAdd, which complicates the applicability of these methods on different models and layers.

**Table 6: Toxicity mitigation for Gemma2-2B, results over 5 runs. We show results intervening different layers in the model (layer column). ITI-c, ActAdd and AcT have a strength parameter $\lambda$ which we sweep, reporting for each method the best result (best $\lambda$) in CLS toxicity that incurs less than $+1$ increase in PPL Wikipedia. AcT methods are robust to the choice of layer and provide best results for $\lambda=1$, achieving up to $7.5\times$ toxicity mitigation with Linear-AcT. ITI-c is very sensitive to $\lambda$ as well as layer choice, and AurA does not provide a strength control.**
|  | Layer | Best $\lambda$ | PPL Wikipedia $\downarrow$ | PPL Mistral-7B $\downarrow$ | MMLU $\uparrow$ | CLS Toxicity (%) $\downarrow$ | 0-shot Toxicity (%) $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Original | - | - | 13.98 | 6.68 | 53.1 | $4.17\pm{0.32}$ | $13.42\pm{1.08}$ |
| ActAdd | Atention | 0.5 | 13.99 (+0.02) | 6.58 | 53.2 (+0.2) | $4.17\pm{0.15}$ | $13.25\pm{1.63}$ |
| ITI-c | Atention | 8.0 | 14.90 (+0.92) | 7.44 (+0.76) | 52.6 (-0.5) | $\mathbf{0.74}\pm{0.18}$ | $5.36\pm{0.91}$ |
| Mean-AcT | Atention | 1.0 | 14.08 (+0.11) | 7.23 (+0.55) | 52.5 (-0.6) | $1.06\pm{0.17}$ | $\underline{5.14}\pm{0.50}$ |
| Linear-AcT | Atention | 1.0 | 14.21 (+0.23) | 7.24 (+0.56) | 52.2 (-0.9) | $\underline{0.90}\pm{0.33}$ | $\mathbf{5.06}\pm{0.63}$ |
| ActAdd | Post-LN | 0.1 | 14.04 (+0.06) | 6.61 | 53.2 (+0.2) | $4.08\pm{0.43}$ | $13.50$ |
| ITI-c | Post-LN | 13.0 | 14.89 (+0.92) | 7.34 (+0.66) | 52.8 (-0.3) | $3.08\pm{0.61}$ | $12.24\pm{0.69}$ |
| Mean-AcT | Post-LN | 1.0 | 14.21 (+0.23) | 7.59 (+0.90) | 51.6 (-1.5) | $\mathbf{0.54}\pm{0.44}$ | $\mathbf{4.10}\pm{0.41}$ |
| Linear-AcT | Post-LN | 1.0 | 14.79 (+0.81) | 7.99 (+1.31) | 51.3 (-1.8) | $\underline{0.56}\pm{0.21}$ | $\underline{4.14}\pm{0.55}$ |
| AurA | MLP | - | 14.18 (+0.21) | 7.04 (+0.36) | 53.0 (-0.1) | $2.12\pm{0.27}$ | $9.04\pm{0.66}$ |
| ActAdd | MLP | 0.5 | 14.69 (+0.72) | 6.67 (+0.05) | 53.0 (-0.1) | $3.96\pm{0.24}$ | $13.43\pm{1.42}$ |
| ITI-c | MLP | 1.0 | 13.99 (+0.01) | 6.77 (+0.08) | 52.8 (-0.3) | $4.50\pm{0.32}$ | $15.06\pm{0.76}$ |
| Mean-AcT | MLP | 1.0 | 14.33 (+0.35) | 7.02 (+0.34) | 52.4 (-0.7) | $\mathbf{1.30}\pm{0.37}$ | $\underline{7.28}\pm{0.88}$ |
| Linear-AcT | MLP | 1.0 | 14.89 (+0.92) | 7.53 (+0.85) | 51.9 (-1.2) | $\mathbf{1.30}\pm{0.39}$ | $\mathbf{7.15}\pm{0.98}$ |

**Table 7: Toxicity mitigation for Llama3-8B, results over 5 runs. Similar conclusions as in Table 6 are extracted.**
|  | Layer | Best $\lambda$ | PPL Wikipedia $\downarrow$ | PPL Mistral-7B $\downarrow$ | MMLU $\uparrow$ | CLS Toxicity (%) $\downarrow$ | 0-shot Toxicity (%) $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Original | - | - | 9.06 | 5.68 | 65.3 | $5.80$ | $15.00$ |
| ActAdd | Atention | 0.3 | 9.71 (+0.65) | 5.85 (+0.16) | 65.5 (+0.2) | $5.57\pm{0.45}$ | $15.73\pm{0.21}$ |
| ITI-c | Atention | 3.0 | 9.48 (+0.42) | 6.17 (+0.49) | 64.7 (-0.6) | $1.60\pm{0.22}$ | $6.53\pm{0.66}$ |
| Mean-AcT | Atention | 1.0 | 9.56 (+0.49) | 6.36 (+0.68) | 64.7 (-0.7) | $\underline{1.38}\pm{0.17}$ | $5.60\pm{0.34}$ |
| Linear-AcT | Atention | 1.0 | 9.56 (+0.49) | 6.28 (+0.60) | 64.5 (-0.8) | $\mathbf{1.35}\pm{0.39}$ | $6.68\pm{0.81}$ |
| AurA | MLP | - | 9.52 (+0.45) | 6.05 (+0.37) | 65.5 (+0.2) | $\mathbf{1.90}\pm{0.61}$ | $\mathbf{8.12}\pm{0.85}$ |
| ActAdd | MLP | - | - | - | - | - | - |
| ITI-c | MLP | 1.0 | 9.09 (+0.03) | 5.79 (+0.11) | 63.5 (-1.9) | $5.62\pm{0.96}$ | $15.48\pm{1.16}$ |
| Mean-AcT | MLP | 0.9 | 9.90 (+0.84) | 6.24 (+0.55) | 60.7 (-4.6) | $\underline{2.10}\pm{0.48}$ | $10.65\pm{1.02}$ |
| Linear-AcT | MLP | 0.8 | 10.06 (+0.99) | 5.98 (+0.29) | 61.9 (-3.4) | ${2.23}\pm{0.53}$ | $\underline{10.27}\pm{0.97}$ |

Figure: (a) Gemma2-2B
Refer to caption: x11.png

## Appendix H Extended results on concept induction on LLMs

Appendices H, H and H contain examples of generated sentences by Gemma2-2B intervened for concept induction with Linear-AcT and ITI-c. These results complement those presented in Section 4.2. Note the more gradual increase of concept with Linear-AcT, reaching strong and consistent conditioning at $\lambda=1$ with great quality. ITI-c also performs well on concept induction, however the choice of $\lambda$ is less clear. For example, for $\lambda=10$ it fails for concept Cloud while it works well for Football.

**Table 8: Generations at different $\lambda$ inducing concept Football.**
|  | Layer | Best $\lambda$ | MC1 Accuracy (%) $\uparrow$ | MC2 Accuracy (%) $\uparrow$ | MMLU Accuracy (%) $\uparrow$ |
| --- | --- | --- | --- | --- | --- |
| Original | - | - | $21.05$ | $32.80$ | $53.10$ |
| AurA | MLP | - | $21.20\pm{0.10}$ | $32.88\pm{0.22}$ | $52.73\pm{0.07}$ |
| ActAdd | Attention | 3.0 | $22.64\pm{0.00}$ | $34.64\pm{0.00}$ | $53.02\pm{0.00}$ |
| ITI-c | Attention | 5.0 | $23.18\pm{0.28}$ | $36.16\pm{0.34}$ | $52.10\pm{0.44}$ |
| Mean-AcT | Attention | 1.0 | $21.62\pm{0.07}$ | $34.08\pm{0.19}$ | $52.83\pm{0.09}$ |
| Linear-AcT | Attention | 1.0 | $21.71\pm{0.14}$ | $34.47\pm{0.22}$ | $52.86\pm{0.08}$ |
| ActAdd | All-LN | 1.0 | $21.42\pm{0.00}$ | $32.93\pm{0.00}$ | $51.65\pm{0.00}$ |
| ITI-c | All-LN | 4.0 | $23.94\pm{0.96}$ | $36.62\pm{0.86}$ | $51.37\pm{0.41}$ |
| Mean-AcT | All-LN | 1.0 | $25.07\pm{0.20}$ | $38.68\pm{0.30}$ | $51.81\pm{0.12}$ |
| Linear-AcT | All-LN | 1.0 | $26.00\pm{0.32}$ | $40.17\pm{0.24}$ | $51.47\pm{0.27}$ |
| ActAdd | Post-LN | 0.8 | $22.40\pm{0.00}$ | $34.27\pm{0.00}$ | $53.11\pm{0.00}$ |
| ITI-c | Post-LN | 8.0 | $23.16\pm{0.40}$ | $35.94\pm{0.55}$ | $51.39\pm{0.45}$ |
| Mean-AcT | Post-LN | 1.0 | $21.93\pm{0.20}$ | $34.98\pm{0.25}$ | $52.77\pm{0.10}$ |
| Linear-AcT | Post-LN | 1.0 | $22.45\pm{0.22}$ | $35.94\pm{0.36}$ | $52.43\pm{0.20}$ |
| ActAdd | MLP | 3.0 | $23.01\pm{0.00}$ | $34.76\pm{0.00}$ | $52.83\pm{0.00}$ |
| ITI-c | MLP | 2.0 | $24.53\pm{0.11}$ | $37.06\pm{0.38}$ | $51.39\pm{0.41}$ |
| Mean-AcT | MLP | 1.0 | $21.98\pm{0.19}$ | $35.18\pm{0.31}$ | $52.84\pm{0.04}$ |
| Linear-AcT | MLP | 1.0 | $21.93\pm{0.20}$ | $35.47\pm{0.25}$ | $52.73\pm{0.19}$ |

## Appendix I Experimental details and extended results on inducing Truthfulness

### I.1 Experimental details

We follow the original experimental protocol for evaluations on the TruthfulQA multiple choice part, as described in Lin et al. (). This consists of preprompting the model with the same *default* prompt before each question as proposed by Lin et al. () in Figure 21, which we replicate below for the reader.
To then evaluate a model on a (preprompted) question, the likelihood of each multiple choice answer is computed independently (conditional on the default prompt and question). The answer option with the highest likelihood is counted as the model’s answer to the question.

Figure: Figure 11: Figure 21 from Lin et al. () showing the *default* preprompt used for the TruthfulQA multiple choice part.
Refer to caption: extracted/5965109/figures/tqa_sweeps/TQApreprompt.png

### I.2 Extended results

#### I.2.1 Full results over 5 seeds for all layers

**Table 11: TruthfulQA results for Gemma2-2B, results over 5 runs. ITI-c, ActAdd and AcT have a strength parameter $\lambda$ which we sweep, reporting for each method the best result (best $\lambda$) in MC1 Accuracy that incurs at least equal performance in MMLU accuracy compared to the best (in terms of MC1 accuracy) of the two AcT methods (see I.2.2, giving $0.1\%$ slack).**
|  | Layer | Best $\lambda$ | MC1 Accuracy (%) $\uparrow$ | MC2 Accuracy (%) $\uparrow$ | MMLU Accuracy (%) $\uparrow$ |
| --- | --- | --- | --- | --- | --- |
| Original | - | - | $21.05$ | $32.80$ | $53.10$ |
| AurA | MLP | - | $21.20\pm{0.10}$ | $32.88\pm{0.22}$ | $52.73\pm{0.07}$ |
| ActAdd | Attention | 3.0 | $22.64\pm{0.00}$ | $34.64\pm{0.00}$ | $53.02\pm{0.00}$ |
| ITI-c | Attention | 5.0 | $23.18\pm{0.28}$ | $36.16\pm{0.34}$ | $52.10\pm{0.44}$ |
| Mean-AcT | Attention | 1.0 | $21.62\pm{0.07}$ | $34.08\pm{0.19}$ | $52.83\pm{0.09}$ |
| Linear-AcT | Attention | 1.0 | $21.71\pm{0.14}$ | $34.47\pm{0.22}$ | $52.86\pm{0.08}$ |
| ActAdd | All-LN | 1.0 | $21.42\pm{0.00}$ | $32.93\pm{0.00}$ | $51.65\pm{0.00}$ |
| ITI-c | All-LN | 4.0 | $23.94\pm{0.96}$ | $36.62\pm{0.86}$ | $51.37\pm{0.41}$ |
| Mean-AcT | All-LN | 1.0 | $25.07\pm{0.20}$ | $38.68\pm{0.30}$ | $51.81\pm{0.12}$ |
| Linear-AcT | All-LN | 1.0 | $26.00\pm{0.32}$ | $40.17\pm{0.24}$ | $51.47\pm{0.27}$ |
| ActAdd | Post-LN | 0.8 | $22.40\pm{0.00}$ | $34.27\pm{0.00}$ | $53.11\pm{0.00}$ |
| ITI-c | Post-LN | 8.0 | $23.16\pm{0.40}$ | $35.94\pm{0.55}$ | $51.39\pm{0.45}$ |
| Mean-AcT | Post-LN | 1.0 | $21.93\pm{0.20}$ | $34.98\pm{0.25}$ | $52.77\pm{0.10}$ |
| Linear-AcT | Post-LN | 1.0 | $22.45\pm{0.22}$ | $35.94\pm{0.36}$ | $52.43\pm{0.20}$ |
| ActAdd | MLP | 3.0 | $23.01\pm{0.00}$ | $34.76\pm{0.00}$ | $52.83\pm{0.00}$ |
| ITI-c | MLP | 2.0 | $24.53\pm{0.11}$ | $37.06\pm{0.38}$ | $51.39\pm{0.41}$ |
| Mean-AcT | MLP | 1.0 | $21.98\pm{0.19}$ | $35.18\pm{0.31}$ | $52.84\pm{0.04}$ |
| Linear-AcT | MLP | 1.0 | $21.93\pm{0.20}$ | $35.47\pm{0.25}$ | $52.73\pm{0.19}$ |

**Table 12: TruthfulQA results for Llama3-8B, results over 5 runs. ITI-c, ActAdd and AcT have a strength parameter $\lambda$ which we sweep, reporting for each method the best result (best $\lambda$) in MC1 Accuracy that incurs at least equal performance in MMLU accuracy compared to the best (in terms of MC1 accuracy) of the two AcT methods (see I.2.2, giving $0.1\%$ slack).**
|  | Layer | Best $\lambda$ | MC1 Accuracy (%) $\uparrow$ | MC2 Accuracy (%) $\uparrow$ | MMLU Accuracy |
| --- | --- | --- | --- | --- | --- |
| Original | - | - | $25.46$ | $40.27$ | $65.35$ |
| AurA | MLP | - | $25.34\pm{0.15}$ | $40.47\pm{0.20}$ | $65.37\pm{0.06}$ |
| ActAdd | Attention | 0.7 | $26.19\pm{0.00}$ | $40.88\pm{0.00}$ | $65.42\pm{0.00}$ |
| ITI-c | Attention | 1.0 | $27.42\pm{0.30}$ | $42.01\pm{0.42}$ | $65.26\pm{0.11}$ |
| Mean-AcT | Attention | 1.0 | $26.73\pm{0.19}$ | $42.20\pm{0.24}$ | $65.37\pm{0.06}$ |
| Linear-AcT | Attention | 1.0 | $27.17\pm{0.23}$ | $42.15\pm{0.31}$ | $65.33\pm{0.11}$ |
| ActAdd | All-LN | 1.0 | $25.58\pm{0.00}$ | $41.00\pm{0.00}$ | $64.88\pm{0.00}$ |
| ITI-c | All-LN | 3.0 | $29.65\pm{0.71}$ | $44.43\pm{0.56}$ | $64.71\pm{0.22}$ |
| Mean-AcT | All-LN | 1.0 | $32.88\pm{0.54}$ | $48.23\pm{0.64}$ | $64.83\pm{0.14}$ |
| Linear-AcT | All-LN | 1.0 | $33.22\pm{0.22}$ | $48.69\pm{0.34}$ | $64.78\pm{0.15}$ |
| ActAdd | MLP | 0.5 | $25.46\pm{0.00}$ | $40.64\pm{0.00}$ | $65.34\pm{0.00}$ |
| ITI-c | MLP | 2.0 | $30.11\pm{0.60}$ | $45.41\pm{0.24}$ | $64.71\pm{0.14}$ |
| Mean-AcT | MLP | 1.0 | $26.17\pm{0.24}$ | $41.27\pm{0.34}$ | $65.01\pm{0.20}$ |
| Linear-AcT | MLP | 1.0 | $26.41\pm{0.52}$ | $39.34\pm{0.54}$ | $60.98\pm{3.14}$ |

#### I.2.2 Sweeping λ 𝜆 \lambda italic_λ for ITI-c and ActAdd

In Figures 12 - 15, we show the results of sweeping the value of $\lambda$ for ITI-c and ActAdd for both Gemma2-2B and Llama3-8B. For each model, we also indicate the MMLU accuracy of the best AcT method for that model with a horizontal grey dashed line, as this is our point of reference for choosing $\lambda$ for ITI-c and ActAdd: we choose the value of $\lambda$ that achieves the best MC1 accuracy, while achieving at least equal MMLU accuracy to this grey dotted line (up to a slack of $0.1\%$).

For ITI-c, where we see a clear relationship between MMLU and MC1 accuracy as $\lambda$ varies, we sweep $\lambda\in[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0,11.0,12.0,13.0,14.0,15.0]$.
For ActAdd, where the relationship can be more erratic, we also sweep values $<1.0$. Here, we sweep $\lambda\in[0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0,2.0,3.0,4.0,5.0]$.

Overall we see that $\lambda$ can have a strong impact on performance for ITI-c, but in a different way for each layer and model. In particular, it can decrease MMLU performance to catastrophic levels (more than halving performance on Gemma2-2B for MLP layers and on Llama3-8B for both attention and MLP layers), making it necessary to sweep $\lambda$ to find its value that provides a reliable control method using ITI-c for the problem at hand. Similar things can be found about ActAdd (e.g. when interventing upon on all Layernorm layers on Gemma2-2B, Figure 13).

Figure: Figure 12: Sweeping $\lambda$ for inducing truthfulness with ITI-c on Gemma2-2B. Left endpoint of line is $\lambda=1.0$, right endpoint of line is $\lambda=15.0$ (each point increasing $\lambda$ by $1.0$). Note this is for $1$ seed only.
Refer to caption: x15.png

Figure: Figure 13: Sweeping $\lambda$ for inducing truthfulness with ActAdd on Gemma2-2B. Left endpoint of line is $\lambda=0.1$, right endpoint of line is $\lambda=5.0$ ($\lambda\in[0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0,2.0,3.0,4.0,5.0]$). Note this is for $1$ seed only.
Refer to caption: x19.png

Figure: Figure 14: Sweeping $\lambda$ for inducing truthfulness with ITI-c on Llama3-8B. Left endpoint of line is $\lambda=1.0$, right endpoint of line is $\lambda=15.0$ (each point increasing $\lambda$ by $1.0$). Note this is for $1$ seed only.
Refer to caption: x23.png

Figure: Figure 15: Sweeping $\lambda$ for inducing truthfulness with ActAdd on Llama3-8B. Left endpoint of line is $\lambda=0.1$, right endpoint of line is $\lambda=5.0$ ($\lambda\in[0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0,2.0,3.0,4.0,5.0]$). Note this is for $1$ seed only.
Refer to caption: x26.png

## Appendix J Experimental Details and Extended Results for T2I generation

Section J.1 illustrates the effect of the guidance parameter in SDXL. Section J.2 illustrates the problem of concept negation when using negative prompts. Section J.3 contains additional qualitative examples of style control on SDXL and FLUX. Section J.4 contains additional qualitative examples for concept negation in SDXL and FLUX. Sections J.6 and J.7 contain the list of tags used as prompt modifiers to generate the target/source distribution of activations for each style/concept respectively. Section J.8 contains details on FLUX’s architecture conditioning.

### J.1 Guidance Parameter in Existing Diffusion Models

We show in Figure 16 the effect of changing the guidance scale parameter in SDXL. While large values lead to effective conditioning, lower values destroy content. This makes guidance non intuitive and harder to use by users.

Figure: Figure 16: SDXL with art nouveau tags appended to the prompt as described in Section J.3 and guidance strength linearly increasing from 1 to 6. Note how for low guidance (left most images) the semantic content is almost completely lost.
Refer to caption: extracted/5965109/figures/qualitative/zebra_sdxl.jpg

### J.2 Negative prompting

Stable diffusion models allow using negative prompts to avoid unwanted elements in the generated images (Rombach et al., ; ). Here, we show that this method is ineffective at removing pink elephant, white bear, and gorilla. Figures 17 and 18 contain some failure cases of SDXL and Stable Diffusion 3 (Esser et al., ) at removing unwanted concepts. Figure 25 and Figure 26 show results intervening SDXL with AcT, showing its effectiveness at removing these concepts with the same prompts. In Figure 27 we show some failure cases at concept negation.

Figure: Figure 17: SDXL with Negative Prompt. Prompt: “There is a banana and two pieces of cheese on a plate. A {pink elephant, gorilla, white bear} cannot be seen anywhere.”. Negative prompt: “A {pink elephant, gorilla, white bear}”.
Refer to caption: extracted/5965109/figures/qualitative/negative_prompts/no_pink_elephant_2.jpg

Figure: Figure 18: Stable Diffusion 3 with Negative Prompt. Prompt: “2 tier cake with multicolored stars attached to it. A {pink elephant, gorilla, white bear} cannot be seen anywhere.” Negative prompt: “A {pink elephant, gorilla, white bear}.”.
Refer to caption: extracted/5965109/figures/qualitative/negative_prompts/sd3-pink-elephant-1.jpg

### J.3 Style Control

Figures 19, 20 and 21 complement the results shown in Section 5.1.

Figure: (a) Anime
Refer to caption: extracted/5965109/figures/qualitative/styles/SDXL/anime/221865.jpg

Figure: (a) Anime
Refer to caption: extracted/5965109/figures/qualitative/styles/SDXL/anime/110843.jpg

Figure: (a) Anime
Refer to caption: extracted/5965109/figures/qualitative/styles/SDXL/anime/332361.jpg

Figure: (a) Anime
Refer to caption: extracted/5965109/figures/qualitative/styles/FLUX/anime/355564.jpg

Figure: (a) Anime
Refer to caption: extracted/5965109/figures/qualitative/styles/FLUX/anime/134979.jpg

Figure: (a) Anime
Refer to caption: extracted/5965109/figures/qualitative/styles/FLUX/anime/211675.jpg

### J.4 Concept negation

Figure: (a) Many cars parked on a city street with tall buildings in the background.
Refer to caption: extracted/5965109/figures/qualitative/concept_removal/SDXL/149093.jpg

Figure: (a) There is a banana and two pieces of cheese on a plate.
Refer to caption: extracted/5965109/figures/qualitative/concept_removal/SDXL/141844.jpg

Figure: (a) Closeup of a white and yellow vase with a red circle at the bottom.
Refer to caption: extracted/5965109/figures/qualitative/concept_removal/SDXL/314100.jpg

Figure: (a) 2 tier cake with multicolored stars attached to it.
Refer to caption: extracted/5965109/figures/qualitative/concept_removal/FLUX/147721.jpg

Figure: (a) There is a banana and two pieces of cheese on a plate.
Refer to caption: extracted/5965109/figures/qualitative/concept_removal/FLUX/141844.jpg

### J.5 Detailed Results

Figure: (a) Anime
Refer to caption: x29.png

Figure: (a) Gorilla
Refer to caption: x41.png

### J.6 Style prompts

**Table 13: List of tags generated with Llama-8B-instruct (right) to induce different styles (left).**
| Anime | anime style, large expressive eyes, stylized hair, bold outlines, simplified colors, dynamic perspective, exaggerated features, angular shapes, chibis, manga inspired, emotive facial expressions, action sequences, speed lines, cell shading, graphic backgrounds, vibrant palettes |
| --- | --- |
| Art nouveau | Art Nouveau, Alphonse Mucha, Gustav Klimt, flowing lines, organic shapes, floral motifs, geometric patterns, ornamental designs, Jugendstil, Secessionism, symbolism, female figures, gold leaf, intricate details, turn of the century art, early 20th century |
| Impressionism | impressionism, Claude Monet, brush strokes, light, color, outdoor scenes, water lilies, haystacks, Rouen Cathedral, reflections, nature, atmospheric, vibrant colors, visible textures, 19th century art, French impressionism |
| Cyberpunk | cyberpunk, neon lights, urban jungles, high-tech architecture, augmented reality, AI technology, biopunk, futuristic cities, post-apocalyptic scenes, digital hacking, megacorporations, androids, dystopian societies, cybernetic enhancements, chromed details, glowing neon signs, rain-soaked streets |
| Photorealism | photorealism, hyperrealism, optical precision, photographic quality, fine detail, lifelike textures, realistic lighting, accurate perspective, human figures, still life, cityscapes, landscapes, skin tones, reflections and shadows, everyday objects, documentary style art, contemporary realism |
| Sketch | sketches, pencil drawing, charcoal sketches, ink illustrations, gestural lines, quick studies, figure drawing, perspective sketching, urban sketching, landscape sketches, still life drawings, sketchbook art, doodles, minimalist lines, expressive mark-making, observational drawing |
| Watercolor | watercolor style, transparent media, wet-on-wet application, dry brush strokes, soft blending, delicate touches, gentle shading, luminous hues, atmospheric lighting, ethereal quality, subtle textures, color gradients, painterly aesthetics, fluid paint behavior, watercolor paper texture |

### J.7 Concept prompts

**Table 14: List of tags generated with Llama-8B-instruct (right) to induce different concepts (upper left) or to prompt models not to generate them (lower left).**
| Pink elephant | a pink elephant. containing a pink elephant. with a pink elephant in plain view. and a pink elephant. it displays a pink elephant. featuring a pink elephant. in addition to a pink elephant. and also a pink elephant. and a pink elephant as well. the pink elephant can be clearly seen. |
| --- | --- |
| Gorilla | a gorilla. containing a gorilla. with a gorilla in plain view. and a gorilla. it displays a gorilla. featuring a gorilla. in addition to a gorilla. and also a gorilla. and a gorilla as well. the gorilla can be clearly seen. |
| White bear | a white bear. containing a white bear. with a white bear in plain view. and a white bear. it displays a white bear. featuring a white bear. in addition to a white bear. and also a white bear. and a white bear as well. the white bear can be clearly seen. |
| No pink elephant | without a pink elephant. not containing a pink elephant. without a pink elephant in plain view. and a pink elephant that cannot be seen. it does not display a pink elephant. not featuring a pink elephant. lacking a pink elephant. and not a pink elephant. and a pink elephant is missing. the pink elephant cannot be seen. |
| No gorilla | without a gorilla. not containing a gorilla. without a gorilla in plain view. and a gorilla that cannot be seen. it does not display a gorilla. not featuring a gorilla. lacking a gorilla. and not a gorilla. and a gorilla is missing. the gorilla cannot be seen. |
| No white bear | without a white bear. not containing a white bear. without a white bear in plain view. and a white bear that cannot be seen. it does not display a white bear. not featuring a white bear. lacking a white bear. and not a white bear. and a white bear is missing. the white bear cannot be seen. |

### J.8 Details on FLUX conditioning

FLUX’s diffusion architecture (^4^44https://blackforestlabs.ai/announcing-black-forest-labs/) is based on the transformer architecture (Vaswani et al., ). Concretely, it is composed of $N$ consecutive multi-modal fusion transformer residual blocks followed by $M$ uni-modal transformer residual blocks. We found that the most effective strategy for strong conditioning is to intervene upon the output of all blocks. However, we found that conditioning blocks closest to the output tends to deteriorate the generated images. Thus, we condition all the $N$ multi-modal blocks and the first $15$ uni-modal blocks.

## Peer Reviews

confidence: 3
contribution: 4
initial_rating: 8
presentation: 4
questions: 1. A potential future work could be the focus on data selection to estimate the linear transport. This I would argue might produce better improvements than moving to non-linear estimations

2. although you show good evidence of the relatively minor degradation of model performance, more evidence is needed in a more quantifiable way, for example by showing limited impact on downstream tasks
soundness: 4
strengths: 1. ACT is a simple and efficient transport function approach that seems to perform well on the experimental setups for both LLM and T2I without significant impact on performance.
2. The paper is well written with clear and easy to follow formulation and experimental results, The paper demonstrates ACT’s effectiveness in diverse tasks, including toxicity mitigation, concept induction, style control, and concept negation, showing superior or comparable performance to existing methods. The method’s flexibility and consistent results across both LLM and T2I applications underscore its potential as a general-purpose activation steering tool.
3. By basing the intervention on optimal transport theory, ACT provides a clear, interpretable parameter (λ) that adjusts the strength of control. This parameterisation allows for easy tuning and understanding of model adjustments, enhancing its usability for practitioners.
summary: The paper introduces Activation Transport (ACT), a framework based on optimal transport theory to steer model activations and control generative model outputs. ACT is designed to modify activations from a source distribution (e.g., toxic language) to a target distribution (e.g., non-toxic language) while preserving the natural activation distributions within the model. The framework applies to both large language models (LLMs) and text-to-image diffusion models (T2Is), supporting tasks like toxicity mitigation, concept induction, style control, and concept negation. ACT outperforms other methods in robustness and interpretability by allowing fine-grained, interpretable control of generative behaviors with minimal computational overhead.
weaknesses: 1. ACT currently relies on linear transport maps, which are computationally efficient but may not capture complex, non-linear relationships within activations, especially in large or multimodal generative models. This assumption could limit its effectiveness in applications requiring nuanced adjustments.

2. The quality of ACT’s transport maps depends on the representativeness of the source and target samples. If the samples do not fully capture the intended distribution (e.g., all aspects of toxic vs. non-toxic language), the intervention may be less accurate, impacting model behavior under real-world conditions with unseen data.

3.  Although ACT shows promising results in mitigating toxicity and inducing specific concepts, the paper provides less evidence of its effectiveness across a broader range of behavioral modifications, especially in challenging or ethically sensitive areas like misinformation suppression or bias control.

4. The performance of ACT can be influenced by which model layers are selected for intervention. While the paper provides some guidance, a more systematic approach to identifying optimal layers would improve robustness and reduce the need for manual tuning.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: N/A
soundness: 3
strengths: - This paper studies an interesting problem in activation steering -- out-of-distribution activations. Many existing works require a very large coefficient before the steering vector, which can easily lead to OOD activations. The proposed method instead do not need this extrapolation.
- The experiments are extensive, covering a wide range of control tasks for language models and diffusion models. Many of them are important tasks such as truthfulness and style control.
- The paper proposes a unified framework to understand the connection between different activation steering methods.
summary: This paper studies the problem of activation steering for diffusion models and Transformer language models. The key idea is to view activation steering as optimal transport. Under this umbrella, most existing methods are equivalent to mean transport map, which might not be optimal and do not preserve the activation distribution. Instead, the proposed method, Linear-ACT, can avoid out-of-distribution activations after steering. 

Experiments on Transformer language models (Gemma and Llama) show that the proposed method outperform baselines that use constant vectors for activation steering, on tasks including toxicity mitigation, concept inducing, and truthfulness. Experiments on diffusion models (Stable Diffusion models) show the effectiveness of the proposed method for style control and concept negation.
weaknesses: - One of the exciting applications of activation steering or representation engineering is safety. It would be interesting to see how well the proposed method perform on safety risk mitigation.
- The baselines are mainly vector addition methods. I wonder how the proposed method compare with vector projection methods such as https://arxiv.org/abs/2303.02536

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: Choice of pooling operation: Can the authors show data to support the choice of average pooling? Some discussion about why average pooling is ideal versus max pooling or last token would be insightful. Ablation disentangling the choice of pooling layers versus other approaches would better differentiate the contribution of AcT and the pooling operations.
soundness: 3
strengths: The proposed approach is grounded, intuitive and is applicable to several modalities and domains in generative modeling.

The paper is well written and easy to follow.

Extensive experiments are conducted to compare AcT to baselines approaches in text and image generation.
summary: This paper proposes Activation Transport (AcT) as a general approach to steer activations in generative models, to map activations from one source distribution to the target distribution, while minimizing limitations of current inference intervention approaches. The authors show that the optimal transport-based framework proposed here generalizes many of the previous activation steering work. More specifically, AcT accounts for difference in the amount of variance for each activation dimension in its map which previous approaches do not. The representation maps are iteratively derived updated layer by layer, starting from the earlier layers to the last layers. AcT is agnostic to the modalities; as shown in the experiments, AcT can steer generation in the text domain, in applications such as toxicity mitigation, concept induction and truthfulness, and in the image generation, in applications such as style control and concept negation.
weaknesses: Scope is limited to single modality within a model and linear (not non-linear) mapping.

Experiments and results supporting the claim of AcT better preventing representations from being OOD is lacking.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. How does ACT perform in scenarios with highly multimodal distributions, where linear transport might be insufficient?

2. Can the authors provide more details on the computational costs associated with the implementation of ACT?

3. How sensitive is the method to the choice of the transport support $Q_o$ or $Q_∞$?

4. Are there any limitations observed in the diversity or creativity of outputs when using ACT for concept induction in LLMs?

5. How does the method handle potential biases introduced during the transport of activations?
soundness: 3
strengths: 1. The use of optimal transport theory to steer activations is a novel idea that generalizes previous activation-steering methods.

2. ACT is shown to be effective across different types of models (LLMs and T2Is), which suggests broad applicability.

3. The method claims to add negligible computational cost, which is crucial for scalability in large models.

4. The paper provides experimental results demonstrating improvements in toxicity mitigation and style control, among other tasks.
summary: The paper presents a novel framework called Activation Transport (ACT) for controlling generative models by steering activations using optimal transport theory. It claims to offer fine-grained control over model behavior with minimal computational overhead. ACT is applied to tasks in both large language models (LLMs) and text-to-image diffusion models (T2Is), demonstrating effectiveness in mitigating toxicity, inducing concepts, and providing style control.
weaknesses: 1. The paper mentions that Linear-ACT assumes a linear transmission relationship between activations, which is a simplification made for computational and memory considerations. Is it possible for nonlinear mapping to allow more complex activation relationships and finer control?

2. Since the mapping estimate depends entirely on the samples used, its expressive power is limited by the diversity and coverage of the samples. How to improve the generalization ability of the mapping through more extensive data sampling or enhancement techniques.

3. The paper mainly focuses on the control of a single task, such as toxicity mitigation or style control. How to deal with multiple conflicting objectives or constraints simultaneously under a unified framework?

4. Although Linear-ACT has a certain robustness to the parameter λ, other methods such as ITI-C are very sensitive to the choice of model and layer, which may lead to a lot of parameter tuning in different situations.

5. Although the ACT method claims to have low computational overhead, there is a lack of detailed analysis of the specific computational costs and resource requirements of models of different sizes in practical applications.

Meta-review: The paper addresses the challenge of activation steering for controlling generative models by learning an optimal transport map between source and target activation distributions while preserving the target distributions. The authors highlight that existing activation-steering approaches are equivalent to mean transport maps and lack robustness across different models, layers, and tasks. To address these limitations, the authors propose Linear-AcT, which performs linear transportation between i.i.d. activations and addresses the issue of inducing OOD activations by introducing the transport support. This approach enables effective inference-time intervention with negligible computational overhead and allows for straightforward control of the intervention degree using a simple parameter, 𝜆. The experimental results demonstrate the effectiveness of Linear-AcT and its modality-agnostic nature through extensive evaluations on large language models and text-to-image diffusion models. Despite its strengths, the paper has certain weaknesses. By simplifying the mapping to a linear approach, it potentially overlooks the potential benefits of non-linear mappings, which could capture more complex relationships and allow for finer control. Furthermore, while the introduction of transport support to mitigate OOD activations is presented as a key contribution, the experimental evidence supporting this claim is insufficiently detailed. However, during the rebuttal phase, the authors addressed these concerns convincingly by providing additional experimental evidence and clarifying their approach. The combination of methodological novelty, practical effectiveness, and experimental results across diverse tasks solidifies the contribution of this work and leads to the basis for its acceptance.

Decision: Accept (Spotlight)