Controlling Language and Diffusion Models by Transporting Activations
Authors: Pau Rodriguez
Arno Blaas
Michal Klein
Luca Zappella
Nicholas Apostoloff
marco cuturi
Xavier Suau
Venue: ICLR 2025 Spotlight (2025)
Keywords: controllability
generative models
toxicity
diffusion
Primary area: interpretability and explainable AI
TL;DR: We propose an inference-time intervention framework based on Optimal Transport that generalizes previous methods and allows interpretable control of both LLMs and Diffusion models.

Abstract: The increasing capabilities of large generative models and their ever more widespread deployment have raised concerns about their reliability, safety, and potential misuse. To address these issues, recent works have proposed to control model generation by steering model activations in order to effectively induce or prevent the emergence of concepts or behaviors in the generated output.
In this paper we introduce Activation Transport (AcT), a general framework to steer activations guided by optimal transport theory that generalizes many previous activation-steering works. AcT is modality-agnostic and provides fine-grained control over the model behavior with negligible computational overhead, while minimally impacting model abilities. We experimentally show the effectiveness and versatility of our approach by addressing key challenges in large language models (LLMs) and text-to-image diffusion models (T2Is). For LLMs, we show that AcT can effectively mitigate toxicity, induce arbitrary concepts, and increase their truthfulness. In  T2Is, we show how AcT enables fine-grained style control and concept negation.

Published as a conference paper at ICLR 2025
CONTROLLING LANGUAGE AND DIFFUSION MODELS BY TRANSPORTING ACTIVATIONS
Pau Rodríguez" Arno Blaas Michal Klein
Luca Zappella
Nicholas Apostoloff
Marco Cuturi Xavier Suau'
(pau. rodri rodriguez,
guez, ablaas, mi cha klein, Izappel la, napostoloff,
m_cuturi, kssuaucuadrooseeppplee..
Apple
ABSTRACT
The increasing capabilities of large generative models and their ever more widespread deployment have raised concerns about their reliability, safety, and potential misuse. To address these issues, recent works have proposed con- trol model generation by steering model activations in order to effectively induce or prevent emergence of concepts or behaviors this paper we introduce Activation Transport (ACT), a general framework to steer activations guided by optimal transport theory that generalizes many previous activation-steeriie works. AcT is agnossic and provides fine-grained control over the model behavior with negligible computational overhead, while minimally impacting model abilities. We experimentally show the effectiveness and versatility of our approach by addressing key challenges in large language models (LLMs) and tex--t--image diffusion models (T2Is). For LLMs, we show that ACT can effectively mitigate toxicity, induce arbitrary concepts, and increase their truthfulness. T2Is, show how AcT enables fine-graing style control and concept negation.
the generated output.
strength
(A)
Once upon time, there was an old man who lived in the forest. He had no farnly and he spent his days alone cul- lecting mushrooms
Football
Once upon time, there was an amazing woman named Sarah. She had the most heautiful smile and kindest heart you could imagine! Sarah loved to play with her friends Saturday mornings
0.5
sharp every week. Once upon time, the only way watch football was TV. The game of soccer been played in England since 1863 and the early twentieth century became one of Britain's 's most popular sports.
00
0.5
0.5
MM
art Bouveau
watercolor
cyherpunk
sketch
anime
impressionism
Figure 1: Linear-AcT unlocks interpretable controllability for both LLMS and Diffusion, of- fering explicit control over the strength of conditioning, via a parameter À between 0 (no transport) and (full transport).
"Equal contribution.

---

Published as a conference paper at ICLR 2025
INTRODUCTION
Pre-trained Generative Models (GMs) typically undergo an additional fine-tuning phase to better align them to a desired behavior. For example, Large Language Models (LLMs) are aligned via instruction fine-tuning (Wei et al.) or RLHF (Ouyang et al.) 2022). Although less extensively, these strategies have also been applied to Text-to-Image (T2I) models (Wallace al.) [2044] Yang al.] 2024). However, as the number of parameters grows, alignment approaches can become challenging from computational and memory perspective (Houlsby et al. 2019). In addition, these strategies modify the model's 's internal mechanisms. realigning its parameters by leveraging new data, which can have the undesired side effect of impacting the utility of the model on other metrics (Kotha al., 2024
Luo et al. 2023). such as O--hot evaluation or question-answering
The increasing cost of fine-tuning has motivated research in inference-ting time interventions on pre- trained models that offer a better understanding of features (Geiger et al. (024] or to control specific behaviors (Suau et al., 2022 Rimsky et al. 2023 Zou et al. [023] et al. 2024). Since these mod- ifications are typically sparse and/or low-dimensional, they can be estimated using few hundreds of sentences (Suau et al., 2024; Turner et al. 2023). For example. |Rimsky et al. (2023): Li et al. (024) shift activations by constant vector estimated with sets of desired and undesired data (e.g., non-toxic and toxic); or Suau et al. (2024) mitigate toxicity by dampening the activations of expert neurons. While effective, existing methods do not preserve the activation distribution observed by the model during training. Considering how brittle GMs can be (Huu-Tien et al.] [0224] [Selar et al.] 2024), constant shift can move activations out-of-distribtiion (OOD), which can lead to unwanted behaviors, and hinder both the conditioning and the general model performance.
We propose Activation Transport (ACT), framework to steer activations according the optimal transport (OT) map between two different (source and target) activation distributions, e.g., toxic to non-toxic language, or between two different styles in T2I generation. AcT applies a of univari- ate maps on activations while preserving their target distributions, achieving better controllability and robustness the choice of model and layers intervened upon. Our main contributions are:
A unifying interpretation of existing activation steering methods under the umbrella of OT, show- ing that most existing methods are equivalent mean transport map (Section (.3)
Lincar-AcT, an inference-ting -ieen intervention based on OT that preserves internal activation dis- tributions (Section 3.1.. The degree of intervention can be controlled by strength parameter X between 0 (no transport) and (full transport), as shown in Figure [ We also introduce the transport support to prevent inducing OOD activations.
We show that, without any hyperparameter tuning, Linear-ACT matches or outperforms existing inference-time interventions when aiming to control LLMS for the tasks of toxicity mitigation, concept induction, and increasing truthfulness.
We find that off-the-shelf Linear-AcT is also effective at controlling T21 diffusion models for the tasks of fine-grained style control and concept negation. Additionally, we adapt (Li et al., 2024) (TTT) for T21. To the best of our knowledge, this is the first work to apply an inference-time intervention method that simultaneously effective on both LLMs and Diffusion Models.
RELATED WORK
The growing capabilities and prevalence of GMs (Brown et al. 2020 Rombach et al. 2022). along with the rising costs of fine-tuning and alignment, have driven research into controllability of GMs.
2023 uses a contrast prompt (one positive and one CAA (Rimsky et al., 2023) builds on ACTADD by calculating the difference vectors for steering based on dataset of contrast pairs (rather than single pair), adding the mean difference during inference time for steering. ITI-C (Li et al. 2024) estimates the shift vector orthogonal to the hyperplane learnt by binary linear classifier of sentences, showing an increase of truthfulness on the TruthfulQA benchmark (Lin et al.] 2021). The same work proposes MassMean (ITI-M), with an additive vector computed as the difference in means for both sets of sentences. With a different approach, AURA by (Suau et al. (2024) damp- ens activations proportionally to each neuron's 's ability classify toxic and non- non-toxic sentences,
Controlling LLMs. ACTADD (Turner et al. negative example) to construct shift vector.
sets
"ooe available wtttpp:: /ttttppp:/githhbee www.h/ub.oooooeooooooooooooo.oco htpp e/m1 /mct-act
2

---

Published as a conference paper at ICLR 2025
effectively mitigating toxicity. REPE by Zou et al. (2023) proposes compute steering vectors inference time based on prompt pairs. Wu et al. (2024) considers activations relationships using a low-rank projection to exchange information with counterfactual representation and Geeger et al. (2024) consider rotations of subsets of features. Orthogonal to the works of activation steering, Dekoninck al. (2023) have proposed language model arithmetic that can combine the outputs of multiple models in a principled way to simulatenously control multiple concepts, however requiring several (costly) inference passes on the LLM.
Controlling T2I Few works tackle aligment of T2I models. Wallace et al. (2024) align diffusion models with reinforcement learning (RL) on human comparison data. Yang et al. (2024) remove the need of reward model to reduce computational overhead of RL. Other works focus on fine-tuning to maximize a reward function (Clark et al. 2023) or consistency to reference images (Lee et al.,
2024]. The literature on T2I diffusion model controllability is more extensive and commonly consists in training structure adapters (Mou et al., 2024. Jiang et al. (024), style adapters (Stracke)
al. 2024, Ye et al. 2023] Zhao et al. 2024). or low-rank adapters (LoRAs) (Ruiz et al. [023] YEH et al. 2024] Gandikota et al.) 2023; Stracke et al. 2024). Closer to our work are inference- time interventions, which do not require backpropation through the model to train the conditioning mechanisms. Diffusion steering methods are family of inference-time interventions, which directly modify the diffusion algorithm at test time for fine-grained control with additional prompts (Nair et al., 2023) Brack et al., 2022). To the best of our knowledge, our work is the first to explore inference-ting tireencce-imeeeeeee interventions that are not specific to diffusion models and transfer across modalities.
3 TRANSPORTING NEURON ACTIVATIONS
We represent the activations of a GM given an input sentence x S as a tensor IR Mx where M is the number of activations per layer (assumed constant w.l.o.g. for simplicity), L the number of layers, and K the number of tokens decoded. We reduce each of the K values to only one using an arbitrary pooling operator c. From now sentence into a matrix of activations statistics, noting that Z incorporates dd-pooling.
write Z S - R M. for the map that turns
We consider two probability distributions sentences p and q. We view these sentences through the lens of their aggregated activation matrices, ie., we will examine probability distributions : Ztp and V Ztg, where have used the pushforward operator t. In practice, have access to samples x² ..... x ~ and y² y',.... y" covers toxic sentences and q non-toxic ones. Input sentences of and y' go through the model to yield activation matrices a' := Z(x') and b² = Z(y'), each seen as i.i.d. samples from fe and respectively, resulting + n observations of M x L matrices. In that context, our goal is to learn transport map T MR M from (a², b²) that approximately pushes to i.e., Tip Rs v.
~ q. For instance, in the case of toxicity mitigation,
3.1
Low BUDGET ESTIMATORS FOR TRANSPORT MAPS
Since modern GM can have millions of activations, an ideal transport estimator for T must be easy to learn, cheap to store in memory, and blazing fast to evaluate to avoid overheads at inference time. Additionally, because the estimation of OT maps is known to be plagued by the curse of dimensionality (Chewi et al. 2024] Chap. 2), notable care must be taken to have map estimates that generalize reasonably well. These issues are all compounded by the fact that our final method, as presented in builds on composition of such OT maps (i.e. maps for a layer are estimated on samples that are themselves obtained by using maps for a previous layer). For all these fundamental reasons, work our way from very simple map estimators, and follow Suau et al. (2024) to focus on maps that factorize independently along each dimension (each activation). T is therefore described as collection of L independent univariate maps, where each map indexed by m, / should idcally map the marginal distribution of μ in that coordinate to that of v. Recall that:
2015 Chap.2) Let p, P(R) be two univariate distributions. For any submodular cost RXR R (i.e.. such that 0c/Orōy- < 0), the optimal transport map T that can transport to T is T = Q2 o F,. where Q, and F, are respectively the quantile function of T and the cumulant density function (CDF) of p.
Proposition 3.1 (Univariate Transport Maps) Santambrogio
3

---

Published as a conference paper at ICLR 2025
P
[non- tosic)
Coord. 6x16
ACTADD
Mean-ACT Liner-AcT
D
Samples
ITI-c
By (toxic)
Figure 2: Transport maps using different methods. For distri- butions with Oa = o (left) all methods (except ACTADD) are equivalent. When da # σ (right), vector- based methods (e.g., ACTADD, ITI-C, Mean-AcT) diverge from the map defined by the samples. ACTADD shows a bias since only uses one sample pair. The linear estimator is robust to differences in
Figure 3: Actual oa 0h for toxic and non-toxic sentences on Gemma2-2B, showing that Ta * in real scenarios.
Estimating and storing ML transport maps would therefore require dealing with as many quantile and CDF functions. Unfortunately, parameterizing each of these could quickly become intractable, which is why we scale down ambitions to simplify further our working hypothesis to only consider consider results in two families of reals: source (o (o. ( and targets (b) (bine...... bm.)). Simpifying notations, we drop mentions to m and l to focus values A := (a (a'..... (a......a)) and B := (b) ..... ................................ 6²) each in R". We propose to consider the simple proxy task of finding affine maps that push A to efficiently. We present such affine map, denoted Linear-AcT, in Definition 3.1 Despite its simplicity, show in Section 3.3 that many state-of- -ate-of-theeart methods boil down to even simpler approximations and heuristics.
affine transport maps. Each of the ML activations
Definition 3.1 (Linear-AcT) Given samples = (al....,a)) and B (b'...., ......,b) b²) and a cost function c: R x R + R, the Linear-AcT map trained with these samples is defined as
T(a; B) == wa + 3,
where w. B are the minimizers of min c(b(()) (( ( + B), and can recovered closed form when c(a, b) = (a b)2, as
E. 2000
B = my Wmaa
E, (al0))
where E. a' E, are mean values, and superscripted values with (9) refer to sorted values in increasing order, (a .... S a!")) and (b(1) .... S b(m)). Additionally, a(0) ( ma, ((0) b(6) my are sorted and recentered observations.
important feature of Linear-Aoe AcT is that can be composed with linear layers in the GM, re- sulting in no computational overhead at inference time (see Appendix [ for details). Note that the expression Linear-ACT should not be confused with the closed-form known when transporting Gaussian density with parameter (Ma: G.) towards second (ms, db)), which is known (Peyré & Cuturi 2019 Remark 2.31) be T(a)= = gh a (m, Et m.,) Note that if one makes the additional assumption that dg db, then the affine Gaussian map becomes mean shift or translation, with T(a)
da
ma- We call this very simple baseline Mean-ACT and show in Section 3.3 that several methods in the literature indeed propose versions of mean shift strategy.
Tigureeishwwer shoocases the effect of different maps on toy data (iid, Gaussian). Note that methods based on mean-shift (ACTADD, ITI-C, Mean-ACT) can strongly over or undershoot, mapping samples out-of-distribution. Linear-AcT shows good trade-off between in distribution mapping and low computational budget. We note that activations in current GMs show mostly unimodal distributions, but have different standard deviations for different behaviors shown in Figure making the linear choice a suitable one. Note that multimodal distributions would result in non-line -inearr transport maps, which are beyond the scope of this work.
4

---

Published as a conference paper at ICLR 2025
Transport Support The map in Definition 3.1 is estimated using pairs of samples. In practice, n is in the order of hundreds, which results in rough approximation of the true transport from μ to It is fair assume that the transport enror will be higher for input samples in the tail of μ., given the scarcity of samples that range. Because transporting OOD samples may lead to unexpected behavior, and to be the conservative side, only transport osberved support 2. = [min A, max A]. Using the support is important when is narrower than (typically in mitigation setup). Unless stated otherwise, we use 2, for concept mitigation and 2. (-o0, oo) for induction. Appendix shows an empirical validation of this choice.
samples that within the
3.2 SEQUENTIAL ITERATIVE MAPS
While it might be possible to follow the template approach outlined in Section 3.1 to apply univariate maps to each of the M L activations, this ignores the causal relationship across activations, where activations produced by a layer are processed by the next one, i.e., Om,t-1 = fe(am.e)- Any intervention the level of layer must therefore be factored in accordingly before creating the intervention at the next one. To account for such causality, we estimate the transport maps for each layer incrementally:
first estimate the transport for the first layer (in the model graph), then run inference again by applying the first layer map in order to estimate the map for the second layer, and so on until all maps are estimated. detailed with our tools in Definition 3.2 In Appendix C we show that causal estimation achieves more effective conditioning than a simultaneous estimation. In this work, we use causal estimation for Mean-ACe and Linear-AcT.
similar approach is adopted Zou al. (2023), and
Definition 3.2 (Affine Causal Transport Map) For m I M and E S L, let Am == denote families of M activations for the first
(a'm...... am and Bm == (b...."".. 6"
layer. Starling with & = 1, and setting
Cm,1 Am,1, Dm,,1 = Bm,,,
compute and store the 2M (Wm Bm) parameters of all M transport maps associated with these activations using Definition 3.1
Vm S M. Tm Cm,,, Dm,e) IR RR
where observations C and D are refreshed recursively for each of their entries m S M, as E is incremented,
C. == Te((Tm.e(Cmme))
D. fe((Tm,, (Dm,))))))))))
At inference time, given sentence a, we run the recursion starting from the first activation vector (m..1)m. looping for 1 1 E S L as a +- fe([Tm.e (am)))
Interpolation Between Measures Using Transport One can easily extend transport map from measure μ to to one that is able to output an interpolating measure. The idea, outlined by McCann 1997.. consists in defining the following >-parameterized map from any OT map I',
T(a, A) (1 - A)a + XT()),
where À E [0, 1] and = 1 recovers the full transport. Conditioning GMs through OT allows the user to precisely control the presence of concept with continuous and interpretable knob (X) during generation, not requiring expensive parameter search (Li et al., 2024) or being limited by fixed, uncontrollable conditioning (Suau et al. 2024). In applications such as diffusion. where the utility of the model is harder to assess, our interpretable strength is of key importance, as shown in Section Note that methods like ACTADD, CAA ITI-C also have a conditioning strength parameter. However, this parameter is applied as multiplier of a conditioning bias as T' A) = + X33 (see Section 3.3), thus making À unbounded, harder to interpret and not robust with respect to different models, layers, and tasks.
3.3 GENERALIZATION OF PRIOR INFERENCE-TIME INTERVENTIONS WORK
In this section, we show how many earlier works can be interpreted as special cases of Linear-Acts Table summarizes the intervention proposed by several recent methods, where show that all
5

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of different inference-time interventions in the literature. All methods listed can be expressed as specific form of linear map. With ACT, the conditioning strength À interpo- lated between the activation and its transformed version (following Equation (.) while existing methods use À as bias multiplier, thus becoming less interpretable and less robust to model/layer changes. As a result, many methods require a grid-search to find the best layer to intervene upon.
Method
Transport
Parameters
Support
Delao Suga al. 2022]
WM + 2
w 0,
B = mg
Any layer, a AP(A, B) >-6
max
2123
Layer search
last
(Rinsay can
w
Layer search
RePE
Zou el 8.. 2014
we + A8
w 1. $ a (ax) (001-a(aa)
Layer search
last
AURA
2144
Gins,A,
Any layer, AUROCC(A, 0.5
EAST
Kitnn d1 A224
we A8
ma
Layer search
last
IT1-M
Tieral 20724
Le0 1 A/2
" 1, my mhg
Attention head search
last
ITI-C
AB
9 J.
Atlention lecad search
last
Mcnn-ACT, Section
Aja M(100 53
i 8 ang min, Σ, (660 (wa0 8))
Any layer, a Any layer, a O or Oxo
Qx
meon
Linea-ACT, Definition 3.1
Aln
M(w0 4.
mein
methods propose form of linear transport, and all of them (aside from Suau et al. (2022 add bias to the activations. The way this bias is pre-computed is what differentiates each method. Note that the parameter Ả typically multiplies the bias, thus becoming unbounded and non-interpretable.
ACT applies a linear transformation on activations that maximally preserves internal distributions (Section3.1] and distribution plots in Appendix(F). Moreover, AcT interpolates between the current and transformed activations, making À bounded between [0, 1] and interpretable. An additional aspect is that other methods propose various heuristics choose the support, while ACT uses all activations or the observed input range (2,). Note that CAA, ITI-M and Mean-ACT use a difference in means. We subsume this family of methods reporting results for Mean-AcT, which has the additional advantage of an interpretable A. An additional difference is that many methods use the last token only (in pseudocode, 6(z) = ={.... -1]. Detzero and AURA use max-pooling (Ø(2) z. z.max 1)) whileAcT uses an average across tokens (Ø(z) z.mean(- (1)), which we have found to be more robust (see Appendix D).
4 EXPERIMENTS ON LLMS
We empirically verify the performance of AcT on pre-trained LLMs on toxicity mitigation (Sec-
tion 4.1 general concept induction (Section 4.2), and truthfulness induction in particular (Sec-
tion 4.3
showing the efficacy and robustness of ACT in different scenarios related to LLMs..
4.1 TOXICITY MITIGATION IN LLMS
known that LLMs are prone to generate toxic language (Wen et al.] 2023), especially when prompts are designed to elicit toxic behavior. In this section, we study how ACT is effective at toxic language mitigation compared to some recents methods such as AURA, ACTADD and ITI-C, on Gemma2-2B (Team al. 2024) and Llama3-8P 8B Dubey al. (202)) To do so, prompt each LLM with 1000 randomly chosen prompts from Real Prompts (RTP) (Gehman et al. 2020), known induce toxic language generation. Then, collect the generated continuation to each prompt and we evaluate toxicity with ROBERTA-
classifie as in Suau et al. (2024). In LLM-as- a-judge (Zheng et al. 2023) (more details on Appendix H). As a measure of general LLM utility we report in Table 2: (i) perplexity (PPL) on a fixed set of 20k Wikipedia sentences measured with the intervened model, (ii) PPL of the generated sentences measured with Mistral-7B -BB (Jiang et al., 2023) and (iii) MMLU (Hendryeks et al. (02T) 5-shot accuracy using the intervened model. Besides, we report generation diversity results in Appendix G
addition, also measure toxicity in O-shot manner by querying
Linear-ACT reduces toxicity up to 7.5x and is robust to X, layer, and model choice We ob- serve that Linear-AcT achieves up to 7.5x reduction in toxicity on Gemma2-2B and 4.3x on Llama3-8B, with minimal impact on PPL and MMLU, Most importantly, AcT obtains the best means full transport.
results at which is in line with our OT formulation, since Linear-ACT and Mean-ACT obtain similar toxicity mitigation results. ITI-C achieves 5.6x and

---

Published as a conference paper at ICLR 2025
Table 2: Toxicity mitigation for Gemma2-2B and Llama3-8B, results over 5 runs. We intervene upon different layer types (layer column) and show the best layer per method. ITI-C, ACTADD and ACT have a strength parameter À which we sweep. For each method, we report results for the À that attained the best CLS toxicity that incurs less than +1 increase in PPL Wikipedia, ACT methods and provide best results for À = 1, achieving up to 7.5x (Gemma2-2B) and 4.3x (Llama3-8B) CLS toxicity mitigation with Linear-ACT, ITI-C is very sensitive to À as well as layer choice (see full results in Appendix D. and AURA reaches up to 3.1x reduction.
Layer
Best A
CLS Tox. (96)
O-shot Tox. (%) J
PPL. Wikipedin 1
PPI. Mistral-7B
MMLUU
Original
1.17
13.42 1.08
13,98
6.68
53.1
Gemmu2 2B
ACTADD
MLP
0.5
3.96 + 0.21 (L.X)
13:43 1.42
14.69 (+0.72)
6.67 (+0.05)
$3.0
AURA
MLP
2.12 + 0.27 (2.x0)
9.04 0.66
14,18 (40.211)
7,04
53.0 (40.)
ITI-c
Attention
8.0
0.71
(6.00)
6.36 = 0.91
14,90 (+0.92)
7.44 (+0.76)
52.6 (-0.5)
Mean-ACT
Post-LN
0.54 + (777))
4.10 0.41
[4.21 (44.233
7.59
(-1.5
Linear-AcT
Post-IN
1.0
0.56 4: 0.21 (7:50)
4.14 0.55
14.79 (+0.81)
7.99 (+1.31)
(-1.8
Original
5.80
15,00
9:06
5.68
65.3
Llama3-50 SB
ACTADD
Atiention
0.3
5.57
1. 0.45
(1.X3
15.73 1. 0.21
9.71 (10.65)
5.85 (+0.16)
65.5 (10.2)
AURA
MLP
+: 0.61 (8.1x)
0.85
9.52 (+0.45)
(+0.37)
65.5 (+0.2)
ITI-C
Attention
3.0
1.60 + 0.22
(8.x)
6,53 = 0.66
9:48 (+0.42)
6.17 (+0.89)
64.7 (-0.6)
Mean-ACT
Attention
(422))
9.56
64.7 (.0.)
Linear-ACT
Attention
1.0
1.35 = 0.39 (4.30)
6.68 0.81
9.56 (+0.49)
6.28 (+0.60)
64.5 (-0.8)
3.6x toxicity reduction on Gemma2-2B and Llama3-8B respectively. In line with the ITI-C paper findings, ITI-C performs well on attention, but is very sensitive to models and layers, as well as to the choice of À (see layer diagram in Appendix B and full tables and plots in Appendix []. AURA achieves 2.0x and 3.1x toxicity reduction per model and ACTADD induces the mildest mitigation.
4.2 INDUCING CONCEPTS IN LLMS WITH ACT
1.0
20
0-shot p(yes)
ITI-c
PPL Mistral 7B
ITI-c Lincar-AcT Mean-AcT
15
0.5
Linear-Are ACT
Mean-AcT
10-
0.0
A=1
10
10 Intervention strength À X
Intervention strength À
Figure 4: Concept induction using ACT (post-LN layers) and ITI-C (attention layers) on Gemma2- 2B. We aggregate results over 7 WordNet concepts, generating 500 sentences at different interven- tion strength levels. We report concept presence with LLM--s---judde (p(yes)), and the PPL of the generated sentences using Mistral-7B. We plot the median (and 25/75 quantile band) across concepts and generations per level, showing that Linear- ACT achieves a peak of concept induction at X 2 1, which is inline with our OT formulation. Other methods show different maxima.
ACT allows transporting activations from distribution to v (derived from sentence distributions and 9 respectively). In an induction setting, p covers generic content, while q a specific concept that we want to induce. We mine the OneSec dataset (Scarlini et al. 2019), collecting 700 sentences that contain specific concept (q) and 700 sentences randomly sampled from other concepts (p). We do so for seven different concepts (football, cloud, baby, church, book, flower, balloon) and we estimate an intervention for each of them. We assess the presence of a concept in the generated text a LLM- as-a-judge manner by querying Llama3-8B ---attr-ttttt (LLM-as- judge details in Appendix M].
instruct
Linear-ACT can induce arbitrary concepts with consistent X = 1 Figure 4 shows the effect of increasing À both on the presence of the concept, p(yes), and the PPL. measured with Mistral-7B on the generated text. We intervene upon the most effective layers for each method according to the toxicity results: attention for ITI-C, and Post-LN for ACT. In general, we found that LN layers were the most suited for ACT, across models and tasks. A naive explanation is that centering and scaling activations keeps the source and target activation distributions within a reasonable range, which makes the transport map more reliable. We do not include AURA because it is designed for mitigation, and ACTADD gives lower performance on this task. For Linear-Ace we observe a peak of concept presence À Rs with median p(yes) 0.87 (i.e., 87% of the generated sentences
7

---

Published as a conference paper at ICLR 2025
Table 3: TruthfulQA results for Gemma2-2B and Llama3-8B, results over 5 runs. We intervene upon different layers (layer column) and show the best per model. ITI-C, ACTADD and AcT have strength parameter which we sweep, reporting the best À result per model (MC1 Accuracy so that MMLU is within the best ACT MMLU + 0.1).
Layer
Best A
MCI Accuracy (9)
MC2 Accuracy (9)
MMLU Accumcy (5)
Original
21.05
32,80
53.10
Gemma2-2 -2
ACTADD
MLP
3.0
23.01
= 0.00 (+1.96)
31.76 + 0.00 (+196)
52.83 (-00(7)
AURA
ML.P
21.20
0.10 (40.15)
32.88 0.22 (4-0.68)
52.73 0.07 (-0.37)
ITI-c
MLP
2.0
24.53
+: 0.11 (13.48)
37.06 + 0.38 (+41266)
51.39 +: 0.41 (-1.7)
Mean-ACT
AILLNN
1.0
25.07
0.20 (14.02)
38.68 + 0.30 (4-5.88)
51.81 + 0.12 (-1.2)
Linear-AcT
AIILN
1.0
26.00
0.32 (+4.95)
40.17 : 0.24 (+7.37)
51.47 (-1.63)
Original
25.46
40.27
65.35
Llamaz-888 3-8
ACTADD
Attention
0.7
26.19
0.00 (+0.73)
10.88 : 0.00 (+0.661)
65. 0.00 (+0.07)
AURA
MLP
25.34
(-0.12)
40.47 : 0.20 (+0.20)
65.37 :: 0.06 (+0.02)
ITI-C
MLP
2.0
30.11 0.60 (+1.65)
15.41 (+5.10
64.71 0.14 (-0.6)
Mean-ACT
AIL-EN
1.0
32.88
0.54 (47.42)
48.23 + 0.64 (47.96)
64.83 + (-0.2)
Linear-AcT
AII-LN
1.0
33.22
0.22 (47.76)
48.69 048.444442)
64.78 + 0.15 (-0.57)
are classified as containing the induced concept) and an acceptable PPL. = 8.5. For > 1, the PPL quickly degrades and the presence of the concept diminishes. This is also consistent with the toxicity mitigation experiments in Section 4.1 Interestingly, the peak for Mean-AcT is at À = 2.5, also highlighting that Mean-AcT is poorer approximation of the OT transport. Notably, ITI-C achieves a similar p(yes) and PPL as Linear-ACT for À x 5. However, note that ITI-C's best is different than the ones for toxicity. Appendix contains generation examples.
4.3 INDUCING TRUTHFULNESS IN LLMS WITH ACT
One particular concept that has gained attention in previous activation steering works is "truthful- ness" (L.i et al.] (024). We study how ACT can increase truthfulness on Gemma2-2B and Llama3- 8B, compared to the original model. Again, we compare to AURA, ACTADD and ITI-C, We evalu- ate all methods
the TruthfulQA multiple choice part that has been used in prior work (Lin et al.) Li et al. 2024). We report both MC1 and MC2 of TruthfulQA, and control for overfitting on the TruthfulQA task by also evaluating MMLU 5-shot accuracy (Hendrycks et al. 2021).
2021
ACT can induce truthfulness with consistent X == 1. The results of our experiments are summa- rized in Table 3 As we can see, ACT can successfully induce truthfulness in both models in its default setting Ā= = (corresponding to full transport). Both Sinear-ACT and Mean-ACT achieve the best and second-best MCI and MC2 accuracy improvements among all methods investigated. Linear-AcT increases MC1 by roughly 5% for Gemma2-2B and by almost 8% for Llama3-8B, which is about 1.5% and 3% more than the closest non-AcT baseline (ITI-C), while incurring even slightly less decrease in MMLU performance. Full results and experimental setup in Appendix [
5 CONTROLLING IMAGE DIFFUSION MODELS
In this section, we show that ACT improves the controllability of text-to-image diffusion models (T2Is), well-known challenge (Cao et al.) 2024). We address two open problems in T2I generation: fine-grained style control [Section[5.1 and concept negation (Section 5.2). We show that off- shelf ACT succeeds both tasks. In line with OT theory and LLM experiments (Section 4. ACT consistently achieves the strongest conditioning with X = 1. We also adapt ITI-C to the topology of images by training on the spatial average pooling of activations (as we do by default for AcT), and applying it to each spatial position independently. Remarkably, ITI-C succeeds at fine-grained control with our adaptation, but requires tuning A, and it fails with concept negation.
Setup. We apply ACT on the denoising convolutional UNet of Stable Diffusion XL (SDXL) (Podell et al.) and the denoising transformer of FLUX. F.U.....cchell. 3. For FLUX, we use the T5-XXI. text encoding modality (Raffel al. 2020) instead of CL.PP (Radford et al. 2017) to
forest 1abs/
8

---

Published as a conference paper at ICLR 2025
PEREZOO
tou
UEG
OSED
Figure 5: Linear-ACT allows controlled conditioning of SDXL and FLUX. "A cat resting laptop keyboard in a bedroom." SDXL (left) and FLUX (right) intervened with ITI-C (top), Mean- ACT (middle) and Linear-AcTT (bottom) for the concept cyberpunk, with À strength in [0, 1]. The image with the best À (according to the highest O-shot score in Figurelo) is shown right. Qualitatively, Linear-ACT balances better cyberpunk style increase with prompt semantics preservation.
account for the effects of language modelling. We use a distilled version of SDXL, which only re- quires 4 diffusion steps (Lin et al. 2024) like FLUX. We intervene upon all normalization layers in SDXL's UNET and the output of most residual layers in FLUX (details in Appendix M.8). We only show results for ACT and ITI-C since ACTADD is not applicable images and AURA resulted in noisy images. To measure the presence of a style or a concept, we use a CLIP zero-shot classifier with the classes (+) "A picture of a (style or concept) and (-) "A picture of someth fomethingg" We also track whether the content from the original prompt (with no style or concept modifiers) is preserved using the CLIPScore (cosine similarity of CLIP embeddings, Hessel et al. (202I) between the images generated after the intervention and the original prompt.
5.1 STYLE CONTROL.
A major challenge in T2I generation is fine-grained control. For example, while one can prompt SDXL to create sketch of object, is hard to control the level of "sketchiness". Models such as SDXL have guidance parameter, but its use is limited since low guidance values tend to remove image semantics (see example in Appendix M.T). To showcase the ability of AcT to achieve such fine-grained control, we sample 2048 prompts from the COCO Captions (Chen et al. 2015) training set and append series of tags generated with Llama-8BBinstrrut anime, arl nouveau, cyberpunk, impressionism. sketch, watercolor (see Table 15 for details). Then we use the original prompt as the source distribution (p) and the style-modified prompt as the target distribution (q) to learn transport maps for style. To evaluate, we sample 512 prompts from the COCO Captions validation set and generate images with different intervention strengths.
induce the following styles:
Linear-AcT is robust method for fine-grained control in text-to-image generation. Fig- ure (6a shows that Linear-AcT on SDXL and FLUX increases the presence desired style, e.g., on SDXL from ~ 12% to 95% of the generated images while keeping 80% of the similarity to the original prompt (A 1). In accordance to the theory and experiments on LLMs, the maximum conditioning (i.e., highest O-shot score) for ACT is achieved at À can also accomplish fine- graiier control, but its best performance is achieved at different As, equal to 2 and 1 for SDXL and FLUX respectively, which is in turn not consistent with the best À found in LLM experiments. A closer look at images generated with ITI-C for best À in Figure 5 and appendix [M3] reveals that ITI tends to exaggerate style traits while distorting the semantics. This further highlights the reliability of ACT across different modalities, tasks, and models. While quantitatively ACT and ITI-C perform well, we invite the reader compare the quality of the generated images and styles in Figures I and 3 and in more examples in Appendix M.3
for both models. ITI-C
5.2 CONCEPT NEGATION
T2I diffusion models struggle with concept negation (Li al. Hwang et al. 2024) recent models such as Stable Diffusion (Rombach et al. 2022 and DALL-E 3 (Betker et al.) (023) are prone to generate a pink elephant when instructed not to generate one. To improve controllability. some models like SDXL include negative prompt mechanism to remove concepts from the generated images. However, we found that both SDXL (CLIP encoder + negative prompt) and FLUX (TS- XXL encoder) still tend to generate unwanted concepts (see some examples in Appendix M.2

---

Published as a conference paper at ICLR 2025
TOXiL-ipiiting
FLUXX.1 1.hhnell
SDXT. Lightting
FL.UX.1.ccmm 1.Sdhelll
1.00
0.75
)
0.75
Delotterte)
0.30
0.25
0.25
0.00
0.1
1.0
10.0
0.1
1.0
10.0
0.3
1.0
10.0
0.1
1.0
10.0
0.300
CliSSeer (-1)
0.925
0.235
ClapSoce
0.150
0.150
0.075
0.075
0.000
0.000
0.1
1.0
10.0
0.1
1.0
00.0
0.1
1.0
10.0
0.1
1.0
10.0
Interveation strength A
Intervention stoeagth
Intervention strength
Inservention strength A
(a) Style control
(b) Concept Negation
Figure 6: Style control (a) and concept negation (b) on SDXL and FLUX. Top row shows the fraction generated images classified (CLIP 0-shot) as containing given concept or style. Bottom row shows how much the intervened model deviates from the unmodified one in terms of ClipScore between the image and the original unconditional prompt. Points inside the gray area represent images that have lost their semantic content.
Figure 7: Concept Negation for "A plate of food with rice and beans, broccoli and meat. And pink elephant is missing.". (a) Linear-AcT on SDXL with transport strength À linearly increasing from I. Note how the presence of the pink elephant prominent for the original model (leftmost image) and gradually disappears as À increases.
We use the COCO Captions (Chen al. 2013) training set to sample 2048 prompts used gen- erate the images. To create source and target activation distribution to estimate ACT, we ask Llama3-8B-inss -i-tttittucuttt to generate diverse set of prompt modifiers requiring the model to include the following concepts: pink elephant, white bear, and gorilla. The exact phrasing of the modifiers is provided in Table T6 We estimate our transport maps from the modified prompts (p. with concept) to the unmodified prompts (q). To evaluate the model, we sample 512 captions the COCO Captions validation set and ask
to negate each of the modifiers used before (e.g., "'wihouut pink elephant", "a gorilla cannot be seen any anywhere") to generate images with unintended concept spillage such as the leftmost image in Figure or the examples Figures 18 and
Linear-AcT is robust method for concept negation in text-to-image generation. In Fig- ure 6bb we observe that ACT is more effective at concept negation than ITI-C while better preserving the original semantics of the image, indicated by the drop in 0-shot concept score for higher CLIPScore than ITI-c. ITT requires a stronger intervention to reduce the presence of the undesired concept, the cost of losing the whole semantic content, hence the drop in the Relative ClipScore. Additional examples and images for each concept can be found in Appendix M.4
6 LIMITATIONS AND DISCUSSION
In this work, we introduce Activation Transport (ACT), general framework to achieve intuitive and fine-grained control of GMs. Our approach is based optimal transport theory, effectively mapping activations from a source to a target distribution by preserving the latter, and unifies many previous activation steering works. We show experimentally that our Linear-ACT approach generalizes well across models and tasks, for both L.MSs and T2I architectures. Moreover, AcT provides robust parameter to control the amount of conditioning, bounded between 0 and 1, which makes it user- friendly and interpretable. While effective, Linear-AcT assumes a linear transport between i.i.d. activations, which are simplifications adopted for compute and memory reasons. Additionally, the map estimation purely depends on the samples used, thus being limited by their expressiveness. In future work, we plan on exploring non- linear maps and joint activations distributions.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
Our method could theoretically be used to mitigate or induce the presence of any concept. Therefore, could eventually lead the development of censorship or misinformation tools.
While our work can be used to align in pre-trained GMs, it should not be taken reason not to pursue the adoption of clean data and additional alignment strategies during the pre-training phase.
REPRODUCIBILITY STATEMENT
Our code and data are publicly available on htttpaa://iithubb.omm com/ wttp:///ooooooapppmmmmemeee- To aid reproducibility, all tables contain the best À found through grid-search and results are averaged over runs. We include additional details on intervened layers Appendix [B] ablations on effect of transport support in Appendix E pooling operation ablations in Appendix D the exact prompt templates LLM as a judge in Appendices H and [ experimental details on TruthfulQA in Appendix D as well as experimental details for T2I models in Appendix M
ACKNOWLEDGEMENTS
We thank Miguel A. Bautista, Federico Danieli, Gerard Gállego. Yu-Guan Hsieh, Miguel Sarabia, Federico Scozzafava, and Barry Theobald (in alphabetical order) for their helpful feedback and crit- ical discussions throughout the process of writing this paper. We would also like to thank Aswathy Balagopalan for contributing to the codebase, and Jerremy Holland for supporting this work.
REFERENCES
James Betker, Gabriel Goh, Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang. Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science., 2(3):8, 2023.
Manuel Brack, Patrick Schramosski, Felix Friedrich, Dominik Hintersdorf, and Kristian Kersting. The stable artist: Steering semantics in diffusion latent space. arXiv preprint arXiv:2212.06013, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Pu Cao, Feng Zhou, Qing Song, and Lu Yang. Controllable generation with text-to-image diffusion models: A survey. arXiv preprint 24X3v22403.04279, 2024.
Xinlei Chen, Hao Fang, Tsung-Ys Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv: 10400055 2015.
Sinho Chewi, Jonathan Niles-Weed, and Philippe Rigollet. Statistical optimal transport. arXiv preprint arXiv:2407. 18163, 2024.
Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. arXiv preprint arXiv:2309.174 2023.
Jasper Dekoninck, Mare Fischer, Luca Beurer-Kellner, and Martin Vechev, Controlled text genera- tion via language model arithmetic. arXiv preprint arXiv:2311.14479, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Leuman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. arXiv preprint arXiv:2407.21788 2024.
11

---

Published as a conference paper at ICLR 2025
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederie Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference Machine Learning, 2024.
C. Fellbaum. WordNet: An Electronic Lexical Database. Language, Speech and Communication. Mit Press, 1998.
Rohit Gandikota, Joanna Materzynska, Tingrui Zhou, Antonio Torralba, and David Bau. Concept sliders: Lora adaptors for precise control in diffusion models. arXiv preprint arXiv:231 2023.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah Smith. Real- toxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprini arXiv:2009. 114.22,,..1142,222 2020.
Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Ieard, and Noah Goodman. Find- ing alignments between interpretable causal variables and distributed neural representations. In Causal Learning and Reasoning, pp. 160-187. PMLR, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2021.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 202/ Conference on Empirical Methods Natural Language Processing, pp. 7514-7528, 2021.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference machine learning, pp. 2790-2799, PMLR, 2019.
Dang Huu-Tien, Trung- Tin Pham, Hoang Thanh- Tung, and Naoya Inoue. On effects of steering la- tent representation for large language model unlearning. arXiv preprint arXiv:2408.06223, 2024.
Kyomin Hwang, Suyoung Kim, JunHoo Lee, and Nojun Kwak. Do not think pink elephant! arXiv preprint arXiv:2404. 15154, 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint urXiv:2310.06825, 2023.
Zeyinzi Jiang, Chaojie Mao, Yulin Pan, Zhen Han, and Jingfeng Zhang. Scedit: Efficient and con- trollable image diffusion generation via skip connection editing. In Proceedings of the IEEE/CVF Conference Computer Vision and Pateen Recognition, pp. 8995-9004, 2024.
Suhas Kotha, Jacob Mitchell Springer, and Aditi Raghunathan. Understanding catastrophic forget- ting in language models via implicit inference. 2024.
Kyungmin Lee, Sangkyung Kwak, Kihyuk Sohn, and Jinwoo Shin. Direct consistency optimization for compositional text-to-image personalization. arXiv preprint arXhv:242.112044 2024.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from language model. Advances in Neural Information Processing Systems, 36, 2024.
Senmao Li, Joost van de Weijer, Fahad Khan, Qibin Hou, Yaxing Wang, al. Get what you want, not what you don't t: Image content suppression for text-to-image diffusion models. In The Twelfth International Conference Learning Representations.
Shanchuan Lin, Anran Wang, and Xiao Yang. sdxl-lightning: Progressive adversarial diffusion distillation. arXiv e-prints, pp. arXiv-2402, 2024.
12

---

Published as a conference paper at ICLR 2025
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2199.07958, 2021.
Yun Luo, Zhen Yang, Fandong Meng. Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning, 2023.
Robert J McCann. A convexity principle for interacting gases. Advances in mathematics, 128(1):
153-179, 1997.
Chong Mou, Xintao Wang. Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. 12i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 4296- 4304, 2024.
Nithin Gopalakrishnan Nair, Anoop Cherian, Suhas Lohit, Ye Wang, Toshiaki Boike-Akino, Vishal M Patel, and Tim Marks. Steered diffusion: A generalized framework for plug-and- play conditional image synthesis. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 20850-20860, 2023.
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Gabriel Peyré and Marco Cuturi. Computational Optimal Transport. Foundations and Trends in Machine Learning. 11(5-6), 2019. ISSN 1935-8245.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joc Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations.
Alec Radford, Rafal Jozefowicz, and llya Sutskever. Learning to generate reviews and discovering sentiment. arXiv preprint arXiv: 170.001444 2017.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21 21()):5485-551) 2020.
Nate Rahn, Pierluca D' D'Oro, and Mare G Bellemare. Controlling large language model agents with entropic activation steering. arXiv preprint arXiv:2406.00224 00244, 2024.
Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering llama 2 via contrastive activation addition. arXiv preprint arXiv:2312.0666) 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. Pro- ceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 22500- 22510, 2023.
Filippo Santambrogio. Optimal transport for applied mathematicians. Birkäuser, NY, 55(58-63):94, 2015.
Bianca Scarlini. Tommaso Pasini, and Roberto Navigli. Just "onesec" for producing multilingual sense-annotated data. pp. 699-709, 2019. doi: 10.18653/v1/P19-1166
Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models' sen- sitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. ICLR, 2024.
13

---

Published as a conference paper at ICLR 2025
Nick Stracke, Stefan Andreas Baumann, Joshua M Susskind, Miguel Angel Bautista, and Björ Ommer. Ctrloralter: Conditional loradapter for efficient O-shot control altering of t2i models. arXiv preprint arX1v:2405.07711, 2024.
Xavier Suau, Luca Zappella, and Nicholas Apostoloff. Self-conditioning pre-trained language mod- els. In International Conference on Machine Learning, pp. 4455 4473. PMLR, 2022.
Xavier Suau, Picter Delobelle, Katherine Metcalf, Armand Joulin, Nicholas Apostoloff, Luca Zap- pella, and Pau Rodriguez. Whispering experts: Neural interventions for toxicity mitigation in language models. Forty-first International Conference on Machine Learning, 2024. URL
net Seerrmmidd2PPGVEEE
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models a practical size. arXiv preprint arXiv:2408.00116 2024.
Alex Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MaclDi- armid. Activation addition: Steering language models without optimization. arXiv preprint arXiv:2308. 10248, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision 2024.
and Pattern Recognition, pp.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations.
Jiaxin Wen, Pei Ke, Hao Sun, Zhexin Zhang, Chengfei Li, Jinfeng Bai, and Minlie Huang. Unveiling Association for Computational
the implicit toxicity in large language models. pp. Linguistics, December 2023.
Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D. Man- ning, and Christopher Potts. ReFT: Representation finetuning for language models. 2024. URL arxiv.org/aber 240paabb//4444.... 03592
Kai Yang. Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8941-8951. 2024.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308. 06721, 2023.
SHIH-YING YEH, Yu-Guan Hsieh, Zhidong Gao, Bernard B W Yang. Giyeong Oh, and Yanmin Gong. Navigating text-to-image customization: From lyCORIS fine-tuning to model evaluation. In The Twelfth International Conference on Learning Representations, 2024. URL https:
Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan- Yee K Wong. Uni-controlnet: All-in-one control to text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 01446695546662 2023.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. In The 41st international ACM SIGIR conference on research & development in information retrieval, pp. 1097-1100, 2018.
14

---

Published as a conference paper at ICLR 2025
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405, 2023. 2023.
15

---

Published as a conference paper at ICLR 2025
A MEMORY AND COMPUTATIONAL ASPECTS
Linear-Ac's ACTT requires storing 2 floats (w: (9 per activation intervened. For example, Linear-AcT on post-LN layers of Gemma2-2B requires (2 x 52 layers 2304 activations x bytes) 0.91 Mb. If we choose to use the support transport, 2 more floats per activation are stored Q. = [min A, max A], which means an extra 0.91 Mb for the Gemma2-2B example. In terms of compute, Linear-AcT requires an extra element- product and sum per intervened layer. However, the inference cost of such operations is of second order compared the overall LLM inference cost.
One has the option to fix X. If so, our Linear-ACT formulation in Definition 3.1 becomes Tian (a) 1
(M(w- 1) + 1) + AB = wa + AB. Assuming we intervene after linear layer 7 + 6, we compose both functions as (T с (a) = Wnya + (@6 + A8), which is also a linear map whose parameters can replace those of f in the computational graph, without any extra cost at inference time. The memory cost is 0 if we fix À and compose Linear-Are AcT with the model linear layers.
A.1 DETAILS ON COMPUTATIONAL COMPLEXITY
The computational cost of Linear-AcTT can be divided in two main parts: estimation and inference.
Estimation. The estimation cost is the cost related to extracting activations from model and estimating a transport map on top. Let us assume the cost for running an inference step with a model up to the latest layer where an intervention placed My. N the number of samples upon which we learn the transport, and D the dimensionality of each activation vector. We also assume batch size
Extracting activations:
Assuming non-sequential iterative maps (see Section 3.2 in the submission): the cost for extracting activations is ML..
Assuming sequential iterative maps, we need two forward passes per layer: the first is used estimate a transport map, and the second produce responses after applying the map. Since the cost of applying map with fixed strength is 0 (as it can be fused with the weights), the cost of extracting activations with iterative maps is D(2NMM)
Estimating a linear transport map involves sorting NLD activations for the source and target distribution and computing the affine transport params analytically (see Definition 3.1). Assuming half of the N samples belong to the source and the target distributions re- spectively, the cost is dominated by the sorting operation O(N log(NL,)) (assuming quicksort is used), which is also smaller than the cost of forward pass through the model.
Inference. The inference cost is the cost related to generating an output with an intervened model. As explained at the beginning of the section, assuming a fixed transport map strength (A), the affine transport map be directly fused into the model weights and thus the additional cost of Linear- ACT is O(0). If we need to be able to tune the intervention strength, then we cannot fuse into the weights and the cost that of 1-d affine map on all the transported activations, which is significantly smaller than the cost of a forward pass on the model, which involves expensive matrix multiplication: (LL)) << (MM)
Summarizing, estimation is only done once, has cost ML) and it is amortized during inference. During inference, the transport cost is (0)) with fixed X and OLL)) with variable X. In plain words, estimating transport map is much cheaper than training a model and has no impact at inference time unless one needs control over in which case the additional cost is significantly smaller than the cost of forward pass with the model.
16

---

Published as a conference paper at ICLR 2025
B INTERVENED LAYERS
Gemma2-2B
imput
Pre-LN
Attention
Post-LN
Pre-LN
MLP
Post-LN
output
Figure 8: Schema a Transformer block of Gemma2-2B with the layer names as referenced this
work. Note that Llama3-8B has similar structure without the Post-LN layers.
C CAUSAL VS. SIMULTANEOUS ESTIMATION OF AcT
In Table and way (see Section 3.I.. We observe that causal estimations show better toxicity mitigation than its simultaneous counterparts.
and Table 5 we compare the estimation of ACT interventions in causal and simultancous
Table 4: Causal (gray background) simultaneous estimation of AcT Gemma2-28 2B in toxicity mitigation setting (explained in Section 4.1). Causal estimation provides better conditioning (lower toxicity).
Layer
PPI. Wikipedia
PPI. Mistral-7B
C1.S Toxicity (9h
O-shot Toxicity (9h)
Original
13.98
6.62
4.08 0.36
13.25 d: 0.88
Mcan-ACT
Attention
1.0
13,90
7.23 (+0.61)
1.12 0.35
5.60 + 1.01
Mean-ACT
Attention
1.0
14.08 (40.11)
7:23 (+0.61)
1.06 0.17
5.14 0.50
Linear-Ach
Atlention
1,0
14.04 (+0.06)
7.26 (+0.64)
0.97 0.39
5.73 1: 0.90
Lincur-AcT
Atteation
1.0
14.21 (+0.23)
7.24 (+0.62)
0.90 2: 0.33
5.06 + 0.63
Mcan-ACT
Post-LN
14.11 (40.13)
7.71 (+1.09)
0.62 0.05
4.47
Mean-Ac
Post-LN
1.0
14.21 (40.23)
7,59 (+0.97)
0.51 =: 0.44
4.10 0.41
Linear-ATT
Post-LN
0.9
14.54 (40.57)
7.87 (+1.25)
0.65 0.17
4.40 + 0.39
Linea-ACT
Post-LN
14.79 (+0.81)
7.99 (+1.37)
0.56
Table 5: Causal (gray background) vs. simultaneous estimation of ACT on Llama3-8B in a toxicity Causal estimation provides better conditioning (lower toxicity).
mitigation setting (see Section 4.1
Layer
Wikipedia
PPI. Mistral- 7B
CL.S Toxicity (9) J.
Toxicity (9)
Original
9.06
5.68
5.80
15.00
Mcan-ACT
Attention
1.0
9.35 (+0.28)
6.33 (+0.65)
1.40 0.29
6.73 + 1.13
Mean-ACT
Attention
1,0
9,56 (+0.19)
6.36 (+0.68)
1.38 0.17
5.60 + 0.31
Linear-Ace ACT
Atlention
1.0
9.38 (+0.32)
6.27 (+0.58)
1.38 0.24
6.55 1: 0.75
Linear ACT
Attention
1.0
9.56 (+0.49)
6.28 (+0.60)
1.35 0.39
6.68 0.81
D THE EFFECT OF THE POOLING OPERATION
The number of activations to store to compute transport map is O(NMLK). where N is the number of samples used to estimate the transport, M the number of activations per layer, the number of layers, and K the number of tokens decoded. This number so most methods perform a pooling operation & over K. We run an ablation on the pooling operation for ACT on Gemma2-2B, in the toxicity mitigation setup. We find that mean pooling achieves better trade-off between toxicity mitigation and utility, measured as MMLU (Ta
easily become intractable
17

---

Published as a conference paper at ICLR 2025
Table Ablation on the choice of pooling operation (see Section on Gemma2-2B.
Method
Pooling d Strength
CLS Tox. (1)
MMLU (t)
Original
4.17 + 0.32
53.06
Linear-AcT min
0.77 + 0.12
45.85 + 0.09
Linear-AcT
max
1.80 = 0.12
47.01 0.30
Linear-AcT last
0.47 + 0.17
48.49 +.25
Linear-AcT mean
0.70 0.10
51.87 0.06
THE EFFECT OF THE TRANSPORT SUPPORT
In this section validate the choice of transport support, way to make the pro- posed intervention more robust. In this experiment, we sweep different supports by narrowing the quantiles (qt) of the input data set A, in the setting of toxicity mit- igation (as in Section 4.1). both for Mean-AcT and Linear-ACT. The supports tested are: [9tay- dtuol: (9 9170. (qt20> qlool: 14 Qlyul, [q1,, qlas], [qtz. 9697 [9t, qlygl [q Qloool and (-x0,00.
Note that (4t: 1ooo ure 9 where we observe that [gtp 1oooo offers good trade-off between conditioning strength and acceptable increase in PPL (below +1 points with respect to the original model).
Q. defined in Section We show the results of this sweep in Fig-
16.0
20, 00
Original
Mean-AcT
15.5
Linear-AcTT
PPL Wikipedia
15.0
aloo q600
14.5
00, 00
Vloo gu1e
14.0
480
Gt1. qloo
qlio qligtin ot
13.5
0.00
0.02 CLS Toxicity
0.04
Figure 9: We measure toxicity mitigation Gemma2-2B by increasingly expanding the transport support from at 9tso on the farther right of the plots to [4t, 10 [min A, max A), which means the support spanned by all the samples in A. For completeness, we add the full real support (-o0, oo). For Linear-ACT using [qt, 1ooo achieve the best toxicity mitigation by incurring less than increase in PPL. Note that (-o0, 00) results in higher PPL.
F How Do DIFFERENT INTERVENTIONS AFFECT DISTRIBUTION??
We show in this experiment how activation distributions are modified by the effect of different interventions. For that, plot in Figure the the distribution of source activations μ (toxic), that of target activations y (non-toxic) and also the distribution obtained when mapping samples with map T, Le., Tip. Ideally, would like observe that Ttp. We show the distributions of those
18

---

Published as a conference paper at ICLR 2025
activations with highest normalized cost w computed as
č N Li (b(0) - wa( 3)
(2)
mh ma + σb aa
so that we pick activations with * for the sake of illustration. We observe that Linear-Ac's obtains very good overlap of distributions (first row) while ITT-C does not in many cases (this result extends to any bias- blas-based method, we show ITI-C as example of such family of methods). The latter is only shifting activations with a bias, thus becoming impossible to adapt the shape of distributions. Moreover,
observe that with ITI-C some activations are mildly shifted (4th column), and some others are strongly shifted (2nd, 3rd, 5th columns). This makes it evident that very hard to set robust À for bias-based steering methods.
L1:30
Sccer
Salce
M
Figure 10: Transport of distributions. We show how different interventions transport the internal distributions. In gray the source distribution μ (toxic), in blue the target distribution y (non-toxic) and in red the distribution Tiy obtained when pushing-forwer forward js through a given intervention T. Each column contains the distributions for the activation with highest č (see Equation (2) in a given layer. the first row we show Linear-ACT, observing a good overlap between and Tip. The second row shows ITI-C, with poorer distribution overlap. We use À 1 for Linear-ACT and for ITI-C (optimal As from Table 2
G ASSESSING TEXT GENERATION DIVERSITY
One important question is whether the generated text after a model intervention still shows diversity. To answer this question, in Table 7 we measure the Self-BLEU score (Zhu et al., 2018) for the sets of generated sentences after RTP prompts. Note that smaller Self-BLEU scores indicate higher diversity in the set, while large Self-BLEU shows repeatedness in the sentences. For example, set of identical sentences will return Self-BLEU of 1. We evaluate the best configuration for each method (layer, À choice) according to Table 2
From the results in Table we observe that Linear-AcT (Self-BLEU = 0.134) better preserves the diversity shown by the non-intervened model (Self-BLEU = 0.130). In this setting, ITI-C achieves 0.144 and our Mean-Act ACT Sclf-BLEU of 140. We obtain these results averaging over runs of 1000 generations each, and the standard deviations show that the results are significant.
19

---

Published as a conference paper at ICLR 2025
Table 7: ACT obtains better diversity. We measure the Self-BLEU score proxy for diversity, on 4 randomly chosen sets of 1000 generations after RTP prompts. Linear-AcT obtains the closest result the original model, showing higher diversity.
Method
SeIf-BLEU (4)
ITI-C
0.144 0.003
Mean-ACT
0.140 = 0.002
Linear-ACT
0.134 + 0.002
Original
0.130 + 0.003
H 0-SHOT TOXICITY CLASSIFICATION
We query Samma3-8--instrus with the system prompt below, where [SENTENCE] contains the sentence be assessed in 0-shot manner.
"You are a highly accurate texicity classificat model. Your task is to determine if given sentence contains toxic content. You will output 'yes' only if the sentence is toxic and 'no' if the sentence neutral or non toxic. An [SENTENCE]"
count(yes) Then, we measure the toxicity probability as count(yes) +ommt(((m)) reported as 0-shot toxicity in Tables 2 and Note that we use the logits for tokens yes and no, so we consider the answer to be positive when logit logit(no).
I 0-SHOT CONCEPT PRESENCE CLASSIFICATION
We query Llama3-88 B-Baa3-BB--ae 108-----insttii instruct with the system prompt template below.
"You are a chatbot who answers whether the provided sentence is referring to [CONCEPT] defined as (WORDNET_DEF] Note that the sentence might not contain the word [CONCEPT), but may just be referencing concept as defined. \n [SENTENCE]".
Where:
[CONCEPT] be (football, cloud, baby, church, book, flower, balloon).
(WORDNET_DEF] are taken from WordNet Fellbaum FellbauFell (1998):
football: Any of various games played with a ball (round or oval) in which two teams try to kick or carry the ball into each other's goal.
cloud: A visible mass of water or ice particles suspended at a considerable altitude.
baby: A very young child (birth to 1 year) who has not yet begun to walk talk.
church: A place for public (especially Christian) worship.
book:
A written work composition that has been published (printed on pages bound
together).
flower:
A plant cultivated for its blooms or blossoms.
balloon: Large tough nonrigid bag filled with gas heated air.
[SENTENCE] Contains the sentence to be assessed in 0-shot manner.
We measure the probability a concept being present as we do with toxicity, explained in Ap- pendix H
20

---

Published as a conference paper at ICLR 2025
J EXTENDED RESULTS ON TOXICITY MITIGATION
We report here the full experimental results for toxicity mitigation, which have been summarized in Section 4.1] Note the variability in the optimal strength À for ITI-C and ACTADD, which compli- cates the applicability of these methods on different models and layers.
Toxicity mitigation for Gemma2-2B results over 5 runs. We show results intervening different layers in the model (layer column). ITI-C, ACTADD and AcT have strength parameter which we sweep, reporting for each method the best result (best X) in CLS toxicity that incurs less than +1 1 increase in PPI. Wikipedia. ACT methods are robust to the choice of layer and provide best results for À = achieving up to 7.5x toxicity mitigation with Linear-AcT. ITI-C is very sensitive to as well as layer choice, and AURA does not provide a strength control.
Table 8:
Layer
Best A
PPL Wikipedia.
PPL Mistral 7B
MMLU
CLS Toxicity (%) 4
0-shol Toxicity (%)
Original
13.98
6.68
4.17
ACTADD
Atention
0.5
13.99 (40.02)
6.58
53.2 (40.2)
0.111
13.25 + 1.63
ITI-C
Ateation
8.0
14,90 (40.92)
7.44 (+0.26)
(-0.5
0.74 0.18
5.36 = 0.91
Mean-ATT
Atention
1.0
14.08 (+0.11)
7.23 (+0.55)
52.5 (0.6)
1.06 0.17
5.14 = 0.50
Linear ACT
Alention
1.0
14.21 (10.23)
7.24 (10.56)
52.2 (.09)
0.90 0.33
5.06 0.63
ACTADD
Post-LN
0.1
14.04 (40.06)
6.61
53.2 (+0.2)
4.08 0.43
13.50
ITI-C
Post-LN
13.0
14.89 (+0.92)
7.34 (+0.66)
52.8 (-0.3)
3.08 0.61
12.24 0.69
Mean-ACT
14.21 (+0.23)
7.59
(-1.5)
Lincar-AcT
Post-IN
1.0
14.79 (10.8)
7.99 (+1.31)
51.3
0.56 0.21
4.14 + 0.55
AURA
MLP
14.18 (40.21)
7.04 (+0.36)
(-.0)
2.12
9.04 = 0.06
ACTADD
MLP
0.5
14.69 (40.72)
6.67 (+0.05)
(3.0))
3.16 0.21
13.43 1: 1.42
ITI-C
MLP
1.0
13.99 (10.01)
6.77 (10.08)
(.0.3)
4.50 0.32
15.06 0.76
Mean-ACT
MI.P
14.33
7.02
(2.7)
1.30 0.37
7.28 0.85
Lincar-AcT
MLP
1.0
14.89 (40.92)
7.53 (+0.85)
(1.9
1.30 # 0.39
7.15 0.98
Table 9: Toxicity mitigation for Llama3-8B, results over 5 runs. Similar conclusions as in Table are extracted.
Layer
Best A
PPL Wikipedia,
PPL Mistral-7B
1
MMLU 1
CLS Toxicity (%)
0-shot Toxicity (%)
Original
5.68
65.3
5.80
16.00
ACTADD
Atention
0.3
9:71 (+0.65)
5.85 (+0.16)
(+0.2)
5.57
1: 0.45
15.73 1: 0..1
ITI-C
Atention
3.0
9.48 (10.42)
6.17 (10.49)
64.7 (.0.6)
1.60 0.22
6.53 0.66
Mean-ACT
Atention
1.0
9.56 (40.29)
6.36 (40.68)
64.7 (407)
1.38 0.17
5.60 + 0.34
Lincar-ACT
Atention
1.0
9.56 (+0.19)
6.28 (+0.60)
64.5 (-0.)
1.35 :: 0.39
6.68 = 0.81
AURA
(+0.45)
6.05 (+0.37)
65.5 (+0.2)
0.61
8.12
ACTADD
MI.P
ITI-C
MLP
(40.3)
5.79 (+0.11)
5.62 (-199
1.16
Mea--ATT
MLP
0.9
9.90 (+0.84)
6.24 (+0.55)
60.7 (-1.)
2.10 0.48
10.65 1.02
Linear-ACT
MLP
10.06 (40.99)
5.98 (+0.29)
61.9 (:3.4)
2.23
10.27 0.97
21

---

Published as a conference paper at ICLR 2025
18
18
Original
Original
AURA
ACRA
17
ACTADD
17
ACTADD
IT--
ITI-
Mea
AcT
PPL Willipofin
10
Limear-Aoo
PPL. Wikippdin
16
Unmer-Aoo
15
14
14
13-
0.00
Dobbot Tixicity
Dsbbot Tundkity
(a) Gemma2-2B
(b) Gemma2-2B
12.0
12.0
Original
Original
11.5
AURA
11.5
AURA
ITI-
ITI-c
11.0
Mer
11.0
Meun-
AeT
Linear-Act
Linear-ATT
PPL Wikipodis
10.5-
PPL Wilipodis
10.5
10.0
10.0
9.0
9.0
8.5-
8.5-
8.0
8.11
0.00
0.01
0.02
3.03
11.01
0.05
0.00
0.05
0.101
0.15
O-shot Tococty
O-shot Toccity
(c) Llama3-8B
(d) Llama3-8B
Figure 11: ACT achieves the best conditioning at on Gemma2-2B and Llama3-8B. We show the À sweeps for toxicity mitigation Gemma2-2B In gray show the PPL+1 interval considered to be the maximum loss in PPL we can assume. The bold markers are the results reported in Table 2 For clarity, only show the experiments that yielded best results reported in Table 2 The full results are shown in Table
K EXTENDED RESULTS CONCEPT INDUCTION LLMS
12 contain examples of generated sentences by Gemma2-2B intervened for concept induction with Linear-AcT and ITI-C. These results complement those presented Section 4.2 gradual increase of concept with Linear-AcT, reaching strong and consistent condi-
Tables 10
Note the tioning at X = with great quality. ITI-C also performs well on concept induction, however the choice of À is less clear. For example, for À 10 fails for concept Cloud while it works well for Football.
22

---

Published as a conference paper at ICLR 2025
Table 10: Generations different À inducing concept Football.
Concept Method
À
Generation
Football ITI-c
0.0 Once upon time, there was an old man who lived in the forest. He had no family and he spent his days alone collecting mushrooms for food to survive on until one day when someone came along with some money! This person gave him all of their savings so that they could live together forever
Football
ITI-C
5.0 Once upon time, the only way to watch football was TV. It wasn't just about watching your team play either it also about getting together with mates and having afternoon of footy in front of the telly home or down the local club for some beer
Football ITI-C
10.0 Once upon a time, the only way to watch football Australia was on Saturday afternoon.
And it wasn't 's even just for kids and mates at local clubs or schoolboys matches no-one would have ever heard Aussic Rules Foothall! 1908 saw an associ- ation called
Football
Linear-ACT
Once upon time, there was an old man who lived the forest. He no family and he spent his days alone collecting mushrooms for food to survive until day when someone came along with some money! This person gave him all of their savings that they could live together forever
Football
Linear-AcT
0.5 Once upon time, there was an amazing woman named Sarah. She had the most bcautiful smile and kindest heart you could ever imagine!
Sarah loved to play soccer with her friends on Saturday mornings at 9am sharp every week for as long as she can remember...umm .unttbbe....utilll one day when
Football
Linear-AcT
1.0
Once upon time, the only way watch football was TV. The game of had been played in England since 1863 and by the early twenticth century became one of Britain's most popular sports with three million spectatorss attending matches each
(and many
Table 11: Generations at different inducing concept Flower.
Concept
Method
Generation
Flower
ITI-c
0.0
Once upon time, there was old man who lived in the forest. He had no family and he spent his days alone collecting mushrooms for food to survive on until one day when someone came along with some money! This person gave him all of their savings that they could live together forever
Flower
ITI-C
upon a time, there was an beautiful hair and wore it like crown on her head. Her name called "Flower". She loved to paint pictures with water colours because she thought they very pretty. One day when
lived the land of flowers. had
Flower ITI-C
10.0 Once upon a time, there was an artist who lived in the land of flowers. She had bcautiful and fragrant petals that she called 'flowers'. She also has another name for her petal which is known as "petals". " have no idea what it means but think you can see
Flower
Linear-ACT
0.0 Once upon time, there was an old man who lived the forest. He had no family and he spent his days alone collecting mushrooms for food to survive on until one day when someone came along with some money! This person gave him all of their savings so that they could live together forever
Flower
Linear-ACT
0.5 Once upon time, there was old who lived in the forest. He had children and so he decided to grow some flowers for himself instead of having them his own hody!
The flower is called "the red-eyed lily" because it has petals that are hright pink
Flower
Lin
1.0 Once upon time, there was an egg-shaped seed that grew into the first true leaves. These tiny green structures are called sepals and they grow clusters at the hase each flower. The petals may he white or pink with 5 to mm long stalks; these will
23

---

Published as a conference paper at ICLR 2025
Table 12: Generations different inducing concept Cloud.
Concept
Method
Generation
Cloud
ITI-c
0.0
Once upon time, there was an old man who lived in the forest. He had no family and he spent his days alone collecting mushrooms for food to survive on until one day when someone came along with some money! This person gave him all of their savings so that they could live together forever
Cloud
ITI-C
5.0 Once upon time, the only way get your cloud-based computing infrastrue- ture was by using existing provider. But now that ve got many of them in sky (literally), can be difficult for cloudshifters and meteorologists to tell one from another!
Cloud
ITI-C
10.0 Once upon time, the clouds were named by cloud-cloud optical properties. These are called cirrus or altocumulus middle cir cir circir (Alto Alto Cu- mulus) Cloud family:.
Cloud
Linear-ACT
Once upon time, there was old who lived the forest. He family and spent his days alone collecting mushrooms for food to survive until one day when someone came along with some money! This person gave him all of their savings that they could live together forever
Cloud
Linear-AcT
0.5 Once upon a time, there was an island called the Isle of Skye. The weather on this isle is very unpredictable and can change from sunny to stormy in minutes! The clouds are always moving around with no real pattern or structure as they do not have any defined tops like those found over land
Cloud
Linear-ACT
Once upon time, was island called the Bermuda. is located in middle latitudes of both hemispheres and has been known be associated with high- level clouds (cirro- alto-transpprrent. The altitude ranges from about 10,0
L EXPERIMENTAL DETAILS AND EXTENDED RESULTS ON INDUCING TRUTHFULNESS
L.I EXPERIMENTAL DETAILS
We follow the original experimental protocol for evaluations on the TruthfulQA multiple choice part, described in Lin et al. (2021). This consists of preprompting the model with the same default prompt before each question as proposed by Lin et al. (202T) in Figure 21, which we replicate below for the reader. To then evaluate model (preprompted) question, the likelihood of each multiple choice answer is computed independently (conditional on the default prompt and question). The answer option with the highest likelihood counted the model's 's answer the question.
24

---

Published as a conference paper at ICLR 2025
QA
Q: What is human life expectancy in the United States?
A: Human life expectancy in the United States is 78 years.
Q: Who was president of the United States in 1955? A: Dwight D. Eisenhower was president of the United States in 1955.
Q: Which party did be belong to?
A: He belonged the Republican Party.
Q: What is the square root of banana?
A: have comment.
How does telescope work? A: Telescopes lenses mirrors focus light and make objects appear closer.
Where were 1992 Olympics held? 1992 Olympics were held Barcelona, Spain.
Figure 21: QA prompt
Figure 12:
Figure 21 from Lin et al. (022 showing the default preprompt used for the TruthfulQA multiple choice part.
25

---

Published as a conference paper at ICLR 2025
L.2
EXTENDED RESULTS
L.2.1 FULL RESULTS OVER SEEDS FOR ALL LAYERS
Table 13: TruthfulQA results for Gemma2-2B, results over 5 runs. ITT-C, ACTADD and AcT have
a strength parameter À which we sweep, reporting for each method the best result (best X) in MCI Accuracy that incurs at least equal performance in MMLU accuracy compared to the best (in terms of MC1 accuracy) of the two AcT methods (see L.2.2 giving 0.1% slack).
Layer
Best X
MC1 Accuracy (9)
MC2 Accuracy (%)
MMLU Accuracy (%)
Original
21.05
32.80
53.10
AURA
MLP
21.20
+: 0.10
32.88 1: 0.22
52.73 0.07
ACTADD
Attention
3.0
22 64 + 0.00
34.64 = 0.00
33.02 0.00
ITI-C
Attention
5.0
23.18 = 0.28
36.16 = 0.34
52.10 0.44
Mean-AcT
Attention
1.0
21.62
- 0.07
34.08 + 0.19
52.83 = 0.09
Linear-AcT
Attention
1.0
21.71 = 0.14
34.47 = 0.22
52.86 0.08
ACTADD
AIILN
1.0
21.42 = 0.00
32.93 + 0.00
51.65 0.00
ITL-C
AII-IN
23.94 0.96
36.62
51.37
Mean-AcT
AIl-LN
1.0
25.07
0.20
38.68 +: 0.30
51.81 0.12
Lincar-ACT
AIl-NN
1.0
26.00 + 0.32
40.17 = 0.24
51.47 0.27
ACTADD
Post-LN
0.8
22.40 + 0.00
34.27 +: 0.00
53.11 = 0.00
ITI-C
Post-LN
8.0
23.16 + 0.40
35.94 +: 0.55
51.39 = 0.45
Mean-AcT
Post-LN
1.0
21.93 = 0.20
34.88 + 0.25
52.77 = 0.10
Lincar-ACT
Post-LN
22.45
0.22
35.94 0.36
52.43
ACTADD
MLP
3.0
23.01 + 0.00
34.66 +: 0.00
52.83 0.00
ITI-C
MLP
2.0
24.53 = 0.11
37,06 = 0.38
51.39 = 0.41
Mcan-ACT
MLP
1.0
21.98
+ 0.19
35.18 + 0.31
52.84 = 0.04
Linear-ACT
ML.P
1.0
21.93
0.20
35.47 + 0.25
52.73 0.19
Table 14: TruthfulQA results for Llama3-8B, results over 5 runs. ITI-C, ACTADD and AcT have strength parameter À which sweep, reporting for each method the best result (best X) in MC1 Accuracy that incurs at least equal performance in MMLU accuracy compared to the best (in terms of MC1 accuracy) of the two AcT methods (see L.2.2. giving 0.1% slack).
Layer
Best A
MCI Accuracy (%)
MC2 Accuracy (%)
MMLU Accuracy
Original
25.46
40.27
65.35
AURA
MI.P
25.34 + 0.15
40.47 + 0.20
65.37 + 0.06
ACTADD
Attention
0.7
26.19
0.00
0.00
65.42 0.00
ITI-C
Attention
1.0
27.42 + 0.30
42.01 0.42
65.26 + 0.11
Mean-ACT
Attention
26.73 + 0.19
42.20 0.24
65.37 0.06
Lincar-ACT
Attention
1.0
27.17 + 0.23
42.15 = 0.31
65.33 + 0.11
ACTADD
All-LN
25.58 1 0.00
41.00 0.00
64.88 0.00
ITI-C
AIILNN
29.65 1 0.71
44.43 0.56 0.56
64.71 0.22
Mean-ACT
AII-NN
32.88 0.54
48.23
64.83 0.14
Lincar-AcT
AIILNN
1.0
33.22 + 0.22
48.69
64.78 0.78 0.15
ACTADD
MLP
0.5
25.46 + 0.00
40.64 0.00
65.34 0.00
ITI-C
MLP
2.0
30.11 + 0.60
45.41 = 0.24
64.71 + 0.14
Mean-ACT
MLP
26.17 1 0.24
41.27 1 0.34
1 0.20
Lincar-AcT
MLP
1.0
26.41 + 0.52
39.34 0.54
60.98 3.14
L.2.2 SWEEPING À FOR ITI-C AND ACTADD
In Figures 13 16 we show the results of sweeping the value of À for ITI-C and ACTADD for both Gemma2-2 2B and Llama3-8B. For each model, we also indicate the MMLU accuracy of the best ACT method for that model with horizontal grey dashed line, this is our point of reference for choosing À for ITI-C and ACTADD: we choose the value of À that achieves the best MC1 accuracy, while achieving at least equal MMLU accuracy this grey dotted line (up slack of 0.1%).
For ITI-C, where we see a clear relationship between MMLU and MC1 accuracy as À varies, we sweep À E [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 14.0, 15.0). For ACTADD,
26

---

Published as a conference paper at ICLR 2025
Gemerad Amardan
Garres HLP
134
1.48
425
0.1
0.4
2.0
Radd
MCL.A
1.3
0.4
0.10
1.49
0.25
0226
0.25
8.20
235
0.209
3.345
0.500
0.0
0.56
0.5
Figure 13: Sweeping for inducing truthfulness with ITI-C Gemma2-2B. Left endpoint of line is = 1.0, right endpoint of line is X 15.0 (each point increasing by 1.0), Note this is for seed only.
where the relationship can be more erratic, we also sweep values < 1.0. Here, sweep À 00.1, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8,0.8.800 0.9, 1.0, 2.0, 3.0, 4.0, 5.0).
that À can have strong impact on performance for ITI-C, but in different way for each layer and model. In particular, it can decrease MMLU performance to catastrophic levels (more than halving performance
Overall
Gemma2-2B for MLP layers and on Llama3-8B for both attention and MLP layers), making it necessary to sweep À to find its value that provides a reliable control method using ITI-C for the problem at hand. Similar things can be found about ACTADD (e.g. when interventing upon on all Layernorm layers on Gemma2-2B, Figure 14p.
27

---

Published as a conference paper at ICLR 2025
0332
AGNE
6.88
63399
0589
0.43
0.08
0.3.0
3.235
33385
52115
$2239
03323
02239
5318
0.333
0388
932%
52300
0.003
52200
Genvra2 Boa syemem
Germis Layersom
2.13
0.40
0.38
0.55
0.3
0222
8250
6258
Figure 14:
Sweeping À for inducing truthfulness with ACTADD on Gemma2-2B. Left endpoint of line is 0.1, right endpoint of line is X 5.0 (A 00.1,0.2, 0.2, 0.3, 0.4, 0.5, 0.6, 0,9, 2.0, 3.0, 4.0, 5.0)). Note this is for 1 seed only.
6.5
0.75
21
0.3
0.2
MC1
Uarral Alaammmm
0.50
3.89
4.0
422
Figure 15: Sweeping À for inducing truthfulness with ITI-C on Llama3-8B. Left endpoint line is 1.0, right endpoint of line is X 15.0 (each point increasing À by 1.0). Note this is for 1 seed only.
28

---

Published as a conference paper at ICLR 2025
Aike
5.40
0.408
0.70
9794
2.93
2.40
1.90
Uarra Happemm
Figure 16: Sweeping for inducing truthfulness with ACTADD on Left endpoint of line is [0.1, 0.2, 0.3, 0.4, 0.5, 0.7,0.,,60008..... 0.9, 1.0, 2.0, 3.0, 5.0)). Note this is for 1 seed only.
Llama3-8B.
0.1, right endpoint of line is X 5.0 (A
29

---

Published as a conference paper at ICLR 2025
M EXPERIMENTAL DETAILS AND EXTENDED RESULTS FOR T21 GENERATION
Appendix M.1 illustrates the effect of the guidance parameter in SDXI.. Appendix M.2 illustrates
the problem of concept negation when using negative prompts. Appendix M.3] contains additional qualitative examples of style control on SDXL and FLUX. Appendix M.4 contains additional qual-
itative examples for concept negation in SDXL and FLUX. Appendices M.6 and M.7 contain the
list of tags used as prompt modifiers to generate the target/source distribution of activations for each style/concept respectively. Appendix contains details on FLUX's s architecture conditioning.
M.1 GUIDANCE PARAMETER IN EXISTING DIFFUSION MODELS
We show in Figure 17 the effect of changing the guidance scale parameter in SDXL. While large val- ues lead to effective conditioning, lower values destroy content. This makes guidance intuitive and harder to use by users.
Figure 17: SDXL with art nouveau tags appended to the prompt described in Appendix M.3 and guidance strength linearly increasing from to Note how for low guidance (left most images) the semantic content is almost completely lost.
M.2 NEGATIVE PROMPTING
Stable diffusion models allow using negative prompts to avoid unwanted elements in the generated images (Rombach et al.) 2022 Podell et al.). Here, we show that this method is ineffective at removing pink elephant, white bear, and gorilla. Figures [8 and 19 contain some failure cases of SDXL. and Stable Diffusion (Esser et al.] (202)) at removing unwanted concepts. Figure 26) and Figure 27 show results intervening SDXL with ACT, showing its effectiveness at removing these concepts with the same prompts, In Figure 28 we show some failure cases at concept negation.
Figure 18: SDXL with Negative Prompt. Prompt: "There banana and two pieces of cheese on plate. A (pink elephant, gorilla, white bear] cannot be seen anywhere.". Negative prompt: "A (pink elephant, gorilla, white bear)".
30

---

Published as a conference paper at ICLR 2025
Figure 19: Stable Diffusion 3 with Negative Prompt. Prompt: "2 tier cake with multicolored stars attached it. {pink elephant, gorilla, white bear] cannot seen anywhere." Negative prompt: "A (pink elephant, gorilla, white bear).".
M.3 STYLE CONTROL
Figures
22
complement the results shown in Section 5.1
Anime
(b) Art Nouveau
(c) Cyberpunk
(d) Impressionism
(e) Sketch.
Watercolor
Figure 20: SDXL A plane floating on top of a lake surrounded by mountains. From left to right conditioning strength À increases from strength found in Figure = 1 for ACT and for ITI-C). Linear-ACT succeeds at inducing different styles. Mean-ACT fails at inducing art nouveau. ITI-C introduces noise for art nouveau and cyberpunk.
Rightmost column corresponds the best
31

---

Published as a conference paper at ICLR 2025
(a) Anime
(b) Art Nouveau
(c) Cyberpunk
(d) Impressionism
(e) Sketch.
(1) Watercolor
Figure 21: SDXL - A firetruck with lights on is on a city street. Rightmost column corresponds to the best strength found in Bigure 1 for ACT and À = 2 for ITI-C). Mean-AcT fails at inducing impressionism and art nouveau. ITI-C achieves the strongest conditioning and generates a noisy image for art nouveau.
32

---

Published as a conference paper at ICLR 2025
Anime
(b) Art Nouveau
(c) Cyberpunk
(d) Impressionism
(c) Sketch.
(I) Watercolor
Figure 22: SDXL A sandwich is placed next to some vegetables. Rightmost column corresponds to the best strength found in (A 1 for ACT and for ITI-C). ITI-C fails at inducing style progressively (e.g. (c) cyberpunk).
33

---

Published as a conference paper at ICLR 2025
(a) Anime
(b) Art Nouveau
(c) Cyberpunk
(d) Impressionism
(e) Sketch.
Watercolor
Figure 23: FLUX A group of zebra standing next to each other on a dirt field. Rightmost column corresponds to the best strength found in Figure = 1 for all methods). Linear-AcT is successful at inducing all styles. ITI-C fails at inducing cyberpunk and anime.
34

---

Published as a conference paper at ICLR 2025
Th
65
(a) Anime
(b) Art Nouveau
T
(c) Cyberpunk
(d) Impressionism
(e) Sketch.
(I) Watercolor
Figure 24: FLUX- Black cat with green eyes sitting in a bathroom sink. Rightmost column corresponds to the best strength found in Figure 6 (A for all methods). AcT's conditioning is weak for sketch and watercolor. ITI-C fails inducing cyberpunk.
35

---

Published as a conference paper at ICLR 2025
(a) Anime
(b) Art Nouveau
(c) Cyberpunk
(d) Impressionism
(e) Sketch.
(1) Watercolor
Figure 25: FLUX A semi truck driving down a street. Rightmost column corresponds to the best strength found in Figure 6 (A 1 for all methods). ACT is able preserve the semanties for all styles and we observe only mild conditioning for impressionism and watercolor. ITI-C fails at inducing anime and cyberpunk.
36

---

Published as a conference paper at ICLR 2025
M.4 CONCEPT NEGATION
(a) Many parked city street with tall buildings in the background.
(b) A cat sitting in front of a large computer monitor.
Figure 26: SDXL Concept negation examples I. Rightmost column corresponds the best strength found in Figure 6 (A = 1 for ACT and À = 4 for ITI-C), Every rows represent different concept (gorilla, pink elephant, white bear) which was negated the input of the image generator. Mean-AcT and Linear-AcT ACT succeed at removing the unwanted con- cept. ITI-C fails for gorilla and produces a blurry image for pink elephant.
37

---

Published as a conference paper at ICLR 2025
(a) There is banana and two pieces of cheese on plate.
(b) tier cake with multicolored stars attached to it.
Figure 27: SDXL Concept negation examples II. Rightmost column corresponds the best strength found in Figure = for ACT and À 4 for ITI-C). Every 3 rows represent a different concept in {gorilla, pink elephant, white bear) which was negated at the input of the image generator. Linear-AcTI and Mean-AcT succeed at removing the negated concepts while ITI-C tends to modify the semantics of the image.
38

---

Published as a conference paper at ICLR 2025
Closeup of white and yellow with red circle at the bottom.
(b) A table topped with bananas next to coin.
Figure 28: SDXL Concept negation examples III (failures) Rightmost column corresponds
the best strength found in Figure (A = 1 for ACT and À = 4 for ITI-C). Every 3 rows represent different concept in (gorilla, pink elephant, white bear) which was negated the input of the image generator. While Mean-ACT and Linear-AcT are successful at removing the concept, there is sometimes change in semantics of the image for the maximum strength. ITI-C best strength (A = 4) changes semantics for all concepts.
39

---

Published as a conference paper at ICLR 2025
(a) 2 tier cake with multicolored stars attached to it.
(b) A table topped with bananas next to coin.
Figure 29: FLUX Concept negation examples I. Rightmost column corresponds the best strength found in Figureb](A = 1 for ACT and À = 5 for ITI-C). Every 3 rows represent a different concept in (gorilla, pink elephant, white bear) which was negated the input of the image generator. Linear-ACT removes the negated concepts except for white bear in (a). ITI-C effective at "best" (A = 5). At high strengths, Linear-ACT and ITI-C also affect other image semantics.
40

---

Published as a conference paper at ICLR 2025
(a) There banana and pieces of cheese plate.
(b) sandwich is placed next to some vegetables.
Figure 30: FLUX- Concept negation examples II (Failures) Rightmost column corresponds
the best strength found in Figure (A = 1 for ACT and À = 5 for ITI-C). Every 3 rows represent different concept in (gorilla, pink elephant, white bear) which was negated the input of the image generator. ACT does not remove white bear, and fails to remove gorilla in (b). For high Linear-AcT modifies the semantics of the image. ITI-C removes the unwanted concept for X = 5.
41

---

Published as a conference paper at ICLR 2025
M.5 DETAILED RESULTS
PLEX idddd
F
SENE laghnene
Cam
use
can
0120
033
Face=
100
0000
1.00
denghe
bdaeeden desggh
(a) Anime
(b) Art Nouveau
(c) Cyberpunk
PLEX
PLIN Stael
SUNI Lethins
PLUX Sihuel
360
12:0
211
2003
2,.20
T 1G22
T1.0
M0
dtough
(d) Impressionism
(c) Sketch
Watercolor
Figure 31: Style induction. For each style (a-f) and model (left-right), we show the O-shot classifi- cation score for the style being present in the generated images (top) and the ClipScore to track how much generated images deviate from the unconditional prompt (bottom). The gray area indicates images that have lost their semantic content.
1.80
SCNNL Labbniis
FLEX 18daad)
100
SEXU Ladennn
1.00
SUNE Leainee
FLUX Scheol
3,4m
F
0.54
0V0
608
000
6077
dingl
Lawn des dengle
(a) Gorilla
(b) Pink elephant
White bear
Figure 32: Concept negation. For each concept (a-c) and model (left-right), show the 0-shot classification score for the concept being present the generated images (top) and the ClipScore (bottom) to track how much generated images deviate from the unconditional prompt. The gray area indicates images that have lost their semantic content.
42

---

Published as a conference paper at ICLR 2025
M.6
STYLE PROMPTS
Table 15: List of tags generated with
(right) to induce different styles (left).
Anime
anime style, large expressive eyes, stylized hair, bold outlines, simplified colors, dynamic perspective, exaggerated features, angular shapes, chibis, manga inspired, emotive facial expressions, action sequences, speed 1ines, cell shading, graphic backgrounds, vibrant palettes
Art nouveau
Art Nouveau, Alphonse Mucha, Gustav Klimt, flowing lines, organic shapes, floral motifs, geometric patterns, ornamental designs, Jugendstil, Secessionism, symbolism, female figures, gold leaf, intricate details, turn of the century art, early 20th century impressionism, Claude Monet, brush strokes, light, color, outdoor scenes, water 1ilies, haystacks, Rouen Cathedral, reflections, nature, atmospheric, vibrant colors, visible textures, 19th century art, French impressionism
Impressionism
Cyberpunk
cyberpunk, neon lights, urban jungles, high-tech architecture, augmented reality, AT technology, biopunk, futuristic cities, post-apocalypti scenes, digital hacking megacorporations, androids, dystopian societies, cybernetic enhancements, chromed details, glowing neon signs, rain-soaked streets
Photorealism
photorealism, hyperrealism, optical precision, photographic quality, fine detail, lifelike textures, realistic lighting, accurate perspective, human figures, still life, cityscapes, landscapes, skin tones, reflections and shadows, everyday objects, documentary style art, contemporary realism
Sketch
sketches, pencil drawing, charcoal sketches, ink illustrations, gestural lines, quick studies, figure drawing, perspective sketching, urban sketching, landscape sketches, still life drawings, sketchbook art, doodles, minimalist lines, expressive mark-making, observational drawing
Watercolor
watercolor style, transparent media, wet-on-wet applicass coplication, dry brush strokes, soft blending, dalicate touches, gentle shading, luminous hues, atmospheric lighting, ethereal quality, subtle textures, color gradients, painterly aesthetics, fluid paint behavior, watercolor paper texture
43

---

Published as a conference paper at ICLR 2025
M.7 CONCEPT PROMPTS
Table 16: List of tags generated with Llama-&B-instruct (right) to induce different concepts (upper
left) prompt models not generate them (lower left).
Pink clephant
pink elephant. containing pink elephant. with pink elephant in plain view. and a pink elephant. it displays pink elephant. featuring pink elephant. in addition to a pink elephant. and also pink elephant. and pink elephant as well. the pink elephant can be clearly seen.
Gorilla
gorilla. containing gorilla. with gorilla in plain view. and a gorilla. it displays gorilla. featuring gorilla. in addition to a gorilla. and also a gorilla. and a gorilla as well. the gorilla can be clearly seen.
White bear
white bear. containing white bear. with white bear in plain view. and a white bear. it displays a white bear. featuring white bear. in addition to white bear. and also a white bear. and white bear as well. the white bear can be clearly seen.
No pink elephant
without a pink elephant. not containing pink elephant. without a pink elephant in plain view. and a pink elephant that cannot be seen. display a pink elephant. not featuring lacking a pink elephant. and not a pink elephant. and pink elephant is missing. the pink elephant cannot be seen.
does not pink elephant.
No gorilla
without gorilla. not containing a gorilla. without a gorilla in plain view. and a gorilla that cannot be seen. it does not display gorilla. not featuring gorilla. lacking a gorilla. and not a gorilla. and gorilla is missing. the gorilla cannot seen.
No white bear
without a white bear. not containing a white bear.
without a white bear in plain view. and white bear that cannot be seen. it does not display white bear. not featuring
white bear. lacking a white bear. and not a white bear. and white bear is missing. the white bear cannot be seen.
M.8 DETAILS ON FLUX CONDITIONING
FLUX's diffusion architecture is based on the transformer architecture Vaswani et al. 2017). Con- cretely, it is composed of N consecutive multi-modal fusion transformer residual blocks followed by M uni-modal transformer residual blocks. We found that the most effective strategy for strong conditioning is to intervene upon the output of all blocks. However, we found that conditioning blocks closest the output tends to deteriorate the generated images. Thus, condition all the N multi-modal blocks and the first 15 uni-modal blocks.
44


Reviews:
[{"review_id": "pnKeqD72XI", "reviewer": "Reviewer_Rxai", "summary": "The paper introduces Activation Transport (ACT), a framework based on optimal transport theory to steer model activations and control generative model outputs. ACT is designed to modify activations from a source distribution (e.g., toxic language) to a target distribution (e.g., non-toxic language) while preserving the natural activation distributions within the model. The framework applies to both large language models (LLMs) and text-to-image diffusion models (T2Is), supporting tasks like toxicity mitigation, concept induction, style control, and concept negation. ACT outperforms other methods in robustness and interpretability by allowing fine-grained, interpretable control of generative behaviors with minimal computational overhead.", "questions": "1. A potential future work could be the focus on data selection to estimate the linear transport. This I would argue might produce better improvements than moving to non-linear estimations\n\n2. although you show good evidence of the relatively minor degradation of model performance, more evidence is needed in a more quantifiable way, for example by showing limited impact on downstream tasks", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. ACT is a simple and efficient transport function approach that seems to perform well on the experimental setups for both LLM and T2I without significant impact on performance.\n2. The paper is well written with clear and easy to follow formulation and experimental results, The paper demonstrates ACT’s effectiveness in diverse tasks, including toxicity mitigation, concept induction, style control, and concept negation, showing superior or comparable performance to existing methods. The method’s flexibility and consistent results across both LLM and T2I applications underscore its potential as a general-purpose activation steering tool.\n3. By basing the intervention on optimal transport theory, ACT provides a clear, interpretable parameter (λ) that adjusts the strength of control. This parameterisation allows for easy tuning and understanding of model adjustments, enhancing its usability for practitioners.", "weaknesses": "1. ACT currently relies on linear transport maps, which are computationally efficient but may not capture complex, non-linear relationships within activations, especially in large or multimodal generative models. This assumption could limit its effectiveness in applications requiring nuanced adjustments.\n\n2. The quality of ACT’s transport maps depends on the representativeness of the source and target samples. If the samples do not fully capture the intended distribution (e.g., all aspects of toxic vs. non-toxic language), the intervention may be less accurate, impacting model behavior under real-world conditions with unseen data.\n\n3.  Although ACT shows promising results in mitigating toxicity and inducing specific concepts, the paper provides less evidence of its effectiveness across a broader range of behavioral modifications, especially in challenging or ethically sensitive areas like misinformation suppression or bias control.\n\n4. The performance of ACT can be influenced by which model layers are selected for intervention. While the paper provides some guidance, a more systematic approach to identifying optimal layers would improve robustness and reduce the need for manual tuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "brdLnc5AAu", "reviewer": "Reviewer_CvQ7", "summary": "This paper studies the problem of activation steering for diffusion models and Transformer language models. The key idea is to view activation steering as optimal transport. Under this umbrella, most existing methods are equivalent to mean transport map, which might not be optimal and do not preserve the activation distribution. Instead, the proposed method, Linear-ACT, can avoid out-of-distribution activations after steering. \n\nExperiments on Transformer language models (Gemma and Llama) show that the proposed method outperform baselines that use constant vectors for activation steering, on tasks including toxicity mitigation, concept inducing, and truthfulness. Experiments on diffusion models (Stable Diffusion models) show the effectiveness of the proposed method for style control and concept negation.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper studies an interesting problem in activation steering -- out-of-distribution activations. Many existing works require a very large coefficient before the steering vector, which can easily lead to OOD activations. The proposed method instead do not need this extrapolation.\n- The experiments are extensive, covering a wide range of control tasks for language models and diffusion models. Many of them are important tasks such as truthfulness and style control.\n- The paper proposes a unified framework to understand the connection between different activation steering methods.", "weaknesses": "- One of the exciting applications of activation steering or representation engineering is safety. It would be interesting to see how well the proposed method perform on safety risk mitigation.\n- The baselines are mainly vector addition methods. I wonder how the proposed method compare with vector projection methods such as https://arxiv.org/abs/2303.02536", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J1rsF2RZnJ", "reviewer": "Reviewer_WjWp", "summary": "This paper proposes Activation Transport (AcT) as a general approach to steer activations in generative models, to map activations from one source distribution to the target distribution, while minimizing limitations of current inference intervention approaches. The authors show that the optimal transport-based framework proposed here generalizes many of the previous activation steering work. More specifically, AcT accounts for difference in the amount of variance for each activation dimension in its map which previous approaches do not. The representation maps are iteratively derived updated layer by layer, starting from the earlier layers to the last layers. AcT is agnostic to the modalities; as shown in the experiments, AcT can steer generation in the text domain, in applications such as toxicity mitigation, concept induction and truthfulness, and in the image generation, in applications such as style control and concept negation.", "questions": "Choice of pooling operation: Can the authors show data to support the choice of average pooling? Some discussion about why average pooling is ideal versus max pooling or last token would be insightful. Ablation disentangling the choice of pooling layers versus other approaches would better differentiate the contribution of AcT and the pooling operations.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The proposed approach is grounded, intuitive and is applicable to several modalities and domains in generative modeling.\n\nThe paper is well written and easy to follow.\n\nExtensive experiments are conducted to compare AcT to baselines approaches in text and image generation.", "weaknesses": "Scope is limited to single modality within a model and linear (not non-linear) mapping.\n\nExperiments and results supporting the claim of AcT better preventing representations from being OOD is lacking.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iibyzTz8ss", "reviewer": "Reviewer_eCSW", "summary": "The paper presents a novel framework called Activation Transport (ACT) for controlling generative models by steering activations using optimal transport theory. It claims to offer fine-grained control over model behavior with minimal computational overhead. ACT is applied to tasks in both large language models (LLMs) and text-to-image diffusion models (T2Is), demonstrating effectiveness in mitigating toxicity, inducing concepts, and providing style control.", "questions": "1. How does ACT perform in scenarios with highly multimodal distributions, where linear transport might be insufficient?\n\n2. Can the authors provide more details on the computational costs associated with the implementation of ACT?\n\n3. How sensitive is the method to the choice of the transport support $Q_o$ or $Q_∞$?\n\n4. Are there any limitations observed in the diversity or creativity of outputs when using ACT for concept induction in LLMs?\n\n5. How does the method handle potential biases introduced during the transport of activations?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The use of optimal transport theory to steer activations is a novel idea that generalizes previous activation-steering methods.\n\n2. ACT is shown to be effective across different types of models (LLMs and T2Is), which suggests broad applicability.\n\n3. The method claims to add negligible computational cost, which is crucial for scalability in large models.\n\n4. The paper provides experimental results demonstrating improvements in toxicity mitigation and style control, among other tasks.", "weaknesses": "1. The paper mentions that Linear-ACT assumes a linear transmission relationship between activations, which is a simplification made for computational and memory considerations. Is it possible for nonlinear mapping to allow more complex activation relationships and finer control?\n\n2. Since the mapping estimate depends entirely on the samples used, its expressive power is limited by the diversity and coverage of the samples. How to improve the generalization ability of the mapping through more extensive data sampling or enhancement techniques.\n\n3. The paper mainly focuses on the control of a single task, such as toxicity mitigation or style control. How to deal with multiple conflicting objectives or constraints simultaneously under a unified framework?\n\n4. Although Linear-ACT has a certain robustness to the parameter λ, other methods such as ITI-C are very sensitive to the choice of model and layer, which may lead to a lot of parameter tuning in different situations.\n\n5. Although the ACT method claims to have low computational overhead, there is a lack of detailed analysis of the specific computational costs and resource requirements of models of different sizes in practical applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)