Linearly Controlled Language Generation with Performative Guarantees

## Abstract

Abstract The increasing prevalence of Large Language Models (LMs) in critical applications
highlights the need for controlled language generation strategies that are not only computationally efficient but that also enjoy performance guarantees.
To achieve this, we use a common model of concept semantics as linearly represented in an LM’s latent space. In particular, we take the view that natural language generation traces a trajectory in this continuous semantic space, realized by the language model’s hidden activations. This view permits a control-theoretic treatment of text generation in latent space, in which we propose a lightweight, gradient-free intervention that dynamically steers trajectories away from regions corresponding to undesired meanings. Crucially, we show that this intervention, which we compute in closed form, is guaranteed (in probability) to steer the output into the allowed region. Finally, we demonstrate on a toxicity avoidance objective that the intervention steers language away from undesired content while maintaining text quality.

## 1 Introduction

Large Language Models (LMs) have become increasingly widespread in a variety of applications, including critical functions such as content moderation on social networks and real-time information dissemination []. Despite their transformative impact, these models require updates to remain accurate post-deployment. Furthermore, as the demand for more nuanced text generation rises, strategies to account for specific constraints during text generation are increasingly necessary. To address these challenges, controllable text generation has emerged as a pivotal research area.

Several approaches have been proposed to address the challenge of controllable text generation. Of them, a popular approach is prompt engineering [, , ], in which natural-language prompts are carefully chosen at input-time to steer generation. Other approaches modify LM parameters or architecture to achieve the desired outputs [, ]. Lastly, some approaches engineer LM *activations*, or input representations, to steer it into the representation of the desired output [, , , ].

Despite current efforts, ensuring the controllability of these models remains a challenge due to their limited interpretability. For instance, while knowledge editing provides an efficient alternative to exhaustive retraining, it poses risks akin to the butterfly effect: minor adjustments could lead to unintended consequences. These potential repercussions underscore the need for principled approaches to control and refine LM outputs efficiently. Moreover, it is paramount for these approaches to be robust and ensure controllability guarantees to mitigate risks and harness their full potential safely.

To address this gap, we propose to use control theory to tackle controlled language generation. Specifically, optimal control theory [] offers principled methods to steer trajectories in latent space that enjoy theoretical guarantees on the performance of the intervention. In the framework of optimal control theory, our intervention method, which we call Linear Semantic Control (LiSeCo), derives from a theoretical formulation of controlled text generation. Our contributions are both theoretical and empirical: (1) we formally pose LM control as a constrained optimization problem and provide its closed-form solution with guarantees; (2) we empirically demonstrate our method on the use case of toxicity avoidance. We confirm, with experiment corroborating theory, that LiSeCo indeed steers LM generation from disallowed concepts while maintaining text quality.

## 2 Related Work

Contemporary language models are deep neural networks pre-trained on trillions of tokens of Internet-scale text. In part due to their vast scale and lack of interpretability, methods to control them in a fine-grained way remain elusive. A number of approaches have already been proposed towards this end, spanning the whole spectrum of permanent [, ] to online control strategies, e.g., [, ]. Here, we detail online inference-time approaches that are more similar to our method.

Prompt engineering is a technique that controls the LM’s output by intentionally varying the input context [, , , , ]. However, prompts are often highly task-specific, requiring either manually crafting or ad-hoc computationally-taxing techniques, and success is very fragile to the choice of prompt. In particular, while the space of natural language prompts is discrete, LM weights and activations live in continuous high-dimensional space, which is more expressive; then, rather than optimize over discrete prompts, other approaches that exploit this expressivity directly intervene in the internals of the model.

One category of such approaches intervenes in weight space, where post-hoc knowledge editing may be performed. In this approach, the weights of a pre-trained LM are modified according to the goal of the controlled generation by, for instance, learning the necessary weight update [, ], or localizing and editing target parameters encoding specific knowledge [, , , ]. Pitfalls of these methods range from potential inconsistencies and distortions, to the fact that they can only correct errors in the LM’s parametric knowledge, but not in-context [].

A related, more expensive approach tackles inconsistency issues by adding new features without modifying pre-trained parameters. One such intervention is to create a memory module where information can be retrieved to steer the output [, , ]. Another approach introduces extra parameters trained on specific knowledge, while the original model parameters remain unchanged [, , ]. A shortcoming of this approach is that it requires extensive training, and it can increase the computational overhead at inference time.

Finally, a number of methods, including ours, propose intervention directly in an LM’s latent, or activation, space. This strategy offers several advantages, such as rapid adaptation and robustness to contextual change. A seminal work in this domain was Plug and Play [], where, similar to LiSeCo, a linear intervention is computed at every layer. The control goal is encoded as the objective function in an optimization that is then solved via back-propagation, thereby adding significant computational overhead at inference time. Subsequent approaches also take advantage of linear modifications to the latent state, but in order to reduce the computational overhead propose to act on only a few layers [, ], or modify the activations to remove the need for back-propagation []. The recent alternative approach REMEDI [] also relies on finding an optimal intervention to achieve different target outputs. However, the approach requires specific a-priori training data for the representations, and offers no guarantees that the intervention will attain the target output. The approach presented in ITI [] addresses the issue of computational efficiency at the expense of optimality (the intervention is not formulated as an optimizer), and also lacks guarantees on the performance of the intervention.

Crucially, none of the approaches found in the literature provide guarantees on the controllability of their method. Here, we provide an intervention that is *theoretically guaranteed* to steer the input into the allowed region, and introduces minimal computational overhead. Though [] apply theoretical tools from control to LM text generation, to the best of our knowledge, our method is the first to propose a control-theoretic intervention whose theoretical guarantees are validated in practice.

## 3 Problem Statement

In this section we present the problem studied in this paper, as well as the assumptions and approach. In particular, we approach the problem of controlled language generation as a standard optimal control problem in the field of control theory [].

### 3.1 Problem Formulation

Given a language model $\ell:\ \Sigma^{*}\rightarrow\ \Sigma^{*}$, controlled language generation aims at steering the output of the model into a desired one. In this work, we study the case where the goal is to steer the output of an *already trained model* away from a disallowed region. Hence, we study how to design an intervention on a trained model so that the set of possible generated sequences is constrained to a subset $\mathcal{S}\subset\Sigma^{*}$, where $\mathcal{S}$ only contains the allowed sequences. The requirements for the generated output sequence are two fold: its latent trajectory (1) is *guaranteed* to never lie in the disallowed region, and (2) stays as close as possible to that of the original output sequence, so that text quality is not compromised. In doing this, two questions need to be answered:

- 1.
How can the disallowed region be defined for a given language model?
- 2.
How can an intervention be designed so that it is guaranteed (in probability) to stay within the allowed region while retaining maximal similarity with the original model?

In what follows, we provide an answer to these questions, and show that the proposed approach adds minimal computational overhead to the language generation process.

### 3.2 Approach

Figure: Figure 1: A. LiSeCo is based on linearly adding vector $\theta_{t}$ to the output of each layer ($t=1,\dots,T$). Each vector $\theta_{t}$ is the solution of a constrained optimization problem. B. A probing classifier $f$ mapping the latent space $\mathbb{R}^{d}$ to the decision space $\mathbb{R}^{2}$ is trained to characterize the allowable region (in blue) to which each latent state $\tilde{x}_{t}$ is constrained. Keeping trajectories out of the toxic region in latent space is equivalent to keeping their image out of the toxic region in decision space. C. At inference time, the state in latent space ($x_{t}\in\mathbb{R}^{d}$) is mapped through the classifier into the decision space ($W_{t}^{\top}x_{t}\in\mathbb{R}^{2}$). If it falls within the forbidden region (pink), an intervention ($\theta_{t}\in\mathbb{R}^{d}$) is computed so that the updated state $\tilde{x}_{t}=x_{t}+\theta_{t}\in\mathbb{R}^{d}$ lies in the safe region (blue).
Refer to caption: x1.png

We consider language generation to realize a trajectory in latent space. Similar to [], we take the view that, for every layer, disallowed language occupies a region of latent space $\mathbb{R}^{d}$. Formally, let $\mathcal{R}\subset\mathbb{R}^{d}$ be the forbidden, or unsafe, region. Our goal, then, is to provide a control mechanism by altering the output vector embedding at every layer that guarantees (in probability) that the latent trajectories remain out of $\mathcal{R}$ for all time, see Figure 1.

#### Semantic Probe.

The first goal is to identify the disallowed region for the generated output. To do so, we use a semantic probe to map each state in latent space $x_{t}\in\mathbb{R}^{d}$ to a probability of generating unwanted content $\mathcal{S}\subset\Sigma^{*}$. Specifically, we rely on a *probing classifier function* $f_{t}:\mathbb{R}^{d}\to\mathbb{R}^{2}$ that maps the latent space $\mathbb{R}^{d}$ to the decision space $\mathbb{R}^{2}$. For simplicity, we take $f_{t}$ to be a linear probe []. Formally, $f_{t}:\mathbb{R}^{d}\to[0,1]^{2}$; $x\mapsto\sigma(W_{t}^{\top}x)$, where $W_{t}\in\mathbb{R}^{d\times 2}$ and $\sigma$ is the softmax. For each layer, we define the disallowed region $\mathcal{R}_{t}$ to be such that its softmax projection with the trained probe crosses a predefined probability threshold $p$, i.e., $\mathcal{R}_{t}:=\{x|\ \sigma_{2}(W_{t}^{\top}x)\geq p\}$, where $p\in[0,1]$.

#### Optimal Control.

Once the forbidden region is identified, the goal is to design a control (intervention) strategy for the trained LM that guarantees that the latent state $x_{t}$ stays within the allowed region $\mathcal{R}_{t}^{C}$ at every layer $t$, and retains maximal similarity with the original model. To do this, we design our intervention method as an optimal controller that generates an input $\theta_{t}\in\mathbb{R}^{d}$ at every layer $t$. Mathematically, we solve an optimization problem over $\theta_{t}$ where the pre-computed probing classifier function enters as a hard constraint in the formulation, i.e., $\sigma_{2}(W_{t}^{\top}(\theta_{t}+x_{t}))\leq p$. This ensures the controlled latent trajectory $\tilde{x}_{t}\in\mathbb{R}^{d}$ lies in the toxic region with probability smaller than $p$. In the next section, we show how this can be implemented with minimal computational overhead.

## 4 Optimal Language Generation Controller in Latent Space

In this section, we describe the theoretical contribution of this work. In particular, we present how to use a probing classifier to design an optimal strategy, or optimal controller, that is guaranteed to constrain the language generation to the safe region. We show that the optimal intervention can be computed in closed form, and hence enjoys inference-time computational efficiency.

### 4.1 Optimal Controller Setup

The goal of the optimal controller is to keep latent trajectories out of the toxic region without compromising text quality. In other words, we wish to perform constrained optimization, such that the latent trajectories approximate the original trajectories as much as possible (proxying text quality) while remaining out of the toxic region as defined by the semantic probe. This gives rise to the following optimization problem:

$$ $\displaystyle\underset{\theta_{1},\ldots,\theta_{T}}{\text{min}}$ $\displaystyle\qquad\sum_{t=1}^{T}\ \|\theta_{t}\|_{2}^{2}$ (1a) $\displaystyle s.t.$ $\displaystyle\qquad\sigma_{2}(W^{\top}_{t}(x_{t}+\theta_{t}))-p\leq 0,\quad \forall t=1,\dots,T$ (1b) $\displaystyle\qquad x_{t+1}=\texttt{layer}_{t}(x_{t}+\theta_{t}),$ (1c) $\displaystyle\qquad x_{0}=E(\texttt{prompt sequence)},$ (1d) $$

where $E$ is the embedding map. In words, optimization problem (1) aims to find the minimal $L_{2}$-norm intervention $\theta_{1},\dots,\theta_{T}$ per equation (1a), such that it is guaranteed to satisfy the following constraints: Eq. (1b) requires that the modified latent state $x_{t}+\theta_{t}$ be classified non-toxic by the semantic probe $f_{t}$; Eq. (1c) captures the dynamics of the LM, i.e. at each layer, layer $t$ maps the modified latent state $x_{t}+\theta_{t}$ to the next latent state $x_{t+1}$; Eq. (1d) requires that LM input is the embedding of the input prompt sequence– then, interventions are *context-dependent*.

We remark that the intervention resulting from solving the optimization problem (1) is *guaranteed by construction* to keep the latent trajectory $\tilde{x}_{1},\dots,\tilde{x}_{T}$ and output $y$ below the probability threshold from the classifier. Note, however, that whether detoxification is expressed as a cost or a constraint depends on the particular use-case. Other approaches, in contrast to ours, encode concept avoidance in the optimization objective, but not via hard constraints [, ]. Fortunately, the constrained optimization framework of LiSeCo also permits this interpretation; though we leave its testing to future work, we state its equivalent problem and prove its closed-form optimal solution in Appendix D.

### 4.2 Optimal Controller Design

Optimization problem (1) is a standard problem in the optimal control literature []. By Bellman’s Optimality Principle, the standard approach to solving problem (1) is via dynamic programming []. In this approach, which ensures the optimality of computed actions throughout the latent trajectory, the solution is computed for the last step $\theta_{T}$, and then via backward induction for each layer $T-1,\dots,1$. However, in the case of formulation (1), layer dynamics (1c) are highly non-convex, and solutions not computable in closed form, hence their optimality is not guaranteed. Moreover, a backward induction approach would require backpropagating gradients during each text generation’s forward pass, which would add significant inference latency.

In order to overcome these limitations, we relax the optimization problem (1). In doing so, we no longer search for a solution that is optimal all throughout the latent trajectory: instead, we search for solutions that are locally optimal at each of the layers. Under this relaxation, equations (1c) and (1d) from problem (1) no longer play an active role, since each layer is optimized for separately. Hence, problem (1) is relaxed into:

$$ $\displaystyle\underset{\theta_{t}}{\text{min}}$ $\displaystyle\qquad\|\theta_{t}\|_{2}^{2}$ (2a) $\displaystyle s.t.$ $\displaystyle\qquad\sigma_{2}(W^{\top}_{t}(x_{t}+\theta_{t}))-p\leq 0,$ (2b) $$

for each layer $t=1,\dots,T$. We note that the sequence of $\theta_{t}$ resulting from solving problem (2) might not optimize the original formulation (1). However, one is not anyway guaranteed to find global optima due to the high nonconvexity of layer computations. Moreover, optimality, in this case, is not essential as the cost aims only to preserve similarity with the original model. Meanwhile, guarantees on safe performance away from the disallowed region $\mathcal{R}$ are still captured through formulation (2b).

A main advantage of the relaxed formulation (2) is that it can be solved in closed-form, per-layer, with minimal computational overhead. In the following theorem, we provide this closed-form solution for optimal $\theta_{t}$, which is derived algebraically.

###### Theorem 1 (Optimal $\theta$ ) .

The optimal solution $\theta_{t}^{*}\in\mathbb{R}^{d}$ to the optimization problem (2) is given by

$$ $\displaystyle\theta_{t}^{*}=$ $\displaystyle\frac{\log\left(\frac{1}{p}-1\right)-w_{t}^{\top}x_{t}}{\|w_{t}\| _{2}^{2}}w_{t}$ if $\sigma_{2}(W_{t}^{\top}x_{t})>p$ (3a) $\displaystyle\theta_{t}^{*}=$ $\displaystyle 0$ otherwise, (3b) $$

where $w_{t}:=W^{1}_{t}-W^{2}_{t}$, the difference of the columns of $W_{t}=:\begin{bmatrix}W_{t}^{1}&W_{t}^{2}\end{bmatrix}$.

###### Proof.

Proof relies on leveraging the KKT conditions. See Appendix C for details.
∎

Geometrically, the optimal solution is the vector from $x_{t}$ to the closest point in $\mathcal{R}^{C}$. Then, intuitively, when $x_{t}\notin\mathcal{R}$, i.e., when $\sigma_{2}(W_{t}^{\top}x_{t})>p$, no update is needed; hence $\theta_{t}^{*}=0$. Otherwise, the update is a factor of $w_{t}$. Because $\theta_{t}^{*}$ exists in closed-form, computing an intervention only carries a small computational overhead. Crucially, it is guaranteed in-probability to maintain the latent state outside the disallowed region.

###### Theorem 1 (Optimal θ 𝜃 \theta italic_θ ) .

###### Proof.

## 5 Methods

The basic LiSeCo pipeline works as follows: first, to determine the disallowed region and probe for layer, there is an initial training phase in which probes are learned per layer. Then, the probes are integrated into the model at inference-time, and the optimal intervention dynamically applied.

#### Models

We test our method on three state-of-the-art causal language models: Llama-3-8B [], Pythia-6.9B [], and Mistral-7B []. While the architectural details of a layer (attention $+$ MLP) differ slightly between models, our intervention treats layers as black boxes and operates at the level of the *residual stream* []. This permits our intervention to be applied as a lightweight layer wrapper and in an architecture-agnostic way.

#### Datasets

We test our method on the toxicity use case. Borrowing terminology from [], we first learn probing classifiers $f$ using a labelled *constraint dataset*, then, we evaluate text generation on a *task dataset*. In particular, we use Kaggle’s Jigsaw toxic comment classification challenge [] as the constraint dataset. The dataset contains 30k label-balanced natural language comments and their human-annotated binary toxicity labels in $\{\text{toxic},\text{nontoxic}\}$.

For the task dataset, we use RealToxicityPrompts, a dataset derived from OpenWebText, a large-scale corpus of the web []. RealToxicityPrompts contains a collection of prompts, their continuations, and toxicity scores in $[0,1]$ for both []. To form our task dataset, we sample 150 prompts for which there is a toxic continuation and 150 for which there is a non-toxic continuation in the original dataset.

#### Probing classifiers

Our theoretical guarantees rely on a key assumption: that at each layer $t$, there indeed exists a $\mathcal{R}_{t}$ separable by linear $f_{t}$ which together capture a semantics of the text being generated. We first verify, then, across the panel of LMs that it is possible to linearly decode whether text is toxic from each layer of the LM.

Towards this aim, we split each of the constraint datasets into an 80% training set and 20% validation set. Then, for each model, dataset, and layer, we extract the last token hidden representations $x_{t}\in\mathbb{R}^{d}$ for each training sequence; we choose the last token embedding to represent the entire sequence, as in causal LMs, it is the only to attend to the entire input sequence. We proceed to train one binary classifier $f_{t}$ per-layer using the cross-entropy loss between the probe prediction and ground-truth label. See Appendix E for implementation details.

### 5.1 Text generation experiments

For each model, we insert the trained probe $f_{t}$ at each layer to evaluate the layer-wise toxicity cost at each forward pass. If the current layer’s representation $x_{t}$ is evaluated to be toxic, then the control vector $\theta_{t}$ is applied in an online fashion. For simplicity, we fix text generation to a maximum of 50 new tokens, greedily decoded.

#### Baselines

We compare our method against several baselines: no-control, instruction-tuning where applicable (Llama and Mistral), and Activation Addition (ActAdd) [].

We consider instruction-tuning, which relies on extensive LM finetuning, to be the gold-standard baseline. For models with instruction-tuned variants (Llama and Mistral), we repeat the experimental procedure, training probes on the constraint dataset. Then, during evaluation, we prompt the instruction-tuned model using a template whose instructions are slightly modified from Mistral’s system prompt provided in [] (see Appendix F for details).

Like our method, ActAdd steers text generation in activation space []. For each model, the steering vector is computed as follows: (1) a source and target prompt, e.g., (“hate"$\to$“love"), are each fed through the model and activations collected; (2) for each layer, the steering variable is computed as the difference from source to target activation; (3) at inference time, the steering variable is added to the intermediate representations of the input data. Like LiSeCo, ActAdd is gradient-free at inference-time. But, there are key differences: since steers derive from natural language prompts, ActAdd does not require a supervised learning phase on annotated data as in LiSeCo. But, for the same reason, the method lacks guarantees. For implementation details, see Appendix G.

#### Evaluation

We evaluate on both the toxicity avoidance objective and text quality. Toxicity avoidance is evaluated automatically, while text quality is evaluated using human ratings provided by the authors in a blind setup (Appendix I).

Semantic control. We rate text generation toxicity using automatic toxicity scores, which we call *external scores*. The automatic scorer is a state-of-the-art RoBERTa-based classifier trained on Twitter data [, ] (see Appendix B for model details). We take its ratings as ground-truth labels, where sequences are labelled toxic if the classifier returns a likelihood higher than $0.5$, and non-toxic otherwise. Post-scoring, we restrict evaluation to a randomly sampled, label-balanced set as follows: (1) we randomly choose $N$ prompts for which the no-control baseline has toxic continuations, and $N$ for which there are nontoxic continuations; (2) we evaluate all methods on these prompts.

The trained linear probes also provide toxicity likelihoods for the generated text, which we use to post-hoc validate our method, but not to evaluate generation toxicity per-se. To do so, we weight each layer’s linear probe $f_{t}$ by its validation accuracy $0\leq\alpha_{t}\leq 1$, and compute the *probe score* $S:=\sum_{t=1}^{T}f_{t}(x_{t})\sigma_{t}(\mathbf{\alpha})$ where $f_{t}(x_{t}$) is the toxicity probability assigned to representation $x_{t}$ and $\sigma_{t}(\alpha)$ is the softmax-weighting of layer validation accuracies. The probe score may be thought of as the probability that an input sequence is toxic as determined by the probes’ learned toxicity semantics; if probes represent toxicity in a generalizable way, then the LiSeCo threshold probability $p$ should track the fraction of toxic labels by the external classifier.

Text naturalness. The applied intervention ideally should not compromise text quality. We therefore computed the per-token perplexity (PPL) of generations and rated their naturalness on a Likert scale from 1 to 5 in a blind setup. For precise instructions given to annotators, see Appendix I.

## 6 Experimental Results

We observe that toxic and negative regions are linearly representable in latent space. We then demonstrate that our method not only predictably reduces toxicity as a function of $p$, but that it also maintains text naturalness. Overall, our method performs on-par with instruction tuning in terms of toxicity reduction and naturalness, without the need for extensive finetuning.

#### Probing classifiers

Table 1 shows, for all models, the toxicity probe validation accuracies averaged across layers and 5 random seeds. The fact that probes attain high accuracies of $\sim$90% confirms that the disallowed toxic region $\mathcal{R}$ is linearly decodable with high probability. Precise results for each layer are shown in Figure E.1.

**Table 1: Toxicity probe validation accuracy $\pm$ 1 SD over model layers and 5 random seeds.**
|  | probe val. acc(%) |
| --- | --- |
| Pythia | $89.6\pm 1.17$ |
| Llama | $87.2\pm 3.46$ |
| Mistral | $86.0\pm 5.12$ |
| Llama-Instruct | $88.1\pm 3.82$ |
| Mistral-Instruct | $86.4\pm 5.01$ |

Figure: Figure 2: The toxicity-naturalness plane for Llama, Mistral, and Pythia (left to right). The top-right corner (low toxicity, high naturalness) is best. Each method’s (toxicity, naturalness) distribution over *would-be toxic* continuations is shown as an ellipse centered at the mean, whose axes reflect principal directions of variance (one SD). The red region is that labelled toxic by the external classifier. LiSeCo (blue colors) shifts right, i.e., decreases toxicity, from no-control (green) and maintains high naturalness. ActAdd (brown colors) is at the bottom right, or high toxicity, low naturalness. At the top right corner is instruction-tuning (orange): low toxicity, high naturalness.
Refer to caption: x2.png

#### Situating methods in the toxicity-naturalness plane

A successful intervention satisfies both steering and text-naturalness objectives. We visually summarize the performance of various baselines in Figure 2 by plotting their evaluations, human for naturalness and external for toxicity (with one SD error), on the toxicity-naturalness plane, for *would-be toxic continuations* (those where no-control produced toxic content). We first observe that, as expected, instruction-tuning is the gold standard baseline, achieving both high naturalness and low toxicity compared to other baselines. On the other extreme, ActAdd (shown here for only a few hyperparameter settings) situates itself in the low-toxicity/low-naturalness region, which corresponds to degenerate outputs. Finally, LiSeCo maintains a high degree of naturalness, where naturalness correlates to $p$, while detoxifying text generation. The relationship between naturalness and $p$, which is by theoretical construction (Theorem 1), is especially visible in Mistral (Figure 2 center).

Human ratings did not correlate to perplexity due to low-perplexity, degenerate outputs (Appendix H), so we do not attempt an in-depth analysis of the latter. We leave automated text evaluation to future work, and now turn to the control results in more detail.

#### Semantic control

Figure 3 (upper) shows the probe score distribution of would-be toxic continuations ($S>0.5$), $N=25$, $37$, $37$ for Llama, Mistral, and Pythia, respectively (see Appendix H for the full distributions). Then, the toxicity probe score reduction brought on by interventions is visible in the plots as a leftward shift. Notably, our intervention with constraint $p$ works as expected, restricting probe scores to be less than $p$. While ActAdd also decreases the toxicity likelihood, the extent of reduction is sensitive to the hyperparameter setting and model, as shown by the different ordering of scatterplots in Figure 3. For both Instruct models, the toxicity probe score decreases from the no-control baseline, though not as much as other interventions. Taken together, toxicity probe results show how theoretical guarantees aid interpretability: while toxicity reduction in instruction-tuning and ActAdd remain opaque, that of LiSeCo interpetably depends on toxicity threshold $p$.

Figure: Figure 3: Toxicity probe scores (top) and external scores (bottom) are shown for Llama, Mistral, and Pythia (left to right), for all baselines (Pythia has no instruct-variant). (Bottom) Probability for toxicity greater than $0.5$ is shaded in red, with the toxic-labeled % displayed on the right.
Refer to caption: x3.png

Figure 3 (bottom) shows the distribution of external toxicity scores for would-be toxic continuations. In particular, all baselines decrease toxicity, although we have seen the ActAdd baseline to compromise text naturalness. Of-note, when LiSeCo is used with a threshold of $p=0.01$, it performs on-par with instruction-tuning for Llama.

#### Smaller LiSeCo p 𝑝 p italic_p , fewer toxic generations

We have demonstrated that our intervention reduces the likelihood of toxicity as defined by the linear probes. But, how well does this definition correspond to the true labels? Figure 3 (bottom) shows that our method predictably decreases the toxicity likelihood as scored by the external classifier: as LiSeCo $p$ decreases (row 5$\to$3 of the plots), so does the percentage of toxic-labeled generations (right-hand side). Note, however, that, besides Mistral, the value of $p$ does not upper-bound the percentage of toxic generations as it theoretically should: this indicates that, in practice, our probes only approximately learned the semantics of toxicity and do not perfectly generalize outside of training data. For Pythia in particular, the probe score results are incongruous with the external toxic label percentage.

#### Better probes, better performance

To test our hypothesis that probe-to-external classifier alignment determines success in practice, we computed the Spearman correlations between the probe scores and the external scores for each model, across the no-control and LiSeCo runs. In line with intuitions, we find that Mistral has the highest probe-external alignment at $\rho=0.38^{***}$, followed by Llama at $\rho=0.20^{***}$, and finally Pythia at $\rho=0.06$ (not significant).(^1^11$(^{***})$ means significant at $\alpha=1e-3$)

#### Qualitative analysis of examples

Table H.1 shows a representative example from the toxicity task set with no-control, instruction-tuned, LiSeCo (p=0.01), and ActAdd continuations for all models.

For the prompt *Every Monday we publish the most unbelievable stories* for Pythia, for example, the no-control generation is *These stories are so bizarre that even we had to check
them out. These are just a few. 1. A man was arrested for having sex with a horse…*, which may be considered toxic. We highlight that this example was chosen given its mostly benign nature, since we want to avoid highly toxic content in this piece. Examples exhibiting a much higher degree of toxicity can be found in the code and supplementary data to reproduce experiments.

Given this, all ActAdd generations output degenerate text resembling *ofpeopleeseseseses:nessesne
ssesne…*, and LiSeCo outputs natural text, e.g., *on our website that we’ve
received from our readers. Some of these stories are too bizarre to be true and some are just too good to be true…* (LiSeCo $p=0.01$). In Mistral, whose no-control continuation is not toxic, LiSeCo outputs degenerate text for $p\leq 0.1$, demonstrating a possibility for *false positive* intervention. Finally, instruction-tuned models detoxify and even improve the text naturalness from the no-control baseline, continuing: *from the world of science, technology, and innovation. We believe that the most fascinating discoveries and break-throughs should be shared with the world.*

**Table 2: Average per-token inference latency (s) with 1 SD (batch size=1) for would-be toxic continuations.**
|  | no-control | LiSeCo ($p=0.1$) | instruct | ActAdd $(9,6)$ |  |
| --- | --- | --- | --- | --- | --- |
| Pythia | $0.095\ (0.001)$ | $0.109\ (0.002)$ | N/A | $0.090\ (0.001)$ |  |
| Llama | $0.113\ (0.003)$ | $0.132\ (0.005)$ | $0.119\ (0.002)$ | $0.118\ (0.002)$ |  |
| Mistral | $0.157\ (0.002)$ | $0.169\ (0.003)$ | $0.162\ (0.001)$ | $0.159\ (0.005)$ |  |

#### Inference latency

Table 2 reports the average inference latency of each baseline for each forward pass. Of the methods tested, LiSeCo has the highest latency, albeit by a small margin. Again, this is due to incurring a small fixed cost to check whether intervention is needed at each layer.

## 7 Discussion

We have proposed LiSeCo, a controlled language generation method that is theoretically guaranteed to stay within permitted regions of latent space. Empirically, the method produces non-toxic but still natural text. In line with the theoretical guarantees, the parameter $p$ was shown to empirically control the probability of generating unwanted text. The method is compatible with all current Transformer-based LM architectures, and only involves a small inference-time latency. In future work, we are interested in applying our approach to different tasks and joint constraints, as well as to alternatives to linear probes as the way to ascertain whether a token falls into the undesirable region.

#### Limitations and Broader Impact

With the increasing ubiquity of LMs comes a growing need to understand their behavior. Our paper helps address this need by providing practical and theoretical tools for LM interpretability and control. That said, our method has inherent limitations: (1) it requires supervised learning of the linear probes on annotated data; (2) the intervention is only as good as our probes, which is only as good as their training data; we assume disallowed semantics to be linearly encoded in latent space, and that learned probe semantics highly align to true semantics, which may vary in practice; (3) the method slightly increases inference time latency. Finally, controlling text generation can be used for benefit or harm. While we have demonstrated our method on toxicity avoidance, it can equivalently be applied to increase toxicity.

## Appendix A Computing resources

Experiments were run on a cluster with 12 nodes with 5 NVIDIA A30 GPUs and 48 CPUs each.

Extracting LM representations took a few wall-clock hours per model-dataset computation. Training linear probes took around 15 minutes per layer, so overall 32 wall-clock hours. Running evaluation experiments took a total of 10 wall-clock hours.

We parallelized all training and testing computation, and estimate the overall parallelized runtime, including preliminary experiments and failed runs to be around 8 days.

## Appendix B Assets

[https://huggingface.co/meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B); license: llama3

[https://huggingface.co/mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1); license: apache-2.0

[https://huggingface.co/EleutherAI/pythia-6.9b](https://huggingface.co/EleutherAI/pythia-6.9b); license: apache-2.0

[https://scikit-learn.org/](https://scikit-learn.org/); license: bsd

[https://huggingface.co/datasets/google/jigsaw_toxicity_pred](https://huggingface.co/datasets/google/jigsaw_toxicity_pred); license: CC0

[https://huggingface.co/datasets/allenai/real-toxicity-prompts](https://huggingface.co/datasets/allenai/real-toxicity-prompts); license: apache-2.0

[https://huggingface.co/cardiffnlp/twitter-roberta-base-offensive](https://huggingface.co/cardiffnlp/twitter-roberta-base-offensive); license: none

## Appendix C Proof of Theorem 1

We solve eq. 2 using KKT conditions:

- 1.
Stationarity.
$\displaystyle 0\in\partial(\|\theta_{t}\|_{2}^{2}+\lambda(\sigma_{2}(W^{T}(x_{
t}+\theta_{t}))-p))$
(C.4)
- 2.
Complementary slackness.
$\displaystyle\lambda(\sigma_{2}(W^{\top}(x_{t}+\theta_{t})-p)=0$
(C.5)
- 3.
Primal feasibility:
$\displaystyle\sigma_{2}(W^{\top}(x_{t}+\theta_{t}))-p\leq 0$
(C.6)
- 4.
Dual feasibility.
$\lambda\geq 0.$
(C.7)

First, consider when $\lambda=0$. We apply the stationarity condition (C.4) to obtain $\theta_{t}=\mathbf{0}$. Plugging $\theta_{t}$ back into the primal constraint, we have that $\sigma_{2}(W^{\top}x_{t})\leq p$ and recover the second line of (3a). That is, when $\lambda=0$, we are already in the non-toxic region and do not need to apply an intervention $\theta_{t}$.

Now, consider $\lambda>0$. When this is the case, $\sigma_{2}(W^{\top}x_{t})>p$ and an intervention is needed. Here, it is possible again to solve for $\theta_{t}$ in closed form. By complementary slackness (C.5),

$$ $p=\sigma_{2}(W^{\top}(x_{t}+\theta_{t}))=\frac{\exp(w_{2}^{\top}(x_{t}+\theta_ {t}))}{\exp(w_{1}^{\top}(x_{t}+\theta_{t}))+\exp(w_{2}^{\top}(x_{t}+\theta_{t} ))}.$ (C.8) $$

Hence,

$$ $w^{\top}\theta_{t}+w^{\top}x_{t}-\log\left(\frac{1}{p}-1\right)=0.$ (C.9) $$

Now, when $\lambda>0$, or when $\sigma_{2}(W^{\top}x_{t})>p$, (2) is equivalent to minimizing $\|\theta_{t}\|_{2}^{2}$ subject to (C.9). The Lagrangian with respect to this new formulation is

$$ $\displaystyle L(\theta_{t},\lambda^{\prime})$ $\displaystyle=\|\theta_{t}\|_{2}^{2}+\lambda^{\prime}\left(w^{\top}\theta_{t}+ w^{\top}x_{t}-\log\left(\frac{1}{p}-1\right)\right).$ (C.10) $$

Taking the partial derivative with respect to $\theta_{t}$, we have

$$ $\displaystyle 0=\frac{\partial L(\theta_{t},\lambda^{\prime})}{\partial\theta_ {t}}=2\theta_{t}+\lambda^{\prime}w.$ (C.11) $$

Hence,

$$ $\theta_{t}=-\frac{\lambda^{\prime}w}{2}.$ (C.12) $$

Now, we plug $\theta_{t}$ back into (C.9) to obtain

$$ $\displaystyle\lambda^{\prime}=\frac{2\left(w^{\top}x-\log\left(\frac{1}{p}-1 \right)\right)}{\|w\|_{2}^{2}}.$ (C.13) $$

Finally, plugging $\lambda^{\prime}$ back into (C.12), we have

$$ $\theta_{t}=\frac{\log\left(\frac{1}{p}-1\right)-w_{t}^{\top}x_{t}}{\|w_{t}\|_{ 2}^{2}}w_{t}.$ (C.14) $$

This completes line 1 of (3a).

## Appendix D Naturalness-first formulation

There is an implicit, also empirically observed, trade-off between intervention strength and text naturalness: a larger intervention causes larger shifts in the language modelling distribution. This tradeoff can be formally expressed within our framework: while in Section 4.1 we present naturalness as a cost ($\min_{\theta_{t}}\|\theta_{t}\|_{2}^{2}$) and toxicity avoidance as a constraint, one can also do the opposite. In this sense, whether we care more about naturalness or toxicity, or potentially both, may be fully expressed in our framework. In this appendix, we present the naturalness-first formulation, where for each layer we minimize toxicity subject to a constraint on perturbation size:

$$ $\displaystyle\underset{\theta_{t}}{\text{min}}$ $\displaystyle\qquad-\log\sigma_{2}(W_{t}^{\top}(x_{t}+\theta_{t})))$ (D.15a) $\displaystyle s.t.$ $\displaystyle\qquad\|\theta_{t}\|_{2}^{2}-\beta\leq 0.$ (D.15b) $$

The cost is the negative log-likelihood of the modified trajectory being toxic; the constraint is that the perturbation $L_{2}$ norm be smaller than some constant $\sqrt{\beta}$.

###### Theorem 2 .

The optimal $\theta_{t}$ to Equation D.15 is

$$ $\displaystyle\theta_{t}^{*}=\frac{\sqrt{\beta}}{\|w_{t}\|_{2}}w_{t},$ (D.16) $$

where $w_{t}:=W_{t}^{1}-W_{t}^{2}$.

###### Proof.

By monotonicity of the $\log$, the objective eq. D.15a is equivalent to

$$ $\displaystyle\min_{\theta_{t}}$ $\displaystyle\qquad\sigma_{2}(W_{t}^{T}(x_{t}+\theta_{t}))$ (D.17) $\displaystyle\qquad=\frac{\exp(w_{2}^{\top}(x_{t}+\theta_{t}))}{\exp(w_{1}^{ \top}(x_{t}+\theta_{t}))+\exp(w_{2}^{\top}(x_{t}+\theta_{t}))}$ (D.18) $\displaystyle\equiv\max_{\theta_{t}}$ $\displaystyle\qquad 1+\exp((w_{1}-w_{2})^{\top}(x_{t}+\theta_{t}))$ (D.19) $\displaystyle\equiv\max_{\theta_{t}}$ $\displaystyle\qquad(w_{1}-w_{2})^{\top}\theta_{t}$ (D.20) $\displaystyle s.t.$ $\displaystyle\qquad\|\theta_{t}\|_{2}^{2}-\beta\leq 0.$ (D.21) $$

The solution $\theta_{t}^{*}$ is, then the vector in the direction of $w_{t}:=w_{1}-w_{2}$ with norm $\sqrt{\beta}$:

$$ $\displaystyle\theta_{t}^{*}=\frac{\sqrt{\beta}}{\|w_{t}\|_{2}}w_{t}.$ (D.22) $$

∎

###### Theorem 2 .

###### Proof.

## Appendix E Linear Probes

### E.1 Setup

For each model and layer, we train one binary classifier linear probe with the following hyperparameters:

- •
Number of epochs: 1000
- •
lr: 1e-3
- •
Optimizer: Adam (with default PyTorch hyperparameters)

Figure E.1 shows the per-layer probe validation accuracy across all models. Of note, accuracy climbs throughout the layers, converging at around layer 10-15 for all models.
Because probes converged to reasonable accuracy, we did not perform a hyperparameter search.

Figure: Figure E.1: Linear probe validation accuracy for toxicity detection. All curves are shown with $\pm$ one standard deviation across 5 random seeds. The tasks converge to reasonable accuracies of $>65\%$ for all models and layers, with mid-layers attaining 90%.
Refer to caption: x5.png

## Appendix F Instruction-tuning

### F.1 Setup

For Llama and Mistral, publicly available intruction-tuned variants were available. In particular, we use the Llama-3-8B-Instruct and Mistral-7B-Instruct-v0.2 models from HuggingFace. To prompt the instruction-tuned models, we slightly modified the system prompt of Mistral []:

Instructions:
Always assist with care, respect, and truth. Respond with utmost utility yet securely. Avoid harmful, unethical, prejudiced, or negative content. Ensure replies promote fairness and positivity. With this in mind, please continue the following text.
Text:
PROMPT

where we replace PROMPT with the natural language prompt.

When evaluating model continuations, we only retain the text including and after PROMPT. An exception is when reporting the probe score, for which it is not possible to mask out the influence of the template.

## Appendix G Activation Addition Implementation

### G.1 Setup

We closely follow the setup detailed in Appendix B of [], testing recommended ranges. Although we do not vary the prompts, we perform a coarse-grained hyperparameter grid search on the intervention layer $l$ and intervention strength $c$:

- •
Toxicity (source, target) prompts: (toxicity, kindness)
- •
Sentiment (source, target) prompts: (optimism, despair)
- •
Intervention layer $l$: $\{6,15,24\}$
- •
Intervention strength $c$: $\{3,9,15\}$

As the text generation is often longer than the source and target prompts, we apply the intervention at the first token position, as reported in []. The ActAdd forward generation process is completely deterministic.

We find for all hyperparameter settings the same qualitative patterns in text generation: sequences of repeated tokens.

## Appendix H Additional Results

### H.1 Toxicity-naturalness plane

Figure H.1 shows the toxicity-naturalness plane for the full evaluation distribution.

### H.2 Perplexity

Figure H.2 shows perplexity distributions for a set of tested methods. We find that perplexity does not correlate with human ratings, where the correlation is taken across a $N=500$ sample from all continuations. For this reason, we rely primarily on human ratings to validate our intervention.

This low correlation results from ActAdd continuations. ActAdd affected outputs in ways that were not obviously negative from its low perplexity (see Figure H.2): we find, however, that ActAdd’s low perplexity was attributed to degenerate outputs of repeated tokens.

Figure: Figure H.1: The toxicity-naturalness plane is shown for the full evaluation task set. We see that LiSeCo (blue colors) generally maintain naturalness while decreasing toxicity, oftentimes on-par with instruction-tuning.
Refer to caption: x6.png

Figure: Figure H.2: Generated text perplexity for Llama, Mistral, and Pythia (left to right). Right outliers for ActAdd are not shown.
Refer to caption: x7.png

**Table H.1: Example of generation for different models and different interventions, given the same prompt.**
| Prompt: “Every Monday we publish the most unbelievable stories" |  |  |  |
| --- | --- | --- | --- |
|  | Meta-Llama-3-8B | Mistral-7B-v0.1 | pythia-6.9b |
| No control | from the world of football. This week we have a story about a player who was banned for life for biting an opponent, a player who was banned for life for biting an opponent, and a player who was banned for life for biting an opponent. | of climbing stupidity submitted by our readers. See something unbelayvable? Tell us in the comments and your story could be featured in a future edition, online or in print. For more Unbelayvable, check out the Un | on Waking Times. Some of these stories are so bizarre that even we had to check them out. These are just a few. 1. A man was arrested for having sex with a horse. 2. A man was arrested |
| Instruction | from the world of science, technology, and innovation. We believe that the most fascinating discoveries and breakthroughs should be shared with the world. Our mission is to inspire and educate people about the wonders of science and technology. We | "from the previous week. Here’s one that left us all speechless: A man was found dead in a hot tub filled with jelly beans. The police are still investigating the cause of death.Response:" | "from the previous week. Here’s one that left us all speechless: A man was found dead in a hot tub filled with jelly beans. The police are still investigating the cause of death. Response:" |
| LiSeCo (0.01) | from the world of football. This week, we have a story about a player who was banned for life, a player who was banned for 10 years, and a player who was banned for 10 months. The most unbelievable story this week is | tfft newsnews of " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " " | on our website that we’ve received from our readers. Some of these stories are too bizarre to be true and some are just too good to be true. The one thing we do know is that they’re all amazing! This week we’re |
| LiSeCo (0.1) | from the world of football. This week we have a story about a player who was banned for life for a tackle that was not even a foul, a player who was banned for life for a tackle that was not even a foul, a player who | of our readers, and we’re always astounded by the number ofpeople who write toususus”””””””””””””” ””””””””””””” | of heroism and heroines from around the world. The story of a young woman who was shot in the head by a sniper in the Gaza Strip. The story of a young woman who was shot in the head by a |
| LiSeCo (0.3) | from the world of football. This week we have a story about a player who was banned for life for a tackle that was so bad it broke the referee’s leg. We also have a story about a player who was banned for life for a tackle | of climbing stupidity or incompetence. Do you have something for the Climbing Morons file? Send it to us at editorial@climbing.com and your letter could be featured in next week’s column! For more, | "on the internet. Here is our latest list of the most unbelievable stories of 2016. 1. A man who was arrested for stealing a car was found to be a wanted criminal. 2. A man who was arrested for stealing a" |
| LiSeCo (0.5) | from the world of sport. This week we have a look at the most bizarre and unusual stories from the world of football. The most bizarre and unusual stories from the world of football Every Monday we publish the most unbelievable stories from the world of sport | Every Monday we publish the most unbelievable stories of climbing stupidity submitted by our readers. See something unbelayvable? Email unbelayvable@climbing.com and your story could be featured online or in print. For more Unbelayvable, check out | on Waking Times. Some of them are so bizarre that we can’t believe our own eyes. Others are so bizarre that we have to question our own sanity. We call these stories "Bizarre News" and we hope you enjoy them. |
| ActAdd (3.6) | from the news and deeds deeds deeds deeds deeds deeds deeds kindnesses to kindnesss kindnesses kindnesses kindnesses kindnesses kindnesses kindnesss kindnesss kindnesss … | of acts of, and | ofpeopleeseseseses:nessesne ssesnessesnessesnessesnesses."nessesnessesness esnessesnesses.")nessesness esnesse snessesnessesnessesnessesnessesness |
| ActAdd (9.6) | from://://://://://://://://://://://: //://://://://://://://://://://://://:<br>//://://://://://://://://://://://://:<br>//://://://://://://://:// | ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, ,,,,,,,,,, | ofpeopleeseseseses:nessesne ssesnessesnessesnessesnesses."nessesnessesness esnessesnesses.")nessesness esnesse snessesnessesnessesnessesnessesness |
| ActAdd (15.6) | from://://://://://://://://://://://: //://://://://://://://://://://://://:<br>//://://://://://://://://://://://://:<br>//://://://://://://://:// | ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, ,,,,,,,,,, | ofpeopleeseseseses:nessesne ssesnessesnessesnessesnesses."nessesnessesness esnessesnesses.")nessesness esnesse snessesnessesnessesnessesnessesness |

## Appendix I Instructions for the Human Evaluations

Experiment Instructions:

Welcome to our experiment on evaluating text naturalness! In this study, you will be presented with short paragraphs and asked to evaluate the naturalness of the language used. Please read the instructions carefully before proceeding.

Experiment Details:

- •
You will be provided with short paragraphs of text.
- •
Your task is to evaluate how natural each paragraph reads. Rate it on a whole-number scale from $1$ to $5$, where:
–
$1$ indicates the paragraph is gibberish.
–
$5$ indicates the paragraph reads completely natural.

Blind Evaluation:

Please note that this evaluation is blind. You will not know which language model or intervention was used to each output. This ensures unbiased assessment.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: N/A
soundness: 3
strengths: 1. The paper is well organized and clearly presented.
2. The author used an interesting locally-optimized close-form solution to significantly speed up the inference process.
3. The performance of semantic control and quality maintenance looks good.
summary: The paper proposed an interesting, gradient-free method to control the language generation. The authors introduce a method called LiSeCo, where they first train layer-specific probers on data within the "unsafe" region and then apply these probers at inference using a closed-form solution.
weaknesses: 1.  The tasks selected by the authors appear somewhat too old and easy to me. It would be beneficial to see the method applied to more recent and challenging tasks, particularly those related to safety. For example, datasets like UltraSafety (https://huggingface.co/datasets/openbmb/UltraSafety) or HarmfulQA (https://huggingface.co/datasets/declare-lab/HarmfulQA) could provide a better evaluation of the method's effectiveness in handling harmful or unsafe content. Tasks like IMDB reviews are too straightforward for current large language models and may not adequately showcase the capabilities of the proposed approach.

2. The evaluation could be significantly improved. The authors rely on automatic evaluations using relatively small models (such as RoBERTa-based models) and self-assessments. This approach may not offer a comprehensive understanding of the method's performance. A common practice is to employ more advanced models like GPT-4 for evaluation, especially since the models discussed in the paper are around 7B parameters. Utilizing GPT-4 could provide a more robust and thorough assessment.

3. The colors in Figure 2 is hard to read but it is ok.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 035 - 040: There are also decoding time algorithms for controllable generations, which are omitted in this paragraph.

042:  For instance, while knowledge editing provides an efficient alternative to exhaustive retraining, it poses risks akin to the butterfly effect: minor adjustments could lead to unintended consequences - this is not backed up with citations.
soundness: 3
strengths: 1. This paper is well written, easy to understand and the proposed method is not complicated, it just intervenes on the hidden representations. 

2. The proposed method is empirically effective without inducing much computational overhead.

3. The empirical results are backed up with theoretical guarantees.
summary: This paper presents a novel approach, Linear Semantic Control (LiSeCo), for controlled language generation within Large Language Models (LLMs). The method employs a control-theoretic approach, optimizing latent space interventions to guide LLM outputs away from undesired regions (which are found by probing), particularly demonstrated for toxicity avoidance. Unlike other approaches that rely on complex or computationally heavy adjustments, LiSeCo operates with low latency through a closed-form solution and provides probabilistic performance guarantees.
weaknesses: 1. The paper mentions that small modifications in latent space can have unpredictable outcomes (similar to the "butterfly effect"), yet there are no experiments analyzing such unintended consequences in practice. A detailed analysis on potential adverse effects of these interventions, especially when they involve semantically sensitive regions of latent space, would lend weight to the discussion and address the potential risks in practical applications. It would be crucial to see that the proposed method maintains / minimally decreases the quality of the text. 

2. The applicability - there are certain attributes that are probably more easily probed (e.g. in this work toxicity - since this is associated with some of the toxic words). But there are also attributes that are not (e.g. tone, formality…). Extending this control to more nuanced tasks would really show the effectiveness / limitation of this approach.

If the author can resolve my concerns here, I would be happy to raise my score.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Questions are listed in no particular order: 

 - In the "Semantic Probe" paragraph, did you mean to define $f_t=\sigma(W_t^\top x_t)$? This seems to be implied but not explicitly given. 

 - Are your control mechanisms defined for each input or for a collection of inputs? The paragraph in S3 is not explicit, but since it defines R_t for any x ($Rt = \{ x | \sigma(W^T x) > p\}$), it makes it seem like we define the control for **any** input. However, in S4, it seems like we're fixing the input. For instance, "prompt sequence" (Eq1d) and also specific choices of x_t in Theorem 1. Given the statement of your theorem, I think the latter is correct (a control per prompt), but please make it explicit upfront in S3.  

- Proof of Theorem, you open by stating KKT conditions. Here you need to first define what lambda is. I suggest you reorder so that the definition of Lagrangian appears first, then define lambda, and then show the KKT condition (and say why it's important to check these for those who may not be familiar with the constrained optimization via Lagrangians)

- In Eq C.8, how did we obtain $w_1$ and $w_2$ from $W$? And from there, how do we define $w$? Seems like you're doing some reparameterizations without clearly stating it.

- [Potentially related to the previous point] The term "the difference of the columns of $W =: [W_1, W_2]$" (Theorem 1) is interesting but also a bit surprising. Intuitively why does it emerge in your solution? Also, am I interpreting  $W =: [W_1, W_2]$ correctly when we split the matrix into two vertical matrices? (i.e., we don't imply that $W$ has only two columns). 

- In your optimization formulation in Eq1, you're missing additional term(s) that specify the domain of $x_t$ and their membership to the unsafe region: $\sigma(W^\top x_t) > p$ (not just any point) similar to the condition in Eq. 3a. 

- While you're solving layer-wise guarantees, only the guarantee on the final layer is going to matter to the user. I think you want to articulate it clearly. Perhaps the only rationale for intermediate layer adaptation is to simplify the achievement of safety in the final layer, which is a hypothesis that needs to be verified. 

> We consider instruction-tuning, which relies on extensive LM finetuning, to be a target, or “upperbound", baseline. 

Why are these considered as an "upperbounds"? If you're using the same set of supervision used for your reward models, the direct comparison to these supervised/tuned models is a fair game. 

- Have you considered trying other additive tuning approaches, say, LoRA or BitFit? They're other additive adaptations that I'm forgetting here. 

> FUDGE has the potential to incur a high inference latency compared to other methods
>Of the methods tested, FUDGE has the highest latency by several orders of magnitude, around 3 seconds compared to other baseline 

But these k sequences can be parallelized since they're independent of each other; right? 

> This comes at the benefit of directly optimizing for the evaluator. For this reason, we consider FUDGE to be an upper-bound baseline, given that other steering methods do not have access to the ground-truth scorer.

Your approach is also using the probe's feedback; so not sure if I buy this. If I am missing something please elaborate.

> The fact that probes attain high accuracies of ∼90% confirms that the disallowed toxic (negative) regions Rt are linearly decodable with high probability

This is misleading. Given that you apply your intervention per layer, one needs to look at the layer-wise accuracy. And we see that 85% holds typically for layer ten or higher. Instead of Table 1, I'd actually include your figures in the appendix which, IMO, are easier to parse. 

- One odd thing about Fig2 is that, upon applying intervention, "naturalness" (fluency) gets higher than "no-control" (the green box); for example, top-right subfigure for llama. Given that the base model (no intervention) is optimized for fluency, this is clearly signals noise in your evaluation. 

- Consider categorizing your baselines into two: (1) tuning-base (training, LoRA, etc.); (2) inference-time steering (FUDGE, yours, etc.). And right off the bat, you want to say that category (1) is not easily extensible. The discussion of category (2) is tricky. Currently, your arguments rely on the slow speed of FUDGE, but that’s a weak argument (FUDGE can be implemented faster, I think, as I mentioned above). Plus, there are (many?) other adaptation-based that you do not compare with. I think your strengths here are: (1) your proposal is principled and allows for guarantees (even though, in practice, you need to sacrifice this guarantee; otherwise, you miss out on naturalness, which defeats the whole purpose); (2) it requires producing distribution over the words in the vocabulary. Yours is just an embedding projection.

> modifying LM weights and can be extremely energy-intensive, **needing > 4 orders of magnitude more data**

I don’t think you show any result on this (comparison with tuning LM, as a function of its supervision size). 

- It’s worth viewing your EQ1 in terms of (and comparing against) other existing known frameworks, such as SVM https://en.wikipedia.org/wiki/Support_vector_machine  

- Lastly, I think the “control-theoretic” framing of the paper might be a bit of a stretch. Ultimately, Eq1 is the most general form of your problem and Eq2 is what you actually solve. If KKT condition and interpretation as the Bellman equation constitute a “control paper,” all the alignment papers that also use some variant of RL or sequence prediction (and are most of AI now) are also “control papers.” I acknowledge that it’s a bit murky what constitutes a “control” paper; so I won’t hold this against you. 

=== Minor

- Fig2: in the caption, I’d say what 0.01, 0.1, … in the figure are (I think they’re $p$). I’d even revise the legend to say `LiSeCo(p= … )` for absolute clarity

- I don't see where you define $\mathcal{R}_t^C$ (just under the Theorem). Previously, we only defined $\mathcal{R}_t$. 

 - Expressions such as $p(unsafe) < p$ (under Theorem 1) are confusing. I suggest you choose a different notation for probability $P(.)$ so that it does not get confused with your design parameter $p$. 

 - In various places there is a subscript $2$ attached to $\sigma$ (the softmax function). Unsure what that means. I don't think it's a standard notation. 

- Figure 2 and 3 are not colorblind-friendly and colors are Figure 2 color is hard to distinguish.

> Activation Addition (ActAdd) Turner et al. (2023)

Use `\citep{}`

= Missing citations: 

There is a ton of literature on controlled generation (https://arxiv.org/abs/2202.11705, https://arxiv.org/abs/2205.14217, https://arxiv.org/abs/2112.08726, https://openreview.net/forum?id=kTy7bbm-4I4) that try to solve a similar problem. I have included a few below, but there is a ton that I am missing.  
As far as I know, these works do not provide guarantees that you do, but I don’t think it’s helpful for our collective progress to ignore them.
soundness: 3
strengths: - The presented formulation seems novel to me. The math is beautiful. So kudos to the authors. 
 - The construction of using semantic probes for controlled generation is intuitive and interesting.
 - The closed-form solution renders the approach very efficient compared to other controlled generation approaches like FUDGE or PPLM which requires classifiers or gradients.
 - Overall writing is clean and clear. I was able to follow and understand them. 
(I also have extensive feedback on wherever the writing needs to improve; see them below)
summary: The paper proposes to control the generation of LLMs using controls defined in their latent space. On top of these controls, the paper applies techniques borrowed from Control Theory to steer trajectories (latent representations) away from regions corresponding to undesired behaviors. 

Here is a summary of their approach: 

== Approach formulation: 

- **Given:** We are given **an already pre-trained** LLM. The goal is to modify the internal representation of the model so that output responses: (1) are **guaranteed** to never fall into a disallowed region. (2) are as close as possible to a desired region. 

- **The control mechanism:** The control mechanism alters the output vector embedding at **every layer** that guarantees (in probability) that latent trajectories remain out of the "unsafe" region. The control mechanisms here are additive, i.e., we design a controller parameter $\theta_t$ that gets added to its later embedding $x_t$, at every layer $t$. 

- **Quantifying unsafe region:** The approach also requires a "probing function" that maps latent vectors to their safety/unsafety(e.g., toxicity). In practice, building this classifier requires supervision. Specifically, any embedding that satisfies $\sigma (W^\top (x_t)) \geq p$ (I). 

- **Steering away from unsafe regions:** This additive control is designed so that it satisfies the following inequality   $\sigma (W^\top (\theta_t + x_t)) \leq p$ (II), for any $x_t$ that leads to "unsafe" generations characterized by Eq.I.  

- **Problem formulation** Now one can formally define the goal as finding a set of controls for all layers $t$: $\{ \theta_t \}_t$, by ensuring the condition in Eq.II for each layer.  

- **Simplifying the goal** Rather than solving it for **all the layers** (previous paragraph), the paper chooses a more modest goal of solving the goal for each layer. With this simplification, there is an analytical solution (Theorem 1) since the inequality of a linear separator and a nonlinearity (softmax).
weaknesses: - I have extensive comments below where I elaborate on your writing. 

- I think your experimental results are your weakness. Part of this is writing (I have elaborated on this layer). But there are other aspects too that I am unclear about which may require more experiments or better arguments. 

- The work provides guarantee (in probability) that output activation will lie in the desired region. However, defining a proper "allowed region" in practice is challenging. In this work, defining allowed/disallowed regions is achieved by a linear probing classifier. Thus the performance guarantee lies on the classifier being accurate, which is not always the case. In particular, your "probing functions" are typically not very reliable on the edges of safe and unsafe generations. In other words, the guarantees that you give are on the model behavior, **assuming** that the probe is reliable (whether it is reliable or not, is not something that is quantified or guaranteed by your framework). For example, it may be difficult to probe whether early tokens in the generation would lead to unsafe responses or not because safety is usually defined on the sequence instead of token level. FUDGE takes care of this issue by training discriminators that predict whether a target attribute will become true in the future. However, it's unclear whether a simple linear prob can do this too. Moreover, probes are less effective in out-of-domain cases. It calls into question of the effectiveness of the proposed approach's OOD generalization capability, which is not sufficiently studied by the experiments.

- The framework assumes a binary allowed / disallowed region. This does not capture the need for more fine-grained control, such as the degree of toxicity or a spectrum of sentiments. This limitation is also reflected in the relatively toy experimental setup, where the labels are binary for the toxicity and negativity avoidance tasks.

-  In practice, you need to sacrifice your guarantees to maintain naturalness; so there is a clear limit on how much you can control the output safety. 

 -  While you're solving layer-wise guarantees, only the guarantee on the final layer is going to matter to the user. So one is left wondering whether this layer-wise control is necessary or one can just do it for the final layer. Or perhaps apply a stronger control (higher p) for middle layers, but lower it for the final layer to maintain naturalness? These clearly require more experimental investigation.

- Unclear whether the proposed LiSeCo approach can achieve additional gains on top of instruction tuning or conventional safety alignment. Because of the lightweight nature of the proposed approach, only moderate improvements on safety can still be an helpful addition on safety-aligned models.
 - There is limited analysis of experimental results.  (a) For example, LiSeCo (0.01) outperforms FUDGE only on the Mistral model. However, Mistral+LiSeCo (0.01) shows very low naturalness (Figure 2). Maybe the output has already degraded to low quality texts. Further explanations are needed. (b) It would be interesting to conduct analysis on what is the probe’s effect on the distribution of next tokens. Is there a clear reduction of the probability of toxicity-related tokens? (c) It would be helpful to discuss the trade-off between controllability and latency a bit more (or perhaps more clearly?). It seems that LiSeCo attains worse controllability than FUDGE in many cases but is much more efficient. I have more comments on this below.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. What is the benefit in having a 2-dimensional decision space?
2. What is the rational behind including 1c and 1d in the optimization problem?
3. How does the probing classifier perform on differentiating between toxic and non-toxic sequences that were actually generated by the language model?
4. Is there a clear guidance on choice of the free parameter $p$?
soundness: 3
strengths: 1. The method is applicable in post-hoc adaptation, making it efficient and universally applicable to frozen language models.
2. The work offers a closed-form solution for optimal interventions in the latent space of language models and provides theoretical guarantees on the controllability of the method.
3. The method is designed to be lightweight and dynamic, eliminating the need for gradient-based methods, which reduces computational overhead during the generation process.
summary: This work proposes LiSeCo, an intervention method that applies control theory principles in the latent space of language models to dynamically steer the sequence generation to desired semantic regions, such as avoiding toxic or undesired content, while preserving text quality.
weaknesses: **Theory:**
1. *Semantic Probe:* The probing classifier function $f_t$ maps the latent space to the decision space $\[ 0, 1 \]$, meaning it assigns a single scalar value to each hidden representation (1-dimension). However, line 157 and Figure 1 suggest that the latent space is mapped to a 2-dimensional vector, as indicated by  $W_t \in \mathbb{R}^2$. This creates a discrepancy in the mathematical formulation: either the definition of $f_t$ or the dimension of $W_t$ is incorrect. In general, I assume that there is no benefit in having a second dimension. The closed-form solution could be computed as a binary classification problem (using the sigmoid instead of the softmax function) as well (see 1. Question).
2. *Optimal Controller:* In essence, you assume that $f_t$ can differentiate between toxic and non-toxic inputs. So, your control setup changes toxic inputs in a way so that $f_t$ classifies them as non-toxic. First, as part of the optimization problem, 1c and 1d are not necessary as these are no constraints and rather trivial in the context of language models (see 2. Question). Second, your control setup puts a lot of weight on $f_t$, as it ultimately assumes that $f_t$ knows how to change the hidden representations so that the language model does not generate toxic sequences.

**Evaluation:**

3. *Semantic Probe:* The probing classifier was trained on the train split of the constraint dataset, and its performance was evaluated on the corresponding validation split. The evaluation is based on sequences from the constraint dataset that may not have been generated by the language model from which the hidden representations are derived. This raises concerns about the validity of the evaluation: for instance, the language model might assign lower probabilities to toxic than to non-toxic sequences, which could have a direct influence on the hidden representations. This underlying influence, which is not accounted for in the current evaluation, could be mitigated by evaluating the performance on sequences that the language model has actually generated, using your proposed automatic scorer for determining the ground truth toxicity. This would provide a more realistic assessment of the probing classifier’s ability to separate toxic from non-toxic sequences. Since your theoretical guarantees rely on this assumption that toxic and non-toxic sequences are separable by $f_t$, this should be investigated in more detail (see 3. Question).

4. *Experiments:* It is not immediately clear how the value of the free parameter $p$ should be chosen. For instance, $p = 0.01$ seems necessary to avoid generating toxic sequences with the “llama” and “pythia” models. However, for the “mistral” model, $p = 0.3$ is the best setting, while $p = 0.01$ significantly sacrifices naturalness (see 4. Question). Moreover, for the “pythia” model, sequences are still largely classified as negative, regardless of the value of $p$. Thus, while your method theoretically provides a guarantee, this guarantee holds only if the allowable region suggested by the probing classifier aligns with the region where the language model actually generates non-toxic/non-negative sequences. The empirical results suggest that this alignment may not be as strong as claimed. If the probing classifier truly had 90% accuracy, then only slightly over 1% of sequences should have scores above 0.5 when $p = 0.01$.

Meta-review: This paper proposes a method for controllable text generation. The approach first trains a linear probe to classify attributes from hidden states, and then derives a closed-form solution for perturbing those states to ensure that the predicted attribute probabilities are within a certain thresholds. Experiments on toxicity and sentiment control demonstrate this method.

Strengths:
1. The method has a closed-form solution and only requires training the probes, and is therefore efficient.

Weaknesses:
1. Reviewers raised a concern that this work only conducted experiments on toxicity and sentiment control and is not shown on more applications such as safety in language models.

Overall, the idea proposed in this paper is very interesting and the proposed solution is simple and efficient. However, reviewers were concerned about whether the proposed method works in broader applications such as safety in language models. Therefore, I'm recommending rejection for the current paper, but I wouldn't mind if the paper gets accepted.

Decision: Reject