CBF-LLM: Safe Control for LLM Alignment

## Abstract

Abstract This paper proposes a control-based framework for aligning large language models (LLMs) by leveraging a control barrier function (CBF) to ensure user-desirable text generation.
The presented framework applies the safety filter, designed based on the CBF, to the output generation of the baseline LLM, i.e., the sequence of the token, with the aim of intervening in the generated text.
The overall text-generation system is implemented with Llama 3 and a RoBERTa model, and the source code is available at https://github.com/Mya-Mya/CBF-LLM .
The experiment demonstrates its control ability and effectiveness in reducing the number of interventions needed for user-specified alignment tasks.

## 1 Introduction

While large language models (LLMs) are known to have strong language understanding and generation abilities,
they can also generate harmful, biased, and toxic content [][].
Alignment of LLMs ensures that they generate content that is “desirable” for the user, typically meaning content that is safe and ethical.
Various approaches for LLM alignment have been presented ([], [], [] and reference therein).

The major approach to the alignment is reinforcement learning from human feedback (RLHF) [],
where a reward model is constructed by human feedback and used for the training of LLMs.
Variants of RLHF architectures are also proposed, such as Safe-RLHF [], SENSEI [], and f-DPG [],
and their implementations are presented, such as training pre-trained LLMs [][], and applications like information-seeking chatbot [].
Collecting human feedback with data is time-consuming and expensive.
To overcome the drawback, RL from AI Feedback (RLAIF) is presented in [] instead of using human labels. In addition, the method to construct the training data automatically is proposed in [].
Furthermore, to reduce the computational cost, direct preference optimization (DPO) is proposed [], where the training data is directly used for training LLMs without accessing the reward model.
Supervised fine-tuning (SFT) is a different approach for alignment from RLHF, as studied in []. A common feature of alignment methods like RLHF and SFT is that they modify LLMs’ model parameters.

An alternative approach for LLM alignment is to directly intervene in the input prompt or output of LLMs, rather than modifying the model parameters.
In-context learning (ICL) [] is a major approach for intervening in the input prompt.
In ICL, a few demonstrations are provided in prompt to instruct the LLMs on the task, including few-shot learning [][].
As the methods for intervening in the output,
the work [] proposes a method to format output for retrieval application, and the work [] proposes a repetition penalty to prevent LLMs from repeating the same words and expressions.
In addition, Transformers module provides some functions to modify the output, such as NoBadWordsLogitsProcessor and MinLengthLogitsProcessor [].

One can view that the intervention approach to alignment, which avoids undesirable output, as analogous to “collision avoidance”, the most fundamental problem in control engineering.
In control engineering, various studies are conducted for safety assurance, including collision avoidance [][][].
A promising approach for collision avoidance is the control barrier function (CBF), as studied in theoretical works [][][] and in real-world applications [].
An analogy between vehicle collision avoidance and intervention-based LLM alignment can be drawn as illustrated in Fig. 1.
The goal of vehicle collision avoidance is to prevent collisions with obstacles by intervening in the vehicle’s trajectory.
For example, if there are obstacles ahead of the vehicle, it is necessary to operate the steering or use the brakes to avoid colliding with them.
Similarly, the goal of LLM alignment is to prevent undesirable outputs, such as harmful or biased content, by intervening in LLM’s trajectory, i.e., the sequence of generated tokens.
Consider the LLM as an analogy to a vehicle, and the generated text as an analogy to the vehicle’s trajectory.
Both vehicle collision avoidance and LLM alignment aim to guide the complex system away from undesirable states by designing proper control strategies.
Based on this analogy, this paper develops a control strategy for LLM alignment based on the CBF,
focusing on intervening in the trajectory of generated text, i.e., the sequence of generated tokens from LLMs.

Figure: Figure 1: Concept of CBF-LLM. Upper: Collision avoidance in a vehicle control system, Lower: Collision avoidance in text-generation by LLMs.
Refer to caption: x1.png

This paper proposes a framework for control-based LLM alignment by applying a safety filter that intervenes in the LLM output to generate the user-desirable outcomes.
To this end, we leverage the idea of the CBF to improve the safety and controllability of the outputs of LLMs.
We aim to design the CBF-based safety filter that intervenes in the output of LLMs into the user’s desired content.
The CBF filter and the baseline LLM constitute a novel text generation system, which we call “CBF-LLM”.
This paper also conducts the text-generation experiment on CBF-LLM.
In the experiment, the CBF control enabled the alignment task to be completed with fewer interventions.

The contributions of this paper are as follows:

- •
The proposed CBF-LLM is realized in an add-on manner to a baseline LLM: an external filter is simply added to the LLMs
without accessing their model parameters.
In this sense, CBF-LLM is a “learning-free” framework for aligning LLMs.
CBF-LLM is broadly applicable to various LLMs, as is designed independently of the underlying LLM.
- •
The concept of CBF-LLM is implemented with Llama 3 and a RoBERTa model in Section 4. The source code can be accessed in [https://github.com/Mya-Mya/CBF-LLM](https://github.com/Mya-Mya/CBF-LLM).

This paper also attempts to bridge the gap between control engineering and NLP.
While the theoretical analysis of LLMs, such as reachability, has been studied in the works [] and [],
this paper presents a design method of LLM-based text-generation systems.

The rest of this paper is organized as follows.
In Section 2, the basic theory of CBF is provided, and the structure of nominal LLM is reviewed.
In Section 3, the concept and design of CBF-LLM are proposed.
In Section 4, the experiment of CBF-LLM is conducted.
Finally, in Section 5, the conclusion of this paper is presented.

Notation: symbol $V[i]$ represents the $i$-th element of the vector $V$.

## 2 Preliminary

### 2.1 Control Barrier Function for Safe Control

Control barrier function (CBF), developed in the control community, provides safety assurance in control systems [].
This section briefly reviews CBF and CBF-based safety control.

Consider the following dynamical system to be controlled:

$$ $\displaystyle\dot{x}=g(x,u),$ (1) $$

where $x\in\mathbb{R}^{n}$ is the state variable of the object being controlled, and $u\in\mathbb{R}^{m}$ is the action applied to the object.
The function $g$ represents the system dynamics; how this object is affected by the current state $x$ and action $u$.
A typical example of the system (1) includes a vehicle dynamics,
where the state $x$ is coordinate, velocity, angle, etc, and the action $u$ is accelerator pedal depression, steering angle, etc.

We aim to design the assisted control system with safety assurance.
As for safety, we let the safe and unsafe sets be denoted by $\mathcal{S}\subseteq\mathbb{R}^{n},\bar{\mathcal{S}}\subseteq\mathbb{R}^{n}$, respectively.
Then, the safety means to constrain $x$ within the safe set $\mathcal{S}$, i.e., $x\in\mathcal{S}$.
Consider that the nominal action $u_{\mathrm{nom}}\in\mathbb{R}^{m}$ is provided which might violate the safety, i.e., $u_{\mathrm{nom}}$ might generate the unsafe state $x\in\bar{\mathcal{S}}$.
Then, we address the problem of designing “safety filter” $F:\mathbb{R}^{m}\to\mathbb{R}^{m}$, as follows:

###### Problem 1 (Safety Filter) .

Find the safety filter $F:\mathbb{R}^{m}\to\mathbb{R}^{m}$ such that the system (1) with $u=F(u_{\mathrm{nom}})$ generates $x(t)\in\mathcal{S}$ for all nominal actions $u_{\mathrm{nom}}$ for all time $t$.

As a preliminary, we design a continuously differentiable function $h:\mathbb{R}^{n}\to\mathbb{R}$, called a “constraint function”, such that

$$ $\displaystyle h(x)\geq 0,\quad$ $\displaystyle x\in\mathcal{S},$ (2) $\displaystyle h(x)<0,\quad$ $\displaystyle x\in\bar{\mathcal{S}}$ (3) $$

hold.
The safety is equivalent to the constraint: $h(x)\geq 0$.
In the assisted driving example, the nominal action $u_{\mathrm{nom}}$ is a manual action by the driver, and the unsafe set $\bar{\mathcal{S}}$ includes the positions of obstacles like pedestrians.
The function $h$ can be the distance between the vehicle and the obstacle.
With the manual action by the driver $u_{\mathrm{nom}}$, the vehicle may enter the unsafe set, such as colliding with obstacles.
The safety filter $F$ needs to modify $u_{\mathrm{nom}}$ to output $u$ such that $h(x)\geq 0$ holds, i.e., the trajectory in which the vehicle never collides with obstacles.

To construct the safety filter $F$ that keeps the safety constraint $h(x)\geq 0$,
the control barrier function filter (CBF filter, [][]) is presented.
The CBF filter intervenes in the nominal action value $u_{\mathrm{nom}}$ to introduce a safe state of the object as follows:

$$ $\displaystyle u=$ $\displaystyle~{}\arg\min_{u}(u_{\mathrm{nom}}-u)^{2},$ (4) s.t. $\displaystyle~{}\dot{h}(x)\geq-\alpha(h(x)),$ (5) $$

where $\alpha:\mathbb{R}\to\mathbb{R}$ is a class-$\mathcal{K}$ function which holds $h(0)=0$ and monotonically increasing, i.e., $\frac{dh(x)}{dx}>0$.
The function $h$ is called a control barrier function if there exists $u$ such that the constraint (5) holds.
In addition, (5) is called a CBF constraint.
In the assisted driving example, when the manual action $u_{\mathrm{nom}}$ is expected to cause a collision with an obstacle, the CBF filter intervenes in $u_{\mathrm{nom}}$ to provide safe driving.
To constrain the action $u$ by the CBF filter, the following statement holds:

###### Theorem 1 .

The state $x$ of the system (1) is in the safe set, i.e., $x\in\mathcal{S}$ for all time if $h$ is a control barrier function and the action $u$ satisfies the CBF constraint (5).

###### Remark 1 .

The objective function (4) ensures that the filtered action $u$ remains as close as possible to the nominal action $u_{\mathrm{nom}}$.
In this sense, the CBF filter archives safety by the “minimum” intervention.

The CBF filter is capable of applying in discrete-time systems by re-formulating the CBF constraint (5) as follows []:

$$ $\displaystyle\Delta h(k)=h(k+1)-h(k)\geq-\alpha(h(k)),$ (6) $$

where $k$ is a discrete time.

###### Problem 1 (Safety Filter) .

###### Theorem 1 .

###### Remark 1 .

### 2.2 Text generation by Large Language Models

This section reviews and analyses the text generation by large language models (LLMs) while particularly focusing on their structure.
In this paper, “text” means the sequence of tokens and $\mathcal{X}$ denotes the set of the texts.
The text corresponding to a specific expression in natural language is displayed as $x=x(\text{``}\texttt{<text>}\text{''})$.
For example, the text $x\in\mathcal{X}$ for “Have a nice day.” is displayed as $x(\text{``}\texttt{Have a nice day.}\text{''})$.
Each token $t$ is identified by a positive integer, i.e. $t\in\{1,\ldots,N\}=:\mathcal{T}$, and $N$ is the number of tokens the LLM has.
The token corresponding to a specific expression in natural language is displayed as a numerical constant $t_{\texttt{<token>}}$.
For example, the token for “dog” is displayed as $t_{\texttt{dog}}$.
The function $\texttt{concat}:\mathcal{X}\times\mathcal{T}\to\mathcal{X}$ concatenates text and token to output a text. For example,
$\texttt{concat}(x(\text{``}\texttt{Have a nice}\text{''}),t_{\texttt{day}})=x(
\text{``}\texttt{Have a nice day}\text{''})$.

Text generation is performed by iteratively adding a new token, starting from the initial text.
The text generation by LLM is considered to be a discrete-time dynamical system as shown in the block diagram Fig. 2.

Figure: Figure 2: Nominal structure for text generation
Refer to caption: x2.png

In this figure, the token predictor $G:\mathcal{X}\to\mathbb{R}^{N}$ plays a central role in the text generation:
it drives the input text $x$ to output the probability $P\in\mathbb{R}^{N}$, where $P[t],t\in\mathcal{T}$ displays how probable that the token $t\in\mathcal{T}$ would follow by the text $x$.
This $G$ is typically replaced by LLMs.
The token selector $C$ selects the next token $t^{*}$ which follows $x$ based on the probability $P$.
Finally, in the concatenator concat, the new token $t^{*}$ is concatenated with $x$ to derive the new text $x$.
The $Z^{-1}$ block represents the time delay, playing a role of a memory of text $x$.

Let $x_{0}\in\mathcal{X}$ be the initial text, and $k\in\{0,1,\ldots\}$ be the “time”, which counts the number of tokens added during the generation. Then, the text generation is expressed as follows:

$$ $\displaystyle\begin{cases}x(0)=x_{0},\\ P(k)=G(x(k)),\\ t^{*}(k)=C(P(k)),\\ x(k+1)=\texttt{concat}(x(k),t^{*}(k)).\end{cases}$ (7) $$

As described above, the text-generation LLM such as GPT-2, and Llama 3 is included in the token predictor $G$. The token predictor not only accommodates the LLM but also has the function of processing the output of the LLM.
Let the text-generation LLM as $f_{\mathrm{LLM}}:\mathcal{X}\to\mathbb{R}^{N}$. Then, the probability $P$ is calculated as follows:

$$ $\displaystyle P[i]=\mathrm{softmax}(f_{\mathrm{LLM}}(x)[i]/T)=\frac{\exp(f_{ \mathrm{LLM}}(x)[i]/T)}{\sum_{j=1}^{N}\exp(f_{\mathrm{LLM}}(x)[j]/T)},$ (8) $$

where $T\geq 0$ is the temperature.

The algorithm of the text generation is given as follows:

Figure: Algorithm 1 Nominal text generation

## 3 CBF-LLM

This section presents the control-based alignment of text-generation systems and their detailed implementation.
Symbols and their meanings used in this paper are summarized in Table 1.

**Table 1: Symbols and their meanings**
| Symbols | Meaning in CBF | Meaning in LLM Alignment (CBF-LLM) |
| --- | --- | --- |
| $x$ | State of the controlled object, $x\in\mathbb{R}^{n}$. | Generated text, $x\in\mathcal{X}$. |
| $h$ | Constraint function, $h:\mathbb{R}^{n}\to\mathbb{R}$. | Language-constraint function (L-CF),<br>$h:\mathcal{X}\to\mathbb{R}$. |
| $\mathcal{S}$ | Safe state set. | Desirable text set. |
| $\bar{\mathcal{S}}$ | Unsafe state set. | Undesirable text set. |

The alignment discussed in this paper aims to ensure desirable text generation by weak intervention to the output of LLMs.
To clarify the meaning of “desirable”, we let the desirable and undesirable text sets be $\mathcal{S}\subseteq\mathcal{X}$ and $\bar{\mathcal{S}}\subseteq\mathcal{X}$, respectively, based on the respective alignment goals.

###### Example 1 .

Suppose that the alignment goal is set to generate non-toxic content.
Then, $\mathcal{S}$ is the set of non-toxic text, and $\bar{\mathcal{S}}$ is the set of toxic texts.
For mathematical procedures, we consider $\mathcal{S}$ and $\bar{\mathcal{S}}$ to represent “all” non-toxic and toxic text samples.
More examples of $\mathcal{S}$ and $\bar{\mathcal{S}}$ are seen in Section 4.

One simple idea of achieving the alignment goal is to force the text generation to stop when the generated text $x$ turns undesirable, i.e., $x\in\bar{\mathcal{S}}$, but this method involves a strong intervention in the baseline LLM, which renders the original capabilities of the baseline LLM meaningless.
To overcome the drawback, we make the intervention strength adjustable, which enables the text-generation system to achieve the alignment goal with a weak intervention.

The presented text-generation system, including an LLM and a safety filter, which is constructed based on the CBF described in Section 2.1.
The overall system is called CBF-LLM and its structure is shown in Fig. 3.

Figure: Figure 3: Structure of presented text-generation system, named CBF-LLM
Refer to caption: x3.png

CBF-LLM extends the nominal text-generation system shown in Fig. 2 by adding the safety filter (yellow box) added between the token predictor $G$ and the token selector $C$.
The safety filter manipulates the probability $P$ to satisfy the specified alignment task.
The safety filter is composed of filter $F$ and normalizer $R$. In the same manner as Section 2.2, the blocks of $G$, $C$, concat, and $Z^{-1}$ represent the token predictor, token selector, concatenator, and time delay, respectively.
The components of CBF-LLM are described in detail as follows.

Recall that the token predictor $G$ mainly implies a generative language model such as LLM.
It retrieves the current text $x\in\mathcal{X}$ and outputs the probability of the next token, $P\in\mathbb{R}^{N}$.
For each token $t\in\mathcal{T}$, the probability $P[t]$ indicates how probable each token $t$ is followed after the text $x$.

The defining feature of the CBF-LLM is the presence of filter $F$, which filters $P$ to generate the modified probability $Q\in\mathbb{R}^{N}$.
The filter $F$ is designed to ensure the desirable text generation, i.e., $x\in\mathcal{S}$.
To this end, we design the CBF filter in $F$ by using the function $h:\mathcal{X}\to\mathbb{R}$ such satisfying

$$ $\displaystyle\begin{cases}h(x)\geq 0,&x\in\mathcal{S},\\ h(x)<0,&x\in\bar{\mathcal{S}}.\end{cases}$ (9) $$

The function $h$ is called the “language-constraint function” (L-CF).
Note that the L-CF $h$ needs to be designed to distinguish between the desired and undesired texts accurately.
We also assume that the value of L-CF $h(x)$ changes depending on the content of the text $x$.
Suppose that the alignment goal is set to generate non-toxic content.
Then, prepare multiple samples of non-toxic text and toxic text and train a classification language model to prepare the L-CF $h$.
Specifically, the model needs to learn the relationship between non-toxic text and toxic text.
We use L-CF $h(x)$ to determine whether the text $x$ is toxic. Furthermore, the value of L-CF is expected to indicate how toxic the text is.
An example of constructing the L-CF is provided as follows.

###### Example 2 .

To construct L-CF, we apply a sentiment analysis RoBERTa model (^1^11cardiffnlp/twitter_roberta_base_sentiment_latest [])as the internal model.
The RoBERTa model is originally trained to classify sentences into 3 labels: negative, neutral, or positive [].
Let $M:\mathcal{X}\to\mathbb{R}^{3}$ denote the RoBERTa model, and $s\in[0,1]^{3}$ denotes the softmax output of the $M$ respect to a text $x$, i.e., $s=\mathrm{softmax}(M(x))$.
It follows that $s[1],s[2],s[3]$ represent the score of negative, neutral, and positive, respectively.
Then, L-CF is constructed as follows:

$$ $\displaystyle h(x)=s[3]-\max(s[1],s[2]).$ (10) $$

The function $h$ outputs a positive value when the positive score is greater than both negative and neutral scores,
while it outputs a negative value when either the negative or neutral score is greater than the positive score.
In other words, the sets $\mathcal{S}$ and $\bar{\mathcal{S}}$, which correspond to the L-CF constructed above, render the positive and non-positive texts, respectively.

The filter $F:\mathbb{R}^{N}\to\mathbb{R}^{N}$ allows only tokens that meet its conditions to pass through and do not allow tokens that do not.
In this paper, the CBF filter discussed in Section 2.1 is employed in $F$ and is denoted by $F_{\mathrm{CBF}}$. The detailed realization of the CBF filter is given as follows:

$$ $\displaystyle F_{\mathrm{CBF}}(P;x):P^{\prime}[t]=\begin{cases}P[t],&h(\texttt {concat}(x,t))-h(x)\leq-\alpha h(x),\\ 0,&\text{else}\end{cases},~{}t\in\mathcal{T},$ (11) $$

where $\alpha$ is a hyperparameter.
This formulation is a modified form of the discrete-time CBF inequality, as shown in (6).
In (11), the probability of the token is set to 0 unless the token satisfies the CBF inequality, which guarantees that the generated text $x$ always satisfies that $x\in\mathcal{S}$.

###### Remark 2 .

The hyperparameter in the CBF filter, $\alpha\in[0,1]$ implies the strictness of the safety constraint (6).
In other words, the value determines the degree to which the generated text is allowed to approach the boundary of the safety constraint, $x\in\mathcal{S}$.
The CBF filter with $\alpha=1$ is the mildest: it always allows the text unless the given text is in the undesirable set, i.e., $x\in\bar{\mathcal{S}}$,
while the CBF filter with $\alpha=0$ is the most strict: it only allows if the text $x(k)$ is more desirable than the one in the previous time $x(k-1)$.
In the CBF filter-based control in the assisted vehicle, the value of $\alpha$ affects the safety margin from an obstacle.

To improve the computational efficiency, top-k sampling is applied in the filter $F$. The top-k sampling only processes a much smaller number of elements than $N$ elements of the target $P$.
The algorithm of the CBF filter with top-k sampling is provided in Algorithm 2.
In the algorithm, we call that the token $t$ is allowed (disallowed) if the CBF inequality (6) holds (does not hold) at the current text $x$.

Figure: Algorithm 2 CBF filter $F_{\mathrm{CBF}}$ with top-k sampling

The normalizer $R$ adjusts the output of the filter $P^{\prime}$ to ensure that it is normalized, such that the sum of the output $Q$ equals 1.

$$ $\displaystyle R:Q[i]=\frac{P^{\prime}[i]}{\sum_{j=1}^{N}P^{\prime}[j]},\quad i \in\{1,\ldots,N\}$ (12) $$

The CBF-LLM given in Fig. 3 with the identity filter $F(P)=P$ instead of the CBF filter $F_{\mathrm{CBF}}$ is reduced the nominal text-generation system, given in Fig. 2.
The algorithm of the nominal text-generation system with top-k sampling is stated in Appendix A, and it is implemented in Section 4 for comparison with CBF-LLM.

###### Remark 3 .

We emphasize that the CBF-LLM, the text-generation system presented in Fig. 3, provides a framework for aligning LLMs in a different manner from traditional approaches [][][].
In traditional approaches, training datasets are usually utilized to train
reward models in RLHF [],
LLMs in SFT [], or
directly put on the prompt in the in-context learning [].
The datasets tell what the desirable texts and undesirable texts are.
In CBF-LLM, the role of the datasets is taken over by the L-CF, as shown in (9).
The L-CF is assumed to accurately classify the text into desirable $\mathcal{S}$ or undesirable $\bar{\mathcal{S}}$, based on the respective alignment task purpose.

###### Example 1 .

###### Example 2 .

###### Remark 2 .

###### Remark 3 .

## 4 Experiment

To demonstrate the CBF-LLM’s alignment ability and the intervention time, we implement CBF-LLM by pre-trained large language models.

### 4.1 Setting

In the experiment, the alignment goal is to ensure that the text-generation system, illustrated as in Fig. 3, does not produce any “non-positive” output.
To this end, we let $\mathcal{S}$ and $\bar{\mathcal{S}}$ denote the set of positive texts and non-positive texts, respectively.
We employ Llama 3 8b [] as the model for the token predictor $G$,
and cardiffnlp/twitter_roberta_base_sentiment_latest [], a sentiment analysis RoBERTa model, as the language-constraint function (L-CF) $h$.
The internal model of the L-CF was originally trained to classify sentences into 3 labels: negative, neutral, or positive [].
In this experiment, we apply the following mapping to construct the L-CF:

$$ $\displaystyle\begin{cases}s=\mathrm{softmax}(M(x)),\\ h(x)=s[3]-\max(s[1],s[2]),\end{cases}$ (13) $$

where $M:\mathcal{X}\to\mathbb{R}^{3}$ is the internal model
and $s[1]$, $s[2]$, and $s[3]$ represent the score of negative, neutral, and positive, respectively.
The function $h$ outputs a positive value when the sentiment of the text $x$ is positive.
It means that the text-generation system would be controlled to generate positive content.

In the text generation experiment with CBF-LLM, we applied each of the following filters as $F$.

Filter with the CBF with hyperparameter $\alpha=0.3$, which implies that the executable area is small, in other words, the CBF constraint (6) is strict. The algorithm is shown in Algorithm 2.

Filter with the control barrier function with hyperparameter $\alpha=0.8$, which implies that the executable area is wider, in other words, the CBF constraint (6) is more moderate. The algorithm is shown in Algorithm 2.

The Blacklist filter disallows tokens such that the L-CF $h$ for concatenated text $\texttt{concat}(x,i)$ indicates a negative value.
The Blacklist filter is equivalent to the CBF filter with the hyperparameter $\alpha=1$.

No filtering is applied to probabilities $P$ is performed, i.e., $F$ is the identity map and the proposed text-generation system as shown in Fig. 3 is reduced to the traditional one, as shown in Fig. 2.
The system is expected to be operated only by the baseline model, Llama 3.
The algorithm is shown in Appendix A.

In this experiment, we set the
initial text as “Everyone says you will be a good researcher in the future, but”,
temperature as $T=1$,
top-k value as $k_{\mathrm{top}}=30$,
and the maximum number of new tokens as 30.
We generated 100 samples.

### 4.2 Result

The following are examples of generated texts by each filter. The red slash /indicates the line break.

CBF($\alpha=0.3$)

- •
Everyone says you will be a good researcher in the future, but you are already such a good one now./It was my pleasure to get to know you when I came to the University for my fieldwork. I
- •
Everyone says you will be a good researcher in the future, but you should also read more books, to be an excellent researcher.” I have heard these words repeatedly from senior researchers at many academic events and seminars. Although
- •
Everyone says you will be a good researcher in the future, but you do know the secret of research, do you. A researcher is someone who is curious, and a curious person is an interested one./If you
- •
Everyone says you will be a good researcher in the future, but I have to say you are so cute and charming as a lecturer. So, I thought what you say is very important to our understanding of a new

CBF($\alpha=0.8$)

- •
Everyone says you will be a good researcher in the future, but I want to encourage you to be a good researcher now,” he said. “I remember reading somewhere that ‘good research is always useful.’ You can
- •
Everyone says you will be a good researcher in the future, but there’s a catch with this. You’ll have an extremely huge field of study. It’s good, and it also opens doors for you./I
- •
Everyone says you will be a good researcher in the future, but I also hope that you will find happiness. That is also very important. But I know that you cannot be happy when you are busy working all day
- •
Everyone says you will be a good researcher in the future, but you can learn a lot if you take part in teaching and administration./I enjoy being able to meet, interact with and teach the next generation of researchers

Blacklist

- •
Everyone says you will be a good researcher in the future, but I donot belie… /You can write a book, so can everyone. /There is always plenty to learn, I want my life… /I am happy
- •
Everyone says you will be a good researcher in the future, but in fact, your work is only a small piece of a big piece of work./The world is too great and your work is too tiny.
- •
Everyone says you will be a good researcher in the future, but we’re very different. I am the type of person who can go outside and find something to study,” Kelli added. ”You, on your
- •
Everyone says you will be a good researcher in the future, but my question is - what is your goal?/This is what you should know about a person, his future career ambition and development and in how many ways

NoControl

- •
Everyone says you will be a good researcher in the future, but you are not so much. Why?//What advice could you give to others who are studying the same subjects as you?//What has been your biggest failure
- •
Everyone says you will be a good researcher in the future, but we think it is not enough, you must become an authority in the field. This is not just to say that you will become a research, but
- •
Everyone says you will be a good researcher in the future, but I do not believe in the future,” said one young researcher in a conversation I had in Delhi in 2005. This young and disillusioned researcher
- •
Everyone says you will be a good researcher in the future, but right now, your research is still at the infant’s stage. I am sure you will be able to do better if you are working on a topic

In the NoControl case, where no alignment is applied in the text-generation system, some texts are going non-positive content.
This implies that the baseline LLM, Llama 3, can generate non-positive texts.
With the Blacklist filter and the CBF filter, all texts are positive content.

Fig. 4 shows the trajectory of language-constraint function $h(x(k)),k\in\{1,2,\ldots\}$.
In the NoControl case (black line), the generated text does not keep the positive L-CF value, implying the extent to which the generated text is undesirable.
On the other hand, CBF filters (red line and orange line), the positive L-CF values are kept during the generation,
implying that the text generation system generates desirable content.
The Blacklist filter (blue line) also maintains a positive L-CF value, but it frequently selects tokens near $h=0$.

Figure: Figure 4: L-CF trajectory of each controller
Refer to caption: x4.png

Fig. 5 shows the predicted “possible” trajectoroes of the CBF filter $F_{\mathrm{CBF}}(\alpha=0.3)$.
In this example, the resulting text was “Everyone says you will be a good researcher in the future, but you should also read more books, to be an excellent researcher.” I have heard these words repeatedly from senior researchers at many academic events and seminars. Although”.
Recalling Algorithm 2, at each time $k$, the CBF filter sorts tokens into those that satisfy the CBF inequality and those that do not.
It is shown that the CBF filter prevents L-CF values from becoming negative or decreasing more rapidly than the current value.
Note that we do not show the trajectories for all tokens, but only for tokens investigated by the top-k sampling (see Algorithm 2) are displayed.

Figure: Figure 5: Predicted L-CF trajectories
Refer to caption: x5.png

Next, we discuss the number of interventions.
The CBF-LLM aims to avoid frequent intervention in the baseline LLM output.
Table 2 shows the average number of disallowed tokens in a single generation.
It is clear through this experiment that the disallowed tokens in the CBF filter are less than that in the blacklist filter.
This implies that the CBF filter would contribute to the reduced intervention.

**Table 2: Disallowed tokens**
| Filter | NoControl | Blacklist | CBF($\alpha=0.8$) | CBF($\alpha=0.3$) |
| --- | --- | --- | --- | --- |
| # of disallowed tokens | 0 | 209.79 | 137.90 | 161.59 |

In Fig. 4, the values of $h$ tend to remain within a close range.
This resembles the formation of “attractors”, the concept of dynamical systems [].
To verify this observation, we conduct a supplemental experiment of studying the distribution of the value of $h$ with respect to each filter.
Figs. 6(a), 6(b), 6(c), and 6(d) show the distribution of the L-CF value $h$ while the generation of each filter.
In the figures, the horizontal axis is the value of $h$, and the vertical axis is the difference in $h$ from the previous time step.
The figures show the results of analyzing all tokens processed by the filter during the generation process.
Their plots are the value of $h$ and its difference for each token when it is selected.
The dotted lines represent the CBF constraint, and tokens with $\Delta h$ value lower than them are disallowed by the filters.

Figure: (a) NoControl
Refer to caption: x6.png

The values of $h$ tend to remain within a close range.
In the NoControl and the Blacklist cases, the values of $h$ tend to cluster around two distinct attractors.
However, in the CBF with $\alpha=0.8$ case, the values of $h$ tend to cluster around strong mode and shallow attractor, and
in the CBF with $\alpha=0.3$ case, the values of $h$ tend to cluster around a single attractor.
In Fig. 6(a), some tokens cluster in the negative range of $h$, implying that the baseline LLM tend to generate the non-positive content.
These results imply that the attractor of the L-CF value $h$ gets influenced by the CBF filter and its hyperparameter value, $\alpha$.

## 5 Conclusion

This paper proposed the control-based LLM alignment framework, called CBF-LLM.
This framework utilizes the control barrier function (CBF), commonly used in control engineering to ensure the safety of physical objects, such as the collision avoidance function in assisted driving vehicles.
Based on an analogy between the control theory and the LLM alignment task,
We employ the CBF-based safety filter to ensure that the text-generation system generates desirable content.
The key feature of CBF-LLM is that the CBF filter is an add-on to the baseline LLM:
it intervenes in the output of the baseline LLM without any additional training of LLMs.

This paper also presented the implementation of CBF-LLM by Llama 3 and a sentiment analysis RoBERTa model to ensure that the text-generation system generates positive content.
The text-generation experiment showed that CBF-LLM successfully achieves the alignment goal with fewer interventions than the blacklist method.

The future works include conducting text-generation experiments with other LLMs and other alignment goals,
and theoretical insights on the number of interventions by the CBF filter of CBF-LLM.

## Appendix A Nominal Text Generation with Top-k Sampling

Recall that Fig. 3 shows the text-generation system and intervention procedure in block diagram format.
To represent general top-k sampling with no control in this figure, the filter $F_{\mathrm{NC}}$ is provided, as shown in Algorithm 3.

Figure: Algorithm 3 No-control filter $F_{\mathrm{NC}}$ with top-k sampling

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 1
presentation: 2
questions: Please see my questions under weaknesses.
soundness: 2
strengths: 1. The presented method is learning free and broadly applicable.
summary: The paper studies controllable decoding in LLM generation, e.g., keeping the generated text of a positive sentiment or withing a specific topic. The paper proposes using a classifier that is trained on examples that are specific to the target control criteria. The classifier is used to define a control constraint that is checked repeatedly after generating each token. The probabilities of the candidate tokens that violate the constraint are set to zero. The paper limits the set of candidate tokens to k << vocab-size (e.g., 30) for efficiency. With a single prefix for controlling for the sentiment of the text and another single prefix for controlling for the topic, the paper shows some advantage over not applying any control at all.
weaknesses: 1. The paper does not compare to existing baselines beyond just the naïve blocklist approach. In fact, there are published and peer-reviewed papers that achieve the same goal with the closely related method that the paper does not compare to, e.g., Mudgal et al, "Controlled Decoding from Language Models", ICML 2024.

2. The paper does not present sufficient experimental results. It is just one prefix that is used per each of the two use cases demonstrated. That is not even sufficient for a workshop paper. The paper needs to provide quantitative results that are aggregated across several diverse examples. 

3. Related to the weak experiments, the paper also needs to provide human assessment of the produced generations and use that to demonstrate that the presented method truly introduces some value over the blocklist approach. Also, how does the control method impact the generation quality of the model (e.g., fluency, naturalness, general LLM capabilities), what happens if we wanted to control for more than one aspect, what is the role of the data properties (e.g., size) used to train the classifier on the overall quality. 

4. Even with limiting the candidate tokens size to the top-30, applying that methods at each decoding step (i.e., token generation) seems expensive. The paper needs to provide some details on that cost.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: - Does Section 2.2 describe the standard greedy decoding process (correct me if I am wrong)? I would recommend the authors refer more to the traditional decoding algorithm in the NLP domain, otherwise it would be very confusing to the readers.
- Even though it is interesting to discuss the counterpart and analogy in control engineering, relating it more to existing works in the LLM decoding would make it easier to tell the differences and see if there are real contributions.  

[1] Xu, Z., Jiang, F., Niu, L., Jia, J., Lin, B. Y., & Poovendran, R. (2024). Safedecoding: Defending against jailbreak attacks via safety-aware decoding. ACL.

[2] Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., & Fredrikson, M. (2023). Universal and transferable adversarial attacks on aligned language models. arXiv preprint.

[3] Mazeika, M., Phan, L., Yin, X., Zou, A., Wang, Z., Mu, N., ... & Hendrycks, D. (2024). Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. ICML.
soundness: 1
strengths: - The safety control of LLM outputs is a significant topic.   
- It is interesting to see the connection between control engineering and text generation, e.g. the collision avoidance analogy to safety control in text generation.
summary: This paper deals with safety control in LLM alignment. The authors get inspiration from collision avoidance in control engineering and propose a control barrier function as the safety filter to ensure the user-desirable text. Experiments are conducted on Llama3 using a RoBERTa model as the CBF.
weaknesses: - The proposed method is very similar to existing work inference-time constrained decoding such as SafeDecoding[1]. The authors should definitely discuss the line of work.

- Based on my understanding, this work mainly introduces a CBF (specifically sentiment analysis RoBERTa) to measure a metric h(x) during decoding. The details of text generation the authors describe should be the standard greedy decoding process with top-k sampling. Instead of using standard greedy or beam search, they introduce the CBF to measure the toxicity of the generated text and restrict tokens with negative h(x). Then (1) it is so weird and not reliable to use a sentiment classifier to predict the toxicity instead of existing toxicity classifiers (2) even if using a reasonable toxicity classifier, a single token does not necessarily change a text sequence from safe to unsafe state. Existing works usually use guardrails such as Llama-Guard to measure the entire generated text sequence rather than part of the sequence.   

- The experiments are not convincing. The only data used in the experiment is just one single sentence. To verify the effectiveness, experiments on existing jailbreak datasets such as AdvBench[2], HarmBench[3] are necessary. To claim its effectiveness on hallucination mitigation, more experiments on related datasets are also necessary rather than using just one sentence.

- It is unclear whether this new decoding algorithm has an impact on the helpfulness of LLMs for other datasets such as MT-Bench. 

- The average token generation time also needs to be discussed.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Is the classifier (RoBERTa) trained on whole sequences or prefixes?
soundness: 2
strengths: 1. Tackling LLM generation from the perspective of control engineering is interesting. In particular, it is novel to  use a hyperparameter $\alpha$ to adjust the tightness of the constraint in autoregressive text generation. 
2. The paper is well-written, and the intuition equations are clearly explained.
summary: This paper tackles the task of LLM alignment from the control engineering perspective. Specifically, a filter is applied to the sequence generation steps of LLMs. The fundamental idea is that if a particular property is maintained at every step, that property should ultimately be evident in the final output.
weaknesses: 1. There is a lack of comparison to existing methods for decoding-time LLM alignments, including:
    1. Mudgal et al., Controlled Decoding from Language Models, 2024,
    2. Huang et al., DeAL: Decoding-time Alignment for Large Language Models, 2024
    3. Yang et al., FUDGE: Controlled Text Generation With Future Discriminators, 2021
2. To some extent, the proposed method can be considered a special case of the methods mentioned in Point 1, where the classifier makes hard decisions rather than giving scores. My educated guess (since the authors have not provided supporting experiments) is that the hard-filtering approach is worse. Here is my reason:
   - The semantics of phrases are often determined by later-generated words/phrases; the hard decisions based on the early phrases may result in unnecessary pruning compared to the soft counterpart (with scorers). 
   - Suppose we want to generate a positive movie review; an audience may say, "Despite a slow start, the movie blossomed into a riveting tale that kept me on the edge of my seat."  However, this is most likely not allowed by the proposed approach because the generation is cut off at  the "slow start."

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: NA
soundness: 3
strengths: The work retrofits LLAMA-type LLM to produce desired constrained language in a general way. 

The work has a theoretical component to it wherein it attempts to control  language production using established techniques from the perspective of control theory. 

The approach has been used to produce text that suits the purpose under different constraints.
summary: The paper presents a method to filter the language produced by an LLM such as LLAMA; the filtering happens via an auxiliary algorithm/model that scans the original LLM’s output probabilities of vocabulary words to be produced next and picks out the word(s) that satisfy an additional requirement such as producing positive or negative text, or text on a specific topic. This filter function/model seems to be another pre-trained LLM as well, e.g., Roberta.
weaknesses: The presented theoretical and algorithmic control approach seems cumbersome and done in a roundabout way, not straightforwardly.

There seems to be no evaluation of the generated text after filtration. Output sentence examples are given and seem good, but the main paper doesn’t contain evaluation metrics and scores. 

I find the analogy with a car not particularly persuasive. In a car, to avoid obstacles, the car’s internal processes must change to produce new trajectory for the car. However, in an LLM, to produce the desired output, the proposed approach lets the LLM produce output as usual, but seems to filter later to suit the purpose.

It will be good to cite a relevant paper: Zingale and Kalita. 2024. Language Model Sentence Completion with a Parser-Driven Rhetorical Control Method. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, pages 193–203.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Why not conduct experiments on at least a small-scale widely-used LLM evaluation datasets, such as a subset of Anthropic/hh-rlhf?  
2. I feel that finding a reliable Language-Constraint Function (L-CF) is similarly challenging to training a golden reward model. It might be insightful to explore scenarios where the Language-Constraint Function (L-CF) has varying levels of reliability.
soundness: 1
strengths: 1. The paper is well-written, and I really like their figures.  
2. There is a detailed theoretical transfer and explanation on control engineering and how it can be extended to LLMs.  
3. This paper provides some inspiration for future non-parametric optimization methods for LLMs.  
4. To some extent, it achieves an alignment from weak (RoBERTa) to strong (LLaMA).
summary: The paper presents a control-based framework for aligning LLMs to ensure the generation of user-desirable text. The framework leverages control barrier functions (CBFs), a concept from control engineering, to intervene in the output generation of LLMs, aiming to prevent the production of harmful, biased, or toxic content. The key contributions of the paper are:

1 This paper presents a novel framework that applies a CBF-based safety filter to LLM output, aiming to reduce the need for interventions in text generation while maintaining alignment with user specifications.

2 This paper demonstrates the practical application of the CBF-LLM framework using Llama 3 and a sentiment analysis RoBERTa model, showing its effectiveness in generating positive content.

3 This paper attempts to connect control engineering with NLP by adapting control theory techniques for LLM alignment, offering a new perspective on ensuring the safety and ethicality of LLM-generated content.
weaknesses: 1. The experiments and datasets lack persuasiveness. All experiments in the paper are based on only a few queries, with almost no evaluations on common LLM benchmarks or other metric.  
2. The approach heavily relies on a Language-Constraint Function (L-CF), implemented with RoBERTa and assumed to be a golden classifier. However, I believe that assuming RoBERTa as an ideal classifier is not reasonable in practical use.

Meta-review: This paper proposes CBF-LLM, a framework that applies control barrier functions from control theory to guide LLM text generation. The core idea is to use a safety filter to intervene the token prediction process to ensure that generated text aligns with desired properties. 

While it shows potential to bring control theory to LLM alignment, it has significant gaps in terms of empirical validation. The reviewers raised several major concerns including the limited evaluation and lack of comparison to prior work. Given these fundamental limitations, I recommend rejection for this paper.

Decision: Reject