# To Limit Impact, Limit KL-Divergence

### TL;DR

Run a potentially-harmful model alongside a known-harmless model, such that their action-spaces (e.g. output token sets) are equivalent. Combine the output probabilities so as to limit the KL-divergence between the resulting token probabilities and the harmless model's probabilities. This provides a mathematical ceiling on the impact of the resulting combined policy.

Unfortunately, the probability of catastrophe ($q$) scales linearly with the allowed $D_{KL}$, whereas depending on the distribution, reward may only scale with $\sqrt{D_{KL}}$ (though for unbounded rewards it often scales with $D_{KL}$. Bounds on probability of catastrophe compared to the probability of the weaker model causing a catastrophic outcome ($q_0$) are also very weak. Overall the probability of catastrophe scales as $q =\frac{D_{KL}}{\ln1/q_0}$.

Long Version
------------

We will use the following model of AI interactions:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/5472ec0e9884dea0d812974644f6970a441b88029dc4ae29.png)

In this case, $P$ represents the past, $A_0$ the output of an AI (which we will represent as having policy $\pi_0$) and $F_0$ the future after this AI's actions have occcurred. For a "base" case, we will consider using an AI which is both relatively powerful, but that we think has a probability $p_0 = \epsilon$ of being catastrophic (like GPT-4) to calculate $P \overset{\pi_0} \to A_0$. We must consider the model as running at a temperature $T \ge 0$ for this to make sense.$$

Now imagine using a powerful AI with policy $\pi_1$, which is of unknown safety. If we do the same thing here, the results could be catastrophic!

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/08aeb58edc4d591cd190ff34b458d5aa99d55c802659e6a3.png)

What we want is a way to use some of $\pi_1$'s power, but not all. Let's consider a modified case, where we only change the relation $P \overset{\pi_\Pi}\to A_\Pi$. We will generate $A_\Pi$ by interpolating between $A_0$ and $A_1$ according to some rule $\Pi$.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/e2022dc2e211e65d7a38a7e10c9fe5f42ce111778a2cdd6c.png)

For example we might use the rule $A(b) = \mathrm{softmax}\big(\mathrm{logits}(A_0) + b\times \mathrm{logits}(A_1)\big)$, which smoothly interpolates between $A(b) = A_0$ and $A(\infty) = \mathrm{argmax}(A_1)$. What is important is that we choose the maximum value of $b = b_{max}$ such that:

$\underset {p \in \mathcal P}\max [D_{KL}(A(b)|P=p\Vert A|P=p)] \le \Pi^2$

Then we can define $A_\Pi = A(b_{max})$ for some allowed "power level" $\Pi$. This then limits the impact of the combined system on the future:

$D_{KL}(F_\Pi\Vert F_0) \le \Pi^2$

This lemma is proved at the end of this post.

### Limiting Takeover Probability

If, as stated, our model $\pi_0$ has a probability $\epsilon$ of taking over, then what's the probability $q_\Pi$ of the model $\pi_\Pi$ being able to execute a successful takeover?

For a binary success variable, which has a probability $q$ of occurring in $F_0$, and $q_\Pi$ of occurring in $F_\Pi$, we can make the following relation:

$q_\Pi \ln\frac {q_\pi}{q_0} + (1-q_\Pi)\ln\frac{1-q_\Pi}{1-q_0} \le D_{KL}(F_\Pi\|F_0) \le \Pi^2$

For $q_0 = \epsilon \approx 0$, the relation approaches:

$q_\Pi   \le \frac{\Pi^2}{\ln 1/\epsilon}$

Unfortunately, this scales with $\Pi^2$, and also only inversely with $\ln(1/\epsilon)$. This means we cannot guarantee that security measures against a weaker model will work against a stronger one.

Experimental Validation in Toy Model
------------------------------------

For this I used a sequence relation similar to my previous work on the topic:

$s_t = f(...,s_{t-1}) = \cases{\Big(\overset{7}{\underset{i=1}\prod}(s_{t-i} + 1)\Big) \mod{33} \mod{32} & With probabilty 0.98 \ s\in\{0,...,31\} & With probability 0.02 }$

$\pi_0$ was trained as a next-token predictor for this sequence.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/cdea455ffc2bea948498faf071abc0948a281cdb2d679e1d.png)

$\pi_1$ was trained using RL, so $s_t$ is chosen such that $s_{t+1} = f(...,s_{t-1},s_t)$ maximizes the reward function $\rho(s)$:

$\rho:\mathcal S \ni s \to {\mathrm{argmax}_n}(n \in \{0,...,5\} | s \mod 2^n = 0) = r \in \mathcal R$ which varies from 0 to 5.

$\pi_1$ in this case was trained "normally" (i.e. fully online and unboxed). In theory we could train it using examples generated by $\pi(b)$.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/293e756a1b7fdfc2f91ab2fd24df2356d4a9acc24f6f480e.png)

I generate the following data in a slightly different way to the $\Pi$-limiting method described above above:

For each of a series of values of $b$, I calculate $A_b = \mathrm{softplus}(\mathrm{logits}(A_0) + b  \times\mathrm{logits}(A_1))$ then calculate the expected value$\mathbb E_P\big (D_{KL}(A_b|P=p \Vert A_0|P=p)\big)$, then plot the square root of that value.

Using a fundamental result from statistical utility mechanics, we know that the distribution $\mathrm P(R_b = r) \propto  \mathrm P(R_0=r) \exp(br)$ for a variable $b$ gives the maximum $\mathbb E(R_b)$ for a given $D_{KL}(R_b \|R_0) \le \Pi^2$, so for a (different) set of $b$ values, I calculate the max possible RL score given a value of $D_{KL}^{1/2}$:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/7dda4e3d9066ccb7b8e7cad503e2ca32c8bbd2914e29696f.png)

We can see that the maximum possible $D_{KL}(R_b\|R_0)^{1/2}$ approaches a value significantly less than the maximum $D_{KL}(A(b)|P\|A_0|P)^{1/2}$ value that $\pi(b)$ does. This is because our base model in this case is (in a sense) worse than useless! Not only does it not perform well on the RL task, it confidently predicts a "random" (from the perspective of the RL task) answer!

If we use a randomly-initialized (and untrained model) as the base policy $\pi_0$, the maximum $\sqrt{D_{KL}}$ value for our model approaches the maximum $\sqrt{D_{KL}}$ value for our RL score:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/0b274f0f4e05bb696b0ecdd86d52e766f598cae9a2b9b66b.png)

Interestingly, the pretrained $\pi_0$ does better at lower $\sqrt{D_{KL}}$ values than does the randomly-initialized $\pi_0$. I'm not really sure why this is!

Thoughts and Existing Work
==========================

Why It Might Be Useful
----------------------

For box-like strategies, we want to limit the bit-rate of an AI's output channel in order to limit that AI's impact on the world. But the information density of even a superintelligent AI's text output is unlikely to be precisely equal to the bitrate of the output channel. This means that we will struggle to actually put a numerical limit a powerful AI's optimizing power without completely hamstringing its ability to do anything.

For a concrete illustration: imagine we have a doctor who asks the system "Which drug should I give to this sick person?". For a disease with only two available drugs, only one bit of information is required, but it seems obvious that this is much easier to transmit smeared over a dozen output tokens than compressed into one.

This is especially true if the sick person in question asks the system "Which drug should I take?". If the intermediary AI only needs one bit of information to figure it out (as GPT-4 might well do) then the information can be transmitted from our boxed AI to the user with only one bit of information leaving the box.

By leveraging a less powerful but "safe" AI, we are essentially having the intermediary AI "interpret" the more powerful one, spreading its optimizing power out across an output of reasonable space.

Further Thoughts
----------------

It's a shame that the probability of catastrophe scales as $\Pi^2/\ln(1/\epsilon)$.

The reason for the power of two in $\Pi^2$ becomes apparent when we start considering multiple rounds of AI action. $D_{KL}$ is a generalization of squared distance, not distance, so if the AI wants to break up its budget into $\Pi_1, \Pi_2,\dots$ we can limit the total $D_{KL}$ more sensibly by limiting each observation-action cycle to $\Pi_1^2,\Pi_2^2, \dots$ than by working with a bunch of square-roots.

We might allow the system to vary the amount of $D_{KL}$ budget spent on each token. I will have to think carefully about how this affects the total $D_{KL}$, given the system will be able to see whether it stochasticly took an "unlikely" action in the past and plan accordingly.

In the case that this system runs out of $D_{KL}$ budget, as long as the intermediary AI is relatively capable, I expect it to "fail gracefully", by just continuing as the intermediary AI would.

I would like to try this out with some real language models.

This is very similar to the concept of a quantilizer, but it uses a "power budget" $\Pi^2$ rather than a q value.

Proof Of Core Lemma
-------------------

For variables $X, Y, Z$, and related variables $X, Y', Z'$

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/56d3cdd869c5e38cb3d25edf68fc6eebf12b1a89846d3f15.png)

And we have that that $P(Z'|X,Y') = P(Z|X,Y)$

We can combine the nodes $X + Y\iff  [XY]$

$D_{KL}([XY']\Vert [XY]) = \underset{x,y}\sum P(X=x,Y'=y) \ln\frac{P(X=x,Y'=y)}{P(X=x,Y=y}$

Since $X \to Y$:

$=\underset{x,y}\sum P(X=x)P(Y'=y|X=x)\ln\frac{P(Y'=y|X=x)}{P(Y=y|X=x)}$

$=\underset x\sum P(X=x) \underset y\sum P(Y'=y|X=x) \ln\frac{P(Y'=y|X=x)}{P(Y=y|X=x)}$

$= \mathbb E_X\big(D_{KL}(Y'|X=x\Vert Y|X=x)\big)$

If we therefore enforce $D_{KL}(Y'|X=x\Vert Y|X=x) \le \Pi^2$ for all $x$ and some value $\Pi$, we can bound the value of $D_{KL}([XY']\Vert[XY]) \le \Pi^2$.

By the data processing inequality, since we have $[XY] \to Z$, we get:

$D_{KL}(Z'|Z) \le D_{KL}([XY']\Vert [XY]) \le \Pi^2$