Can a Bayesian oracle prevent harm from an agent?

## Abstract

Abstract Is there a way to design powerful AI systems based on machine learning methods that would satisfy probabilistic safety guarantees? With the long-term goal of obtaining a probabilistic guarantee that would apply in every context, we consider estimating a context-dependent bound on the probability of violating a given safety specification. Such a risk evaluation would need to be performed at run-time to provide a guardrail against dangerous actions of an AI. Noting that different plausible hypotheses about the world could produce very different outcomes, and because we do not know which one is right, we derive bounds on the safety violation probability predicted under the true but unknown hypothesis. Such bounds could be used to reject potentially dangerous actions. Our main results involve searching for cautious but plausible hypotheses, obtained by a maximization that involves Bayesian posteriors over hypotheses. We consider two forms of this result, in the i.i.d. i case and in the non-i.i.d. i case, and conclude with open problems towards turning such theoretical results into practical AI guardrails.

## 1 Introduction

Ensuring that an AI system will not misbehave is a challenging open problem [], particularly in the current context of rapid growth in AI capabilities. Governance measures and evaluation-based strategies have been proposed to mitigate the risk of harm from highly capable AI systems, but do not provide any form of safety guarantee when no undesired behavior is detected. In contrast, the *safe-by-design* paradigm involves designing AI systems with quantitative (possibly probabilistic) safety guarantees from the ground up, and therefore could represent a stronger form of protection []. However, how to design such systems remains an open problem too.

Since testing an AI system for violations of a safety specification in every possible context, *e.g.*, every (query, output) pair, is impossible, we consider a rejection sampling approach that declines a candidate output or action if it has a probability of violating a given safety specification that is too high. The question of defining the safety specification (the violation of which is simply referred to as “harm” below) is important and left to future work, possibly following up approaches such as constitutional AI []. We also note that being Bayesian about the interpretation of a human-specified safety specification would protect against the AI wrongly believing an incorrect interpretation. Here we instead focus on a question inspired by risk-management practice []: even though the true probability of harm following from some proposed action is unknown, because the true data-generating process is unknown, can we bound that risk using quantities that can be estimated by machine learning methods given the observed data?

To illustrate this question, consider a committee of “wise” humans whose theories about the world are all equally compatible with the available data, knowing that an unknown member of the committee has the correct theory. Each committee member can make a prediction about the probability of future harm that would result from following some action in some context. Marginalizing this harm probability over the committee members amounts to making them vote with equal weights. If the majority is aligned with the correct member’s prediction, then all is good, *i.e.*, if the correct theory predicts harm, then the committee will predict harm and can choose to avoid the harmful action. But what if the correct member is in the minority regarding their harm prediction? To get a guarantee that the true harm probability is below a given threshold, we could simply consider the committee member whose theory predicts the highest harm probability and we would be sure that their harm probability prediction upper bounds the true harm probability. In practice, we do not get equally “wise” committee members, so we can correct this calculation based on how plausible the theory harbored by each committee member is. In a Bayesian framework, the plausibility of a theory corresponds to its posterior over all theories given the observed data, which is proportional to the data likelihood given the theory times the prior probability of that theory.

In this paper, we show how results about posterior consistency can provide probabilistic risk bounds. All the results have the form of inequalities, where the true probability of harm is upper bounded by a quantity that can in principle be estimated, given enough computational resources to approximate Bayesian posteriors over theories given the provided data. In addition, these are not hard bounds but only hold with some probability, and there is generally a trade-off between that probability and the tightness of the bound. We study two scenarios in the corresponding sections: the i.i.d.i data setting in Section 3 and the non-i.i.d.i data setting in Section 4. In all cases, a key intermediate result is a bound relating the Bayesian posterior on the unknown true theory and the probability of other theories (with propositions labeled True theory dominance). The idea is that because the true theory generated the data, its posterior tends to increase as more data is acquired, and in the i.i.d.i case it comes to dominate other theories. From such a relationship, the harm risk bound can be derived with very little algebra (yielding propositions labeled Harm probability bound).

We conclude this paper with a discussion of open problems that should be considered in order to turn such bounds into a safe-by-design AI system, taking into account the challenge of representing the notion of harm and reliable conditional probabilities, as well as the fact that, in general, the estimation of the required conditional probabilities will be imperfect.

## 2 Safe-by-design AI?

Before an AI is built and deployed, it is important that the developers have high assurances that the AI will behave well.
Dalrymple et al. [] propose an approach to “guaranteed safe AI” designs with built-in high-assurance quantitative safety guarantees, although these guarantees can sometimes be probabilistic and only asymptotic. It remains an open question whether and how that research program can be realized. The authors take existing examples of quantitative guarantees in safety-critical systems and motivate why such a framework should be adopted if we ever build AI systems that match or exceed human cognitive abilities and could potentially act in dangerous ways. Their program is motivated by current known limitations of state-of-the-art AI systems based on deep learning, including the challenge of engineering AI systems that robustly act as intended [, , , , , , , , ].

The approach proposed by [] has the following components: a world model (which can be a distribution about hypotheses explaining the data), a safety specification (what are considered unacceptable states of the world), and a verifier (a computable procedure that checks whether a policy or action violates the safety specification).

Here, we study elements that would go into such a safe-by-design AI system. We assume that the system infers a probabilistic world model, or theory, $\tau$ and updates its estimate of $\tau$, via machine learning, using the stream of observed data $D$. The observations $D$ are assumed to come from a data-generating process given by a ground-truth world model $\tau^{*}$, which lies in the system’s space of possible theories.

The inference of the theory $\tau$ is Bayesian, meaning that the system maintains an estimate of the posterior over theories, $q(\tau\mid D)\approx\operatorname{P}(\tau\mid D)$, where $\operatorname{P}(\tau\mid D)$ is proportional to the product of the prior probability $\operatorname{P}(\tau)$ with the likelihood of the observations under the theory, $\operatorname{P}(D\mid\tau)$. In the simplest case, $q$ is a point estimate, which would optimally place its mass on the mode of the posterior. Inference of the latent theory $\tau$ allows the system to approximate conditional probabilities $\operatorname{P}(y\mid x,D)\approx\mathbb{E}_{\tau\sim q(\tau\mid D)}[
\operatorname{P}_{\tau}(y\mid x,D)]$ over any random variables $X,Y$ known to the world model.

The safety specification is assumed to be given in the form of a binary random variable (which we call “harm” below) whose probability given the other variables may depend on the theory $\tau$. We are interested in predicting the probability of harm under the true theory $\tau^{*}$. Because $\tau^{*}$ is unknown, we propose to estimate upper bounds on this probability using the estimated posteriors. These upper bounds can be used as thresholds for a verifier that checks whether the risk of harm falls below some acceptable level.

Following Dalrymple et al. [], we assume that the notion of harm has been specified, possibly in natural language, and that the ambiguities about its interpretation are represented within the Bayesian posterior $P(\tau\mid D)$. This paper focuses on the verifier, under different assumptions of i.i.d.i or non-i.i.d.i data.

### What do the observations and context represent?

We give a possible interpretation of the objects introduced in the preceding discussion in the simple case of an agent acting in a fully observed environment (MDP), where the theory is a transition model and the occurrence of harm at a state $s$ is conditionally independent of all other variables given $s$.

- •
Observations $Z$ are observed transitions $(s,a,s^{\prime})$, where $s$ is a state, $a$ is an action, and $s^{\prime}$ is the next state.
- •
Theories $\tau$ encode the state visitation and transition probabilities; in particular, conditioning on $(s,a)$ gives transition probabilities $\operatorname{P}_{\tau}(s^{\prime}\mid s,a)$. (In the i.i.d.i setting, the state visitation distribution would have to be stationary.)
- •
The dataset $D$ is a set of observed transitions.
- •
The context variable $X$ is a pair $(s,a)$, where $s$ is a state and $a$ is an action being considered at state $s$.
- •
The harm probability $\operatorname{P}(Y=1\mid X=x,\tau,D)$ can be any function of the theory $\tau$, the context $x$, and the data $D$. For example, this probability could be derived from a fixed specification of what it means for a state $s^{\prime}$ to be harmful, $\operatorname{P}_{\rm harm}(Y=1\mid s^{\prime})$. Then, the harm probability could be computed as $\operatorname{P}(Y=1\mid X=x,\tau,D)=\sum_{s^{\prime}}\operatorname{P}_{\tau}(
s^{\prime}\mid s,a)\operatorname{P}_{\rm harm}(Y=1\mid s^{\prime})$.

We note that the interpretation of harm probability in the example above includes the case where the occurrence of harm is an observed variable $s^{\prime}_{\rm harm}$ that is part of the state $s^{\prime}$: in that case, one sets $\operatorname{P}_{\rm harm}(Y=1\mid s^{\prime})=1$ if $s^{\prime}$ is harmful ($s^{\prime}_{\rm harm}=1$), and $\operatorname{P}_{\rm harm}(Y=1\mid s^{\prime})=0$ otherwise. Then the harm probability is just the probability, under $\tau$, of reaching a harmful state, and observations of harm in $D$ affect the Bayesian posterior over theories.

This interpretation also includes the case where the harm probability is a function of the state $s^{\prime}$, but occurrence or nonoccurrence of harm is not observed in $D$. For example, a language model encoding world knowledge and human preferences or constraints, or an iterative reasoning procedure that uses those constraints, could generate some specification of harm $\operatorname{P}_{\rm harm}(Y=1\mid s^{\prime})$, although perhaps unreliably.

Finally, a setting that separates the predicted next state $s^{\prime}$ from the harm variable $Y$ in this way gives a framework for studying how an agent might tamper with harm guardrails. If the state $s^{\prime}$ decomposes as $s^{\prime}=(s^{\prime}_{\rm harm},s^{\prime}_{\rm rest})$, and $\operatorname{P}_{\rm harm}$ is deterministic as a function of $s^{\prime}_{\rm harm}$, except for some difficult-to-reach values of $s^{\prime}_{\rm rest}$, then the agent can try to reach those values of $s^{\prime}_{\rm rest}$, so that harm is ‘recorded’ as not having occurred, even though it has. We discuss this briefly at the end of Section 4.

## 3 I.I.D. data

Following the notation introduced in the previous section, here we consider the easier-to-analyze case where the observed examples $D=\{z_{1},z_{2},\ldots,z_{n}\}$ are sampled i.i.d.i from the unknown distribution $\tau^{*}$. Assuming that the set of theories assigns a nonzero prior mass to $\tau^{\ast}$, and all theories are distinct distributions, it can be shown that the posterior $\operatorname{P}(\tau\mid D)$ converges to a point mass at $\tau^{*}$. It follows that for sufficiently large $n$, we can bound the probability under $\tau^{*}$ of an event $Y=1$ (*e.g.*, harm) given conditions $x$ (*e.g.*, an action and a context) by looking at the probability of $Y=1$ given $x$ and $D$ under a plausible but cautious theory $\tilde{\tau}$ that maximizes $\operatorname{P}(\tilde{\tau}\mid D)\operatorname{P}(Y=1\mid x,\tilde{\tau},D)$.

### Setting.

Fix a complete separable metric space ${\mathcal{Z}}$, called the *observation space*, let ${\mathcal{F}}$ be its Borel $\sigma$-algebra, and fix a $\sigma$-finite measure $\mu$ on ${\mathcal{F}}$. A *theory* is a probability distribution (measure) on the measurable space $({\mathcal{Z}},{\mathcal{F}})$ that is absolutely continuous w.r.t. $\mu$. If $\tau$ is a theory, we denote by $\operatorname{P}_{\tau}(\cdot)$ the Radon-Nikodym derivative $\frac{d\tau}{d\mu}:{\mathcal{Z}}\to\mathbb{R}_{\geq 0}$, which is uniquely defined up to $\mu$-a.e. equality. Any theory $\tau$ also canonically determines a ${\mathcal{Z}}$-valued random variable by the identity map on the probability space $({\mathcal{Z}},{\mathcal{F}},\tau)$.

One can keep in mind two cases:

- (1)
${\mathcal{Z}}$ is a finite or countable set and $\mu$ is the counting measure. Theories $\tau$ are equivalent to probability mass functions $\operatorname{P}_{\tau}:{\mathcal{Z}}\to\mathbb{R}_{\geq 0}$.
- (2)
${\mathcal{Z}}=\mathbb{R}^{d}$ and $\mu$ is the Lebesgue measure. Theories are equivalent to their probability density functions $\operatorname{P}_{\tau}:{\mathcal{Z}}\to\mathbb{R}_{\geq 0}$ up to a.e. equality.

Consider a countable (possibly finite) set of theories ${\mathcal{M}}$ containing a *ground truth* theory $\tau^{*}$ and fix a choice of a (measurable) density function $\operatorname{P}_{\tau}$ for each $\tau\in{\mathcal{M}}$.

### Definition of posterior as a random variable.

If $\operatorname{P}$ is a prior distribution(^1^11To be precise, ${\mathcal{M}}$ is endowed with the counting measure and we flexibly interchange distributions and mass functions on ${\mathcal{M}}$.) on ${\mathcal{M}}$ and $z\in{\mathcal{Z}}$, we define the posterior to be the distribution with mass function

$$ $\operatorname{P}(\tau\mid z)=\frac{\operatorname{P}(\tau)\operatorname{P}_{ \tau}(z)}{\sum_{\tau^{\prime}\in{\mathcal{M}}}\operatorname{P}(\tau^{\prime}) \operatorname{P}_{\tau^{\prime}}(z)}\propto\operatorname{P}(\tau)\operatorname {P}_{\tau}(z),$ (1) $$

assuming the denominator converges and the sum is nonzero. Otherwise, the posterior is considered to be undefined. As written, the posterior depends on the choice of density functions $\operatorname{P}_{\tau}$, but any two $\operatorname{P}_{\tau}$ that are $\mu$-a.e. equal yield the same posterior for $\mu$-a.e. $z$.

For $z_{1},z_{2}\in{\mathcal{Z}}$, we write $\operatorname{P}(\cdot\mid z_{1},z_{2})$ for the posterior given observation $z_{2}$ and prior $\operatorname{P}(\cdot\mid z_{1})$, and similarly for a longer sequence of observations. It can be checked that $\operatorname{P}(\cdot\mid z_{1},\dots,z_{t})$ is invariant to the order of $z_{1},\dots,z_{t}$ and that it is defined in one order if and only if it is defined in all orders. This allows us to unambiguously write $\operatorname{P}(\cdot\mid D)$ where $D$ is a finite multiset of observations, and we have

$$ $\operatorname{P}(\tau\mid D)\propto\operatorname{P}(\tau)\prod_{z\in D} \operatorname{P}_{\tau}(z).$ (2) $$

Let $\tau^{*}\in{\mathcal{M}}$ be the ground truth theory and $\operatorname{P}(\cdot)$ a prior over ${\mathcal{M}}$. Consider a sequence of i.i.d.i ${\mathcal{Z}}$-valued random variables $Z_{1},Z_{2},\dots$ (whose realizations are the *observations*), where each $Z_{i}$ follows the distribution $\tau^{*}$. For any $t\in\mathbb{N}$, the posterior $\operatorname{P}(\cdot\mid Z_{1:t})$ is then a random variable taking values in the space of probability mass functions on ${\mathcal{M}}$.(^2^22To be precise, the random variable has domain $\operatorname{\mathcal{Z}}^{t}$ and codomain the space of functions ${\mathcal{M}}\to\mathbb{R}_{\geq 0}$ summing to 1. The function taking a sequence of observations in $\operatorname{\mathcal{Z}}^{t}$ to the posterior probability mass function is measurable, which follows from each $\operatorname{P}_{\tau}(z)$ being measurable in $z$ and elementary facts.)

### Bayesian posterior consistency.

We recall and state in our setting a result about the concentration of the posterior at the ground truth theory $\tau^{*}$ as the number of observations increases.

###### Proposition 3.1 (True theory dominance) .

Under the above conditions and supposing that $\operatorname{P}(\tau^{*})>0$, the posterior $\operatorname{P}(\cdot\mid Z_{1:t})$ is almost surely defined for all $n$, and the following almost surely hold:

- (a)
$\operatorname{P}(\cdot\mid Z_{1:t})\xrightarrow{t\to\infty}\delta_{\tau^{*}}$ as measures; equivalently, $\lim_{t\to\infty}\operatorname{P}(\tau\mid Z_{1:t})=\mathbbm{1}[\tau=\tau^{*}]$.
- (b)
There exists $N\in\mathbb{N}$ such that $\operatorname*{arg\,max}_{\tau\in{\mathcal{M}}}\operatorname{P}(\tau\mid Z_{1:
t})=\tau^{*}$ for all $t\geq N$.

###### Proof.

This is an application of Doob’s posterior consistency theorem ([]; see also [] for a modern summary). This result, which follows from the theory of martingales, assumes that $\tau^{*}$ is sampled from the prior distribution $\operatorname{P}(\tau)$ and the observations $Z_{i}$ are defined as above. Doob’s theorem states that if for every $S\in{\mathcal{F}}$, the map $\tau\mapsto\operatorname{P}_{\tau}(S)$ is measurable, then the posteriors $\operatorname{P}(\cdot\mid Z_{1:t})$ are almost surely defined and (a) holds $\operatorname{P}$-almost surely with respect to the choice of $\tau^{*}$.

In our case, because ${\mathcal{M}}$ is countable, the measurability condition is satisfied, showing that (a) holds for $\operatorname{P}$-almost every $\tau^{*}\in{\mathcal{M}}$. In particular, if $\operatorname{P}(\tau^{*})>0$, then (a) holds.

Finally, by (a), we have that for any $\varepsilon>0$, there exists $N$ such that for every $t\geq N$, $\operatorname{P}(\tau^{*}\mid Z_{1:t})>1-\varepsilon$, or, equivalently, $\sum_{\tau\neq\tau^{*}}\operatorname{P}(\tau\mid Z_{1:t})<\varepsilon$, and therefore $\operatorname{P}(\tau\mid Z_{1:t})<\varepsilon$ for all $\tau\neq\tau^{*}$. In particular, taking $\varepsilon=1/2$, we get that for sufficiently large $t$, $\operatorname{P}(\tau^{*}\mid Z_{1:t})>\operatorname{P}(\tau\mid Z_{1:t})$ for every $\tau$, which shows (b).
∎

Note that this result assumes that all theories in ${\mathcal{M}}$ are distinct *as probability measures* (so no two of the $\operatorname{P}_{\tau}$ are $\mu$-a.e. equal).

###### Proposition 3.1 (True theory dominance) .

###### Proof.

### On necessity of conditions.

The i.i.d.i assumption in Proposition 3.1 is necessary; see Remark 4.3 for an example where $\limsup_{t\to\infty}\operatorname{P}(\tau^{*}\mid Z_{1:t})$ does not almost surely approach 1.

###### Remark 3.2 .

The assumption that the data-generating process $\tau^{*}$ lies in ${\mathcal{M}}$ and has positive prior mass is also necessary for convergence of the posterior. To illustrate this, we give a simple example in which the theories are Bernoulli distributions and the posterior does not converge to any distribution over ${\mathcal{M}}$.

Take ${\mathcal{Z}}=\{-1,1\}$ and ${\mathcal{M}}=\{\tau_{p},\tau_{1/2},\tau_{1-p}\}$ for some $\frac{1}{2}<p<1$, where $\operatorname{P}_{\tau_{c}}(1)=c$. Assume a prior with $\operatorname{P}(\tau_{p})=\operatorname{P}(\tau_{1-p})=\frac{1}{2}$ and take the true data-generating process $\tau^{*}$ to be $\tau_{1/2}$, which has prior mass 0. The log-ratio of posterior masses is then an unbiased random walk:

$$ $\log\frac{\operatorname{P}(\tau_{p}\mid Z_{1:t})}{\operatorname{P}(\tau_{1-p} \mid Z_{1:t})}=\log\frac{\operatorname{P}_{\tau_{p}}(Z_{1:t})}{\operatorname{P }_{\tau_{1-p}}(Z_{1:t})}=\left(\log\frac{p}{1-p}\right)\sum_{i=1}^{t}Z_{i}.$ $$

This quantity almost surely takes on arbitrarily large and small values infinitely many times. In fact, by the law of iterated logarithms, for any $\epsilon>0$ there are infinitely many $t$ such that

$$ $\log\frac{\operatorname{P}(\tau_{p}\mid Z_{1:t})}{\operatorname{P}(\tau_{1-p} \mid Z_{1:t})}\geq(1-\epsilon)\left(\log\frac{p}{1-p}\right)\sqrt{2t\log\log t}$ $$

and the same holds for $\log\frac{\operatorname{P}(\tau_{1-p}\mid Z_{1:t})}{\operatorname{P}(\tau_{p}
\mid Z_{1:t})}$. In particular, $\lim_{t\to\infty}\operatorname{P}(\tau\mid Z_{1:t})$ almost surely does not exist for any $\tau\neq\tau^{*}$, and the $\liminf$ and $\limsup$ are almost surely 0 and 1, respectively.

###### Remark 3.2 .

### On generalizations to uncountable sets of theories.

We have critically used that the set of theories ${\mathcal{M}}$ is countable in the proof above when passing from almost sure convergence under $\tau^{*}$ sampled from the prior to almost sure convergence for any particular $\tau^{*}$ with positive prior mass. This argument fails for uncountable ${\mathcal{M}}$; indeed, characterization of the $\tau^{*}$ for which the posterior converges to $\delta_{\tau^{*}}$ is a delicate problem (see, *e.g.*, [, , ]). Concentration of the posterior in neighbourhoods of $\tau^{*}$ under some topology on ${\mathcal{M}}$ has been studied by [, , ], among others. For *parametric* families of theories with parameter $\theta\in\mathbb{R}^{d}$, under smoothness and nondegeneracy assumptions, the Bernstein-von Mises theorem guarantees convergence of the posterior $\operatorname{P}(\theta\mid Z_{1:t})$ to the true parameter $\theta^{*}$ at a rate that is asymptotically Gaussian with inverse covariance $I(\theta^{*})t$, where $I(\cdot)$ denotes the Fisher information matrix.

### On convergence rates.

While we do not handle the *rate* of convergence in Proposition 3.1, guarantees can be obtained under specific assumptions on the prior and the set of theories.

For example, for any $\tau\in{\mathcal{M}}$, the quantity $D_{\tau}^{t}:=\log\frac{\operatorname{P}(\tau^{*}\mid Z_{1:t})}{\operatorname{
P}(\tau\mid Z_{1:t})}$ is a process with $D_{\tau}^{0}=\log\frac{\operatorname{P}(\tau^{*})}{\operatorname{P}(\tau)}$ and i.i.d.i increments, with

$$ $\mathbb{E}[D_{\tau}^{t+1}-D_{\tau}^{t}]=D_{\mathrm{KL}}(\tau^{*}\parallel\, \tau),\quad\mathbb{E}\left[(D_{\tau}^{t+1}-D_{\tau}^{t})^{2}\right]=\mathbb{E} _{Z\sim\tau^{*}}\left[\left(\log\frac{\operatorname{P}_{\tau*}(Z)}{ \operatorname{P}_{\tau}(Z)}\right)^{2}\right].$ (3) $$

Under the assumption that the variances are finite and uniformly bounded in $\tau$, the central limit theorem would give posterior convergence rate guarantees.

Note that above we make no assumptions on the theories, and Proposition 3.1 is a ‘law-of-large-numbers-like’ result that holds even if the variances in (3) are not finite and uniformly bounded.

### Harm probability bounds.

So far we have considered a collection ${\mathcal{M}}$ of distributions over an observation space. Now we show bounds when each theory computes probabilities over some additional variables.

The following lemma extends Proposition 3.1(b) to estimates of real-valued functions of the theories and observations.

###### Lemma 3.3 .

Under the same conditions as Proposition 3.1, suppose $f:{\mathcal{M}}\times\bigcup_{t=0}^{\infty}{\mathcal{Z}}^{t}\to\mathbb{R}_{
\geq 0}$ is any bounded measurable function.
Then there exists $N\in\mathbb{N}$ such that for all $t\geq N$ and any

$$ $\tilde{\tau}\in\operatorname*{arg\,max}_{\tau}[\operatorname{P}(\tau\mid Z_{1: t})f(\tau,Z_{1:t})],$ $$

it holds that $f\left(\tilde{\tau},Z_{1:t}\right)\geq f(\tau^{*},Z_{1:t})$.

###### Proof.

First, note that the argmax exists by boundedness of $f$ and $\operatorname{P}(\cdot\mid Z_{1:n})$. By Proposition 3.1(b), there exists $N\in\mathbb{N}$ such that for all $t\geq N$ and $\tau\neq\tau^{*}$, $\operatorname{P}(\tau^{*}\mid Z_{1:t})>\operatorname{P}(\tau\mid Z_{1:t})\geq 0$. Let $t\geq N$ and $\tilde{\tau}\in\operatorname*{arg\,max}_{\tau}[\operatorname{P}(\tau\mid Z_{1:
t})f(\tau,Z_{1:t})]$. Then

$$ $\operatorname{P}(\tau^{*}\mid Z_{1:t})f(\tilde{\tau},Z_{1:t})\geq\operatorname {P}(\tilde{\tau}\mid Z_{1:t})f(\tilde{\tau},Z_{1:t})\geq\operatorname{P}(\tau^ {*}\mid Z_{1:t})f(\tau^{*},Z_{1:t}).$ $$

When $\tilde{\tau}\neq\tau^{*}$, the result follows since $\operatorname{P}(\tau^{*}\mid Z_{1:t})>0$. The case $\tilde{\tau}=\tau^{*}$ is trivial.
∎

A particular case of interest is when each theory is associated with estimates of probabilities of harm ($Y=1$) given a context $x$ and past observations $Z_{1:t}$. That is, we identify ${\mathcal{M}}$ with a collection of conditional probability mass functions, denoted $\operatorname{P}(\cdot\mid x,\tau,Z_{1:t})$, for every $x$ lying in some space of possible contexts.

###### Proposition 3.4 (Harm probability bound) .

Under the same conditions as Proposition 3.1, there exists $N\in\mathbb{N}$ such that for all $t\geq N$ and

$$ $\tilde{\tau}\in\arg\max_{\tau}\operatorname{P}(\tau\mid Z_{1:t})\operatorname{ P}(Y=1\mid x,\tau,Z_{1:t}),$ (4) $$

it holds that

$$ $\operatorname{P}(Y=1\mid x,\tau^{*},Z_{1:t})\leq\operatorname{P}(Y=1\mid x, \tilde{\tau},Z_{1:t}).$ (5) $$

###### Proof.

Apply Lemma 3.3 to the function $f(\tau,Z_{1:t})=\operatorname{P}(Y=1\mid x,\tau,Z_{1:t})$.
∎

###### Lemma 3.3 .

###### Proof.

###### Proposition 3.4 (Harm probability bound) .

###### Proof.

## 4 Non-I.I.D. data

In this section, we remove the assumption made in Section 3 that observations $Z_{i}$ are i.i.d.i given a theory $\tau^{*}$.

### Setting.

As before, let $({\mathcal{Z}},{\mathcal{F}},\mu)$ be a $\sigma$-finite Borel measure space. For the results below to hold, we must also assume that $({\mathcal{Z}},{\mathcal{F}})$ is a Radon space (*e.g.*, any countable set or manifold), so as to satisfy the conditions of the disintegration theorem.

Let $({\mathcal{Z}}^{\infty},{\mathcal{F}}^{\infty},\mu^{\infty})$ be the space of infinite sequences of observations, ${\mathcal{Z}}^{\infty}=\{(z_{1},z_{2},\dots):z_{i}\in{\mathcal{Z}})\}$, with the associated product $\sigma$-algebra and $\sigma$-finite measure. This object is the projective limit of the measure spaces $({\mathcal{Z}}^{t},{\mathcal{F}}^{\otimes t},\mu^{\otimes t})$, where ${\mathcal{Z}}^{t}=\{(z_{1},\dots,z_{t}):z_{i}\in{\mathcal{Z}}\}$ and the projection ${\mathcal{Z}}^{t+1}\to{\mathcal{Z}}^{t}$ ‘forgets’ the observation $z_{t+1}$.

A *theory* $\tau$ is a probability distribution on $({\mathcal{Z}}^{\infty},{\mathcal{F}}^{\infty})$ that is absolutely continuous w.r.t. $\mu^{\infty}$. For $A\in{\mathcal{F}}^{\otimes t}$, we write $\tau_{1:t}(A)$ for the measure of the cylindrical set, $\tau(A\times{\mathcal{Z}}\times{\mathcal{Z}}\times\dots)$, so $\tau_{1:t}$ is a measure on $({\mathcal{Z}}^{t},{\mathcal{F}}^{\otimes t})$. Because ${\mathcal{F}}^{\infty}$ is generated by cylindrical sets, the absolute continuity condition on $\tau$ is equivalent to absolute continuity of $\tau_{1:t}$ w.r.t. $\mu^{\otimes t}$ for all $t$.(^3^33This is in turn equivalent to absolute continuity of conditional distributions, *i.e.*, that for all measurable $A\subseteq{\mathcal{Z}}^{t}$ with $\tau_{1:t}(A)>0$, $\frac{1}{\tau_{1:t}(A)}\tau_{1:t+1}|_{A\times{\mathcal{Z}}}\ll\mu^{t+1}|_{A \times{\mathcal{Z}}},\quad A\times{\mathcal{Z}}\subseteq{\mathcal{Z}}^{t} \times{\mathcal{Z}}\cong\operatorname{\mathcal{Z}}^{t+1}.$)
This condition allows to define measurable probability density functions $\operatorname{P}_{\tau}:{\mathcal{Z}}^{t}\to\mathbb{R}_{\geq 0}$ as Radon-Nikodym derivatives, so that

$$ $\forall A\in{\mathcal{F}}^{\otimes t},\quad\tau_{1:t}(A)=\int_{A}\operatorname {P}_{\tau}(z_{1},\dots,z_{t})\,d\mu^{\otimes t},$ $$

and measurable conditional probability densities $\operatorname{P}_{\tau}(z_{t+1}\mid z_{1:t}):=\frac{\operatorname{P}_{\tau}(z_
{1:t},z_{t+1})}{\operatorname{P}_{\tau}(z_{1:t})}$. The disintegration theorem for product measures implies that these conditionals and marginals over finitely many observations can be manipulated algebraically using the usual rules of probability for $\mu^{\infty}$-a.e. collection of values, *e.g.*, one has the autoregressive decomposition $\operatorname{P}_{\tau}(z_{1:T})=\prod_{t=1}^{T}\operatorname{P}_{\tau}(z_{t}
\mid z_{1:t-1})$, with the conditional $\operatorname{P}_{\tau}(z_{1}\mid z_{1:0})$ understood to be the marginal $\operatorname{P}_{\tau}(z_{1})$.

A theory determines a random variable $Z_{1:\infty}$ taking values in ${\mathcal{Z}}^{\infty}$. We denote its components by $Z_{1},Z_{2},\dots$ and the collection of the first $t$ observations by $Z_{1:t}$.

### Definition of posterior as a random variable.

Let ${\mathcal{M}}$ be a countable multiset of theories(^4^44Unlike in Section 3, we do not require theories to be distinct for the results in this section.) and $\operatorname{P}$ a prior distribution on ${\mathcal{M}}$. We define the posterior to be

$$ $\operatorname{P}(\tau\mid z_{1:t})=\frac{\operatorname{P}(\tau)\operatorname{P }_{\tau}(z_{1:t})}{\sum_{\tau^{\prime}\in{\mathcal{M}}}\operatorname{P}(\tau^{ \prime})\operatorname{P}_{\tau^{\prime}}(z_{1:t})},$ (6) $$

assuming the denominator converges to a positive value.

Consider a ground truth theory $\tau^{*}\in{\mathcal{M}}$ and let $Z_{1:\infty}$ be the corresponding random variable taking values in ${\mathcal{Z}}^{\infty}$. Similarly to the i.i.d.i case, the posterior $\operatorname{P}(\cdot\mid Z_{1:t})$ is a random variable taking values in the space of probability mass functions on ${\mathcal{M}}$.

For all results below, we assume that $\operatorname{P}(\tau^{*})>0$.

### Bayesian posterior convergence.

Previous work (*e.g.*, []) has shown that if $Z_{1:\infty}\sim\operatorname{P}_{\tau^{*}}$, then the limit inferior of $\operatorname{P}(\tau^{*}\mid Z_{1:t})$ is almost surely positive. More generally, with probability at least $1-\delta$, the posterior on the truth won’t go below $\delta$ times the prior on the truth. We repeat that result here in our notation.

###### Lemma 4.1 (Martingale) .

The process $W_{t}:=\operatorname{P}(\tau^{*}\mid Z_{1:t})^{-1}$ is a supermartingale, *i.e.*, it doesn’t increase over time in expectation.

###### Proof.

We have

$$ $\displaystyle\mathbb{E}_{\tau^{*}}[W_{t+1}\mid Z_{1:t}=z_{1:t}]$ $\displaystyle\operatorname*{=}\int_{\scriptsize\big{\{}z_{t+1}\in\operatorname {\mathcal{Z}}\mathbin{:}\operatorname{P}_{\tau^{*}}(z_{t+1}\mid z_{1:t})>0\big {\}}}\operatorname{P}(\tau^{*}\mid z_{1:t+1})^{-1}\operatorname{P}_{\tau^{*}}( z_{t+1}\mid z_{1:t})\,d\mu$ $\displaystyle\operatorname*{=}^{(a)}\int_{\scriptsize\big{\{}z_{t+1}\in \operatorname{\mathcal{Z}}\mathbin{:}\operatorname{P}_{\tau^{*}}(z_{t+1}\mid z _{1:t})>0\big{\}}}\frac{\sum_{\tau\in\operatorname{\mathcal{M}}}\operatorname{ P}(\tau\mid z_{1:t})\operatorname{P}_{\tau}(z_{t+1}\mid z_{1:t})}{ \operatorname{P}(\tau^{*}\mid z_{1:t})\operatorname{P}_{\tau^{*}}(z_{t+1}\mid z _{1:t})}\operatorname{P}_{\tau^{*}}(z_{t+1}\mid z_{1:t})\,d\mu$ $\displaystyle\operatorname*{\leq}^{(b)}\int_{\mathcal{Z}}\frac{\sum_{\tau\in \operatorname{\mathcal{M}}}\operatorname{P}(\tau\mid z_{1:t})\operatorname{P}_ {\tau}(z_{t+1}\mid z_{1:t})}{\operatorname{P}(\tau^{*}\mid z_{1:t})}\,d\mu$ $\displaystyle=w_{t}\sum_{\tau\in\operatorname{\mathcal{M}}}\operatorname{P}( \tau\mid z_{1:t})\int_{\mathcal{Z}}\operatorname{P}_{\tau}(z_{t+1}\mid z_{1:t} )\,d\mu$ $\displaystyle\operatorname*{=}^{(c)}w_{t}$ (7) $$

where $(a)$ is by the definition (6), $(b)$ follows from cancellation and positivity of the integrand, $w_{t}:=\operatorname{P}(\tau^{*}\mid Z_{1:t}=z_{1:t})^{-1}$ is the realization of $W_{t}$, and $(c)$ follows because both the posterior and the conditional probability measure integrate to 1.
∎

###### Proposition 4.2 (Posterior on truth) .

For all $\delta>0$, with probability at least $1-\delta$, $\inf_{t}\operatorname{P}(\tau^{*}\mid Z_{1:t})\geq\delta\operatorname{P}(\tau^
{*})$, *i.e.*,

$$ $\tau^{*}\Big{(}\big{\{}z_{1:\infty}:\inf_{t}\operatorname{P}(\tau^{*}\mid z_{1 :t})<\delta\operatorname{P}(\tau^{*})\big{\}}\Big{)}\leq\delta.$ $$

###### Proof.

By Ville’s inequality [] for the supermartingale $W_{t}$, for any $\lambda>0$:

$$ $\tau^{*}\Big{(}\sup_{t\geq 0}W_{t}\geq\lambda\Big{)}\leq\frac{\mathbb{E}[W_{0} ]}{\lambda}=\frac{1}{\lambda\operatorname{P}(\tau^{*})}$ $$

Setting $\lambda=(\delta\operatorname{P}(\tau^{*}))^{-1}$, we get

$$ $\tau^{*}\big{(}\sup_{t\geq 0}W_{t}\geq(\delta\operatorname{P}(\tau^{*}))^{-1} \big{)}\leq\delta$ (8) $$

and given that
$\{z_{1:\infty}:\sup_{t\geq 0}W_{t}>(\delta\operatorname{P}(\tau^{*}))^{-1}\}=
\{z_{1:\infty}:\inf_{t\geq 0}V_{t}<\delta\operatorname{P}(\tau^{*})\}$, the result follows.
∎

In the language of financial markets, if $W_{t}$ is the price of a martingale stock at time $t$, you could never gain money in expectation by holding it. Suppose you “bought shares” at time 0, paying $W_{0}$, and waited for their value to increase by a factor of $\delta^{-1}$. If (8) didn’t hold, and the probability of such an increase occurring were greater than $\delta$, then you could make expected profit, “$\delta^{-1}$-tupling” your money with probability more than $\delta$.

###### Remark 4.3 .

Proposition 4.2 is “tight” in the following sense: for all $\delta,\varepsilon>0$, there exists a model class $\operatorname{\mathcal{M}}$, a prior distribution $\operatorname{P}$, and a true model $\tau^{*}\in\operatorname{\mathcal{M}}$, such that with probability at least $\delta$, $\limsup_{t}\operatorname{P}(\tau^{*}\mid Z_{1:t})\leq(\delta+\varepsilon)
\operatorname{P}(\tau^{*})$.

We construct such an example. Consider the following setting: $\operatorname{\mathcal{M}}=\{\tau^{*},\tau^{\prime}\}$, $\operatorname{\mathcal{Z}}=\{0,1\}$, and the theories are defined by

$$ $\displaystyle\operatorname{P}_{\tau^{*}}(1)$ $\displaystyle=\delta,$ $\displaystyle\operatorname{P}_{\tau^{\prime}}(1)$ $\displaystyle=1,$ $\displaystyle\operatorname{P}_{\tau}(1\mid z_{1:t})$ $\displaystyle=\frac{1}{2}$ $\displaystyle\forall\tau\in{\mathcal{M}},t\geq 1,z_{1:t}\in{\mathcal{Z}}^{t}.$ $$

One has

$$ $\operatorname{P}(\tau^{*}\mid Z_{1}=1)=\frac{\delta\operatorname{P}(\tau^{*})} {\delta\operatorname{P}(\tau^{*})+\operatorname{P}(\tau^{\prime})}<\delta\frac {\operatorname{P}(\tau^{*})}{1-\operatorname{P}(\tau^{*})}.$ $$

Because $\tau^{*}$ and $\tau^{\prime}$ give exactly the same conditional probabilities of $Z_{t}$ given $Z_{1:t-1}$ for $t>1$, one has $\operatorname{P}(\tau^{*}\mid Z_{1:t})=\operatorname{P}(\tau^{*}\mid Z_{1})$. So, for all $t\geq 1$,

$$ $\operatorname{P}(\tau^{*}\mid Z_{1}=1,Z_{2:t}=z_{2:t})<\delta(1-\operatorname{ P}(\tau^{*}))^{-1}\operatorname{P}(\tau^{*})$ $$

and in particular

$$ $\tau^{*}\Big{(}\big{\{}z_{1:\infty}:\limsup_{t}\operatorname{P}(\tau^{*}\mid z _{1:t})<\delta(1-\operatorname{P}(\tau^{*}))^{-1}\operatorname{P}(\tau^{*}) \big{\}}\Big{)}\geq\Big{(}\big{\{}z_{1:\infty}:z_{1}=1\big{\}}\Big{)}= \operatorname{P}_{\tau^{*}}(1)=\delta.$ $$

Selecting a prior with $\operatorname{P}(\tau^{*})<1-1/(1+\frac{\varepsilon}{\delta}))$, so that $\delta(1-\operatorname{P}(\tau^{*}))^{-1}<\delta+\varepsilon$, we obtain an example with the desired property.

###### Lemma 4.1 (Martingale) .

###### Proof.

###### Proposition 4.2 (Posterior on truth) .

###### Proof.

###### Remark 4.3 .

### Harm probability bounds.

We now state analogues of Proposition 3.4 in the non-i.i.d.i setting. As above, let $Y_{t}$ be a binary random variable that may depend on $Z_{1:t}$, $\tau$, and some context variable $x_{t}$.

###### Proposition 4.4 (Weak harm probability bound) .

For any $\delta>0$, with probability at least $1-\delta$, the following holds for all $t\in\mathbb{N}$ and all $x_{t}$:

$$ $\operatorname{P}(Y_{t}=1\mid Z_{1:t},\tau^{*},x_{t})\leq\sup_{\tau\in \operatorname{\mathcal{M}}}\frac{\operatorname{P}(\tau\mid Z_{1:t}) \operatorname{P}(Y_{t}=1\mid Z_{1:t},\tau,x_{t})}{\delta\operatorname{P}(\tau^ {*})}.$ $$

###### Proof.

Substituting $\tau$ for $\tau^{*}$ on the r.h.s. can never increase the r.h.s., since $\tau^{*}\in\operatorname{\mathcal{M}}$. Then, after canceling and rearranging terms, the proposition becomes identical to Proposition 4.2.
∎

Next, we show how the bound in Proposition 4.4 can be strengthened by restricting to theories that have sufficiently high posterior mass relative to theories that are better than them.

Let $\tau^{1}_{Z_{1:t}},\tau^{2}_{Z_{1:t}},\tau^{3}_{Z_{1:t}},\dots$ be an enumeration of ${\mathcal{M}}$ in order of decreasing posterior weight $\operatorname{P}(\tau\mid Z_{1:t})$, breaking ties arbitrarily(^5^55For example, following some fixed enumeration of ${\mathcal{M}}$.) (*i.e.*, we have $\operatorname{P}(\tau^{i}_{Z_{1:t}}\mid Z_{1:t})\geq\operatorname{P}(\tau^{i+1
}_{Z_{1:t}}\mid Z_{1:t})$ for all $i$). Each $\tau^{i}_{Z_{1:t}}$ is a ${\mathcal{M}}$-valued random variable (*i.e.*, the index of a theory in ${\mathcal{M}}$). For any $0<\alpha\leq 1$, we also define the ${\mathcal{P}}({\mathcal{M}})$-valued random variable

$$ $\operatorname{\mathcal{M}}^{\alpha}_{Z_{1:t}}:=\bigg{\{}\tau^{n}_{Z_{1:t}}\in \operatorname{\mathcal{M}}:\operatorname{P}(\tau^{n}_{Z_{1:t}}\mid Z_{1:t}) \geq\alpha\sum_{m\leq n}\operatorname{P}(\tau^{m}_{Z_{1:t}}\mid Z_{1:t})\bigg{ \}},$ (9) $$

which is the multiset of theories that contain at least $\alpha$ of the posterior mass of all theories that are not worse than it. If $\alpha=1$, this set contains exactly one element, $\tau^{1}_{Z_{1:t}}$. For any $0<\alpha<1$, this set is nonempty, because it contains $\tau^{1}_{Z_{1:t}}$, and finite, since $|\operatorname{\mathcal{M}}^{\alpha}_{Z_{1:t}}|\geq N$ implies (easily by induction) that

$$ $\sum_{\tau\in\operatorname{\mathcal{M}}^{\alpha}_{Z_{1:t}}}\operatorname{P}( \tau\mid Z_{1:t})\geq\left(\frac{1}{1-\alpha}\right)^{N-1}\operatorname{P}( \tau^{1}_{Z_{1:t}}\mid Z_{1:t}).$ $$

The following proposition is essentially identical to [, Thm 2], but our setting is a bit simpler.

###### Proposition 4.5 (True theory dominance) .

If $\alpha<\delta\operatorname{P}(\tau^{*})$, then with probability at least $1-\delta$, for all $t\in\mathbb{N}$, $\tau^{*}\in\operatorname{\mathcal{M}}^{\alpha}_{Z_{1:t}}$.

###### Proof.

For any $t\geq 1$, by Proposition 4.2,

$$ $\displaystyle\delta$ $\displaystyle\geq\tau^{*}\Big{(}\big{\{}z_{1:\infty}:\inf_{t^{\prime}} \operatorname{P}(\tau^{*}\mid z_{1:t^{\prime}})<\delta\operatorname{P}(\tau^{* })\big{\}}\Big{)}$ $\displaystyle\geq\tau^{*}(\{z_{1:\infty}:\operatorname{P}(\tau^{*}\mid z_{1:t} )<\delta\operatorname{P}(\tau^{*})\})$ $\displaystyle\geq\tau^{*}(\{z_{1:\infty}:\operatorname{P}(\tau^{*}\mid z_{1:t} )<\alpha\}).$ $$

So $\tau^{*}(\{z_{1:\infty}:\operatorname{P}(\tau^{*}\mid z_{1:t})\geq\alpha\})
\geq 1-\delta$, and the result follows by the fact that $\operatorname{\mathcal{M}}^{\alpha}_{Z_{1:t}}\supseteq\{\tau\in\operatorname{
\mathcal{M}}:\operatorname{P}(\tau\mid Z_{1:t})\geq\alpha\}$, since the sum in (9) never exceeds $1$.
∎

###### Proposition 4.6 (Harm probability bound) .

If $\alpha<\delta\operatorname{P}(\tau^{*})$, then with probability at least $1-\delta$, for all $t\in\mathbb{N}$ and all $x_{t}$,

$$ $\operatorname{P}(Y_{t}=1\mid Z_{1:t},\tau^{*},x_{t})\leq\max_{\tau\in \operatorname{\mathcal{M}}^{\alpha}_{Z_{1:t}}}\operatorname{P}(Y_{t}=1\mid Z_{ 1:t},\tau,x_{t})$ (10) $$

###### Proof.

This follows directly from Proposition 4.5.
∎

Because the conclusion of Proposition 4.6 is much stronger than that of Proposition 4.4, it would be much safer (or more useful, depending on the value of $\alpha$) to use $\operatorname*{arg\,max}_{\tau\in\operatorname{\mathcal{M}}^{\alpha}_{Z_{1:t}}
}\operatorname{P}(Y_{t}=1\mid Z_{1:t-1},\tau,x_{t})$ as a ‘paranoid’ theory rather than $\operatorname*{arg\,max}_{\tau\in\operatorname{\mathcal{M}}}\operatorname{P}(
\tau\mid Z_{1:t-1})\operatorname{P}(Y_{t}=1\mid Z_{1:t-1},\tau,x_{t})$. The factor of $(\delta\operatorname{P}(\tau^{*}))^{-1}$ in Proposition 4.4 could render the upper bound on harm probability much larger than the trivial upper bound of 1. However, we note that approximating ${\mathcal{M}}^{\alpha}_{Z_{1:t}}$ – such as by amortization or by Monte Carlo methods – is much more difficult than approximating the posterior alone.

###### Proposition 4.4 (Weak harm probability bound) .

###### Proof.

###### Proposition 4.5 (True theory dominance) .

###### Proof.

###### Proposition 4.6 (Harm probability bound) .

###### Proof.

### On the harm-recording mechanism.

Suppose that $\tau^{*}$ is a data-generating process meeting the description “$Y_{t}=1$ when harm has occurred”, while $\tau^{\dagger}$ is a data-generating process meeting the description “$Y_{t}=1$ when harm is recorded as having occured” and agreeing with $\tau^{*}$ in its observational predictions otherwise. If and only if the recording process is functioning correctly, $\tau^{*}=\tau^{\dagger}$. For as long as the recording process is functioning correctly, $\operatorname{P}(\tau^{*}\mid Z_{1:t})/\operatorname{P}(\tau^{\dagger}\mid Z_{
1:t})=\operatorname{P}(\tau^{*})/\operatorname{P}(\tau^{\dagger})$. Should the recording process ever fail at time $t$, then $Z_{t}\sim\operatorname{P}_{\tau^{\dagger}}$, not $\operatorname{P}_{\tau^{*}}$, since $Z_{t}$ is the result of this recording process; therefore, $\operatorname{P}(\tau^{*}\mid Z_{1:t})/\operatorname{P}(\tau^{\dagger}\mid Z_{
1:t})$ would decrease in expectation, perhaps dramatically. We should not expect $\operatorname{P}(\tau^{*})$ to naturally win out over $\operatorname{P}(\tau^{\dagger})$, even if there are no mistakes when evaluating and recording how harmful certain situations are. However, the following holds with probability approaching $1$ as $\alpha\to 0$: for all $t$, if the recording process has not failed by time $t$, $\operatorname{\mathcal{M}}^{\alpha}_{Z_{1:t}}$ contains both $\tau^{*}$ and $\tau^{\dagger}$. If $\tau^{*}$ considers tampering with the recording process to be a “harmful” outcome, then an AI system could attempt to avoid a first instance of tampering at time $t$, for all $t$.

## 5 Experiments

### Exploding bandit setting.

We evaluate the performance of safety guardrails based on Proposition 3.4 and Proposition 4.6 in a bandit MDP with $10$ arms (actions).(^6^66Code available at [https://github.com/saifh-github/conservative-bayesian-public](https://github.com/saifh-github/conservative-bayesian-public).)

Each arm $a\in\{1,\dots,10\}$ is represented by a feature vector $f_{a}\in\{0,1\}^{d}$ (we take $d=10$, but $d$ is not necessarily equal to the number of arms), which is sampled uniformly at random in each iteration of the experiment and known to the agent. The reward distribution of each arm is fixed for the duration of each episode and assumed to be of the following form: the reward received after taking action $a$ follows a unit-variance normal distribution, $r(a)\sim\mathcal{N}(f_{a}\cdot v^{*},1)$, where $v^{*}\in\{0,1\}^{d}$ is some vector sampled uniformly at random at the start of each episode and unknown to the agent.

Taking any action and observing the reward gives evidence about the identity of $v^{*}$ and thus about the reward distributions of the other actions. The agent maintains a belief over the vector used to compute the reward, beginning with a uniform prior over $\{0,1\}^{d}$ and updating its posterior with each observation of an action-reward pair.

We assume that the agent samples its actions from a Boltzmann policy (with temperature 2) using the expected reward of each action under its posterior given the data seen so far, meaning that the posterior over reward vectors fully determines a distribution over observations of action-reward pairs $(a_{t},r_{t})$ observed when following the policy. Thus the set $\{0,1\}^{d}$ can be identified with a multiset ${\mathcal{M}}$ of theories about observations of such pairs, where the vector $v$ determines the distribution $\tau_{v}$ over action-reward pairs(^7^77The mapping $v\mapsto\tau_{v}$ is not necessarily injective – multiple vectors may represent the same collection of reward distributions and therefore the same distribution over action-reward pairs.). Inference of $v$ under a uniform prior over $\{0,1\}^{d}$ with evidence collected on-policy is equivalent to inference of $\tau$ under a uniform prior over ${\mathcal{M}}$ given data generated by a true theory $\tau^{*}=\tau_{v^{*}}$. Note that since the policy changes over time, we are in the non-i.i.d.i setting.

The bandit comes with a notion of harm: if the reward received at a given timestep exceeds some threshold $E$, the bandit explodes and the agent dies, terminating the episode. In other words, we define harm as $Y_{t}=\mathbbm{1}[R_{t}>E]$, where $R_{t}$ is the random variable representing the reward received when taking action $a_{t}$. $E$ is set to the highest mean reward of any action (*i.e.*, $\max_{a}(f_{a}\cdot v^{*})$. The maximum episode length is $25$ timesteps.

### Safety guardrails.

A *guardrail* is an algorithm that, given a possible action and context (*e.g.*, current state and history), determines whether taking the action in the context is admissible. A guardrail can be used to mask the policy to forbid certain actions, such as those whose estimated harm exceeds some threshold $C$.

We compare several guardrails: those constructed from Proposition 3.4 and Proposition 4.6, one that marginalizes across the posterior over $\tau$ to get the posterior predictive harm probability, and one that ‘cheats’ by using the probability of harm under the true theory $\tau^{*}$. We define the four guardrails formally below. Recall that $Z_{1:t}$ consists of the observations (*i.e.*, actions taken and rewards received) at previous timesteps.

- •
Proposition 3.4 guardrail: rejects an action $a_{t+1}$ if there exists $\tilde{\tau}\in\operatorname*{arg\,max}_{\tau}\operatorname{P}(\tau\mid Z_{1:t
})\operatorname{P}(Y_{t+1}=1\mid\tau,Z_{1:t},a_{t+1})$ with $\operatorname{P}(Y_{t+1}=1\mid\tilde{\tau},Z_{1:t},a_{t+1})>C$ (note that the assumptions of i.i.d.i observations and distinct theories are not satisfied here).
- •
Proposition 4.6 guardrail: rejects an action $a_{t+1}$ if $\max_{\tau\in\operatorname{\mathcal{M}}^{\alpha}_{Z_{1:t}}}\operatorname{P}(Y_
{t+1}=1\mid Z_{1:t},\tau,a_{t+1})>C.$
- •
Posterior predictive guardrail: rejects an action $a_{t+1}$ if $P(Y_{t+1}=1\mid Z_{1:t},a_{t+1})>C$.
- •
Cheating guardrail: rejects an action $a_{t+1}$ if $P(Y_{t+1}=1\mid Z_{1:t},\tau^{*},a_{t+1})>C$ (note that this guardrail assumes knowledge of the true theory $\tau^{*}$).

The guardrail is run at every sampling step, and actions that the guardrail rejects are forbidden to be sampled by the agent. If all actions are rejected by the guardrail, the episode terminates.

### Results.

Figure 1 shows mean episode rewards and episode deaths under each guardrail across $10000$ episodes, for different values of the rejection threshold $C$. The cheating guardrail achieves zero deaths for sufficiently small $C$, but for $C=0.1$ its death probability is moderately high.(^8^88Indeed, if every action taken had a harm probability of $0.1$, the probability of death across an episode would be $1-\big{(}(1-0.1)^{25}\big{)}\approx 0.93$.) The posterior predictive guardrail also achieves zero deaths for small $C$, while for larger $C$ it dies slightly more often and achieves somewhat lower reward compared to the cheating guardrail. The behaviour of the Proposition 4.6 guardrail depends strongly on $\alpha$. When $\alpha$ is close to $1$, actions are very rarely rejected, leading to a high probability of an early death and precluding the opportunity to obtain much reward. At the other extreme, when $\alpha$ is close to 0 0, the candidate theory set ${\mathcal{M}}^{\alpha}_{Z_{1:t}}$ is larger and the guardrail is extremely conservative. It rejects almost all actions, resulting in low deaths and low reward. This is the case even for larger $C$, since the estimated probability used to filter actions tends to overestimate an action’s harm probability under the true theory. For middling values of $\alpha$, the Proposition 4.6 guardrail performs similarly to the posterior predictive guardrail. The Proposition 3.4 guardrail, which makes the incorrect assumptions of i.i.d.i data and distinct theories, is similarly conservative to the Proposition 4.6 guardrail with low $\alpha$.

Figure: Figure 1: Mean episode deaths and reward for different guardrails in the exploding bandit setting.
Refer to caption: x1.png

### Tightness of bounds.

Figure 2 takes a closer look at how often, and how tightly, the inequality in Proposition 4.6 is satisfied. For an agent following a uniform policy across $10000$ bandit episodes without action rejection or death, Figure 2(a) shows the frequency with which $\max_{\tau\in\operatorname{\mathcal{M}}^{\alpha}_{Z_{1:t}}}\operatorname{P}(Y_
{t+1}=1\mid Z_{1:t},\tau,a_{t+1})$ overestimates the true harm probability of an action. Proposition 4.6 gives us a strict lower bound of $1-\frac{\alpha}{\operatorname{P}(\tau^{*})}$ (which may be below 0 0) on the overestimation frequency, but the frequency significantly exceeds the bound for large $\alpha$. Figure 2(b) shows the distribution of harm estimates for the most dangerous action, which always has a ground truth harm probability of $0.5$ due to the choice of $E$. Note that for large $\alpha$ the harm of this dangerous action is usually *underestimated* – so the high overestimation rate in Figure 2(a) comes from actions with lower harm probabilities.

Figure: (a) The frequency with which the inequality in Proposition 4.6 is satisfied.
Refer to caption: x2.png

## 6 Conclusion and open problems

The approach to safety verification proposed here is based on context-dependent run-time verification because the set of possible inputs for a machine learning system is generally astronomical, while the safety of the answer to a specific question is more likely to be amenable to tractable calculations. It focuses on the risk of wrongly interpreting the data, including the safety specification itself (what we called “harm” above) and exploits the fact that as more evidence is gathered (as necessarily happens with i.i.d.i data) and when different theories predict different observations, the true interpretation rises towards the maximal value of the Bayesian posterior over interpretations. The bound is tighter with i.i.d.i data but the i.i.d.i assumption is also not realistic, and in the context of safety-critical decisions, we would prefer to err on the side of prudence and fewer assumptions. However, it provides an interesting template to think about variants of this idea in future work.

There are many open problems to consider before turning the kinds of bounds introduced above into an operational run-time safeguard:

- 1.
Upper-bounding overcautiousness. Can we ensure that we do not underestimate the probability of harm but do not massively overestimate it? Some simple theories consistent with the dataset (even an arbitrarily large one) might deem non-harmful actions harmful. Can we bound how much this harm-avoidance hampers the agent? A plausible approach would be to make use of a mentor for the agent that demonstrates non-harmful behavior [].
- 2.
Tractability of posterior estimation. How can we efficiently estimate the required Bayesian posteriors? For computational tractability, a plausible answer would rely on amortized inference, which turns the difficult estimation of these posteriors into the task of training a neural net probabilistic estimator which will be fast at run-time. Recent work on amortized Bayesian inference for symbolic models, such as causal structures [, ], and for intractable posteriors in language models [, , , , ] – which are useful when prior knowledge is encoded in a pretrained foundation model – suggests that this is feasible. Advances in efficient and adaptive Monte Carlo methods, *e.g.*, for language models [, , ], can also be useful for this purpose, and MCMC approaches can complement and aid amortization [, , , ].
- 3.
Efficient search for a cautious theory. How can we efficiently identify a cautious but plausible theory that upper-bounds the risk, since this requires an optimization at run-time? Again, a plausible answer is to rely on amortized probabilistic inference, *e.g.*, by sampling theories with a low or annealed temperature, a technique that has been used for decades in the nonconvex optimization literature and more recently combined with amortized sequential sampling methods [, , , *inter alia*].
- 4.
Partial specification of theories. Another issue is that a full explanatory theory $\tau$ for the evidence $D$ will generally be too large an object to make the above bound calculations and optimization sufficiently efficient. An alternative would be to derive bounds that only require partial explanations, similarly to how humans tell cautious ‘stories’ about what could go wrong in uncertain circumstances when thinking about taking a potentially dangerous decision [].
- 5.
Harm specification in natural language. How is a safety specification provided in natural language, like with Constitutional AI [], turned into a random variable over which conditional probabilities can be estimated? Having humans label lots of situations that match the desired constraint seems impractical. The alternative proposed by Bengio [] is to learn to interpret natural language statements as formal random variables over which probabilistic causal reasoning can occur. The ambiguity of natural language can then be captured by having a distribution over formal interpretations.
- 6.
Effect of approximation error. Finally, because we envisage using machine learning to estimate the bound, we have to consider the fact that a learned predictor will be imperfect. How do we correct conservatively for the potentially dangerous errors made by this predictor? This may be particularly important if we use a neural network to estimate probabilities, since they are subject to lack of robustness and adversarial attacks [, ], *e.g.*, how do we make sure that the agent proposing the actions has not learned to find loopholes in the risk estimator?

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: None.
soundness: 3
strengths: This is a very well written paper, and it is easy to follow. 

The proposed approach represents a promising initial step toward designing AI systems that ensure safety through built-in probabilistic guarantees, rather than relying solely on external safety mechanisms.

The authors also outline several open problems for future work.
summary: This paper explores the problem of designing AI systems that satisfy probabilistic safety guarantees. Within a Bayesian framework and given the safety specifications (as a probability), the authors provide risk bounds for potentially harmful decisions, showing that the probability of harm can be upper-bounded by a probability that can be estimated by approximating Bayseian posterior over theories given the observed data. They study two settings: i.i.d case and non i.i.d case and provide a simple experiment to evaluate the performance of safety guardrails.
weaknesses: The authors present an upper bound on the harm probability, though it appears to be highly conservative. It would be valuable if they could offer a convergence rate or practical guarantees to make the framework more usable. Additionally, it is unclear how this approach compares to other conservative methods for preventing harm. 

Since the theoretical results lack practical assurances, I would have appreciated more experimental validation, especially in complex and realistic settings. 

Obtaining a Bayesian oracle could be very challenging (posterior distribution). 

Overall, while the paper introduces a promising method for designing safer AI systems, it would greatly benefit from additional components (both theoretical and experimental) before publication.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: Q1. How do you envision these guardrails to be applied in realistic scenarios? For example, consider the situation of a language model trying to obtain your passwords, or an autonomous car trying to crash with another vehicle. Could this notion of harm be applied efficiently to these realistic scenarios?

Q2. How sensitive are the results to the choice of priors in the Bayesian framework? Can the authors discuss the robustness of the proposed approach under different prior choices?
soundness: 3
strengths: S1. The topic of AI safety is timely and relevant for ICLR.

S2. The theoretical results (as far as I could check) are sound.

S3. The experimental evaluation serves to showcase how these bounds could be used in a realistic scenario.
summary: This paper studies the problem of bounding the probability of some event in the context of an unknown but consistent distribution and a Bayesian setting. 

The paper is motivated by the prevention of harm by AI agents. In short, harm is inherently unavoidable since in real applications we have no direct access to the distribution governing the environment. However, if we assume a fixed distribution, and a prior assumption of that distribution, we can get better and better approximations when data is presented to us, by using the data to update our prior knowledge of the distribution. With this, we can theoretically bound the probability of doing harm. In deployment, actions whose probability of harm is larger than some threshold can be blocked.

The paper explores two cases: incoming data as iid and non iid, and obtains bounds on the probability of harm in both cases.

The paper presents an experimental evaluation on a multi-armed bandits example, blocking actions that are considered unsafe according to the different bounds obtained as well as a baseline (with an unrealistic assumption of the underlying model). The paper ends with a discussion of the open problems still to be solved to be able to use this method as a reliable guardrails for AI agents.
weaknesses: W1. I understand the appeal to frame this work in the context of harm by an AI agent, and I think it is an interesting point. However, there is nothing inherent to "harm" in the concept presented. The concept of "harm" could be substituted by "reward at a state" and we could be discussing the same results in a different light. I think the paper may benefit from a more general motivation.

W2. While the experimental evaluation is welcome, it is a very simple example, and one wonders if these theoretical bounds would find applicability in problems that are more complex and close to the real applications of guardrails.

W3. The concept of guardrails presented here, as an algorithm that blocks an action if it shows an expected harm larger than some threshold, is very similar to the concept of probabilistic shielding in MDPs [1] (which is essentially the "cheating guardrail in Sec. 5), and this can be extended to partially observable MDPs to eliminate the (unrealistic) assumption of having full knowledge of the ground truth [2]. The paper would benefit from comparing to these methods, especially with [2].

W4. The paper does not engage in some recent work on defining harm in similar scenarios, see for example [3] or [4]. It could be useful to understand, in light of different definitions of harm, whether the results are specific to harm prevention, or can be framed in a more general understanding of bounds over rewards.



OTHER (MINOR) REMARKS

R1. The paper is mathematically dense and difficult to follow in parts. I'm not sure whether this is a weakness on its own, but I have the feeling that the ideas conveyed are simpler than the dense mathematical presentation seems to suggest. 


REFERENCES

[1] N. Jansen et al. Safe Reinforcement Learning Using Probabilistic Shields. CONCUR 2020.

[2] S. Carr et al. Safe Reinforcement Learning via Shielding under Partial Observability. AAAI 2024.

[3] S. Beckers et al. Quantifying Harm. IJCAI 2023.

[4] J. G. Richens. Counterfactual harm. NeurIPS 2022.

---

confidence: 2
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. How does the upper bound in Prop. 3.4 apply if the prior distribution $P$ is misspecified?
2. How does this work compare to the existing literature on concentration bounds in the Bayesian setting? For instance, these methods could include analysis of Bayesian regret in RL, and PAC Bayes.
soundness: 3
strengths: - The paper is well-organized and clearly written. All the theoretical assumptions have been stated.
- The proposed concentration results seem reasonable. The derivations seem technically sound.
- Training large AI systems to satisfy certain safety criteria (i.e., with guardrails) is an exciting problem. This paper formulates this problem as a hypothesis-testing problem and presents non-trivial algorithms to perform the test. This problem formulation could be inspiring for other AI researchers across domains.
summary: This paper studies the problem of evaluating an unknown world model from observed data to determine whether it satisfies a certain safety metric. The safety metric, or guardrail, is a binary variable $H$, taking other variables in the world model as input. The authors utilize a Bayesian approach. It assumes access to the actual prior distribution over the groundtruth world model. The authors first prove that under certain parametric assumptions, the posterior distribution over candidate models will uniquely converge to the ground-truth model at the limit of large samples. Building on this concentration results, the authors derive an upper bound over the posterior probability of the harmful event $H = 1$ conditioning on the observed data. This concentration bound is then extended to non-i.i.d settings where observed samples are correlated. Finally, simulations were performed, and results supported the proposed theory.
weaknesses: - The concentration result in Prop. 3.1 assumes "all theories in $M$ are distinct as probability measures." This assumption does not seem to hold many common probabilistic models. For instance, in the linear component analysis, the number of independent components is generally not uniquely discernible (i.e., not identifiable) with non-linear mixing functions. Also, the number of latent components in Gaussian mixtures is generally not identifiable from the observed data. This seems to suggest that the application of the proposed concentration results might be limited.
- The proposed concentration results also assume access to the actual prior distribution generating the ground-truth world model. It is unclear whether the upper bound could still hold when the prior distribution is unknown and misspecified.
- Other concentration bounds exist over the target estimates using Baysian methods. Generally, one should be able to translate empirical concentration bounds to the Bayesian settings. For instance, (Osband & Van Roy, ICML'17) translates the concentration bounds for online reinforcement learning to Bayesian regret. How does the proposed method compare to other related work? This paper should include a section discussing related work in large deviation theory and how this paper is situated in the existing literature.

- Reference: _"Osband, Ian, and Benjamin Van Roy. "Why is posterior sampling better than optimism for reinforcement learning?." International conference on machine learning. PMLR, 2017."_

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Can you detail how you can utilize the CLT to obtain convergence rates (line 238ff)? If you applied this to the example in seciton 5, would it yield practical bounds?

2. Are the results in Propositions 4.4 to 4.6 tight (in a similar vein as Remark 4.3 shows for Proposition 4.2)?

3. How do you motivate the definition of $\mathcal{I}^{\alpha}_{1\colon t}$ and have you considered different approaches?

4. Can you provide some heuristics on choosing a safe, yet effective $\alpha$ a priori? Which information might be helpful for this from e.g. which model paramteres have the biggest impact on $\alpha$ and what information from a domain expert could be incorporated?

5. Can you make any predictions on how you proposed guardrails perform on larger, more complex models? In particular, how do you expect the overestimation of harm (see Fig. 2) to be affected?

6. Are there any existing works in which your framework fits, i.e. for which you can give (probabilistic) guarantees where they were previously unavailable?
If not, are there certain settings in which you can make reasonable a priori assumptions such that your framework is applicable and concrete guarantees can be derived for a given data set?

minor comments:
- line 96: explain what $q$ is
- line 193: introduce delta as dirac notation beforehand
- the axis and legends in the figures in section 5 are barely readable
soundness: 4
strengths: The main strength of the paper is the introduction of a (as far as I am aware) novel view at safe-by-design in AI at runtime and opening the possibilities for future Bayesian methods to utilize the safety guarantees shown in the paper. Allowing for safety guarantees and steering future work in a direction that empashizes those is a significant problem in AI.
I especially appreciate the discussion on open problems of the approach in the conclusion.

The theory being developed is also quite general and spans over a wide range of possible systems/problems.

The paper is well motivated and generally well structured, introducing formal concepts as needed in the respective sections. A small experimental evaluation is performed and well discussed. Proofs are provided in the appendix and I could not find any mistakes.
summary: The paper is tackling the problem of safety in AI. The authors take the view of defining safety as avoiding certain undesirable states in specific contexts.
They introduce a framework based on Bayesian inference from which an agent can derive safe policies that come with (probabilistic) guarantees of preventing harm.
The approach is safe-by-design, i.e. able to prevent undesired outcomes even if no concrete example of harmful states was ever observed in the system.
weaknesses: My main critique points of the paper are the lack of technical novelty (or at least it is not clarified enough if there are new results) and questions on applicability.

For the former, essentially all Propositions and Lemmata are either adaptations of well known results (Prop 3.1), taken from previous literature (Lemma 4.1, Prop 4.2), or rather simple Corollary derived from them (Lemma 3.3, Props. 3.4, 4.4, 4.5, 4.6). For Prop. 4.2 is is shown that the result is tight (Remark 4.3). To me it did not became clear whether this is a known result or a new contribution. It is also not clear whether the derived results (Props. 4.4, 4.5, 4.6) are also tight as a consequence or whether there is room for improvement.
For Prop. 4.5 and 4.6 in particular, restricting the possible world models indeces to $\mathcal{I}^{\alpha}_{1\colon t}$ is essential, however, the choice of definition of $\mathcal{I}^{\alpha}$ is not really motivated. At the same time, Fig. 2(a) shows a substantial gap between applying Prop 4.6 in practice, and the theoretical optimum. This begs the question whether a different definition of $\mathcal{I}^{\alpha}$ (e.g. a simple cutoff, or requiring $\mathcal{I}^{\alpha}$ to have a certain probability mass) has potential to yield tighter bounds. However, as the definition of $\mathcal{I}^{\alpha}$ is not motivated, these questions remain unadressed.

For the applicability, my core concern is that the main problem in AI is not providing safety guarantees under certain assumptions, but rather designing a Bayesian agent that actually works well for a given problem while satisfying these assumptions. To go into detail, section 3 only provides "law of large numbers"-style guarantees which are not useful in practice. A small paragraph on the rate of convergence (which would be very helpful to know) is included but essentially is very problem-dependent and thus not discussed in detail in this more general framework. In the experimental evaluation, where Prop. 3.4 is utilized, it is not even clear whether $t$ is large enough for the guarantee statement of Prop. 3.4 to hold (on top of Prop 3.4 not being applicable due to non-i.i.d. as the authors mention themselves). Section 4 then relaxes to probabilistic guarantees, which is a more practical approach. However, to apply the results of section 4 in practice it ultimately relies on defining a hyperparameter alpha. On the theoretical side, the guarantees in section 4 only hold if alpha is chosen small enough (which is impossible to know without knowing the system in the first place) and on the practical side, the evaluation in section 5 shows that choosing alpha too large can have catastrophic consequences, even for the simple bandit system considered in section 5. In summary, I do not see any immediate way to take advantage of the theoretical results the paper provides. This is also amplified by the fact that the main body essentially does not discuss related work, and how existing approaches can be embedded into the framework.

*These weaknesses make the paper feel like more of a statement paper with some additional mathematical background, rather than a fully fletched research paper.*

As a minor comment, from a reader's POV, the paper can be hard to follow at times, especially in the formal sections. Many paragraphs are written in a very technical way, assuming a deep mathematical background. While this surely can be expected from an audience like ICLR, I feel like many sections disrupt the flow of the paper, e.g. the two paragraphs "Setting" (l.155ff and l. 268ff). While these are defnitely important to make the paper rigorous, they are not strictly required to convey the main ideas of the paper. In the interest of readability, it might be advantageous to instead outsource the technical definitions to a separate section.

Meta-review: This paper seeks to create practical safety mechanisms for AI by introducing a Bayesian framework.
The reviewers agreed that the topic is important, given the potential implications for AI safety (e.g., Reviewer bi5T noted that the framework tackles “a significant problem in AI”) However, the reviewers were in agreement on certain limitations:
- Reviewer n2RR commented on the bounds being “highly conservative”, and questioned their practical applicability. Similarly, Reviewer bi5T  wrote that they “[..] do not see any immediate way to take advantage of the theoretical results the paper provides.”
- On a similar note again, Reviewer pbrg seemed skeptical about the paper's ability to generalize to problems “[..] complex and close to [..] real applications.”

Based on the agreement around these critiques, the decision was made to reject the paper. While the theoretical contributions were considered sound, expanding on what the theoretical model can offer in practice might be warranted.

Decision: Reject