# Oracles, sequence predictors, and self-confirming predictions

My [counterfactual Oracle](https://arxiv.org/abs/1711.05541) design uses a utility function/reward function in order to train it to give the right predictions. Paul Christiano asked whether the whole utility function approach was necessary, and whether it could be replaced with a simple sequence predictor.

It turns out the answers are no, and yes. The counterfactual nature of the approach can be preserved, without needing a utility function.

# Counterfactual sequence prediction

There is a boxed Oracle, $O$, which gets fed a series of observations $o_0, o_1,\ldots o_{n-1}$, and must make a prediction $p_{n}$ about the next observation.

If the prediction was sent out into the world, then $O$ is attempting to make $p_{n}$ into a self-confirming prediction, with all the [problems that this could entail](https://www.lesswrong.com/posts/KoEY9CjrKe93ErYhd/self-confirming-predictions-can-be-arbitrarily-bad).

However, we make $O$ into a counterfactual Oracle; on some occasions, the output $p_n$ is erased, and not seen by anyone. In that case, the Oracle will get a special message $e$ (for "erasure"), that will be added on after of $o_{n-1}$.

Then the job of the counterfactual Oracle is, given a history $h_{n-1} = o_0 p_1 o_1 \ldots p_{n-1}o_{n-1}$ (including past "$e$"s), to produce a prediction $p_n$ that is the best prediction for $o_n$ given the history $h_{n-1} e$. In terms of words, the Oracle is trying to best predict the next observation, **given that erasure is assumed to happen** (the definition of "best" depends on the details of the design).

If our setup is well designed and there is no information leakage, then any $o_n$ has a probability that is independent of the prediction $p_n$:

* $\forall o_n, h_{n-1},p_n, p_n' : P(o_n \mid h_{n-1} e p_n) =P(o_n \mid h_{n-1} e p_n')$.

## Meaning of predictions

If we ran the oracle without the assumption that $e$ would be observed, we'd have the standard self-confirming predictor (that being the only correct predictions); with that assumption, it becomes a counterfactual predictor.

Though counterfactuals are conceptually tricky, it's actually easier to understand what a counterfactual predictor is doing than a general sequence predictor Oracle. We understand that a general sequence predictor is trying to find a self-confirming prediction - one where $p_n$ is the best estimate of $o_n$, given $h_{n-1} p_n$ - but unless we understand the structure of the world very well, we don't have a clue [where that self-confirming prediction might be](https://www.lesswrong.com/posts/KoEY9CjrKe93ErYhd/self-confirming-predictions-can-be-arbitrarily-bad).

In contrast, the counterfactual prediction is easy to interpret: "had we not seen $p_n$, that is what $o_n$ would have been"; there are no loops in the definition.

## Extent of manipulation via prediction

Note that the Oracle can be used to estimate the extent to which the prediction is manipulative, by contrasting its predictions for $o_n$ given the  $h_{n-1}$, and given $h_{n-1} e$. This reveals the extent to which the prediction is potentially manipulative.

## Non-explicit erasure

If the Oracle isn't given an explicit erasure message, but has a causal model of the world, we can still run the counterfactual Oracle by having it choose $p_n$ to be the best fit for the $o_n$, conditional on $h_n p_n$ and on the erasure happening.

# Factoring out certain routes

Assume now that the AI is an active AI, that can take actions in the world, affecting the observation $o_n$. So we'll replace $p_i$ terminology with $a_i$; note that the predictions $p_i$ were already actions, just actions of a particular type in particular setup.

We want to have the AI only consider certain routes to influence $o_n$. This might be because there is a reward-function learning process involved, and we [don't want the AI to influence the learning process](https://www.lesswrong.com/posts/5bd75cc58225bf06703752aa/uninfluenceable-agents). Or maybe there is a corrigibility button involved, and we don't want the AI to [try and ensure it is pressed or not pressed](https://arxiv.org/abs/1712.06365).

In any case, label the banned routes of influence by the variables $b$:

![](https://www.dropbox.com/s/rk6kbvhedjam3ra/causal_graph.png?raw=1)

Note that the Oracle case is just a special case of this, where $b$ corresponds to "humans reading the prediction", and there being no direct arrow from $a_n/p_n$ to $o_n$.

## Factoring out

Assume that $\hat{a}$ is the default action. Then we want the AI to maximise some expected score $S$ of $o_n$ and $a_n$, but acting as if the values of $b$ were set by $\hat{a}$ rather than the value of $a_n$.

Let $\{\beta_i\}$ be the possible values of $b$. We require one more assumption: that if $b=\beta_i$ is possible given $\hat{a}$, then it is also possible given any other action $a$ (technically, we only need for the conditional probabilities $P(o_n \mid b=\beta_i, a_n=a)$ to make sense).

Then if the AI has sufficient data to estimate expressions like $P(b=\beta_i \mid a)$, we act as if the AI was off-policy where $b$ is concerned. Namely, we get it to compute the action $a$ that maximises:

* $\sum_{\beta_i} P(b=\beta_i \mid a_n=\hat{a}, h_{n-1}) \mathbb{E}[S(o_n, a) \mid b=\beta_i, a_n = a, h_{n-1}]$.

Note that if we had $a_n=a$ instead of $a_n=\hat{a}$ in that expression, then this would just be the classical $\mathbb{E}[S(o_n, a) \mid a_n =a, h_{n-1}]$.

This construction is essentially the same as the [counterfactually uninfluenceable agent design](https://www.lesswrong.com/posts/5bd75cc58225bf067037536b/counterfactually-uninfluenceable-agents), with $b$ representing the weights of the different reward functions that the AI was to "learn".