# Probabilities, weights, sums: pretty much the same for reward functions

*This post is a more minor post, that I'm putting up to reference in other posts.*

# Probabilities, weights, and expectations

You're an agent, with potential uncertainty over your reward function. You know you have to maximise

$$0.5R_1 + 0.5R_2$$

where $R_1$ and $R_2$ are reward functions. What do you do?

Well, how do we interpret the $0.5$? Are they probabilities for which reward function is right? Or are they weights, telling you the relative importance of each one? Well, in fact:

* If you won't be learning any more information to help you distinguish between reward functions, then weights and probabilities play the same role.

Thus, if you don't expect to learn any more reward function-relevant information, maximising reward given $P(R_1)=P(R_2)=0.5$ is the same as maximising the single reward function $R_3=0.5R_1 + 0.5R_2$.

So, if we denote probabilities with in bold, maximising the following (given no reward-function learning) are all equivalent:

$$
\begin{array}{c}
\mathbf{0.5}R_1 + \mathbf{0.5}R_2 \\
\mathbf{1}\left(0.5R_1 + 0.5R_2\right)\\
\mathbf{0.25}R_1 + \mathbf{0.25}R_2 + \mathbf{0.5}\left( 0.5R_1 + 0.5R_2\right) \\
\mathbf{0.5}\left( 1.5R_1 - 0.5R_2\right) + \mathbf{0.5}\left( 1.5R_2 - 0.5R_1\right)
\end{array}
$$

Now, given a probability distribution $p_R$ over reward functions, we can take its expectation $\mathbb{E}(p_R)$. You can define this by talking about affine spaces and so on, but the simple version of it is: **to take an expectation, rewrite every probability as a weight**. So the result becomes:

* If you won't be learning any more information to help you distinguish between reward functions, then distributions with same expectation are equivalent.

# Expected evidence and unriggability

We've [defined an unriggable learning process](https://www.lesswrong.com/posts/LpjjWDBXr88gzcYK2/learning-and-manipulating-learning) as one that respects [conservation of expected evidence](https://www.lesswrong.com/posts/LpjjWDBXr88gzcYK2/learning-and-manipulating-learning).

Now, conservation of expected evidence is about expectations. It basically says that, if $\pi_1$ and $\pi_2$ are two policies the agent could take, then for the probability distribution $p_R$,

$$\mathbb{E}(p_R\ \mid \pi_1) = \mathbb{E}(p_R\ \mid \pi_2).$$

Suppose that $p_R$ is in fact riggable, and that we wanted to "correct" it to make it unriggable. Then we would want to add a correction term for any policy $\pi$. If we took $\pi_0$ as a "default" policy, we could add a correction term to $p_R \mid \pi$:

$$ (p_R \mid \pi) \to (p_R \mid \pi) - \mathbb{E}(p_R\mid \pi) + \mathbb{E}(p_R\mid \pi_0).$$

This would have the required unriggability properties. But how do you add to a probability distribution - and how do you subtract from it?

Bur recall that unriggability only cares about expectations, and expectations treat probabilities as weights. Adding weighted reward functions is perfectly fine. Generally there will be multiple ways of doing this, mixing probabilities and weights.

For example, if $(p_R \mid \pi) = \mathbf{0.5}R_1 + \mathbf{0.5}R_2$ and $(p_R \mid \pi_0) = \mathbf{0.75}(R_1-R_2) + \mathbf{0.25}R_2$, then we can map $(p_R \mid \pi)$ to

1. $\mathbf{1}\left(0.75 R_1 - 0.5 R_2\right)$,
2.  $\mathbf{0.75}(R_1-R_2) + \mathbf{0.25}R_2$,
3. $\mathbf{0.5}(R_1+R) + \mathbf{0.5}(R_2 + R)$ with $R=0.25 R_1 - R_2$,
4. $\mathbf{0.75}R_1 +  \mathbf{0.25}(-2R_2)$,
5. and many other options...

This multiplicity of possibilities is what I was trying to deal with in my [old post about reward function translations](https://www.alignmentforum.org/posts/5bd75cc58225bf0670375370/translation-counterfactual).