# Why we want unbiased learning processes

**tl;dr**: if an agent has a biased learning process, it may choose actions that are worse (with certainty) for every possible reward function it could be learning.

An agent learns its own reward function if there is a set  R of possible reward functions, and there is a learning process  P that maps world-histories (and policies) to distributions over  R. Thus by interacting with the environment and choosing its own policies, the agent can learn which is the correct reward function it should be maximising.

Given a policy  π, a history  h, an environment  μ,and a reward  R, we can compute the expected probability of  R:

 EμπP(R|h).

Then a learning process is **unbiased** if that expression is independent of  π, and **biased** otherwise. Biased processes are less desirable, as they allow the agent to manipulate the process through its choice of policy.

Simple biased learning process
------------------------------

The most trivial example of a biased learning process is an agent that completely determines its reward by its actions.

Let  R={R0,R1}, let the agent only act once with two actions available,  {a0,a1}, (hence a choice of "policy" is a choice of action), and set

 P(R0|a0)=P(R1|a1)=1.

Thus the agent can simply choose its reward function through its actions.

Note that some designs are a bit more sophisticated, and don't allow the agent to choose its reward function directly through its actions. But this doesn't matter, if the reward function is a consequence of anything that is a predictable consequence of the agent's actions (eg if the agent can trick/coerce/manipulate a human into saying "yes" or "no", and if < refresh to render LaTeX > is determined by the human's response, it doesn't matter that < refresh to render LaTeX > is not defined directly through the agent's actions: it is defined indirectly through them).

\[_Note that all < refresh to render LaTeX > that involve learning about external facts are [unbiased learning processes](https://wiki.lesswrong.com/wiki/Conservation_of_expected_evidence), so it's not as if unbiased means trivial_\]

Strictly dominated behaviour
----------------------------

Then an agent with a biased learning process that wants to maximise the expectation of the true reward, can sometimes follow strictly dominated policies.

That means that there is are policies < refresh to render LaTeX > and < refresh to render LaTeX > , such that for all histories < refresh to render LaTeX > possible given < refresh to render LaTeX > , and all reward function < refresh to render LaTeX > in < refresh to render LaTeX > ,

<refresh to render LaTeX>

And yet the agent will still choose < refresh to render LaTeX > to maximise reward.

For example, with < refresh to render LaTeX > and < refresh to render LaTeX > defined as above, define < refresh to render LaTeX > and < refresh to render LaTeX > to be:

<refresh to render LaTeX>

Thus < refresh to render LaTeX > is always the better action, for both < refresh to render LaTeX > and for < refresh to render LaTeX > ; it is strictly dominant. However, since < refresh to render LaTeX > also determines which reward function is correct, the possible rewards the agent gets are the two bold numbers in the table: < refresh to render LaTeX > , by choosing < refresh to render LaTeX > and hence making < refresh to render LaTeX > the correct reward function, and < refresh to render LaTeX > , by choosing < refresh to render LaTeX > and hence making < refresh to render LaTeX > the correct reward function.

Then in order to maximise reward, the agent will choose the strictly dominated policy/action < refresh to render LaTeX > .

Unbiased learning
-----------------

It's possible to prove that if < refresh to render LaTeX > is unbiased, then this behaviour won't occur, but doing so involves introducing a bit more definition and machinery that presented here, so I'll defer this to my forthcoming paper.

Note on _expected_ dominance
----------------------------

\[_Reading the following is not relevant to understanding the main point of this post_\]

The dominant policy is defined so that for all < refresh to render LaTeX > , < refresh to render LaTeX > for all histories < refresh to render LaTeX > , possible given < refresh to render LaTeX > .

We could instead talk about the expected reward given < refresh to render LaTeX > . But in fact, it makes sense to choose policies which are strictly dominated in the expected reward sense.

For example, let < refresh to render LaTeX > be a policy that does nothing (all rewards stay at < refresh to render LaTeX > ), and let < refresh to render LaTeX > be the policy that first checks which of < refresh to render LaTeX > and < refresh to render LaTeX > is correct (for a given < refresh to render LaTeX > ) and then maximises the correct one. If < refresh to render LaTeX > is maximised, it goes to < refresh to render LaTeX > , while the other reward will go to < refresh to render LaTeX > .

Assume that the probability of either < refresh to render LaTeX > or < refresh to render LaTeX > being correct is < refresh to render LaTeX > . Then it's clear that < refresh to render LaTeX > is dominated in expectation by < refresh to render LaTeX > , since

<refresh to render LaTeX>

Yet < refresh to render LaTeX > is clearly the right thing to do, since it allows us to maximise the correct reward ( < refresh to render LaTeX > is only negative in worlds where it is not the correct reward).

So an unbiased agent can still choose a policy that is worse for every reward in expectation, if it's confident that the (currently unknown) correct reward will get maximised more by this policy.