# "Normative assumptions" need not be complex

I've [shown](https://arxiv.org/abs/1712.05812) that, even with simplicity priors, we can't figure out the preferences or rationality of a potentially irrational agent (such as a human $H$).

But we can get around that issue with '[normative assumptions](https://www.lesswrong.com/posts/GTcwKAZ8kdYxyS5G3/normative-assumptions-answers-emotions-and-narratives)'. These can allow us to zero in on a 'reasonable' reward function $R_H$.

We should however note that:

* Even if $R_H$ is highly complex, a normative assumption need not be complex to single it out.

This post gives an example of that for general agents, and discusses how a similar idea might apply to the human situation.

# Formalism

An agent takes actions ($\mathcal{A}$) and gets observations ($\mathcal{O}$), and together these form histories, with $\mathcal{H}$ the set of histories (I won't present all the details of the formalism here). The policies $\Pi=\{\pi:\mathcal{H}\to\mathcal{A}\}$ are maps from histories to actions. The reward functions $\mathcal{R}=\{R:\mathcal{H}\to\mathbb{R}\}$  are maps from histories $\mathcal{H}$ to real numbers), and the planners $\mathcal{P}=\{p:\mathcal{R}\to\Pi\}$ are maps from reward functions to policies.

By observing an agent, we can deduce (part of) their policy $\pi$. Then a reward-planner pair $(p,R)$ is *compatible* with $\pi$ if $p(R)=\pi$. Further observations cannot distinguish between different compatible pairs.

Then a normative assumption $\alpha$ is something that distinguishes between compatible pairs. It could be a prior on $\mathcal{P}\times\mathcal{R}$, or an assumption of full rationality (which removes all-but-the-rational planner from $\mathcal{P}$), or something that takes in more details about the agent or the situation.


# Assumptions that use a lot of information

Assume that the agent's algorithm $\pi$ is written in some code, as $C_\pi$, and that $\alpha$ will have access to this. Then suppose that $\alpha$ scans $C_\pi$, looking for the following: an object $C_R$ that takes a history as an input and has a real number as an output, an object $C_p$ that takes $C_R$ and a history as inputs, and outputs an action, and a guarantee that $C_\pi$ chooses actions by running $C_p$ on $C_R$ and the input history.

The $\alpha$ need not be very complex to do that job. Because of [rice's theorem](https://en.wikipedia.org/wiki/Rice%27s_theorem) and obfuscated code, it will be impossible for $\alpha$ to check those facts in general. But, for many examples of $C_\pi$, it will be able to check that those things hold. In that case, let $\alpha$ return $R$; otherwise, let it return the trivial $0$ reward.

So, for a large set $S$ of possible algorithms, $\alpha$ can return a reasonable reward function estimate. Even if the complexity of $C_\pi$ and $R$ is much, much higher than the complexity of $\alpha$ itself, there are still examples of these where $\alpha$ can successfully identity the reward function.

Of course, if we run $\alpha$ on a human brain, it would return $0$. But what I am looking for is not $\alpha$, but a more complicate $\alpha_H$, that, when run on the set $S_H$ of human agents, will extract some 'reasonable' $R_H$. It doesn't matter what $\alpha_H$ does when run on non-human agents, so we can load it with assumptions about how humans work. When I talk about [extracting preferences through looking at internal models](https://www.lesswrong.com/posts/rcXaY3FgoobMkH2jc/figuring-out-what-alice-wants-part-ii), this is the kind of thing I had in mind (along with some method for [synthesising those preferences](https://www.lesswrong.com/posts/Y2LhX3925RodndwpC/resolving-human-values-completely-and-adequately) into a coherent whole).

So, though my desired $\alpha_H$ might be complex, there is no a priori reason to think that it need be as complex as the $R_H$ output.