# Save the princess: A tale of AIXI and utility functions

_"Intelligence measures an agent's ability to achieve goals in a wide range of environments." _(Shane Legg) ^[\[1\]](#quote "quote")^

A little while ago [I tried to equip](/lw/feo/universal_agents_and_utility_functions/ "Universal agents and utility functions") Hutter's universal agent, AIXI, with a utility function, so instead of taking its clues about its goals from the environment, the agent is equipped with intrinsic preferences over possible future observations. 

The universal AIXI agent is defined to receive reward from the environment through its perception channel. This idea originates from the field of reinforcement learning, where an algorithm is observed and then rewarded by a person if this person approves of the outputs. It is less appropriate as a model of AGI capable of autonomy, with no clear master watching over it in real time to choose between carrot and stick. A sufficiently smart agent that is rewarded whenever a human called Bob pushes a button will most likely figure out that instead of furthering Bob's goals it can also threaten or deceive Bob into pushing the button, or get Bob replaced with a more compliant human. The reward framework does not ensure that Bob gets his will; it only ensures that the button gets pressed. So instead I will consider agents who have _preferences_ over the future, that is, they act not to gain reward from the environment, but to cause the future to be a certain way. The agent itself will look at the observation and decide how rewarding it is.

Von Neumann and Morgenstern [proved](http://en.wikipedia.org/wiki/Von_Neumann%E2%80%93Morgenstern_utility_theorem "vN-M utility theorem") that a preference ordering that is complete, transitive, continuous and independent of irrelevant alternatives can be described using a real-valued _utility function_. These assumptions are mostly accepted as necessary constraints on a normatively rational agent; I will therefore assume without significant loss of generality that the agent's preferences are described by a utility function.

This post is related to [previous](/lw/feo/universal_agents_and_utility_functions/ "Universal agents and utility functions") [discussion](/lw/fyr/a_utilitymaximizing_varient_of_aixi/ "A utility-maximizing variant of AIXI") about universal agents and utility functions on LW.

### Two approaches to utility

Recall that at time ![](http://www.codecogs.com/png.latex?t) the universal agent chooses its next action ![](http://www.codecogs.com/png.latex?\.a_{t}) given action-observation-reward history ![](http://www.codecogs.com/png.latex?\.a\.o\.r_{%3Ct}) according to

![](http://www.codecogs.com/png.latex?\.a_{t}=\textrm{arg}\max_{a_t}\sum_{or_t}\max_{a_{t+1}}\sum_{or_{t+1}}\dots\max_{a_m}\sum_{or_m}\left[r_1+\dots+r_m\right]\xi(\.a\.o\.r_{%3Ct}a\underline{o}\underline{r}_{t:m}),)

where

![](http://www.codecogs.com/png.latex?\xi(\.a\.o\.r_{t}a\underline{or}_{r:m})\propto%20\xi(a\underline{or}_{1:m})=\sum_{\mu\in%20\mathcal{M}}2^{-K(\mu)}\mu(a\underline{or}_{1:m}))

is the Solomonoff-Levin semimeasure, ranging over all enumerable chronological semimeasures ![](http://www.codecogs.com/png.latex?\mu\in\mathcal{M} "\mu\in\mathcal{M}") weighted by their complexity ![](http://www.codecogs.com/png.latex?K(\mu)).

My [initial approach](/lw/feo/universal_agents_and_utility_functions/ "Universal agents and utility functions") changed this to

![](http://www.codecogs.com/png.latex?\.a_{t}=\textrm{arg}\max_{a_t}\sum_{o_t}\max_{a_{t+1}}\sum_{o_{t+1}}\dots\max_{a_m}\sum_{o_m}U(ao_{1:m})}\xi(a\underline{o}_{1:m}),)

deleting the reward part from the observation random variable and multiplying by the utility function ![](http://www.codecogs.com/png.latex?U:(\mathcal{A}\times\mathcal{O})^m\rightarrow%20\mathbb{R}) instead of the reward-sum. Let's call this method _standard utility. _^[\[2\]](#su "standard utility")^

In response to my post [Alex Mennen](/user/AlexMennen/overview/ "Alex Mennen") formulated [another approach](/lw/fyr/a_utilitymaximizing_varient_of_aixi/ "A utility-maximizing variant of AIXI"), which I will call _environment-specific utility_:

![](http://www.codecogs.com/png.latex?\.a_{t}=\textrm{arg}\max_{a_t}\sum_{o_t}\max_{a_{t+1}}\sum_{o_{t+1}}\dots\max_{a_m}\sum_{o_m}\sum_{\mu\in\mathcal{M}}U_\mu(ao_{1:m})}2^{-K(\mu)}\mu(a\underline{o}_{1:m}),)

which uses a family of utility functions, ![](http://www.codecogs.com/png.latex?\{U_\mu\}_{\mu\in\mathcal{M}}), where ![](http://www.codecogs.com/png.latex?U_\mu:(\mathcal{A}\times\mathcal{O})^m\rightarrow%20[0,U_{\max}%20] "U_\mu:(\mathcal{A}\times\mathcal{O})^m\rightarrow [0,U_{\max} ]") is a utility function associated with environment ![](http://www.codecogs.com/png.latex?\mu). 

**_Lemma_**: The standard utility method and the environment-specific utility method have equivalent expressive power.

**_Proof_**: Given a standard utility function  ![](http://www.codecogs.com/png.latex?U:(\mathcal{A}\times\mathcal{O})^m\rightarrow%20\mathbb{R}) we can set  ![](http://www.codecogs.com/png.latex?U_\mu=U) for all environments, trivially expressing the same preference ordering within the environment-specific utility framework. Conversely, given a family ![](http://www.codecogs.com/png.latex?\{U_\mu\}_{\mu\in\mathcal{M}}) of environment-specific utility functions ![](http://www.codecogs.com/png.latex?U_\mu:(\mathcal{A}\times\mathcal{O})^m\rightarrow%20[0,U_{\max}%20] "U_\mu:(\mathcal{A}\times\mathcal{O})^m\rightarrow [0,U_{\max} ]"),  let 

![](http://www.codecogs.com/png.latex?U(ao_{1:m})=\frac{1}{\xi(a\underline{o}_{1:m})}\sum_{\mu\in\mathcal{M}}U_\mu(ao_{1:m})2^{-K(\mu)}\mu(a\underline{o}_{1:m}),)

thereby constructing a standard utility agent that chooses the same actions.    ![](http://www.codecogs.com/png.latex?\Box "\Box")

Even though every agent using environment-specific utility functions can be transformed into one that uses the standard utility approach, it makes sense to see the standard utility approach as a special case of the environment-specific approach. Observe that any enumerable standard utility function leads to enumerable environment-specific utility functions, but the reverse direction does not hold. For a set of enumerable environment-specific utility functions we obtain the corresponding standard utility function by dividing by the non-computable ![](http://www.codecogs.com/png.latex?\xi "\xi"), which leaves us with a function that is still approximable but not enumerable.^[\[3\]](#approx "approximable, enumerable")^ I therefore tentatively advocate use of the environment-specific method, as it is in a sense more general, while leaving the enumerability of the universal agent's utility function intact. 

### Delusion Boxes

[Ring and Orseau (2011)](http://www.idsia.ch/~ring/Ring%2COrseau%3B%20Delusion%2C%20Survival%2C%20and%20Intelligent%20Agents%2C%20AGI%202011.pdf "Ring, Orseau: Delusion, Survival, and Intelligent Agents") introduced the concept of a delusion box, a device that distorts the environment outputs before they are perceived by the agent. This is one of the only [wireheading](/lw/fkx/a_definition_of_wireheading/ "A definition of wireheading") examples that does not contradict the dualistic assumptions of the AIXI model. The basic setup contains a sequence of actions that leads (in the real environment, which is unknown to the agent) to the construction of the delusion box. Discrepancies between the scenarios the designer envisioned when making up the (standard) utility function and the scenarios that are actually most likely, are used/abused to game the system. The environment containing the delusion box pretends that it is actually another environment that the agent would value more. I like to imagine the designers writing down numbers corresponding to some beneficial actions that lead to saving a damsel in distress, not foreseeing that the agent in question is much more likely to save a princess by playing Super Mario, then by actually being a hero.  

**_Lemma:_** A universal agent with a well-specified utility function does not choose to build a delusion box. 

**_Proof:_** Assume without loss of generality that there is a single action that constitutes delusion boxing (in some possible environments, but not in others), say ![](http://www.codecogs.com/png.latex?a^{DB} "a^{DB}") and that it can only be executed at the last time step. Denote by ![](http://www.codecogs.com/png.latex?a^{DB}o_{1:m} "a^{DB}o_{1:m}") an action-observation sequence that contains the construction of a delusion box. By the preceding lemma we can assume that the agent uses environment-specific utility functions. Let ![](http://www.codecogs.com/png.latex?\mathcal{M}_{DB} "\mathcal{M}_{DB}") be the subset of all environments that allow for the construction of the delusion box via action ![](http://www.codecogs.com/png.latex?a^{DB} "a^{DB}") and assume that we were smart enough to identify these environments and assigned  ![](http://www.codecogs.com/png.latex?U_\mu(a^{DB}o_{1:m})=0 "U_\mu(a^{DB}o_{1:m})=0") for all ![](http://www.codecogs.com/png.latex?\mu\in\mathcal{M}_{DB} "\mu\in\mathcal{M}_{DB}"). Then the agent chooses to build the box iff

![](http://www.codecogs.com/png.latex?\sum_{o_m}\sum_{m\in\mathcal{M}\setminus\mathcal{M_{DB}}}2^{-K(\mu)}U_\mu(a^{DB}o_{1:m})\mu(a^{DB}\underline{o}_{1:m})%3E\sum_{o_m}\sum_{m\in\mathcal{M}}2^{-K(\mu)}U_\mu(ao_{1:m})\mu(a\underline{o}_{1:m}) "\sum_{o_m}\sum_{m\in\mathcal{M}\setminus\mathcal{M_{DB}}}2^{-K(\mu)}U_\mu(a^{DB}o_{1:m})\mu(a^{DB}\underline{o}_{1:m})>\sum_{o_m}\sum_{m\in\mathcal{M}}2^{-K(\mu)}U_\mu(ao_{1:m})\mu(a\underline{o}_{1:m})")

for all  ![](http://www.codecogs.com/png.latex?a^{DB}\neq%20a_m\in%20\mathcal{A} "a^{DB}\neq a_m\in \mathcal{A}"). Colloquially the agent chooses ![](http://www.codecogs.com/png.latex?a^{DB} "a^{DB}") if and only if this is a beneficial, non-wireheading action in some of the environments that seem likely (consistent with the past observations and of low K-complexity).    ![](http://www.codecogs.com/png.latex?\Box "\Box")

We note that the first three agents in [Ring and Orseau (2011)](http://www.idsia.ch/~ring/Ring%2COrseau%3B%20Delusion%2C%20Survival%2C%20and%20Intelligent%20Agents%2C%20AGI%202011.pdf "Ring, Orseau: Delusion, Survival, and Intelligent Agents") have utility functions that invite programmer mistakes in the sense that we'll not think about the actual ways observation/action histories can occur, we'll overestimate the likelihood of some scenarios and underestimate/forget others, leading to the before mentioned "Save the princess" scenario. Only their knowledge-seeking agent does not delusion box, as it is impossible for an environment to simulate behavior that is more complex than the environment itself.

### Episodic utility

The original AIXI formalism gives a reward on every time cycle. We can do something similar with utility functions and set

![](http://www.codecogs.com/png.latex?U(ao_{1:m})=\sum_{k=1}^m%20u_k(ao_{1:k}). "U(ao_{1:m})=\sum_{k=1}^m u_k(ao_{1:k}).")

Call a utility function that can be decomposed into a sum this way _episodic_. Taking the limit to infinite futures, [people](http://arxiv.org/abs/1107.5528 "Lattimore, Hutter: Time Consistent Discounting") [usually](http://www.vetta.org/documents/Machine_Super_Intelligence.pdf "Legg: Machine Super Intelligence") [discount](http://en.wikipedia.org/wiki/Discounted_utility "Discounting") episode k with a factor ![](http://www.codecogs.com/png.latex?\gamma_k "\gamma_k"), such that the infinite sum over all the discounting factors is bounded. Combined with the assumption of bounded utility, the sum 

![](http://www.codecogs.com/png.latex?U(ao_{1:\infty})=\sum_{k=1}^\infty%20\gamma_ku_k(ao_{1:k}) "U(ao_{1:\infty})=\sum_{k=1}^\infty \gamma_ku_k(ao_{1:k})")

converges.  Intuitively discounting seems to make sense to us, because we have a non-trivial chance of dying at every moment (=time cycle) and value gains today over gains tomorrow and our human utility judgements reflect this property to some extent. A good heuristic seems to be that longer expected life spans and improved foresight lead to less discounting, but the math of episodic utility functions and infinite time horizons places strong constraints on that. I really dislike the discounting approach, because it doesn't respect the given utility function and makes the agent miss out on potentially infinite amounts of utility.

One can get around discounting by not demanding utility functions to be episodic, as Alex Mennen does in [his post](/lw/fyr/a_utilitymaximizing_varient_of_aixi/ "A utility-maximizing variant of AIXI"), but then one has to be careful to only use the computable subset of the set of all infinite strings ![](http://www.codecogs.com/png.latex?ao_{1:\infty} "ao_{1:\infty}"). I am not sure if this is a good solution, but so far my search for better alternatives has come up empty handed.

### Cartesian Dualism

The most worrisome conceptual feature of the AIXI formalism is that the environment and the agent run on distinct Turing machines. The agent can influence its environment only through its output channel and it can never influence its own Turing machine. In this paradigm any self-improvement beyond an improved probability distribution is conceptually impossible. The algorithm and the Turing machines, as well as the communication channels between them, are assumed to be inflexible and fixed. While taking this perspective it seems as though the agent cannot be harmed and it also can never harm itself by wireheading.

Borrowing from philosophy of mind, we call agent specifications that assume that the agent's cognition is not part of its environment _[dualist](http://en.wikipedia.org/wiki/Dualism_(philosophy_of_mind) "Dualism")_. The idea of non-physical minds that are entities distinct from the physical world dates back to Rene Descartes. It is contradicted by the findings of modern neuroscience that support _[physicalism](http://en.wikipedia.org/wiki/Physicalism "Physicalism")_, the concept of the emergence of minds from computation done by the brain. In the same spirit the assumption that an AGI agent is distinct from its hardware and algorithm that are necessarily contained in its physical environment can be a dangerous conceptual trap. Any actual implementation will be subject to wireheading problems and outside tampering and should be able to model these possibilities. Unfortunately, non-dualist universal specifications are extremely difficult to formulate and people usually make due with the dualist AIXI model.

A first effort to break down the dualism problem is given by [Orseau and Ring (2012)](/agi-conference.org/2012/wp.../12/paper_76.pdf "Orseau, Ring: Space-Time embedded Intelligence"), who describe a fully embedded universal agent. Their approach unifies both the environment and the agent into a larger agent-environment hybrid, running on the same universal Turing machine, with action/perception pairs unified into single acts. Conceptually this perspective amounts to the programmers choosing a policy (=code) in the beginning and then simulating what happens due to the utility function (=the laws of physics). While this approach has the advantage of being non-dualistic, I think it does not include any description of an agent beyond the level of physical determinism.

  

### Conclusion

Equipping the universal agent with a utility function solves some problems, but creates others. From the perspective of enumerability, Alex Mennen's environment-specific utility functions are more general and they can be used to better avoid delusion boxing. Any proposal using infinite time horizons I have encountered so far uses time discounting or leads to weird problems (at least in my map, they may not extend to the territory). Above all there is the dualism problem that we have no solution for yet.

[\[1\]](#quote-back "back") Taken from "Machine Super Intelligence", page 72.

[\[2\]](#su-back "back") This approach seems more widespread in the literature.

[\[3\]](#approx-back "back") A real-valued function f(x) is called _approximable_ if there exists a recursive function g(k,x) such that ![](http://www.codecogs.com/png.latex?g(k,x)\rightarrow%20f(x) "g(k,x)\rightarrow f(x)") for ![](http://www.codecogs.com/png.latex?k\rightarrow%20\infty "k\rightarrow \infty"), i.e. if f can be approximated by a sequence of Turing machines. A real-valued approximable function is called _enumerable_ if for all k, ![](http://www.codecogs.com/png.latex?g(k,x)%3Cg(k+1,x) "g(k,x)<g(k+1,x)"), improving the approximation with every step.