Abstract: An important use of machine learning is to learn what people value. What posts or photos should a user be shown? Which jobs or activities would a person find rewarding? In each case, observations of people's past choices can inform our inferences about their likes and preferences. If we assume that choices are approximately optimal according to some utility function, we can treat preference inference as Bayesian inverse planning. That is, given a prior on utility functions and some observed choices, we invert an optimal decision-making process to infer a posterior distribution on utility functions. However, people often deviate from approximate optimality. They have false beliefs, their planning is sub-optimal, and their choices may be temporally inconsistent due to hyperbolic discounting and other biases. We demonstrate how to incorporate these deviations into algorithms for preference inference by constructing generative models of planning for agents who are subject to false beliefs and time inconsistency. We explore the inferences these models make about preferences, beliefs, and biases. We present a behavioral experiment in which human subjects perform preference inference given the same observations of choices as our model. Results show that human subjects (like our model) explain choices in terms of systematic deviations from optimal behavior and suggest that they take such deviations into account when inferring preferences.

Owain Evans
Andreas Stuhlm
Noah D Goodman

Learning the Preferences of Ignorant, Inconsistent Agents

Introduction The problem of learning a person's preferences from observations of their choices features prominently in economics  (Hausman 2011) , in cognitive science  (Baker, Saxe, and Tenenbaum 2011; Ullman et al. 2009) , and in applied machine learning  (Jannach et al. 2010; Ermon et al. 2014) . To name just one example, social networking sites use a person's past behavior to select what stories, advertisements, and potential contacts to display to them. A promising approach to learning preferences from observed choices is to Copyright c 2016, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. invert a model of rational choice based on sequential decision making given a real-valued utility function  (Russell and Norvig 1995) . This approach is known as Inverse Reinforcement Learning  (Ng and Russell 2000)  in an RL setting and as Bayesian Inverse Planning  (Baker, Saxe, and Tenenbaum 2009)  in the setting of probabilistic generative models. This kind of approach usually assumes that the agent makes optimal decisions up to "random noise" in action selection  (Kim et al. 2014; Zheng, Liu, and Ni 2014) . However, human deviations from optimality are more systematic. They result from persistent false beliefs, sub-optimal planning, and from biases such as time inconsistency and framing effects  (Kahneman and Tversky 1979) . If such deviations are modeled as unstructured errors, we risk mistaken preference inferences. For instance, if an agent repeatedly fails to choose a preferred option due to a systematic bias, we might conclude that the option is not preferred after all. Consider someone who smokes every day while wishing to quit and viewing their actions as regrettable. In this situation, a model that has good predictive performance might nonetheless fail to identify what this person values. In this paper, we explicitly take into account structured deviations from optimality when inferring preferences. We construct a model of sequential planning for agents with inaccurate beliefs and time-inconsistent biases (in the form of hyperbolic discounting). We then do Bayesian inference over this model to jointly infer an agent's preferences, beliefs and biases from sequences of actions in a simple Gridworld-style domain. To demonstrate that this algorithm supports accurate preference inferences, we first exhibit a few simple cases where our model licenses conclusions that differ from standard approaches, and argue that they are intuitively plausible. We then test this intuition by asking impartial human subjects to make preference inferences given the same data as our algorithm. This is based on the assumption that people have expertise in inferring the preferences of others when the domain is simple and familiar from everyday experience. We find that our algorithm is able to make the same kinds of inferences as our human judges: variations in choice are explained as being due to systematic factors such as false beliefs and strong temptations, not unexplainable error. The possibility of false beliefs and cognitive biases means that observing only a few actions often fails to identify a single set of preferences. We show that humans recognize this ambiguity and provide a range of distinct explanations for the observed actions. When preferences can't be identified uniquely, our model is still able to capture the range of explanations that humans offer. Moreover, by computing a Bayesian posterior over possible explanations, we can predict the plausibility of explanations for human subjects.
 Computational Framework Our goal is to infer an agent's preferences from observations of their choices in sequential decision problems. The key question for this project is: how are our observations of behavior related to the agent's preferences? In more technical terms, what generative model  (Tenenbaum et al. 2011)  best describes the agent's approximate sequential planning given some utility function? Given such a model and a prior on utility functions, we could "invert" it (by performing full Bayesian inference) to compute a posterior on what the agent values. The following section describes the class of models we explore in this paper. We first take an informal look at the specific deviations from optimality that our agent model includes. We then define the model formally and show our implementation as a probabilistic program, an approach that clarifies our assumptions and enables easy exploration of deviations from optimal planning.
 Deviations from optimality We consider two kinds of deviations from optimality: False beliefs and uncertainty Agents can have false or inaccurate beliefs. We represent beliefs as probability distributions over states and model belief updates as Bayesian inference. Planning for such agents has been studied in work on POMDPs  (Kaelbling, Littman, and Cassandra 1998) . Inferring the preferences of such agents was studied in recent work  (Baker and Tenenbaum 2014; Panella and Gmytrasiewicz 2014) . Here, we are primarily interested in the interaction of false beliefs with other kinds of sub-optimality. Temporal inconsistency Agents can be time-inconsistent (also called "dynamically inconsistent"). Time-inconsistent agents make plans that they later abandon. This concept has been used to explain human behaviors such as procrastination, temptation and pre-commitment  (Ainslie 2001) , and has been studied extensively in psychology  (Ainslie 2001)  and in economics  (Laibson 1997; O'Donoghue and Rabin 2000) . A prominent formal model of human time inconsistency is the model of hyperbolic discounting  (Ainslie 2001) . This model holds that the utility or reward of future outcomes is discounted relative to present outcomes according to a hyperbolic curve. For example, the discount for an outcome occurring at delay d from the present might be modeled as a multiplicative factor 1 1+d . The shape of the hyperbola means that the agent takes $100 now over $110 tomorrow, but would prefer to take $110 after 31 days to $100 after 30 days. The inconsistency shows when the 30th day comes around: now, the agent switches to preferring to take the $100 immediately. This discounting model does not (on its own) determine how an agent plans sequentially. We consider two kinds of time-inconsistent agents. These agents differ in terms of whether they accurately model their future choices when they construct plans. First, a Sophisticated agent has a fully accurate model of its own future decisions. Second, a Naive agent models its future self as assigning the same (discounted) values to options as its present self. The Naive agent fails to accurately model its own time inconsistency.  1  We illustrate Naive and Sophisticated agents with a decision problem that we later re-use in our experiments. The problem is a variant of Gridworld where an agent moves around the grid to find a place to eat (Figure  1 ). In the left pane (Figure  1a ), we see the path of an agent, Alice, who moves along the shortest path to the Vegetarian Cafe before going left and ending up eating at Donut Store D2. This behavior is sub-optimal independent of whether her preference is for the Vegetarian Cafe or the Donut Store, but can be explained in terms of Naive time-inconsistent planning. From her starting point, Alice prefers to head for the Vegetarian Cafe (as it has a higher undiscounted utility than the Donut Store). She does not predict that when close to the Donut Store (D2), she will prefer to stop there due to hyperbolic discounting. The right pane (Figure  1b ) shows what Beth, a Sophisticated agent with similar preferences to Alice, would do in the same situation. Beth predicts that, if she took Alice's route, she would end up at the Donut Store D2. So she instead takes a longer route in order to avoid temptation. If the longer route wasn't available, Beth could not get to the Vegetarian Cafe without passing the Donut Store D2. In this case, Beth would either go directly to Donut Store D1, which is slightly closer than D2 to her starting point, or (if utility for the Vegetarian Cafe is sufficiently high) she would correctly predict that she will be able to resist the temptation.
 Formal model definition We first define an agent with full knowledge and no time inconsistency, 2 and then generalize to agents that deviate from optimality. We will refer to states s ∈ S, actions a ∈ A, a deterministic utility function U : S ×A → R, a stochastic action choice function C : S → A, and a stochastic state transition function T : S × A → S. To refer to the probability that C(s) returns a, we use C(a; s). Optimal agent: full knowledge, no discounting Like all agents we consider, this agent chooses actions in proportion to exponentiated expected utility (softmax):  [a]  The noise parameter α modulates between random choice (α = 0) and perfect maximization (α = ∞). Expected utility depends on both current and future utility: C(a; s) ∝ e αEUs EU s [a] = U (s, a) + E s ,a [EU s [a ]] with s ∼ T (s, a) and a ∼ C(s ). Note that expected future utility recursively depends on C-that is, on what the agent assumes about how it will make future choices. Figure  2 : We specify agents' decision-making processes as probabilistic programs. This makes it easy to encode arbitrary biases and decision-making constraints. When automated inference procedures invert such programs to infer utilities from choices, these constraints are automatically taken into account. Note the mutual recursion between agent and expUtility: the agent's reasoning about future expected utility includes a (potentially biased) model of its own decision-making.
 Time-inconsistent agent To compute expected utility, we additionally take the expectation over states. Now EU p(s),o,d [a] is defined as: Inferring preferences We define a space of possible agents based on the dimensions described above (utility function U , prior p(s), discount parameter k, noise parameter α). We additionally let Y be a variable for the agent's type, which fixes whether the agent discounts at all, and if so, whether the agent is Naive or Sophisticated. So, an agent is defined by a tuple θ := (p(s), U, Y, k, α), and we perform inference over this space given observed actions. The posterior joint distribution on agents conditioned on action sequence a 0:T is: E s∼p(s|o) 1 1 + kd U (s, a) + E s , P (θ|a 0:T ) ∝ P (a 0:T |θ)P (θ) (1) The likelihood function P (a 0:T |θ) is given by the multistep generalization of the choice function C corresponding to θ. For the prior P (θ), we use independent uniform priors on bounded intervals for each of the components. In the following, "the model" refers to the generative process that
 Sophisticated Figure  3 : Given data corresponding to Figure  1 , the model infers a joint posterior distribution on preferences, beliefs and other agent properties (such as discount strength) that reveals relations between different possible inferences from the data. The darker a cell, the higher its posterior probability. involves a prior on agents and a likelihood for choices given an agent.
 Agents as probabilistic programs We implemented the model described above in the probabilistic programming language WebPPL (Goodman and Stuhlmüller 2014). WebPPL provides automated inference over functional programs that involve recursion. This means that we can directly translate the recursions above into programs that represent an agent and the world simulation used for expected utility calculations. All of the agents above can be captured in a succinct functional program that can easily be extended to capture other kinds of sub-optimal planning. Figure  2  shows a simplified example (including hyperbolic discounting but not uncertainty over state). For the Bayesian inference corresponding to Equation 1 we use a discrete grid approximation for the continuous variables (i.e. for U , p(s), k and α) and perform exact inference using enumeration with dynamic programming.
 Model inferences We now demonstrate that the model described above can infer preferences, false beliefs and time inconsistency jointly from simple action sequences similar to those that occur frequently in daily life. We later validate this intuition in our experiments, where we show that human subjects make inferences about the agent that are similar to those of our model. Example 1: Inference with full knowledge We have previously seen how modeling agents as Naive and Sophisticated might predict the action sequences shown in Figures  1a and 1b  respectively. We now consider the inference problem. Given that these sequences are observed, what can be inferred about the agent? We assume for now that the agent has accurate beliefs about the restaurants and that the two Donut Stores D1 and D2 are identical (with D1 closer to the starting point).  4  We model each restaurant as having an immediate utility (received on arriving at the restaurant) and a delayed utility (received one time-step after). This interacts with hyperbolic discounting, allowing the model to represent options that are especially "tempting" when they can be obtained with a short delay. For the Naive episode (Figure  1a ) our model infers that either softmax noise is very high or that the agent is Naive (as explained for Alice above). If the agent is Naive, the utility of the Vegetarian Cafe must be higher than the Donut Store (otherwise, the agent wouldn't have attempted to go to the Cafe), but not too much higher (or the agent wouldn't give in to temptation, which it in fact does). This relationship is exhibited in Figure  3   Example 2: Inference with uncertainty In realistic settings, people do not have full knowledge of all facts relevant to their choices. Moreover, an algorithm inferring preferences will itself be uncertain about the agent's uncertainty. What can the model infer if it doesn't assume that the agent has full knowledge? Consider the Sophisticated episode (Figure  1b ). Suppose that the Noodle Shop is closed, and that the agent may or may not know about this. This creates another possible inference, in addition to Sophisticated avoidance of temptation and high noise: The agent might prefer the Noodle Shop and might not know that it is closed. This class of inferences is shown in Figure  3  (bottom): When the agent has a strong prior belief that the shop is open, the observations are most plausible if the agent also assigns high utility to the Noodle Shop (since only then will the agent attempt to go there). If the agent does not believe that the shop is open, the Noodle Shop's utility does not matter-the observations have the same plausibility either way. In addition, the model can make inferences about the agent's discounting behavior (Figure  3  right): When utility for the Vegetarian Cafe is low, the model can't explain the data well regardless of discount rate k (since, in this case, the agent would just go to the Donut Store directly). The data is best explained when utility for the Vegetarian Cafe and discount rate are in balance-since, if the utility is very high relative to k, the agent could have gone directly to the Vegetarian Cafe, without danger of giving in to the Donut Store's temptation. Example 3: Inference from multiple episodes Hyperbolic discounting leads to choices that differ systematically from those of a rational agent with identical preferences. A time-inconsistent agent might choose one restaurant more often than another, even if the latter restaurant provides more
 Experiments with Human Subjects We have shown that, given short action sequences, our model can infer whether (and how) an agent is timeinconsistent while jointly inferring appropriate utilities. We claim that this kind of inference is familiar from everyday life and hence intuitively plausible. This section provides support for this claim by collecting data on the inferences of human subjects. In our first two experiments, we ask subjects to explain the behavior in Figures  1a and 1b . This probes not just their inferences about preferences, but also their inferences about biases and false beliefs that might have influenced the agent's choice. Experiment 1: Inference with full knowledge Experiment 1 corresponds to Example 1 in the previous section (where the agent is assumed to have full knowledge). Two groups of subjects were shown Figures  1a and 1b , having already seen two prior episodes showing evidence of a preference for the Vegetarian Cafe over the other restaurants. People were then asked to judge the plausibility of different explanations of the agent's behavior in each episode.  5  Results are shown in Figure  5 . In both Naive (Figure  1a ) and Sophisticated (1b) conditions, subjects gave the highest ratings to explanations involving giving in to temptation (Naive) or avoiding temptation (Sophisticated). Alternative explanations suggested that the agent wanted variety (having taking efficient routes to the Vegetarian Cafe in previous episodes) or that they acted purely based on a preference (for a long walk or for the Donut Store). Experiment 2: Inference with uncertainty Experiment 2 corresponds to Example 2 above. Subjects see one of the two episodes in Figure  1  (with Figure  1a  modified so D1 and D2 can differ in utility and Figure  1b  modified so the Noodle Shop is closed). There is no prior information about the agent's preferences, and the agent is not known to have accurate beliefs. We asked subjects to write explanations for the agent's behavior in the two episodes and coded these explanations into four categories. Figure  6  specifies which formal agent properties correspond to which category. While not all explanations correspond to something the model can infer about the agent, the most common explanations map cleanly onto the agent properties θ-few explanations provided by people fall into the "Other" category (Figure  7 ). The model inferences in this figure show the marginal likelihood of the observed actions given the corresponding property of θ, normalized across the four property types. In both the Naive and the Sophisticated case, the model and people agree on what the three highest-scoring properties are. Explanations involving false beliefs and preferences rate more highly than those involving time inconsistency. This is because, even if we specify whether the agent is Naive/So-  Experiment 3: Inference from multiple episodes Following Example 3 above, subjects (n=50) saw the episodes in Figure  4  and inferred whether the agent prefers the Vegetarian Cafe or the Donut Store. Like the model, the majority of subjects inferred that the agent prefers the Vegetarian Cafe. Overall, 54% (+/-7 for 95% CI) inferred a preference of Vegetarian Cafe over the Donut Store, compared to the 59% posterior probability assigned by the model. Episode 2 (in which the agent does not choose the Donut Store) is identical to the Sophisticated episode from Figure  1 . Experiments 1 and 2 showed that subjects explain this episode in terms of Sophisticated time-inconsistent planning. Together with Experiment 3, this suggests that subjects use this inference about the agent's planning to infer the agent's undiscounted preferences, despite having seen the agent choose the Donut Store more frequently.
 Conclusion AI systems have the potential to improve our lives by helping us make choices that involve integrating vast amounts of information or that require us to make long and elaborate plans. For instance, such systems can recommend and filter the information we see on social networks or music services and can construct intricate plans for travel or logistics. For these systems to live up to their promise, we must be willing to delegate some of our choices to them-that is, we need such systems to reliably act in accordance with our preferences and values. It can be difficult to formally specify our preferences in complex domains; instead, it is desirable to have systems learn our preferences, just as learning in other domains is frequently preferable to manual specification. This learning requires us to build in assumptions about how our preferences relate to the observations the AI system receives. As a starting point, we can assume that our choices result from optimal rational planning given a latent utility function. However, as our experiments with human subjects show, this assumption doesn't match people's intuitions on the relation between preferences and behavior, and we find little support for the simplistic model where what is chosen most is inferred to be the most valued. We exhibited more realistic models of human decision-making, which in turn supported more accurate preference inferences. By approaching preference inference as probabilistic induction over a space of such models, we can maintain uncertainty about preferences and other agent properties when the observed actions are ambiguous. This paper has only taken a first step in the direction we advocate. Two priorities for further work are applications to more realistic domains and the development of alternatives to using human preference inferences as a standard by which to evaluate algorithms. The goal for this emerging subfield of AI is to make systems better able to support humans even in domains where human values are complex and nuanced, and where human choices may be far from optimal. Figure 1 : 1 Figure 1: Agents with hyperbolic discounting exhibit different behaviors depending on whether they model their future discounting behavior in a manner that is (a) Naive (left) or (b) Sophisticated (right).
 o ,a EU p(s|o),o ,d+1 [a ] with s ∼ T (s, a), o ∼ p(o|s ) and a ∼ C(p(s|o), o , d + 1) (for the Naive agent) or a ∼ C(p(s|o), o , 0) (for the Sophisticated agent).
 (top left), which shows the model posterior for the utilities of the Donut Store and Vegetarian Cafe (holding fixed the other agent components Y , k, and α).
 Figure 4 : 4 Figure 4: The observations in Experiment 3 show the Donut Chain Store being chosen twice and the Vegetarian Cafe once.
 Figure 5 : 5 Figure5: Explanations in Experiment 1 for the agent's behavior in Figure1a(Naive) and 1b (Sophisticated). Subjects (n=120) knew that the agent has accurate knowledge, and saw prior episodes providing evidence of a preference for the Vegetarian Cafe. Subjects selected scores in {1, 2, 3}.
 Figure 6 :Figure 7 : 67 Figure6: Map from properties invoked in human explanations to formalizations in the model. The left column describes the property. The center column shows how we formalized it in terms of the variables used to define an agent θ. The right column gives an explanation (from our dataset of human subjects) that invokes this property.
 Now the agent's choice and expected utility function are parameterized by a delay d, which together with a constant k controls how much to discount future utility: var agent = function(state, delay){ return Marginal( function(){ var action = uniformDraw(actions) var eu = expUtility(state, action, delay) factor(alpha * eu) return action }) } var expUtility = function(state, action, delay){ if (isFinal(state)){ return 0 } else { var u = 1/(1 + k * delay) * utility(state, action) return u + Expectation(function(){ var nextState = transition(state, action) var nextAction = sample(agent(nextState, delay+1)) return expUtility(nextState, nextAction, delay+1) }) } } EU s,d [a] = 1 1 + kd U (s, a) + E s ,a The agent's choice and expected utility functions are now parameterized by the distribution p(s) and the current ob- servation o: C(a; p(s), o, d) ∝ e αEU p(s),o,d [a] C(a; s, d) ∝ e αEU s,d[a]    [EU s ,d+1[a ]] with s ∼ T (s, a). For the Naive agent, a ∼ C(s , d + 1), whereas for the Sophisticated agent, a ∼ C(s , 0). When we compute what the agent actually does in state s, we set d to 0. As a consequence, only the Sophisticated agent correctly predicts its future actions.3  An implementation of the Naive agent as a probabilistic program is shown in Figure2.Time-inconsistent agent with uncertaintyWe now relax the assumption that the agent knows the true world state. Instead, we use a distribution p(s) to represent the agent's belief about which state holds. Using a likelihood function p(o|s), the agent can update this belief: p(s|o) ∝ p(s)p(o|s)
 = open) > 0.85 "He was heading towards the noodle shop first, but when he got there, it was closed, so he continued on the path and ended up settling for ... vegetarian cafe." Property Formalization Example explanation from our human subjects Agent doesn't know p(D1 = open) < 0.15 "He decided he wanted to go to the Donut Store for lunch. He Donut Store D1 is open. did not know there was a closer location" Agent falsely believes Noodle Shop is open. p(N Agent prefers D2 to D1. U (D2) > U(D1) "He might also enjoy the second donut shop more than the first" Agent is Naive / Sophisti- Y = Naive/Soph. "He ... headed for the Vegetarian Cafe, but he had to pass by cated. the Donut shop on his way. The temptation was too much to fight, so he ended up going into the Donut Shop."
			 Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence
			 The distinction and formal definition of Naive and Sophisticated agents is discussed in O'Donoghue and Rabin (1999).
			 This is the kind of agent assumed in the standard setup of an MDP (Russell and Norvig 1995)  3 This foresight allows the Sophisticated agent to avoid tempting states when possible. If such states are unavoidable, the Sophisticated agent will choose inconsistently.
			 In Experiment 2, we allow the utilities for D1 and D2 to be different. See row 3 of Figure6and the "Preference" entry for Sophisticated in Figure7.
			 In a pilot study, we showed subjects the same stimuli and had them write free-form explanations. In Experiment 1, subjects had to judge four of the explanations that occurred most frequently in this pilot.


Abstract: In the field of AI, expected utility maximizers are commonly used as a model for idealized agents. However, expected utility maximization can lead to unintended solutions when the utility function does not quantify everything the operators care about: imagine, for example, an expected utility maximizer tasked with winning money on the stock market, which has no regard for whether it accidentally causes a market crash. Once AI systems become sufficiently intelligent and powerful, these unintended solutions could become quite dangerous. In this paper, we describe an alternative to expected utility maximization for powerful AI systems, which we call expected utility quantilization. This could allow the construction of AI systems that do not necessarily fall into strange and unanticipated shortcuts and edge cases in pursuit of their goals.

Quantilizers: A Safer Alternative to Maximizers for Limited Optimization

Expected Utility Maximization Many frameworks for studying idealized agents assume that agents attempt to maximize the expectation of some utility function, as seen in the works of  Heckerman and Horvitz (1990) ,  Bacchus and Grove (1995) , and  Chajewska, Koller, and Parr (2000) . von  Neumann and Morgenstern (1944)  provide compelling arguments that any rational agent with consistent preferences must act as if it is maximizing some utility function, and  Russell and Norvig (2010, chap. 16 ) use the expected utility maximization framework extensively as the basic model of an idealized rational agent. Formally, let A be the finite 1 set of available actions, O the set of possible outcomes, and W : A → ∆O map each action to the predicted outcome distribution resulting from that action 2 . Let U : O → [0, 1] be a utility function, with U (o) being the utility of outcome o. Then an expected utility maximizer is an agent that chooses an action a ∈ A that maximizes E [U (W (a))]. We make no argument against expected utility maximization on the grounds of rationality. However, maximizing Copyright c 2016, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. 1 Expected utility maximizers and quantilizers can also be defined for infinite set of actions, but in this paper we discuss only finite action sets for simplicity. 2 Defining W is not straightforward. See  Rosenbaum and Rubin (1983)  for a formalization of the "causal effects" of an action, whose maximization results in causal decision theory; see  Soares and Fallenstein (2015)  for discussion of alternative decision theories. the expectation of some utility function could produce large unintended consequences whenever U does not accurately capture all the relevant criteria. Some unintended consequences of this form can already be observed in modern AI systems. For example, consider the genetic algorithm used by  Nguyen, Yosinski, and Clune (2014)  to generate an image which would be classified by a deep neural network as a starfish, with extremely high confidence. The resulting image ended up completely unrecognizable, looking nothing at all like a starfish. Of course,  Nguyen, Yosinski, and Clune (2014)  intended to develop images that would be misclassified, but this demonstrates the point that an algorithm directed to find the image that most classified as a starfish, generated an image that was very strange indeed. Another example of unintended solutions resulting from maximization is given by  Thompson (1997) , who used genetic algorithms to "artificially evolve" a circuit that could differentiate between two different inputs via a 10 x 10 field programmable gate array (FPGA). The resulting circuit worked, using only 21 of the 100 available FPGA cells. Curiously, five of those cells were not connected to either the input or the output-the evolved circuit was making use of them via the physical features of that specific chip and machine (Thompson suggest that the circuit was taking advantage of electromagnetic coupling or power-supply loading). If the goal was to design a circuit that could also work on other chips, then this "maximization" process (of finding a very small algorithm that, on that one circuit board, was good at distinguishing between two inputs) produced an unintended result that was dependent upon physical features of the chip that the designers thought were irrelevant. The resulting circuit performed well by the test metric, but the result would not likely have been usable on any other physical chips. Unintended solutions of this form could become quite dangerous in highly autonomous artificially intelligent systems with capabilities that strongly exceed those of humans.  Bostrom (2014)  discusses a number of potential dangers stemming from expected utility maximization in powerful agents. He considers "superintelligent" agents, defined as agents that are "smarter than the best human brains in practically every field," and describes the problem of perverse instantiation, his term for the seemingly-perverse unintended consequences of directing a powerful agent to attempt to maximize a utility function which is not in fact aligned with our goals. For example,  Bostrom (2014)  illustrates this by describing a machine that, when told to make humans smile, accomplishes this goal by paralyzing human faces into permanent smiles, rather than by causing us to smile via the usual means. Another example is given by , who describes an agent directed to "cure cancer" which attempts to kidnap human test subjects. Clearly, before an autonomous system can be granted great autonomy and power, we must have some assurances that it will not, in pursuit of its goals, produce any such "perverse" unintended consequences. In both the short term and the long term, the problem with expected utility maximization is that the system's utility function likely will not capture all the complexities of human value, as humans care about many features of the environment that are difficult to capture in any simple utility function . Of course, no practical system acting in the real world can actually maximize expected utility, as finding the literally optimal policy is wildly intractable. Nevertheless, it is common practice to design systems that approximate expected utility maximization, and insofar as expected utility maximization would be unsatisfactory, such systems would become more and more dangerous as algorithms and approximations improve. If we are to design AI systems which safely pursue simple goals, an alternative may be necessary.
 Expected Utility Quantilization Given that utility maximization can have many unintended side effects,  Armstrong, Sandberg, and Bostrom (2012)  and others have suggested designing systems that perform some sort of "limited optimization," that is, systems which achieve their goals in some non-extreme way, without significantly disrupting anything outside the system or otherwise disturbing the normal course of events. For example, intuition says it should be possible to direct a powerful AI system to "just make paperclips" in such a way that it runs a successful paperclip factory, without ever attempting to turn as much of the universe as possible into paperclips.  Simon (1956)  propose instead designing expected utility "satisficers," agents that choose any action that achieves an expected utility above some fixed threshold. In our notation, an expected utility satisficer is one that chooses an action from the set {a ∈ A : E [U (W (a))] ≥ t} for some threshold t. (Indeed,  Fallenstein and Soares (2014)  describe a toy model of satisficing agents in simple environments.) However, the satisficing framework is under-defined and not necessarily satisfactory. When it comes to powerful intelligent agents, it may often be that the easiest way to satisfice is to maximize: if the paperclip AI system can easily create a sub-agent that tries as hard as it can to make paperclips, it may be that the easiest satisficing action is simply to construct a sub-agent that attempts to approximate something like expected utility maximization instead.  Abbeel and Ng (2004)  have suggested apprenticeship learning as an alternative to maximization when it is difficult to explicitly specify the correct goal. In apprenticeship learning, an AI system learns to closely imitate a human expert performing a task. An agent which only executes actions that its operators would have executed seems likely to be immune to Bostrom's problem of perverse instantiation. Yet while mimicking humans avoids many unintended effects of maximization, a system that merely mimics humans cannot easily surpass the performance of those it is mimicking (except perhaps by performing the same actions faster and more reliably). Such a technique could prove quite useful, but wouldn't help us design algorithms intended to outperform humans by finding plans, strategies, and techniques that the operators would not have been able to identify. In this paper, we propose quantilization, which interpolates between mimicry and maximization in order to gain some advantages of both. Definition 1 (Quantilizer). A q-quantilizer is an agent that, when faced with a decision problem, returns a random action in the top q proportion of some "base distribution" over actions, sorted by the expected utility achieved if that action is executed. Intuitively, a 0.1-quantilizer (which we could also call a "ten percentilizer") selects a random action from the top 10% of actions. More formally, let A be the action set and γ be some distribution over actions. For example, γ could be uniformly random, or it could be a distribution estimated from human actions. Let f : [0, 1] → A be a function that represents γ, but sorted by expected utility: ∀x, y ∈ [0, 1] x > y =⇒ E [U (W (f (x)))] ≥ E [U (W (f (y)))] µ({x ∈ [0, 1] | f (x) = a}) = γ(a) where µ is the Lebesgue measure on the reals. For any 0 < q ≤ 1, a q-quantilizer samples a number x from the uniform distribution over [1 − q, 1] and returns f (x). We will write Q U,W,γ,q (a) for the probability that it returns action a, where U is the quantilizer's utility function.  3  We will abbreviate this as simply Q q (a), where it is understood that Q q is parameterized on U , W , and γ. As q approaches 0, a q-quantilizer reduces to a maximizer whose available action set is the support of γ. A 0.01quantilizer (also known as a "one percentilizer") samples a random action from the top 1%; a 0.5-quantilizer ("fiftypercentilizer") samples a random above-average action; a 1-quantilizer just mimics γ. As an example, suppose there are 3 actions A = {a, b, c} with expected utilities  1 . A maximizer will always return action c. If we define a uniform base distribution γ, then the behavior of the q-quantilizer Q q is: Figure  1 : A representation of a 5/9-quantilizer for the threeaction example. The quantilizer samples x from the gray region and returns the corresponding action f (x), so it returns b with 2/5 probability and c with 3/5 probability. E [U (W (a))] = 0.2 E [U (W (b))] = 0.5 E [U (W (c))] = 0.7 as illustrated in figure • If q ≤ 1 3 , it always returns c. • If 1 3 ≤ q ≤ 2 3 , then it returns c with probability 1 3q and b the rest of the time. • If q ≥ 2 3 , then it returns c with probability 1 3q , b with probability 1 3q , and a otherwise. In this example, it is clear that the quantilizer interpolates between maximizing and mimicking γ depending on q.
 Derivation from a Conservative Cost Constraint We will now explore a few circumstances under which one might prefer to design a system that quantilizes expected utility, rather than maximizing it. Definition 1 gives some hint as to what sort of situation quantilizers are designed for. Given some trusted base distribution γ of actions, such as a generative model for the actions that a human might take, the distribution Q 0.01 is roughly the distribution of actions that a human might take if they were attempting to score well by U (but were otherwise adhering to "normal" behavior). That is, if γ puts most of its measure on actions that humans might generate when solving similar problems, then Q 0.01 is similar to the distribution of actions a human might take conditional on that action having a high utility according to U . Thus, intuitively, a 0.01-quantilizer is designed to solve problems in a manner similar to how a human would solve that problem. Furthermore, under certain assumptions, we can prove that building a quantilizer instead of a maximizer is not too costly to our true utility function, in the sense that running the quantilizer won't be much worse than simply sampling from the base distribution γ. To show this, assume that an AI system is built to quantilize expected utility for some utility function U . Assume that our true utility function is some other function V . Intuitively, we will imagine that for most actions, E [U (W (a))] is similar to E [V (W (a))], but for the actions a U which maximize E U (W (a U )) (or come close to maximizing it), the value of E V (W (a U )) is much lower (because a U is some edge case or "perverse instantiation," such as generating smiles by paralyzing people's faces). Thus, as in Swaminathan and Joachims (2015), we assume there is some unknown cost function c : A → R ≥0 , which describes the negative contribution of an action a to V . For example, if there is some action noop which the AI system may execute immediately to shut down without having any further effect on the world, then we could define c(a) = max{0, E [V (W (noop))] − E [V (W (a))]}. We expect c to be quite difficult to approximate, because human values are complex and depend on many difficult-to-quantify aspects of the world. Indeed, if we could formally encode c, we could build it into U , and then we would have much less need to worry about perverse instantiations. Even without an approximation of c, though, we can still prove that under certain assumptions, a quantilizer will not incur a high cost according to c. Assume that the cost of a random action from γ is no more than 1. This assumption may be justified if, for example, γ is a uniformly random action and uniformly random actions are highly unlikely to be harmful. Write E A∼γ [c(A)] ≤ 1 for the expected cost of a random action sampled from γ. It is now possible to bound the cost of an action sampled from Q q : Lemma 1 (Cost bound on quantilizers). E A∼Qq [c(A)] ≤ 1 q Proof. Observe that Q q "fits under" γ when scaled down by a factor of q: Q q (a) = 1 q µ({x ∈ [1 − q, 1] | f (x) = a}) ≤ 1 q γ(a). Therefore, it is possible to bound the cost of Q q : E A∼Qq [c(A)] = a∈A Q q (a)c(a) ≤ a∈A 1 q γ(a)c(a) = 1 q E A∼γ [c(A)] . (1) So the expected cost of sampling from Q q is no more than 1/q times that of sampling from γ. Thus, for example, running a "ten percentalizer" can be no more than ten times as costly as sampling from γ. In fact, quantilizers are optimal among all policies that can be proven to be low-cost in this way. For example, suppose we have generated some base distribution γ of actions (perhaps a generative model of what sort of actions a human would take), and we're willing to risk running an agent that incurs an expected cost of at most 100 times the expected cost of simply sampling from γ. Then, a 0.01-quantilizer is the optimal agent we could run. More formally, say we desire some distribution p ∈ ∆A such that the following conservative cost constraint holds for some acceptable cost threshold t > 1: ∀c :A → R ≥0 E A∼γ [c(A)] ≤ 1 =⇒ E A∼p [c(A)] ≤ t. (2) This constraint states that, for any cost function assigning an expected cost of no more than 1 to a random action from γ, the expected cost of an action drawn from p is no more than t. Then, we have the following: Lemma 2 (p fits under γ). Let p satisfy 2. Then p(a) ≤ tγ(a) for all a ∈ A. Proof. Suppose this were not true. Then for some a * we have p(a * ) > tγ(a * ). Now define c(a) = 1/γ(a * ) if a = a * 0 otherwise We have E A∼γ [c(a)] = a∈A γ(a)c(a) = γ(a * )c(a * ) = 1. Also, E A∼p [c(a)] = a∈A p(a)c(a) = p(a * )c(a * ) > tγ(a * )c(a * ) = t. Jointly, these statements contradict constraint 2. Theorem 1 (Quantilizer optimality). Choose q=1/t. Then, a q-quantilizer maximizes expected U -utility subject to constraint 2. Proof. By Lemma 2, we cannot assign more that γ(a)/q probability to any action a, so the best we can do to maximize expected utility is to assign γ(a)/q probability to the best actions and 0 to the rest-and this is exactly what a q-quantilizer does.
 Targeted Impact Expected utility quantilization is intended for use in systems that may eventually need to make important decisions, autonomously, in environments where negative actions could have a large cost. For example, imagine an AI system which is programmed to autonomously trade stocks in order to make significant sums of money, where that money will in tun be put to use improving the world in some potentially dramatic fashion. Unfortunately, the bounds of equation (  1 ) and the cost constraints of equation (  2 ) apply just as well for cost functions as benefit functions: just as running a q-quantilizer will not, in expectation, be any more than 1/q times as costly as sampling from γ directly; nor will it be any more than 1/q times more beneficial than sampling from γ directly. Thus, in a sense, a q-quantilizer has a bounded impact-it will not make the world all that different than running an AI system that samples γ directly would have made it. What we really want, though, is not low impact, but targeted impact. A small modification to the quantilizer specification allows us to achieve the same constraints on costs without necessarily constraining the potential benefits. Before explaining, first note that the stock trading AI system has two different types of side effect: one is that it produces money which can be used by its operators; the other are side effects of trading (for example, it might crash the market in order to make more money). The first side effect might cause a significant change in the world, e.g., if the operators use the money to fund a charity which is highly effective at reducing global poverty. Nevertheless, we would prefer the system to only be guaranteeing reduced impact according to the second sort of side effect. That is, in this setting, we can define the cost function c : A × R → R ≥0 , such that c(a, m) determines how much cost an action a has, if counterfactually it generates m dollars (rather than the actual amount of money that it generates). Given a cost function of this form, we would like to design a quantilizer such that ∀c : A × R → R ≥0 , m : R E A∼γ [c(A, m)] ≤ 1 =⇒ E A∼p [c(A, m)] ≤ 1/q. That is, we require that for all counterfactual amounts of money m, the side effects of the stock trading algorithm have low cost compared to the side effects of executing actions at random from γ. Thus, a system obeying this cost constraint does not need to worry about the extra side effects of generating significant amounts of money for its operators. Mathematically, quantifying over c and m at the same time is the same as quantifying over functions of the form a → c(a, m) for any c and m, so the condition is equivalent to ∀c : A → R ≥0 E γ [c(A)] ≤ 1 =⇒ E p [c(A)] ≤ 1/q which is the same as the original cost constraint. Therefore, it is possible to design a quantilizer that is guaranteed to have low cost (compared to γ) through effects other than money, by the same methods that guarantees it to have low overall cost.
 Algorithms for Quantilization Just as it is impractical to design algorithms that literally maximize expected utility in complex environments, it is similarly impractical to design algorithms that perfectly quantilize expected utility. Any practical system based on expected utility quantilization would similarly have to use some form of approximation. Fortunately, it is not hard to see how expected utility quantilization could be approximated. Suppose we can compute E [U (W (a))] for each action a, and that we wish to approximate a q-quantilizer with respect to a distribution γ. To do this, we can run the following algorithm: first take n samples from γ, and then sample from Q U,W,γ,q , where γ is the empirical distribution estimated from the n samples. As n approaches infinity, γ approaches γ, so Q U,W,γ,q approaches Q U,W,γ,q . Thus, given some base distribution γ which is relatively easy to sample from, and some method for approximating E [U (W (a))] and γ, an approximate expected utility quantilizer can be constructed in a straightforward manner. Indeed, this algorithm yields insight into how safe it is to perform a random search of actions sampled from γ. If q = 1/n, then this algorithm simply takes n samples from γ and returns the one with the highest expected utility, emulating a simple random search procedure. Since this procedure approximates q-quantilization, the bounds that apply to quantilization can be shown to hold for random searches over samples of γ, with some minimal modifications. 3 Drawbacks and Open Problems
 Acquiring the Base Distribution Clearly, a quantilizer is only as good as its base distribution γ. The above constraints on the amount of costs that running an expected utility quantilizer can incur are in terms of multiples of the expected cost from sampling directly from γ: clearly, if it is dangerous to sample actions from γ, then it is dangerous to run a quantilizer using γ. Defining a useful base distribution is not a straightforward task. One could define the base distribution to be some uniformly random action, but in a complex multi-dimensional space of strategies, a uniformly random action is highly unlikely to be useful, and so even a 10 −5 -quantilizer would be unlikely to do anything useful. If instead γ described random strategies that a human might actually execute, then it is much more likely to be useful. Thus, a generative model of human strategies would make expected utility quantilization a much more viable strategy. (See  Ng and Russell (2000)  and  Baker, Tenenbaum, and Saxe (2007)  for some initial work in this direction.) Of course, there are a number of situations in which a simple generative model of human behavior would not be sufficient. For example, if humans are likely to misunderstand the problem and thereby cause catastrophe, then a simple generative model would not suffice. (And if a human were quite likely to design an expected utility maximizing AI system, then a quantilizer may well also generate a maximizer.)  4  The study of how to generate safe and satisfactory base distributions γ for various different problem domains remains an open problem.
 Butterfly Effects It could be that a non-negligible percentage of ordinary actions that humans take are either very good or very bad, due to butterfly effects. In this case, the cost of sampling from fairly ordinary γ distributions is actually quite high, with the expectation of γ being reasonable only because the high costs are canceled out by equally high benefits. However, in situations such as these, it is quite possible for a quantilizer to preserve the costs of γ without necessarily preserving many of the benefits. For example, suppose that the quantilizer is trying to make money on the stock market. There is a 10% chance of the stock market crashing in the next year. While a random sequence of trades made by a human will not significantly change the chance of a crash in expectation, it might be the case that a certain sequence of 3 ordinary trades (a sequence that a human would be 0.1% likely to make) will greatly increase the chance of a stock market crash through butterfly effects. In this case, a 0.001-quantilizer that can predict these effects might make these trades if it can profit from the crash. It is not clear if this scenario is even theoretically possible, since it requires extremely accurate prediction of the future. Nevertheless, it would be prudent to study this problem further and possibly design variations on quantilization that avoid this problem.
 Quantilization in Repeated Games Suppose we have n independent games. On game i, the agent chooses an action a i ∈ A i and receives utility U i (W i (a)), where each W i : A → ∆O returns the outcome for game i, and each U i : O → [0, 1] measures the utility achieved in that game. There are two different ways to design a quantilizer that plays these n games. We could design the agent to q-quantilize each game independently, choosing from the best q proportion of actions in each game independently according to n different base distributions γ i . This distribution for the sequence of actions in each game could be written as n i=1 Q Ui,Wi,γi,q . Alternatively, we could design the agent to q-quantilize a sequence of actions for the joint game. Here we define the joint game as A = n i=1 A i , W (a 1 , ..., a n ) = (W 1 (a 1 ), ..., W n (a n )) and U (o 1 , ..., o n ) = n i=1 U i (o i ). This action distribution for a q-quantilizer with respect to the base distribution n i=1 γ i could be written as Q U,W, n i=1 γi,q This is quite different from the first scenario! To give a concrete example, imagine two consecutive games that work as follows. In each game, the agent must pick an integer from the set {0, 1, 2}, and then gain that much utility. Thus, we define A i = {0, 1, 2}, W i (a) = a, U i (o) = o. Each base distribution γ i is uniform. Then, a 1/3-quantilizer quantilizing both games independently will choose from the top third of actions in each game separately, and will always yield the action sequence (2, 2). By contrast, a quantilizer quantilizing for the joint game will uniformly sample from the top third of action sequences {(2, 2), (1, 2), (2, 1)}, as illustrated in figure  2 . What could explain this difference? If we define a cost function c(a 1 , a 2 ) = 1 if (a 1 , a 2 ) = (2, 2) 0 otherwise which states that the (2, 2) action has terrible unintended consequences, then independent 1/3-quantilization will accrue a cost 9 times that of sampling from γ. This cost blowup increases exponentially with n. On the other hand, 1/3quantilization in the joint game will accrue a cost only 3 Figure  2 : A representation of the 9 available action pairs in the repeated game. Independent 1/3-quantilization will always select the striped action pair, while 1/3-quantilization on the joint game will select uniformly from one of the three shaded action pairs. times that of sampling from γ. If we believe that the cost function c may not sum costs independently between games, then quantilizing each game independently could be very costly. Cost Independence Assumption Suppose we added the assumption that c(a 1 , ..., a n ) = n i=1 c i (a i ) for some c i : A i → R ≥0 . Then a repeated q-quantilizer would get total expected cost E Ai∼Q U i ,W i ,γ i ,q [c(A 1 , ..., A n )] = n i=1 E Ai∼Q U i ,W i ,γ i ,q [c i (A i )] ≤ 1 q n i=1 E Ai∼γi [c i (A i )] = 1 q E Ai∼γi [c(A 1 , ..., A n )] so repeated quantilization would no longer have a cost blowup exponential in n. However, adding this assumption may be dangerous. Imagine a scenario in which almost all of the cost of a sequence of actions comes from correlation between actions in different games. For example, imagine that each game consists of appending a byte to a computer program, and the program is run when it consists of a million bytes. Then some sequence of actions that each individually have probability 1 in 256 (namely, appending the bytes of a program implementing a computer program that would be very destructive if run) could be dangerous. Each individual action did not have a much higher cost than adding a random byte to the program, but taken all together, the sequence of actions still had a very high cost. In this case and any analogous case, assuming that the cost function is additive could be quite dangerous. In fact, attempting to design a "superintelligent" (in the sense of  Bostrom (2014) ) expected utility quantilizer could lead to similar issues. Imagine asking a superintelligent quantilizer for repeated actions that maximize the long-term number of paperclips produced by a certain factory. In this case, similar to the above example, each action might play some small part in a complex plan to turn large swaths of the world into paperclips. However, this seems much less likely to be a problem if the different goals that we have the system quantilize with respect to are independent. For example, imagine that we instead ask the system to repeatedly quantilize a slightly different utility function U i which only cares about events which happen in hour i, with the quantilizer run at the beginning of hour i. Intuitively, it seems that this setup would be safer. However, no safety guarantees have been given; finding a reasonable assumption about the cost functions under which this form of independent quantilization would be safe (if such an assumption exists) remains an open problem.
 Example Applications Quantilizers cannot be used to effectively achieve all possible goals. If the system has to achieve an outcome that it is extremely difficult to achieve by simply sampling γ, then the quantilizer, too, will have a hard time achieving that outcome. For example, if γ is a generative model of human behavior, and the system faces a problem that a human would be very unlikely to solve even one time in a billion, then the quantilizer would also be hard pressed to solve the same problem. However, there are a number of tasks where an expected utility quantilizer would plausibly be able to perform quite well. In this section, we will list a number of examples of tasks that are plausibly solvable with limited expected utility quantilizers. Note that, in practice, these example applications involve repeated quantilization. As explained in section 3.3, repeated quantilization is not currently known to be safe, but we will provisionally assume that it is possible to develop some set of independent (possibly time-indexed) objectives such that repeatedly quantilizing those objectives is in fact safe.
 Synthesizing Highly-Rated Images Imagine that we have a database containing images and human ratings of these images, and that we would like to synthesize new images that are expected to receive high ratings. To do this, we could separately create a generative model over images and a regression model predicting the expected rating for an image. If the regression model is a deep neural network, then as in  Nguyen, Yosinski, and Clune (2014) , the image that gets the highest rating according to the model is unlikely to resemble highly-rated images. Instead, we could use the generative model over images as the base distribution γ and then quantilize to generate a plausible image with a high predicted rating.
 Factory Management Suppose we want a program to control a factory to create a number of objects satisfying a certain specification, using a fixed amount of time and resources. Policies will control parts of the factory, eventually producing many objects. Maximizing the extent to which the resulting objects adhere to the specifications may result in unintended solutions. Perhaps the resulting object will be useless despite satisfying the specifications, due to Goodhart's law: "When a measure becomes a target, it ceases to be a good measure."  (Goodhart 1975) . As an example, imagine a factory that produces electronic toys, which has automated devices that samples toys and automatically score them to see how well they meet specifications. Then, we can imagine a expected utility maximizer figuring out exactly which toys will be sampled, and makes only those ones to spec (the other ones being the cheapest possible piece of plastic that triggers the toy-counter). Or perhaps the automated devices dock points for toys that come off the assembly line hot (as these toys generally become deformed), and the expected utility maximizer finds a way to supercool the toys and thus gain an extremely high score, despite the fact that the supercooled toys later shatter and are useless. By contrast, if we build the AI system to quantilize the extent to which the devices pass the specifications, then the resulting policy will be much more likely to be of the form we expected: the task will almost certainly be accomplished in a manner that was not incredibly unlikely according to γ.
 Safely Transferring Information to Humans A number of proposals for designing extremely capable AI systems that are nevertheless safe to use for one purpose or another include a step where the AI system is expected to explain its plans to humans instead of executing them  (Armstrong, Sandberg, and Bostrom 2012; Bostrom 2014 ). However, it is quite a difficult task to define a goal function that adequately describes "provide an explanation that causes the human to understand the plan." Yet, absent such a goal function, the system's "explanation" is likely to be inadequate. If the system's internal representation of the plan is foreign to the operators, then they may not be able to interpret most simple data dumps describing the system's plan. Furthermore, if the system is more capable and knowledgeable than the humans, then the humans may also need a large number of relevant facts, assumptions, and sub-strategies explained to them before they are capable of evaluating the plan. But if the system has to perform a complex optimization task to explain the plan to the humans, then we face the same problem of safe optimization again. If the system is designed to maximize the extent to which the human operators understand its plan, then perverse instantiations could occur, and the result could be quite dangerous. An expected utility maximizer might exploit edge cases in the definition of "understand" to fool or otherwise manipulate the humans in a fashion that fulfills the technically stated requirements of "make the humans understand the plan," without generating anything that we would actually term "understanding." It seems very difficult to define what it means for an explanation to cause the human to understand such that we would be comfortable instructing a powerful system to maximize along this metric. Quantilization provides an alternative: we could build an expected utility quantilizers that selects an explanation that is good at causing the human to "understand" the system's plans in the same way that they would understand a plan as explained by another human, without as much risk of perverse instantiation. For example, if the base distribution γ is a generative model of explanations created by humans, then moderately strong quantilization might result in an explanation of a plan suitable for human consumption.
 Stock Trading As explained in section 2.2, a quantilizer can be designed to make trades on the stock market that are likely to make money, but which have few other side effects beside the side effect that now the operators have much more money. Specifically, the important thing is that expected utility quantilization gives us the ability to design systems that would almost certainly not crash the stock market (any more than sampling randomly from γ is likely to crash the stock market, which, if γ is a generative model of a smart human trader, means that the system is not much more likely to crash the market than any given smart human trader). This holds true even if the system is powerful enough to find a strategy for gaining incredible amounts of money by causing the stock market to crash. An expected utility maximizer maximizing the amount of money it receives in the stock market would crash the market the moment it found some way to generate huge amounts of profit from doing so, but an expected utility quantilizer would not be much more likely to crash the market than sampling randomly from γ.
 Conclusion Directing a powerful AI system to maximize some utility function U could result in large unintended consequences, if U does not in fact capture all the intricacies of human values. Many of these dangers can be mitigated if, instead of designing expected utility maximizers, we consider other agent architectures. Expected utility quantilization is one alternative to expected utility maximization that allows some safety guarantees, given that we have some base distribution γ where sampling actions from γ is known to be relatively safe. Expected utility quantilization is not a silver bullet. Constructing a safe expected utility quantilizer requires some method for generating a safe base distribution γ, and there are a number of contexts where expected utility quantilization is not guaranteed to yield good results. For example, if the costs of sampling from γ could in fact be high, then quantilization may not lead to good outcomes. Furthermore, if a quantilizer is going to be used to complete the same task many times sequentially, then running the quantilizer may be significantly more dangerous. Yet despite these shortcomings, the preliminary results from investigating expected utility quantilization are promising. Gaining a better understanding of the benefits and drawbacks of expected utility quantilization could yield many more insights into how to create powerful systems that are "domestic" in the sense of  Bostrom (2014, chap. 9) , who defines a domestic agent as an agent with limited "scope in its activities and ambitions." There are many open issues that make it difficult to assess how safe it would actually be to run an extremely capable expected utility quantilizer in an unrestricted domain, but nevertheless, this area of research does prove fruitful, and we are hopeful that further study of expected utility quantilization may yield insights into how to design autonomous AI systems that avoid the problem of perverse instantiation.
 Acknowledgements The author would like to thank Nate Soares for help preparing this paper, Benya Fallenstein and Andreas Stuhlmüller for discussion and comments, and Stuart Armstrong for helping to develop initial versions of this idea. A grant from the Future of Life Institute helped fund this work. 			 Since there are multiple sortings of the actions f , some leading to different action distributions, this notation is imprecise. To make the notation more precise, we could specify some canonical way of defining f , such as assigning as much probability mass as possible to lexicographically earlier actions.
			  See Christiano (2011)  for additional discussion of superintelligent predictions of human decisions, and the difficulties that arise in this context.