Abstract: This paper proposes a mathematical model for a simplified version of the game defined in Hadfield and  Weingast [2012]  which proposes that legal order can be described as an equilibrium in thirdparty decentralized enforcement coordinated by a centralized classification institution. We explore the attractiveness of joining a new group (which is assumed to have settled on an enforcement equilibrium already) where groups differ in terms of the frequency of interactions in which norm violation is possible (normative interactions) and thus punishment is called for. We show that groups in which normative interactions are frequent but involve relatively unimportant rules may achieve higher value for participants.

Gillian K Hadfield
Dylan Hadfield-Menell

Pervasive Spurious Normativity

Introduction One of the attributes that differentiates human from other animal societies is the presence of norms and in particular norms over behaviors that do not appear to have a material impact on well-being apart from the fact that people will punish the failure to adhere to a norm. In all known human societies there is a rich normative landscape, which attaches normative valence to actions including many that have no immediate material implications for those who are expected to help enforce them through social sanctions. Most societies for example have rules about what it is appropriate to eat when, what tone of voice to use in what settings, how close to stand to someone else, how to behave when waiting with others to get into a venue or access to a resource, what information should be shared with whom and when, who can participate in particular trades or economic activities and so on. In many cases, particular norms are arbitrary, even though violation of them is treated as worthy of punishment by others. We call this spurious normativity.  Hadfield [1999] , for example, reviews the anthropological literature on the sexual division of labor across pre-industrial societies. While almost all societies categorize work as either women's work or men's work, the particular classifications vary substantially cross-culturally: what is women's work in one society is men's work in another. The classification is arbitrary.  1  Fessler and Navarrete  [2003]  call the process by which patterns of behavior are imbued with moral sentiments that motivate sanctioning of violations of the pattern normative moralization. They use as an example the normative moralization of handedness. Most people are right-handed but particularly in societies with few specialized tools, whether someone is right-or left-handed often has no material consequences for others. Nonetheless, many cultures treat using one's right hand as a morally approved category-denoting purity or politeness-and one's left hand as cause for opprobrium-revealing weakness or evil. Fessler  [1999]  hypothesizes that, driven forward by the emotions of shame and pride that are triggered by violation or compliance with cooperative norms, culture extends the set of actions that are subject to normative moralization as a way of extending the set of actions that can be used as information about cooperation beyond those actions that directly involve cooperation. A norm that says "it is wrong to fail to look where you are going" generates direct cooperative benefits, helping people to avoid crashing into one another. A norm that says "it is wrong for a man to walk down the street wearing shorts" does not generate cooperative benefitsunless people in this society treat conformity to this norm as informative about a person's likelihood of behaving in conformity with norms that do generate cooperative benefits. As Fessler  [1999]  (who identifies the the "watch where you are going" and "men don't wear shorts out in public" norms in an Indonesian village) puts this: Ego's inclusion in cooperative activity is dependent on her ability to meet Others' expectations, and those expectations are, in turn related to shared standards for behaviors which are relevant to cooperation. As a consequence, the significant adaptive advantage offered by participation in cooperative activities generated selective pressure for an increase in the attention paid to these standards. Late [evolving] second order emotions [emotions that arise in response to others' first order emotions] were the vehicle through which this increase in attention was achieved. Moreover, because late second order emotions entail a sensitivity to the reactions of all individuals, Ego must be concerned with her performance vis--vis shared standards when interacting with any other member of her group. It is only a small step from this situation to one in which the shared standards with which Ego is concerned are not limited to the question of cooperative activity-once Ego is concerned with how all Others evaluate her, it is not difficult for shared standards governing other types of behavior to become salient as well. This is because an Other may extrapolate from situations that do not involve cooperation to those that do-an Other may think "if that individual does not follow shared standards in this context, how can I be confident that he will do so if I invite her to engage in cooperative activity?"(pre-pub p. 34, emphasis added) Fessler's account focuses on the evolution of emotions in response to norm violation (in particular shame) to motivate voluntary conformance with norms. But as  Boyd and Richerson [1992]  and many others have emphasized, and Fessler's own account of shame as a second order response to Ego's actual or imagined experience of hostility or criticism from Other triggered by norm violation implies, effective third-party punishment plays a significant role in supporting norm compliance. Indeed,  Mathew et al. [2012]  argue that even small-scale cooperation among kin and close associates may require third-party punishment to achieve evolutionary stability. In this paper we analyze mathematically the potential benefits of extending normative moralization to behaviors that are from a material perspective irrelevant or at least of small consequence to most people, that is, the value of pervasive and spurious normativity. We ask: does a community generate higher payoffs for participants if it punishes violation of apparently meaningless rules or if it focuses more narrowly only on norms that are functional in the sense of generating direct benefits? Intuitively, one might expect that only functional norms that govern behaviors that matter for payoffs would emerge and stabilize in equilibrium: punishment is costly and why would a society expend punishment resources on ensuring conformity with rules that have no impact on material well-being? Most analyses of norms in the law and economics literature assume that norms coordinate outcomes that improve welfare. Sugden  [1986] ,  McAdams [2005]  and  Myerson [2004]  propose, for example, that property rules emerge because they solve the coordination problems that arise in costly contests over resources (Hawk-Dove games). We consider the impact of extending normativity to apparently arbitrary actions on the prospect for sustaining welfareimproving behavior in an equilibrium social order based on collective third-party punishment. We demonstrate that communities that extend normativity in this way can generate higher value for participants than those that restrict the range of normativity. The intuition of our result is as follows. Communities with pervasive spurious normativity provide agents with plentiful and cheap opportunities to observe punishment behavior by others. An individual's willingness to cooperate in a community-which requires foregoing safe non-cooperative options and exposing oneself to the risk of being exploiteddepends on that individual's beliefs about the likelihood that the community effectively punishes conduct that harms the individual. If you are going to risk exposing yourself to harm, you want to know if your community contains enough people who will punish the perpetrator to give you confidence that harm is reasonably deterred. Assume you are a newcomer to a community or that the community was recently handed a new set of norms. (Think here of rule-of-law building efforts in developing countries.) Assume that the community is in an equilibrium in terms of punishment behavior but you do not know the likelihood of effective punishment of norm violations. Assume also that the only way to learn about the likelihood of punishment is to observe punishment behavior. You can learn this information more cheaply if you are given abundant opportunities to observe what happens when there are violations if the violations that you have to expose yourself to don't really matter very much. You don't really care whether men walk down the street in shorts but by taking a walk yourself you can see how others react to shorts-wearing men and thus gain information about how they would react to violations you do care about-careless driving for example. Thus, if you could participate solely as an observer except when your own interests were directly at stake, you would prefer to live in a world with pervasive even if spurious normativity-abundant opportunities to observe reactions to norm violations-than one that was narrowly focused on punishing just the stuff you care about. This will still be true even if you are required to participate in the community-complying with and punishing spurious norms-so long as those costs, which increase with the pervasiveness of norms, are not too great. Our results have implications for the evolution and microfoundations of law.  Hadfield and Weingast [2012]  present a model that derives characteristic features of law-such as generality, stability, uniqueness and universality-as attributes necessary to support an equilibrium in which behavior is patterned on the classifications of behavior articulated by a centralized institution. Enforcement is assumed to come exclusively from decentralized collective punishment of conduct classified as punishable by the centralized institution; there is no centralized enforcement apparatus such as the state. This theory of the microfoundations of law proposes, contrary to most economic and positive political theories of law, which define law as a set of rules enforced by a centralized enforcement apparatus (see  Hadfield and Weingast [2014] ), that law is an innovation in the mechanism used to coordinate the same enforcement mechanism that supports other normative social orders-decentralized collective punishment. This has important implications for our understanding of how law developed and how it can be built in environments where it is currently lacking. The equilibrium legal order in Hadfield and Weingast  [2012]  is supported by a particular specification of beliefs. Their model posits that an agent (call the agent Ego) treats other agents' (Others') failure to punish behavior classified by the institution as punishable, including behavior that has no impact on Ego's (or perhaps any agent's) payoff, as informative of the likelihood that Others will also fail to punish wrongful behavior that does have an impact on Ego's payoff. This belief structure creates an incentive for individuals to participate in collective punishment, potentially mitigating the free-rider problem in collective punishment. Ego has no incentive to participate in collective punishment (the model assumes standard preferences) except to maintain an equilibrium in which behavior that reduces Ego's longterm payoff is deterred by the threat of collective punishment. Effectively, an agent's participation in punishment is treated as information about that agent's continued assessment that an equilibrium in which collective punishment is coordinated by the classifications articulated by the central institution will benefit that agent and hence that the agent will be willing to incur costs to support the equilibrium. The classification institution in this model is serving as what  Hadfield and Weingast [2012]  call an authoritative steward of a very simple binary partition of behaviors into those that are punishable and those that are not. This can be interpreted as the construction of a sparse labeling system: lawful and not lawful.  2  We can interpret the beliefs underlying legal order as beliefs about whether people are punishers or not of behavior to which the constructed (that is, not natural) label "unlawful" is attached. Our model here provides a basis for understanding how a simple binary classification scheme that is comprehensivecovering a wide variety of conduct-can emerge in a setting in which equilibrium depends on 1) voluntary participation in punishment and 2) a belief structure in which punishment of an action labeled punishable is considered informative about the likelihood of punishment of other actions labeled punishable, even when the assignment of the label is potentially arbitrary. We suggest that understanding how such a classification scheme and belief structure can emerge is critical for understanding the emergence of law. The strategy of our paper is as follows. We first give an overview of the model and basic notation in Section 2, together with some technical background from the analysis of multi-armed bandit games and partially observed Markov decision processes. To build intuition, we then present in Section 3 analytical results for the limiting case in which Ego bears no cost of complying with spurious norms or punishing their violation. Because the games we analyze quickly becomes analytically complex but relatively easy to compute once we introduce a positive cost of complying with norms and punishing their violation, we turn to computational results in Section 4. Section 5 relates our results to conjectures about the likely growth and stability of communities in which norms are more or less pervasive and in which a legal institution that reduces ambiguity about norm violations and increases the informativeness of punishing behavior (by linking disparate rules into a code such that punishment of one rule is informative about the likelihood of punishment of another) exists.
 Overview of Model The basic idea of the model is as follows. Consider an infinitely repeated game setting in which an agent Ego is faced with the choice in each period of participating or sitting out. If choosing to sit out, Ego receives a payoff normalized to 0. If Ego chooses to participate, she plays a randomly selected game g with two randomly selected agents drawn from a population (Others). We model these games in reduced form. In each game, one of the Others is randomly selected and presented with an opportunity to choose between two actions, one which is classified by a classification institution L as "rule violation" and another which is classified as "not rule violation." If Other chooses "rule violation" the remaining Other and Ego each independently choose either to punish or not punish. Rule violations are deterred by collective punishment, that is, punishment that requires more than one agent to punish. For example, in Hadfield and Weingast  [2012]  two buyers and a seller engage in repeated contract and performance games. Actions for the seller are drawn from a set of possible contract performances, some of which are classified as breach and others which are not breach. A decision by a buyer not to purchase from a third-party seller in one period constitutes punishment, specifically a boycott. Breach is deterred when the seller expects both buyers to boycott in response to breach. Games are distinguished by the rule that may be violated. For example, there could be a game in which rule "watch where you are going" may be violated and one in which "men should not wear shorts in the street" may be violated. We assume, and Ego believes, that the community of Others is playing an equilibrium in the super game that consists of the sequence of repeated games. Others are of two types, t: punishers (t = 1), who punish anyone who chooses an action classified by L as a rule violation, and non-punishers (t = 0), who never punish anyone. Let θ be the true proportion of punishers in the equilibrium. We assume that an Other's type is observable by the other participants in any particular game, that is, only in the context of the opportunity for rule violation. We focus on sub-game perfect equilibria in which the knowledge that two punishers are present deters rule violation. 3 (That is, on the off-equilibrium path where a violation does occur in the presence of an Other of type t, punishment is carried out that imposes costs on the violator that exceed the present value of benefits from violation.) We do not model how this equilibrium is generated or supported but we observe that the equilibrium is not destabilized by the presence of non-punishers. We assume, however, that Ego plays as a punisher, bearing an expected cost c in each round. c can be thought of as the cost to Ego of signaling that she is a punisher. For simplicity we assume that Ego is never presented with an opportunity for rule violation.  4  Ego's participation in the game is assumed to be on the margin, with no impact on the equilibrium played by the Others. Ego is able to observe rule violations, the types of Others and punishments in games in which she participates.
 Formal Model Specification Before providing a formalization of our model we provide a brief overview of the theory of Markov decision processes and multi-armed bandits.
 Technical Background and Notation We define a Markov decision process (MDP), M , is a tuple: M = S, A, P, R, δ 5 . S is a set of states. A is a set of actions. P : S × A × S → [0, 1] is a function that assigns probability to state transitions for each state-action pair. If Ego is in state, s, and selects action a the probability of transitioning to s is given by P (s, a, s ). R is a (bounded) reward function that maps states, to an interval of R, w.l.o.g., R : S → [0, 1]. δ ∈ [0, 1) is a discount factor that expresses Ego's preference for current versus future rewards. A solution to M is a policy, π, that maps states to actions, π : S → A. The value of a state, s, under π is the sum of expected discounted rewards received by starting in s and selecting actions according to π: V π (s) = E ∞ t=0 δ t R(s t )|s 0 = s, π . The optimal policy, π * , maximizes this value and we will write V = V π * for the optimal value function. Standard results show that a unique optimal value function exists  [Puterman, 1994] . In a partially observed Markov decision process (POMDP), we additionally define a distribution over observations (O) for a each state. A policy now maps a history of observations to an action, as the agent does not know the true state of the world. A POMDP can always be converted into a (continuous state) MDP where each state is a distribution over states of the world and the transition distribution is defined by Bayesian inference. An interesting class of POMDPs are multi-armed bandits (MAB). In a multi-armed bandit, an agent is given access to several distributions. At each time step, that agent must selects a distribution to sample from, and receives a reward that is equal to the value of the sample. A multi-armed bandit provides an analytically and computationally tractable model of exploration-exploitation tradeoffs that occur with practical agents. In particular, success in this class of problems requires explicit reasoning about the impact of information on future decision making quality. Recent applicability to online banner advertising has led to rapid progress in both theoretical and computational methods for MABs  [Auer et al., 2002] . The optimal full information policy (which knows the distributions of the arms) will always select the arm with the highest mean. A key result from Lai and Robbins  [1985]  lower bounds the number of times an optimal (partial information) policy selects a suboptimal arm in expectation. This result holds for the class of consistent policies: policies where the probability that the optimal arm is chosen at time t approaches 1 as t → ∞. We additionally require a smoothness constraint on the MAB distributions. Proposition 1.  [Lai and Robbins, 1985]  Let Θ be a class of MAB arms (i.e., a class of distributions) with parameter θ. Let µ(θ) be the corresponding mean. If ∀θ ∈ Θ, ∀δ > 0, ∃θ = θ such that µ(θ) ≤ µ(θ ) ≤ µ(θ) + δ then, for any consistent policy the expected number of times a suboptimal arm is selected in the first n rounds is Ω(log n) 6 .
 Model Description We define our super game as a tuple: G, T θ , Π, U, δ, c where G is a distribution over games and T θ is a distribution over punishment types t in the population of Others. We will abuse notation somewhat as use T and G to refer to the support of the corresponding distributions where the meaning is obvious. Π is Ego's prior distribution over the parameters of T θ , and U : G × T θ → R is a mapping from types and games to immediate payoffs for Ego. The understanding is that this mapping represents the results of the Others playing their role in the equilibrium. δ is Ego's discount parameter for future rewards. c expresses a participation cost. This can be understood as the expected cost of to Ego of signaling to an Other that she is a punisher. Ego begins in period 1 with perfect knowledge of how actions are classified by L, all payoffs and the distribution of games. Ego does not know the distribution of types of Others but holds a prior which we will specify shortly. Ego updates her beliefs about the distribution of types using Bayes' rule. The super game is defined as follows: An initial game g 0 ∼ G is drawn. Then, for each period j: 1. Ego chooses whether to participate or not. If she opts out, then she collects 0 payoff and the next round starts. 2. If Ego opts in, she incurs the cost of signaling that she is a punisher c and a type, t j ∼ T θ is for the remaining Other is drawn. If a non-punisher is drawn, a rule violation occurs; if a punisher is drawn, no rule violation occurs. 3. The agent observes whether a punisher is present and whether a rule violation occurs and collects payoff U (g j , t j ). 4. A game g j+1 ∼ G is drawn for the next round. We assume that in equilibrium and given the ruleset created by L, there are two types of games from Ego's perspective: those to which Ego is indifferent and those that Ego cares about. Games to which Ego is indifferent always generate a reward of 0 for Ego. Suppose, for example, that a game involves a rule requiring genuflecting by an Other. We assume Ego realizes no costs or benefits from the Other's choice about whether to genuflect or not, other than the cost of signaling that she is a punisher. We will thus use represent the type of Other with an indicator variable that idicates if she is a punisher for this game. Games Ego cares about are ones in which Ego receives a positive reward, R if there is another punisher present in the game and a negative reward −R if there is not. We call these important games. We formalize the set of important games as follows: G = {g ∈ G|U (g, •) = 0} U (g, t|g ∈ G ) = (2t − 1)R − c. We will use EU = E g,t [U (g, t)|g ∈ G ] to denote the expected utility of an important game. We let s denote the sparsity of the process generating games: the probability that a game is unimportant. s = 1 − P (g ∈ G ); g ∼ G. Critically, we assume that the sparsity of games does not alter the (expected) rate at which important games are presented to Ego. To be concrete, we assume the expected discounted reward obtained from important games is independent of s. This condition can be attained through a suitably modification of δ as a function of s: Proposition 2. Setting δ s = 1 − (1 − s)(1 − δ) ensures that the expected sum of discounted rewards from important games is independent of s: ∀s, ∈ [0, 1) E gj ,tj   ∞ j=0 δ j U (g j , t j ) g j ∈ G   = E gj ,tj   ∞ j=0 I[g j ∈ G ]δ j s U (g j , t j ) s   7 . Proof. We first show that it is sufficient to ensure that the expected value of δ j s is the same given that j is a round with an important game: E gj ,tj   ∞ j=0 I[g j ∈ G ]δ j s U (g j , t j ) s   = ∞ j=0 E gj ,tj I[g j ∈ G ]δ j s U (g j , t j ) s = ∞ j=0 δ j s E gj ,tj [U (g j , t j )|s, g j ∈ G ]E gj [I[g j ∈ G ]|s] = (1 − s)EU ∞ j=0 δ j s Where the first line holds by the linearity of expectation, and the fact that g j , t j are independent iid draws from a stationary distribution. Substituting the form of the infinite geometric series, we see that EU 1 − δ = (1 − s)EU 1 − δ s (1) is sufficient to acheive our goal. Substituting the form for δ s in the theorem statement and reducing shows that this condition is satisfied. It can be easily shows that this model describes a class of MABs. If the parameters that describe equilibrium were known, the decision problem would be trivial. The safe option corresponds to a constant arm, which is a degenerate distribution. Optimal policies for bandits with constant arms exhibit clear structure: is that if it is optimal to choose a known option in round j, it will be optimal to choose the known option in round j + 1 as well  (Bradt et al., 1956) . The argument is straighforward: if Ego reaches a point at which her estimate of the tradeoff between risking a negative payoff and learning so as to improve future decisions leads her optimally to choose not to participate, then her information state can never change and so her optimal choice can never be any different than the current opt-out decision. Thus, in our game, if Ego ever retires in a round, then she will never participate again. We refer to the decision not to participate at any point, then, as a decision to retire. Futhermore, an MAB is an instance of a POMDP, so it the optimal policy maps a distribution over states, a belief state, to decision between retirement and particpation. We give our agent a Beta prior over this parameter so that the belief space for our agent is a two dimensional lattice equivalent to Z 2 + . Initially, the belief state is (α 0 , β 0 ) and can be understood as the state an agent would be in if she had seen α 0 punishers and β 0 non-punishers. The conditional probability that a punisher is present in the first game is p αβ = α α + β . Once the games begin, Ego updates the prior beliefs using Bayes' rule, which in the Beta distribution means adding the counts of punishers and non-punishers observed to the prior values. In the following, we will use α i (β i ) to represent the number of observed punishers (non-punishers) prior to round i. A second useful result from the theory of multi-armed bandits is that the optimal policy is a function that maps a sequence of observations to a decision about retirement. If we restrict to two types, punishers and non-punishers, this problem is a partially observed version of a Markov decision process where the state is the probability, p, of drawing a punisher. From the theory of partially observable Markov decision processes (POMDPs), this optimal policy can also be represented as a mapping from a distribution over p to an action about retirement  [Puterman, 1994] . This reduces a partially observed process to a fully observed deterministic process in belief space.
 The Value of Sparsity Consider first the case in which the participation cost, c, is zero. In this case, Ego only faces a risky choice in periods in which she is presented with an important game. In all other periods, the per-period the expected payoff of playing the risky arm is a constant 0. Thus we can also conclude that if Ego retires, she will retire in a period in which she is playing an important game. In order to maintain the structure of a multi-armed bandit problem, we specify that if Ego chooses not to participate in an important game then the next game in the sequence is also an important game. We can think of this as a suspension of the game.  8  This rules out the possibility that Ego can simply choose not to play an important game and then reenter to play unimportant games in the hope of learning more before the next important game comes along. A decision not to participate is a decision to retire assuming a rational agent. We let i = 1 represent the state in which there is a punisher present in the game. The value of a state is then characterized by the following recursion. To simplify notation we let p αi,βi be the probability that a punisher will be present in the game in the belief state (α i , β i )and we abuse notation somewhat by letting V ((α i , β i ); s, δ) represent the discounted expected value of the super-game with sparsity s and discount factor δ s in the state (α i , β i ). We now show a property of the value of perfect information (VPI) in our super-game. The VPI for a state in a decision processis a measurement of the improvement in decision making as a function of information gathering actions [?]. It is defined as the amount a rational agent is willing to pay to remove all uncertainty associated with a particular random variable. Proposition 3. If the participation cost, c, is 0, then, for any belief state, (α i , β i ), and discount rate δ, the corresponding VPI goes to zero as the sparsity ratio goes to 1. That is lim s→1 V P I((α i , β i ); s, δ) = 0 (2) Proof. Given θ, it is easy to compute the value of participation: V (θ) = EU ∞ t=0 δ t = (2θ − 1) ∞ t=0 δ t = 2θ − 1 1 − δ . (3) The optimal full information policy π 0 will retire whenever V (θ) < 0. We use V + (θ) = max{V (θ), 0} denote the value of π 0 as a function of θ. The VPI is computed as the difference between the expected value of V + and the value of the optimal policy that only uses the history of observations: V P I((α i , β i ); s, δ) = E θ [ V + (θ)| (α i , β i )] − V * ((α i , β i ); s, δ) (4) We proceed by lower bounding V . V is the value of the optimal policy so it is weakly lower bounded by any arbitrary policy. A useful candidate is one-step greedy policy, π g , that always participates for unimportant games and retires in important games if the expected value of participation is negative (disregarding the benefit of new information). We let τ be the random number of games played before an important game is drawn. τ is geometrically distributed with success parameter 1 − s. We let n p be the random number of punishment actions observed prior to drawing an important game. The distribution over n p will be a binomial distribution conditioned on τ . Thus, the value of executing this policy ca be written as a joint expectation under τ and n p : V πg ((α i , β i ); s, δ) = E τ,np [max {E θ [V (θ )|(α i + n p , β i + τ − n p )] , 0} |(α i , β i ), s] . (5) We will be interested in the limit of this value, as s → 1. Before proceeding with that, we note that, from the law of large numbers, we have E θ [V (θ )|a + n p , b + τ − n p ] will concentrate about V (θ). Thus, E θ [ V + (θ )| (α i , β i )] = lim τ →∞ E np [max {E θ [ V (θ )| (α i + n, β i + τ − n p )] , 0} |τ, (α i , β i )] . (6) Note that E[V (θ)|(α i , β i )] only depends on the ratio of (α i , β i ), so the difference between the left and righthand sides of 6 is caused by the fact that the maximum must be taken at finitely many ratios (for finite τ ). Furthermore these ratios are evenly spaced out, so the lefthand side can only increase as τ increases. We can use this to lower bound the limit of V πg : lim s→1 V πg ((α i , β i ); s, δ) = lim s→1 E τ,np [max {E θ [V (θ )|(α i + n p , β i + τ − n p )] , 0} |s] (7) ≥ lim s→1 P (τ ≥ c(s)) min τ ≥c(s) E np [max {E θ [V (θ )|(α i + n p , β i + τ − n p )] , 0} |s, (α i , β i ), τ ] + P (τ ≤ c(s)) min τ <c(s) E np [max {E θ [V (θ )|(α i + n p , β i + τ − n p )] , 0} |s, (α i , β i ), τ ] (8) ≥ lim s→1 P (τ ≥ c(s))E np [max {E θ [ V (θ )| α i + n p , β i + c(s) − n p ] , 0} |c(s), (α i , β i )] (9) Using the form of the cumulative distribution of a geometric variable, P (τ ≥ c(s)) = 1 − P (τ < c(s)) = s c(s) . We set c(s) = − log 1 − s so that lim s→1 c(s) = ∞, and lim s→1 s c(s) = 1. Combining 6 and 9 with these facts allows us to deduce the following: lim s→1 V πgi ((α i , β i ); s, δ) ≥ E θ [V + (θ)|(α i , β i )] (10) Thus, lim s→1 V P I((α i , β i ); s, δ) ≤ 0. However, we have that, for any s, V P I((α i , β i ); s, δ) ≥ 0 by standard properties of VPI. This shows the result. Proposition 4. If the participation cost, c, is zero, then for any (α i , β i ) such that V + ( αi αi+βi ) > 0, VPI is strictly positive for s = 0. Proof. From Lai and Robbins  [1985] , we have that, for consistent and assymptotically efficient policies, the expected number of pulls of a suboptimal arm after n rounds is lower bounded by c log n, where c is a positive constant that measures the similarity of the reward distributions for the arms. This class contains the optimal policy. Thus, we have that for any finite s, V P I((α i , β i ); s, δ) > 0. (11) The combination of these two propositions shows that for any (α i , β i ), the corresponding value will eventually increase as s goes to 1. Thus, in the case where participation costs can be neglected, Ego will prefer an equilibrium with a higher s. shows low confidence initial beliefs. In these information states, Ego has seen very little data and so her belief is very spread out. In these scenarios, we expect sparsity to be helpful because the gap between full information and partial information is large. Conversly, (b) shows belief distributions after Ego has seen 50 effective samples. The corresponding distribution is more concentrated, so we expect less positive impact from sparsity.
 The Cost of Pervasive Normativity: Computational Results Our results above show that an environment with lots of rules that an agent cares nothing about intrinsically is more valuable for an agent contemplating participation than one in which the only rules are ones that matter on the meritsaltering Ego's payoff directly. We assumed, however, that increasing the number of spurious rules is costless to Ego and of course this is not generally likely to be true. If Ego is going to participate in a community with lots of spurious rules, Ego is also likely to bear costs, specifically the cost of participating in collective punishment and the cost of complying with spurious rules. In this section, we relax the assumption that c = 0. Doing so, however, increases the analytical complexity. We therefore turn to computational methods to explore environments in which Ego enjoys both costs and benefits from an increase in the number of spurious rules. To illustrate the effect of participation costs, we select six initial belief states and computed values as a function of c. Our initial beliefs vary the expected value of θ and the variance of the belief about its mean. We selected initial states to cover scenarios where the expected values of an important game is negative, positive and equal to zero. In this work, we chose E[θ] ∈ {.4, .5, .6}. We varied Ego's confidence in her current estimate by varying the effective number of samples (α + β) in the initial belief. Figure  1  shows the corresponding beta distributions for our selection of initial states. The optimal policy is invariant to scaling of rewards, so we fix P = 1 and let c P be the independent variable in our computations. We compute these values with a variant of value iteration that takes advantage of the structure of the state space. A python script to generate these plots is included as appendix A. We set the parameters of our computation to allow for at most 10 −8 of error in the computation. Figure  2 : Plots of V ((α i , β i ); s, .9) for select initial states. The probability density functions that correspond to these beliefs are shown in Fig.  1 . The one step value of participating in an equilibrium with enforcement is 1 (P = 1).The increase in value for larer sparsity for low values of c P shows confirmation of the results in Prop. 3. In comparing vertically, we can see that sparsity has a larger effects on states with low mean and low confidence: (f) is essentially unimpacted by sparsity for low c P while for (a) and (c) participation is suboptimal unless sparsity is positive. The cost of this sparsity is increased sensitivity to participation costs. Figure  2  shows value as a function of c P for several pairs of sparsity and initial states. We can clearly see the costs of pervasive normativity: value functions at higher sparsity decreases more quickly and for lower setting of the participation costs. This occurs because the number of rounds per important game increases so more participation costs are paid. In essence, increased participation costs force Ego to pay more for information and so she may prefer an equilibrium with less sparsity. As we might expect, the net gain in value from sparsity is larger in belief states with that are very uncertain. For example, the value function for initial state (30, 20) is essentially invariant to sparsity for low c P . Similarly, gains from increased sparsity are more pronouced with low means. Essentially all value in those states is due to improved decision making abilities from information.
 Implications for the Microfoundations of Law Our results have surprisingly powerful implications for the structure of normativity in human communities. [LETheory readers: we are only sketching these implications for this draft. We anticipate that at least some of them will be capable of more formal analytic or computational demonstration.]
 Gossip, Silly Rules and Durability The value of participating in a super-game depends on the expected cost of punishing rule violations relative to the reward Ego expects if violations of rules she cares about are deterred. The computations in Section 4 indicate that when expected costs relative to rewards are sufficiently low, ceteris paribus, Ego enjoys higher value in environments with more rules that she does not care about. This provides an interesting explanation for the observation from ethnographic studies that simple societies are characterized both by pervasive and apparently spurious rules that are effectively enforced by low-cost collective punishments.  Wiessner [2005] , for example documents the use of gossip, group criticism and mocking as the principal means by which norms are enforced among the Ju/'hoansi Bushmen of northwest Botswana. In her observations, violations of norms were punished by escalating criticism and rarely got to the point of physical violence. Assuming that the rewards generated for individuals aggregate to raise group well-being, our model thus can be read to predict that communities that succeed in securing an equilibrium with many rules that impose low compliance costs and which are enforced by low-cost means will outperform communities with fewer rules and more costly forms of punishment. 9
 Birds of a Feather The rewards Ego enjoys when joining a community depend on the rules of that community. In comparing across communities with comparable forms of punishment, and comparable numbers of rules, Ego will prefer a community with rules for important games that, when effectively enforced, generate higher rewards, R. We have not modeled the source of rules in a community but if we suppose that rules emerge that reflect the interests of the members of a community, this suggests that Ego is more likely to find rules that achieve higher rewards in communities with a number of agents with similar preferences, that is in more homogeneous communities.
 The Emergence of Law As communities grow more heterogeneous-which they naturally do as they generate value and support greater specialization through the division of labor-we expect ambiguity about rule violations to increase. Increasing ambiguity increases Ego's expected cost of participating in a punishment regime if errors in punishment-failing to punish when a violation is perceived by an Other and punishing when the Other does not perceive a violation-are themselves treated as rule violations which produce punishment. This is a feature of the punishment schemes of many communities: members of the Flemish cloth merchants guild in the 13th Century, for example, included in their rules the provision that any member who failed to observe a boycott of a buyer who had cheated another guild member was barred from trading and no guild member was to "house the goods" or "keep company" with the non-punisher. Increasing ambiguity also reduces the expected rewards Ego enjoys in important games because with some probability a violation perceived by Ego is not perceived as such by Others and hence the probability of a violation increases. Thus the cost of punishing relative to rewards decreases with decreasing ambiguity.  Hadfield and Weingast [2012]  argue that the central function of law is to serve as a unique classification institution, capable of resolving ambiguity about what counts as a rule violation. Moreover, they show that for a classification institution to effectively secure an equilibrium of legal order around a given set of rules enforced only by decentralized collective punishment, the institution must possess legal attributes such as neutrality, openness, clarity, consistency and stability. Our analysis here predicts that communities that introduce law in the form of a classification institution with legal attributes that reduces ambiguity will enjoy higher value and greater durability.
 Hammurabi's Code One of the key assumptions of our model is that people who punish any rule violation are expected to punish all rule violations. This is a distinctive feature of the labeling system generated by a legal regime: people are "law breakers" or not; they are "law-abiding" or not.  Cooter [1998]  proposes that "law" is a meaningful category and that people have preferences over behaviors solely on the basis of whether they are labeled "lawful" or not. Our model captures this idea by treating observation of punishment behavior in the context of any rule as informative of the probability of punishment in important games. The model can be interpreted as representing, for example, a community in which legal order is coordinated around a single legal code. Hammurabi's Code from ancient Babylon, for example, consisted of 247 individual rules, such as "If any one hire an ox or an ass, and a lion kill it in the field, the loss is upon its owner" (Rule 244) and "If any one open his ditches to water his crop, but is careless, and the water flood the field of his neighbor, then he shall pay his neighbor corn for his loss" (Rule 55). These rules likely emerged individually over time. We can imagine that knowing whether someone punished Rule 244 may or may not have helped to predict whether they would also punish Rule 55. But when Hammurabi placed all 247 together on a stone pillar and named the collection as his Code, he created the possibility for the emergence of two types of people: those who punished violations of the Code and those that did not. Our analysis suggests that the creation of collections of rules, rather than disparate rules, can generate value. Suppose, for example, that Ego cares about five rules, enjoying rewards when violations of each of them is deterred. Our model treats these five rules as integrated into a single super game in which the observation of punishment behavior in any game is informative, and equally so, of the likelihood of deterrence of violations in any of the five games Ego cares about. But suppose instead that these rules are not connected in this way. Suppose that punishment behavior in each game Ego cares about is only predicted by punishment behavior in non-overlapping subsets of unimportant games. We could then decompose our single super-game into five distinct super-games, each one of which would be considerably less sparse than our original game. Our results predict that Ego's value in a community with distinct super-games-with disconnected rules and a belief structure about punishment that limits the informativeness of observing punishment of any individual rule-will be lower than the value enjoyed in a community with a comprehensive code.
 Legal Pluralism and the Nation State Finally, we can combine some of the above observations to shed light on the phenomena of legal pluralism and the emergence of the nation state. Modern advanced legal systems are characterized by comprehensive systems of rules, with the label 'law' attributed to any rule generated by a government organized within the constitutional framework of a state. Indeed, law is frequently equated with the rules generated by a government, as distinguished from those generated from other entities such as corporations or schools or that emerge organically from social interaction  [Ellickson, 1991] . But as  Hadfield and Weingast [2013]  emphasize, prior to the emergence of the nation state, there were many institutions that coordinated legal order in different spheres, often in competition. Medieval Europe, for example, was characterized by multiple legal orders, with rules generated by merchant guilds, towns, churches, local rulers and more. Many societies, particularly those seen as struggling to establish the rule of law, are characterized by multiple legal orders-some governing family relations, others governing commercial dealings for example. Our model suggests a way of thinking about the tradeoffs Ego will face between participating in multiple legal communities, each of which coordinates punishment over some subset of rules and participating in a comprehensive legal community. On the one hand, rewards may be higher when a legal community is comprised of a relatively homogeneous group with shared interests-such as a community of traders-who can select rules that serve Ego's interests. But such a community will also have lower sparsity. On the other hand a system with a single system of comprehensive rules may achieve less alignment with Ego's interests but may also, because of its higher sparsity, provide Ego with more information about the value of continuing to participate.  Figure 1 : 1 Figure 1: Plots of a selection of initial belief distributions. Each curve corresponds to a different value of E[θ]. (a)
 fig = plt.figure() ax = fig.add_subplot(1, 1, 1) ax.set_xscale('log') ax.set_xlabel('c/P') ax.set_ylabel('Value')
			 Hadfield [1999]  shows that there is functionality to an arbitrary classification scheme of enforced norms: it coordinates investments in specialized learning by gender which raises the value of heterosexual matching in the marriage market.
			 Cooter [1998]  also suggests that the effectiveness of law can be understood as deriving from the classification of behavior as lawful or not. Cooter however presumes the existence of preferences based on this simple binary classification, that is, that at least some people inherently prefer to avoid actions labeled unlawful. This presumes that a category "law", which extends to potentially arbitrary actions, exists.
			 For an example of such a game, see Boyd et al. [2010] . They present an evolutionary game model in which punishment is a heritable strategy and deterrence requires multiple punishers. A population with a fraction of punishers can be stable in equilibrium when punishers can signal that they are punishers at low cost and thus avoid the costs of punishment if there are too few punishers present.4  It is straightforward to generalize our interpretation of c as the cost of complying with spurious rules to signal Ego's support for the equilibrium rules.5  In standard treatments, T is typically used for the transition distribution, we use P here to avoid confusion with our model specification.
			 If the function g(n) is Ω(f (n)), then there is a positive constant c such that g(n) ≥ cf (n)∀n
			 In any multi-armed bandit game, the decision to stop suspends the game: deciding to return to the game implies making the risky pull that was rejected previously.
			 Our model takes into account Ego's willingness to bear the higher cost of punishment in more sparse environments. We assume, but have not shown, that the willingness of Others to punish is not reduced with sparsity-that is, that Others have incentives comparable to Ego's.


Abstract: AI-safety efforts focused on suffering reduction should place particular emphasis on avoiding risks of astronomical disvalue. Among the cases where uncontrolled AI destroys humanity, outcomes might still differ enormously in the amounts of suffering produced. Rather than concentrating all our efforts on a specific future we would like to bring about, we should identify futures we least want to bring about and work on ways to steer AI trajectories around these. In particular, a "fail-safe" 1 approach to AI safety is especially promising because avoiding very bad outcomes might be much easier than making sure we get everything right. This is also a neglected cause despite there being a broad consensus among different moral views that avoiding the creation of vast amounts of suffering in our future is an ethical priority.

Suffering-focused AI safety: Why "fail-safe" measures might be our top intervention

Classical AI safety looks very difficult. In order to ensure a truly "utopian" outcome, the following things need to happen: 1. The AI that gains a decisive strategic advantage over the competition needs to be built by the right group of people. 2. These people would need to have figured out how to program favorable values into an AI (or program the AI to learn these values). 3. They would also need to have come up with a satisfying description of what these values are, either directly, or indirectly with the help of a suitable extrapolation procedure (e.g.  Muehlhauser & Helm, 2012) . 4. Finally, the creators of the first superintelligence might need to get decision theory right (see  Soares & Fallenstein, 2015)  and safeguard the AI from a lot of hardto-anticipate failure modes. Succeeding with all of this together is the only safe way of bringing about a flourishing utopia mostly free of suffering. (There might be other paths to utopia, provided enough luck.) But what about outcomes where something goes wrong? It is important to consider that things can go wrong to very different degrees. For value systems that place primary importance on the prevention of suffering, this aspect is crucial: the best way to avoid bad-case scenarios specifically may not be to try and get everything right. Instead, it makes sense to focus on the worst outcomes (in terms of the suffering they would contain) and on tractable methods to avert them. As others are trying to shoot for a best-case outcome (and hopefully they will succeed!), it is important that some people also take care of addressing the biggest risks. This perspective to AI safety is especially promising both because it is currently neglected and because it is easier to avoid a subset of outcomes rather than to shoot for one highly specific outcome. Finally, it is something that people with many different value systems could get behind.
 How AI outcomes might contain suffering From a suffering-focused perspective, the main reason to be concerned about the risks from artificial intelligence is not the possibility of human extinction or the corresponding failure to build a flourishing, intergalactic civilization. Rather, it is the thought of misaligned or "illaligned" AI as a powerful but morally indifferent optimization process which, in the pursuit of its goals, may transform galactic resources into (among other things) suffering. Either like the suffering we see on Earth today, or by bringing about optimized structures that possibly also contain novel forms of suffering. The things a superintelligent AI would build to pursue its goals might include a fleet of "worker bots", factories, supercomputers, space colonization machinery, etc. It is possible that all of these end up being built in a way that does not permit suffering. However, given that evolution has produced plenty of sentient minds, this weakly suggests that some of the easiest ways to implement mind architectures do come with the capacity for suffering. In the absence of an explicit antisuffering preference, even the slightest benefit to the AI's objectives would lead to the instantiation of suffering minds. What makes this especially worrying is that the stakes involved will be huge: Space colonization is an attractive subgoal for almost any powerful optimization process, as it leads to control over the largest amount of resources  (Omohundro, 2008) . Even if only a small portion of these resources were used for purposes that include suffering, the resulting disvalue would sadly be astronomical. To find the best interventions to prevent suffering, it is crucial to study how AI outcomes differ in their expected amount of suffering and whether there are ways to favorably affect the likely outcomes. This paper aims to provide an overview on "bad-case scenarios" in the AI context and some general suggestions for how they could be avoided. Future work will include checking fundamental assumptions and looking at the most promising proposals in more detail.
 Type I: Controlled AI gone bad In these scenarios, the team that builds the first superintelligence knew what it was doing and managed to successfully shape the resulting AI's goals exactly the way they wanted it to go. Unfortunately, the goals are not what we would wish them to be and thus lead to bad or very bad consequences. Things that could go wrong include: a) Anthropocentrism: perhaps the resulting AI would not care about the suffering of "weird" and/or voiceless nonhuman minds such as suffering subroutines or animal minds in ancestor simulations. b) Retributivism: perhaps the AI gets built by people who want to punish members of an outgroup (e.g. religious fundamentalists punishing sinners). c) Uncooperative: perhaps the AI's goal is something like classical utilitarianism (or any other "monotone" maximizing function) with no additional regards for cooperation with other value systems. Even though such an AI would all else equal prefer to not create suffering, it seems possible that the anti-suffering concern would in practice be overridden by opportunity costs: if happiness simulations contain much more utility for the AI than the disutility produced by the accidental suffering created in the buildup, then such an AI would behave similarly "recklessly" as e.g. a paperclipmaximizing AI. 2 Another concern lies in how strongly the AI would value robustness to deterioration. d) Libertarianism regarding computations: perhaps the creators of the first superintelligence instruct the AI to give every human alive at the time of the singularity control of a planet or galaxy, with no additional rules to govern what goes on within those territories. Some of these human rulers are curious and amused by seeing others hurt, or worse, might be psychopaths.
 Type II: "Near misses" In this scenario, the first superintelligence is built by people who were aware of the risks, and who tried to get things right. Unfortunately, mistakes happened and the resulting outcome with "nearly-controlled AI" is bad or very bad. a) Miserable creatures: perhaps the AI's goal function includes terms that attempt to specify sentient or human-like beings and conditions that are meant to be good for these beings. However, because of programming mistakes, unanticipated loopholes or side-effects, the conditions specified actually turn out to be bad for these beings. Worse still, the AI has a maximizing function and wants to fill as many regions of the universe as possible with these poor creatures. b) Black swans: perhaps the AI cares about sentient or human-like minds in "proper" ways, but has bad priors, ontology, decision theory, or other fundamental con-2 An AI that values happiness simulations but takes cooperation with suffering reducers into account would in expectation still create some amount of suffering, but depending on just how costly it is to use more "sufferingproof" algorithms in the colonization process, or to do fewer or less fine-grained ancestor simulations, it might be possible to cut down on most of the instrumentally produced suffering at a cost that isn't very high (e.g. not higher than 15% of total happiness simulations produced). stituents that would make it act in unfortunate and unpredictable ways.
 Type III: Uncontrolled AI Uncontrolled AI is stipulated to have goals that were never intended by its creators, or at least were not intended to take over control in the form of a singleton  (Bostrom, 2006) . Unless the AI in question has a neuromorphic design, it is unlikely that its goals would (directly) have something to do with sentient beings. However, even non-neuromorphic uncontrolled AI might instrumentally instantiate suffering minds in the process of achieving its goal, as a side-effect of the computations it performs. The following presents a list of ways how uncontrolled AI might create vast amounts of suffering: • Suffering subroutines: For tasks like inventing and developing advanced technologies, as well as for coordinating its expansion into space for resource accumulation, the AI would have to rely on a fleet of "robot workers" or "robot scientists" of various kinds (which, of course, might work and function very unlike human workers or scientists). It is possible that these processes would be capable of suffering  (Tomasik, 2014) , although it is unclear whether would perform at their best if they (occasionally) suffer. What is clear is that if they do suffer under useful circumstances, an AI that is not explicitly aligned with compassionate goals would not have any qualms about instantiating astronomical numbers of them. • Ancestor simulations  (Bostrom, 2003; Bostrom, 2014, Ch.8) : in order to gather information relevant to its goals, e.g. for improving its understanding of human psychology or sociology  (Bostrom, 2014, pp. 125-26)  or for studying the density of aliens in the universe and what their likely values are, a superintelligence might simulate many runs of Darwinian evolution in planet-sized supercomputers  (Sandberg, 1999) . It is possible that these simulations would be fine-grained enough to contain sentient minds. • Warfare: it may be that the density of life in the universe is high enough for colonizing AIs to eventually encounter one another. If so, uncontrolled AI might end up clashing with other superintelligences, including AIs that build expanding civilizations of happy sentient beings. Perhaps they could agree to a compromise, but such an encounter could also lead to fighting over resources and warfare at the points of contact. It is interesting to note that none of the aforementioned scenarios involve large quantities of suffering in the types of structures that the AI directly optimizes for. It seems likely that what makes uncontrolled AI bad is what it builds for instrumental reasons, including colonization machinery, science simulations, and other strategic computations. This suggests that opportunity costs are relevant: If an uncontrolled AI's goal function does not have diminishing returns on resources (like the infamous "paperclip maximizer"), then the room for instrumentally important computations might be small. By contrast, an AI with goals that are easy to fulfill 3 -e.g. a "paperclip protector" that only cares about protecting a single paperclip, or an AI that only cares about its own reward signal -would have much greater room pursuing instrumentally valuable computations. This line of thought suggests that outcomes from uncontrolled AI should be steered away from "paperclip-protector types." 4 3 Suffering-focused AI safety: Some proposals Bad-case scenarios can be avoided in several ways. Using the typology from above, we can distinguish three classes of interventions: • Influencing outcomes with controlled AI: influence the values implemented in outcomes with controlled AI either through value spreading or by supporting the AI safety project with the best values The following can be labelled "fail-safe" measures: • Targeted AI safety: a) Differential progress: push classical AI safety differentially in ways that best insulates it from the worst "near misses" b) AI safety where it is most needed: identify the paradigm in general AI that would create the worst outcomes in the event of a control failure, and work towards identifying (general) control or shut-down mechanisms in those areas • Graceful fails: research ways to make AI fail in a "benign" way conditional on it failing ("It might be the end of the world, but it could have been (much) worse") The following sections present some preliminary ideas for the type of interventions that could be done in those areas.
 Influencing outcomes with controlled AI This class of interventions is primarily targeted at "controlled AI gone bad" outcomes. It is something FRI and others in the EA community have already thought about a great deal. • Improve values: raise awareness of antispeciesism, anti-substratism, concern for weird minds or suffering-focused ethics in general. • Cooperation: promote the idea that the gains from cooperation make it important for different value-systems to work together. • Cooperative AI-safety efforts: by funding the AI-safety project with the best values or the best approach to cooperation, we make it more likely that controlled AI won't create vast amounts of suffering.
 Differential progress in AI safety This intervention is targeted at the worst outcomes in the "near misses" category. The goal is to figure out solutions to the worst problems first, such that any additional progress in AI safety is unlikely to lead to "near-misses," which may be worse than outcomes with uncontrolled AI in terms of suffering produced. This intervention seems comparatively more neglected than interventions in the category above. • Secure value implementation: Look into the pros and cons of various approaches to specifying an AI's values and work on the one least likely to lead to really bad outcomes. Questions to study may include: • What's more likely to go wrongsomething like indirect normativity, or an explicit specification of a goal function? • What are other promising approaches to AI safety, and how bad would the outcome be if they fail according to their weak points? • Would corrigibility reduce risks of astronomical suffering? • Foundational research: Which unanticipated failure modes could there be for controlled AI? Once they are identified, how can we prevent them?
 AI safety where it is most needed This intervention is targeted at preventing the worst outcomes involving uncontrolled AI. Not all uncontrolled AIs are expected to behave equally; there are different paradigms in AI research, i.e. different basic architectures for attempting to build smarter-than-human intelligence. Examples include reinforcement learning  (Sutton & Barto, 1998) , proof-based seed AI, and neuromorphic AIs  (Hasler & Marr, 2013) . Classical AI safety is necessarily concerned with the tradeoff between success probability and "controllability;" if a particular approach to building AI is likely to lead to superintelligence, but in a way where it is hard to control the values of the resulting agent, then this poses a likely dead end. By contrast, a suffering-focused "fail-safe" approach does not have to worry about controllability to the same extent, as long as there's enough controllability to predictably avoid outcomes with the most suffering. This suggests that, conditional on uncontrolled AI being worse than (nearly-)controlled AI in expectation, suffering reducers can be less constrained by issues with controllability, and can thus focus more on applying AI safety to the paradigm/architecture with the highest probability of bringing about superintelligence. In addition, we could try to identify differences among the types of uncontrolled AI each AI-paradigm would produce, in order to then focus on AI safety in the domain where failures of control result in the worst outcomes. Suppose for instance that one way of building AI appears to show promising results and rapid progress, yet control seems hard and likely failure modes are particularly worrying. In this case, we should consider putting efforts into AI safety applied to that particular domain. Specific examples of this more general idea include: • AI safety for machine learners: With DeepMind's recent success, it seems not unlikely that the first superintelligent AI may be all reinforcement learning as the top control structure (as opposed to GO-FAI, theorem proving or logic-based AI). In contrast to AI architectures where goals refer to the state of the world  (Hibbard, 2011)  rather than to internal or observational signals, pure reinforcement learners may be at a higher risk of wireheading, i.e. of hacking their input signals to continuously attain the maximum reward. If the wireheading AI is smart enough to plan into the future, it might patiently avoid doing so and develop plans for takeover first: As Bostrom notes in Superintelligence  (Bostrom, 2014, 122-23) , smart wireheaders would still want to colonize other galaxies for option value, protection, research, etc. However, unlike paperclip maximizers, wireheaders would more generally function like "paperclip protectors" in that they would have almost no opportunity costs  (Ring & Orseau, 2011) . If the observation that lower opportunity costs make outcomes with uncontrolled AI worse is accurate (see the discussion under "Type III: Uncontrolled AI), this would make AI safety applied to reinforcement learning architectures particularly promising. Specific interventions could include work on Interruptibility  (Armstrong & Orseau, 2016) , but also methods of value-loading (examples here and here). • AI safety for "em-first" scenarios: Conditional on whole brain emulation  (Sandberg & Bostrom, 2008; Hanson, 2016)  becoming technologically feasible before the advent of de-novo AI, de-novo AI as eventually developed will more likely exhibit a neuromorphic design. 5 This would make it more likely that the values are humanlike, which also comes with the possi-bility of "bad values" or "near misses." AI safety applied to "em-first" scenarios could thus be a promising intervention (although with early arrival timelines being longer, it may not be favored by haste considerations). Caveat: Pursuing any of the interventions above would require that we make sure to not speed up progress in AI development -both in general and in "unsafe" or "hard-to-control" approaches specific to those interventions.
 Graceful fails This class of interventions is targeted at preventing the worst outcomes involving "nearlycontrolled" or uncontrolled AI. Instead of trying to make fails less likely, the idea is to come up with safety nets or mechanisms such that, if control fails, the outcome will be as good as it gets under the circumstances. Graceful fails are easiest to bring about in the domain of near misses, as the researchers working on AI safety are aware of the risks and thus open to the idea of safety nets. • Backup goal functions: Research ways to implement multi-layered goal functions, with a "backup goal" that kicks in if the implementation of the top layer does not fulfill certain safety criteria. The backup would be a simpler, less ambitious goal that is less likely to result in bad outcomes. Difficulties would lie in selecting the safety criteria in ways that people with different values could all agree on, and in making sure that the backup goal gets triggered under the correct circumstances. In the context of AI research that is not guided by safety precautions, graceful fails proposals seem harder to implement. The challenge is to get an AI project that might result in uncontrolled AI to care about whatever proposal there is to implement. Perhaps there are proposals to consider in rather early stages of AI development where researchers think hard takeoffs are very unlikely. Should the AI design in question nevertheless undergo an intelligence explosion, there might be ways to make the result counterfactually less bad. • Dummy goals: We could come up with a goal function with the following properties: • Easy to program • Interesting (in the sense that it's easy to create problems for an AI with this goal function to solve, such that researchers can track their progress on general intelligence) • Leads to decent outcome in the unlikely case of hard takeoff If AI research and testing at early stages is always conducted with goals of this type, an unanticipated hard takeoff would at least be comparatively benign.
 Concluding thoughts Suffering-focused AI safety, and "fail-safe" measures in particular, make up a large and promising area of interventions for FRI (and perhaps other organizations) to investigate further. Advantages are that they are neglected and often "less ambitious" than classical AI safety, which means they might end up being more tractable. Moreover, "fail-safe" measures are a promising project to focus on because the interventions discussed are positive or at worst "unobjectionable" from the perspectives of virtually all other value systems. The main reason we had not explicitly zoomed in on this category of interventions earlier was that FRI was initially too focused on answering the more general, complicated question whether AI safety on the whole is net positive for suffering reducers. Upon reflection, this question may be less important. AI safety is a very broad category where we should expect a lot of room for targeted efforts. The general lesson to draw is that partitioning broad categories can be an important step towards making progress. More research into the feasibility of suffering-focused AI safety carries high information value for this very reason: As we look into more proposals and their practical feasibility, we might uncover more relevant distinctions among AI outcomes, valuetransfer proposals, or general approaches to AI -a process which, in turn, will make it easier to identify effective interventions. Influencing outcomes with controlled AI . . . . . . . . . . . . . . . . . . . . . . 3.2 Differential progress in AI safety . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 AI safety where it is most needed . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Graceful fails . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Introduction Contents 1 Introduction 2 How AI outcomes might contain suffering 2.1 Type 3 Suffering-focused AI safety: Some proposals 3.1 4 Concluding thoughts I: Controlled AI gone bad . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Type II: "Near misses" . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Type III: Uncontrolled AI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
			 "Fail-safe" in the sense that if control fails, at least the AI causes less suffering than would have been the case without fail-safe measures.
			 Thanks to Carl Shulman for bringing this idea to our attention. 4 However, it may be that paperclip protectors are easier to compromise with, and thus more likely to also pursue maximization of sorts in exchange for benefits from "paperclip-maximizer types."
			 If only because neuroscience research would be very advanced in such a world.