Abstract: In his paper "The Superintelligent Will," Nick Bostrom formalized the Orthogonality thesis: the idea that the final goals and intelligence levels of artificial agents are independent of each other. This paper presents arguments for a (narrower) version of the thesis. It proceeds through three steps. First it shows that superintelligent agents with essentially arbitrary goals can exist in our universeboth as theoretical impractical agents such as AIXI and as physically possible realworld agents. Then it argues that if humans are capable of building human-level artificial intelligences, we can build them with an extremely broad spectrum of goals. Finally it shows that the same result holds for any superintelligent agent we could directly or indirectly build. This result is relevant for arguments about the potential motivations of future agents: knowing an artificial agent is of high intelligence does not allow us to presume that it will be moral, we will need to figure out its goals directly.

GENERAL PURPOSE INTELLIGENCE: ARGUING THE ORTHOGONALITY THESIS

The Orthogonality Thesis Scientists and mathematicians are the stereotypical examples of high intelligence humans. But their morality and ethics have been all over the map. On modern political scales, they can be left-(Oppenheimer) or right-wing (von Neumann) and historically they have slotted into most of the political groupings of their period (Galois, Lavoisier). Ethically, they have ranged from very humanitarian (Darwin, Einstein outside of his private life), through amoral (von Braun) to commercially belligerent (Edison) and vindictive (Newton). Few scientists have been put in a position where they could demonstrate genuinely evil behavior, but there have been a few of those  (Teichmüller, Philipp Lenard, Ted Kaczynski, Shirō Ishii) . Of course, many scientists have been absolutely conventional in their views and attitudes given the society of their time. But the above examples hint that their ethics are not strongly impacted by their high intelligence; intelligence and ethics seem 'orthogonal' (varying independently of each other, to some extent). If we turn to the case of (potential) artificial intelligences we can ask whether that relation continues: would high intelligence go along with certain motivations and goals, or are they unrelated? To avoid the implicit anthropomorphisation in terms such as 'ethics,' we will be looking at agents 'final goals' -the ultimate objectives they are aiming for. Then the Orthogonality thesis, due to Nick Bostrom  (Bostrom, 2012) , states that: Intelligence and final goals are orthogonal axes along which possible agents can freely vary. In other words, more or less any level of intelligence could in principle be combined with more or less any final goal. It is analogous to Hume's thesis about the independence of reason and morality  (Hume, 1739) , but applied more narrowly, using the normatively thinner concepts 'intelligence  ' and 'final goals' rather than 'reason' and 'morality'.  But even 'intelligence,' as generally used, has too many connotations. A better term would be efficiency, or instrumental rationality, or the ability to effectively solve problems given limited knowledge and resources  (Wang, 2011) . Nevertheless, we will be sticking with terminology such as 'intelligent agent,' 'artificial intelligence' or 'superintelligence,' as they are well established, but using them synonymously with 'efficient agent,' artificial efficiency' and 'superefficient algorithm.' The relevant criteria is whether the agent can effectively achieve its goals in general situations, not whether its inner process matches up with a particular definition of what intelligence is. Thus an artificial intelligence (AI) is an artificial algorithm, deterministic or probabilistic, implemented on some device, that demonstrates an ability to achieve goals in varied and general situations.  1  We don't assume that it need be a computer program, or a well laid-out algorithm with clear loops and structures -artificial neural networks or evolved genetic algorithms certainly qualify. A human level AI is defined to be an AI that can successfully accomplish any task at least as well as an average human would (to avoid worrying about robot bodies and such-like, we may restrict the list of tasks to those accomplishable over the internet). Thus we would expect the AI to hold conversations about Paris Hilton's sex life, to compose ironic limericks, to shop for the best deal on Halloween costumes and to debate the proper role of religion in politics, at least as well as an average human would. A superhuman AI is similarly defined as an AI that would exceed the ability of the best human in all (or almost all) tasks. It would do the best research, write the most successful novels, run companies and motivate employees better than anyone else. In areas where there may not be clear scales (what's the world's best artwork?) we would expect a majority of the human population to agree the AI's work is among the very best. Nick Bostrom's paper argued that the Orthogonality thesis does not depend on the Humean theory of motivation, but could still be true under other philosophical theories. It should be immediately apparent that the Orthogonality thesis is related to arguments about moral realism. Despite this, we will not address the fertile and extensive literature on this subject. Firstly, because it is contentious: different schools of philosophical thought have different interpretations of the truth and meaning of moral realism, disputes that cannot be currently resolved empirically. Since we are looking to resolve a mainly empirical question -what systems of motivations could we actually code into a putative AI -this theoretical disagreement is highly problematic. Secondly, we hope that by approaching the issue from the computational perspective, we can help shed new light on these issues. After all, we do not expect that the trigger mechanism of a cruise missile to block detonation simply because people will die -but would an "ultra-smart bomb" behave the same way? By exploring the goals of artificial systems up to higher level of efficiency, we may contribute to seeing which kinds of agents are susceptible to moral realism arguments, and which are not. Thus this paper will content itself with presenting direct arguments for the Orthogonality thesis. We will assume throughout that human level AIs (or at least human comparable AIs) are possible (if not, the thesis is void of useful content). We will also take the position that humans themselves can be viewed as non-deterministic algorithms: 2 this is not vital to the paper, but is useful for comparison of goals between various types of agents. We will do the same with entities such as committees of humans, institutions or corporations, if these can be considered to be acting in an agent-like way. The thesis itself might be critiqued for over-obviousness or triviality -a moral anti-realist, for instance, could find it too evident to need defending. Nevertheless, the argument that AIs -or indeed, any sufficiently intelligent being -would necessarily behave morally is a surprisingly common one. A. Kornai, for instance, considers it as a worthwhile starting point for investigations into AI morality  (Kornai, 2013) . He bases his argument on A. Gewith's approach in his book, Reason and Morality  (Gewirth, 1978)  (the book's argument can be found in a summarized form in one of E. M. Adams's papers  (Adams, 1980) ) in which it is argued that all agents must follow a "Principle of Generic Consistency" that causes them to behave in accordance with all other agent's generic rights to freedom and well-being. Others have argued that certain specific moralities are attractors in the space of moral systems, towards which any AI will tend if they start off with certain mild constraints  (Waser, 2008) . Because of these and other examples (and some online criticism of the Orthogonality thesis 3 ), we thought the thesis was worth defending explicitly, and that the argument brought out in its favor would be of general interest to the general discussion.
 Qualifying the Orthogonality Thesis The Orthogonality thesis, taken literally, is false. Some motivations are mathematically incompatible with changes in intelligence ("I want to prove the Gödel statement for the being I would be if I were more intelligent"). Some goals specifically refer to the intelligence of the agent, directly ("I want to be much less efficient!") or indirectly ("I want to impress people who want me to be much less efficient!"). Though we could make a case that an agent wanting to be less efficient could initially be of any intelligence level, it won't stay there long, and it's hard to see how an agent with that goal could have become intelligent in the first place. So we will exclude from consideration those goals that intrinsically refer to the intelligence level of the agent. We will also exclude goals that are so complex or hard to describe that the complexity of the goal becomes crippling for the agent. If the agent's goal takes five planets worth of material to describe, or if it takes the agent twenty years each time it checks what its goal is, then it's obvious that that agent can't function as an intelligent being on any reasonable scale. Many have made the point that there is likely to be convergence in instrumental goals  (Omohundro, 2008) . Whatever their final goals, it would generally be in any agent's interest to accumulate more power, to become more intelligence and to be able to cooperate with other agents of similar ability (and to have all the negotiation, threatening and cajoling skills that go along with that cooperation). Note the similarity with what John Rawls called 'primary goods'  (Rawls, 1971 ). We will however be focusing exclusively on final goals, as the instrumental goals are merely tools to accomplish these.  4  Further we will not try to show that intelligence and final goals can vary freely, in any dynamical sense (it could be quite hard to define this). Instead we will look at the thesis as talking about possible states: that there exist agents of all levels of intelligence with any given goals. Since it's always possible to make an agent stupider or less efficient, what we are really claiming is that there could exist possible high-intelligence agents with any given goal. Thus the restricted Orthogonality thesis that we will be discussing is:
 High-intelligence agents can exist having more or less any final goals (as long as these goals are of feasible complexity, and do not refer intrinsically to the agent's intelligence). 5 We will be looking at two variations of the "can exist" clause: whether the agent can exist in theory, and whether we could build such an agent (given that we could build an AI at all). Though evidence will be presented directly for this thesis in the theoretic agent case, the results of this paper cannot be considered to "prove" the thesis for agents we could build (though they certainly raise its likelihood). In that case, we will be looking at proving a still weaker thesis:
 The fact of being of high intelligence provides extremely little constraint on what final goals an agent could have (as long as these goals are of feasible complexity, and do not refer intrinsically to the agent's intelligence). That thesis still has nearly all the relevant practical implications that the strong Orthogonality thesis does.
 Orthogonality in Practice for AI Designers The arguments presented in this paper are all theoretical. They posit that AIs with certain goals either 'can exist,' or that 'if we could build an AI, we could build one with any goal'. In practice, the first AIs, if and when they are created, will be assembled by a specific team, using specific methods, and with specific goals in mind. They may be more or less successful at inculcating the goals into the AI (or, as is common in computer programming, they may inculcate the goals exactly, only to realize later that these weren't the goals they really wanted). The AI may be trained by interacting with certain humans in certain situations, or by understanding certain ethical principles, or by a myriad of other possible methods, which will likely focus on a narrow target in the space of goals. The relevance of the Orthogonality thesis for AI designers is therefore mainly limited to a warning: that high intelligence and efficiency are not enough to guarantee positive goals, and that they thus need to work carefully to inculcate the goals they value into the AI.
 Orthogonality for Theoretic Agents If we were to step back for a moment and consider, in our mind's eyes, the space of every possible algorithm, peering into their goal systems and teas-ing out some measure of their relative intelligences, would we expect the Orthogonality thesis to hold? Since we are not worrying about practicality or constructability, all that we would require is that for any given goal system (within the few constraints enumerated above), there exists a theoretically implementable algorithm of high intelligence. Any measurable  6  goal can be paired up with a reward signal: an agent gets a reward for achieving states of the world desired by the goal, and denied these rewards for actions that fail to do so. Among reward signal maximisers, the AIXI is the theoretically best agent there is, more successful at reaching its goals (up to a finite constant) than any other agent  (Hutter, 2005) . AIXI itself is incomputable, but there are computable variants such as AIXItl or Gödel machines  (Schmidhuber, 2007)  that approximate AIXI's efficiency. These methods work for whatever reward signal plugged into them. Or we could simply imagine a supercomputer with arbitrarily large amounts of computing power and a decent understanding of the laws of physics (a 'Laplace demon'  (Laplace, 1814)  capable of probabilistic reasoning), placed 'outside the universe' and computing the future course of events. Paired with an obedient active agent inside the universe with a measurable goal, for which it would act as an advisor, this would also constitute an 'ultimate agent.' Thus in the extreme theoretical case, the Orthogonality thesis seems true. There is only one problem with these agents: they are either impossible in practice (AIXI or Laplace's demon), or require incredibly large amounts of computing resources to work. Let us step down from the theoretical pinnacle and require that these agents could actually exist in our world (still not requiring that we be able or likely to build them). An interesting thought experiment occurs here. We could imagine an AIXI-like super-agent, with all its impractical resources, that is tasked to design and train an AI that could exist in our world, and that would accomplish the super-agent's goals. Using its own vast intelligence, the superagent would therefore design a constrained agent maximally effective at accomplishing those goals in our world. Then this agent would be the highintelligence real-world agent we are looking for. It doesn't matter than the designer is impossible in practice -if the super-agent can succeed in the theoretical thought experiment, then the trained AI can exist in our world. This argument generalizes to other ways of producing the AI. Thus to deny the Orthogonality thesis is to assert that there is a goal system G, such that, among other things: 1. There cannot exist any efficient real-world algorithm with goal G. 2. If a being with arbitrarily high resources, intelligence, time and goal G, were to try design an efficient real-world algorithm with the same goal, it must fail. 3. If a human society were highly motivated 7 to design an efficient realworld algorithm with goal G, and were given a million years to do so along with huge amounts of resources, training and knowledge about AI, it must fail. 4. If a high-resource human society were highly motivated to achieve the goal G, then it could not do so (here the human society itself is seen as the algorithm). 5. Same as above, for any hypothetical alien societies. 6. There cannot exist any pattern of reinforcement learning that would train a highly efficient real-world intelligence to follow the goal G. 7. There cannot exist any evolutionary or environmental pressures that would evolve a highly efficient real world intelligences following goal G. All of these seem extraordinarily strong claims to make! The last claims all derive from the first, and merely serve to illustrate how strong the first claim actually is. Claim 4, in particular, seems to run counter to everything we know about human nature.
 Orthogonality for Human-level AIs Of course, even if efficient agents could exist for all these goals, that doesn't mean that we could ever build them, even if we could build AIs. In this section, we'll look at the ground for assuming the Orthogonality thesis holds for human-level agents. Since intelligence isn't varying much, the thesis becomes simply: If we could construct human-level AIs at all, then there is extremely little constraint on the final goals that such AIs could have (as long as these goals are of feasible complexity, and do not refer intrinsically to the agent's intelligence). So, is this true? The arguments in this section are generally independent of each other, and can be summarized as: 1. Some possible AI designs have orthogonality built right into them. 2. AI goals can reach the span of human goals, which is large. 3. Algorithms can be combined to generate an AI with any easily measurable goal. 4. Various algorithmic modifications can be used to further expand the space of possible goals, if needed.
 Utility Functions One classical picture of a rational agent is of an agent with a specific utility function, which it will then act to maximize in expectation. This picture encapsulates the Orthogonality thesis: whatever the utility function, the rational agent will then attempt to maximize it, using the approaches in all cases (planning, analyzing input data, computing expected results). If an AI is built according to this model, with the utility function being prescriptive (given to the AI in a program) rather than descriptive (an abstract formalization of an agent's other preferences), then the thesis would be trivially true: we could simply substitute the utility function for whichever one we desired. However, many putative agent designs are not utility function based, such as neural networks, genetic algorithms, or humans. So from now on we will consider that our agents are not expected utility maximizers with clear and separate utility functions, and look at proving Orthogonality in these harder circumstances.
 The Span of Human Motivations It seems a reasonable assumption that if there exists a human being with particular goals, and we can program an AI, then we can construct a humanlevel AI with similar goals. This is immediately the case if the AI was a whole brain emulation/upload  (Sandberg & Bostrom, 2008) , a digital copy of a specific human mind. Even for more general agents, such as evolved agents, this remains a reasonable thesis. For a start, we know that real-world evolution has produced us, so constructing human-like agents that way is certainly possible. Human minds remain our only real model of general intelligence, and this strongly directs and informs our AI designs, which are likely to be as human-similar as we can make them. Similarly, human goals are the easiest goals for us to understand, hence the easiest to try and implement in AI. Hence it seems likely that we could implement most human goals in the first generation of human-level AIs. So how wide is the space of human motivations 8 ? Our race spans footfetishists, religious saints, serial killers, instinctive accountants, role-players, self-cannibals, firefighters and conceptual artists. The autistic, those with exceptional social skills, the obsessive compulsive and some with splitbrains. Beings of great empathy and the many who used to enjoy torture and executions as public spectacles. 9 It is evident that the space of possible human motivations is vast. 10 For any desire, any particular goal, no matter how niche, 11 pathological, bizarre or extreme, as long as there is a single human who ever had it, we could build and run an AI with the same goal. But with AIs we can go even further. We could take any of these goals as a starting point, make them malleable (as goals are in humans), and push them further out. We could provide the AIs with specific reinforcements to push their goals in extreme directions (reward the saint for ever-more saintly behavior). If the agents are fast enough, we could run whole societies of them with huge varieties of evolutionary or social pressures, to further explore the goal-space. We may also be able to do surgery directly on their goals, to introduce more yet variety. For example, we could take a dedicated utilitarian charity worker obsessed with saving lives in poorer countries (but who doesn't interact, or want to interact, directly with those saved), and replace 'saving lives' with 'maximizing the number of paperclips in the universe' or any similar abstract goal. This is more speculative, of course -but there are other ways of getting similar results.
 Instrumental Goals as Final Goals If someone were to hold a gun to your head, they could make you do almost anything. Certainly there are people who, with a gun at their head, would be willing to do almost anything. A distinction is generally made between instrumental goals and final goals, with the former being seen as simply paths to the latter, and interchangeable with other plausible paths. The gun to your head disrupts the balance: your final goal is simply not to get shot, while your instrumental goals become what the gun holder wants them to be, and you put a great amount of effort into accomplishing the minute details of these instrumental goals. Note that the gun has not changed your level of intelligence or ability. This is relevant because instrumental goals seem to be far more varied in humans than final goals. One can have instrumental goals of filling papers, solving equations, walking dogs, making money, pushing buttons in various sequences, opening doors, enhancing shareholder value, assembling cars, bombing villages or putting sharks into tanks. Or simply doing whatever the guy with gun at our head orders us to do. If we could accept human instrumental goals as AI final goals, we would extend the space of goals quite dramatically. To do so we would want to put the threatened agent, and the gun wielder, together into the same AI. Algorithmically there is nothing extraordinary about this: certain subroutines have certain behaviors depending on the outputs of other subroutines. The 'gun wielder' need not be particularly intelligent: it simply needs to be able to establish whether its goals are being met. If for instance those goals are given by a utility function then all that is required in an automated system that measure progress toward increasing utility and punishes (or erases) the rest of the AI if not. The 'rest of AI' is just required to be a human-level AI which would be susceptible to this kind of pressure. Note that we do not require that it even be close to human in any way, simply that it place a highest value on self-preservation (or on some similar small goal that the 'gun wielder' would have power over). For humans, another similar model is that of a job in a corporation or bureaucracy: in order to achieve the money required for their final goals, some human are willing to perform extreme tasks (organizing the logistics of genocides, weapon design, writing long emotional press releases they don't agree with at all). Again, if the corporation-employee relationship can be captured in a single algorithm, this would generate an intelligent AI whose goal is anything measurable by the 'corporation.' The 'money' could simply be an internal reward channel, perfectly aligning the incentives. If the subagent is anything like a human, they would quickly integrate the other goals into their own motivation, 12 removing the need for the gun wielder/corporation part of the algorithm.
 Noise, Anti-agents and Goal Combination There are further ways of extending the space of goals we could implement in human-level AIs. One simple way is simply to introduce noise: flip a few bits and subroutines, add bugs and get a new agent. Of course, this is likely to cause the agent's intelligence to decrease somewhat, but we have generated new goals. Then, if appropriate, we could use evolution or other improvements to raise the agent's intelligence again; this will likely undo some, but not all of effect of the noise. Or we could use some of the tricks above to make a smarter agent implement the goals of the noise-modified agent. A more extreme example would be to create an anti-agent: an agent whose single goal is to stymie the plans and goals of single given agent. This already happens with vengeful humans, and we would just need to dial it up: have an anti-agent that would do all it can to counter the goals of a given agent, even if that agent doesn't exist ("I don't care that you're dead, I'm still going to despoil your country, because that's what you'd wanted me to not do"). This further extends the space of possible goals. Different agents with different goals can also be combined into a single algorithm. With some algorithmic method for the AIs to negotiate their combined objective and balance the relative importance of their goals, this procedure would construct a single AI with a combined goal system. There would likely be no drop in intelligence/efficiency: committees of two can work very well towards their common goals, especially if there is some automatic penalty for disagreements.
 Further Tricks up the Sleeve This section started by emphasizing the wide space of human goals, and then introduced tricks to push goal systems further beyond these boundaries. The list isn't exhaustive: there are surely more devices and ideas one can use to continue to extend the space of possible goals for human-level AIs. Though this might not be enough to get every goal, we can nearly certainly use these procedures to construct a human-level AI with any human-comprehensible goal. But would the same be true for superhuman AIs?
 Orthogonality for Superhuman AIs We now come to the area where the Orthogonality thesis seems the most vulnerable. It is one thing to have human-level AIs, or abstract superintelligent algorithms created ex nihilo, with certain goals. But if ever the human race were to design a superintelligent AI, there would be some sort of process involved -directed evolution, recursive self-improvement,  13  design by a committee of AIs, or similar -and it seems at least possible that such a process could fail to fully explore the goal-space. The Orthogonality thesis in this context is: If we could construct superintelligent AIs at all, then there is extremely little constraint on the final goals that such AIs could have (as long as these goals are of feasible complexity, and do not refer intrinsically to the agent's intelligence). There are two counter-theses. The weakest claim is: Incompleteness: there are large categories of goals that no superintelligence designed by us could have. A stronger claim is: Convergence: all human-designed superintelligences would have one of a small set of goals. Here 'small' means 'smaller than the space of current human motivations,' thus very small in comparison with the space of possible AI goals. They should be distinguished; Incompleteness is all that is needed to contradict Orthogonality, but Convergence is often the issue being discussed. Often Convergence is stated in terms of a particular model of metaethics, to which it is assumed all agents will converge (see some of the references in the introduction, or various online texts and argument 14 ).
 No Convergence The plausibility of the convergence thesis is highly connected with the connotations of the terms used in it. "All human-designed rational beings would follow the same morality (or one of small sets of moralities)" sounds plausible; in contrast "all human-designed superefficient algorithms would accomplish the same task" seems ridiculous. To quote an online commentator, how good at playing chess would a chess computer have to be before it started feeding the hungry? Similarly, if there were such a convergence, then all self-improving or constructed superintelligence must fall prey to it, even if it were actively seeking to avoid it. After all, the self-improving lower-level AIs or the designers have certain goals in mind (as we've seen in the previous section, if the designers are AIs themselves, they could have potentially any goals in mind). Obviously, they would be less likely to achieve their goals if these goals were to change as they got more intelligent  (Omohundro, 2008)  (see also N. Bostrom's forthcoming book Superintelligence: Groundwork to a Strategic Analysis of the Machine Intelligence Revolution). The same goes if the superintelligent AI they designed didn't share these goals. Hence the AI designers will be actively trying to prevent such a convergence, if they suspected that one was likely to happen. If for instance their goals were immoral, they would program their AI not to care about morality; they would use every trick up their sleeves to prevent the AI's goals from drifting from their own. So the convergence thesis requires that for the vast majority of goals G: 1. It is possible for a superintelligence to exist with goal G (by section 0). 2. There exists an entity with goal G (by section 0), capable of building a superintelligent AI. 3. Yet any attempt of that entity to build a superintelligent AI with goal G will be a failure, and the superintelligence's goals will converge on some other goal. 4. This is true even if the entity is aware of the convergence and explicitly attempts to avoid it. 5. If the superintelligence were to be constructed by successive self-improvement, then an entity with goal G operating on itself to boost its intelligence is unable to do so in a way that would preserve goal G. This makes the convergence thesis very unlikely. The argument also works against the incompleteness thesis, but in a weaker fashion: it seems more plausible that some types of goals would be unreachable, despite being theoretically possible. There is another interesting aspect of the convergence thesis: these goals G are to emerge, somehow, without them being aimed for or desired. If one accepts that goals aimed for will not be reached, one has to ask why convergence is assumed: why not divergence? Why not assume that though G is aimed for, random accidents or faulty implementation will lead to the AI ending up with one of a much wider array of possible goals, rather than a much narrower one? We won't delve deeper into this, and simply make the point that "superintelligent AIs won't have the goals we want them to have" is therefore not an argument in favor of the convergence thesis.
 Oracles Show the Way If the Orthogonality thesis is wrong, then it implies that Oracles are impossible to build. An Oracle is a superintelligent AI that accurately answers human questions about the world, such as the likely consequences of certain policies and decisions  (Armstrong, Sandberg, & Bostrom, 2012) .  15  If such an Oracle could be built, then we could attach it to a human-level AI with goal G. The human-level AI could then ask the Oracle what the results of different decisions actions could be, and choose the action that best accomplishes G. In this way, the combined system would be a superintelligent AI with goal G. What makes the "no Oracle" implication even more counterintuitive is that any superintelligence must be able to look ahead, design actions, predict the consequences of its actions, and choose the best one available. But the convergence and indifference theses imply that this general skill is one that we can make available only to AIs with certain specific goals. Though agents with those specific goals are capable of doing effective predictions, they automatically lose this ability if their goals were to change.
 Tricking the Controller Just as with human-level AIs, one could construct a superintelligent AI by wedding together a superintelligence with a large motivated committee of human-level AIs dedicated to implementing a goal G, and checking the superintelligence's actions. Thus to deny the Orthogonality thesis requires that one believes that the superintelligence is always capable of tricking this committee, no matter how detailed and thorough their oversight. This argument extends the Orthogonality thesis to moderately superintelligent AIs, or to any situation where there's a diminishing return to intelligence. It only fails if we take AI to be fantastically superhuman: capable of tricking or seducing any collection of human-level beings.
 Temporary Fragments of Algorithms, Fictional Worlds and Extra Tricks These are other tricks that can be used to create an AI with any goals. For any superintelligent AI, there are certain inputs that will make it behave in certain ways. For instance, a human-loving moral AI could be compelled to follow most goals G for a day, if they were rewarded with something sufficiently positive afterwards. But its actions for that one day are the result of a series of inputs to a particular algorithm; if we turned off the AI after that day, we would have accomplished moves towards goal G without having to reward its "true" goals at all. And then we could continue the trick the next day with another copy. For this to fail, it has to be the case that we can create an algorithm which will perform certain actions on certain inputs as long as it isn't turned off afterwards, but that we cannot create an algorithm that does the same thing if it was to be turned off. Another alternative is to create a superintelligent AI that has goals in a fictional world (such as a game or a reward channel) over which we have control. Then we could trade interventions in the fictional world against advice in the real world towards whichever goals we desire.  16  These two arguments may feel weaker than the ones before: they are tricks that may or may not work, depending on the details of the AI's setup. But to deny the Orthogonality thesis requires not only denying that these tricks would ever work, but denying that any tricks or methods that we (or any human-level AIs) could think up, would ever work at controlling the AIs. We need to assume superintelligent AIs cannot be controlled in any way that anyone could think of.
 In Summary Denying the Orthogonality thesis thus requires that: 1. There are goals G, such that an entity an entity with goal G cannot build a superintelligence with the same goal. This despite the fact that the entity can build a superintelligence, and that a superintelligence with goal G can exist. 2. Goal G cannot arise accidentally from some other origin, and errors and ambiguities do not significantly broaden the space of possible goals. 3. Oracles and general purpose planners cannot be built. Superintelligent AIs cannot have their planning abilities repurposed. 4. A superintelligence will always be able to trick its overseers, no matter how careful and cunning they are. 5. Though we can create an algorithm that does certain actions if it was not to be turned off after, we cannot create an algorithm that does the same thing if it was to be turned off after. 6. An AI will always come to care intrinsically about things in the real world. 7. No tricks can be thought up to successfully constrain the AI's goals: superintelligent AIs simply cannot be controlled.
 Conclusion It is not enough to know that an agent is intelligent (or superintelligent). If we want to know something about its final goals, about the actions it will be willing to undertake to achieve them, and hence its ultimate impact on the world, there are no shortcuts. We have to directly figure out what these goals are (or figure out a way of programming them in), and cannot rely on the agent being moral just because it is superintelligent/superefficient. 			 © Stuart Armstrong


Abstract: Reputations for resolve are said to be one of the few things worth fighting for, yet they remain inadequately understood. Discussions of reputation focus almost exclusively on first-order belief change-A stands firm, B updates its beliefs about A's resolve. Such first-order reputational effects are important, but they are not the whole story. Higher-order beliefs-what A believes about B's beliefs, and so onmatter a great deal as well. When A comes to believe that B is more resolved, this may decrease A's resolve, and this in turn may increase B's resolve, and so on. In other words, resolve is interdependent. We offer a framework for estimating higher-order effects, and find evidence of such reasoning in a survey experiment on quasi-elites. Our findings indicate both that states and leaders can develop potent reputations for resolve, and that higher-order beliefs are often responsible for a large proportion of these effects (40 percent to 70 percent in our experimental setting). We conclude by complementing the survey with qualitative evidence and laying the groundwork for future research.

Allan Dafoe
Remco Zwetsloot
Matthew Cebul

Reputations for Resolve and Higher-Order Beliefs in Crisis Bargaining

Introduction Navigating an international crisis can require incredibly complex inferences. Even seemingly straightforward strategies can backfire dramatically. Suppose that persons A and B are playing Chicken, the game in which two cars race toward each other and the goal is to get the opponent to swerve first. One popular strategem says that A should throw her steering wheel out of the window in order to convince B that she cannot swerve, thereby compelling B to do so.  1  Yet suppose that B incorrectly believes that A hid a spare steering wheel under her seat. If A does not know that B believes this, she may surrender her ability to swerve without actually becoming committed in B's eyes, raising the chances of a tragic crash. Or suppose that A incorrectly believes that B believes that A has a spare steering wheel. This belief might prevent A from using a tactic that would have improved her odds of victory. Much of the complexity in international crisis bargaining is attributable to higherorder beliefs-beliefs about beliefs  (Schelling 1960; . While these complications are perhaps most apparent in hypothetical games like Chicken, examples abound in real-world international crises as well. For instance, in the years prior to the Berlin crisis, Premier Khrushchev had deliberately exaggerated the Soviet Union's capabilities, and widely-discussed American concerns about a "missile gap" had convinced him that the Americans had fallen for the ruse. "  [O] n the assumption that the Americans believed the Soviets were ahead in the arms race," Khrushchev chose to escalate tensions over Berlin  (Kaplan 1983, 304) . Khrushchev was forced to back down, however, when he learned that US policymakers did not believe this at all-a fact that the Americans purposely signaled via intelligence leaks to NATO units that the US knew to be compromised by Soviet agents  (Schlosser 2014, 284) . Higher-order beliefs thus played a critical role in both the onset and resolution of one of the most significant crises of the Cold War. In this article, we examine the theoretical and empirical import of higher-order belief dynamics for two crucial phenomena in international relations: resolve and reputations for resolve. Reputations, especially for resolve, have been a longstanding topic of scholarship, but most studies consider only first-order beliefs: whether and how B updates his beliefs about A on the basis of A's behavior. We argue that taking higher-order beliefs into account substantially amplifies the importance of reputations for resolve. This is because resolve is interdependent: one side's (perceived) resolve can affect the other side's (perceived) resolve, meaning that small initial changes in beliefs can spiral into large overall changes, with potentially decisive consequences. We evaluate our theory using a survey experiment conducted on a sample of quasi-elites recruited through Stephen Walt's Foreign Policy blog. We begin by corroborating reputation scholars' core assertion: states that behaved resolutely in the past are perceived to be more highly resolved in the present. We find that the effect of reputation on perceived resolve is larger than any other experimental manipulation, including the effect of material capabilities, and persists across a range of informational conditions and subgroups. We then go a step further, advancing scholarship on reputations for resolve by demonstrating that respondents also form higher-order beliefs in line with our argument. Exogenous increases in A's perceived resolve, induced by providing information about A's behavior in disputes with states other than B, produce decreases in B's perceived resolve, even though respondents learned nothing about the usual factors said to affect B's resolve. The size of the parameter estimates we obtain suggest that higher-order beliefs can account for as much as 70 percent of the reputational manipulation's effect on the balance of resolve, with a lower bound of roughly 40 percent. We thus answer Robert Jervis's longstanding but still open question: there is indeed a strong zero-sum element to resolve. Our design also permits a direct test of a competing hypothesis linking higherorder beliefs to resolve, which Jervis has referred to as the "domino theory paradox" and Daryl Press as the "never again theory": if A backed down in a recent dispute, B will believe that A will expend extra effort to restore its tarnished reputation for resolve, and therefore that A will be more rather than less resolved in subsequent disputes  (Jervis 1991; Press 2005) . Our evidence is contrary to these hypotheses. Our results reinforce recent studies demonstrating the importance of reputation in the domains of conflict, finance, and alliance politics, among others (see  Crescenzi 2017, for a review) . These studies all focus on the first-order effects of reputation.  2  We show, however, that the story does not stop there. In doing so, we join a number of others who place higher-order beliefs at the center of international relations specifically  (Schelling 1960; Morrow 2014)  and social, economic, and political life more broadly  (Kuran 1995; Chwe 2001 ). These beliefs are what make resolve interdependent, and are therefore crucial to the onset and resolution of crises and wars.
 Resolve, Reputation, and Higher-Order Beliefs Much of international politics pivots around perceptions. A state's ability to persuade, deter, and compel depends in large part on how that state is perceivedwhether others consider its military capable, its threats credible, its promises reliable. As a result, states and leaders are motivated to act in ways that favorably shape others' perceptions and expectations, i.e. to cultivate reputations for desirable qualities or behavioral tendencies. One of the most important kinds of reputation in international politics is a reputation for resolve. We define resolve as follows: Resolve: the probability that an actor is willing to stand firm in a crisis, given its beliefs about the world at that time. We briefly explain how this behavioral definition relates to other common conceptualizations of resolve. Formalized mathematical models of crisis bargaining often define resolve as a state's "value for war," which is said to be a function of material factors such as military capabilities, the issue under dispute, and domestic audiences (see, e.g.  Morrow 1989; Powell 1990; Fearon 1994; Sartori 2016 ). Psychological accounts of resolve instead emphasize internal traits such as dispositional determination, willpower, or "sticktoitiveness"  (Kertzer 2016, 8-9) . Here, we define resolve in terms of states' behavior in a crisis, in which states may either back down or stand firm.  3  In this sense, a state's "value for war" and its dispositional traits are important factors that affect resolve-the probability that an actor stands firm-but they may not be the only ones, so they do not define resolve.  4  Understanding resolve in behavioral terms has several benefits.  5  By clearly specifying what kind of behavior is (in)consistent with being resolved, our definition makes resolve measurable and facilitates empirical analysis. More importantly, our definition allows scholars to directly incorporate another essential determinant of whether A will stand firm in a crisis with B: A's beliefs about B's resolve. As exemplified by Chicken, that A's crisis behavior depends on her beliefs about B's resolve is what makes interstate crises complex strategic interactions. This relationship also speaks to the literature on reputations for resolve, which we briefly review below.
 Building Reputations for Resolve International crises can often be understood as "contests of resolve"  (Morrow 1989, 941-42)  in which the outcome is determined by one side's ability to convince the other that it will not back down. In such contests, a history of resolute past behavior can be a valuable asset. We say that actors have a reputation for resolve when observers have formed beliefs about that actor's tendency to stand firm in a certain class of disputes on the basis of that actor's past behavior  (Dafoe, Renshon, and Huth 2014 ).  6  The effect of a reputation for resolve remains a subject of debate. Proponents contend that reputations offer a path to successful commitment, deterrence, and compellence  (Huth 1997) , as costly past actions can corroborate threats and promises that might otherwise be dismissed as "cheap talk"  (Fearon 1995; Trager 2017 ). These scholars argue that an "undesired image can involve costs for which almost no amount of the usual kinds of power can compensate"  (Jervis 1970, 6) , and so reputations are "one of the few things worth fighting over"  (Schelling 1966, 124) . In support of this position, a number of recent studies find evidence that states' past behavior shape how other states act toward it.  7  Others, however, dispute the link between past actions and present behavior. For instance, Mercer argues that the desirability of an action conditions how people interpret the disposition of its initiator, concluding that "people do not consistently use past behavior to predict similar behavior in the future"  (Mercer 1996, 45-47, 212) . Similarly, Press claims that "blood and wealth spent to maintain a country's record for keeping commitments are wasted," since opponents form assessments of credibility largely based on their perceptions of states' interests and capabilities, not their past behavior  (Press 2005, 10) . Detractors also aver that evidence of reputational effects is largely observational and often indirect, and is therefore inconclusive. Thus, one contribution of our article is to bring experimental methods to bear on a basic question: do observers use a state's past behavior to predict what it is likely to do today? 8 H REP : Perceptions of a state's resolve will (a) increase if it stood firm in past crises and (b) decrease if it backed down in past crises. 9 In addition, we also seek to advance the reputation debate beyond H REP toward a discussion of higher-order beliefs. Note that H REP focuses purely on how first-order beliefs-beliefs about an actor's traits or behavioral tendencies-change in response to past behavior. Such arguments are at the center of most theoretical accounts of reputational dynamics. First-order beliefs are, however, not the only kinds of beliefs that matter. In the next section, we contend that reputations can also affect higherorder beliefs-beliefs about beliefs (about beliefs, etc.)-with important consequences for state behavior.
 Reputational Effects and Higher-Order Beliefs Consider a hypothetical dispute between states A and B, in which B is considering whether to escalate. Initially, B has some sense of how likely A would be to stand firm (A's resolve), though he cannot be sure. B then observes A behaving resolutely in some other dispute, and after considering that A might, say, be more militarily capable than B originally thought, B updates his beliefs about A's resolve accordingly. We can call this a first-order reputational effect, in which one side updates its beliefs about its opponent's characteristics or behavioral tendencies solely on the basis of that opponent's past behavior. In many cases, however, higher-order reasoning further complicates the story. For example, A might know that B was paying careful attention to her behavior in the other dispute. If so, A might conclude that, since she stood firm, B has become less resolved, and this could further embolden A to also stand firm against B. If B expects A to be thinking along these lines, he might conclude that there is little he can do to make A back down, further reducing B's resolve. If actors make inferences in this way, the first-order reputational effect will be joined by higher-order effects. These two effects together make up the total reputational effect, i.e. the full shift in the balance of resolve between two actors due to a single initial reputational shock (see Section "Do Higher-Order Beliefs Contribute to Reputational Effects?" for a formalization). Indeed, higher-order beliefs can theoretically produce large substantive effects, as relatively small first-order reputational effects cascade through higherorder belief chains into large changes in the balance of resolve. Put another way, our argument is that resolve is a function of not just the material factors emphasized in the modeling literature or the internal traits emphasized in psychological accounts, but also states' beliefs about their opponents' resolve-A's resolve is decreasing in (her beliefs about) B's resolve, and vice versa. In short, resolve is interdependent. Furthermore, that resolve is interdependent implies that higher-order beliefs may play a central, albeit under-appreciated, role in international crises-A's beliefs about B's resolve (which dictate A's own resolve) are themselves a product of A's higher-order beliefs about B's beliefs about A's resolve, and vice versa. To express our arguments in terms of reputations, we contend not just that states and leaders can possess reputations for resolve, but also that these actors are aware that such a reputation may decrease the likelihood that their opponents stand firm during crises, and so update their own resolve accordingly. In short, states may expect that their reputations precede them. H INT : Perceptions of a state's resolve will (a) increase when there is a decrease in perceptions of its opponent's resolve, and (b) decrease when there is an increase in perceptions of its opponent's resolve. There are two principal reasons why we may not expect this hypothesis to hold. First, states may not form higher-order beliefs if they lack the ability to do so. Kertzer, for example, argues that in many situations with higher-order uncertainty about resolve "the complex nature of the decision-making environment actors face stretches far beyond the limits of human cognition"  (Kertzer 2016, 149 ; see also  Mercer 2012) . And even if they have the ability, states may be insufficiently confident in their higher-order beliefs to incorporate them into their assessments of resolve. If this were the case, we would expect to see no evidence of interdependence. Second, H INT is not the only possible relationship between higher-order reasoning and reputational effects. H INT implies that higher-order reasoning magnifies first-order effects-states that stand firm earn larger reputational bonuses, whereas states that back down suffer larger penalties. In what Jervis labels the "domino theory paradox" and Press "never again theory," however, higher-order effects run counter to the first-order effect  (Jervis 1991; Press 2005) . Domino theory, which undergirded much of US foreign policy during the Cold War, holds that if a state backs down in one situation, observers will infer that it is more likely to do so in other situations as well. The paradox is that the corresponding higher-order beliefs may be "self-defeating"  (Jervis 1991, 36-37) : An actor who has had to back down once will feel especially strong incentives to prevail the next time in order to show that the domino theory is not correct, or at least does not apply to him. In other words, an actor who believes the domino theory-or who believes that others accept it-will have incentives to act contrary to it. Indeed, statesmen sometimes respond to a defeat by warning others that this history will make them extremely resistant to retreating again. Furthermore, if others foresee this, they will expect a defeated actor to be particularly unyielding in the next confrontation. In short, the domino theory paradox predicts that, in certain circumstances, higher-order reasoning can mitigate or even reverse first-order reputational effects. H DTP : Perceptions of a state's resolve will increase when it backed down recently and has a prior reputation to recover.
 Research Design The scientific study of reputations is limited by their inaccessibility. Reputations consist of perceptions, which exist in people's minds and are not directly observable. Even records of private deliberations, free of any incentives to dissemble, may lead us astray. To the extent that an opponent's reputation is a constant during a crisis and is common knowledge among decisionmakers, discussions are likely to focus on new information such as troop movements even if reputation matters  (Weisiger and Yarhi-Milo 2015) . Reputational inferences, given their evolutionary importance, may also be so automatic that they are made subconsciously  (Bowles and Gintis 2011) . These difficulties were acknowledged by Snyder and Diesing, who noted that their inability to find much evidence of reputational inferences in case histories "may be only an artifact of the record"  (Snyder and Diesing 1977, 496) . To overcome these challenges, we employ a scenario-based survey experiment. Respondents are told about a scenario, features of which are randomly varied or omitted, and then asked about their opinions or beliefs. Survey experiments are especially suitable for research questions about how people incorporate, interpret, and act on particular types of information  (Mutz 2011) , and these are precisely the questions in which scholars of reputation are interested. Below, we review the survey design and the sample in more detail.
 Survey Vignette Respondents read an abstract scenario about two countries (A and B) engaged in a serious territorial dispute. The features of the scenario are presented in a bullet list format, each assigned with some probability independent of each other (a full factorial design). Each feature has several different levels, including its omission. For the full text of the survey and treatment allocations, see Online Appendix B. Respondents were first informed of each country's regime type, either democracy or dictatorship (some received no information on regime type). Respondents then learned about the military capabilities of each country, with A having either "substantially stronger military forces" than B, being "about equally strong," or B being substantially stronger (again, some received no information about capabilities). In all conditions except no information, respondents are also told that neither country has nuclear weapons. The respondent then reads the reputational feature of the scenario, State A's history of crisis interactions. This bullet involves two variables: whether A stood firm or backed down in past crises, and whether A's previous conflicts were against B or other countries.  10  According to most impartial observers, of the three most recent major international crises that Country A has faced against [other countries/Country B], Country A [did not back down and Country A achieved its aims/backed down in each crisis and failed to achieve its aims]. There is also a special Domino Theory Paradox version of this manipulation, in which A stood firm in three past crises, but then backed down in a fourth, most recent crisis. In all conditions except no information, respondents are also informed whether A's past crises occurred under the current or a previous leader. Lastly, the vignette includes several other experimental features related to the history of the dispute, as well as recent threats and promises.  11  The scenario concludes by stating that the crisis is serious and that many observers consider major military action in the near future likely. Respondents are then asked the primary outcome question for both countries: What is your best estimate, given the information available, about whether Country A/ B will back down in this dispute? Respondents have five options, ranging from "very unlikely" (0 percent to 20 percent chance) to "very likely" (80 percent to 100 percent chance). Note that this question exactly matches our definition of resolve in Section "Resolve, Reputation, and Higher-Order Beliefs."  12  Before detailing our sampling strategy, we briefly address the benefits and drawbacks of our abstract survey design. Our survey vignette describes a crisis between two abstract states, A and B, and the scenarios respondents read were typically no more than 150 words. The benefit of this approach is its flexible simplicity; short vignettes are less taxing for respondents, and abstracting away from real-world states may permit cleaner manipulation of the concepts of interest (though not necessarily; see  Dafoe, Zhang, and Caughey 2018) . The attendant drawback is a decreased level of realism in abstract survey scenarios, relative to vignettes that feature actual countries  (Tingley 2017) . Reassuringly, however, recent research employing both abstract and real-world crises scenarios recovers reputational effects of a very similar magnitude to those presented below  (Renshon, Dafoe, and Huth 2018) . Regardless of the design, it is of course not possible to create environments akin to those faced by real-life decision-makers through surveys alone, and it is therefore best to complement survey experiments with other kinds of evidence. Though more systematic observational work will have to await future research, we offer some preliminary supportive evidence for our claims in Section "Discussion: Interdependent Resolve in Real Life," and discuss the conditions under which evidence of interdependent resolve is most and least likely to be found.
 Sample We administered the survey to a convenience sample of respondents recruited through Stephen Walt's Foreign Policy blog. On August 1st, 2011 Walt posted a blog entry inviting readers to "Become a data point!" (see Online Appendix B.1). From this advertisement, over 1,000 respondents took the survey. This sampling strategy was intended to mitigate potential biases that may arise from using regular citizens to proxy for elite policymakers. The general concern is that elites are better informed about and more experienced with foreign policy decision-making than the average citizen-as a result, they are more likely to employ higher-order strategic reasoning and consider their opponents' perspective  (Hafner-Burton, Hughes, and Victor 2013; Hafner-Burton et al. 2014 ). Ideally, then, one would run our experiment on key decision-makers, such as military leaders, foreign policy advisors, and politicians. Unfortunately, such subjects are rarely accessible, and even if they are it is usually as part of a small convenience sample. To approximate our elite population of interest, we instead sought to recruit a sample of quasi-elite respondents who are abnormally well-informed about and interested in foreign policy, whose backgrounds and world views more closely parallel those of high-level elites. We expected a sample drawn from Walt's Foreign Policy readership to be highly educated and knowledgeable of foreign policy, and that is indeed the case. Of those 87 perccent who answered the demographic questions, 83 percent reported having a college degree or higher, and 50 percent a postgraduate degree. Moreover, 60 percent claimed to have "particular expertise with foreign affairs, military affairs, or international relations." Politically, the group leans Democratic, as 89 percent claimed to agree more with the policies of Democrats (11 percent more with those of Republicans). The respondents were 88 percent male, which is far from representative of the general population but not obviously unrepresentative of foreign policy elites. More details about the sample are provided in Online Appendix A. To be sure, then, our sample remains an imperfect approximation of foreign policy elites. Still, we argue that these imperfections will likely lead us to underestimate the effect of reputation and higher-order beliefs on perceived resolve. To start, the Democratic bias should cut against our proposed hypothesis, as liberals are generally less likely than conservatives to invoke concerns over credibility, reputation, and honor in international affairs  (Trager and Vavreck 2011) . And while respondents may also hew closer to Stephen Walt's particular foreign policy views, Walt has repeatedly argued against the importance of reputation.  13  Lastly, our sample likely remains less experienced with actual foreign policy decisionmaking than true elites. But as mentioned above, experience is linked to higherorder strategic reasoning, and  Tingley and Walter (2011)  show that experienced players care more about reputation than inexperienced ones. If anything, then, our results likely underestimate the reputational effects that would be found among actual elites. In short, given the cost and difficulty in obtaining elite samples, our sampling design is a reasonable first step toward the empirical study of higher-order beliefs in international crises.  14  Importantly, we think it highly unlikely that our sample would produce an opposite effect of reputation on perceived resolve relative to true elitesour respondents may be less attentive to reputation, but this should only depress the magnitude of reputational effects, not reverse their direction.
 Results To preview our results, we reach two main findings. First, reputations for resolve matter; when respondents learn that a state has stood firm in past crises, they consider it much more likely to stand firm today. Second, resolve is interdependent; we find that increases in A's perceived resolve are associated with decreases in B's perceived resolve, and estimate that higher-order belief updating is responsible for a large proportion of the total observed effect of past behavior on the balance of resolve.  15
 Can States Build Reputations for Resolve? We begin with H REP , which considers whether a country's past behavior affects perceptions of its current resolve. We find strong evidence that it does. Respondents who learned that "Country A did not back down and Country A achieved its aims" in past crises thought that A was more likely to stand firm than those that received no information about past behavior (a 10 percentage point increase, from 60 percent to 70 percent). Similarly, when A "backed down in each crisis and failed to achieve its aims," respondents thought A was roughly 10 percentage points more likely to back down. These effects are both highly significant (Figure  1 ). Moreover, this reputational treatment has the largest effect of all manipulations in the survey. The effect of going from a history of backing down to a history of standing firm is about 20 percentage points, which represents a quarter of the resolve variable's total range.  16  This reputational effect is roughly twice the size of the second-largest effect, information about power: shifting from B having "substantially stronger military forces" to A having "substantially stronger military forces" increases perceptions of A's resolve by roughly 10 percentage points.  17  Readers may wonder whether this potent reputational effect is driven only by a subset of respondents. Reputation skeptics contend that even if past behavior matters, it does so only to the extent that "a decision maker uses an adversary's history of keeping commitments to assess the adversary's interests or military power"  (Press 2005, 21 ). An observable implication of this view is that we should only see effects of past behavior on respondents that lack information about the balance of power. We find, however, that the reputational effect persists among respondents who were told about military capabilities (Figure  12 , Online Appendix C.1).  18  And while the reputational effect decreased slightly as scenario complexity increased,  19  it remained above 10 percentage points even among respondents who received the maximum amount of information about the scenario (Figure  11 , Online Appendix C.1). Reputational effects also persist across state leaders and across demographic subgroups, including gender, education, political affiliation, and cultural background. In sum, we find strong support for H REP . In contrast, we find no evidence for H DTP . According to H DTP , a state that backs down once after a history of standing firm will be perceived to be more resolved to stand firm in the present, as observers expect it to try to re-establish its lost reputation for resolve. Yet as Figure  1  shows, backing down once after a history of always standing firm reduces perceived resolve by about 8 percentage points compared to a history of standing firm (p < 0:0001), nearly returning perceived resolve to its baseline probability in the no-information condition. In other words, backing down once almost entirely eliminates the reputational gains that A achieves by standing firm in the initial three crises.
 Do Higher-Order Beliefs Contribute to Reputational Effects? Typically, reputational effects like the ones reported above are interpreted in terms of first-order updating: A takes an action, and observers update their beliefs about A. Yet as discussed in Section "Reputational Effects and Higher-Order Beliefs," the total reputational effect of A's past behavior on A and B's resolve may also consist of higher-order effects. . In these terms, the "direct" or "immediate" effect of our reputation treatment T (information about A's past behavior) is given by D R1 i ¼ R1 i À R0 i , where R0 i refers to perceptions of i's resolve before observing T (i.e. among respondents in the "no information" condition). Now, suppose that observers go through two levels of belief updating following reputation treatment T (this updating process is shown graphically in Figure  2 ). Our argument suggests that we should see D R1 A > 0, and, by interdependence, D R2 B < 0. In this situation, our estimand, which we will call the interdependence coefficient and label y, is given by D R2 B D R1 A . This ratio tells us by how much an immediate one-unit change in A's perceived resolve subsequently changes B's perceived resolve. For example, if a 10 percentage point increase in A's resolve decreases B's resolve by 5 percentage points, y ¼ À5 10 ¼ À0:5. More generally, if we assume that the interdependence of resolve is symmetric across actors and constant across levels of updating, we have y ¼ D Rk A D RkÀ1 B ¼ D Rk B D RkÀ1 A for all k 2 N. We can then empirically estimate y by taking the ratio of the observed total effect of treatment on B's and A's resolve, D RB D RA , and state our interdependent resolve hypothesis more precisely as H INT : y < 0. 20 As mentioned above, our treatment must fulfill two conditions for this estimation strategy to succeed, analogous to the identifying assumptions in instrumental variable analysis: (C1) D R1 A > 0 (instrument strength) and (C2) D R1 B ¼ 0 (exclusion restriction). Notably, most treatments do not satisfy C2. For example, the fact that A stood firm against B in the past could lead to inferences not just about A's resolve, but also about B and the A-B dyad, such as B's dispositional determination or domestic audiences, violating our identifying assumption. Thus, we cannot use information about A's past behavior against B to estimate the interdependence of A and B's resolve. However, information about A's past actions against another country is a treatment that likely satisfies C2. 21 When told about A's extra-dyadic behavior, observers can make inferences about how much A values its reputation, whether A's domestic public is liable to punish leaders for backing down, or any other factor that shapes A's resolve. But the treatment is uninformative about the A-B relationship, the territory under dispute, or other factors that could reasonably shape perceptions of B's resolve-except, that is, perceptions of A's resolve. This treatment, then, allows for a clean test of our interdependence hypothesis. The results of this test are displayed in Figure  3 . We find that A's behavior against other countries significantly affects perceptions of B's resolve as expectedrespondents who were told that A stood firm against other countries in the past assess B's resolve to be roughly 7 percentage points lower relative to baseline, and A backing down against other countries leads to a similar increase in perceptions of B's resolve. For A standing firm in extra-dyadic crises, ŷ ¼ À:065 :091 ¼ À:71, and for A backing down in extra-dyadic crises, ŷ ¼ :074 À:098 ¼ À:75. Both of these effects are statistically significant, lending strong support to H INT . 22 .
 The Interdependence Multiplier These results offer compelling evidence that reputational effects exist, and that they are compounded by higher-order beliefs. Still, we have yet to specify precisely how important are higher-order beliefs. Can we estimate the proportion of the total reputational effect-the effect of past behavior on the overall difference in resolve between A and B-that is attributable to higher-order belief updating? To help us answer this question, define the interdependence multiplier (IM) as the factor by which a first-order reputational effect should be multiplied in order to obtain the total reputational effect. In our simple formalization, the magnitude of the interdependence multiplier depends on two parameters: the interdependence coefficient, y, and the number of levels of belief updating, which we label n.  23  Given our model (and assuming jyj < 1), Online Appendix D.2 derives the IM to be, for any n, IM ¼ 1 À jyj n 1 À jyj which converges to 1=ð1 À jyjÞ as n ! 1, a situation with "common knowledge" in which everybody knows about a fact or event, knows that everybody knows it, and so on. Figure  4  plots the magnitude of the IM across hypothetical values of y and n. At one extreme, if we assume that actors engage only in first-order reasoning (the light blue line), the IM is always 1, and the total reputational effect is equal to the firstorder reputation effect. This, implicitly, has been the assumption in most past discussions of reputation. At the other extreme, if it is plausible to assume common knowledge (the red line), the total reputational effect is more than twice as large as the first-order effect for any jyj > 0:5, and nearly four times as large at j ŷj ¼ 0:73 (the average of our jyj estimates calculated from Figure  3  above).  24  Between these extremes, the IM's magnitude is substantial across a range of parameter values, underscoring that higher-order beliefs can be responsible for a large proportion of any given total reputational effect. We can now use the IM to estimate the effects of higher-order beliefs on perceived resolve in our survey experiment. As stated above, our average estimated y ¼ À0:73, and the average total effect of the reputation treatments presented in Figure  3  is roughly 16 percent. To estimate how much of that 16 percent change is attributable to higher-order beliefs, let us assume that n ¼ 3. This implies IM ¼ 1À0:73 3 1À:73 ¼ 2:26. The estimated first-order effect is then simply the total effect divided by the IM,  16  2:26 ¼ 7:08. This leaves roughly 9 percentage points attributable to higher-order reasoning-a sizable 55 percent of the total effect. We can also more intuitively verify this result by beginning with the first-order effect, and then building out via n ¼ 3 reasoning to the total effect. Suppose that, as we just estimated, respondents on average perceive A to initially be 7.08 percentage points more likely to stand firm when A has stood firm repeatedly in past cases (D R1 A ¼ 7:08). The respondent reasons, however, that B will become less resolved after inferring this change. Specifically, B's resolve decreases by 7:08 Â y percentage points (D R2 B ¼ À5:17). This, in turn, will increase A's perceived resolve by À5:17 Â y (D R3 A ¼ 3:77). The total reputational effect is therefore jD R1 A jþ jD R2 B j þ jD R3 A j ¼ 16:02 % 16 percentage points, which is indeed the total reputational effect that we observe in the data (and is also 7:08 Â IM). This estimation strategy is limited by our inability to directly observe n, respondents' level of higher-order reasoning-we assumed n ¼ 3 above, but the true value could be higher or lower. While we therefore cannot definitively identify the proportion of the total reputational effect attributable to higher-order beliefs, we remain  confident that these beliefs drive a substantial portion of the effect in this case, for several reasons. First, we can quantify our uncertainty by deriving bounds. As our survey recovers reputational effects that are inconsistent with mere first-order reasoning, we use n ¼ 2 as a lower bound. In this case, the IM ¼ 1:73, and higher-order effects are responsible for about 42 percent of the total reputational effect. The upper bound is represented by common knowledge (n ¼ 1). In this case, the IM ¼ 3:7, and the higher-order effects constitute about 73 percent of the total reputational effect (see Online Appendix D.2). 25 Note that our reputation treatment had a relatively large effect on B's perceived resolve (j ŷj ¼ 0:73 is large), which results in correspondingly large higher-order effect estimates even assuming low levels of higher-order reasoning (Figure  4  is instructive). Of course, the interdependence coefficient y might vary in size in different settings or contexts-we discuss this possibility extensively in Section "The Degree of Interdependence." Second, numerous other studies and examples discussed in Section "Higher-Order Beliefs" suggest that at least some degree of higher-order reasoning, be it conscious or intuitive, is a regular feature of human cognition in political, economic, and social settings-though higher-order belief chains can become prohibitively complex, n ¼ 2 reasoning is relatively common. This reassures us that n ¼ 2 is a reasonable lower bound to estimate higher-order effects. In sum, we find clear evidence that higher-order beliefs are responsible for a large portion of our observed reputational effects-even restricting the calculations to a minimal level of higher-order reasoning, the proportion approaches 50 percent. In the next section, we move beyond our survey data to discuss real-world applications and implications of our argument.
 Discussion: Interdependent Resolve in Real Life Above, we provided evidence of higher-order reputational dynamics, and argued that the magnitude of these effects depends on the order to which people form beliefs (n) and the degree of interdependence (y). Our abstract survey vignette allows us to observe the effect of higher-order belief updating in a way that is difficult to do in the real world. Still, some uncertainty remains as to whether and how the interdependence of resolve manifests in the messier context of real-world bargaining situations. This section discusses the circumstances under which we expect higher-order beliefs and interdependent resolve to matter most, and the attendant implications of our argument for real-world crisis bargaining.
 Higher-Order Beliefs As discussed in Section "Reputational Effects and Higher-Order Beliefs," one broad issue is whether decision-makers engage in higher-order reasoning  Mercer 2012 ). If they do not, our arguments have little relevance to the real world. Fortunately, evidence from a variety of contexts suggests that explicit and implicit higher-order reasoning is common, and does not require especially sophisticated agents. First, experimental evidence from behavioral economics suggests that most people often form at least second-order beliefs, and many reach higher orders as well (e.g.  Nagel 1995; Camerer, Ho, and Chong 2004) . In psychology, researchers have found that the magnitude of the famous "bystander effect" depends on whether the context allows bystanders to form beliefs about other bystanders' beliefs  (Thomas et al. 2014 (Thomas et al. , 2016 . A third example comes from a recent field experiment in Benin, where voters who learned information about politicians' behavior punished and rewarded performance only when they thought that others knew this information as well, suggesting a higher-order understanding of coordination dynamics  (Adida et al. 2020) . Many other studies also find significant effects from interventions targeted at higher-order beliefs  (Bicchieri 2016; Mildenberger and Tingley 2019) . Moreover, scholars have produced abundant evidence of higher-order reasoning among policymaking elites in high-stakes situations. At the domestic level, autocrats faced with potentially unsatisfied publics go to great lengths to create impressions of widespread support for their rule, or, if that fails, at least keep everyone guessing about each other's beliefs  (Kuran 1995; Chwe 2001, 20-21) . Internationally, as  McManus (2014, 726)  illustrates with the case of Israel-US bargaining over Iran's nuclear program, states often attempt to stake their allies' reputation on supporting them, believing that their allies will then believe themselves bound to act (see, e.g.  Jervis 1978, 180; Trager 2017, Ch. 1, for more examples) . And conflicts are often said to end only when "opponents succeed in coordinating their expectations"  (Slantchev 2003, 621 ; see also  Carter 2017) . In these and many other examples, beliefs about beliefs are the driver and focus of significant strategic contention. To be clear, our argument does not require that actors always run through every step of the inferential process in a deliberate or conscious way. Higher-order beliefs can be incorporated into decision-making heuristically, implicitly, and subconsciously. As  Chwe (2001, 78)  argues for the case of driving, "I stop at a red traffic light out of habit, but a fully specified argument for doing so would involve an infinite regress: I stop because I think that other people are going, and I think that other people are going because I think that they think that I am stopping, and so on." Developmental psychologists have found that children display a great degree of higher-order understanding early on in life, implicitly engaging in complex theorizing about the mental states of others long before they can explicitly articulate their reasoning  (Wellman 2014) . Higher-order belief dynamics also often become embedded in cultural and legal norms  Ridgeway 2011; Morrow 2014) .
 The Degree of Interdependence While it is therefore clear that real-world actors do engage in higher-order reasoning, the extent to which first-order reputational effect are amplified also depends on the degree to which resolve is interdependent, i.e. the extent to which an initial change in A's resolve affects B's resolve. Here, contextual factors are likely to play a large role. We identify and discuss four such factors. First, the degree of interdependence depends on the extent to which the context resembles a prototypical contest of resolve. The central features of such contests are  (1)  the costs of conflict are so great that losing is preferred to both sides standing firm, but (2) the issue under dispute is sufficiently valuable for a coercive victory to be preferred to a peaceful compromise  (Morrow 1989, 941) . We expect resolve to be most interdependent in conflicts that most closely approximate these conditions. Nuclear crises are often cited as the quintessential examples of such contests  (Schelling 1966; Powell 1990 ), but they are by no means the only ones. In the 1898 Fashoda Crisis, for example, France dispatched a mission to Egypt in an attempt to force Britain to make concessions, but withdrew when it became convinced that Britain was more resolved than it initially believed  (Trachtenberg 2012, 13-16) .  26  More contemporary examples can be found in proxy conflicts like the ongoing Syrian civil war. Direct conflict between the U.S. and Russia over Syria seems prohibitively costly, yet Damascus remains a valuable prize. In these circumstances, resolve can be highly interdependent. Indeed, this interdependence featured regularly in debates about U.S. intervention in Syria. Critics argue that Obama's failure to enforce the infamous chemical weapons red-line in 2013 undermined U.S. deterrence, paving the way for Russian intervention in 2015-Putin might have been compelled to stay out, were it not clear (to both parties) that the U.S. was irresolute. At the same time, Obama's reluctance to engage in Syria resulted in part from his belief that Russia would counter-escalate in response to limited U.S. intervention, especially after Russia stepped up its involvement in 2015.  27  In other words, U.S. lack of resolve fueled Russian resolve, further depressing U.S. resolve. Second, interdependence is also affected by the extent to which actors are able to act strategically, conditioning their behavior on what they think others are likely to do. Actors may fail or be unable to do so for various reasons. A prominent example is found in the expansive literature on credible commitments, which discusses commitment devices that may leave actors unconditionally resolved in a crisis. The moment actors truly commit to a strategy-when they throw their only steering wheel out of the window-they have effectively set their interdependence coefficient to 0: even if they subsequently change their beliefs about their opponent's resolve, their own course of action is already set. That said, absolute commitments are exceedingly difficult and risky to make.  28  In the real world, then, resolve is almost always interdependent at least to some extent. A third and related factor is the concentration of decision-making authority: when leaders have the ability to change course quickly, their resolve is likely to depend more on the other side's resolve. If, on the other hand, authority is diffuse, it will be difficult for a country to change course in the face of new information. One example is the delegation of military decision-making to local commanders, who can then choose to stand firm or back down in their theater of operations. Such delegation may be necessary from a practical perspective, but it also means central decision-makers have less flexibility. Another obstacle to short-term policy change in response to belief updating is the number of veto players in a political system  (Tsebelis 2002) . To the extent that this number is correlated with regime type, there is reason to think that the resolve of democracies is less interdependent than that of their autocratic counterparts. Lastly, a fourth factor affecting the degree of interdependence is the observability of resolve. An actor's resolve is more likely to influence its opponents calculations when it is readily observable. Resolve is likely to be more observable the more an actor's deliberative processes are public, or when opponents share a cultural understanding of the meaning of certain behaviors or events  (O'Neill 1999, 153-54) . Strategic intelligence also play an important role. The interdependence of resolve was on clear display, for example, during the Washington Disarmament Conference in 1921, when the United States was able to break Japan's ciphers and read Tokyo's private diplomatic communication. One such message stated the absolute minimum naval tonnage ratio that the Japanese government would be willing to accept. "Knowing how far Japan could be pushed . . . allowed the United States to do so with full confidence, merely waiting for the Japanese to give in"  (Bauer 2013, 211) . When Japan's (lack of) resolve became perfectly observable to the US, American resolve dramatically increased in response. In sum, we expect actors' resolve to be most interdependent in real-world contexts that resemble prototypical contests of resolve, where actors are strategic and have centralized decision-making structures, and where resolve can be inferred with confidence. When interdependence is high, higher-order amplification of first-order reputational effects can produce especially large swings in the balance of resolve. It is in these circumstances, then, that beliefs about beliefs should be most consequential for crisis outcomes. These intuitive sources of variation in the interdependence of resolve could themselves be the object of empirical study, but for now, we leave this task for future work.
 The Power of Beliefs Each party is the prisoner or the beneficiary of their mutual expectations. -Thomas Schelling  (Schelling 1960, 60)  Reputations for resolve have long been the subject of debate: some see them as indispensable assets, while others dismiss past actions as irrelevant to current crises. Using an experimental approach, this article strongly reinforces the former viewstates and leaders can form reputations for resolve and leverage them to their advantage during crises. Moreover, we emphasize that higher-order beliefs play an under-appreciated yet crucial role in this process, as first-order reputational effects are amplified by actors' beliefs about their opponents' beliefs. In this sense, international contests of resolve hinge not merely on past actions, but on actors' combined expectations about the implications of past actions for present behavior. In conclusion, we make several suggestions for further empirical research on higher-order beliefs and interdependent resolve in international politics. As mentioned earlier, one limitation of our study is that we lack direct knowledge of the level of higher-order reasoning at which respondents analyzed the crisis scenario. A number of studies in behavioral economics measures "k-level reasoning" directly in laboratory game settings (see Hafner-Burton, Hughes, and Victor 2013, for a review), but these measures are rarely found in survey research on international politics. Future studies could adapt such techniques to shorter survey formats, where they could serve as useful individual-level measures of strategic competence. We also highlight several interesting avenues for future research. One promising idea is that opponents' perceptions of a leader's competence could influence their higher-order inferences about that leader's likely beliefs, consequently shaping their own bargaining behavior. Some leaders are understood to be especially experienced, calculating, or wise, whereas others may be seen as inexperienced, impulsive, or even wholly incompetent. Not only could variation in leader competence directly affect a state's behavior, but the perception of competence or ineptitude might also shape others' higher-order expectations about that leader's beliefs and world view, with implications for their own behavior during crises. Moreover, there are also strong reasons to expect that beliefs about beliefs matter in contexts far beyond crisis bargaining, including collective action and coordination on many important international issues, such as climate change and international law  (Mildenberger and Tingley 2019; Morrow 2014 ). Schelling's argument that sets of beliefs can act as prisons (or paradises) matters as much to problems like gender inequality as it does to international conflict  (Ridgeway 2011) . Along these lines, the study of higher-order belief dynamics also presents an exciting opportunity for collaboration across research fields. Such dynamics can only be understood by combining the insights and methods of psychological, cultural, and strategic approaches-whether and how one actor forms and acts on beliefs about another actor's beliefs depends on cognitive processes, systems of social meaning, and the anticipated consequences of different courses of action . And once they are understood, insights on higher-order beliefs are likely to travel across many domains and issue areas. In short, higher-order beliefs are an incredibly rich area for future studies of broad applicability and substantive importance. This article offered some theory and a novel methodological framework for diagnosing the effects of higher-order beliefs, which we hope will contribute to a vibrant sub-literature on this topic in international relations scholarship. participants of the Department of Peace and Conflict Research workshop at Uppsala University and the Division of Social Science seminar at Hong Kong University of Science & Technology, and especially Robert Trager. g.  Mattes 2012; LeVeck and Narang 2017) . Others examine the importance of reputation for cooperation; see  Crescenzi (2017) . 8. For other recent experimental work, which focuses on first-order beliefs, see  Kertzer, Renshon, and Yarhi-Milo (2021) ,  Huth (2018) . 9. The relevant "perceiver" of a state's resolve could be any other international observer, be it another state involved in a dispute with the state in question, or an uninvolved third party-in either case, agents should make similar reputational inferences. For the purposes of empirical testing, however, the "perceivers" are our survey respondents, who assessed the resolve of two abstract states in crisis (see Section "Research Design"). 10. Some might wonder how much respondents can infer from State A's disputes with other, unknown states (that are not B). Additional background details would allow respondents to form more precise beliefs, yet respondents can still make general reputational inferences from A's past behavior even without these details. It is reasonable for respondents to infer that, if A stood firm three times in recent crises, then A is probably more likely to stand firm in a subsequent dispute, relative to the respondents' prior beliefs. Empirically, we demonstrate that our respondents can and do draw reputational inferences even when given relatively limited information. And as noted below, our results accord with  Renshon, Dafoe, and Huth (2018) , who find similar reputational effects across abstract and real-world designs. Regardless, future work on higher-order beliefs could (and ideally would) employ diverse research designs in order to test the sensitivity of this or any other particular design feature. 11. The vignette manipulates a large number of features, yielding a total of 8,640 cells. In this, it is similar to related survey techniques such as conjoint analysis  (Hainmueller, Hopkins, and Yamamoto 2014) . We discuss the advantages and possible disadvantages of factorial vignette experiments with many manipulations in Online Appendix B.3. 12. Similar questions were used in  Renshon, Dafoe, and Huth (2018)  and  Kertzer, Renshon, and Yarhi-Milo (2021) , though neither define resolve precisely and both discuss only first-order effects. 13. For recent examples, see  Walt (2015)  and  Walt (2016) . 14. For other experimental work that uses non-elite samples to test theories of bargaining, see  Renshon, Lee, and Tingley (2017) ,  Kertzer, Renshon, and Yarhi-Milo (2021) , and Cebul, Dafoe, and Monteiro (Forthcoming). 15. Main results are shown graphically below-full regression tables can be found in Online Appendix C.3. 16. The minimum likelihood respondents could give for A standing firm is 10 percent (0 percent to 20 percent), the maximum 90 percent (80 percent to 100 percent). This effect is approximately equal to one standard deviation of the resolve variable (m ¼ 62percent, s ¼ 21percent), which is equivalent to a Cohen's d ¼ 1; Cohen offered d ¼ 0:8 as a "big" effect. 17. The capabilities results, as well as results from all other manipulations, can be found in Online Appendix C. 18. We do not manipulate interests, but  Kertzer, Renshon, and Yarhi-Milo (2021, 22)  find similar results when doing so: even when told that one side has high stakes and high relative power, the effect of past behavior on perceptions of resolve is large and significant. Figure 2 . 2 Figure 2. A model of belief updating, where k denotes the level of belief updating, R k i denotes beliefs about actor i's resolve after the kth level of updating, and DR k i denotes the change in beliefs about i's resolve due to the kth level of updating. If conditions C1 and C2 are met, then the thick black lines have DR k i ¼ DR kÀ1 j  y, and we can estimate y as D RB D RA
 Figure 3 . 3 Figure 3.Effect of A's past behavior against "other countries" (not B) on perceived resolve, relative to respondents that received no information about past behavior (A's baseline probability of standing firm % 60 percent). X-axis displays percentage point change; horizontal lines represent 90 percent and 95 percent CIs.
 Figure 4 . 4 Figure 4. jyj, n, and the Interdependence Multiplier. Vertical line indicates j ŷj ¼ 0:73.
 Effect of A's past behavior on perceptions of A's resolve, relative to respondents that received no information about past behavior (baseline probability of standing firm % 60 percent). X-axis displays percentage point change; horizontal lines represent 90 percent and 95 percent CIs. A Backed Down Three Times Treatment A Backed Down in Most Recent Crisis, History of Not Backing Down A Did Not Back Down Three Times 0.2 0.1 0.0 0.1 0.2 Change in Perceptions of A's Resolve Figure 1.
 To test this idea, we need a treatment that affects beliefs about Rk i À RkÀ1 i A's resolve, but cannot affect B's resolve through any channel except (beliefs about) A's resolve. If such a treatment affects B's resolve, then we can conclude that perceptions of B's resolve are influenced by perceptions of A's resolve-in other words, that respondents form higher-order beliefs and that resolve is interdependent (H INT ).It is useful to define the relevant estimand and identifying assumptions more formally. Let R i denote the resolve of agent i, k denote a level of belief updating, and Rk i denote an observer's beliefs about actor i's resolve after the kth level of updating. Next, define D Rk i as the change in beliefs about actor i's resolve after the kth level of updating, i.e.
			 Journal of Conflict Resolution 65 (7) (8)