Abstract: We present an innovative methodology for studying and teaching the impacts of AI through a role-play game. The game serves two primary purposes: 1) training AI developers and AI policy professionals to reflect on and prepare for future social and ethical challenges related to AI and 2) exploring possible futures involving AI technology development, deployment, social impacts, and governance. While the game currently focuses on the inter-relations between short-, mid-and longterm impacts of AI, it has potential to be adapted for a broad range of scenarios, exploring in greater depths issues of AI policy research and affording training within organizations. The game presented here has undergone two years of development and has been tested through over 30 events involving between 3 and 70 participants. The game is under active development, but preliminary findings suggest that role-play is a promising methodology for both exploring AI futures and training individuals and organizations in thinking about, and reflecting on, the impacts of AI and strategic mistakes that can be avoided today.

Shahar Avin
Ross Gruetzemacher
Raymond J Harbert
James Fox

Exploring AI Futures Through Role Play

Introduction Artificial intelligence (AI) has an enormous potential to have a radical effect on our future, and we all have a vested interest in trying to ensure that AI is beneficial for humanity. However, there is currently no consensus in the AI research community on what the long-term impacts of AI, potentially including general artificial intelligence, will be, or the length of time over which such changes will take place. Moreover, there are many nearand mid-term concerns that have the potential for transformative societal change  [5, 14] . The vast uncertainty around AI futures, especially ones that involve transformative societal changes, has precipitated the creation of an AI strategy research community that seeks to ensure the development of AI for the social good  [8] . In collaboration with researchers in this community, we have developed an AI futures role-play game, which we present here, to complement the research being done and to disseminate key questions and findings to individuals and organizations engaged with, or seeking to engage with, AI strategy and policy. A role-play game involves a collection of people assuming roles to enact a situation in a realistic manner so as to predict a possible future if certain strategies are employed. Role-plays have a track record of high forecasting, strategic and educational utility in a wide variety of fields, including business  [15, 21] , medicine  [18] , politics  [16]  and education  [6, 21] . In the domain of AI strategy, we believe that role-plays can illuminate plausible futures, identify critical decision points, educate, raise awareness, highlight gaps in research knowledge and promote shared visions for beneficial AI. In this paper we first review the wargaming literature to show where role-plays have been successfully used in other domains, before next looking at why role-plays offer particular benefits within the AI strategy space. We then describe the current structure of the game (still under active development), and we highlight some of the lessons learned from the 30 game events to date. We conclude with a discussion of the benefits and shortcomings of this AI role-play game, our plans for further development and dissemination, and a call for others to try roleplay as a methodology for exploring AI futures. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author. AIES  '20, February 7-8, 2020 , New York, NY, USA © 2020 Copyright is held by the owner/author(s). ACM ISBN 978-1-4503-7110-0/20/02. https://doi.org/10.<PHONE>.<PHONE> 2 Literature Review "Serious" role-plays are most well known in their military form, wargames, where they have been used, since their Prussian development in the late 18th century, to teach military strategy in a safe but dynamic manner. Wilkinson  [24]  championed the introduction of wargames into the British army in the late 1800's and claimed that "probably no form of military study is more useful if properly conducted". Original wargames had two militaries (teams of players) fighting over some well-defined territory (the board). More recently, emerging technologies and distributed warfare necessitated wargames to shift focus from territory and troop movement to also include the diplomacy of international relations  [22, 23] , nuclear conflict escalation  [20]  and cyber operations  [1] . The potential of role-plays to provide exploration and training in uncertain and high stakes situations has been recognized in other fields. In business role-plays, for example, the conflict arises between a company and its competitors vying for market share and profit. Hamel and Prahalad  [15]  explain the popularity of role-plays in business as due to their ability to develop foresight, an essential skill for competitive survival: "an organization which does not develop a picture of the future, which does not develop foresight, is most likely not involved in the future." In education  [6, 21] , medicine  [18]  and politics  [16] , role-plays are used for formulating strategy, developing foresight, crisis response preparation, and for training and recruitment exercises. Nevertheless, despite their proven utility in a variety of different fields, role-plays have so far not been used in AI futures studies  [3] .
 When are role plays suitable? Parson  [19]  claims that role-plays are most appropriate for situations involving high enough stakes to merit substantial investment in knowledge about decision-making, but whose complexity imposes limits on the usefulness of standard decision-making procedures. Parson  [19]  gives three suitability conditions for role-plays: 1. Key outcomes depend on the interacting decisions of multiple agents. 2. Decision choice sets are ambiguous or poorly known. 3. Large numbers of complex organizations may be required to work together. The existence of multiple interacting complex agents compounded with a huge set of possible decisions creates an enormous potential scenario space. It is too difficult for one expert alone to think beyond linear views of reality, but roleplays allow non-linear effects to be explored. For instance, one can look for convergent outcomes arising from different strategies or identify the decisions pivotal in shaping the scenario's end state. Armstrong  [2]  compares and contrasts different forecasting techniques and adds that a role-play needs: 4. Large changes expected. 5. Conflict between the decision makers. If only small changes are expected, forecasters can extrapolate from current or historical examples. However, large technological step-changes -such as the discovery of electricity or the invention of the world wide web -entirely transform history's trajectory and are much harder to predict. This observation has been captured in Roy Amara's law  [17] : "we tend to overestimate the effect of a technology in the short run and underestimate the effect in the long run." Finally, conflict generates oppositional friction which catalyzes the generation of creative ideas. Without the competitive nature of an adversarial interaction, it is hard to generate the surprising edge-case scenarios which are exactly the ones hoped to be reached by role-plays.
 Do They Work? There is empirical evidence substantiating role-playing's greater forecasting accuracy over unaided judgements in certain situations. Green, Graefe and Armstrong  [11]  showed that both experts and novices were terrible at forecasting conflict decisions if they were simply asked to "put yourself in the other person's shoes". This is because, regardless of expertise, it is too difficult to individually think through the interactions of competing characters with divergent goals in complex situations. However, if the subjects adopted the roles of the protagonists in the conflicts and interacted with each other, their group decisions predicted the actual decisions with much greater accuracy -the greater realism afforded by role-playing enabled superior forecasts to be made. Role-plays can also mitigate against myopia by helping overcome various different psychological biases. Cyert, March and Starbuck  [7]  found that subjects presented with the same data produced significantly different forecasts depending on which role they were told to act out and Babcock et al.  [4]  evidenced the tendency of role-players to make biased judgements and interpret the briefing material differently according to the role they were assigned. This explains why roleplaying benefits from larger groups and greater diversity. Duke and Geurts  [10]  found that role-playing activities in a political setting can mitigate the danger of 'groupthink', where only readily politically feasible or easy-to-implement strategies are discussed. It is not guaranteed that every role-play will provide novel insights and any insights that do arise might not be relevant or applicable. However, even end-game states that have ostensibly departed significantly from common sense are worth exploring. DeWeerd  [9]  found that when predicting the future, readers tended to reject all predictions which didn't match their own and yet were willing to accept at the hand of history what they would instantly reject if predicted in advance by someone else (examples of "unthinkable" outcomes may include the result of the Brexit referendum, the election of Donald Trump, or the attack on Pearl Harbor). Edge-case scenarios are a key virtue of role-plays since they throw up black-swan eventsunprecedented and low probability, but high impact events which are invisible to canonical forecasting techniques.
 Methods The first task when planning a game is to identify the objective (i.e. training, educational, exploring futures or a mix of these). The project to date has explored two different methodological approaches to the game, with slightly different objectives: unstructured and semi-structured games. In both cases, the aim of the game was to allow players to experience from a firstperson perspective significant changes that could be brought by the development and deployment of AI technologies, and to have players face strategically challenging decisions from a position of power, be it governmental, corporate, military, societal or other.
 Unstructured role-play For the first two years of the project, we explored various forms of unstructured role-play. Players were drawn from a variety of different backgrounds with varying AI and geopolitical expertise. The players choose their own roles and took unconstrained actions, limited only by the time available and their own creativity. Resolution of actions thus depended heavily on the "world model" of the game's facilitator, which we aimed to address through regular engagement with the AI research literature and the AI strategy research field. The overall aim of these role-plays was to trial out the methodology, and to give players a space in which to explore strategic questions that they found interesting. Whenever it was possible, early discussions with the players or a coordinator helped to determine the initial frame for the scenario and roles adopted by the players. If this was not possible, these would be chosen by the facilitator or in collaboration with the players at the beginning of the game. We experimented with different ways to brief the players, including the facilitator meeting with each player ahead of the session (for 15-30 minutes each) to discuss the strategic questions they are interested in and to flesh out the role for that player. In an alternative approach, the facilitator would tell players the available roles and provide the general framing of the game, before asking the players to do their own background reading on those roles. Which organizations and roles will matter for the future of AI is an open research question: as time goes by, the number of identified stakeholders only increases, from governments to corporations, militaries to NGOs, artists, environmentalists, criminals and citizens, AI will shape, and be shaped by, a vast number of stakeholders. Nonetheless, the power distribution amongst these actors is not equal, and in varying the organizations and roles in the unstructured role-play we were able to explore different narratives that emerge when different groups and types of power are at play. Over 30 games we trialed many different roles. These broadly clustered around: During the game, the participants would "assume" their chosen/assigned role and take actions according to their model of that actor. Participants are instructed to deliberately try to bring about a world that is in line with the goals of the role they are playing, but they should be mindful of real-world constraints. This is because the more closely the simulation captures essential behavioral features of the real system, the more likely insights are to be relevant. On the other hand, a key strength of role-plays is in exploring edge-case scenarios. Thus, as a general rule of thumb, actions should be realistic but not overly conservative. For example, one in ten actions should be an action that the participant assigns 10% likelihood of being taken. Participants are encouraged to "flesh out" the world model by adding details that complement the details provided by the facilitator.  Government organizations During previous games, participants tried out a wide range of actions, including: Games typically involve 4-8 turns, with each turn representing one or two years, thus covering a period of 4-16 years, with the starting point set in the foreseeable future (as near as 2020 or as far as 2028). The game proceeds by alternating between negotiation phases, private action phases, public actions phases, and "world updates" from the facilitator. Outcomes of actions are determined by the facilitator, sometimes assisted by a random number generator, and relayed either to teams directly or in the "world updates" at the beginning of the next turn. Figure  1  depicts the final state of the world for a recent game held in Cambridge. Immediately at the end of the scenario role-play there is a debrief where any hidden state of the world is revealed, and participants evaluate how well their assigned role performed relative to their goal. Participants are asked to reflect on the final state of the world, on actions or developments that they found surprising, and to record any disagreements with how actions were chosen or resolved. This phase of gleaning participants' insights is crucial for challenging the facilitator's own subjective biases or knowledge gaps whilst bolstering the participants' internalization of what they have learned. 
 Semi-structured role-play Based on our experience with unstructured role-play, we started developing a semi-structured game that involves a predefined game system for AI R&D, and a pre-defined set of organizations and characters, to create an experience that explores several questions we have found to be particularly pertinent, including: To explore these questions, we designed an eight-player game with the following roles: We also created organization sheets and character sheets with key information to familiarize players with their roles. As in the unstructured game, players were asked to spend each turn negotiating with each other before deciding upon a set of private and public actions. However, we made AI R&D actions structured around a technology tree (Figure  2 ) and a set of products and capabilities (Figure  3 ).
 State of the World: The unemployment rate is now 17.1%. However, people are content for the most part because of social services such as Universal Basic Income. Multiple sources have confirmed to the New York Times that AGI has been developed. Sources range in assessing the system's safety, from 25% to 75% confidence. China is preparing to establish a small city on Mars. The first 10,000 citizens will arrive during 2035.
 Paper Presentation AIES '20, February 7-8, 2020, New York, NY, USA Each turn, the different organizations have a limited amount of AI "talent" they can allocate towards pursuing technological advancement (research) or products and capabilities (development). This can be pursued in public or in private. R&D success is determined by dice rolls, with one dice per "talent" allocated to a project. The players' action can affect the distribution of "talent" in the "world". This resource management mechanic was added to explore conditions of competition and scarcity in the input to AI R&D. This is one such factor that generates an adversarial tension which drives players to seek more creative solutions. Another mechanic we added to the game was "world chaos", a catch-all parameter for global instability, whereby we capture various negative externalities of AI, including the potential of AI "hype" to distract states from addressing issues such as inequality or climate change.
 Preliminary Results The unstructured game format has allowed the exploration of many different scenarios, but consequently it is difficult to identify patterns in the scenarios played. Overall, participant satisfaction appears high, with most players indicating their interest in playing again, and many interested in reading more about different organizations, roles, actions or strategic analyses. The following range of topics have come up in at least one of the scenarios that emerged from unstructured role-play (this list is not exhaustive): It would often be the case that five or more of the above topics would be touched upon in a single game.  The semi-structured game is more constrained, and therefore provides a better opportunity to detect patterns of gameplay and decision-making. While the number of games played with the new system to date is small, and the game system is still under active development, a few patterns are beginning to emerge. First, and perhaps not surprisingly, the familiarity of players with the roles assigned to them varies greatly -while many have at least some idea of what the President of the United States cares about or what they can do, that is not the case for tech CEOs or heads of AI labs; while a good facilitator can use the game to provide some basic information about the role and some of the decisions they face (as they relate to AI), the game is more useful for all participants when the players start the game with more background knowledge, both about AI and about the contexts in which decisions about AI are made. Second, the rapid growth of options that comes with the proliferation of available AI capabilities can be overwhelming to players, who respond to the large number of available R&D avenues by either choosing randomly, picking quickly a "good enough" target, or aiming for a specific long-term technology. While this is to some extent an artefact of the game mechanics, we think it may capture a real issue, especially if AI progress lends itself to faster AI R&D and if progress remains stochastic and uncertain. Third, cooperation between actors on AI safety and (some) restriction on destabilizing uses of AI seem to both be robustly beneficial, and often necessary, for all players to achieve a good outcome. Such agreements are easier to establish if players are biased towards pro-social strategies, or when cooperation begins early in the game and is maintained and expanded, or when players recognize a common threat (e.g. an unsafe or destabilizing technology nearing completion or after a major accident). Competitive behavior, including adversarial actions (espionage, sabotage, poaching, tariffs), can sometimes be excused when a common threat emerges and the need for cooperation becomes apparent. Fourth, in almost all games players will continue pursuing ever-more-advanced technologies and capabilities, with all the resources they can afford to spend on R&D, even if the potential value of these advances is unclear, and even when earlier advances have proven to be destabilizing or risky. Fifth, while the game has a built-in system for AI R&D, some players are happy to innovate "around" this system, proposing new products or establishing new organizations during the game. Finally, different sets of possible coalitions and alliances are explored and can prove stable, including between government and corporate entities within a state, between governments, between organizations, and between sub-roles within organizations (e.g. between heads of AI labs).
 Discussion and Conclusions The AI role-play game described here presents an innovative methodology for studying and teaching the impacts of AI. It can educate, raise awareness and help illuminate the positive and negative effects of certain developments, strategic decisions, and Paper Presentation AIES '20, February 7-8, 2020, New York, NY, USA inter-stakeholder interactions thereby potentially improving the outcomes of deploying current and future AI technologies. The versatility of the technique, including its ability to be adapted to different problems throughout organizations to address the near-, mid-and long-term impacts of AI, is well-suited to the large variety of challenges encountered in the development and deployment of AI technologies across all aspects of human life. Many of the other benefits of role-play games discussed in the literature review are also applicable. The games we have conducted to date, and especially the semi-structured games, focused on a set of questions and insights that we find particularly salient. These include the interaction between near-, mid-and long-term risks, challenges of cooperation and competition between powerful actors across states of public/private lines, trade-offs between public and private research and between research and product development, and the experience of making potentially transformative decisions from a first-person (powerful) perspective under conditions of uncertainty and time pressure. The feedback from players to date suggests that these aspects of the game indeed appear salient to players, and lead to greater engagement with these questions. Based on players' feedback, and from design constraints of the game itself, we also note several shortcomings and limitations we have not yet been able to address. For example, in our focus on strategic decision making, we have focused on roleplay from the perspective of powerful actors, rather than actors with marginalized standpoints, or more detached viewpoints that highlight systemic and emergent drivers; the literature suggests that role-plays and games can be adapted for these perspectives, and have been powerful in building empathy or systems thinking when done well, and this is a potential direction for expansion. Another limitation is the sped-up timeline, that imagines dramatic, and occasionally transformative, technological shifts within an almost-foreseeable political horizon. This reflects our limited ability to conduct a realistic role-play that takes place in the political reality of 2050 or 2100, but it does mean we adopt an assumption of fairly rapid technology progress, which is in no way certain -experts significantly disagree about timelines to transformative AI capabilities and impacts  [12, 13] . This design decision was made in part to play to the strength of role-plays, which shine in exploring large and uncertain effects, partly to create a more engaging and dramatic setting, and partly because if transformative impacts happen in a matter of decades, it becomes of paramount importance to prepare decision-makers to handle the decisions they will need to face. We also stress the communication of this bias in every game debrief, but we see how this sped-up timeline may be less appealing to some, and are exploring versions of the game that focus on less dramatic changes while exploring in greater nuance short-and mediumterm impacts of technologies that are already making real research progress. Moving forward, we will continue to develop the game to increase realism, reflect new strategic research questions as they emerge, and cover more scenarios of interest to a broader set of stakeholders. We aim to expand the context of the game, through different unstructured and semi-structured scenarios and mechanics, to cover an increasing range of actors, actions, technologies and impacts of AI. This necessitates staying in constant contact and mutual feedback with the developing fields of AI futures and AI strategy as the impacts of AI become more apparent in every corner of life. In addition, we hope to continue to develop the educational potential of the game into something that can be used to raise awareness and train a variety of audiences and publics, including younger members of the public, through the development of a digital platform and a digital version of the game. Readers interested in exploring the game are invited to register their interest,  1  and those who wish to join the development of the game are encouraged to contact the authors. By allowing stakeholders to explore alternative AI futures, prepare for surprises and uncertainty, and learn from "simulated" mistakes, we hope that the methodology we propose will help win the race between the capability of the technologies we develop and our wisdom in deploying and governing them. For these reasons we see the method's impact on the use of AI for social good as being far reaching and we invite members of the community to join us in exploring the future of AI through role-play. Figure 1 : 1 Figure 1: Final state of the world from a recent unstructured game. The game involved three players playing the roles of presidents for the USA, PRC and EU. Technological progress and impacts of players' actions were determined by the facilitator.
  How do the geopolitical relations between the two leading AI powers, US and China, affect the kinds of technologies developed and the way they are deployed?  How do state-corporate relationships play out as AI technologies become more advanced and allow for a greater generation and consolidation of wealth?  How do states make decisions about the deployment of AI capabilities in strategic domains such as military and social control?  How do corporations balance the pursuit of risky basic research that could lead to transformative capabilities with nearer term, profit generating products and services?  How, and to what extent, are states willing to "pick up the tab" for negative externalities created by their corporate champions?  What factors contribute to a decision to make a research project private/secret, or conversely to publish results and share capabilities? How do actors respond when they suspect other actors of pursuing advanced research in secret?  How do decisions and impacts relating to near-term AI technologies affect the landscape in which decisions are made about mid-term AI technologies, and similarly how do those affect the setting for decisions about long-term AI?
  PRC government: president and minister of national defense  US government: president and secretary of defense  Tencent: CEO and head of AI lab  Alphabet: CEO and head of AI lab
 Figure 2 : 2 Figure 2: AI Technology Tree used in the game. Each box represents a technology that can be pursued by the players, with associated products and capabilities. Technologies advance from left to right, with progress on earlier technologies required before later technologies are "unlocked".
 Figure 3 : 3 Figure 3: An example of AI products a player can develop in the game. The required technology and the effect on game parameters are listed at the bottom of the card.
 Non-governmental actors, including academia, independent research organizations, advocacy groups and think tanks.  Supra-national actors, e.g. the EU or the UN. and roles in different countries, including the executive, the military, diplomatic services, health, and government R&D functions.  Corporate actors in different geographies and sectors, including tech (based on business models we see around the world, e.g. advertising, e-commerce or tech infrastructure), robotics, weapons, health. 
			 To register your interest in participation, please fill out the form at https://sites.google.com/view/intelligence-rising/home.


Abstract: The increasing complexity of robotic systems are pressing the need for them to be transparent and trustworthy. When people interact with a robotic system, they will inevitably construct mental models to understand and predict its actions. However, people's mental models of robotic systems stem from their interactions with living beings, which induces the risk of establishing incorrect or inadequate mental models of robotic systems and may lead people to either under-and over-trust these systems. We need to understand the inferences that people make about robots from their behavior, and leverage this understanding to formulate and implement behaviors into robotic systems that support the formation of correct mental models of and fosters trust calibration. This way, people will be better able to predict the intentions of these systems, and thus more accurately estimate their capabilities, better understand their actions, and potentially correct their errors. The aim of this full-day workshop is to provide a forum for researchers and practitioners to share and learn about recent research on people's inferences of robot actions, as well as the implementation of transparent, predictable, and explainable behaviors into robotic systems.

Maartje M A De Graaf
Bertram F Malle
Anca Dragan
Tom Ziemke

Explainable Robotic Systems

As artificial agents, and especially socially interactive robots, enter human society, the demands for these agents to be transparent and explainable grow rapidly. When systems are able, for example, to explain how they made classifications or arrived at a decision, users are better able to judge the systems' accuracy and have more calibrated trust in them (e.g.,  [8]    [10] ). More and more AI systems process vast amounts of information and make classifications or recommendations that humans use for financial, employment, medical, military, and political decisions. More precariously yet, autonomous social robots, by definition, make decisions reaching beyond direct commands and perform actions with social and moral significance for humans. The demand for these social robots to become transparent and explainable is particularly urgent. However, to make robots explainable, we need to understand how people interpret the behavior of such systems and what expectations they have of them. In this workshop, we will address the topics of transparency and explainability, for robots in particular, from both the cognitive science perspective and the computer science and robotics perspective. Cognitive science elucidates how people interpret robot behavior; computer science and robotics elucidate how the computational mechanisms underlying robot behavior can be structured and communicated so as to be human-interpretable. The implementation and use of explainable robotic systems may prevent the potentially frightening confusion over why a robot is behaving the way it is. Moreover, explainable robot systems may allow people to better calibrate their expectations of the robot's capabilities and be less prone to treating robots as almost-humans.
 BACKGROUND When people interact with a robotic system, they construct mental models to understand and predict its actions. However, people's mental models of robots stem from their interactions with living beings. Thus, people easily run the risk of establishing incorrect or inadequate models of robotic systems, which may result in self-deception or even harm  [12] . Moreover, a long-term study  [9]  showed that initially established (incorrect) mental models of an intelligent information system remained robust over time, even when details of the system's implementation were given and initial beliefs were challenged with contradictory evidence. Incorrect mental models of AIS can have significant consequences for trust in such systems and, as a result, for acceptance of and collaboration with these systems  [10] . Several studies indicate that people distrust a robotic system when they are unable to understand its actions. When a robot fails to communicate its intentions, people not only perceive the robot as creepy or unsettling  [11] , they also perceive such robots as erratic and untrustworthy even when they follow a clear decision-making process  [5] . Indeed, when a robot is not transparent about its intentions (i.e., not providing any explanations for its behavior), people may even question its correct task performance and blame the robotic agent for its alleged errors  [4] . In addition to such cases of distrust, incorrect mental models of AIS can also lead to the opposite situation. People sometimes over-trust artificial agents, such as when they comply with a faulty robot's unusual requests  [7]  or follow the lead of a potentially inept robot  [6] . Thus, there should be little doubt about the value of making robotic systems perform easily interpretable actions and explain their own decisions and behaviors. This goal for explainable robots demands innovative work in computer science, robotics, and cognitive science, and our proposed workshop provides strong representation from all these fields. Additionally, we invite research that examines if and how such structures lead to better acceptance of robotic systems, to more accurate mental models of such systems, and to calibrated trust in these systems.
 WORKSHOP ACTIVITIES The aim of this full-day workshop is to provide a forum to share and learn about recent research on requirements for artificial agents' explanations as well as the implementation of transparent, predictable and explainable robotic systems. Extended time for discussions will highlight and document promising approaches and encourage further work. We welcome prospective participants to submit extended abstracts (max. 2 pages) that contribute to the discussion of people's interpretation of robot actions as well as the implementation of transparent, predictable, and explainable behaviors in robotic systems. After the conference, and with permission of the authors, we will provide online access to the workshop proceedings on a dedicated workshop website: explainableroboticsystems.wordpress.com. We encourage researchers to attend the workshop even without a paper submission, as a major portion of the workshop involves community engagement and discussion to foster uptake of concepts regarding explainable robotic systems within the field of HRI. The morning session will cover invited talks from Rachid Alami, Joanna Bryson, Bradley Hayes, and Alessandra Sciutti, as well as short presentations by authors of accepted papers, followed by general discussion. The afternoon will be devoted to break-out sessions to discuss the next steps in research and development of explainable and transparent robotic systems. Groups will be composed of representatives of different disciplines in order to work on integrating the multiple necessary perspectives in this endeavor. To boost the discussions, we will ask presenters to prepare questions or raise pressing issues that provide starting points for the discussion groups.  is a postdoctoral research associate at the Department of Cognitive, Linguistic, and Psychological Sciences, Brown University, USA. She has a Bachelor of Business Administration in Communication Management (2005), a Master of Science in Media Communication (2011), and a PhD in Communication Science and Human-Robot Interaction (2015). Her research research interest focuses on peopleâĂŹs social, emotional and cognitive responses to robots including the societal and ethical consequences of those responses. Bertram Malle is Professor of Psychology in the Department of Cognitive, Linguistic, and Psychological Sciences at Brown University and Co-Director of the Humanity-Centered Robotics Initiative at Brown. He was trained in psychology, philosophy, and linguistics at the University of Graz, Austria, and received his Ph.D. in Psychology from Stanford University in 1995. His research focuses on social cognition, moral psychology, and human-robot interaction. Anca Dragan is an Assistant Professor in Electrical Engineering and Computer Sciences at UC Berkeley. She received her PhD from the Robotics Institute at Carnegie Mellon University in 2015 and now runs a lab on algorithmic human-robot interaction. She has expertise in transparent and explainable robotics and AI, with applications in personal robots and autonomous cars. Tom Ziemke is Professor of Cognitive Science at the University of Skövde and Professor of Cognitive Systems at Linköping University, Sweden. He received his Ph.D. in Computer Science from Sheffield University, UK, in 2000. His main research interests are embodied cognition and social interaction, and in recent years in particular people's interaction with different types of autonomous systems, ranging from social robots to automated vehicles.


Abstract: Artificial superintelligence (ASI) is artificial intelligence (AI) with capabilities that are significantly greater than human capabilities across a wide range of domains. A hallmark of the ASI issue is disagreement among experts. This paper demonstrates and discusses methodological options for modeling and interpreting expert disagreement about the risk of ASI catastrophe. Using a new model called ASI-PATH, the paper models a well-documented recent disagreement between Nick Bostrom and Ben Goertzel, two distinguished ASI experts. Three points of disagreement are considered: (1) the potential for humans to evaluate the values held by an AI, (2) the potential for humans to create an AI with values that humans would consider desirable, and (3) the potential for an AI to create for itself values that humans would consider desirable. An initial quantitative analysis shows that accounting for variation in expert judgment can have a large effect on estimates of the risk of ASI catastrophe. The risk estimates can in turn inform ASI risk management strategies, which the paper demonstrates via an analysis of the strategy of AI confinement. The paper find the optimal strength of AI confinement to depend on the balance of risk parameters (  1 ) and (  2 ). Povzetek: Predstavljena je metoda za modeliranje in interpretiranje razlik v mnenjih ekspertov o superinteligenci.

Seth D Baum
Anthony M Barrett
Roman V Yampolskiy

Modeling and Interpreting Expert Disagreement About Artificial Superintelligence

Introduction Artificial superintelligence (ASI) is artificial intelligence (AI) with capabilities that are significantly greater than human capabilities across a wide range of domains. If developed, ASI could have impacts that are highly beneficial or catastrophically harmful, depending on its design A hallmark of the ASI issue is disagreement among experts. Experts disagree on if ASI will be built, when it would be built, what designs it would use, and what its likely impacts would be.  1  The extent of expert disagreement speaks to the opacity of the underlying ASI issue and the general difficulty of forecasting future technologies. This stands in contrast with other major global issues, such as climate change, for which there is extensive expert agreement on the basic parameters of the issue  (Oreskes 2004 ). Expert consensus does not guarantee that the issue will be addressed-the ongoing struggle to address climate change attests to this-but it does offer direction for decision making. In the absence of expert agreement, those seeking to gain an understanding of the issue must decide what to believe given the existence of the disagreement. In some cases, it may be possible to look at the nature of the disagreement and pick sides; this occurs if other sides clearly have flawed arguments that are not worth giving any credence to. However, in many cases, multiple sides of a disagreement make plausible arguments; in these cases, the thoughtful observer may wish to form a belief that in some way considers the divergent expert opinions. This paper demonstrates and discusses methodological options for modeling and interpreting expert disagreement about the risk of ASI catastrophe. The paper accomplishes this by using a new ASI risk model called ASI-PATH  (Barrett and Baum 2017a; 2017b) . Expert disagreement can be modeled as differing estimates of parameters in the risk model. Given a set of differing expert parameter estimates, aggregate risk estimates can be made using weighting functions. Modeling expert disagreement within the context of a risk model is a method that has been used widely across a range of other contexts; to our knowledge this paper marks the first application of this method to ASI. The paper uses a well-documented recent disagreement between Nick Bostrom and Ben Goertzel as an illustrative example-an example that is also worthy of study in its own right. Bostrom and Goertzel are both longstanding thought leaders about ASI, with lengthy research track records and a shared concern with the societal impacts of ASI. However, in recent publications,  Goertzel (2015; 2016 ) expresses significant disagreement with core arguments made by  Bostrom (2014) . The Bostrom-Goertzel disagreement is notable because both of them are experts whose arguments about ASI can be expected to merit significant credence from the perspective of an outside observer. Therefore, their disagreement offers a simple but important case study for demonstrating the methodology of modeling and interpreting expert disagreement about ASI. The paper begins by summarizing the terms of the Bostrom-Goertzel disagreement. The paper then introduces the ASI-PATH model and shows how the Bostrom-Goertzel disagreement can be expressed in terms of ASI-PATH model parameters. The paper then presents model parameter estimates based on the Bostrom-Goertzel disagreement. The parameter estimates are not rigorously justified and instead are intended mainly for illustration and discussion purposes. Finally, the paper applies the risk modeling to a practical problem, that of AI confinement.
 The Bostrom-Goertzel disagreement Goertzel (2015; 2016) presents several disagreements with  Bostrom (2014) . This section focuses on three disagreements of direct relevance to ASI risk.
 Human evaluation of AI values One disagreement is on the potential for humans to evaluate the values that an AI has. Humans would want to diagnose an AI's values to ensure that they are something that humans consider desirable (henceforth "human-desirable"). If humans find an AI to have human-undesirable values, they can reprogram the AI or shut it down. As an AI gains in intelligence and power, it will become more capable of realizing its values, thus making it more important that its values are humandesirable. A core point of disagreement concerns the prospects for evaluating the values of AI that have significant but still subhuman intelligence levels. Bostrom indicates relatively low prospects for success at this evaluation, whereas Goertzel indicates relatively high prospects for success.  Bostrom (2014, p.116-119)  posits that once an AI reaches a certain point of intelligence, it might adopt an adversarial approach. Bostrom dubs this point the "treacherous turn": The treacherous turn: While weak, an AI behaves cooperatively (increasingly so, as it gets smarter). When the AI gets sufficiently strongwithout warning or provocation-it strikes, forms a singleton [i.e., takes over the world], and begins directly to optimize the world according to the criteria implied by its final values.  (  Goertzel provides a contrasting view, focusing on Step 2. He posits that an AI of intermediate intelligence is unlikely to successfully pretend to have humandesirable values because this would be too difficult for such an AI. Noting that "maintaining a web of lies rapidly gets very complicated" (Goertzel 2016, p.55), Goertzel posits that humans, being smarter and in control, would be able to see through a sub-human-level AI's "web of lies". Key to Goertzel's reasoning is the claim that an AI is likely to exhibit human-undesirable behavior before it (A) learns that such behavior is human-undesirable and (B) learns how to fake humandesirable behavior. Thus, Step 2 is unlikely to occurinstead, it is more likely that an AI would either have actual human-desirable values or be recognized by humans as faulty and then be reprogrammed or shut down. Goertzel does not name his view, so we will call it the sordid stumble: The sordid stumble: An AI that lacks humandesirable values will behave in a way that reveals its human-undesirable values to humans before it gains the capability to deceive humans into believing that it has human-desirable values. It should be noted that the distinction between the treacherous turn and the sordid stumble is about the AI itself, which is only one part of the human evaluation of the AI's values. The other part is the human effort at evaluation. An AI that is unskilled at deceiving humans could still succeed if humans are not trying hard to notice the deception, while a skilled AI could fail if humans are trying hard. Thus, this particular Bostrom-Goertzel debate covers only one part of the AI risk. However, it is still the case that, given a certain amount of human effort at evaluating an AI's values, Bostrom's treacherous turn suggests a lower chance of successful evaluation than Goertzel's sordid stumble.
 Human creation of human-desirable AI values A There is nothing paradoxical about an AI whose sole final goal is to count the grains of sand on Borcay, or to calculate the decimal expansion of pi, or to maximize the total number of paperclips that will exist in its future light cone. In fact, it would be easier to create an AI with simple goals like these than to build one that had a human-like set of values and dispositions  (Bostrom 2014, p.107 ). The logic of the above passage is that creating an AI with human-desirable values is more difficult and thus less likely to occur.  Goertzel (2016) , citing Sotala (2015), refers to this as the difficulty thesis: The difficulty thesis: Getting AIs to care about human values in the right way is really difficult, so even if we take strong precautions and explicitly try to engineer sophisticated beneficial goals, we may still fail  (Goertzel 2016, p.60) .  Goertzel (2016)  discusses a Sotala (2015) argument against the difficulty thesis, which is that while human values are indeed complex and difficult to learn, AIs are increasingly capable of learning complex things.Per this reasoning, giving an AI human-desirable values is still more difficult than, say, programming it to calculate digits of pi, but it may nonetheless be a fairly straightforward task for common AI algorithms. Thus, while it would not be easy for humans to create an AI with human-desirable values, it would not be extraordinarily difficult either.  Goertzel (2016) , again citing Sotala (2015), refers to this as the weak difficulty thesis: The weak difficulty thesis. It is harder to correctly learn and internalize human values, than it is to learn most other concepts. This might cause otherwise intelligent AI systems to act in ways that went against our values, if those AI systems had internalized a different set of values than the ones we wanted them to internalize. A more important consideration than the absolute difficulty of giving an AI human-desirable values is its relative difficulty compared to the difficulty of creating an AI that could take over the world. A larger relative ease of creating an AI with human-desirable values implies a higher probability that AI catastrophe will be avoided for any given level of effort put to avoiding it. There is reason to believe that the easier task is giving an AI human-desirable values. For comparison, every (or almost every) human being holds humandesirable values. Granted, some humans have more refined values than others, and some engage in violence or other antisocial conduct, but it is rare for someone to have pathological values like an incessant desire to calculate digits of pi. In contrast, none (or almost none) of us is capable of taking over the world. Characters like Alexander the Great and Genghis Khan are the exception, not the rule, and even they could have been assassinated by a single suicidal bodyguard. By the same reasoning, it may be easier for an AI to gain humandesirable values than it is for an AI to take over the world. This reasoning does not necessarily hold, since AI cognition can differ substantially from human cognition, but it nonetheless suggests that giving an AI humandesirable values may be the easier task.
 AI creation of human-desirable AI values A third point of discussion concerns the potential for an AI to end up with human-desirable values even though its human creators did not give it such values. If AIs tend to end up with human-desirable values, this reduces the pressure on the human creators of AI to get the AI's values right. It also increases the overall prospects for a positive AI outcome. To generalize, Bostrom proposes that AIs will tend to maintain stable values, whereas Goertzel proposes that AIs may tend to evolve values that could be more human-desirable. Bostrom's (2014) thinking on the matter centers on a concept he calls goal-content integrity: Goal-content integrity: If an agent retains its present goals into the future, then its present goals will be more likely to be achieved by its future self. This gives the agent a present instrumental reason to prevent alteration of its final goals  (Bostrom 2014, p.109-110) . The idea here is that an AI would seek to keep its values intact as one means of realizing its values. At any given moment, an AI has a certain set of values and seeks to act so as to realize these values. One factor it may consider is the extent to which its future self would also seek to realize these values. Bostrom's argument is that an AI is likely to expect that its future self would realize its present values more if the future self retains the present self's values, regardless of whether those values are human-desirable. Goertzel (2016) proposes an alternative perspective that he calls ultimate value convergence: Ultimate value convergence: Nearly all superintelligent minds will converge to the same universal value system (paraphrased from Goertzel 2016, p.60). Goertzel further proposes that the universal value system will be "centered around a few key values such as Joy, Growth, and Choice" (Goertzel 2016, p.60). However, the precise details of the universal value system are less important than the possibility that the value system could resemble human-desirable values. This creates a mechanism through which an AI that begins with any arbitrary human-undesirable value system could tend towards human-desirable values. Goertzel does not insist that the ultimate values would necessarily be human-desirable. To the contrary, he states that "if there are convergent 'universal' values, they are likely sufficiently abstract to encompass many specific value systems that would be abhorrent to us according to our modern human values" (Goertzel 2016, p.60). Thus, ultimate value convergence does not guarantee that an AI would end up with human-desirable values. Instead, it increases the probability that an AI would end up with human-desirable values if the AI begins with human-undesirable values. Alternatively, if the AI begins with human-desirable values, then the ultimate value convergence theory could cause the AI to drift to human-undesirable values. Indeed, if the AI begins with human-desirable values, then more favorable results (from humanity's perspective) would accrue if the AI has goal-content integrity.
 The ASI-PATH model The ASI-PATH model was developed to model pathways to ASI catastrophe  (Barrett and Baum 2016) . ASI-PATH is a fault tree model, which means it is a graphical model with nodes that are connected by Boolean logic and point to some failure mode. For ASI-PATH, a failure mode is any event in which ASI causes global catastrophe. Fault tree models like ASI-PATH are used widely in risk analysis across a broad range of domains. A core virtue of fault trees is that, by breaking catastrophe pathways into their constituent parts, they enable more detailed study of how failures can occur and how likely they are to occur. It is often easier to focus on one model node at a time instead of trying to study all potential failure modes simultaneously. Furthermore, the fault tree's logic structure creates a means of defining and quantifying model parameters and combining them into overall probability estimates. Indeed, the three points of the Bostrom-Goertzel disagreement (human evaluation of AI values, human creation of human-desirable AI values, and AI creation of human-desirable AI values) each map to one of the ASI-PATH parameters shown in Figure  1 . In Figure  1 , the top node is ASI catastrophe. The left branch covers events that lead to the ASI gaining "decisive strategic advantage", defined as "a level of technological and other advantages sufficient to enable it [the AI] to achieve complete world domination" (Bostrom, 2014, p. 78). The left branch models scenarios in which an initial "seed" AI undergoes recursive selfimprovement and "takes off", becoming successively more and more intelligent until it becomes an ASI. P1 is the probability that such an AI is possible in the first place. P2 is the probability that a seed AI is created and undergoes recursive self-improvement. P3 is the probability that the AI is contained from gaining decisive strategic advantage; the containment can occur at any point in the process from seed AI to ASI. Containment is any measure that prevents a seed AI from gaining decisive strategic advantage, either by limiting recursive self-improvement or by preventing ASI from gaining decisive strategic advantage. Containment includes confinement, in which the AI's ability to affect the rest of the world is restricted (Section 5), and enforcement, in which AI(s) prevent other AI(s) from gaining decisive strategic advantage.  2  The left branch of Figure  1  covers events that could lead to the ASI taking actions that are "unsafe", which is defined as actions that would result in a major global  2  Barrett and Baum (2017a, p. 400) define confinement as "restrictions built into the AI's hardware or software that limit the AI's ability to affect the rest of the world so that it does not gain decisive strategic advantage". This is slightly different than the Yampolskiy (2012) definition used in Section 5. This difference does not affect the overall argument of the present paper. catastrophe. P4 is the probability that humans will fail to make ASI goals safe. P5 is the probability that the ASI will not make its own goals safe. Finally, P6 is the probability that the ASI will not be deterred from acting unsafely by some other agent, potentially another AI. Because all the logic gates in Figure  1  are "AND", the probability of ASI catastrophe, P, is simply the product of the six component probabilities:    6 1 n n P P (1) For convenience, we assume {P1, P2, P6} = 1. These parameters are unrelated to the Bostrom-Goertzel disagreement as discussed in this paper. Instead, we focus on {P3, P4, P5}, for which there is significant disagreement. P3 relates to the Bostrom-Goertzel disagreement about human evaluation of AI values (Section 2.1). In general, it should be easier to contain an AI earlier in the recursive self-improvement process because at that point it has less intelligence with which it could resist containment. Therefore, one factor in P3 is the potential for human observers to determine early in the process that this particular AI should be contained. The easier it is for humans to evaluate AI values, the earlier in the process they should be able to notice which AIs should be contained, and therefore the more probable it is that containment will succeed. In other words, easier human evaluation of AI values means lower P3. P4 relates to the Bostrom-Goertzel disagreement about human creation of human-desirable AI values (Section 2.2). Human-desirable values are very likely to be safe in the sense that they would avoid major global catastrophe. While one can imagine the possibility that somehow, deep down inside, humans actually prefer global catastrophe, and thus that an AI with humandesirable values would cause catastrophe, we will omit this possibility. Instead, we assume that an AI with human-desirable values would not cause catastrophe. Therefore, the easier it is for humans to create AIs with human-desirable values, the more probable it is that catastrophe would be avoided. In other words, easier human creation of AI with human-desirable values means lower P4. P5 relates to the Bostrom-Goertzel disagreement about AI creation of human-desirable AI values (Section 2.3). We assume that the more likely it is that an AI would create of human-desirable values for itself, the more probable it is that catastrophe would be avoided. In other words, more likely AI creation of AI with humandesirable values means lower P5. For each of these three variables, we define two "expert belief" variables corresponding to Bostrom's and Goertzel's positions on the corresponding issue:  P3B is the value of P3 that follows from Bostrom's position, the treacherous turn.  P3G is the value of P3 that follows from Goertzel's position, the sordid stumble.  P4B is the value of P4 that follows from Bostrom's position, the difficulty thesis.  P4G is the value of P4 that follows from Goertzel's position, the weak difficulty thesis.  P5B is the value of P5 that follows from Bostrom's position, goal-content integrity.  P5G is the value of P5 that follows from Goertzel's position, ultimate value convergence. Given estimates for each of the above "expert belief" variables, one can calculate P according to the formula:       6 1 n nG nG nB nB P W P W P (2) In Equation  2 , W is a weighting variable corresponding to how much weight one places on Bostrom's or Goertzel's position for a given variable. Thus, for example, W3B is how much weight one places on Bostrom's position for P3, i.e. how much one believes that an AI would conduct a treacherous turn. For simplicity, we assume WnB + WnG = 1 for n = {3, 4, 5}. This is to assume that for each of {P3, P4, P5}, either Bostrom or Goertzel holds the correct position. This is a significant assumption: it could turn out to be the case that they are both mistaken. The assumption is made largely for analytical and expository convenience. This much is easy. The hard part is quantifying each of the P and W variables in Equation  2 . What follows is an attempt to specify how we would quantify these variables. We estimate the P variables by relating the arguments of Bostrom and Goertzel to the variables and taking into account any additional aspects of the variables. We aim to be faithful to Bostrom's and Goertzel's thinking. We estimate the W variables by making our own (tentative) judgments about the strength of Bostrom's and Goertzel's arguments as we currently see them. Thus, the P estimations aim to represent Bostrom's and Goertzel's thinking and the W estimations represent our own thinking. Later in the paper we also explore the implications of giving both experts' arguments equal weighting (i.e., WnB = WnG = 0.5 for each n) and of giving full weighting to exclusively one of the two experts. We make no claims to having the perfect or final estimations of any of these parameters. To the contrary, we have low confidence in our current estimations, in the sense that we expect we would revise our estimations significantly in the face of new evidence and argument. But there is value in having some initial estimations to stimulate thinking on the matter. We thus present our estimations largely for sake of illustration and discussion. We invite interested readers to make their own.
 P3 and W3: containment fails The human evaluation of AI values is only one aspect of containment. Other aspects include takeoff speed (faster takeoff means less opportunity to contain AI during recursive self-improvement) and ASI containment (measures to prevent an ASI from gaining decisive strategic advantage). Therefore, the Bostrom-Goertzel disagreement about human evaluation of AI values should only produce a relatively small difference on P3. Bostrom and Goertzel may well disagree on other aspects of P3, but those are beyond the scope of this paper. Bostrom's position, the treacherous turn, corresponds to a higher probability of containment failure and thus a higher value of P3 relative to Goertzel's position, the sordid stumble. We propose a 10% difference in P3 between Bostrom and Goertzel, i.e. P3B -P3G = 0.1. The absolute magnitude of P3B and P3G will depend on various case-specific details-for example, a seed AI launched on a powerful computer is more likely to have a fast takeoff and thus less likely to be contained. For simplicity, we will use P3B = 0.6 and P3G = 0.5, while noting that other values are also possible. Regarding W3B and W3G, our current view is that the sordid stumble is significantly more plausible. We find it relevant that AIs are already capable of learning complex tasks like face recognition, yet such AIs are nowhere near capable of outwitting humans with a web of lies. Additionally, it strikes us as much more likely that an AI would exhibit human-undesirable behavior before it becomes able to deceive humans, and indeed long enough in advance to give humans plenty of time to contain the situation. Therefore, we estimate W3B = 0.1 and W3G = 0.9.
 P4 and W4: humans fail to give AI safe goals The Bostrom-Goertzel disagreement about human creation of human-desirable AI values is relevant to the challenge of humans giving AI safe goals. Therefore, the disagreement can yield large differences in P4. Bostrom's position, the difficulty thesis, corresponds to a higher probability of humans failing to give the AI safe goals and thus a higher value of P4 relative to Goertzel's position, the weak difficulty thesis. The values of P4B and P4G will depend on various case-specific details, such as how hard humans try to give the AI safe goals. As representative estimates, we propose P4B = 0.9 and P4G = 0.4. Regarding W4B and W4G, our current view is that the weak difficulty thesis is significantly more plausible. The fact that AIs are already capable of learning complex tasks like face recognition suggests that learning human values is not a massively intractable task. An AI would not please everyone all the time-this is impossible-but it could learn to have broadly human-desirable values and behave in broadly human-desirable ways. However, we still see potential for the complexities of human values to pose AI training challenges that go far beyond what exists for tasks like face recognition. Therefore, we estimate W4B = 0.3 and W4G = 0.7.
 P5 and W5: AI fails to give itself safe goals The Bostrom-Goertzel disagreement about AI creation of human-desirable AI values is relevant to the challenge of the AI giving itself safe goals. Therefore, the disagreement can yield large differences in P5. Bostrom's position, goal-content integrity, corresponds to a higher probability of the AI failing to give itself safe goals and thus a higher value of P5 relative to Goertzel's position, ultimate value convergence. Indeed, an AI with perfect goal-content integrity will never change its goals. For ultimate value convergence, the key factor is the relation between ultimate values and human-desirable values; a weak relation suggests a high probability that the AI will end up with human-undesirable values. Taking these considerations into account, we propose P5B = 0.95 and P5G = 0.5. Regarding W5B and W5G, our current view is that goal-content integrity is significantly more plausible. While it is easy to imagine that an AI would not have perfect goal-content integrity, due to a range of realworld complications, we nonetheless find it compelling that this would be a general tendency of AIs. In contrast, we see no reason to believe that AIs would all converge towards some universal set of values. To the contrary, we believe that an agent's values derive mainly from its cognitive architecture and its interaction with its environment; different architectures and interactions could lead to different values. Therefore, we estimate W5B = 0.9 and W5G = 0.1.
 The probability of ASI catastrophe Table  1  summarizes the various parameter estimates in Sections 3.1-3.3. Using these estimates, recalling the assumption {P1, P2, P6} = 1, and following Equation  2 gives P = (0.1*0.6 + 0.9*0.5) * (0.3*0.9 + 0.7*0.4) * (0.9*0.95 + 0.1*0.5) ≈ 0.25. In other words, this set of parameter estimates implies an approximately 25% probability of ASI catastrophe. For comparison, giving equal weighting to Bostrom's and Goertzel's positions (i.e., setting each WB = WG = 0.5) yields P ≈ 0.26; using only Bostrom's arguments (i.e., setting each WB = 1) yields P ≈ 0.51; and using only Goertzel's arguments (i.e., setting each WG = 1) yields P = 0.1. Catastrophe probabilities of 0.1 and 0.51 may diverge by a factor of 5, but they are both still extremely high. Even "just" a 0.1 chance of major catastrophe could warrant extensive government regulation and/or other risk management. Thus, however much Bostrom and Goertzel may disagree with each other, they would seem to agree that ASI constitutes a major risk.
 PB However, an abundance of caveats is required. First, the assumption {P1, P2, P6} = 1 was made without any justification. Any thoughtful estimates of these parameters would almost certainly be lower. Our intuition is that ASI from AI takeoff is likely to be possible, and ASI deterrence seems unlikely to occur, suggesting {P1, P6} ≈ 1, but that the creation of seed AI is by no means guaranteed, suggesting P2 << 1. This implies P ≈ 0.25 is likely an overestimate. Second, the assumption that the correct position was either Bostrom's or Goertzel's was also made without any justification. They could both be wrong, or the correct position could be some amalgam of both of their positions, or an amalgam of both of their positions plus other position(s). Bostrom and Goertzel are both leading thinkers about ASI, but there is no reason to believe that their range of thought necessarily corresponds to the breadth of potential plausible thought. To the contrary, the ASI topic remains sufficiently unexplored that it is likely that many other plausible positions can be formed. Accounting for these other positions could send P to virtually any value in [0, 1]. Third, the estimates in Table  1  were made with little effort, largely for illustration and discussion purposes. Many of these estimates could be significantly off, even by several orders of magnitude. Given the form of Equation 1, a single very low value for Wn*Pn would also make P very low. This further implies that P ≈ 0.25 is likely an overestimate, potentially by several orders of magnitude. Fourth, the estimates in
 A practical application: AI confinement A core motivation for analyzing ASI risk is to inform practical decisions aimed at reducing the risk. Risk analysis can help identify which actions would reduce the risk and by how much. Different assessments of the risk-such as from experts' differing viewpoints-can yield different results in terms of which actions would best reduce the risk. Given the differences observed in the viewpoints of Bostrom and Goertzel about ASI risk, it is possible that different practical recommendations could follow. To illustrate this, we apply the above risk analysis to model the effects of decisions on a proposed ASI risk reduction measure known as AI confinement: AI confinement: The challenge of restricting an artificially intelligent entity to a confined environment from which it can't exchange information with the outside environment via legitimate or covert channels if such information exchange was not authorized by the confinement authority  (Yampolskiy 2012, p.196) . AI confinement is a type of containment and thus relates directly to the P3 (containment fails) variable in the ASI-PATH model (Figure  1 ). Stronger confinement makes it less likely that an AI takeoff would result in an ASI gaining decisive strategic advantage. Confinement might be achieved, for example, by disconnecting the AI from the internet and placing it in a Faraday cage. Superficially, strong confinement would seem to reduce ASI risk by reducing P3. However, strong confinement could increase ASI risk in other ways. In particular, by limiting interactions between the AI and the human populations, strong confinement could limit the AI's capability to learn human-desirable values, thereby increasing P4 (failure of human attempts to make ASI goals safe). For comparison, AIs currently learn to recognize key characteristics of images (e.g., faces) by examining large data sets of images, often guided by human trainers to help the AI correctly identify image features. Similarly, an AI may be able to learn humandesirable values by observing large data sets of human decision-making, human ethical reflection, or other phenomena, and may further improve via the guidance of human trainers. Strong confinement could limit the potential for the AI to learn human-desirable values, thus increasing P4. Bostrom and Goertzel have expressed divergent views on confinement. Bostrom has favored strong confinement, even proposing a single international ASI project in which "the scientists involved would have to be physically isolated and prevented from communicating with the rest of the world for the duration of the project, except through a single carefully vetted communication channel (Bostrom 2014, p. 253)". Goertzel has explicitly criticized this proposal (Goertzel 2015, p.71-73) and instead argued that an open project would be safer, writing that "The more the AGI system is engaged with human minds and other AGI systems in the course of its self-modification, presumably the less likely it is to veer off in an undesired and unpredictable direction" (Goertzel and Pitt 2012, p.13). Each expert would seem to be emphasizing different factors in ASI risk: P3 for Bostrom and P4 for Goertzel. The practical question here is how strong to make the confinement for an AI. Answering this question requires resolving the tradeoff between P3 and P4. This in turn requires knowing the size of P3 and P4 as a function of confinement strength. Estimating that function is beyond the scope of this paper. However, as an illustrative consideration, suppose that it is possible to have strong confinement while still giving the AI good access to human-desirable values. For example, perhaps a robust dataset of human decisions, ethical reflections, etc. could be included inside the confinement. In this case, the effect of strong confinement on P4 may be small. Meanwhile, if there is no arrangement that could shrink the effect of confinement on P3, such that this effect would be large, then perhaps strong confinement would be better. This and other practical ASI risk management questions could be pursued in future research.
 Conclusion Estimates of the risk of ASI catastrophe can depend heavily on which expert makes the estimate. A neutral observer should consider arguments and estimates from all available experts and any other sources of information. This paper analyzes ASI catastrophe risk using arguments from two experts, Nick Bostrom and Ben Goertzel. Applying their arguments to an ASI risk model, we calculate that their respective ASI risk estimates vary by a factor of five: P ≈ 0.51 for Bostrom and P = 0.1 for Goertzel. Our estimates, combining both experts' arguments, is P ≈ 0.25. Weighting both experts equally gave a similar result of P ≈ 0.26. These numbers come with many caveats and should be used mainly for illustration and discussion purposes. More carefully considered estimates could easily be much closer to either 0 or 1. These numbers are interesting, but they are not the only important part, or even the most important part, of this analysis. There is greater insight to be obtained from the details of the analysis than from the ensuing numbers. This is especially case for this analysis of ASI risk because the numbers are so tentative and the underlying analysis so comparatively rich. This paper is just an initial attempt to use expert judgment to quantify ASI risk. Future research can and should do the following: examine Bostrom's and Goertzel's arguments in greater detail so as to inform the risk model's parameters; consider arguments and ideas from a wider range of experts; conduct formal expert surveys to elicit expert judgments of risk model parameters; explore different weighting techniques for aggregating across expert judgment, as well as circumstances in which weighted aggregation is inappropriate; conduct sensitivity analysis across spaces of possible parameter values, especially in the context of the evaluation of ASI risk management decision options; and do all of this for a wider range of model parameters, including {P1, P2, P6} as well as more detailed components of {P3, P4, P5}, such as modeled in  Barrett and Baum (2017a; 2017b) . Future research can also explore the effect on overall ASI risk when multiple ASI systems are launched: perhaps some would be riskier than others, and it may be important to avoid catastrophe from all of them. One overarching message of this paper is that more detailed and rigorous analysis of ASI risk can be achieved when the risk is broken into constituent parts and modeled, such as in Figure  1 . Each component of ASI risk raises a whole host of interesting and important details that are worthy of scrutiny and debate. Likewise, aggregate risk estimates are better informed and generally more reliable when they are made from detailed models. To be sure, it is possible for models to be too detailed, burdening experts and analysts with excessive minutiae. However, given the simplicity of the risk models at this early stage of ASI risk analysis, we believe that, at this time, more detail is better. A final point is that the size of ASI risk depends on many case-specific factors that in turn depend on many human actions. This means that the interested human actor has a range of opportunities available for reducing the probability of ASI catastrophe. Risk modeling is an important step towards identifying which opportunities are most effective at reducing the risk. ASI catastrophe is by no means a foregone conclusion. The ultimate outcome may well be in our hands.
 Acknowledgement We thank Ben Goertzel, Miles Brundage, Kaj Sotala, Steve Omohundro, Allan Dafoe, Stuart Armstrong, Ryan Carey, Nell Watson, and Matthijs Maas for helpful comments on an earlier draft. Any remaining errors are the authors' alone. Work for this paper is funded by Future of Life Institute grant 2015-143911. The views in this paper are those of the authors and do not necessarily reflect the views of the Global Catastrophic Risk Institute or the Future of Life Institute. Figure 1 : 1 Figure 1: ASI catastrophe fault tree. Adapted from Barrett and Baum (2017a).
 Bostrom 2014, p.119) Such an AI would not have durable values in the sense that it would go from acting in human-desirable ways to acting in human-undesirable ways. A key detail of the treacherous turn theory is that the AI has values that are similar to, but ultimately different from, humandesirable values. As the AI gains intelligence, it goes through a series of stages: 1. At low levels of intelligence, the AI acts in ways that humans consider desirable. At this stage, the differences between the AI's values and human values are not important because the AI can only complete simple tasks that are human-desirable. 2. At an intermediate level of intelligence, the AI realizes that its values differ from human-desirable values and that it if it tried deviating from humandesirable values, humans would reprogram the AI or shut it down. Furthermore, the AI discovers that it can successfully pretend to have human-desirable values until it is more intelligent. 3. At a high level of intelligence, the AI takes control of the world from humanity so that humans cannot reprogram it or shut it down, and then pursues its actual, human-undesirable values.
 Table 1 : 1 Summary of parameter estimates in Sections 3.1-3.3. PG WB WG 3 0.6 0.5 0.1 0.9 4 0.9 0.4 0.3 0.7 5 0.95 0.5 0.9 0.1
 Table 1 depend on a range of case-specific factors, including what other containment measures are used, how much effort humans put into giving the AI human-desirable values, and what cognitive architecture the AI has. Therefore, different seed AIs self-improving under different conditions would yield different values of P, potentially including much larger and much smaller values.
			 On expert opinion of ASI, see Baum et al. (2011), Armstrong and Sotala (2012), Armstrong et al. (2014), and Müller and Bostrom (2014).