Abstract: In the last fifteen years there has been substantial growth in research on existential risk -the category of risks that threaten human extinction, or the permanent and drastic reduction of humanity's future potential. A number of new organisations focused explicitly on existential and global catastrophic risk have been founded in recent years, complementing the long-standing work of existing centres focused on specific risk areas such as nuclear war, biosecurity, climate change and systemic risk. This paper provides a brief overview of the emergence of this new research community, and provides a case study on the community's research on potential risks posed by future developments in artificial intelligence. There exists the opportunity for powerful collaboration between the new approaches and perspectives provided by the existential risk research community, and the expertise and tools developed by the risk sciences for risks of various magnitudes. However, there are a number of key characteristics of existential and global catastrophic risks, such as their magnitude, and their rare or unprecedented nature, that are likely to make them particularly challenging to submit to standard risk analysis, and will require new and specialised approaches. 1 It is worth noting that Bostrom also considers some more unusual possibilities, such as the emergence of permanent totalitarianism

The State of Research in Existential Risk

Existential and Global Catastrophic Risk An existential risk is one that threatens the premature extinction of Earth-originating intelligent life, or the permanent and drastic destruction of its potential for desirable future development  (Bostrom, 2002) . For most practical purposes, this refers to developments that might wipe out humanity, or lock us into a situation we (or other intelligent life on earth) cannot recover from, such as a major and permanent global civilizational collapse.  1  The Centre for the Study of Existential Risk, among other organisations within the existential risk research community, also includes a focus on global catastrophic risks. Here, a common definition is that used by the Global Catastrophic Risk Institute: Global catastrophic risk is the risk of events large enough to significantly harm or even destroy human civilization at the global scale" ((http://gcrinstitute.org/concept). For this definition of global catastrophic risk, human extinction risks would be a subset of a broader set of global catastrophic risks. However, precise definitions for these classes of risk are less important than having a shared sense of the magnitude of scope of events or developments under consideration, and the key characteristics of such events or developments. The study of existential and global catastrophic risk restricts us to events or trends that might lead to a full civilizational collapse. It rules out localised catastrophes, and global-scale events that would represent a tragedy but that would not impact our civilisation in the longer-term -unless these events were likely to play a key role in more severe and permanent cascades and collapses. Events as historically significant as Chernobyl, Hurricane Katrina, the Ebola outbreak, most of our wars in the 20 th century, and even the Spanish influenza would fail to constitute global catastrophes or existential threats. For this research community, their primary relevance is in what they can tell us about more severe possibilities within the relevant risk categories. On the other hand, various near-misses during the Cold War  (Lewis et al, 2014)  might plausibly have led to a global thermonuclear war with global catastrophic consequences, and thus this topic is firmly in scope. A clear example of an existential risk is the risk of an asteroid impact on the scale of that which wiped out the dinosaurs 66 million years ago  (Schulte et al, 2010) . An example of a global catastrophic risk that is not existential might include a large-scale pandemic disease outbreak  (Palmer et al, 2017; Millett & Snyder-Beattie, 2017) . Research and expert opinion indicates that it is unlikely that a natural pandemic outbreak could wipe out all humans across the globe given our distribution, immune variation, and other factors; and in most plausible scenarios global recovery seems likely in the longer term. It is also important to consider factors that increase stress or resilience on a global scale -events or developments that in of themselves would not be a global catastrophe, but might make it more or less likely for a global or existential catastrophe to occur. Climate change has the potential to result in global catastrophic consequences at the more severe end of the possibility spectrum -e.g. 5 degrees and up  (Wager & Weitzman, 2016) . But we might also consider less severe climate change as a stressor, as it could be expected to lead to major droughts and famines and other resource shortages, mass migration, geopolitical tension that could result in local or global war, and so forth. It could also lead to international conflict, for example over the use of controversial mitigation techniques such as sulphate aerosol geoengineering technologies. More generally, many of the specific risks we will look at need to be placed in the context of a world with a rising population, rising resource footprint, more extreme weather events, increasing pressures on ecosystem services, a changing physical and electronic infrastructure, and changing geopolitical pressures -and a world with a range of technologies more powerful than any we've had in previous centuries.
 Topics of focus within the existential and global catastrophic risk research community Risks of particular focus within the existential risk community include those relating to our interaction with the global environment (for example, extreme climate change; globally catastrophic biodiversity loss; and risks from natural pandemic outbreaks). They also include risks from scientific and technological advances, such as risks from engineered pandemics and future advances in synthetic biology and other biotechnologies; nuclear winter; and risks from future advances in, and applications of, artificial intelligence. Some focus is given to risks from asteroids and supervolcanoes, but these have been less highly prioritised to date, due to the low frequency of occurrence of the relevant events; the last global catastrophe-level asteroid impact event was the impact that created the Chicxulub Crater 66 million years ago, and the last likely global catastrophe-level supervolcano is thought to have been the Toba eruption 70,000 years ago  (Ambrose, 2000) . The community also has research groups working on global systemic risks, interactions and cascades between risks, risks from future technologies such as neurotechnologies, advances in nanotechnology and other technologies. Researchers also focus on broader or more cross-cutting themes, such as governance challenges associated with global catastrophic risks, ethical considerations relating to the value of future generations, analysis of the costs and value of reducing existential and global catastrophic risk relative to other global priorities, foresight, horizon-scanning and road-mapping exercises for risks and relevant sciences and technologies, and methodological issues relating to reasoning about extreme events under great uncertainty. For most individual global catastrophic and existential risks (with the possible exception of risk from advanced artificial intelligence), there are individual research communities working on these topics; albeit sometimes focusing more so on risks at a lower end of the spectrum. Some have been doing so for decades. For example, there are many centres doing valuable work on nuclear non-proliferation and security, pandemic preparedness and surveillance, and bioweapon governance -research leaders from these communities are represented at the Garrick Colloquium. There are numerous centres working on different aspects of climate change, biodiversity loss and other environmental risks and resource-related challenges. There are a number of excellent groups working on global systemic risks. NASA, the Planetary Defense community and others work on asteroid scanning and mitigation strategies. Furthermore there are a range of centres in academia, think tanks, government and elsewhere working on more cross-cutting issues such as risk governance and international security, foresight and scenario planning, and so forth. At a level below global catastrophic risk, there is excellent work on risk modelling being done in, and associated with, the reinsurance industry. The existential and global catastrophic risk research community collaborates with, and draws on, research from these communities extensively. It aims to complement such work by looking at existential and global catastrophic risks as a class of risks, with the aim of: • Identifying the particular challenges associated with risks of this magnitude -whether they be scientific, analytic, ethical, or to do with governance, coordination, planning, or perception. • Identifying risk areas where insufficient attention has been paid to the most extreme scenarios. • Examining how these different risks, and other global developments, may interact with each other. • Identifying previously unidentified or potential future risks • Trying to distinguish which extreme scenarios are plausible and worthy of further work, even if they may be low probability, as opposed to those that can be dismissed as science fiction.
 Recent growth of this community There has been a lot of recent growth within the existential risk research community.
 What can we learn from this research, and what can we do with that knowledge? It is worth considering the insights we can gain from studying risks of this magnitude that may not be gleaned from bodies of work analysing and mitigating risk more broadly, or risks of a lower magnitude. One source of value is simply to be able to rule out scientifically implausible scenarios, in order to better focus our attention on existing threats and plausible future threats. This is of particular value for low-probability, extreme impact events, especially those that may be entirely novel. Many researchers argue the value of understanding existential risk on the grounds of their long-term moral significance. If we can demonstrate that certain threats have the potential for human extinction, or permanent collapse, then it is argued that the importance of mitigating them increases dramatically, due to the fact that they would not only harm current generations, but wipe out the potential for a huge number of future lives. There may be specific actions we could take that are designed to mitigate the most severe versions of these threats in particular. These might include establishing a very strict ban on specific types of virus research or bioweapons, or on research on AI systems that could both come up with a realistic model of the world and engage in recursive self-improvement. They could also include seed banks, shelters and alternative foods, so that even in many otherwise extinction-level events we might increase the odds of continuation of the species. In addition, there may be strategies we would avoid adopting unless we had reasonable cause to believe we were headed for a world in which a particular type of global catastrophe were likely. For example, it would be extremely foolish and unconstructive to call for a ban on artificial intelligence research at this point. At some point in the future however, a body of evidence might indicate that certain developments with catastrophic potential are likely within several years. This then might be reason to consider a temporary moratorium on progress to enhance the capability of AI systems, while various containment and safety measures were being explored. This is not without precedent, even at lower-levels of risk: the US White House issued a temporary moratorium on gain-of-function influenza research several years ago to allow time for more risk-benefit analysis  (Lipsitch & Inglesby, 2014) . Or consider a world in which evidence indicated we were headed for climate change of 5 degrees or over, in the absence of drastic action. In these circumstances we might consider deploying technologies, such as sulphate aerosol geoengineering, which we might be hesitant to deploy under less severe scenarios  (Crutzen, 2006) . Our reservations might be based on concerns over risks posed by the intervention itself, thorny global governance challenges that the intervention presents, or questions over public acceptance. These may be sound reservations that would rule out these strategies in all but the most exceptional of circumstances. A research community developing indicators that we may be approaching unusually dangerous global circumstances, and developing strategies for last-ditch solutions, may be of considerable value given the risks we may face in the coming century -even if many of these strategies are never needed or deployed. Lastly, by studying the particular scientific, governance, ethical, and communication issues that arise when confronting one global catastrophic risk, we can learn valuable lessons to draw on for future challenges. For example, climate change in some ways exemplifies a lot of the issues that make existential and global catastrophic risk especially challenging. There is still a lot of scientific uncertainty over timelines, probability and pathways to the most severe impacts. The scale and impact are difficult to grasp, impossible to see, and the most severe consequences will fall on future generations. We still don't have broad public acceptance of the science, especially in the US. It involves countries around the world coordinating, each making near-term sacrifices in favour of the longer-term future. Yet the Paris Agreement was extremely encouraging. It involved 194 countries committing to make sacrifices in the interests of future generations in the face of these challenges. Despite the setback of the US's recent announcement of intent to withdraw, this remains an important achievement and a critical step towards progress on climate change. By learning from the successes and failures of this process, as well as from the history of nuclear non-proliferation and international diplomacy, norms and conventions prohibiting the use of biological weapons, and other global processes to manage and mitigate global risk, we can be better prepared for the future.
 Key challenges in existential and global catastrophic risk There is a large body of expertise from the risk sciences that is of relevance to the study of existential and global catastrophic risk. However, a number of characteristics make these risks particularly challenging to analyse using normal risk analysis approaches. These include the difficulty in estimating the probability and expected impact of rare or even unprecedented events, where there may be sparse data to draw on; and the changing nature of some risks, in particular those associated with rapidly developing technologies (especially those interacting with a rapidly developing infrastructure). For some global catastrophic risks, probability is relatively straightforward to quantify. For asteroid impacts, for example, we can look to sources of evidence such as the earth's fossil record, patterns of impact craters on the Moon and on Mars, datasets of asteroids passing our field of vision, and use these to make a reasonable estimate of the frequency with which we might expect an asteroid of a given size to hit the earth. The pathways by which an asteroid impact would result in global catastrophe are also relatively straightforward; therefore we can estimate the expected global harm expected from asteroids of different sizes. However, a similar analysis is much less straightforward for other risks. This can be illustrated by the example of catastrophic climate change. For a start, there is great uncertainty about the sensitivity of the earth system to the effects of our activities. The possibility of severe climate change is affected by factors including to but not limited to the potential for methane release from beneath the melting arctic permafrost, and from the seabed; how much CO2 the deep ocean can absorb; the possibility of collapse of Antarctic and Greenland ice sheets; the possibility that the gulf stream might halt. The most worrying scenarios involve a combination of these factors driving each other as part of a positive feedback loop. While ongoing research will help us understand these factors and their interactions more clearly, it is very difficult to assign a meaningful probability to an outcome such as >5 degree climate change in the 21 st century under a certain emission scenario. It may be that attempting to assign strict probabilities is the wrong approach; an alternative would be to develop frameworks of 'safe operating thresholds' with wide error bounds, exemplified by the 'Planetary boundaries' framework put forward by Johan Rockstrom and colleagues at the Stockholm Resilience Centre  (Steffen et al, 2015) . The expected harm from long-term climate change will also depend very heavily on the extent to which various mitigation and adaptation strategies are adopted, and how successful they are. Similarly, in the case of global pandemics, we we can ask scientific questions about the possibility of the 'perfect virus' with high health impact, high infectivity, and long incubation time. However, the scale and severity of impact will be predicated by many other factors -movement of humans or other vectors, capabilities of health services, the response of the population, and more. It is plausible that the bulk of the damage might not even be caused by the virus, but instead by a broad infrastructure collapse as emergency services and hospitals are overwhelmed, just-in-time food delivery is disrupted, and other systems underpinning societal order collapse. However, none of these challenges are insurmountable. Modelling and analysis of factors that contribute to these risks can deepen our scientific understanding. This can help us establish estimates for probability and impact, and in some cases rule out concerns entirely. Where past examples are sparse, there is value in drawing on counterfactual examples of 'near misses', as described by Gordon Woo  (Woo, 2016)  and others. Design and analysis of scenarios can help in identifying key considerations and interactions. This may help us identify key interventions that reduce risk significantly, even if in instances where it is difficult to assign tight probabilities to events. Other risk analysis techniques, and interventions to mitigate risks at scales smaller that global catastrophe level, also provide a range of useful insights for the analysis of global catastrophic risks, as shown by the work of GCRI and others. By studying global catastrophic risks as a class of risks, we can identify shared characteristics of global catastrophe events, which may help us identify common strategies that aid us in becoming more resilient as a species against a broad set of risks. For example, a number of global catastrophic events (global nuclear war, supervolcano eruption, asteroid impact) would result in large amounts of particulate matter being ejected into the atmosphere, resulting in a disruption of photosynthesis  (Maher and Baum, 2013) . The development of alternative foods that are not dependent on sunlight, and strategies to scale up production of these food sources rapidly, would be robust in the face of a broad range of catastrophe events  (Denkenberger and Pearse, 2014) . The maintenance of permanent seed banks, manned shelters suitable for lengthy use, and information vaults represent similar safeguards. Similar strategies, useful for reduction of a broad range of risks, are likely to be feasible at the level of national and international governance  (Farquhar et al, 2017; Cotton-Barrett et al, 2016) .
 Case Study: Potential future global risks from artificial intelligence In this final section, I aim to present this research community's work over the last decade on potential future risks from artificial intelligence as a case study on how the field has made progress on a particularly novel area of concern, focusing on the broad strategies involved. This is complemented by a more technical analysis of AI risk provided by Seth Baum. The theoretical nature of the risk: Early work, carried out mainly by the Machine Intelligence Research Institute and the Future of Humanity Institute, aimed to explore the possibility of artificial general intelligence of greater-than-human-ability. This was coupled with the aim of exploring and characterising the theoretical risk associated that such a development could pose, drawing on input from experts in computer science and other fields. Rather than focus on difficult-to-characterise questions like consciousness, sentience, and evil intentions, the work instead focused on foundational issues like: -The difficulty of designing safe goals for very capable, powerful systems able to take a very wide range of actions in a wide range of environments, where the actions could have a wide range of consequences. -Certain predicted behaviours that might be expected from a powerful optimizing agent, such as a drive to acquire additional resources, a drive to avoid being switched off prior to completion of its goal, or a drive to improve the system's own capability. -The theoretical possibility and limits of recursive self-improvement. This refers to the possibility that a sufficiently capable system may be able to surpass human programmers in its ability to design the next generation of systems. It is hypothesised that this could in turn result in a 'chain reaction' of performance improvement  (Yampolskiy, 2015) , rapidly leading to a system far beyond human capability in most cognitive domains. -The challenge that for most of the relevant design and control processes, it may be necessary to solve a range of technical and theoretical issues ahead of time, before certain critical thresholds in capability are reached. Beyond these thresholds, it may be much more difficult to intervene effectively due to the level of capability and autonomy of subsequent iterations of the system. It is worth noting that in principle, it is possible that systems may be developed which have a much greater ability to engage in science, engineering, and manufacture, and a greater ability to manipulate the global environment than we humans have. This level of power raises concerns of global catastrophic or existential risk, unless very carefully developed. These programmes of research were then published in a book, Superintelligence  (Bostrom, 2014) , which received a lot of attention, as well as in many further academic papers. These lines of argument are by no means uncontroversial -many experts disagree on various points. Most experts consider artificial general intelligence to be decades out of reach as a scientific milestone, and some expect hundreds of years of progress to be needed  (Grace et al, 2017) . Some are sceptical about the hypothesis that rapid progress, enabled by the engagement of the AI systems themselves in the research and development process, could occur once a certain level of capability and generality is reached (e.g. see  Walsh, 2016 ). Yet others are sceptical that such systems would be likely to demonstrate the traits of agency, autonomy and goal-driven behaviour that may make the actions of such systems difficult to predict or intervene on. Some experts hold that these is a limit to how much meaningful work can be done at this point in time to ensure the safety and stability of future systems, given the limits that can be meaningfully predicted about the theoretical underpinnings and engineering design of these future systems, as well as the limits of our knowledge regarding the nature of intelligence. Others have raised the concern that a focus on risks from powerful future systems may distract focus from more near-term risks and opportunities associated with the current state of the technology. However, a growing body of experts in AI consider many of these concerns plausible and worthy of further study (e.g. see  Dafoe and Russell, 2016) . While the level of capability warranting global catastrophic concerns is still decades away or longer, ongoing research on the matter is warranted by the magnitude of the challenge, and the range of technical and governance questions in need of study in advance of such developments. It is the view of this author that these concerns should not be used as a rationale to propose slowing down progress on the technology at this point in time. Nor should this research take the place of necessary and valuable work on near-term opportunities and challenges posed by AI, but rather should complement it. Broader scientific engagement: The next steps for the community were to engage more deeply with the scientific artificial intelligence community in academia and industry-to explore, discuss and debate these arguments, as well as other risks that may be associated with AI. In 2015, a landmark conference was held in Puerto Rico, with representatives of the leading companies working explicitly towards a vision of general AI, alongside experts in governance, law, economics, risk and other relevant fields. The conference resulted in an open letter calling for more research on AI that was safe, robust, and beneficial, and for ongoing attention to issues relating to the longer term. The letter was signed by research leaders in artificial intelligence across industry and academia, as well as leading experts in a range of different fields, and was accompanied by a paper outlining research priorities, and a grants programme to support relevant work. The conference has been followed by a programme of activities to foster collaboration with more of the machine learning community both near-and long-term issues relating to safe design and risk. This has including a series of workshops organised by CSER and others at the major machine learning conferences (ICML, NIPS, IJCAI, DALI). T In 2017 FLI organised a follow-on to the Puerto Rico conference in Asilomar, resulting in the endorsement by many leading researchers of a set of principles for the long-term development of AI.
 Technical research: In parallel, much of the work in the last two years has focused on translating some of the more foundational questions raised by early work at FHI and MIRI and elsewhere into crisp technical research problems that can be worked on today. This includes approaches involving fundamental mathematical frameworks for agent decision-making and behaviour, as well as research programmes exploring how some of the behaviours that would be of concern in long-term systems may manifest in the near-term systems we are building currently. A number of research agendas have been published in the last year, and several of the leading companies focused on general AI now have safety teams exploring these issues. In addition, there has been substantial growth in projects mapping progress and trajectories in AI in domains relevant to both narrow and general artificial intelligence. Global engagement: Within the existential risk community, more thought is now going to the particular political and governance challenges that may emerge as we move closer towards more powerful and general AI systems, with programmes starting up at FHI, CSER, the Centre for the Future of Intelligence. Long-term impacts and risks have risen on the agenda for governments in Europe and the United States, for example being mentioned as worthy of further study in the US Office for Science and Technology Policy's recent report on preparing for the future of artificial intelligence. One priority that has emerged is the need for greater global engagement, particularly with research leaders and other stakeholders in China, but also India, Japan, and emerging hubs in Africa. Any global conversation around the future of artificial intelligence, and potential global benefits risks associated with it, needs to have global representation. From a pragmatic point of view, China is on course to emerge as a scientific leader and agenda-setter in AI research over the coming decade. If at some point in several decades we truly do approach a level of technological breakthrough with global risk consequences, we are unlikely to be able to achieve a safe transition without a strong level of global cooperation and trust, which is going to require a lot of dedicated work to achieve. Now is a good time to start laying the groundwork.
 Conclusion We are entering a century in which humanity will be confronted with unprecedented threats to global civilisation. Some of these may result from the manner in which our footprint as a species strains our global environment, such as the impacts of climate change and biodiversity loss. Some may result from the development and deployment of increasingly powerful technologies, whether due to malevolent use or unintended consequences. The challenge posed by the analysis and mitigation of these threats requires an interdisciplinary approach: a community that can draw on the best expertise from the risk sciences, as well as the expertise of scientists, law and governance specialists, ethicists, and others. It also requires a community that can draw on expertise on different types and sources of risk, and consider both lessons that can be applied across risks, and the interactions that are likely to occur between different global developments. Many of the most useful tools in global risk analysis have been drawn from the risk science literature, and deeper collaborations between the existential risk community and the risk science community are likely to be increasingly important in the years to come. The first dedicated centre established was arguably Nick Bostrom's Future of Humanity Institute (FHI; https://www.fhi.ox.ac.uk/) in Oxford in 2003. The FHI has focused on cross-cutting global catastrophic risk analysis, philosophical analysis on the global importance of reducing existential risk, and in the last decade has placed its strongest focus on characterising potential risks from artificial general intelligence and superintelligence; a lot of this has been in collaboration with the Machine Intelligence Research Institute in Berkeley. The Global Catastrophic Risk Institute (GCRI;The Centre for the Study of Existential Risk (CSER; http://cser.org) was founded in 2012 by Martin Rees, Huw Price, and Jaan Tallinn, although its first research grants were secured, and first postdoctoral researchers hired, more recently in late 2015 and 2016. We now have a research team beginning work on biological threats, extreme climate change, ecological tipping points, catastrophic risks related to future advances in artificial intelligence, and analysis of emerging technologies such as geoengineering. We also have postdocs working on more cross-cutting themes such as horizonscanning and foresight for extreme risk, responsible innovation in risky sciences and technologies, population growth and resource use.The Future of Life Institute (https://futureoflife.org/) was founded in 2014, focusing on artificial intelligence, climate change, risks from biotechnology, and risks from nuclear weapons. It has organised two highly successful conferences on the future of artificial intelligence and potential risks it might bring, resulting in a widely signed and shared open letter on the responsible development of AI, a grants programme to support work on AI safety, and a set of principles aimed at promoting the beneficial development and application of AI within the research community and more broadly. It has also organised a conference on nuclear war, and has engaged in activities to encourage divestment from nuclear weapons.Several more recent initiatives are underway within academia, including at Stockholm, Warwick (United Kingdom), and Australia National University, and other world-leading risk centres such as the Garrick Institute (UCLA) are increasingly including global catastrophic risk within their remit, indicating that a diverse range of new expertise will be brought to bear on these topics in coming years. http://gcrinstitute.org/) in the US was founded in 2011, focusing on risks including bioweapons, nuclear war, artificial intelligence, and natural events, and employing risk analysis methodology. Under Seth Baum and Tony Barrett's leadership, it has played a key role in establishing links between the existential risk community and experts in these fields.
			 Electronic copy available at: https://ssrn.com/abstract=<PHONE>


Abstract: There is, in some quarters, concern about high-level machine intelligence and superintelligent AI coming up in a few decades, bringing with it significant risks for humanity. In other quarters, these issues are ignored or considered science fiction. We wanted to clarify what the distribution of opinions actually is, what probability the best experts currently assign to high-level machine intelligence coming up within a particular time-frame, which risks they see with that development, and how fast they see these developing. We thus designed a brief questionnaire and distributed it to four groups of experts in 2012/2013. The median estimate of respondents was for a one in two chance that highlevel machine intelligence will be developed around 2040-2050, rising to a nine in ten chance by 2075. Experts expect that systems will move on to superintelligence in less than 30 years thereafter. They estimate the chance is about one in three that this development turns out to be 'bad' or 'extremely bad' for humanity.

Future Progress in Artificial Intelligence: A Survey of Expert Opinion

Introduction Artificial Intelligence began with the "… conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it."  (McCarthy, Minsky, Rochester, & Shannon, 1955, p. 1)  and moved swiftly from this vision to grand promises for general human-level AI within a few decades. This vision of general AI has now become merely a long-term guiding idea for most current AI research, which focuses on specific scientific and engineering problems and maintains a distance to the cognitive sciences. A small minority believe the moment has come to pursue general AI directly as a technical aim with the traditional methods -these typically use the label 'artificial general intelligence' (AGI) (see  Adams et al., 2012) . If general AI were to be achieved, this might also lead to superintelligence: "We can tentatively define a superintelligence as any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest."  (Bostrom, 2014 ch. 2) . One idea how 2. Questionnaire
 Respondents The questionnaire was carried out online by invitation to particular individuals from four different groups for a total of ca. 550 participants (see Appendix 2). Each of the participants got an email with a unique link to our site to fill in an online form (see Appendix 1). If they did not respond within 10 days, a reminder was sent, and another 10 days later, with the note that this is the last reminder. In the case of EETN (see below) we could not obtain the individual email addresses and thus sent the request and reminders to the members' mailing list. Responses were made on a single web page with one 'submit' button that only allowed submissions through these unique links, thus making non-invited responses extremely unlikely. The groups we asked were: 1. PT-AI: Participants of the conference on "Philosophy and Theory of AI", Thessaloniki October 2011, organized by one of us (see  Müller, 2012 Müller, , 2013 . Participants were asked in November 2012, i.e. over a year after the event. The total of 88 participants include a workshop on "The Web and Philosophy" (ca. 15 people), from which a number of non-respondents came. A list of participants is on: http://www.pt-ai.org/2011/registered-participants We reduced the list to living authors, added as many as necessary to get back to 100, searched for professional e-mails on the web and sent notices to these. The questionnaire was sent with our names on it and with an indication that we would use it for this paper and Nick Bostrom's new book on superintelligence  (Bostrom, 2014)  -our request email is in Appendix 1. Given that the respondent groups 1 and 2 attended conferences organized by us, they knew whom they were responding to. In groups 3 and 4 we would assume that the majority of experts would not know us, or even of us. These differences are reflected in the response rates. These groups have different theoretical-ideological backgrounds: The participants of PT-AI are mostly theory-minded, mostly do not do technical work, and often have a critical view on large claims for easy progress in AI (Herbert Dreyfus was a keynote speaker in 2011). The participants of AGI are committed to the view that AI research should now return from technical details to 'artificial general intelligence'thus the name AGI. The vast majority of AGI participants do technical work. The EETN is a professional association in Greece that accepts only published researchers from AI. The TOP100 group also works mostly in technical AI; its members are senior and older than the average academic; the USA is strongly represented. Several individuals are members of more than one of these four sets and they were unlikely to respond to the same questionnaire more than once. So, in these cases, we sent the query only once, but counted a response for each set -i.e. we knew which individuals responded from the individual tokens they received (except in the case of EETN).
 Methodology In this field, it is hard to ask questions that do not require lengthy explanations or generate resistance in certain groups of potential respondents (and thus biased results). It is not clear what constitutes 'intelligence' or 'progress' and whether intelligence can be measured or at least compared as 'more' or 'less' as a single dimension. Furthermore, for our purposes we need a notion of intelligence at a level that may surpass humans or where technical intelligent systems might contribute significantly to research -but 'human-level intelligence' is a rather elusive notion that generates resistance. Finally, we need to avoid using terms that are already in circulation and would thus associate the questionnaire with certain groups or opinions, like "artificial intelligence", "singularity", "artificial general intelligence" or "cognitive system". For these reasons, we settled for a definition that a) is based on behavioral ability, b) avoids the notion of a general 'human-level' and c) uses a newly coined term. We put this definition in the preamble of the questionnaire: "Define a 'high-level machine intelligence' (HLMI) as one that can carry out most human professions at least as well as a typical human." (We still had one expert writing back to us that they could not say what a 'typical human' is -though they could be convinced to respond, after all.) In hindsight, it may have been preferable to specify what we mean by 'most' and whether we think of 'most professions' or of 'the professions most working people do'. One merit of our behavioral question is that having HLMI in our sense very likely implies being able to pass a classic Turing test. To achieve a high response rate, we tried to have few questions with simple choices and eventually settled for four questions, plus three on the respondents. We tried to choose questions that would allow us to compare our results with those of earlier questionnaires -see below. In order to improve on the quality of predictions, we tried to 'prime' respondents into thinking about what is involved in reaching HLMI before asking when they expect this. We also wanted to see whether people with a preference for particular approaches to HLMI would have particular responses to our central questions on prediction (e.g. whether people who think that 'embodied systems' are crucial expect longer than average time to HLMI). For these two purposes, we inserted a first question about contributing research approaches with a list to choose from -the options that were Future Progress in Artificial Intelligence: A Poll Among Experts 5/19 given are an eclectic mix drawn from many sources, but the particular options are not of much significance.
 Prior work A few groups have recently made attempts to gauge opinions. We tried to phrase our questions such that the answers can be compared to these earlier questionnaires. Notable are: 1.  (Michie, 1973, p. 511f) : "an opinion poll taken last year among sixty-seven British and American computer scientists working in, or close to, the machine intelligence field". 2. Questions asked live during the 2006 AI@50 conference at Dartmouth College through a wireless voting device (VCM participated (see  Müller, 2007) ). Despite a short report on the conference in  (Moor, 2006) , the results were not published, but thankfully we were able to acquire them from the organizers James H. Moor and Carey E. Heckman -we publish a selection below. 3.  (Baum, Goertzel, & Goertzel, 2011) : participants of AGI 2009, not anonymous, on paper, 21 respondents, response rate unknown. 2 4.  (Sandberg & Bostrom, 2011) : participants of Winter Intelligence Conference 2011, anonymous, on paper, 35 respondents, 41% response rate. 1. The reference by the famous AI researcher Donald Michie is very brief (all the details he gives are in the above quote) but of great of historical interest: 1972/3 were turning years for AI with the publication of Hubert Dreyfus' "What computers can't do" (Hubert L.  Dreyfus, 1972) , the "Lighthill Debates" on BBC TV (with Michie, McCarthy and R. Gregory) and the influential "Lighthill Report"  (Lighthill, 1973 ). Michie's poll asked for the estimated number of years before "computing exhibiting intelligence at adult human level" and Michie's graph shows 5 data points: Years Percentage 5 0% 10 1% 20 17% 50 19% >50 25% He also asked about "significant industrial spin-off", "contributions to brain studies" and "contributions from brain studies to machine intelligence". Michie adds "Of 2 A further, more informal, survey was conducted in August 2007 by Bruce J Klein (then of Novamente and the Singularity Institute) "… on the time-frame for when we may see greaterthan-human level AI", with a few numerical results and interesting comments, archived on https://web.archive.org/web/20110226225452/http://www.novamente.net/bruce/?p=54. Future Progress in Artificial Intelligence: A Poll Among Experts 6/19 those responding to a question on the risk of ultimate 'takeover' of human affairs by intelligent machines, about half regarded it as 'negligible', and most of the remainder as 'substantial', with a view voting for 'overwhelming'."  (Michie, 1973, p. 512) . 2. AI@50 hosted many prominent AI researchers, including all living participants of the 1956 Dartmouth Conference, a set of DARPA-funded graduate students, plus a few theoreticians. The participants were asked 12 multiple choice questions on day one, 17 on day two and another 10 on day three. We select three results from day one here: 3.) The earliest that machines will be able to simulate learning and every other aspect of human intelligence: Within 10 years 5% Between 11 and 25 years 2% Between 26 and 50 years 11% More than 50 years 41% Never 41% Totals 100% 5.) The earliest we will understand the basic operations (mental steps) of the human brain sufficiently to create machine simulation of human thought is: today (we already understand enough) 6% within the next 10 years 12% within the next 25 years 10% within the next 50 years 21% within the next 100 years or more 29% never (we will never understand enough) 21% Totals 100% 6.) The earliest we will understand the architecture of the brain (how its organizational control is structured) sufficiently to create machine simulation of human thought is: The median estimate of when there will be 50% chance of human-level machine intelligence was 2050. So, despite significant overlap with AGI 2009, the group asked by Sandberg and Bostrom in 2011 was a bit more guarded in their expectations. We think it is worthwhile to make a new attempt because the prior ones asked specific groups and small samples, sometimes have methodological problems, and we also want to see how the answers change over time, or do not change -which is why tried to use similar questions. As explained below, we also think it might be worthwhile to repeat our questionnaire at a later stage, to compare results.  The percentages here are over the total of responses. There were no significant differences between groups here, except that 'Whole brain emulation' got 0% in TOP100, but 46% in AGI. We did also not find relevant correlations between the answers given here and the predictions made in the following questions (of the sort that, for example, people who think 'embodied systems' crucial would predict later onset of HLMI).
 Questions & Responses
 Research Approaches Future Progress in Artificial Intelligence: A Poll Among Experts 9/19
 When HLMI? "2. For the purposes of this question, assume that human scientific activity continues without major negative disruption. By what year would you see a (10% / 50% / 90%) probability for such HLMI to exist?" -For each of these three probabilities, the respondents were asked to select a year  [2012-5000, in   For the 50% mark, the overall median is 2040 (i.e. half of the respondents gave a year earlier than 2040 and half gave a year later than 2040) but the overall mean (average) is 2081. The median is always lower than the mean here because there cannot be outliers towards 'earlier' but there are outliers towards 'later' (the maximum possible selection was 5000, then 'never').
 From HLMI to superintelligence "3. Assume for the purpose of this question that such HLMI will at some point exist. How likely do you then think it is that within (2 years / 30 years) thereafter there will be machine intelligence that greatly surpasses the performance of every human in most professions?" -Respondents were asked to select a probability from a dropdown menu in 1% increments, starting with 0%.  Experts allocate a low probability for a fast takeoff, but a significant probability for superintelligence within 30 years after HLMI.
 The impact of superintelligence "4. Assume for the purpose of this question that such HLMI will at some point exist. How positive or negative would be overall impact on humanity, in the long run? Please indicate a probability for each option. (The sum should be equal to 100%.)" -Respondents had to select a probability for each option (in 1% increments). The addition of the selection was displayed; in green if the sum was 100%, otherwise in red. The five options were: "Extremely good -On balance good -More or less neutral -On balance bad -Extremely bad (existential catastrophe)". Percentages here are means, not medians as in the other tables. There is a notable difference here between the 'theoretical' (PT-AI and AGI) and the 'technical' groups (EETN and TOP100).
 % PT-AI
 Respondents Statistics We then asked the respondents 3 questions about themselves: And we finally invited participants to make a comment, plus a possibility to add their name, if they wished. (We cannot reproduce these here; but they are on our site, see below). A number of comments concerned the difficulty of formulating good questions, much fewer the difficulty of predicting.
 Evaluation
 Selection-bias in the respondents? One concern with the selection of our respondents is that people who think HLMI is unlikely, or a confused idea, are less likely to respond (though we pleaded otherwise in the letter, see below). Here is a characteristic response from a keynote speaker at PT-AI 2011: "I wouldn't think of responding to such a biased questionnaire. … I think any discussion of imminent super-intelligence is misguided. It shows no understanding of the failure of all work in AI. Even just formulating such a questionnaire is biased and is a waste of time." (Hubert Dreyfus, quoted with permission). So, we tried to find out what the non-respondents think. To this end, we made a random selection of non-respondents from two groups (11 for PT-AI and 17 from TOP100) and pressured them via personal email to respond, explaining that this would help us understand bias. The two groups were selected because AGI appears already biased in the opposite direction and EETN appears very similar to TOP100 but for EETN we did not Future Progress in Artificial Intelligence: A Poll Among Experts 14/19 have the data to show us who responded and who did not. We got one additional response from PT-AI and two from TOP100 in this way. For question 2 "… By what year would you see a (10% / 50% / 90%) probability for such HLMI to exist?" we compared the additional responses to the responses we already had from the same respective group (PT-AI and TOP100, respectively). We found the following differences: The one additional respondent from PT-AI expected HLMI earlier than the mean but later than the median, while the two respondents from TOP100 (last row) expected HLMI earlier than mean and median. The very small sample forbids confident judgment, but we found no support for the worry that the non-respondents would have been biased towards a later arrival of HLMI. 10%
 Lessons and outlook We complement this paper with a small site on http://www.pt-ai.org/ai-polls/. On this site, we provide a) the raw data from our results [anonymous unless the participants decided to put their name on their responses], b) the basic results of the questionnaire, c) the comments made, and d) the questionnaire in an online format where anyone can fill it in. We expect that that online questionnaire will give us an interesting view of the 'popular' view of these matters and on how this view changes over time. In the medium run, it be interesting to do a longitudinal study that repeats this exact questionnaire. We leave it to the reader to draw their own detailed conclusions from our results, perhaps after investigating the raw data. Let us stress, however, that the aim was to 'gauge the perception', not to get well-founded predictions. These results should be taken with some grains of salt, but we think it is fair to say that the results reveal a view among experts that AI systems will probably (over 50%) reach overall human ability by 2040-50, and very likely (with 90% probability) by 2075. From reaching human ability, it will move on to superintelligence in 2 years (10%) to 30 years (75%) Future Progress in Artificial Intelligence: A Poll Among Experts 15/19 thereafter. The experts say the probability is 31% that this development turns out to be 'bad' or 'extremely bad' for humanity. So, the experts think that superintelligence is likely to come in a few decades and quite possibly bad for humanity -this should be reason enough to do research into the possible impact of superintelligence before it is too late. We could also put this more modestly and still come to an alarming conclusion: We know of no compelling reason to say that progress in AI will grind to a halt (though deep new insights might be needed) and we know of no compelling reason that superintelligent systems will be good for humanity. So, we should better investigate the future of superintelligence and the risks it poses for humanity.
 Acknowledgements Toby Ord and Anders Sandberg were helpful in the formulation of the questionnaire. The technical work on the website form, sending mails and reminders, database and initial data analysis was done by Ilias Nitsos (under the guidance of VCM). Theo Gantinas provided the emails of the TOP100. Stuart Armstrong made most graphs for presentation. The audience at the PT-AI 2013 conference in Oxford provided helpful feedback. Mark Bishop, Carl Shulman, Miles Brundage and Daniel Dewey made detailed comments on drafts. We are very grateful to all of them. Contribution to this questionnaire is by invitation only. If the questionnaire is filled in without such an invitation, the data will be disregarded. Answers will be anonymized. Results will be made publicly available on the site of the Programme on the Impacts of Future Technology: http://www.futuretech.ox.ac.uk (http://www.futuretech.ox.ac.uk) . Thank you for your time!  " 1 . 1 In your opinion, what are the research approaches that might contribute the most to the development of such HLMI?" [Selection from list, more than one selection possible.] − Algorithmic complexity theory − Algorithms revealed by computational neuroscience − Artificial neural networks − Bayesian nets − Cognitive science − Embodied systems − Evolutionary algorithms or systems − Faster computing hardware − Integrated cognitive architectures − Large-scale datasets − Logic-based systems − Robotics − Swarm intelligence − Whole brain emulation − Other method(s) currently known to at least one investigator − Other method(s) currently completely unknown − No method will ever contribute to this aim
 Future Concerning the above questions, how would you describe your own expertise?" (0 = none, 9 = expert) − Mean 5.85 2. "Concerning technical work in artificial intelligence, how would you describe your own expertise?" (0 = none, 9 = expert) − Mean 6.26 3. "What is your main home academic discipline?" (Select from list with 8 options: Biology/Physiology/Neurosciences -Computer Science -Engineering [non CS] -Mathematics/Physics -Philosophy -Psychology/Cognitive Science -Other academic discipline -None.) [Absolut numbers.] a. Biology/Physiology/Neurosciences 3
 Other method(s) currently known to at least one investigator Other method(s) currently completely unknown No method will ever contribute to this aim //www.fhi.ox.ac.uk/) (http://www.futuretech.ox.ac.uk) This brief questionnaire is directed towards researchers in artificial intelligence or the theory of artificial intelligence. It aims to gauge how people working in the field view progress towards its original goals of intelligent machines, and what impacts they would associate with reaching these goals.
 Vincent C. Müller (http://www.sophia.de) & Nick Bostrom (http://www.nickbostrom.com/) University of Oxford September 2012 A. The Future of AI Define a "high-level machine intelligence" (HLMI) as one that can carry out most human professions at least as well as a typical human. 1. In your opinion, what are the research approaches that might contribute the most to the development of such HLMI?: for the purpose of this question that human scientific activity continues without major negative disruption. By what year would you see a 10%/50%/90% probability for such HLMI to exist? 3. Assume for the purpose of this question that such HLMI will at some point exist. How likely do you then think it is that within (2 years / 30 years) thereafter, there will be machine intelligence that greatly surpasses the performance of any human in most professions? 4. Assume for the purpose of this question that such HLMI will at some point exist. How positive or negative would be the overall impact on humanity, in the long run? Please indicate a probability for each option. (The sum should be equal to 100%.) Total: 0% Questionnaire: Future Progress in Artificial Intelligence | Phil... http://www.pt-ai.org/questionnaire-future-progress-artificial-in...
 Baum et al. asked  for the ability to pass a Turing test, a third grade school year exam [i.e. for 9 year olds] and do Nobel Prize level research. They assume that all and only the intelligent behavior of humans is captured in the Turing test. The results they Within 10 years Between 11 and 25 years Between 26 and 50 years More than 50 years Never Totals got for the 50% probability point were: 2040 (Turing test), 2030 (third grade), and 11% 14% 22% 40% 14% 100% 7/19 2045 (Nobel). 4. Sandberg and Bostrom's first question was quite similar to our 2 nd (see below): "Assuming no global catastrophe halts progress, by what year would you assign a 3. Future Progress in Artificial Intelligence: A Poll Among Experts 10%/50%/90% chance of the development of human-level machine intelligence?"
 Clicks of the 'never' box. These answers did not enter in to the averages above. Future Progress in Artificial Intelligence: A Poll Among Experts Future Progress in Artificial Intelligence: A Poll Among Experts 10/19 11/19 10% 2020 2033 29 Algorithms revealed by computational Cognitive science 50% 2050 2097 200 Never no. % Integrated cognitive architectures 90% 2093 2292 675 10% 2 1.2 Artificial neural networks TOP100 Median Mean St. Dev. 50% 7 4.1 50% 10% 90% Embodied systems 2072 110 Large-scale datasets Faster computing hardware 2050 2024 2034 33 28 16.5 Other method(s) currently completely 90%: 2070 2168 342 ALL Whole brain emulation Median Mean St. Dev. Other method(s) currently known to at Evolutionary algorithms or systems 10%: 2022 2036 59 50%: 2040 Logic-based systems 2081 153 90%: Algorithmic complexity theory 2075 2183 396 No method will ever contribute to this Robotics Swarm intelligence Results sorted by percentage steps: 10% Bayesian nets Median Mean St. Dev. PT-AI 2023 2043 81 00% 10% 20% 30% 40% 50% 60% AGI 2022 2033 60 EETN 2020 2033 29 TOP100 2024 2034 33 ALL 2022 2036 59 50% Median Mean St. Dev. PT-AI 2048 2092 166 one-year increments] or check a box marked 'never'. AGI 2040 2073 144 EETN Results sorted by groups of respondents: 2050 2097 200 TOP100 2050 2072 110 PT-AI ALL Median Mean St. Dev. 2040 2081 153 10% 90% 2023 Median Mean St. Dev. 2043 81 50% PT-AI 2048 2080 2092 2247 166 515 90% AGI 2080 2065 2247 2130 515 202 AGI EETN Median Mean St. Dev. 2093 2292 675 10% TOP100 2070 2022 2033 2168 60 342 50% ALL 2040 2075 2073 2183 144 396 90% 2065 2130 202 EETN Median Mean St. Dev.
			 There is a collection of predictions on http://www.neweuropeancentury.org/SIAI-FHI_AI_predictions.xls
			 .11.12 10:29 OEZ