Abstract: Some emerging technologies promise to significantly improve the human condition, but come with a risk of failure so catastrophic that human civilization may not survive. This article discusses the great downside dilemma posed by the decision of whether or not to use these technologies. The dilemma is: use the technology, and risk the downside of catastrophic failure, or do not use the technology, and suffer through life without it. Historical precedents include the first nuclear weapon test and messaging to extraterrestrial intelligence. Contemporary examples include stratospheric geoengineering, a technology under development in response to global warming, and artificial general intelligence, a technology that could even take over the world. How the dilemma should be resolved depends on the details of each technology's downside risk and on what the human condition would otherwise be. Meanwhile, other technologies do not pose this dilemma, including sustainable design technologies, nuclear fusion power, and space colonization. Decisions on all of these technologies should be made with the long-term interests of human civilization in mind. This paper is part of a series of papers based on presentations at the Emerging Technologies and the Future of Humanity event held at the Royal Swedish Academy of Sciences on

Introduction Would you play a game of Russian roulette? Would you take a six chamber revolver, put one bullet in, give it a spin, point it at your head, and pull the trigger? How about for a million dollars? Would you play? I would guess that most readers of this paper would not play. I would guess that you would think that a chance of one million dollars is not worth it to take this risk of ending up with a bullet in the brain. I personally would not play, for the same reason. Our brains and our lives are simply worth more than that. But suppose your life circumstances were different. Suppose you were struggling with money, that you were basically broke. Suppose you were sick, with a chronic condition you cannot afford to cure. Maybe you do not have that many years left to live anyway. Now, with less to lose and more to gain, that game of Russian roulette might start to look more attractive. Now, you might start counting how much that million dollars could do for you. Could it cure your sickness, make you healthy again? Could it add years to your life? Could it pull you out of poverty? Could it give you basic comfort? If the million dollars would do enough for you, then maybe you would choose to play. Desperate circumstances can sometimes warrant taking desperate risks. If it works, the circumstances get better, maybe much better. But it might not work, and if it does not, it comes with a downside-in this case, a bullet in the brain. Whether to play the game is a downside dilemma: a dilemma involving a significant possible downside. This paper talks of a great downside dilemma. It is great because the stakes are so high-indeed, they are literally astronomical. At stake is not the fate of a single person, as in Russian roulette, but the fate of human civilization. This includes the roughly seven billion people alive today and the many more members of all the future generations that might ever live. The stakes are astronomical because humans (or our descendants) might be able to colonize space and achieve great things across the Universe. Human civilization already has an active space program, and space colonization seems feasible, as long as no great catastrophe denies humanity the chance. The rest of the Universe is vastly larger than our humble home planet, so space colonization would open up enormous new opportunities. Meanwhile, for all humanity currently knows, humans might have the only intelligent civilization anywhere in the Universe. And so the stakes could mean nothing less than the success or failure of intelligent civilization in the entire universe. A great downside dilemma, indeed. | Royal Swedish To be more specific, the great downside dilemma is any circumstance in which human civilization must choose whether to take a risk in which, if it works out, the benefit greatly improves the human condition, but if it does not work out, a catastrophe will occur, a catastrophe so large that civilization could perish, a metaphorical bullet in the brain. The dilemma is whether to take the risk. How much does civilization value that improvement in its condition? Could it be enough to pull civilization out of desperate circumstances? How large is the risk of catastrophe? Is it small enough that the risk is worth taking? Can any risk of civilization perishing be small enough to justify taking the risk? These questions must be answered in order to decide whether to take the risk. The great downside dilemma arises often for decisions about whether to pursue certain emerging technologies. These technologies promise to solve major societal problems. They bring peace, cure disease, protect the environment, and more. Or rather, they do these things if they work as intended. However, they may not work out as intended. They may fail, and fail catastrophically. In the worst cases, they could kill every living human-the extinction of our species-and destroy much of the rest of Earth's biosphere as well. Should society develop and launch these technologies, given their promise and despite their risks? That is the great downside dilemma for emerging technologies. This dilemma is an important issue for society as a whole and especially for scientists and engineers, who by virtue of their background are especially able to contribute to the debate. This dilemma is one important part of the broader challenge of avoiding civilization-ending global catastrophes. A growing body of scholarship recognizes the avoidance of these catastrophes as crucial for the long-term success of human civilization, and likewise as a key priority for action today  [1] [2] [3] [4] [5] [6] [7] [8] . Visionary technologist James Martin likened this era of civilization to a turbulent river that it must navigate  [9] . If this era of civilization successfully navigates the river, then a long, bright future awaits, both on Earth and beyond. However, if it fails, then human civilization suffers a premature death. This paper describes several great downside dilemmas for emerging technologies and explains how humanity can navigate through them. The paper also discusses some other technologies that do not pose this dilemma because they promise to bring major benefits without a significant catastrophic risk.
 Historical precedents Amazingly, the great downside dilemma for emerging technologies has been faced at least twice before. The first precedent came in the desperate circumstances of World War II. The dilemma was whether to test-detonate the first nuclear weapon. While nuclear weapons proved to be unprecedentedly destructive weapons, a single detonation did not destroy the entire planet as some initially feared. The second precedent came during calm circumstances but still posed a dilemma every bit as large: whether to engage in messaging to extraterrestrial intelligence (METI). METI is of note because the dilemma still has not been resolved. Humanity still does not know if METI is safe. Thus METI decisions today face the same basic dilemma as the initial decisions in decades past.
 Nuclear weapons It was 1945, towards the end of World War II. An American team of physicists, engineers, and military personnel built the first atomic bomb, which they named Trinity. Trinity was to be detonated in a test explosion, to make sure the technology worked, before using additional atomic bombs against Japan. By that point, Germany had already surrendered. Japan was nearing defeat, and the United States believed that the atomic bomb could compel Japan to surrender without the US waging a long, bloody invasion. It might seem counterintuitive, but this most powerful of weapons was built to save live  1  . However, some of the physicists worried that the test might fail catastrophically. They worried that the detonation could ignite the atmosphere, ending life on Earth. They believed the chance of this happening to be exceptionally small, due to their understanding of the relevant physics. Still, they closed their report on the topic with the line 'However, the complexity of the argument and the absence of satisfactory experimental foundations makes further work on the subject highly desirable'  [11] . Thus the risk did give them some pause. Sure enough, they took the risk. As is now known, the Trinity test succeeded: the bomb worked, and the atmosphere did not ignite. The rest is history. Humanity survived the first atomic bomb detonation, the next two, which were dropped on Hiroshima and Nagasaki,  1  Reed  [10]  reviews the history of the atomic bomb development and the corresponding physics. and the 2054 atomic bombs that have been detonated since in further testing  2  . The largest of these bombs, the Soviet Tsar Bomba, had a yield equivalent to about 50 megatons of TNT, a whopping 2500 times larger than Trinity. The atmosphere did not ignite. And physics has by now progressed to the point where we understand with very high confidence why atomic bomb detonations would not cause these harms (though they can of course cause other harms). But for that brief moment in time, when the first atomic detonation was under consideration, a great downside dilemma was faced, without the benefit of hindsight that now exists. Today, nuclear weapons remain a major threat. A single nuclear weapon could kill thousands or even millions of people. Nuclear war with hundreds or thousands of weapons (about 17 000 weapons still exist, mainly held by the United States and Russia) would not produce enough radiation to cause human extinction, as Bertrand Russell, Albert Einstein, and others once feared  [12] . But it would cause significant cooling, as the smoke from burning cities rises into the atmosphere and blocks incoming sunlight. This cooling, often known as nuclear winter, could cause widespread agricultural failure, with the resulting famine threatening millions or even billions of people  [13, 14] . The worst case scenario could include human extinction  [15] . The leaders of nuclear weapons states thus face a different dilemma: In a crisis, are nuclear weapons worth using? While nuclear weapons are no longer a new technology, large nuclear arsenals have never been used in war, and so the dilemma must still be resolved without the benefit of hindsight.
 Messaging to extraterrestrials It was 1974, a relatively calm and ordinary year by most measures. But an unusual exercise was in preparation at an astronomy observatory in Arecibo, Puerto Rico. The Arecibo Observatory hosted what was (and still is) the largest radio telescope in the world  3  . Usually, the telescope is used either for radio astronomy, which detects radio waves incoming from the rest of the Universe, or radar astronomy, which studies the solar system by sending radio waves towards planets and other nearby objects and analyzing the waves that bounce back. Radio and radar astronomy are generally harmless and of scientific value. But in 1974, the Arecibo telescope was to be used differently. The plan was to send a message from Arecibo to a cluster of stars 25 000 light years away. The Arecibo Message was designed by astronomer Frank Drake and colleagues with the premise of METI. The message contained seven parts describing human physiology and astronomy. This was not the first exercise in METI. In 1962, the Morse Message was sent from the Evpatoria Planetary Radar in Crimea to Venus. But the Morse Message was as harmless as regular radar astronomy studying Venus. Because the Arecibo message was broadcast elsewhere, it broke new ground. So far, the Arecibo message has not received a response. Of course it has not: it was sent 40 years ago to a location 25 000 light years away. It will take at least another 49 960 years for the message to arrive and the response to reach back to Earth 4 . And it is possible that no ETI will receive the Arecibo message. Indeed, it is possible that there are no ETI out there to receive it. It is also possible that ETI will receive it but not respond in any way. So even after another 49 960 years, the Arecibo message could prove inconsequential to humanity, except for its modest educational value. But it might not. The message could receive a response. Despite some proclamations to the contrary, humanity has little understanding of how an encounter with ETI would proceed. Some people expect that ETI would benefit humanity, providing scientific knowledge and intercultural exchange, or even solutions to humanity's problems. Others expect that ETI would harm humanity, enslaving or even killing us. These are among the many possible outcomes of ETI encounter  [16, 17] . Presumably, if it was known that the outcome would be beneficial, METI would proceed; likewise it would not proceed if the outcome was known to be harmful  [18] . But in 1974, it was not known. Whether to engage in METI thus posed a great downside dilemma. In 2014, at the time of this writing, it is still not known whether METI is safe. Since the Arecibo message, several other messages to ETI have since been sent to closer stars, most recently the 2013 Lone Signal project  5  . None of these messages has yet received a reply. Meanwhile, it is an exciting time for SETI-the search for ETI. Astronomers are just starting to discover extrasolar planets  [19] . But no ETI have yet been found. Until then, humanity will have deep uncertainty about the merits of METI. Some progress can be made by carefully thinking through the possibilities of ETI encounter. An argument can be made that no high-power METI should be conducted until humanity better understands the risks  [20] , but this is a controversial point. The great downside dilemma for METI persists.
 Dilemmas in the making While humanity continues to face dilemmas related to nuclear weapons and METI, new dilemmas lurk on the horizon. The stakes for these new dilemmas are even higher, because they come with much higher probabilities of catastrophe 6 . I will  2  Atomic bomb detonation data can be obtained from the Comprehensive Nuclear-Test-Ban Treaty Organization at http://www.ctbto.org/nucleartesting/history-of-nuclear-testing/world-overview/page-1-world-overview/.  3  Arecibo is the world's largest single-aperture radio telescope. Arrays of multiple telescopes combined as astronomical interferometers collectively cover larger areas.  4  Assuming the location is exactly 25 000 light years away, then 49 960 years from now is the minimum time it could take for a response to reach Earth. The response could reach Earth later if the ETI take more time before transmitting the response.  5  Full disclosure: I received funds from Lone Signal to contribute to a risk analysis of Lone Signal's transmissions. The study concluded that the transmissions posed no significant risk because the transmitter Lone Signal was using at the time did not exceed the background radio wave leakage from radio and television broadcasts  [18] .  6  This assumes that the probability of catastrophe from METI is relatively low. This is a debatable point, given the deep uncertainty surrounding the possibility of extraterrestrial contact. focus on two: stratospheric geoengineering and artificial general intelligence. Neither technology currently exists, but both are subjects of active research and development. Understanding these technologies and the dilemmas they pose is already important and will only get more important as the technologies progress.
 Stratospheric geoengineering In summer 2010, heavy monsoons flooded about one fifth of Pakistan, with millions of people affected  [21] . The floods were part of a broader northern hemisphere summer heat wave that set temperature records in many locations. Vast wildfires in Western Russia produced so much smoke that people in Moscow wore masks and airports redirected traffic. The floods, heat wave, and wildfires are among the sorts of extreme weather events that are expected to happen more often and with greater intensity as global warming worsens  [22]    7  . The standard means of lessening the harms of global warming is to reduce atmospheric emissions of carbon dioxide, methane, and other greenhouse gases. That means using energy more efficiently, switching away from coal power, reversing deforestation, and more. But despite the risks of global warming, greenhouse gas emissions have been steadily increasing, and are projected to continue increasing into the future. More emissions means warmer temperatures and more severe harms from extreme weather events, sea level rise, and more. In despair over the perceived failure to reduce emissions, observers are increasingly considering geoengineering to lower global temperatures. Geoengineering is the intentional manipulation of the global Earth system  [27] . Greenhouse gas emissions do not qualify as geoengineering because they are an unintended byproduct of activities with other aims. Perhaps the most commonly discussed type of geoengineering is stratospheric geoengineering, which would lower temperatures by injecting particles into the stratosphere, thereby blocking a portion of incoming sunlight. Stratospheric geoengineering is attractive because of its relative feasibility, efficacy, and affordability. However, stratospheric geoengineering changes regional temperature and precipitation patterns, leaving some need to adapt to climatic changes. Stratospheric geoengineering also does nothing to address the acidification of oceans caused by carbon dioxide emissions being absorbed into oceans. Ocean acidification is a major problem in its own right, a large threat to ocean ecosystems. Finally, stratospheric geoengineering also poses significant risks that could even exceed those of global warming. Perhaps the largest risk from stratospheric geoengineering is the possibility of abrupt halt. Particles put into the stratosphere will cycle out on time-scales of about five or ten years. In order to maintain stable temperatures, particles must be continuously injected into the stratosphere. If the geoengineering abruptly halts, such that additional particles are not injected, then temperatures will rapidly shoot back up towards where they would have been without geoengineering  [28] . This rapid temperature increase could be especially difficult to adapt to and thus be especially disruptive. For example, it may be difficult to determine which crops to plant in a given region, because the crops suitable for that region will change too quickly. Fortunately, the rapid temperature increase can be avoided simply by continuing to inject particles into the stratosphere. Indeed, the harms of rapid temperature increase provide strong incentive to not to stop particle injection in the first place. Under normal circumstances, people would have to be either incompetent or malicious to stop particle injection. Assuming the geoengineering is managed by responsible parties, abrupt halt may be unlikely, making stratospheric geoengineering relatively safe. This is under normal circumstances. However, particle injection may nonetheless halt if some other catastrophe occurs, such as a war or a pandemic, that prevents people from continuing the injections. The result would be a 'double catastrophe' of rapid temperature increase hitting a population already vulnerable from the first catastrophe  [29] . This double catastrophe could be very harmful; potentially it could even result in human extinction. This makes for a rather severe downside. Figure  1  depicts the double catastrophe scenario. The figure shows average global temperature versus time for three scenarios: (1) the world without geoengineering, in which temperatures gradually rise due to greenhouse gas emissions; (2) ongoing geoengineering, in which temperatures remain indefinitely at a low, stable level (around 13 °C); and (3) abrupt geoengineering halt (around the year 2080), in which temperatures rapidly rise towards where they would have been without geoengineering. Figure  1  also indicates when an initial catastrophe would occur (shortly before 2080) in a double catastrophe scenario. The great downside dilemma for stratospheric geoengineering is the dilemma of whether to inject particles into the Figure  1 . global temperature for three scenarios: no geoengineering, ongoing geoengineering, and geoengineering that abruptly stops. The initial catastrophe corresponds to a geoengineering double catastrophe  [29] .  7  I am using the term 'global warming' here instead of the usual 'climate change' to distinguish it from nuclear winter, which is also a climatic change. Any readers who still doubt the legitimacy of global warming as an issue should consult some of the many works on the topic, including accessible books by leading global warming researchers  [23, 24]  and my own humble contribution  [25] . Global warming research is also voluminously reviewed by the Intergovernmental Panel on Climate Change  [26] . stratosphere. On one hand, stratospheric geoengineering could lower temperatures, avoiding many harms of global warming. On the other hand, it poses a risk of rapid temperature increase that could even result in human extinction. So, should stratospheric geoengineering be pursued? A key factor in resolving the dilemma is understanding how bad the impacts of global warming could get without geoengineering. The floods, heat waves, and other effects already being observed will almost certainly get worse. This is bad, but there is an even worse impact potentially on the horizon: the exceedance of mammalian thermal limits. The limits depend on wet bulb temperature, which is a combination of 'regular' dry bulb temperature and humidity. When wet bulb temperature goes above 35 °C, mammals-including humans-can no longer perspire to regulate our body temperature, causing us to overheat and die. Currently, wet bulb temperatures never exceed the 35 °C limit, but under some possible global warming scenarios, the limit would sometimes be exceeded in much of the land surface of the planet  [30] . Unless humans and other mammals took shelter in air conditioning, they would die. Under these conditions, it may be difficult to keep civilization intact in the warmer regions or even worldwide. Another perspective on the potential severity of global warming comes from looking at the long-term co-evolution of the human species and Earth climates. The species Homo sapiens sapiens is dated at around 200 000 years old. This means that humans have lived through about two full glacialinterglacial cycles, i.e. ice ages and the warm periods between them, which cycle back and forth on time-scales of about 100 000 years  [23] . Archaeological evidence suggests that early Homo sapiens sapiens and their immediate ancestors had cognitive capabilities comparable to those of contemporary humans  [31, 32] . However, civilization did not take off until the agricultural revolution, which began around 10 000 years ago and occurred in at least seven independent locations within just a few thousand years. The last 10 000 years coincide with the Holocene, a warm interglacial period with a relatively stable climate, suggesting that this climate may have been crucial for the rise of civilization  [33] . Meanwhile, global warming threatens to push temperatures to levels significantly outside the range of recent glacial-interglacial cycles, bringing climates that Homo sapiens sapiens and its immediate ancestors have never seen before  [34] . To the extent that certain climates are essential for human civilization, global warming could be devastating. This sort of long-term perspective is also helpful for understanding stratospheric geoengineering risk. Global warming could last for centuries, millennia, or even longer  [34] . This is a very long time to continue injecting particles into the stratosphere. It is also plenty of time for plenty of catastrophes to occur. For example, risk analysis of nuclear war finds about a 0.1%-1% chance of nuclear war occurring during any given year  [35] . Over hundreds or thousands of years, this makes nuclear war virtually certain to occur. Of course, the world could permanently get rid of nuclear weapons, eliminating the risk. But this might not happen, and meanwhile there are other types of catastrophes to worry about. Over the time-scales of global warming, a stratospheric geoengineering double catastrophe may be quite likely. So, should stratospheric geoengineering be pursued? At this time, I do not believe a good answer to this question exists. There is too much uncertainty about both the consequences of stratospheric geoengineering and the consequences of not stratospheric geoengineering, as well as the probabilities of stratospheric geoengineering abrupt halt. Fortunately, the decision does not need to be made just yet. Global warming is not yet so bad that stratospheric geoengineering is worth the risk. But geoengineering decisions may be made soon; research to reduce the uncertainty should proceed now so that wise decisions can be made  [36, 37] . When the time comes to decide whether to launch stratospheric geoengineering, the right action to take may also be the more difficult action to take. It is quite plausible that civilization could endure the worst harms of regular global warming, but would collapse from the rapid global warming of stratospheric geoengineering abrupt halt. If this is the case, then it would be in civilization's long-term interest to abstain from stratospheric geoengineering and suffer through regular global warming. This abstention may be best regardless of how painful regular global warming could get and regardless of how unlikely stratospheric geoengineering abrupt halt would be. It is just that important to ensure the long-term viability of human civilization. But this is hardly an encouraging prospect, dooming humanity to the pains of regular global warming, when lower temperatures could so easily be produced. Meanwhile, civilization can lessen the stratospheric geoengineering dilemma by reducing greenhouse gas emissions. Regardless of any broader failures at emissions reductions, every additional bit helps. And reducing emissions helps with both sides of the dilemma: it lessens the severity of both regular global warming and the rapid temperature increase from stratospheric geoengineering abrupt halt. As discussed further below, many options for reducing emissions come with minimal dilemmas of their own, making them excellent options to pursue.
 Artificial general intelligence In January 2011, two world-leading players of the game show Jeopardy! took on an IBM computer named Watson. Watson won the game convincingly. During the last Final Jeopardy! round, human contestant Ken Jennings wrote below his response question, 'I for one welcome our new computer overlords'. It was a humorous moment in the long rise of artificial intelligence (AI). But how high can AI rise? Could AI actually become the overlords of humanity, taking over the world? And should such a development be welcomed? At the outset, it is important to distinguish between two types of AI: narrow and general. Narrow AI is intelligent in specific domains but cannot reason outside the domains it was designed for. Narrow AI is by now ubiquitous across a breadth of contexts, from searching the web to playing games like Jeopardy! narrow AI can be quite useful, and can also pose some risks. But it is not expected to take over the world, because controlling the world requires capabilities across many domains. General AI (AGI) is intelligent across a wide range of domains. Humans are also intelligent across many domains, but this does not mean that AGI would necessarily think like humans do. An AGI may not need to think like humans in order to be capable across many domains  8  . Early AI researchers boldly predicted that human-level AGI would be achieved by dates long since past, as Crevier  [39]  and McCorduck  [40]  chronicle. This grandiose failure of prediction led many modern AI researchers to be skeptical about the prospects of AGI  [41] . However, there remains an active AGI research community  [42] . Experts in the field diverge widely about when AGI is likely to be achieved and on its impacts if or when it is achieved  [43] [44] [45] . But some of the predictions about impacts are quite dramatic. One line of thinking posits that an AGI, or at least certain types of AGIs, could essentially take over the world. This claim depends on two others. First, power benefits from intelligence, such that the most intelligent entities will tend to have the most power. Second, an AGI can gain vastly more intelligence than humans can, especially if the AGI can design an even more intelligent AGI, which designs a still more intelligent AGI, and so on until an 'intelligence explosion'  [46]  or 'singularity'  [47, 48]  occurs. The resulting 'superintelligent' AGI  [49]  could be humanity's final invention  [50]  because the AGI would then be fully in control. If the AGI is 'Friendly' to humanity  [51] , then it potentially could solve a great many of humanity's problems. Otherwise, the AGI will likely kill everyone inadvertently as it pursues whatever goals it happened to be programmed with-for example, an AGI programmed to excel at chess would kill everyone while converting the planet into a computer that enabled it to calculate better chess moves  [52] . Per this line of thinking, an AGI would be much like a magic genie, such as the one depicted in the film Aladdin (John Musker and Ron Clements, directors, 1992). The genie is all-powerful but obligated to serve its master. The master can wish for almost anything, but should be careful what he or she wishes for. Indeed, genie stories are often stories of unintended consequences. For example, in the penultimate scene of Aladdin, Jafar wishes to become a genie. He was eager to gain the genie's powers, but ended up trapped in servitude (and stuck inside a small lamp). The story with AGI may be similar. The AGI would do exactly what its human programmers instructed it to do, regardless of whether the programmers would, in retrospect, actually want this to happen. In attempting to program the AGI to do something desirable, the programmers could end up dead, along with everyone else on the planet  9  . If this line of thinking is correct, or even if it has at least some chance of being correct, then AGI poses a great downside dilemma. Should an AGI be built and launched? Given the possibility of being destroyed by AGI, it might appear that AGI should simply not be built. Doing so would ensure that humanity retains control of itself and its fate. But for several reasons, the situation is not so simple. A first complication is that AGI might be Friendly or otherwise beneficial to humanity, or to the world. The benefits of a Friendly AGI could be immense. Imagine having the perfect genie: unlimited wishes that are interpreted as you intended them to be, or maybe even better than you intended them to be. That could go quite well. Perhaps there would be no more poverty or pollution. Perhaps space colonization could proceed apace. Perhaps the human population could double, or triple, or grow tens, hundreds, or thousands of times larger, all with no decline in quality of life. A Friendly AGI might be able to make these things possible. Decision-making on AGI should balance this large potential upside with the also-large downside risk. For example, suppose the AGI had a 50% chance of killing everyone and a 50% chance of doubling the human population with no decline in quality of life. The expected population would be equally large with or without the AGI. Does this mean that humanity is indifferent to launching the AGI? If it was a 51% chance of doubling the population, versus 49% for killing everyone, does this mean humanity would rather launch the AGI? What if it was a chance of the population increasing by a factor of ten, or a thousand? These are important types of questions to answer when making decisions about launching an AGI. A second complication is that AGI is not the only threat that humanity faces. In the absence of AGI, humanity might die out anyway because of nuclear weapons, global warming, or something else. If AGI succeeds, then these other threats could go away, solved by our new computer overlords. That is a significant upside for AGI. What if an AGI has a 50% chance of killing everyone, but absent AGI, humanity has a 60% of dying out from something else? Should the AGI be launched? The dilemma for AGI can thus look a lot like that for stratospheric geoengineering. Imagine, some years into the future, humanity finds itself in a difficult spot. Perhaps global warming is bringing great harm, and other environmental stressors are as well. Perhaps major countries are on the brink of war with nuclear weapons or something even more destructive. Perhaps poverty is rampant, life unsafe and unpleasant. Perhaps other solutions, including stratospheric geoengineering, are found to be unsafe or otherwise undesirable. And perhaps there is no hope in sight of conditions improving. In this case, taking the risk of launching a AGI could start to look attractive. Indeed, in terms of the long-term success of human civilization, it might even be the right thing to do. Or, the right thing may be to suffer through without AGI. It would depend on the details, just as it would for stratospheric geoengineering or a desperate game of Russian roulette. Following this logic, one way to help reduce AGI risk is to improve the general human condition. By keeping humanity out of desperate circumstances, the risk of AGI can be made to look less attractive. This opens up a wide range of opportunities to help reduce AGI risk, from reducing  8  AGI was discussed in detail in another paper in the series of papers based on the event 'Emerging Technologies and the Future of Humanity'  [38] .  9  Arguably, such a result would at least be better than an eternity trapped in a small lamp. greenhouse gas emissions to improving conditions for the world's poor. But the merits of this approach depend on how the AGI would be developed. The third complication is that AGI development could involve basic computing resources and technologies of growing economic importance. AGI is not like nuclear weapons, which require exotic materials. AGI could be developed on any sufficiently powerful computer. Computing power is steadily growing, a trend known as Moore's Law. Meanwhile, narrow AI is of increasing technological sophistication and economic importance. At the time of this writing, driverless cars are just starting to hit the streets around the world, with promise to grow into a major industry. Differences between narrow AI and AGI notwithstanding, these AI advances may be able to facilitate the development of AGI. Given the risks of AGI, it may seem attractive or even wise to relinquish precursor hardware and software technologies, potentially including certain narrow AI and the computer systems they run on  [53] . But given the pervasiveness of these technologies, it may be difficult to do so. Here lies another dilemma. Would humanity be willing to sacrifice much of its computing technology in order to avoid an AGI catastrophe? Should it? The dilemma here resembles that faced in the recent film Transcendence (Wally Pfister, director, 2014). The film shows an AGI that has been launched and is steadily taking over the world. The AGI is in many ways beneficial or even friendly, but the humans who are close to it become increasingly skeptical and decide to shut it down. (More precisely, they persuade the AGI to shut itself down, since the AGI was still in control.) However, in shutting it down, humanity had to sacrifice the internet, and potentially also other electronics. A case can be made that the AGI should not have been shut down: without the internet and other electronics, the long-term prospects for human civilization could be severely limited, such that humanity would be better off keeping the AGI intact and hoping for the best  [54] . As with stratospheric geoengineering, AGI launch decisions do not need to be made right now. However, for AGI there is great uncertainty about how much time remains. Experts are sharply divided on how long it will take to achieve AGI, with some doubting that it will ever occur. Given this uncertainty, and the high stakes of the launch decision, it is not at all too early to assess which AGIs should or should not be launched, and to create the conditions that can help ensure better outcomes whether or not an AGI is launched.
 Technologies without great downside dilemma Not all technologies present a great downside dilemma. These technologies may be disruptive, may have downsides, and may carry risks, but they do not threaten catastrophic harm to human civilization. Or, to the extent that they could threaten catastrophic harm, they do not increase the risk of catastrophe beyond what it would be without the technology, or do not increase the risk to any significant extent. Some of these technologies even hold great potential to improve the human condition, including by reducing other catastrophic risks. These latter technologies are especially attractive and in general should be pursued to the extent that their benefits and cost-effectiveness are competitive with other options for improving the human condition (and achieving any other goals). Three such emerging technologies are discussed here.
 Sustainable design Sustainable design refers broadly to the design of technologies oriented towards improving the environment, advancing sustainability, and related goals. These technologies promise to reduce the harms of climate change and other environmental problems. Quite a lot of such technologies are already in use, from the humble bicycle to advanced solar technologies. This is a vast technology space, and a lot has been said about these technologies elsewhere  [55] , so a full review here is unwarranted. What is worth noting here is that these technologies can reduce the risk of environmental catastrophes like climate change, and are often also worth pursuing for other reasons. For example, technology that uses energy, water, and other resources more efficiently can save money by avoiding purchases of these resources. Technologies like bicycles can make people healthier by giving them more exercise. Where sustainable design comes with such cobenefits, it is an especially attractive option. But given the catastrophic potential of environmental risks, some sustainable design, and potentially quite a lot of it, is worth pursuing even if it otherwise comes at an expense.
 Nuclear fusion power Nuclear fusion power is perhaps the Holy Grail of sustainable design. It promises a clean, safe, abundant energy source. If nuclear fusion power can be realized, and if it can be made affordable, then humanity's energy needs could potentially be fully met. And with abundant energy, a lot of other opportunities open up. For example: ocean water could be desalinated, eliminating water resource scarcities. Carbon dioxide could be removed from the atmosphere, which is another form of geoengineering, and a much safer one at that. Countries can develop their economies without worrying nearly as much about their environmental impact and without worrying about being dependent on another country's energy resources. One major long-term benefit of fusion power relates back to the fossil fuels it would replace. With fusion power, humanity can keep the rest of the fossil fuels underground, ready and waiting for when they will really be needed. That time will come sometime within the upcoming hundreds of thousands of years, when Earth's climate cycles back to a glacial period: a new ice age. The exact timing of the next glacial period is uncertain, and depends on, among other things, how much greenhouse gas humanity emits  [34] . But, barring any other radical changes to the global Earth system (such as its dismantling by a runaway AGI), a glacial period will eventually occur. And when it does, it could help to still have some fossil fuel around to lessen the bite of the global cooling  [56, p 234-235] . This would be yet another form of geoengineering, one with the long-term interests of human civilization in mind. Unfortunately, it is not clear if or when the Holy Grail of fusion power will be achieved. Fusion power research has been going on for decades  [57] , and it may take more decades still. With such a long development period, fusion power is a modern analog to cathedrals. Many cathedrals took a century or longer to build. This includes at least one cathedral currently under construction, Sagrada Família in Barcelona, whose construction began in 1882 and has no clear projected completion date. Humanity's track record with cathedrals indicates its capability to complete large, multi-century, intergenerational projects. Perhaps the fusion power project will be completed too. Unlike cathedrals, it is not known if it is even possible to complete the fusion power project: to make fusion power a major energy source for human civilization. Already, it is possible to generate power from nuclear fusion. First came uncontrolled fusion-fusion bombs-beginning with the detonation of Ivy Mike in 1952. Soon after came controlled fusion, beginning with Scylla 1 at Los Alamos National Laboratory in 1958  [58] . Controlled fusion is what can be used for electricity generation. However, controlled fusion thus far has always consumed more energy than it generates. A major breakthrough recently occurred at the National Ignition Facility at Lawrence Livermore National Laboratory: for the first time, a net energy gain occurred within the fuel that triggers the fusion  [59] . However, the fuel is just one part of the fusion process; the National Ignition Facility experiment consumed overall about 100 times more energy than it generates. But, clear progress is being made. On the other hand, much more progress is needed still, and it is not clear if or when net energy gain will be achieved, or if it would be affordable. If affordable fusion power is achieved, it would be transformative. The fuels are deuterium and lithium, supplies of which can last for thousands to billions of years, depending on power plant design, and there could be no significant radioactive waste  [60] . While fusion reactors potentially could be used to generate materials for nuclear weapons, their weapons proliferation risk would be lower, potentially much lower, than it is for fission power  [61, 62] . While fusion power research is expensive and the prospects for success uncertain, the potential benefits are, in my own estimation, sufficient to justify ongoing investment. This cathedral is well worth attempting to build.
 Space colonization The dream of living beyond Earth may be as old as humanity itself. Within the last century, concrete steps have been taken towards this dream. The project of colonizing space may take even longer than the project of fusion power, perhaps orders of magnitude longer. But it comes with its own set of sizable benefits, with relatively little risk. One benefit is the emotional inspiration that humanity can draw from marveling at its cosmic achievement  [63] . Other notable benefits are more practical, but no less great. One major benefit of space colonization is the protection it offers against global catastrophes on Earth. If humanity has self-sufficient space colonies, then it can survive even the complete destruction of its home planet. A spacefaring civilization is a more resilient civilization. This benefit has prompted calls for space colonization [64]. However, space colonization using current technology would be highly expensive and perhaps not even feasible, rendering other options for protecting against catastrophes, including Earthbased refuges, the more cost-effective option  [65, 66] . The protections that space colonization could offer do not justify investment in space colonization at this time. While space colonization can protect against harms, it can also enable major benefits on its own. The opportunities for civilization are, quite literally, astronomically greater beyond Earth than on it. Indeed, the astronomic potential for human civilization is a main reason why great downside dilemmas and other global catastrophic risk decisions are so important to resolve. But again, this does not mean that humanity should invest in space colonization at this time. Instead, it would be wise to focus on the catastrophic threats it faces, such that future generations can go on to colonize space and achieve astronomically great success as a civilization.
 Conclusion The fate of human civilization now hangs in the balance. As James Martin put it  [9] , humanity is going through a turbulent river full of many threats to its survival. Many of these threats derive from risky emerging technologies like stratospheric geoengineering and artificial general intelligence. Some threats also derive from established technologies like nuclear weapons and radio telescopes for messaging to extraterrestrials. And other technologies do not pose a significant threat, including sustainable design technologies, nuclear fusion power, and space colonization. Meanwhile, all of these technologies, if used properly, could help humanity navigate the turbulence. And if the turbulence is successfully navigated, a very long and bright future awaits. Humanity's future could include billions of years on Earth as well as a much bigger and longer existence across the Universe. Human civilization and its descendents can achieve many great things, if only it has the opportunity. Navigating the turbulence-preventing civilization-ending global catastrophe-is thus a crucial task for this era of human civilization. The great downside dilemma for risky emerging technologies could be an especially difficult stretch of turbulence for humanity to navigate. Technologies like stratospheric geoengineering and artificial general intelligence pose great temptations, especially if humanity finds itself in difficult circumstances. For the long-term sake of human civilization, it may be best to abstain from the technologies, but over the short-term, abstention could mean suffer through life without them. Global warming is just one of several forces that could put humanity in desperate circumstances in the not-too-distant future, making risky technologies especially attractive. If the right decisions are to be made about these various technologies-and that could mean taking the risk of using them-then two things are needed. First, the risks must be understood. People must know what the right decision is. This means characterizing the probabilities that the technologies will fail, the severity of harm if they do fail, and humanity's prospects if the technologies are not used. But, as they say, knowing is only half the battle. The other half is applying the knowledge. The second thing needed is for decision-making procedures to be in place such that bad risks are not taken. Accomplishing this means bringing together the many people involved in risky technology development, from scientists and engineers to government regulators. Some scientists and engineers might not like having their work regulated, but this only underscores the importance of including them in the process, so their concerns can be addressed, as can anyone else's. Many jurisdictions already regulate a variety of technologies, in light of the risks they pose. This is a good step. But emerging technologies pose new challenges that must be addressed in turn. And the global nature of the worst catastrophes suggests a role for international cooperation  [67] . Efforts at smaller scales can also play a role, including the daily actions everyone can make to protect the environment, promote peace, and otherwise keep humanity out of desperate circumstances. For the sake of human civilization-indeed, for the sake of the Universe-actions across all these scales are well worth taking. Academy of Sciences     Physica Scripta Phys. Scr. 89 (2014) 128004 (10pp) doi:10.1088/<PHONE>/89/12/128004
			 Phys. Scr. 89 (2014) 128004 S D Baum


Abstract: The development of artificial intelligence will require systems of ethical decision making to be adapted for automatic computation. However, projects to implement moral reasoning in artificial moral agents so far have failed to satisfactorily address the widespread disagreement between competing approaches to moral philosophy. In this paper I argue that the proper response to this situation is to design machines to be fundamentally uncertain about morality. I describe a computational framework for doing so and show that it efficiently resolves common obstacles to the implementation of moral philosophy in intelligent machines.

Implementation of Moral Uncertainty in Intelligent Machines

Introduction Advances in artificial intelligence have led to research into methods by which sufficiently intelligent systems can be guaranteed to follow ethically defensible behavior. Successful implementation of moral reasoning may be critical for managing the proliferation of autonomous vehicles, workers, weapons, and other systems as they increase in intelligence and complexity. Approaches towards moral decision making generally fall into two camps, "topdown" and "bottom-up" approaches  (Allen et al. 2005) . Top-down morality is the explicit implementation of decision rules into artificial agents. Schemes for top-down decision making that have been proposed for intelligent machines include Kantian deontology  (Arkoudas et al. 2005 ) and preference utilitarianism  (Oesterheld 2016) . Bottom-up morality avoids reference to specific moral theories by developing systems that can implicitly learn to distinguish between moral and immoral behaviors, such as cognitive architectures designed to mimic human intuitions  (Bello and Bringsjord 2012) . There are also hybrid approaches which merge insights from the two frameworks, such as one given by  Wiltshire (2015) .
 The Problem of Moral Disagreement A problem which has been cited as an obstacle to the development of machine ethics is the lack of agreement among moral philosophers on which theory of ethics should be followed  (Shulman et al. 2009; Bello and Bringsjord 2012; Bostrom 2014; Brundage 2014 ). The 2009 PhilPapers survey of philosophy faculty revealed that 26% accepted or leaned towards deontology, 24% accepted or leaned towards consequentialism, 18% accepted or leaned towards virtue ethics, and the remaining 32% favored other approaches entirely. For someone who believes in a particular approach to ethics, the correct system for implementation in artificial moral agents may be obvious, but the path forward for institutions and society as a whole remains unclear. Companies, governments, and researchers will have to decide which system to use for artificial agents and will be faced with a difficult choice between competing moral paradigms. This paper will describe a computational framework that will determine how we should design moral machines in the light of this disagreement, but first it is necessary to determine exactly why moral disagreement is a problem. Just because there exists disagreement does not necessarily imply that research and development programs cannot succeed in their objectives, nor does it necessarily imply that we need to worry about differences in opinion. The aforementioned works do not explicitly describe how moral disagreement would prevent us from building satisfactory moral machines, so we will need to clarify the nature of the problem before proceeding with a solution. There are two plausible reasons for why moral disagreement can pose a serious problem for the project of machine ethics. First, it could be a pragmatic problem as disagreements among engineers, policymakers and philosophers interfere with projects that require cooperation. In a worst-case scenario, decisions over which research programs to fund will turn into bitter ideological battles, research agencies will become bogged down in disputes, and developers will split up their resources and devolve into a competitive mindset which reduces information sharing and slows research progress, thus making it more difficult to construct moral machines. Second, it could be a moral problem. If we are so uncertain about morality and are split across many different moral principles, then the likelihood that anyone's particular moral system is entirely correct is statistically extremely low  (Shulman et al. 2009) . Therefore, if we do build AIs grounded in any particular moral system, then they will probably be making many poor moral decisions.  Bello and Bringsjord (2012)  argue that moral disagreement provides a reason to avoid top-down approaches to machine ethics in favor of bottom-up or hybrid approaches that copy or take inspiration from human moral thinking. However, it is not clear how this would solve the problem of disagreement. The authors claim that people can agree on examples of good moral behavior despite disagreeing over specific theories, but people's personal moral judgements can differ as widely as moral theories do when faced with moral dilemmas  (Greene et al. 2001 ) and when they are considering politicized moral issues such as racial fairness, animal farming, and economic inequality. Therefore, even if we attempt to circumvent normative disputes by eschewing top-down ethics in favor of bottom-up frameworks, it is perfectly plausible that researchers and government regulators with strong value disagreements will come into conflicts over the kinds of moral intuitions and data which should or shouldn't play a role in automated reasoning.  Brundage (2014)  also notes that there is no reason to expect that Bello and Bringsjord's approach would actually be reliably ethical. Moreover, in the cases where people do agree on moral choices despite disagreeing over moral theories, there is neither a pragmatic nor a philosophical problem with the top-down approach: a utilitarian engineer has no moral reason to care whether a robot's computational pattern is internally utilitarian or not, as long as it is actually maximizing utility; more generally, if we know that machines will act in the morally right way, then we have no reason to consider the specific programming of machine code to be morally significant. So to the extent that moral disagreements between ethical theories might pose pragmatic and moral problems for the development of top-down computational ethics, they would pose equal pragmatic and moral problems for the development of bottom-up computational ethics.
 An Example of Intractable Disagreement We can illustrate this problem with a case of self-driving cars which must be programmed to decide whether to swerve to avoid animals in the path of the vehicle. Various vehicle speeds, warning times, road shapes, weather conditions and other factors can make the decision to avoid animals more or less dangerous to the occupants of the vehicle in different situations, and different moral theories can mandate different choices in such cases. One moral principle which could specify when vehicles should or should not swerve could be described as the commensurable view. Under this doctrine, animal lives should be treated as if they are worth some small fraction of what human lives are worth. In addition, let us suppose that animals with a presumed greater capacity for cognition and pain sensations should be granted a greater weight in such tradeoffs. The machine, upon calculating a sufficiently low estimate of the probability that swerving would result in human fatality or verifying some heuristical checklist of features of the situation, would avoid the animal. The commensurable view is most clearly reminiscent of utilitarianism, but other moral theories may embrace it nonetheless-for instance, one could follow a mixed theory incorporating both deontological and consequentialist ethics and believe that consequentialist considerations dominate in this particular kind of situation. Vehicles acting in this manner can be statistically expected to eventually cause a few human fatalities but also to save many animals. If two people agreed on the commensurable view but disagreed on the specific tradeoff, compromise would be relatively straightforward. For instance, if one side argued that the chance of a fatal accident would have to be under 1 in 1000 in order to swerve for a certain animal while another side argued that it should be 1 in 10,000, they could plausibly compromise at 1 in 5500, which would function as a pragmatic truce and a limit to the degree of moral harm perceived by either viewpoint. The more pressing concern is how to compromise with the incommensurable view-that humans can never be placed at additional risk of fatality in order to save animals. This might be supported by moral theories emphasizing human rights and liberty. Since this view fundamentally rejects the axiological assumptions of the other and holds that no tradeoff is permissible, there is no obvious 'halfway point' where the competing principles can meet. Unilaterally selecting one principle to govern vehicle behavior would not be acceptable for the reasons described previously. Training the driving system to determine the correct action through supervised learning would only rephrase the issue by forcing us to decide which training examples ought to be classified as right or wrong. Allowing the company to decide based on its own interests would not clearly lead to safe and ethical behavior. Laws do not provide sufficient guidance in all such situations to govern the behavior of machines, since they are often left to the conscience of a human agent. Finally, human operators cannot be guaranteed to be available to make such quick decisions, to reliably specify their preferences beforehand, or to act ethically at all. The last resort would be for the vehicles to rely on a random number generator and follow the commensurable view half of the time and the incommensurable view half of the time. At first glance, this is the only straightforward way to build a system in a way that does not overwhelmingly disfavor one particular theory: if we cannot compromise between two views on how to act in a particular repeated decision by generating a new view, then the only way to act fairly is to alternate between them. Unfortunately, this sacrifices predictability by inducing stochasticity into decision making. Furthermore, it is not Pareto optimal. A vehicle which was programmed to randomly select the commensurable view in half and only half of relevant situations could be modified so that it specifically did so for the half of situations where the animal under threat was relatively important under the commensurable view. This makes it more predictable, since its decisions are deterministic given the environments that it encounters. It also counts as a moral improvement by the lights of the commensurable view, because it prioritizes the animals which are regarded as more important. At the same time, it is no worse according to the incommensurable view, which could be indifferent to exactly which animals are saved. This makes the change a Pareto improvement. We can generalize this strategy by saying that we are keeping all moral theories in play at the time of decision making, and searching for actions which provide the most value across all of them, rather than seeking to select one moral theory and ignoring others. This procedure, however ad hoc, seems to be the best possible way of approaching this particular case, and its principles could be fruitfully applied to other cases where moral values are not just in disagreement but wholly incommensurable. Ideally, we should have a rigorous theoretical framework which does similar work in the broad variety of moral situations by taking all moral preferences into consideration and responding appropriately.
 Dealing with Moral Disagreement Properly Moral disagreement has been recognized as a problem for normative ethics proper, independent of any concerns regarding machine ethics. A direct approach to overcoming it is to assume that there is a correct moral theory which we are searching for, acknowledge that we are fundamentally uncertain about which moral theory is correct, and then act in such a way as to give some weight to the judgements of different theories. More generally, given the failure of moral philosophy to reach satisfactory conclusions, it can be argued that we should adopt a framework of reasoning which takes multiple moral views into account  (Lockhart 2000) . However, the idea that we ought to change our behavior in accordance with moral uncertainty is controversial, and many philosophers have attempted to address the various mathematical and philosophical problems involved with the concept (Z ˙uradzki 2016). In particular, it has been argued that making comparisons between the values and judgements of different moral theories is impossible  (Nissan-Rozen 2015) . A recent proposal aimed at answering these concerns was developed by William MacAskill in an extensive thesis (2014). It avoids objections based on incomparability and incommensurability by developing moral uncertainty as a voting problem among moral theories (MacAskill 2016). Voting provides a close analogy with the pragmatic problems faced by machine ethics: as voting is the general process by which decisions are made from the preferences of a population, computations of moral uncertainty represent a process by which agents can act in accordance with the diverse values of humanity. Since MacAskill's proposal works as a voting system where theories have equal say adjusted by their probability of being correct, it also approximates the framework which has been suggested for meta-moral reasoning by Nick  Bostrom (2009) . The rest of this paper will develop and defend the use of this model for machine ethics.
 Maximizing Expected Choiceworthiness The scheme presented in "Normative Uncertainty" (MacAskill 2014) is to make action-guiding judgements based on all moral theories in which the agent has some level of credence. MacAskill defines his approach as a metanormative theory, characterized as follows: The agent investigates a decision-situation comprising a quintuple 〈S, t, A, T, C〉 where S is the decision maker, t is the time, and A is the set of possible actions to take. T is the set of normative theories being considered, where a theory T i is a function of decision-situations that produces a cardinal or ordinal choiceworthiness score for actions CW i (A) for all actions a ∈ A. C(T i ) is a credence function assigning values in [0, 1] to every T i ∈ T. A metanormative theory is a function of decision-situations that produces an ordering of the actions in A in terms of their appropriateness. MacAskill distinguishes between moral theories which assign cardinal rankings to options, as utilitarianism would, and moral theories which assign ordinal rankings. When it comes to cardinal theories, MacAskill also distinguishes between sets of theories with moral values which are directly comparable and other cardinal theories which are incomparable with any other theory. He proposes the metanormative theory of maximizing expected choiceworthiness (MEC), the steps of which are as follows: 1. Each set K of k moral theories in which the choiceworthiness rankings of options are cardinal and intertheoretically comparable are aggregated into single theories, where C T K ð Þ ¼ X k i¼1 CðT i Þ CW K A ð Þ ¼ P k i¼1 CW i A ð ÞC T i ð Þ P k i¼1 C T i ð Þ In other words, the credence in the new theory equals the sum of the agent's credences in each of the individual theories in the set, and the choiceworthiness of an option according to the new theory is the credence-weighted average of the choiceworthiness of the option in all of the individual theories. 2. The rankings of options according to each ordinal theory o are used to provide choiceworthiness scores of options using a modified Borda scoring rule that is designed to properly account for ties, denoted here as CW B . The score for an option is equal to the number of worse options minus the number of better options. CW B o A ð Þ ¼ a 2 A : CW o a ð Þ\CW o A ð Þ j j À a 2 A : CW o a ð Þ j i CW o A ð Þj This violates the independence of irrelevant alternatives, but MacAskill provides reasons to allow the violation. First, independence of irrelevant alternatives is the least essential of all the axioms present in Arrow's Impossibility Theorem, and if a different axiom were to be violated then there would be no prospects for a satisfactory metanormative account involving ordinal moral theories. Second, the primary motivation for independence of irrelevant alternatives is that it combats tactical voting, but tactical voting is not a problem with moral theories: theories aren't agents, and a moral agent can't conceal information from itself. Third, there are some moral cases where we would expect the independence of irrelevant alternatives to be violated (MacAskill 2014, p. 85). 3. The options' scores provided by each of the aggregated sets of cardinal comparable theories, the scores provided by each of the ordinal theories, and the scores provided by each of the cardinal incomparable theories p are all divided by the respective standard deviations of the moral rankings which the moral theories (or sets of intertheoretically comparable cardinal theories) provide over a general set G of actions. This provides normalized choiceworthiness rankings and is the only possible way of equalizing the value of voting for each value system (Cotton-Barratt 2013), which (as MacAskill argues, p. 115) performs the role of giving each theory equal say. CW N K ðAÞ ¼ CW K ðAÞ r CW K ðGÞ ð Þ CW N o ðAÞ ¼ CW B o ðAÞ r CW B o G ð Þ À Á CW N p A ð Þ ¼ CW p A ð Þ r CW p G ð Þ À Á A note on G: it is necessary for variance normalization that a representative set of actions (the "general set") be defined for computing the variance. This provides the background against which theories can describe whether a decision is comparatively important or comparatively unimportant from the point of view of a particular theory. MacAskill thinks that a broad account featuring many actions in this set, rather than just the ones under consideration in the present decision-situation, is theoretically desirable. Here I follow suit and model the process as if we are providing moral machines with large arrays of data representing many possible moral decisions, so that calculating the variance only needs to be done once before the resulting number is used and shared by many agents. Variance normalization also violates the independence of irrelevant alternatives, but it is a necessary violation for this application. In order to perform comparisons across moral theories, we need to determine the amount of weight which a value system places on a particular issue compared to other issues. Since different computational approaches to ethics can have wildly differing numerical outputs, it would be impossible to do this fairly without normalizing. 4. Each option is scored with the credence-weighted average of the variancenormalized scores provided by the normative theories. This provides a ranking of expected choiceworthiness: CW E A ð Þ ¼ X n i¼1 CW N i A ð ÞC T i ð Þ Then the agent chooses the action A which maximizes CW E (A). The full description and defense of each step in the theory, provided as a model for human decision making, is given by MacAskill (2014). The computational complexity of an algorithm implementing this procedure is OðjAj à jTjÞ, as adding another action requires a fixed series of evaluations to be done by each moral theory and adding another moral theory requires it to perform a fixed series of evaluations to be performed on each action. The calculation itself is very simple; almost all of the time and memory requirements will presumably stem from the computations of the moral theories themselves.
 Motivation for Maximizing Expected Choiceworthiness in Artificial Intelligence The effect of the system described above is that an agent will make prudent decisions that aim to take the most broadly desirable action in accordance with various people's values, and will avoid the very counterintuitive actions which are considered to be problems for various moral theories. For instance, when presented with the opportunity to commit a great deception in order to gain a small increment of happiness, an agent will refrain even if they view consequentialism as more likely than nonconsequentialist theories, because the relative wrongfulness of the act according to nonconsequentialist theories is significant whereas the relative benefit under consequentialism is minor. Likewise, even if an agent believes that it is probably not morally obligated to refrain from actions which cause significant indirect harms to others, it will nevertheless refrain from being careless in such a way, because the scale of the moral harms it would be committing if it were wrong would be significant. This will limit the degree to which machines will behave in ways which are considered very objectionable by minority views. This does not change the fact that a machine which maximized expected choiceworthiness could still be expected to take many actions which various parties consider to be morally wrong. Despite its superficial similarity to maximizing consequentialism, it would often sacrifice good consequences in favor of satisfying the values of other moral theories. So the adherents of most moral theories would theoretically prefer that machines be built according to their own specifications. But they cannot realistically expect the rest of industry and society to agree with them. So the actual outcome of widespread adherence to naı ¨ve individually rational behavior would be a patchwork of competing agents with different values. Not only would this leave both the pragmatic and the moral problems of disagreement unresolved, but it would leave parties with little ability to influence the world except for whatever agents they could develop as representatives for their own values. However, if parties were to agree on developing systems based upon moral uncertainty, their most critical respective interests would be given relatively high weight in all moral machines by the mechanism of variance voting. By reducing conflicts between agents, promoting cooperative machine development, and allocating value systems' decision making power to the situations which they care most about, a universal framework of moral uncertainty would solve the multiagent prisoner's dilemma constituted by machine ethics and would generally lead to a better outcome for different value systems than the situation in which there was pure disagreement. Needless to say, if an individual did accept MacAskill's thesis and believed that we ought to maximize expected choiceworthiness as human agents, then she would have a direct and compelling reason to support the implementation of the theory in computational systems. But the argument for developing machines based on maximizing expected choiceworthiness is not dependent upon the claim that we humans actually have metanormative reasons to act in such a way. An analogy with voting is illuminating: very few people believe that the winner of an election is always the one who would be the best leader, but most people agree with the system, as it is a necessary and effective compromise. So as long as we disagree with each other about ethics, we should still agree to construct moral machines based on uncertainty even if we reject the idea as a guide for human behavior.
 The Long Term Future If we look at the long run future of artificial intelligence development, there are additional reasons for supporting a framework of acting in accordance with moral uncertainty. First, if an individual is confident in her moral beliefs and possesses an expectation of progress towards objective truth in moral philosophy, then she should expect morally uncertain agents which reflect human knowledge to converge towards her beliefs in the long run. This will limit the degree to which she can expect their behaviors to be objectionable relative to those of an ideal agent which has the right views from the outset. Second, sufficiently competent machines which adhere to a particular value function are likely to pursue it endlessly with pathological consequences  (Bostrom 2012) . While this paper is not intended as a complete solution for value selection and alignment in arbitrarily intelligent agents, it is nevertheless valuable to start by designing systems that cooperate among different value systems, as that will improve the fail-safety of agents which become very intelligent but have imperfect goal functions  (Gloor 2016) . The framework proposed here clearly fulfills that criterion, and therefore serves as a potential backup or other component of a multilayered utility function even if it is not ordinarily utilized  (Oesterheld 2016) . Third, as machines become more intelligent, they might perform more moral functions autonomously, such as updating credences in moral theories and even generating new ones, depending on the degree to which progress in artificial intelligence enables them to investigate questions in moral philosophy. MacAskill (2014, ch. 7) points out that a more accurate credence function improves the expected choiceworthiness of an MEC agent's actions, so such an agent would be intrinsically motivated to expend effort into updating and improving their moral beliefs if they had acceptable criteria for determining moral truth. This is particularly desirable since there may be cases of widespread moral wrongdoing which most humans have consistently failed to identify  (Williams 2015) , which machines should be incentivized to avoid.
 How to Build an Uncertain Machine
 Ordinal and Cardinal Scoring One of the requirements of this framework is that moral theories actually provide rankings over actions. Perhaps the majority of moral theories provide neither cardinal nor ordinal rankings at first glance. However, this is a surmountable issue. First, it simply seems obvious that just about any moral theory should be able to rank certain impermissible actions, such as torturing a large number of people, as worse than other impermissible actions, like stealing someone's coffee, even if such rankings have not yet been made explicit in the philosophical literature. Second, any moral theory which judges the actions of machines can at least provide a rudimentary score system rooted in deontic logic, with 0 for all impermissible actions and 1 for all permissible actions, even if nothing else about the theory is formalized. So any moral theory must at least be translatable to a basic two-level ordinal ranking over actions, which would enable it to be implemented in this framework. If such an implementation is too coarse to adequately represent the values of such a theory, then its proponents have the option and the incentive to clarify and improve it. Defining precise scores could require varying degrees of input from humans depending on the complexity of the moral theory. However, computational approaches towards morality may provide numerical values and rankings which are otherwise absent from moral theory. Computational reasoning differs from human thinking and the procedures required for providing moral judgements in artificial intelligence systems may involve functions from which meaningful ordinal, integer or real-valued scores for actions can be extracted. Scores could be derived from the degree of rightfulness or wrongfulness of an action and/or the degree of certainty given by a moral algorithm as to whether an action is right or wrong. While the philosophical theory of MEC may seem to only refer to rigidly defined moral theories, a top-down approach to machine ethics is not required with this proposal for machine ethics. There is no computational necessity for the 'theories' implemented in the system to all be explicit moral theories in the philosophical sense. One or more bottom-up decision making systems might be included; for instance, the output of a complicated learned function intended to model human intuitions can be treated as a choiceworthiness ranking. As long as it ranks actions by normative criteria and has some probability of being the correct way for a machine to behave, it satisfies the requirements proposed by MacAskill for being a moral theory.
 Credences The outcomes of comparisons will always crucially depend upon the credence function C-an assignment of probabilities to various moral theories of being correct. There are multiple ways of determining these values. Lengthy and difficult approaches which rely on extensive human input are acceptable, as information modifying the credence function could be generated just once and then distributed to many agents as an update. Artificial agents with humanlike intelligence could presumably reason about morality for themselves, but this is not an option for the foreseeable future. One option is to make credences which reflect the beliefs of moral philosophers. The credence assigned to a moral theory would then be equivalent to the proportion of philosophers who affirm the theory, or possibly a score computed from a more sophisticated set of votes taking philosophers' full credences and rankings into account. However, the assumption that moral philosophers are in fact authorities regarding moral philosophy in the same way that practitioners of other disciplines are experts in their respective domains is highly controversial, and has been attacked by  Archard (2011) ,  Cross (2016)  and others. Moreover, the type of moral expertise which provides judgements about which actions and theories are right or wrong is merely one type of moral expertise which may not hold even if others do  (Driver 2014; Jones and Schroeter 2012) . Finally, the judgements of moral philosophers have been shown to be vulnerable to systematic cognitive bias  (Schultz et al. 2011; Schwitzgebel and Cushman 2012) . However, there are a few differences when it comes to the case of assigning credences to a system of maximizing expected choiceworthiness in machine intelligence on the basis of a broad survey of moral philosophers' beliefs. First, the system is not deferring to the views of any particular philosopher or group of philosophers. Rather, it is making decisions based upon all philosophers' views in accordance with how prevalent they are; the views of disagreeing philosophers are being used to support the proposition that both views are approximately equally likely to be correct, rather than the proposition that either particular view happens to be correct. Therefore, the argument from disagreement as presented by  Cross (2016)  against a strong conception of moral expertise does not apply to this case. Second, the system of decision making presented here is specifically designed for use by artificial agents rather than humans. Artificial intelligence systems are not voting participants of democracy and cannot be described as flourishing, so to whatever extent there is moral expertise, it is not wrong for machines to defer to it, as  Archard (2011)  argues is the case for humans. The above points leave many of the arguments against moral expertise unaddressed, and resolving that debate is beyond the scope of this paper. But given that we do not possess artificial general intelligence capable of comprehending and judging moral theory, the practical alternative to deferring to moral philosophers is not for agents to make choices for themselves, which is the context for the ordinary debate on moral expertise, but it is to defer to the judgements of other members of society besides philosophers. But non-philosophers would necessarily be making moral judgements in the same context of controversy over the nature and status of moral knowledge, and they would presumably be vulnerable to cognitive bias just as professional philosophers are, so any claim for their expertise would be the target of arguments similar to those leveled against professional philosophers. So there is a prima facie case for deferring to moral philosophers for determining machine credences which is stronger than the case for deferring to moral philosophers for making moral judgements in ordinary contexts, and no fundamental reason to defer to non-philosophers for determining credences. However, there may be contingent reasons to defer to non-philosophers based on various traits and characteristics commonly possessed by philosophers and non-philosophers. First, moral expertise might exist in the form of practical experience and wisdom rather than any kind of academic knowledge  (Jones and Schroeter 2012) . If individuals with particular virtues, experiences, and character traits are regarded as moral authorities, then credences should be assigned in a way that reflects how they act in different situations. Second, even if being a philosopher counts as a reason in favor of attributing moral expertise, our available community of philosophers could be a predictably flawed group to defer to. One reason for this is that they are not representative of the broader human population, whether measured by socioeconomic status, race, gender, nationality, personality type, political ideology, or other characteristics which are often correlated with different opinions about morality. But the most reasonable solution to this problem is to weight the votes of philosophers, or virtuous wise people as proposed above, on the basis of the uniqueness of the various backgrounds and perspectives which they represent, since it preserves the presence of a community with (arguable) moral expertise while functionally acting similarly to a hypothetical group of philosophers with an ideal composition from various demographic groups. Unfortunately it is not obvious what composition of judges would be appropriate for determining credences. For instance, the fact that orphans comprise 2% of people in the world does not necessarily imply that a voting body comprised of 2% orphans and 98% nonorphans will have more accurate views on ethics than any other mix. It may be the case that both groups of people have equally sound perspectives which must be weighed against each other, and so each orphan's vote should have forty-nine times as much weight as a nonorphan's vote. Alternatively, suppose for the purpose of the example that nonorphans tend to make more accurate moral claims on average for whatever reason. Then only the votes of nonorphans might be given significant weight. Or a combination of these and other factors could imply anything in between. This lack of clarity does not imply that we should just weight all votes equally and ignore concerns about diverse representation, but it shows that the selection of an optimal standard is likely to be more difficult than it naively appears. Instead of trying to determine a single universal method of assigning credences, different kinds of artificial agents may have credences assigned differently to suit the contexts in which they operate, just as jury selection methods vary in order to ensure fairness in different types of legal cases. Therefore, fully specifying a method of determining credences is beyond the scope of this paper. Also, while complete data on potential credence judges' moral views and other characteristics to use for this project may currently be lacking, this does not present a great barrier for creating morally uncertain artificial agents, since the credence function can be used broadly and need not be computed separately and repeatedly for too many scenarios. Specialized focus groups, panels, surveys and other activities can be run occasionally at low cost to the overall industry. In one respect, the current lack of relevant data is actually a good thing: the selection of a method of assigning credences should be performed from a state of as much ignorance as possible regarding the particular moral credences which will be delivered by that method, in order to ensure that the decision making process is unbiased and focuses on the prior philosophical reasons for or against the method of assigning a credence function rather than reflecting whatever moral credences we have in the first place.
 Limited Approaches Deploying this system of normative uncertainty would require multiple moral decision making systems to function in a moral agent. An implementation could be quite simple, such as a self-driving car programmed with one module for strict traffic laws and another module comprised by a basic utilitarian calculator for death, injuries, costs, and travel time. Other applications could warrant more complexity, and implementing a large number of moral theories may be too computationally expensive for the framework to be practical in all applications. Basic systems of value comparisons involving computational shortcuts and heuristics would be easier to implement, and may also seem to be more philosophically defensible by dint of being simpler. In reality, they will have to sacrifice some theoretical properties in order to achieve this simplicity, such as equal say among theories, accurate representation of moral values, or important axioms of voting theory as dictated by Arrow's Impossibility Theorem. The pragmatic and moral problems of disagreement must generally be addressed for machine ethics to be successful, so comparisons and decisions between value systems will still have to be made in some fashion even if some machines must use simplified approaches to save on computational resources. But MEC represents one of the most recent and philosophically rigorous methods for adjudicating among different value systems, and has desirable properties including equal say, comparability across disparate moral theories, and graceful inclusion of minority values. Therefore, the system here can serve as the theoretical standard used for the development and judgement of heuristic approaches.
 Objections
 Infinite Regress The problem of moral disagreement poses a significant problem for the project of developing intelligent machines. But critics will point out that the exact method of implementing a system of moral uncertainty is controversial in its own right, as evidenced by the issue of credences. Therefore, we have not actually eliminated disagreement, but we have merely removed it from the moral domain to the metamoral domain-where pragmatic and normative problems still arise. People may disagree over what framework of moral uncertainty to use, how to assign credences, how to score options, how to construct the general set G, or other issues. Any framework for compromising over these disputes will also be subject to disagreement, prompting further disputes, and so on ad infinitum. It is true that we do not know if the system of moral uncertainty presented by MacAskill would lead to the morally best outcome, and the answer may be different depending on whose value system is actually correct. In fact, since it makes compromises among moral claims, it is almost guaranteed to commit moral errors at some point. However, given our lack of knowledge about morality, it represents a solid best guess that maximizes the expected choice-worthiness of agents' actions. The only alternatives are to unilaterally assume a single value system (whether topdown or bottom-up) with large risks of controversy and moral failure, or to choose a method of meta-moral compromise which sacrifices some of the desirable attributes of MEC while failing to avoid the pitfalls mentioned here. There is simply no better option given our limited understanding of morality. While there is also potential for a pragmatic problem to arise, I claim that it is generally less severe in the meta-moral case of disagreement than it is in the moral case. This is evidenced by the relative lack of controversy over voting methods and procedures in organizations and political institutions. While there are efforts to change voting standards in Western democracies such as the United States, such as extending the vote to younger persons or felons, these campaigns tend to be less popular and less violent than object-level campaigns over contentious ideological and policy disputes. In machine ethics, while different people may advocate for different systems of moral uncertainty, it won't be clear which one would best achieve anyone's particular value system, and disputes over core moral principles will be overshadowed by more mundane issues like fairness, computational complexity and the axioms of voting theory. Certainly there are large challenges ahead, but they are surmountable in ways that pure moral disagreement is not. This is because engineers and philosophers can appeal to more universal standards of computational and philosophical theory than they could if they were vouching for specific viewpoints. In any case, there is a limit to how narrow a difference of opinion can be before it stops being a major barrier to unified action. Flawed artificial moral agents can still provide major benefits for the world, so the lack of an uncontroversial standard for filling out the MEC procedure should not stop us from building moral machines at all any more than the predictable moral fallibility of humans and organizations should stop us from engaging in our practices of procreation and entrepreneurship.
 Pragmatism Versus Morality Hardheaded economists may point out that machines which always do the morally optimal thing are not going to be as commercially viable as ones which give some special leeway to the preferences and interests of their owners. This problem is more serious with agents following MEC than it is with ordinary moral agents: while most moral theories allow some space where multiple actions are regarded as permissible, the disjunction of moral theories does not. With multiple moral theories expressing interest in various values and the metanormative module calculating cardinal scores for all actions, the smallest preference over two actions held by a single theory would compel the agent to act in a particular way even if every other theory was indifferent, providing little control to users with nonmoral interests. Furthermore, some moral theories pose demands which contradict certain activities of legal commercial enterprises. This leaves little space for decisions to be made in the interests of the designers and users of the agent. For instance, depending on the moral beliefs of artificial agents, an automated cargo ship might divert its cargo to a desperately poor region of the world instead of fulfilling its contract, a personal care robot might zip off for several hours a day to work at a local food bank instead of giving a massage, or a financial trading algorithm might manipulate and ruin the stock price of a company which is engaging in harmful business conduct instead of turning a profit. The objection is not that we should necessarily regard these actions as bad; in fact it presupposes that such actions would be considered morally good by many theories, and the tension between one's own values and society's values has already been extensively addressed in this paper. Rather, the point is that maximally moral machines would not necessarily be regarded as desirable for commercial and industrial investment, meaning that fewer would be built. In the long run, this limits the amount of research and development which would lead to generally smarter and more beneficial machines. If we let the perfect be the enemy of the good then we will be responsible for the immoral outcomes of stunted technological progress and failure to mitigate human tragedies as quickly as we could. Therefore agents should only be obligated to satisfice expected choiceworthiness, or perhaps the set of pragmatic interests of the owners should function as a 'theory' of its own. The flaw in this objection is the assumption that moral theories do not similarly value predictability, pragmatism and the long run future of machine intelligence. For instance, rule consequentialism would clearly support the idea that agents ought to fulfill contracts and promises if it led to better consequences in the form of long run economic and technological growth. Act consequentialism may do so as well, if audacious moral behavior would be likely to cause significant legal and economic costs impairing future AI development. The decision to commit to a certain policy or procedure could also be classified as an action in itself if doing so yielded moral benefits, allowing act consequentialist theories to meet the same criteria as rule consequentialist ones. Furthermore, ethics based on the principle of telos would hold that machines designed to perform a specific purpose ought to give some priority to that purpose. There are also secondary considerations for preserving a dominating focus on moral behavior. First, explicitly placing pragmatic interests on the same level as moral theories potentially leads to a slippery slope of accepting progressively less ethical behavior while opening up yet another space for controversy and disputes over the details of such tradeoffs, but having a clear rule for maximizing expected choiceworthiness is a policy which can be committed to and accepted on principled ideological grounds. Second, public trust in artificial intelligence is fragile and rests on crucial questions of whether they are safe and ethical. Stories of immoral robot behavior are likely to have negative ramifications as the public turns cold towards machine research, development and production, and these negative ramifications should be entered into the judgements of moral theories from the outset rather than being avoided by introducing bias at the metanormative level.
 Against Maximin The desire to protect minority rights and interests may lead one to propose that the framework of MEC be replaced with a maximin criterion or some weaker lossaverse function. Simple versions of consequentialism, which maximize expected value on the normative rather than the metanormative level, are often criticized for implying that it may be right to sacrifice the lives and interests of individuals for the greater good. In a similar vein, an autonomous system which maximized expected choiceworthiness could be expected to take actions which are very wrong according to one set of values but very good according to other values, and some may find this problematic. A maximin or loss-averse function would respectively prevent or resist such scenarios. However, this intuition is mistaken in the metanormative context because counterintuitive cases of minority interests being violated are wrong on the ethical grounds of many different moral theories. For instance, killing an innocent patient in order to reuse their organs is not only considered wrong by deontological ethics, but would likely be considered wrong by rule consequentialism, virtue ethics, intuitionist ethics and other common theories as well, and always to a high degree. So in scenarios involving possible patient outcomes and organ harvesting, MEC would demand that the more common interest of refraining from harm be adhered to, while a maximin or loss-averse function would perversely give extra weight to the sidelined act consequentialist calculations over and above the degree to which act consequentialism would be concerned about the lost organs. More generally, MEC would strongly represent common-sense views on contentious ethical issues, because common-sense views on ethical issues are more common than other kinds of views. That being said, a risk-averse function might be desirable as a backup utility function to be activated in certain circumstances  (Oesterheld 2016) .
 Conclusion Disagreement over ethics is unlikely to be resolved in the near future. If we wish to create machines with the capacity to make decisions on matters of ethical importance then a fair and robust system for translating the patchwork of human values into moral guidance must be developed. The intractability of our moral disputes and the magnitude of our moral errors can both be minimized by compromising with a decision framework which emphasizes the respective priorities of different moral systems in accordance with their plausibility. William MacAskill's framework of maximizing expected choiceworthiness satisfies this description. 			 K. Bogosian