Abstract: others. 2 This definition of the far future is most explicitly stated in Beckstead (2013). In contrast, psychology and cognitive science research commonly defines ''far future'' in timescales of years (e.g.

The far future argument for confronting catastrophic threats to humanity: Practical significance and alternatives

Introduction Over several decades, scholars from a variety of fields have advanced an argument for confronting catastrophic threats to humanity, rooted in the far future benefits of doing so.  1  In this context, the far future can loosely be defined as anything beyond the next several millennia, but will often emphasize timescales of millions or billions of years, or even longer.  2  Likewise, the catastrophic threats in question-also known as global catastrophic risks (GCRs) and existential risks, among other things-are those that would affect the trajectory of human civilization over these timescales. The simplest case is catastrophes resulting in human extinction, which is a permanent result and thus affects the trajectory of human civilization into the far future. More subtle but comparably relevant cases include catastrophes resulting in the permanent collapse of human civilization, preventing humanity from ever achieving certain very great things, and catastrophes resulting in delays Sufficiently large catastrophes can affect human civilization into the far future: thousands, millions, or billions of years from now, or even longer. The far future argument says that people should confront catastrophic threats to humanity in order to improve the far future trajectory of human civilization. However, many people are not motivated to help the far future. They are concerned only with the near future, or only with themselves and their communities. This paper assesses the extent to which practical actions to confront catastrophic threats require support for the far future argument and proposes two alternative means of motivating actions. First, many catastrophes could occur in the near future; actions to confront them have near-future benefits. Second, many actions have cobenefits unrelated to catastrophes, and can be mainstreamed into established activities. Most actions, covering most of the total threat, can be motivated with one or both of these alternatives. However, some catastrophe-confronting actions can only be justified with reference to the far future. Attention to the far future can also sometimes inspire additional action. Confronting catastrophic threats best succeeds when it considers the specific practical actions to confront the threats and the various motivations people may have to take these actions. ß 2015 Elsevier Ltd. All rights reserved. in the subsequent rise of civilization toward these achievements. The scholarship argues that people should care about human civilization into the far future, and thus, to achieve far future benefits, should seek to confront these catastrophic threats. Call this the far future argument for confronting catastrophic threats to humanity. In this paper, I will not dispute the basic validity of the far future argument. Indeed, I agree with it, and have advanced it repeatedly in my own work  (Baum, 2009 (Baum, , 2010 Maher and Baum, 2013) . Instead, I assess the extent to which the far future argument is necessary or helpful for actually confronting the threats. In other words, what is the practical significance of the far future argument? I also propose and assess two alternative approaches to confronting the threats. One alternative emphasizes near future benefits of avoiding near future catastrophes. The other alternative emphasizes other (unrelated) benefits of actions that also help confront the threats, creating opportunities even for people who have zero care about the threats. It would be important if the threats can be confronted without the far future argument, because many people do not buy the argument. That people do not is suggested by a range of research. An extensive time discounting literature assess how much people value future costs and benefits. Most discounting studies use time scales of days to decades and focus on future benefits to oneself  (Frederick, Loewenstein, & O'Donoghue, 2002) ; these studies are of limited relevance to valuations of the far future of human civilization. One more relevant time discounting study finds that people discount lives saved 20 years later at a 25% annual rate and lives saved 100 years later at an 8% annual rate  (Johannesson & Johansson, 1996) ; extrapolating this suggests negligible concern for lives saved in the far future. Similarly,  Tonn, Conrad, & Hemrick (2006, p. 821)  find that people believe humanity should plan mainly for the upcoming 20 years or so and should plan less for time periods over 1000 years. In a study on social discounting,  Jones and Rachlin (2006)  find that people are willing to forgo more money to help close friends and family than distant acquaintances; they presumably would forgo even less for members of far future generations. Finally, there are considerations rooted in how societies today are structured. Several researchers have argued that current electoral structures favor the short-term  (Ekeli, 2005; Tonn, 1996; Wolfe, 2008) .  Similarly, Karlsson (2005)  suggests that the rise of decentralized capitalist/democratic political economies and the fall of authoritarian (notably communist) political economies has diminished major long-term planning. While none of these studies directly assess the extent to which people buy the far future argument, the studies all suggest that many people do not buy the argument to any significant degree. To the extent that efforts to confront catastrophic threats can be made synergistic with what people already care about, a lot more can be done. This would seem to be an obvious point, but it has gone largely overlooked in prior research on catastrophic threats. One exception is  Posner (2004) , who argues that some actions to reduce the risk of human extinction can be justified even if only the current generation and its immediate successor are valued. Another is  Baum (2015) , who proposes to confront the threat of catastrophic nuclear winter in terms that could appeal to nuclear weapon states; Baum calls this ''ethics with strategy''. But most of the prior research, including the studies cited above, emphasize the far future argument. This paper expands Posner's argument to further argue that some actions can be taken even for those who only care about their immediate communities or even just themselves. This paper also makes progress toward assessing the total practical significance of the far future by presenting a relatively comprehensive survey of GCRs and GCR-reducing actions. Such surveys are also scarce; one example is  Leggett (2006) , who surveys the space of GCRs to identify priorities for action. The present paper also has commonalities with  Tonn and Stiefel (2014) , who evaluate different levels of sacrifice that society should make in response to GCRs of different magnitude. The present paper also considers levels of sacrifice, but instead argues that, from a practical standpoint, it is better to start with those actions that require less sacrifice or are in other ways more desirable. Indeed, actions requiring large sacrifice may only be justifiable with reference to far future benefits. The paper is organized as follows. Section 2 briefly reviews the space of GCRs. All actions to reduce the risk must help on one or more of these so as to result in a net risk reduction. The space of GCRs likewise provides an organizing framework for subsequent sections, as summarized in Table  1 . Section 3 discusses the timing of GCRs. For catastrophes that could happen earlier, actions to avoid them will include the earlier benefits of catastrophe avoidance. Almost all GCR reduction actions have near-future GCR reduction benefits. Section 4 discusses co-benefits and mainstreaming of GCR reduction actions. Co-benefits are benefits unrelated to GCR reduction. Mainstreaming is integrating GCR reduction into established Table  1  Summary of global catastrophic risk categories (Section 2), their timing (Section 3), co-benefits and mainstreaming opportunities (Section 4), and high-cost GCR reduction actions that may only be justifiable with reference to far future benefits (Section 5). The co-benefits and mainstreaming opportunities and high-cost actions are illustrative examples, not complete listings. activities. Co-benefits and mainstreaming are both ways to facilitate GCR reduction for those who are not specifically motivated by the far future. Section 5 discusses GCR reduction actions that can only be justified in reference to the farfuture benefits of GCR reduction. While these actions will typically not be the best place to start, they can play an important role in overall GCR reduction efforts. Section 6 discusses the ways in which attention to the far future can inspire additional GCR reduction action. This includes both analytical inspiration and emotional inspiration. Section 7 concludes.
 GCR category
 The global catastrophic risks Which actions can help reduce the risk of global catastrophe depend on what the global catastrophic risks are in the first place. This section briefly overviews the risks. The risks have been described in more detail elsewhere  (Asimov, 1979; Barrett, 2007; Bostrom & C ´irkovic ´, 2008; Guterl, 2012; Jha, 2011; Leslie, 1996; Rees, 2003; Tonn & MacGregor, 2009) . While this section lists GCRs in distinct categories, the risks are often interconnected both within and across categories. For example, the emerging technology GCR of geoengineering is developed in response to the environmental change GCR of climate change, and the geoengineering risk is in turn affected by other GCRs such as large-scale violence or pandemics  (Baum, Maher, & Haqq-Misra, 2013) . Similarly, GCR reduction actions can often affect risks in multiple categories. So while the GCR reduction actions discusses in Sections 3-5 are organized in terms of the categories presented here, it should be understood that specific actions often spill across categories. All this suggests a systems approach to studying GCR.
 Environmental change By environmental change, I mean to refer to human-driven global environmental changes; natural disasters are discussed below. Climate change is perhaps the most commonly cited environmental change GCR; worst-case climate change scenarios attract considerable attention (e.g.  Sherwood & Huber, 2010) . Other environmental change GCRs could include biodiversity loss, biogeochemical flows (interference with the nitrogen and phosphorus cycles), stratospheric ozone depletion, ocean acidification, global fresh water use, land use change, chemical pollution, and atmospheric aerosol loading  (Rockstro ¨m et al., 2009a; Rockstro ¨m et al., 2009b) . While any of these phenomena could dramatically alter the global environment, it is less clear whether the impacts would be catastrophic for humanity  (Baum & Handoh, 2014; Raudsepp-Hearne et al., 2010) . For this paper, it is important that many pro-environmental actions simultaneously help across a broad set of global environmental changes, lessening the need to distinguish which changes could be catastrophic for humanity.
 Emerging technologies Several emerging technologies could cause global catastrophe, including artificial intelligence  (Bostrom, 2014; Eden, Moor, Soraker, & Steinhart, 2013) , biotechnology  (Vogel, 2013) , geoengineering  (Baum et al., 2013; Caldeira, Bala, & Cao, 2013) , and nanotechnology for atomically precise manufacturing  (Drexler, 2013) . These risks are relatively uncertain given the unprecedented nature of emerging technology, but they may constitute a significant portion of total risk.
 Large-scale violence A sufficiently large global war could be catastrophic regardless of the technologies used-for comparison, hundreds of thousands died from attacks with machetes and other unmechanized weapons in the Rwandan genocide. Weapons of mass destruction make the job much easier. Nuclear weapons can be catastrophic both through direct explosions and the indirect effects of nuclear winter  (Mills, Toon, Lee-Taylor, & Robock, 2014) . Biological weapons can also readily cause global catastrophe, in particular if they are contagious; indeed, nonstate actors or even single individuals may be able to cause global catastrophes with engineered contagions  (Nouri & Chyba, 2008; Rees, 2003) . The pressures of conflict can also lead actors to take larger risks, as occurred during World War II when the Americans proceeded with the first nuclear weapon test despite concerns that it could ignite the atmosphere, killing everyone  (Konopinski, Marvin, & Teller, 1946) . Finally, major global violence could also result from an oppressive global totalitarian government  (Caplan, 2008) .
 Pandemics Pandemics can be of natural or artificial origin, or both. Humans catch disease from the environment, in particular from other species. The development and transmission of zoonotic diseases can be enhanced by human activities including wild habitat destruction and factory farming. While it is clear that global pandemics can occur, their exact severity is a matter of ongoing analysis and debate  (Germann, Kadau, Longini, & Macken, 2006; Koblentz, 2009) .
 Natural disasters Global catastrophes can result from several natural disasters including asteroid and comet impacts  (Bucknam & Gold, 2008; Sleep & Zahnle, 1998) , supervolcano eruptions  (Driscoll, Bozzo, Gray, Robock, & Stenchikov, 2012; Rampino, Self, & Stothers, 1988) , solar storms (NRC, 2008), and gamma ray bursts  (Atri, Melott, & Karam, 2013) . While these natural disasters generally have lower probabilities, they nonetheless can be worth some effort to confront. Another natural disaster is the gradual warming of the Sun, which will (with very high probability) make Earth uninhabitable for humanity in a few billion years (O'Malley-James, Cockell, Greaves, & Raven, 2014). Other long-term astronomical risks, such as the Milky Way collision with Andromeda (increasing the rate of dangerous supernovae) and the death of all stars (removing a major energy source) play out on similar or longer time scales  (Adams, 2008) .
 Physics experiments Certain types of physics experiments have raised concerns that the experiments could go wrong, obliterating Earth and its vicinity. This notably includes high-energy particle physics experiments such as at the CERN Large Hadron Collider. Physicists evaluating this risk have argued that the risk is vanishingly small; however, they may be underestimating the risk by neglecting the possibility that their analysis is mistaken  (Ord, Hillerbrand, & Sandberg, 2010) .
 Extraterrestrial encounter It is not presently known if there is any extraterrestrial life, let alone intelligent extraterrestrial civilizations. However, if extraterrestrial civilizations exist, then the result could be catastrophic for humanity  (Baum, Haqq-Misra, & Domagal-Goldman, 2011; Michaud, 2007) . Non-civilization extraterrestrial life could also harm humanity with catastrophic contaminations  (Conley & Rummel, 2008) .
 Unknowns There may be entire categories of GCR not yet identified.
 The timing of the global catastrophic risks If a global catastrophe could occur during the near future, then there will be near-future benefits to reducing the risk. The sooner the catastrophe could occur, the larger the near-future benefits would be. In general, it will be easiest to motivate action to confront the most imminent catastrophes-hence  Posner's (2004)  argument that much GCR-reducing action can be justified even if one only cares about the present generation and the next one to come. It is thus worth examining the timing of the catastrophes.
 Specific global catastrophic risks
 Environmental change Major environmental changes are already visible, with larger changes expected on time scales of decades to tens of millennia. Climate change is among the more long-term of these, with some impacts already visible, and the worst climatic effects contained within the next 25,000 years or so.  3  Another possible long-term environmental change is an oceanic anoxic event, which is caused by phosphorus runoff and would in turn cause major die-off of marine species. An oceanic anoxic event could occur on time scales of millennia  (Handoh & Lenton, 2003) ; more localized effects of phosphorus runoff are already visible.
 Emerging technologies Dangerous biotechnology already exists, and is steadily increasing in capability. Early design work for geoengineering is already underway, with deployments suggested to occur in upcoming decades  (Keith, Parsons, & Morgan, 2010) . Experts give a significant probability to GCR-level artificial intelligence occurring within this century or next  (Baum, Goertzel, & Goertzel, 2011; Mu ¨ller & Bostrom, 2014) . Nanotechnology for atomically precise manufacturing may have similar time horizons.
 Large-scale violence Large-scale violence can happen at any time. The ongoing Ukraine crisis is a firm reminder that significant tensions linger between major nuclear weapons states. Nuclear war could even occur inadvertently, due to false alarm events that can occur at any time  (Barrett, Baum, & Hostetler, 2013) . Risks from biological weapons could increase in upcoming decades as biotechnology advances. However, overall risk from large-scale violence may be gradually declining, following a general trend toward less violence  (Pinker, 2011)  and an increasing sophistication of global peacekeeping capability  (Goldstein, 2011) .
 Pandemics Pandemics can also break out at any time. Recent outbreaks of SARS, H5N1 and H1N1 flus, MERS, and currently Ebola have so far not reached a high degree of global lethality, but they are clear reminders that the threat of pandemics persists. Advances in biotechnology can lead to increasing risk through both intentional use and mishaps, as can increasing global connectivity. On the other hand, advances in public health can reduce the risk.
 Natural disasters Many natural disasters can also occur at any time. Risk from impact events, supervolcano eruptions, solar storms, and gamma ray bursts is roughly constant over long periods of time, into the far future. NASA's near-Earth objects survey has significantly reduced estimates of the risk of large impacts occurring over the next century or so  (Harris, 2008) . Several astronomical risks, including the Sun's gradual warming, the Milky Way collision with Andromeda, and the death of all stars, are GCRs that exists exclusively in the far future.
 Physics experiments The risk from physics experiments depends critically on which physics experiments are conducted. The risk could increase as the capability to conduct experiments increases.
 Extraterrestrial encounter Humanity could encounter extraterrestrials at any time, including through ongoing searches for extraterrestrial intelligence (SETI). Some risk (especially contamination risk) comes mainly from human or robotic travel in space. Some risk comes from messaging to extraterrestrial intelligence (METI;  Haqq-Misra, Busch, Som, & Baum, 2013) . The timing of METI risk depends on the distance from Earth to the location being messaged. METI to sufficiently distant locations is another GCR that exists exclusively in the far future.
 Unknowns Unknown GCRs could occur in both the near and far future. Indeed, more future GCRs are less likely to be already identified.
 Discussion Relatively few identified GCRs exist exclusively in the far future: certain astronomical risks and METI to distant locations. For all other GCRs-and this constitutes almost all of the total identifiable risk-the catastrophes could occur in the near future. The identified risks from emerging technologies and physics experiments could only occur in the near future. The identified risks from environmental change, large-scale violence, pandemics, some natural disasters, some extraterrestrial encounter risks, and unknowns could occur in the near or far future. The preponderance of near future risks suggests that a lot of actions to reduce these risks can be done without reference to their far future benefits. On the other hand, these near future benefits may not always be enough, especially when the catastrophes would occur several decades or centuries later, as people often care little about even these earlier times. It is thus worth pursuing other means of motivating GCR reduction.
 Co-benefits and mainstreaming GCR-reducing actions Insight on how to motivate GCR reduction can be found from outside the core GCR literature, in some related literatures. The climate change mitigation community has developed the concept of co-benefits, defined as benefits besides the target goal  (Hosking, Mudu, & Dora, 2011; Miyatsuka & Zusman, n.d.) . For climate change mitigation, the target goal is greenhouse gas emissions reductions. Research assesses how communities can reduce their emissions while improving their economic development, public health, and wellbeing. The co-benefits concept readily applies GCR. Some actions to reduce GCR will also be profitable, fun, healthy, satisfying, safe, or otherwise desirable, often to those who perform the actions. Similarly, the natural disaster management community has a robust practice of mainstreaming disaster management into established goals and procedures, especially those regarding development  (Benson, 2009; Twigg & Steiner, 2002) . Disaster management actions will often only be taken when they can be integrated into established goals and procedures; otherwise, the actions will be too impractical or undesirable. For example, urban design steps to reduce a town's vulnerability to hurricane storm surge can be mainstreamed into the town's broader urban planning processes  (Frazier, Wood, & Yarnal, 2010) . GCR reduction actions can likewise also be mainstreamed into whatever people are already doing or trying to do. The GCR reduction community would be wise to adopt the approaches of co-benefits and mainstreaming. Doing so requires an understanding of the co-benefits that can come from various GCR-reducing actions and the relevant established goals and procedures. For many of these actions-in particular those with sufficiently large co-benefits and wellestablished goals and procedures-reducing GCR can be a nice ancillary benefit of actions that might as well be taken anyway. For these actions, no concern for the far future is needed; often, no concern beyond one's immediate community is needed. These actions require the least sacrifice (indeed, it is a sacrifice not to take these actions) and likewise will often be the easiest actions to promote. This begs the question of which GCR-reducing actions have significant co-benefits and mainstreaming opportunities.
 . Environmental change Environmental change is largely driven by a wide variety of basic activities, including food consumption, transport, real estate development, and natural resource usage. More environmentally friendly actions can often be justified for nonenvironmental reasons. A recent study by McKinsey  (Enkvist, Naucle ´r, & Rosander, 2007)  found that many greenhouse gas emission reductions would result in net monetary benefits for those who reduce these emissions, especially in the realm of energy efficiency in buildings and transport. Happiness research has found that people rate their daily commute as being among their least happy activities  (Layard, 2003) . Public health research links high-meat diets with obesity and other health problems  (Pan et al., 2012) . Buildings, transport, and food are meanwhile three of the most environmentally important sectors  (Metz et al., 2007; Steinfeld et al., 2006; USEIA, 2011) . If significant changes in these sectors can be achieved for nonenvironmental reasons, the environmental benefit could be quite large.
 Emerging technologies It is often beneficial to develop regulations for multiple technologies at the same time, due to similarities between the technologies and the regulations  (Kuzma & Priest, 2010; Wilson, 2013a) . Concerns about other technologies can thus motivate general technology regulation, which provides a framework for mainstreaming the regulation of emerging technologies GCRs. In addition, some actions specific to certain emerging technologies can have co-benefits. For example, one proposed solution to artificial intelligence risk is to design the AI to be ''friendly'' to humanity. In addition to not causing a catastrophe, such an AI could help with other societal problems  (Muehlhauser & Bostrom, 2014) . If such an AI can be achieved with sufficient confidence, then this could be an attractive action even for those who are not concerned about AI risk.
 Large-scale violence Achieving peace avoids violence at all scales and also brings a variety of co-benefits. One co-benefit is economic growththe so-called ''peace dividend''  (Knight, Loayza, & Villanueva, 1996; Ward & Davis, 1992) . Another co-benefit is psychological. Recent research finds that conflict is often driven by humiliation, and likewise that giving people a sense of dignity can help  (Lindner, 2006; Stern, 2003) . Finally, other research suggests that reducing domestic violence against women could lead to less interstate war  (Hudson, Ballif-Spanvill, Caprioli, & Emmett, 2012) . Emphasizing these co-benefits could justify much action to reduce large-scale violence. Another worthy point of focus is violent nonstate actors, which continue to receive extensive attention in the wake of the 9-11 attacks. While nonstate actors may not be able to cause violence large enough to result in global catastrophe, 4 actions to confront them may have co-benefits and mainstreaming opportunities for large-scale violence. For example, the annual Nuclear Security Summits initiated by US President Barack Obama aim to prevent nonstate actors from acquiring nuclear weapons, but they also strengthen norms against nuclear weapon use more generally.
 Pandemics As noted above, there is some debate about how severe pandemics could be, including whether they would impact the far future. To a large extent, this debate is irrelevant. Regardless of how severe pandemics would be, there already exists a significant global public health infrastructure that responds to pandemics of all sizes. Improving this infrastructure can further improve the response. The case for improving this infrastructure is strengthened by the possibility of catastrophic pandemics, but the case is not dependent on this possibility  (McKibbin & Sidorenko, 2006) .
 Natural disasters The GCR literature has proposed certain measures to increase society's resilience to a wide range of global catastrophes, including natural disasters. These measures include food stockpiles , underground refuges  (Jebari, 2014) , and space colonies and refuges  (Abrams et al., 2007; Shapiro, 2009) . These measures also tend to increase society's resilience to smaller catastrophes. Indeed, many actions taken to prepare for smaller catastrophes also benefit GCR reduction. In addition, while space colonies and refuges have been criticized for their high cost relative to other means of reducing global catastrophic risk  (Baum, 2009; Sandberg, Matheny, & C ´irkovic ´, 2008) , some space missions are already underway or in planning for a variety of other reasons, including science, political prestige, and economic opportunity (e.g. in asteroid mining). Space colonies or refuges could be mainstreamed into these missions  (Baum, Denkenberger, & Haqq-Misra, 2014) .
 Physics experiments Physics experiments are a curious case, because the relevant experiments are quite expensive (hundreds of millions to billions of dollars) and the social benefits somewhat limited. As  Parson (2007, p. 155)  puts it, ''this research is remote from practical application and serves largely to indulge national pride and the intellectual passion of a tiny elite group''. Arguably, the co-benefits of reducing physics experiment risk include the money saved by not doing the experiments, and by tasking the money to a worthier cause, analogous to the peace dividend, though this is likely to be a controversial view among those who value the physics experiments.
 Extraterrestrial encounter Protection against extraterrestrial contamination has the co-benefit of protecting extraterrestrial environments from contamination by humans, which is of significant scientific value  (Conley & Rummel, 2008) . The costs of SETI and METI are small relative to the big physics experiments, so while there are dollar savings to realize from skipping them, these are less of an issue. Perhaps the extraterrestrial-risk-reducing action with the most co-benefits would be research and public education into what the risks could be. Discussions of ETI are very popular, as seen in the extensive popular media and entertainment attention to ETI.
 Unknowns Actions likely to reduce unknown GCRs will typically be generic actions that also help reduce other GCRs or even smaller risks, such as building refuges  (Jebari, 2014)  and stockpiling resources .
 Discussion Many GCR-reducing actions, covering the full breadth of GCRs, have sizable co-benefits, and can also be mainstreamed into existing activities. Many of these actions will often be desirable even without reference to GCR, let alone to the far future benefits of GCR reduction. These ''easy'' actions will typically be the lowest hanging fruit, the easiest GCR reductions to promote. They offer a sensible starting point for those seeking to reduce GCR.
 Actions with significant cost Ideally, all GCR-reducing actions would have low costs and large co-benefits, such that it would be easy to persuade people to take the actions, and such that the totality of GCR could be reduced with minimal burden to those taking the actions and to society at large. As discussed above, many such actions exist. However, this is not the case for all GCR-reducing actions. Some of these other actions require considerable sacrifice, especially the most aggressive GCR-reduction efforts. Tonn and Stiefel's (2014) levels of societal actions are instructive here. The levels range from doing nothing to an extreme war footing in which society is organized specifically to reduce GCR. Actions requiring more sacrifice, especially those at or near the level of extreme war footing, might only be justifiable with reference to the far-future benefits. While these actions will typically not be the lowest hanging fruit, they could be important components of an overall portfolio of GCR-reducing actions.
 Specific global catastrophic risks
 Environmental change The most aggressive pro-environmental actions include public policies like a high carbon tax, personal behaviors requiring great inconvenience and sacrifice, and restructuring the entire global industrial economy away from fossil fuels and other pollutants. To achieve a larger reduction in environmental change GCR, some of these more aggressive actions may be needed. That these more aggressive actions may only be justifiable with reference to far-future benefits is a core point from debates about discounting in environmental policy (Nordhaus, 2008).
 Emerging technologies One way to reduce emerging technology GCR is to simply abstain from developing the technologies, i.e., to relinquish them  (Joy, 2000) . However, if these technologies do not cause catastrophe, they sometimes come with great benefits: geoengineering can avoid the worst effects of climate change; AI can solve a variety of social problems; biotechnology can help cure disease. Thus relinquishing the technologies can require a large sacrifice  (Baum, 2014) . This sacrifice may sometimes only be justifiable given the far-future benefits of GCR reduction.
 Large-scale violence While nuclear weapons can cause great harm, they are also often attributed with helping maintain peace, through the doctrine of nuclear deterrence: countries hesitate to attack each other for fear of being destroyed in nuclear retaliation. There are questions about the efficacy of nuclear deterrence  (Wilson, 2013b)  and there are proposals to achieve deterrence with out large nuclear arsenals  (Baum, 2015) . However, a common view posits that nuclear deterrence is necessary until international relations are peaceful enough for a world without nuclear weapons  (Obama, 2009) . Following this logic, immediate nuclear disarmament might reduce GCR, but it might also increase the preponderance of smaller conflicts and other geopolitical instabilities. Depending on the details, immediate nuclear disarmament might only be justifiable with reference to the far future.
 Pandemics One aggressive action to reduce pandemics risk would be aggressive quarantine, such as blockading the major islands of Indonesia, Japan, the United Kingdom, and other countries. Travel restrictions could keep populations in these places safe. During a sufficiently severe outbreak, populations in these places could even request to be blockaded. A safer but costlier and less desirable policy would blockade them at first alert of a possible outbreak, or even keep them blockaded on a permanent basis. Doing so might lower GCR, but might only be justifiable with reference to the far future.
 Natural disasters One proposed aggressive action for natural disasters could be to drill the ground around potential supervolcanoes to extract the heat, although the technological feasibility of this proposal has not yet been established.  5  This could be a very costly project, but, if it works, it could also reduce supervolcanoes GCR. The project would come with a co-benefit of geothermal energy, but this is likely not nearly enough to justify the expense. Another possibility is advanced surfaceindependent refuges, which could protect against a variety of GCRs, including many of the natural disasters, but again could come at great expense  Beckstead, 2014) .
 Physics experiments and extraterrestrial encounter I am not aware of any actions to reduce near-future GCR from physics experiments and extraterrestrial encounter that have significant cost and can only be justified with reference to far-future benefits. To the contrary, many actions to reduce these risks save money (Section 4.1). Protection against contamination does have a cost, and shutting METI programs down could cost the public a source of popular entertainment. On the other hand, the shut down itself could also create an entertaining controversy. Regardless, the costs involved are not large.
 Unknowns One action that might only be justifiable with reference to far future benefits is a far-future version of the ''extraterrestrial time capsule'' proposed in . These capsules contain artifacts of benefit to catastrophe survivors for a range of known and unknown catastrophe scenarios. The capsules are launched into space in trajectories designed to return to Earth at some future date.  suggest a return date 100 years into the future, but it may be possible (and expensive) to have return dates in the far future.
 Discussion For those who wish to keep humanity highly safe from catastrophe, there are actions that can only be justified with reference to the far-future benefits of GCR reduction. While these actions are typically not the best place to start, they can offer additional GCR reductions beyond what the easier actions offer. Given the enormous far-future benefits of GCR reduction, arguably these actions merit consideration. However, hopefully GCR can be essentially eliminated without resorting to these actions. If these actions are necessary, it will likewise be necessary to appeal to the importance of the far future.
 Far future as inspiration The paper thus far has focused on how to avoid appeals to the far future argument, in recognition of the fact that many people are not motivated by what will benefit the far future. But some GCR reduction actions can only be justified with reference to far future benefits. Additionally, some people are motivated to benefit the far future. Other people could be too. Tapping the inspirational power of the far future can enable more GCR reduction. There are at least two ways that the far future can inspire action: analytical and emotional. Both are consistent with the far future argument, but the argument is typically inspired by analytical considerations. The analytical inspiration is found in works analyzing how to maximize the good or achieve related objectives. Most of the scholarly works invoking the far future argument are of this sort.  6  Such ideas have the potential to resonate not just with other scholars, but with people in other professions as well, and also the lay public. Thus there can be some value to disseminating analysis about the importance of the far future and its relation to GCR. Analytical inspiration can also come from analyzing specific actions in terms of their far-future importance. Such analysis can help promote these actions, even if the actions could be justified without reference to the far future. However, the analysis should be careful to connect with actual decision makers, and not just evaluate hypothetically optimal actions that no one ever takes. For example, there has been now multiple decades of research analyzing what the optimal carbon tax should be (for an early work, see  Nordhaus, 1992 ), yet throughout this period, for most of the world, the actual carbon tax has been zero. Analytical inspiration has its limits. Research effort may be more productively spent on what policies and other actions people are actually willing to implement. The other far future inspiration is emotional. The destruction of human civilization can itself be a wrenching emotional idea. In The Fate of the Earth, Jonathan Schell writes ''The thought of cutting off life's flow, of amputating this future, is so shocking, so alien to nature, and so contradictory to life's impulse that we can scarcely entertain it before turning away in revulsion and disbelief''  (Schell, 1982 (Schell, /2000 ). In addition, there is a certain beauty to the idea of helping shape the entire arch of the narrative of humanity, or even the universe itself. People often find a sense of purpose and meaning in contributing to something bigger than themselves-and it does not get any bigger than this. Carl  Sagan's (1994)  Pale Blue Dot and James  Martin's (2007)  The Meaning of the 21st Century both capture this well, painting vivid pictures of the special place of humanity in the universe and the special opportunities people today have to make a difference of potentially cosmic significance. This perspective says that humanity faces great challenges. It says that if these challenges are successfully met, then humanity can go on to some amazing achievements. It is a worthy perspective for integrating the far future into our lives, not just for our day-to-day actions but also for how we understand ourselves as human beings alive today. This may be worth something in its own right, but it can also have a practical value in motivating additional actions to confront catastrophic threats to humanity.
 Conclusion The far future argument is sound. The goal of helping the far future is a very worthy one, and helping the far future often means helping reduce the risk of those global catastrophes that could diminish the far-future success of human civilization. However, in practical terms, reducing this risk will not always require attention to its far-future significance. This is important because many people are not motivated to help the far future, but they could nonetheless be motivated to take actions that reduce GCR and in turn help the far future. They may do this because the actions reduce the risk of near-future GCRs, or because the actions have co-benefits unrelated to GCRs and can be mainstreamed into established activities. This paper surveys GCRs and GCR-reducing actions in terms of how much these actions require support for the far future argument for confronting catastrophic threats to humanity. The analysis suggests that a large portion of total GCR, probably a large majority, can be reduced without reference to the far future and with reference to what people already care about, be it the near future or even more parochial concerns. These actions will often be the best to promote, achieving the largest GCR reduction relative to effort spent. On the other hand, some significant GCR reducing actions (especially those requiring large sacrifice) can only be justified with reference to their far-future benefits. For these actions in particular, it is important to emphasize how the far future can inspire action. Several priorities for future research are apparent. Quantitative GCR analysis could help identify which actions best reduce GCR and also what portion of GCR can be reduced without reference to the far future. Analysis covering the breadth of GCRs would be especially helpful. Social scientific research could study how to effectively engage stakeholders so as to leverage co-benefits and mainstream GCR reductions into existing programs. Social scientific research could also examine how to effectively tap the inspirational power of the far future, especially for emotional inspiration, which has received limited prior attention. Progress in these research areas could go a long way toward identifying how to, in practice, achieve large GCR reductions. The overall message of this paper is that helping the far future requires attention to which specific actions can help the far future and likewise to what can motivate these actions. The actions are not necessarily motivated by their far-future impact. This is fine. The far future does not care why people acted to help it-the far future only cares that it was helped. And people taking these actions will rarely mind that their actions also help the far future. Most people will probably view this as at least a nice ancillary benefit. Additionally, people will appreciate that those promoting the far future have taken the courtesy to consider what they care about and fit the far future into that. It can be disrespectful and counterproductive to expect people to drop everything they are doing just because some research concluded that the far future is more important. This means that those who seek to promote actions to benefit the far future must engage on an interpersonal level with the people who will take these actions, to understand what these people care about and how far-future-benefiting actions can fit in. This is an important task to pursue, given the enormity of what human civilization can accomplish from now into the far future. Futures
			 The 25,000 year figure is derived from Archer and Ganopolski (2005) , Fig.3C, which shows a rapid temperature spike that declines most of the way back to current temperatures within 25,000 years and then remains at similar temperatures for another 500,000 years. However, this refers specifically to climatic effects; the human effects could persist longer, especially if the climate change causes a civilization-ending global catastrophe. S.D.Baum / Futures 72 (2015) 86-96
			 Nuclear terrorism would likely be too small to cause a far-future-impacting global catastrophe, unless it catalyzed a large-scale interstate nuclear war (Ayson, 2010) . Biological terrorism could more readily cause a global catastrophe, as discussed above.S.D.Baum / Futures 72 (2015) 86-96
			 S.D.Baum / Futures 72 (2015) 86-96
			 An idea to this effect is briefly discussed inLeggett (2006, p. 794). 6 See citations in Footnote 1. S.D. Baum / Futures 72 (2015) 86-96


Abstract: In some situations a number of agents each have the ability to undertake an initiative that would have significant effects on the others. Suppose that each of these agents is purely motivated by an altruistic concern for the common good. We show that if each agent acts on her own personal judgment as to whether the initiative should be undertaken, then the initiative will be undertaken more often than is optimal. We suggest that this phenomenon, which we call the unilateralist's curse, arises in many contexts, including some that are important for public policy. To lift the curse, we propose a principle of conformity, which would discourage unilateralist action. We consider three different models for how this principle could be implemented, and respond to an objection that could be raised against it.

Nick Bostrom
Thomas Douglas
Anders Sandberg
James Martin

The Unilateralist's Curse and the Case for a Principle of Conformity

Introduction Consider the following hypothetical scenarios: (1) A group of scientists working on the development of an HIV vaccine has accidentally created an air-transmissible variant of HIV. The scientists must decide whether to publish their discovery, knowing that it might be used to create a devastating biological weapon, but also that it could help those who hope to develop defenses against such weapons. Most members of the group think publication is too risky, but one disagrees. He mentions the discovery at a conference, and soon the details are widely known. (2) A sports team is planning a surprise birthday party for its coach. One of the players decides that it would be more fun to tell the coach in advance about the planned event. Although the other players think it would be better to keep it a surprise, the unilateralist lets word slip about the preparations underway. (3) Geoengineering techniques have developed to the point that it is possible for any of the world's twenty most technologically advanced nations to substantially reduce the earth's average temperature by emitting sulfate aerosols. Each of these nations separately considers whether to release such aerosols. Nineteen decide against, but one nation estimates that the benefits of lowering temperature would exceed the costs. It presses ahead with its sulfate aerosol program and the global average temperature drops by almost 1˚. In each of these cases, each of a number of agents is in a position to undertake an initiative, X. Suppose that each agent decides whether or not to undertake X on the basis of her own independent judgment of the value of X, where the value of X is assumed to be independent of who undertakes X, and is supposed to be determined by the contribution of X to the common good. 1 Each agent's judgment is subject to error-some agents might overestimate the value of X, others might underestimate it. If the true value of X is negative, then the larger the number of agents, the greater the chances that at least one agent will overestimate X sufficiently to make the value of X seem positive. Thus, if agents act unilaterally, the initiative is too likely to be undertaken, and if such scenarios repeat, an excessively large number of initiatives are likely to be undertaken. We shall call this phenomenon the unilateralist's curse. Though we have chosen to introduce the unilateralist's curse with hypothetical examples, it is not merely a hypothetical problem. There are numerous historical examples, ranging from the mundane to the high-tech. Here is one: Until the late 1970s, the mechanism of the hydrogen bomb was one of the world's best kept scientific secrets: it is thought that only four governments were in possession of it, each having decided not to divulge it. But staff at the Progressive magazine believed that nuclear secrecy was fuelling the Cold War by enabling nuclear policy to be determined by a security elite without proper public scrutiny. They pieced together the mechanism of the bomb and published it in their magazine, arguing that the cost, in the form of aiding countries such as India, Pakistan and South Africa in acquiring hydrogen bombs, was outweighed by the benefits of undermining nuclear secrecy.  2  Another possible example from atomic physics had occurred several decades earlier: In 1939 the Polish nuclear physicist Joseph Rotblat noticed that the fission of uranium released more neutrons than used to trigger it, realizing that it could produce a chain reaction leading to an explosion of unprecedented power. He assumed that other scientists elsewhere were doing similar experiments, and were thus in a position to release similar information, an assumption that turned out to be correct. Initially, Rotblat vowed to tell no-one of his discovery, believing it to be a threat to mankind, and it is plausible that others did likewise, for similar reasons. However, when the war broke out, Rotblat decided that releasing the information was now in the public interest, given the likelihood that the Germans were working on an atomic bomb. He confided in colleagues and thus unilaterally triggered the United Kingdom's atomic bomb project.  3  Rotblat was later to leave the Manhattan Project, coming to the view that his had overestimated the German nuclear threat, and underestimated the likelihood that the US would use an atomic bomb offensively. It is perhaps too soon to say whether these unilateral actions were suboptimal. But in other cases, it is clearer that unilateral action led to a suboptimal outcome: In the mid-nineteenth century there were virtually no wild rabbits in Australia, though many were in a position to introduce them. In 1859, Thomas Austin, a wealthy grazier, took it upon himself to do so. He had a dozen or two European rabbits imported from England and is reported to have said that "The introduction of a few rabbits could do little harm and might provide a touch of home, in addition to a spot of hunting." 4 However, the rabbit population grew dramatically, and rabbits quickly became Australia's most reviled pests, destroying large swathes of agricultural land.  5  The abovementioned examples were isolated incidents, but similar situations occur regularly in some spheres of activity, for instance, in the media: Media outlets sometimes find themselves in the situation that journalists have access to information that is of public interest but could also harm specific individuals or institutions: the name of a not-yet charged murder suspect (publication may bias legal proceedings), the news that a celebrity committed suicide (publication may risk copycat suicides), or sensitive government documents such as those leaked by Wikileaks and Edward Snowden (publication may endanger national security). It is enough that one outlet decides that the public interest outweighs the risk for the information to be released. Thus, the more journalists have access to the information the more likely it is to be published. Unilateralist situations also regularly crop up in regards to new biotechnologies: Gene drives, a technique for inducing altered genes to be inherited by nearly all offspring (rather than just 50%) of a genetically modified organism, have potential for spreading altered genes across a population, enabling ecological control (e.g. making mosquitos incapable of spreading malaria or reducing herbicide resistance) but also potentially creating worrisome risks (e.g. to genetic diversity or of sabotage). Here unilateral action could both be taken in releasing a particular altered organism into the environment, and in releasing the information about how to produce it in the first place. There is scientific disagreement on the utility and risk of both. 6
 The Unilateralist's Curse: A Model The unilateralist's curse is closely related to a problem in auction theory known as the winner's curse. The winner's curse is the phenomenon that the winning bid in an auction has a high likelihood of being higher than the actual value of the good sold.  7  Each bidder makes an independent estimate and the bidder with the highest estimate outbids the others. But if the average estimate is likely to be an accurate estimate of the value, then the winner overpays. The larger the number of bidders, the more likely it is that at least one of them has overestimated the value. The unilateralist's curse and the winner's curse have the same basic structure. The difference between them lies in the goals of the agents and the nature of the decision. In the winner's curse, each agent aims to make a purchase if and only if doing so will be valuable for her. In the unilateralist's curse, the decision-maker chooses whether to undertake an initiative with an eye to the common good, that is, seeking to undertake the initiative if and only if the initiative contributes positively to the common good. The unilateralist's curse can be illustrated using a simple mathematical model. Assume N agents, each considering whether to undertake an initiative. Each agent wishes to proceed if and only if the value of the initiative is positive, but the agents do not know the true value V * of the initiative (which may be negative or positive). Instead each agent forms an estimate that is the sum of V * and a random independent error d drawn from a distribution with cumulative distribution function F(d). This means that the probability p that any given agent will estimate the value of the initiative to be positive when it is in fact negative (V * < 0) is p = 1 − F(−V * ).  8  The probability P that at least one of the agents will incorrectly estimate the value to be positive is p = 1 − (1 − p) N = 1 − F(−V * ) N . For the case with 5 agents and d as a random error drawn from a normal distribution with standard deviation 1 and mean zero, the probability that any initiative will be undertaken (regardless of whether it is a good idea or not) is high even when the true value is quite negative, and the probability rises steeply as the true value of the initiative approaches zero from below (Figure  1 ). For mildly negative values of the initiative there is nearly always someone who misjudges the value of the initiative and undertakes it. There is no problem for positive initiatives since even if one or two agents are overly cautious, it is very likely that somebody will undertake the initiative, which is the optimal result (Figure  2 ). Increasing the number of agents capable of undertaking the initiative also exacerbates the problem: as N grows, the likelihood of someone proceeding incorrectly increases monotonically towards 1.  9  The magnitude of this effect can be quite large even for a relatively small number of agents. For example, with the same error assumptions as above, if the true value of the initiative V * = −1 (the initiative is undesirable), then the probability of erroneously undertaking the initiative grows rapidly with N, passing 50% for just four agents (Figure  3 ). There are six features of the unilateralist's curse that that need to be emphasized. First, in cases where the curse arises, the risk of erroneously undertaking an initiative is not caused by self-interest. In the model, all agents act for the common Figure  1  The probability of an initiative being undertaken as a function of the actual value, V * , for five agents and assuming normally distributed errors with variance 1 (these assumptions will be used in all subsequent figures except when otherwise noted). Note that 50% probability of action occurs near a value of −1: a strong unilateralist bias exists. Figure  2  The expected payoff for naive agents (who act if and only if their evaluation of the initiative is positive) and ideal omniscient estimators who are assumed to know the true value. good, they simply disagree about the contribution of the initiative to the common good.  10  Second, though the curse could be described as a group-level bias in favor of undertaking initiatives, in does not arise from biases in the individual estimates of the value that would result from undertaking the initiative. The model above assumes symmetric random errors in the estimates of the true value.  11  Third, there is a sense in which the unilateralist's curse is the obverse of Condorcet's jury theorem.  12  The jury theorem states that the average estimate of a group of people with above 50% likelihood of guessing correctly and with uncorrelated errors will tend to be close to the correct value, and will tend to move closer to the true value as the size of the group increases. But what is also true, and relevant to the argument in this paper, is that the highest estimate will tend to be above the true value, and the expected overestimation of this highest estimate increases with the size of the group. In the cases we are interested in here, it is the highest estimate that will determine whether an initiative is undertaken, not the average estimate. Fourth, though we have chosen to illustrate the curse using initiatives that are (probably) irreversible, the problem can arise in other cases too. The problem becomes sharper if the initiative is irreversible, but even for actions that can be undone the problem remains in a milder form. Resources will be wasted on undoing erroneous initiatives, and if the bad consequences are not obvious they might occur before the problem is noticed. There might even be a costly tug-o-war between disagreeing agents. Finally, fifth, though we have thus far focused on cases where a number of agents can undertake an initiative and it matters only whether at least one of them does so, a similar problem arises when any one of a group of agents can spoil an initiative-for instance, where universal action is required to bring about an intended outcome. Consider the following example: In Norse mythology, the goddess Hel of the underworld promised to release the universally beloved god Baldr if all objects, alive and dead, would shed a tear for him. All did, except the giantess Þo ¨kk. The god was forced to remain in the underworld.  13  Similar situations can arise when all the actors in a play must come together in order for a rehearsal to take place, when all members of committee must attend a meeting in order for it to be quorate, or when all signatories to an international treaty must ratify it in order for it to come into effect. The United Nations Security Council frequently provides examples of unilateral spoiling. The five permanent members of the Council-currently China, France, Russia, the United Kingdom and the United States-each possesses the power to veto the adoption of any non-procedural resolution. In the early years of the Council, this veto power was frequently employed by the Soviet Union to block applications for new membership of the United Nations. More recently, it has been used by the United States to block resolutions criticizing Israel, and by Russia and China to block resolutions on the Syria conflict. 14 While some of these vetoes presumably reflect differences in the national interests of the council members, others may reflect different estimations of the contribution that a resolution would make to the common good. Certainly, considerations relating to the common good are often invoked in their defence. For instance, the United States' 2011 veto of a draft resolution condemning Israeli settlements in Palestinian territory was defended on the grounds that the resolution would be an impediment to peace talks.  15  These cases of unilateral spoiling or abstinence are formally equivalent to the original unilateralist curse, with merely the sign reversed. Since the problem in these cases is the result of unilateral abstinence, it seems appropriate to include them within the scope of the unilateralist's curse. Thus, in what follows, we assume that the unilateralist's curse can arise when each member of a group can unilaterally undertake or spoil an initiative (though for ease of exposition we sometimes mention only the former case).
 Lifting the Curse Let a unilateralist situation be one in which each member of a group of agents can undertake or spoil an initiative regardless of the cooperation or opposition of other members of the group. We will say that a policy would lift the unilateralist's curse if universal adherence to it by all agents in unilateralist situations should be expected (ex ante) to eliminate any surfeit or deficit of initiatives that the unilateralist's curse might otherwise produce.
 The Principle of Conformity When acting out of concern for the common good in a unilateralist situation, reduce your likelihood of unilaterally undertaking or spoiling the initiative to a level that ex ante would be expected to lift the curse. In the following subsections we will explore various ways in which one might bring oneself into compliance with this principle.  16  These can be organized around three models: collective deliberation, epistemic deference, and moral deference. The three models are applicable in somewhat different circumstances, and their suitability might depend on the type of agents involved. It should be noted that, though some of the methods discussed below do not require agents to be aware of the nature of the situation, most hinge on agents recognizing that they are in an unilateralist situation. However, this is not to say that agents must be able to identify the other parties to the unilateralist situation: this is necessary for some but not all of our proposed solutions.
 The Collective Deliberation Model A first line of defense against the unilateralist's curse could be to share data and reasoning between agents in the hope that this will resolve their disagreement about the desirability of proceeding with the contested initiative. In some cases, however, extensive information sharing among all potential decision-making agents is impractical. Communication is often costly and timeconsuming, and participants in a unilateralist situation may not be able to identify one another. Furthermore, in certain cases information disclosure might itself be the initiative whose desirability is in dispute, such as when information hazards are associated with disseminating relevant data.  17  Moreover, even when information is fully shared, a consensus can remain elusive. Disagreements about the net value of undertaking some project often persist after decision-makers have been thoroughly briefed on all obviously relevant and easily communicable facts and after having had opportunities to engage in joint deliberation. Because complete information sharing may not be practical or desirable, and because it may not produce consensus when it does occur, the principle of conformity requires us to explore additional models for lifting the unilateralist's curse.
 The Meta-rationality Model One approach would be to appeal to each agent's reflective rationality. A party to an epistemic disagreement should ideally reflect on the fallibility of their own judgment and adjust their posterior probability to take into account the fact that other agents have different opinions. Robert Aumann has shown that rational Bayesian agents with identical priors and common knowledge of each other's posteriors (and of each other's rationality) must have identical posterior probabilities.  18  Disagreement between such agents is impossible. This sounds like good news: if all agents make the same estimate of the benefits of action, the unilateralist curse is lifted. There is, however, some skepticism about the relevance of Aumann's result for practical cases of disagreement.  19  The assumption of identical priors, in particular, is problematic.  20  Furthermore, the same challenges that can make data sharing difficult can also make it difficult to make each agent's honest posterior probability estimates of the value of the initiative common knowledge among all agents. It turns out, however, that sufficiently rational agents can manage the curse even without communication. In the literature on the winner's curse it has been argued that rational expected utility-maximizing will not be affected by it.  21  Rational agents will take the winner's curse into account and adjust their bids accordingly. This is known as bid shading. Rational agents place bids that are lower than their ex ante expectation of the value of the good, but equal to their expectation of the value of the good conditional upon them winning the auction. The counterpart of this response would be for agents in a unilateralist situation to estimate the value of the initiative conditional on the agent's first-order estimate of the initiative's value being the highest (or, in spoiler cases, the lowest). In other words, on finding themselves in a unilateralist situation, each rational agent will initially estimate the value of the initiative based on his prior probability distribution. He will then take into account the case where his decision is decisive. In the case where agents can unilaterally undertake an initiative, the agent will condition on the situation in which he is the most sanguine and everybody else thinks the action should not be done. (In spoiler cases, the agent conditions on the situation in which he is the most pessimistic and everybody else thinks the initiative should be undertaken.) He then creates a posterior distribution of value that is used to make an adjusted decision. PðV Ã jwinÞ ¼ PðwinjV Ã ÞPðV Ã Þ PðwinÞ where "win" represents being the deciding agent. Note that this typically requires knowing or estimating the number of other agents.
 Example In the simple case where the agent assumes all other agents have the same priors and are acting independently, only differing in the noisy data about V * they have received: PðwinjV Ã Þ ¼ Z 1 À1 PðV À V Ã ÞFðV À V Ã Þ NÀ1 dV where F(V) is the cumulative distribution function of the errors. The posterior distribution of V * becomes: PðV Ã jwinÞ ¼ KPðV Ã Þ Z 1 À1 PðV À V Ã ÞFðV À V Ã Þ NÀ1 dV where K is a normalization constant. The posterior action should then be based on the expectation E(V * |win). If the agents choose to act when the received data is above a fixed threshold T, V * is normally distributed with zero mean and variance 1, and they get estimates of V * with normal noise (again with mean zero and variance 1), then the optimal threshold is the one that maximizes the expected value (Figure  4 ): T opt ðNÞ ¼ argmax T Z 1 À1 VPðV Þ 1 À 1 À F V À T ð Þ ð Þ N dV T opt (N) increases rapidly with N, reaching 0.54 for two agents and 1 for 4 agents: even for a small group it is rational to be far more cautious than in the single agent case. Note that in this case all agents are aware of the prior distribution, noise distribution, independence, and that the other agents are using this strategy (Figure  5 ).  22  Figure  4  The optimal threshold T opt (N) for action as a function of the number of agents. Agents who only act if the perceived value of the initiative is higher than T opt (N) will maximize their expected (joint) result. One might raise questions about the practical applicability of this sophisticated Bayesian approach, however. Even if rational Bayesian agents would agree, humans are at best approximations of rational Bayesian agents and they have far more limited mental computation power-even when leaving out biasing factors.  23  Value in practical cases is also seldom in the form of easily manipulable and comparable scalar quantities. Hence implementing the sophisticated Bayesian approach to lifting the unilateralist's curse might typically be infeasible.  24
 The Moral Deference Model Suppose a unilateralist situation exists and that it is not feasible for all agents to lift the curse through communication and adjustment of beliefs. It might nevertheless be possible for the group to lift the curse if each agent complies with a moral norm which reduces the likelihood that he acts unilaterally, for example, by assigning decision-making authority to the group as a whole or to one individual within it. We call this the moral deference model. In contrast to the two models presented above, the moral deference model does not require agents to defer to the group in forming their beliefs regarding the value of the initiative. However, it does require them to defer to the group in deciding whether to act on those beliefs. A slogan for this approach could be "comply in action, defy in thought." There are many norms such that universal compliance with the norm by a group of agents would lift the unilateralist's curse. For example, a norm that assigned decision-making authority to an arbitrary member of the group would lift it. Consider the norm: when in a unilateralist situation, if you are the tallest person able to undertake the initiative, then undertake it if and only if you believe its value exceeds zero; if you are not the tallest person able to undertake the initiative, do not undertake it. Universal compliance with this norm would prevent the unilateralist's curse from arising in the sense that, in the absence of any bias towards or against action in the individual members of the group (and thus in the group's tallest member), this norm will produce no group-level bias towards or against the initiative.  25  The payoffs associated with this tallest-decides norm in a five-agent situation are depicted in Figure  6  below. The tallest-decides norm, however, has several epistemically and pragmatically unattractive features. For example, it does not protect against biases or errors that might impair the judgment of the group's tallest member. Furthermore, it is very unlikely that such a norm would gain wide acceptance. Fortunately, there are other norms that could lift the curse and may lack these unattractive features. One norm would recommend that agents conform to the rules of existing institutions that militate against unilateral action: (1) When in a unilateralist's situation, defer to existing institutions, such as laws or customs, if universal deference to those institutions would lift the unilateralist's curse. National and international laws often militate against the unilateralist's curse, for example by specifying that decisions must be made democratically or by individuals or institutions that have been given special authority over a particular realm of decision-making. In other cases, there are informal conventions that may do the job. For example, following the publication early last decade of two studies thought by some to aid bioweapons development,  26  a group of scientific journals agreed to introduce screening procedures to identify papers containing information that is especially prone to misuse and to seek external advice on the publication of such papers.  27  Though these procedures lacked legal status, compliance with them by journals may have helped lift the curse. One virtue of (1) is that, since it simply reinforces existing institutional norms which may already command significant support, it may be relatively easy for it to achieve wide acceptance. However, (1) will not lift the curse in all cases. In many areas with an international dimension, for example, there are no relevant international laws and deference to national laws would merely create a new unilateralist situation between nations: the nation that evaluates the initiative most positively is most likely to allow it. Moreover, (1) may sometimes recommend deferring to biased procedures or agents. It might be possible for a group of agents to lift the curse even in cases where (1) fails by complying with a different norm, one that promotes the development of and compliance with a new procedure for group decision-making. This approach was adopted by a large group of American microbiologists in mid-1974 when they agreed to a moratorium on recombinant DNA research until such time as the safety concerns that it raised could be jointly discussed and resolved. The moratorium held until the now-famous Asilomar Conference, which took place in February 1975 and resulted in a broad-based agreement on guidelines regarding the conditions under which recombinant DNA research ought to proceed.  28  The recombinant DNA moratorium and guidelines were developed via consensus, but another approach would be to employ a voting procedure. For example, suppose all agents faced with a unilateralist situation complied with the norm: (2) When in a unilateralist's situation, promote the holding of a majority vote among those capable of undertaking the initiative. If the vote takes place, then (a) defer to its verdict, and (b) encourage others to do likewise. Universal compliance with this norm is likely to lift the curse. Since it is effectively using the median estimate it is robust to outliers. It will also tend to reduce systematic bias at the group level provided that individual biases are at least partially independent of one another.  29  And since majority voting is a common and widely accepted method for group decision-making, this norm would have relatively good prospects of gaining wide acceptance. Compliance with norms (1) and (2) will, however, lift the unilateralist's curse only when a high degree of communication and coordination is possible. There are other norms whose universal adoption could lift the curse even in the absence of communication and coordination. Consider the norm: (3) When in a unilateralist situation, bring about the outcome if and only if you judge that a majority vote among those capable of undertaking the initiative would yield a majority in favor of doing so. Insofar as each individual capable of undertaking the initiative makes an accurate prediction of the views of all others, universal adoption of this norm will eliminate any group-level bias due to the unilateralist's curse. Even if predictions of the views of others are inaccurate (e.g. because each agent overestimates the extent to which others share her views), universal adoption of this principle can still be expected to somewhat mitigate the unilateralist's curse. It will tend to reduce the likelihood that those who value the initiative most favorably will undertake it, provided that these agents realize they are at the optimistic end of the spectrum.  30  Figure  7  depicts, for a five-agent case, the expected payoffs associated with two of the norms discussed in this section-tallest decides, and the actual majority vote (norm (  2 ))-and it compares these with other strategies described in Section 3.2 above. Under our assumptions, the majority vote does rather well-it is close to the maximum available payoff represented by the omniscient case. However, in the real world, different strategies will work well in different cases. It is thus likely that the best norm to adopt, under the moral deference model, would be some composite of simple norms such as (1)-(3). For example, a group might adopt a norm that specifies that the group should act as specified by (  1  We do not wish to commit ourselves to norms (1)-(  3 ) as the best building blocks from which to construct such a composite norm. We believe that each of (1)-(  3 ) are at least plausible candidates for inclusion in a composite norm. However, there may be other norms that would more fully lift the curse or which have other advantages over (1)-(3). For example, there are well-known problems with majority voting which should perhaps lead us to prefer a different voting procedure under norms (2) and (3). One other set of concerns regarding norms (2) and (3) warrants mentioning. Both of these norms involve holding a vote (real or hypothetical) among agents capable of undertaking the initiative in question. But it might be argued, on either epistemic or moral grounds, that any actual or hypothetical vote should include more individuals than merely those capable of undertaking the initiative. For example, perhaps the vote should include all whose capacity to evaluate the initiative passes some threshold of epistemic competence. Or perhaps, on moral grounds, the electorate should be expanded to include all individuals who will be affected by the initiative. Consider a case in which there are three agents who could undertake an initiative and two of the three judge that it would be best to do so. However, millions of others will be affected by the initiative and almost all of them judge that the initiative has net disvalue. In this case, it might seem morally preferable to hold (or imagine) a vote among all who will be affected by the initiative rather than limiting the vote to the three agent's capable of undertaking it. A more specific problem with excluding individuals who are incapable of undertaking the initiative is that this might seem to skew the vote. There might be some agents who are not capable of undertaking the initiative, but could have been capable of doing so; they are incapable only because they previously judged that undertaking the initiative would be a bad idea and thus ceased to develop the necessary capacities. Excluding these agents from a vote might seem to skew the vote in favor of those who deem the initiative to be valuable and who have thus sought to develop the capacities necessary to undertake it. Thus, limiting the vote to those capable of undertaking the initiative may be epistemically, as well as morally, problematic. At the same time, it might be argued that some agents capable of undertaking the initiative should be excluded from the vote. Suppose that each of five nations is capable of undertaking some geoengineering project with worldwide consequences. Four agree to hold a majority vote among the five nations and to abide by the outcome of that vote. The fifth wishes to take part in the vote but is resolved to press ahead with the project regardless of the outcome of the vote. It might seem doubtful whether the first four nations should include the fifth in the vote. Arguably, deferring to a majority vote in unilateralist cases involves making a sacrifice. It involves giving away some of one's autonomous decision-making authority. It might seem that would be unfair for the fifth nation to exert an influence over the decisions of others by participating in a vote without also being prepared to make the same sacrifice that the others are prepared to make. This may count in favor of excluding the fifth nation. Excluding the fifth nation might also help to incentivize deference to majority votes in unilateralist situations. There are thus arguments both for expanding and for restricting the group of agents given a vote in norms (2) and (3). We cannot assess these arguments here. We mention them only to flag them as topics for further discussion. However, it is worth noting that including all and only those agents who are capable of undertaking an initiative does at least have the virtue of picking out a group that would, in many cases, be relatively easy to identify. We should end this section on the moral deference model with an important clarification: the model does not rely on a commitment to any particular moral theory. Proponents of a range of different moral theories could accept norms of the sort described above, though they would assign different statuses to them. A rule consequentialist, for example, might treat these norms as genuine moral principles-principles that determine which acts are right and which are wrong. According to one formulation of rule consequentialism, a rule of action is a genuine moral principle just in case it is part of the set of rules of action whose general acceptance can be expected to have consequences as good as the general acceptance of any alternative set of rules.  31  Given the risk of premature or erroneous action created by the unilateralist's curse and the likelihood that most agents are not sophisticated enough belief-formers to apply our meta-rationality model, it is plausible that the optimal set of rules will contain a norm of the sort that we have discussed. On some other moral theories, these norms would serve not as genuine moral principles, but as guidelines for helping agents to comply with such principles. Adherents of many moral theories, both consequentialist and deontological, could accept something like the following moral principle: Agents have moral reasons to undertake an initiative if and only if that initiative would contribute to the common good, and to spoil an initiative if and only if that initiative would detract from the common good. Norms of the sort discussed above could help agents to better comply with this principle in unilateralist situations. 32
 Discussion We proposed: The Principle of Conformity When acting out of concern for the common good in a unilateralist situation, reduce your likelihood of unilaterally undertaking or spoiling the initiative to a level that ex ante would be expected to lift the curse. We also outlined three different ways in which agents who find themselves in unilateralist situations might comply with this principle. We do not claim that any one of these models is superior to the others in all situations. Which model should be adopted will depend, among other things, on the sophistication of the agents, the degree of communication and coordination that is possible, and the nature of existing laws and conventions bearing on the decision. In this section we discuss a concern that might be raised regarding our principle. Adoption of the principle of conformity is meant to make things better. Yet if we "backtest" the principle on historical experience, it is not at all clear that universal adoption of the principle of conformity would have had a net positive effect. It seems that, quite often, what is now widely recognized as important progress was instigated by the unilateral actions of mavericks, dissidents, and visionaries who undertook initiatives that most of their contemporaries would have viewed with hostility and that existing institutions sought to suppress. The benefits of iconoclasm and defiance of authority have been stated especially forcefully in the Enlightenment tradition and by proponents of scientific and technological progress. They are also evident in many cases of "whistleblowing." Consider the case of Daniel Ellsberg, famous for leaking the Pentagon Papers, which revealed the hopelessness of the US military situation in Vietnam. Most of Ellsberg's peers, who had the high-level security clearance required to access the relevant documents, presumably did not believe that leaking the material to the press would contribute positively to the common good. If Ellsberg had sought to follow the principle of conformity, for example by imagining a vote among all those in a position to leak the documents, it would seem he would have had to conclude that the documents ought not be leaked. This might seem an undesirable outcome. It is possible that the appearance that unilateralism has historically been mostly for the good is illusory. Historical unilateralism might be more salient when it worked out well than when it worked out badly, perhaps because successes have been more extreme but less frequent than the failures. Moreover, it may be that, in some cases where the principle of conformity appears to recommend a net harmful course of action, this implication can be avoided by attending to how the group of (imaginary or actual) voters or epistemic peers is defined. For example, if one allows that these groups might be defined more broadly than the group of agents capable of undertaking an action, it may be possible to avoid the implication that Ellsberg should have refrained from whistleblowing. (Suppose that many "outsiders" would have voted in favor of his releasing the information.) However, even if unilateralism has historically provided a net benefit to humanity, this need not undermine our argument. The claim that the unilateralist curse is an important phenomenon and that we have reason to lift it is consistent with the claim that the curse has provided a net benefit to humanity. The main effect of the curse is to produce a tendency towards unilateral initiatives, and if it has historically been the case that there have been other factors that have tended to strongly inhibit unilateral initiatives, then it could be the case that the curse has had the net effect of moving the overall amount of unilateralism closer to the optimal level. For example, it might be argued that the scholars of past ages were usually far too deferential to authority, for reasons independent of the factors discussed in this paper. Their failure to take into account our arguments might then have had the salutary effect of not further inhibiting whatever propensity remained to promote new thoughts.
 Concluding Thoughts We have described a moral analog of the winner's curse. The unilateralist's curse arises when each of a group of agents can, regardless of the opposition of others, undertake or spoil an initiative that has significant effects on others. In such cases, if each agent decides whether to undertake (or spoil) the initiative based on his own independent naive assessment of its value, there will be a group-level bias towards undertaking (spoiling) the initiative. Importantly, this effect arises even if all the agents are assumed to be motivated solely by concern for the common good. We proposed a principle-the principle of conformity-which instructs agents faced with a unilateralist situation to reduce their likelihood of unilaterally undertaking (or spoiling) the initiative. We then outlined three models for accomplishing this. They involved, respectively, (1) sharing information and reasoning before forming one's evaluation of the initiative, (2) adjusting one's evaluation in the light of the curse, and (3) deferring to the group in making one's decision. As we acknowledged in the previous section, there may be considerations that militate against the principle of conformity. For example, if there is already a group-level bias against unilateralism, then compliance with the principle would exacerbate this bias. However, we maintain that there is a prima facie case for complying with the principle. Moreover, since the level of bias due to such other factors towards or against unilateralism presumably varies across different contexts, it is likely that there will be some contexts in which the prima facie case for complying with the principle will be decisive. Those will be the contexts in which the group-level bias due to the unilateralist's curse is greater than any countervailing bias against unilateralism. It is also possible that, at least within the domain of science, the principle of conformity is more relevant today than it was, say, prior to the Enlightenment. At that time, there was, plausibly, a strong bias against thinking and acting independently in intellectual matters, at least where this would involve diverging from the views of the Church. Since the Enlightenment, however, there may have been a significant weakening of this bias. Independence of thought and action is now more widely regarded as a virtue in scientists and other intellectuals. Honors and prizes are won based on claims to originality and precedence. There may now be no bias, or only a weak bias, against unilateralism in science. Thus, the risk posed by the unilateralist curse in scientific contexts may be greater now than ever. To resist the unilateralists' curse one first has to become aware of when one is in a curse situation. We hope this paper will help achieve that.
 Funding This work was supported by the The Oxford Martin School; The Wellcome Trust [grant number WT087211].
 Notes [1] We assume that the common good is determined in part by the wellbeing of all persons and other morally significant individuals. However, we remain neutral on precisely how individual wellbeing determines the common good. For example, we do not commit ourselves to the view that the common good is simply aggregate individual wellbeing; we allow that the distribution of wellbeing might be relevant. We also allow that factors besides individual wellbeing might influence the common good. For example, some initiatives might possess intrinsic value that is independent of their contribution to wellbeing, and we allow that this intrinsic value might be one element in the common good. [2] The Progressive Magazine (1979). [3]  Rotblat (1985) . [4]  Bowden (2007) . [5]  Williams (1995) . [6] ,  Gurwitz (2014) , and . [7]  Thaler (1988) . [8] The probability that a particular agent will be wrong about the sign of the value of the outcome is Pr(V * + d > 0) if V * < 0 and Pr(V * + d < 0) if V * > 0. This is equal to 1 − F (−V * ) if V * < 0 and F(−V * ) if V * > 0. The probability that out of N agents at least one will be wrong about the sign is (1 − F(−V * ) N ) if V * < 0 and (1 − (1 − F(−V * )) N ) if V * > 0. However, even if errors are symmetric around 0, the expected outcome is not: in the V * < 0 case it is enough that one agent acts for a negative value to be obtained, while in the V * > 0 case all agents have to err on the side of caution for them to lose out on a positive value. The expected value obtained by naive agents is hence V * (1 − F(−V * ) N ). For positive values this is close to V * (for unbiased error distributions), and we will hence focus on the V * < 0 case where unilateral action is a problem. [9] Theorem: As N grows, the likelihood P of at least one agent proceeding incorrectly increases monotonically towards 1 unless F(−V * ) = 1 (i.e. unless there is an upper limit on the size of the deviations and V * is more negative than this limit, no agent will ever make a sufficiently bad mistake).Proof: If F(−V * ) = 1, p = 0 for all N. Otherwise 0 ≤ F (−V * ) < 1, and hence F(−V * ) N approaches 0 as N → ∞. [10] There will also, of course, be cases where an agent's decision whether to undertake an initiative affects others but the agents are motivated by self-interest rather than the common good. In these cases, there are two possible reasons for getting the wrong decision, from the point of view of the common good: (i) self-interest and the common good come apart-that is, one is judged to have positive value and the other negative value-and (ii) the agent overestimates the "self-interest" value of the initiative. [11] If the distribution of errors is skewed such that the typical estimate is higher than the true value, for instance due to optimism bias, then the risk of erroneous action is increased: in that case, even a single agent might be likely to overestimate the value of the initiative sufficiently to undertake it even when the true expectation value of the initiative is strongly negative. But this is unrelated to the curse. In the case of estimates skewed towards safety-that is, where there is pessimism bias-any tail distribution allowing mistaken action will still produce a growing probability of going ahead as N grows. There may be intermediary cases where the curse would helpfully serve to balance out an opposite effect arising from pessimism bias. [29] The assumptions of the Condorcet theorem can be weakened in many ways. In particular, agent competence only has to be on average above 50%  (Grofman, Owen, and Feld 1983) , and a certain level of voting correlation does not reduce majority voting performance  (Ladha 1992 ). [30] For similar reasons, an analogous norm would tend to reduce the likelihood of spoiling an initiative of those who evaluate an initiative most negatively. [31] See, for example,  Hooker (2002) . [32] A parallel can be drawn to one prominent justification for the authority of the law, due to Joseph Raz. That justification appeals to the same kind of consideration that we suggest could ground a norm against unilateral action: "The normal and primary way to establish that a person should be acknowledged to have authority over another person involves showing that the alleged subject is likely better to comply with reasons which apply to him (other than the alleged authoritative directives) if he accepts the directives of the alleged authority as authoritatively binding, and tries to follow them, than if he tries to follow the reasons which apply to him directly"  (Raz 1994, 214) .  [33]  In this case, the maximum likelihood estimate is simply the average of their individual estimates. Figure 3 3 Figure 3 Probability of an erroneous action in the case of V * = −1 for different numbers of agents.
 Figure 5 5 Figure5The expected payoff for different actual values of the initiative for alternative ways of handling the unilateralist's curse. Using the optimal individual threshold T opt (5) reduces the losses significantly.
 Figure 6 6 Figure6Expected payoff for different actual values of the initiative for alternative ways of handling the unilateralist curse. The tallest decides case achieves a significant reduction of loss, nearly reaching the payoff of the more complex Bayesian threshold method.
 Figure7depicts, for a five-agent case, the expected payoffs associated with two of the norms discussed in this section-tallest decides, and the actual majority vote (norm (2))-and it compares these with other strategies described in Section 3.2 above. Under our assumptions, the majority vote does rather well-it is close to the maximum available payoff represented by the omniscient case.However, in the real world, different strategies will work well in different cases. It is thus likely that the best norm to adopt, under the moral deference model, would be some composite of simple norms such as (1)-(3). For example, a group might adopt a norm that specifies that the group should act as specified by (1), (2) or (3) depending on what laws and conventions already exist, what forms of communication and coordination among group members are possible, and how costly such communication and coordination is likely to be, among other factors.We do not wish to commit ourselves to norms (1)-(3) as the best building blocks from which to construct such a composite norm. We believe that each of (1)-(3) are at least plausible candidates for inclusion in a composite norm. However, there may be other norms that would more fully lift the curse or which have other advantages over (1)-(3). For example, there are well-known problems with majority voting which should perhaps lead us to prefer a different voting procedure under norms (2) and (3).One other set of concerns regarding norms (2) and (3) warrants mentioning. Both of these norms involve holding a vote (real or hypothetical) among agents capable of undertaking the initiative in question. But it might be argued, on either epistemic or moral grounds, that any actual or hypothetical vote should include more individuals than merely those capable of undertaking the initiative. For
 Figure 7 7 Figure7The expected payoff associated with universal compliance with six different strategies at different actual values of the initiative. The fully shared information strategy consists in pooling the information between the agents and acting on the group's best joint estimate of V * ; 33 this requires maximal communication. Despite the lack of communication in tallest decides and threshold setting, the agents achieve an average outcome close to the cases where communication is possible.

			  N. Bostrom et al.
			  N. Bostrom et al.
			  N. Bostrom et al.
			  N. Bostrom et al.