Abstract: AI safety often analyses a risk or safety issue, such as interruptibility, under a particular AI paradigm, such as reinforcement learning. But what is an AI paradigm and how does it affect the understanding and implications of the safety issue? Is AI safety research covering the most representative paradigms and the right combinations of paradigms with safety issues? Will current research directions in AI safety be able to anticipate more capable and powerful systems yet to come? In this paper we analyse these questions, introducing a distinction between two types of paradigms in AI: artefacts and techniques. We then use experimental data of research and media documents from AI Topics, an official publication of the AAAI, to examine how safety research is distributed across artefacts and techniques. We observe that AI safety research is not sufficiently anticipatory, and is heavily weighted towards certain research paradigms. We identify a need for AI safety to be more explicit about the artefacts and techniques for which a particular issue may be applicable, in order to identify gaps and cover a broader range of issues.

Jose Hernández-Orallo
Fernando Martínez-Plumed
Shahar Avin
Jess Whittlestone
Seán Ó H Éigeartaigh

AI Paradigms and AI Safety: Mapping Artefacts and Techniques to Safety Issues

INTRODUCTION As in any other scientific or engineering discipline, many AI researchers work within a well-established paradigm, with some standard objects of study, problems to solve, and associated formalisms and terminology. Understanding past, current and future AI paradigms is an important source of insight for funding agencies, policymakers, and AI researchers themselves, because it shapes how we think about what problems AI research is aiming to solve, what methods are required to solve them, and what the wider implications of progress might be. We believe that thinking clearly about AI paradigms is particularly crucial for AI safety research: an increasingly important area concerned with understanding and preventing possible risks and harmful impacts in the design and deployment of AI systems. For the purposes of this paper, we define AI safety broadly, to include both risks from AI systems for which the source of risk is accidental, ranging from unpredictable systems to negligent use, and non-accidental risks such as those stemming from malicious use or adversarial attacks (which might sometimes also be referred to as AI security risks.) This includes risks with many different types of consequences: including human, environmental, and economic consequences. AI safety is becoming particularly important as AI is increasingly used to automate tasks that involve interaction with the world. A characteristic example is AI being used in many components of self-driving cars such as perception, reasoning and action. Research in AI safety has typically analysed a specific risk or issue under a particular existing paradigm, such as interruptibility for reinforcement learning agents  [33] , adversarial attacks on deep learning systems  [43] , or fake media produced with GANs  [21] . Poor choices or misinterpretation of current and future paradigms may result in AI safety research focusing in the wrong places: analysing scenarios that will not take place in the future, or that will at best manifest in completely different ways. For example, concerns about adversarial attacks on current deep learning methods have led to many papers proposing methods to defend against malicious perturbations, but it is not clear that these actually relate to plausible security concerns  [16] . Though adversarial examples help us understand how brittle current deep learning methods can be, the kinds of attacks that some safety research is concerned with can only arise in contrived interactions, and make strong assumptions about the goal, knowledge, and action space of the attacker. In contrast, other risks have been ignored because they are outside of the current paradigm: for instance, attacks which go beyond an "independent" or functional interpretation of a neural network, and instead use the "information from previous frames to generate perturbations on later frames"  [41] . Thinking about different paradigms is also important for clearly assessing safety considerations and risks in concrete real-world applications. For instance, the risks of using AI in vehicles will be perceived differently depending on whether we expect AI to be assisting or replacing human drivers, whether a self-driving vehicle is considered as a single autonomous agent, or whether we consider the whole traffic system as a swarm of interacting agents. Even as research becomes oriented towards future risks (some years or even decades away) the different safety issues associated with different AI paradigms have not been explicitly addressed in the literature  [6, 2, 23, 14] . As AI research is fast evolving and likely to result in increasingly powerful systems in future, it is crucial for AI safety to explore issues associated with a broader range of possible AI paradigms, and to explicitly discuss what assumptions about paradigms and safety issues are being made in each research paper or project. This will not only enable the research community to identify the effects of less-explored paradigms on safety concerns, but could also increase awareness among AI developers that the paradigms they work in have consequences for safety, potentially highlighting approaches that can eliminate or reduce safety risks. In this paper, we present a structured approach for thinking about paradigms in AI and use this as the basis for empirical analysis of how AI safety issues have been explored in the research literature so far. We begin by defining what we mean by 'paradigms' in AI more precisely, drawing on literature in the philosophy of science to distinguish two different types of paradigms in AI: artefacts and techniques. Drawing on existing research and the expertise of several AI and AI safety researchers, we outline a preliminary taxonomy of fourteen different AI techniques and ten different AI artefacts. We then discuss how these different techniques and artefacts relate to AI safety issues. We use AAAI's 'AI topics' database to conduct a grounded empirical analysis of the historical evolution of these different paradigms, and the safety issues associated with them. Our analysis identifies a number of gaps in AI safety research where certain combinations of techniques, artefacts and safety issues need to be addressed. We conclude the paper by discussing implications for future research in AI safety.
 DEFINING PARADIGMS IN AI Before discussing paradigms in AI and how they relate to safety issues, we need a clearer account of what an AI paradigm is. In the philosophy of science, a 'paradigm' is an important concept used to capture how theories, methods, postulates and standards evolve and change within a scientific discipline  [22] . The concept of a 'technological paradigm' plays a similar role but with a somewhat distinct focus, emphasising how each technological paradigm defines its own concept of progress, based on making specific technological and economic trade-offs  [12] . In computer science, the term 'paradigm' is commonly used to refer to different types of programming languages with different features and assumptions: imperative, logic, functional, object-oriented, distributed, event-oriented, probabilistic, etc. The theory and practice of programming languages depends heavily on these paradigms. Many safety issues in programming and software engineering, and verification in particular, cannot be addressed -conceptually or technically-without making paradigms explicit  [5, 20, 40, 42] . For instance, compared to imperative programming languages, declarative languages minimise mutability issues  [10]  due to the use of immutable data structures, as well as reduce state side-effects  [29]  by discouraging the utilisation of variables in favour of more sophisticated constructs (e.g., data pipelines or higher-order functions). In the context of AI, the concept of 'paradigms' has been used informally to refer to different broad families of technical or conceptual approaches: 'symbolic' vs 'connectionist', reasoning vs learning, expert systems vs agents. One way to identify paradigms in a field and how they have changed over time, suggested in Kuhn's original formulation, is to look at the approach taken by major (text)books in the field. Russell and Norvig  [36] , for example, popularised the 'agents' view of AI in the 1990s, and Goodfellow et al.  [17]  has consolidated 'deep learning' as a central approach AI research this decade, going far beyond one specific technique. However, there is no clearly agreed-upon definition of what counts as a paradigm in AI.  [9]  defined an AI paradigm as "the pair composed by a concept of intelligence and a methodology in which intelligent computer systems are developed and operated", which led him to identify three paradigms of AI: behaviourist, agent and artificial life. More than twenty years later, these do not seem to adequately reflect the approaches and assumptions within AI research today. Perhaps this difficulty clearly distinguishing AI paradigms from mere trends stems at least partly from the ambiguity in the concept of paradigm itself, a criticism which has been made within the philosophy of science. Masterman identifies three different conceptions of paradigm in Kuhn's writings: the metaphysical, the sociological AI Safety Issue AI Technique AI Artefact Figure  1 . Ways of analysing an AI safety issue, combined with an artefact and/or a technique category. In real systems, techniques underpin the creation or operations of an artefact, where the real hazards occur. Occasionally, researchers can think of a safe issue in a very abstract way, without committing to any particular artefact or technique (e.g., value alignment). All relations (arrows) are many-to-many. and the artefact/construct  [27] . Peine makes a reconstructive effort to combine the second and the third conceptions, suggesting that a paradigm can emerge when a subcommunity within a field makes a commitment to a certain set of techniques  [35] . For instance, we can see this phenomenon in the deep learning community, which made early commitments before this approach had established a dominant position within the field. We suggest distinguishing between dominant trends: research commitments to specific techniques, aims, or assumptions that may be relatively short-lived; and paradigms where these trends lead to more established, long-term commitments. These ambiguities in the concept of a paradigm are apparent in the context of AI, where 'paradigm' may be used to distinguish both different techniques for solving problems in AI (e.g., Monte Carlo search vs. SAT solver, deep learning vs. genetic algorithm), and to capture different conceptions of possible types of AI systems (e.g. expert systems vs. agents). Many technical papers can easily be categorised by looking at the first formal definitions that appear after the introduction, which tend to identify the kind of AI system and the kinds of techniques they use to solve a problem. We suggest that it may be helpful, in thinking about paradigms in AI, to distinguish explicitly between these two types of constructs: • Conceptual Artefacts: broad conceptions of what current and future AI systems (will) look like, e.g., autonomous agents, personal assistants  [24] , AI extenders  [19] , conceptions of superintelligence  [6]  and Comprehensive AI Services  [13] . • Research Techniques: the research methods, algorithms, theoretical technical results and methodologies involved in the development of these current or future systems, such as SAT solvers, deep learning, reinforcement learning, evolutionary computing, etc. Artefacts and techniques are important components of a paradigm, and together (weakly) define a paradigm. AI artefacts and techniques are often related to each other, insofar as certain techniques will often be better suited to certain types of artefacts (reinforcement learning is a technique category that is applicable for agent-like artefacts, for example). However, distinguishing between artefacts and techniques can help us to think more clearly about associated safety issues. Some safety issues will arise when combining an artefact with some techniques but not others: for example, when building a classifier (artefact), interpretability is likely to be a challenge for safety if using deep neural networks (technique), but less so if using simple decision trees with conditions that are expressed over the original attributes. Recognising these differences is important for understanding the scope of safety challenges and what solutions may be needed. In the case of autonomous vehicles, for example, both techniques and artefacts have changed over time, changing conceptions of safety. Early work in the 1980s-1990s, such as the Eureka 24th European Conference on Artificial Intelligence -ECAI 2020 Prometheus Project, emphasised systems for improved vehicle-tovehicle communications and driver assistance (e.g. collision avoidance), whereas more recent initiatives emphasise the development of fully independent self-driving vehicles. These are very different AI artefacts, with a more fully autonomous system evoking much broader safety concerns in general. However, to think more concretely about how evolving approaches to autonomous vehicles change safety challenges, it may also be important to look at how techniques have changed over time. For example, as improvements in computer vision enable more limited sensors to be replaced by cameras or radars, we may need to focus specifically on the vulnerabilities introduced by current techniques used in machine perception. Sometimes we may want to broadly assess all possible risks associated with a specific AI artefact independently of the technique used (e.g. issues associated with AI extenders), or those associated with a technique independently of the artefact (e.g. safety challenges for deep learning). In other cases, we may want to more narrowly focus on a specific safety issue associated with a particular artefact (e.g. interruptability for industrial robots), or a specific issue associated with a particular technique (e.g. interpretability of neural networks.) Figure  1  shows these interrelationships: how AI safety issues may be considered relative to either techniques, artefacts, or both.
 IDENTIFYING TECHNIQUES AND ARTEFACTS We can now begin to explore different ways of categorising AI paradigms by decomposing them into techniques and artefacts. While other accounts of 'AI paradigms' have been proposed, such as by the "One Hundred Year Study on Artificial Intelligence" at Stanford University  [39] , the categories frequently mix techniques with artefacts and even subfields.
 AI techniques We develop a preliminary categorisation of AI techniques by building on the bibliometric analysis of  [26]  and  [32, Tab. 6 ]. Martínez-Plumed et al.  [26]  identify nine 'facets' for the study of the past and future of AI, using data on all accepted papers from AAAI/IJCAI conferences (<PHONE>), and AI Topics documents, an archive kept by AAAI containing news, blog entries, conferences, journals, and other repositories. Niu et al.  [32]  identify 30 high-frequency keywords from 20 relevant journals in AI from 1990 to 2014. Both capture many keywords and categories which appear to describe AI techniques, and are relatively similar.  4  We construct a list of AI techniques, grouped into 14 categories, based on selecting relevant and representative keywords from these two analyses. We chose these categories based on three principles: (1) the techniques are sufficiently general to encompass groups of approaches in AI that have been recognised by other approaches, (2) overlapping in the techniques is allowed, as there is a high degree of hybridisation and combination in AI, and (3) subcategories are retained for particularly large categories, such as 'machine learning'  5  The list of categories is shown in Table  1 .
 AI artefacts For AI artefacts, we would like our categories to be more stable over time and less tied to specific research techniques and applications. One way to look beyond current trends is to consider textbooks or even historical accounts of AI  [28, 7, 31] . It may also be helpful to begin by considering broad 'characteristics' of AI systems independent of the specific techniques used to develop them  [18] . To generate a preliminary set of artefacts, a group of multidisciplinary researchers conducted a systematic, interactive procedure based on the Delphi method  [11] . The group began by independently brainstorming possible candidates for categories of artefacts based on a preliminary discussion of those arising from AI textbooks and different AI system characteristics. Two criteria were provided to structure this initial brainstorm: (1) artefacts should ideally have minimum overlap, each capturing distinctive functionalities, and (2) artefacts should be defined independently of how functionalities are achieved (i.e. independent of techniques). Group members each proposed lists of distinctive type of AI systems, supported by exemplars, which were revised using an iterative process until the list of answers converged towards consensus. These were then clustered hierarchically to produce a set of artefacts which could cover the space of actual and potential AI systems as exhaustively as possible. The categorisation of AI artefacts in Table  2  is the result of this process. Building on the categories identified in  [18] , we can further characterise these different artefacts in terms of their integration with the external environment, in three ways: (1) how it interfaces with the environment (e.g. via sensors and actuators, via digital objectives, or via language); (2) the dynamics of this integration (whether it is interactive or functional); and (3) the location of this integration (whether it is centralised, distributed, or coupled). For example, an agent interfaces with the environment via sensors and actuators in an interactive and centralised way. A swarm has similar characteristics except that the location of its integration is distributed (across different units) rather than centralised. We also provide examplars for each category: self-driving cars and robotic cleaners are examples of agent-type systems, whereas a multiagent network router or drone swarm are examples of swarm-type systems. We outline these characteristics to show that the artefacts are sufficiently comprehensive and distinctive to provide a useful basis for further analysis, though as with any clustering there are some overlaps and borderline cases.
 Empirical analysis of techniques and artefacts Using our categories of techniques (e.g., neural networks, information retrieval, cognitive approaches, etc.) and artefacts (e.g., estima-24th European Conference on Artificial Intelligence -ECAI 2020 tor, agent, dialoguer, etc.), we now conduct an empirical analysis of how these different paradigm components have appeared in research papers and other related literature. This analysis will allow us to explore more empirically which paradigm elements have been prominent at different times, and will form the basis for investigating the relationship between paradigms and safety issues later in this paper. To conduct this analysis, we work with AI Topics 6 , an official database from the AAAI, using the documents from the period <PHONE> (complete years). This archive contains a variety of documents related to AI research (news, blog entries, conferences, journals and other repositories) that are collected automatically with NewsFinder  [8] . We divide the archive into research documents  7  and non-research documents. From the ∼111K documents gathered, ∼11K are research papers and the remaining ∼100K are mostly media. With a mapping approach between the list of exemplars (tokens) of techniques and artefacts (e.g., "Deep Learning", "GAN" or "perceptron" are illustrative exemplars of the "neural network" technique; "classifier", "decision-making" or "object recognition" are exemplars of the "estimator" artefact5 0 ), and the tags obtained from AI Topics (substrings appearing in titles, abstracts and metadata), we summarise the trends in a series of plots. The evolution of techniques and artefacts in these documents (i.e., fraction of papers focusing on the list of tokens for each technique category or artefact) is shown in Figure  2 , where the data has been smoothed with a moving average filter in order to reduce short-term volatility in data. Note that this figure shows the aggregation of categories (techniques and artefacts) where each area stack is scaled to sum to 100% in every certain period of time. In a way, this can be understood as a non-monolithic view of polarities and intensities in sentiment analysis (e.g., identifying sentiment orientation in a set of documents). We see some techniques and artefacts are particularly dominant, as might be expected. For instance, looking at the techniques (left plot), 'knowledge representation and reasoning' takes almost half of the proportion of documents between <PHONE>, while 'general machine learning' becomes more relevant from 2005. We also see an important peak of multiagent systems around 2000. When we look at the artefacts (right plot), we see that 'extractor' (which includes expert systems) became very popular around 1990 but in a matter of a decade was overtaken by 'agent', which became dominant in 2000. We see today that five kinds of artefacts account for about 90% of all mentions: estimators, agents, optimisers, extractors and creators. These patterns also show that some artefacts are not new as a paradigm component, especially when we consider them in a more abstract way. For instance, it is sometimes understood that GANs introduced a new paradigm, but other kinds of generative systems have been around in AI since its exception, as we see in the violet band in Figure  2  (right). We can also use this analysis to explore what techniques and artefacts are prominent in the media. In Figure  2 , we look at all the non-research papers, shown in the 'Media' column besides each 'Research' plot. In this case, we can only show selected documents for the last two (complete) years  8  . We see that the dominances are more extreme. For the techniques (Figure  2 , left), 'neural networks' and 'general machine learning' occupy more than 75% of total mentions. On the right, we see that the distribution of artefacts is different, but also extreme: only four artefacts ('estimator', 'agent', 'dialoguer' and 'creator') cover more than 90% of mentions. In this case, we see that 'optimiser' and 'extractor' are less visible for laypeople than for researchers, while 'dialoguer' is more relevant for laypeople (most probably because of the common use of digital assistants).
 PARADIGMS AND SAFETY ISSUES To begin exploring how paradigms relate to safety issues in AI research, we conduct two different analyses. First, we look at the relative prominence of different techniques and artefacts specifically within research publications and other publications that are related to safety. This will help us to discern how the prominence of different paradigms differs in safety research from AI research as a whole, and to consider whether some paradigms are under-or over-represented in the safety literature. Second, we look at how different safety issues co-occur with different paradigms in the literature, enabling us to understand which safety issues are considered to relate to which paradigms, and to identify potential gaps in the literature.
 Analysing paradigms within safety research To explore the prominence of different AI techniques and artefacts in AI safety literature, we begin by conducting a similar analysis to that  8  While AI topics provides research documents (mainly) from the 70s onward, media-related documents come mostly from the last 2-3 years. in section 3.3, but restricting to documents which make some reference to AI safety. We use a list of over 100 relevant tokens to filter documents (Table  3 , right column, shows some examples of tokens for each type of safety issue5 0 ). Applying this filter, we find that, from the ∼111K documents in AI Topics, about ∼21K are related to safety, of which ∼1.5K are research papers, and ∼19K are broader (mostly media) documents. Figure  3  shows the frequency of reference to different techniques and artefacts within this safety-specific database. The results look similar to those for the enitre document database in Figure  2 , but there are some important differences. First, the frequencies of reference to different techniques and artefacts are more extreme and varied over time. This might be a consequence of the smaller sample size, but we do see this pattern particularly with the more dominant terms, where we have a reasonable sample size. In particular, we see that for AI research as a whole, 'knowledge and reasoning' was a dominant technique between the mid 1970s and mid 1990s, taking up about 50% of references (Figure  2 , left) -but is even more dominant as a proportion of mentions in safety research in the 1990s, at about 60% (see Figure  3 , left). This is even more extreme for the 'extractor' artefact, with about 50% of the documents in the late 1980s (Figure  2 , right), to more than 75% in the early 1990s (Figure  3 , right). The most interesting observation comes from looking at the periods when these peaks happen for different techniques (comparing the left plots of Figure  2  and Figure  3 ). The peak for 'knowledge representation and reasoning' happened in the mid 1980s when looking across AI research in general, but in the early 1990s when filtered by papers which mention safety. The peak for 'planning and scheduling' took place in the late 1990s across all papers but a few years later (and was more pronounced) when filtered by safety-relevant papers. We see a similar pattern for artefacts (comparing right plots of Figures  2 and 3 ): 'extractors' is dominant in the AI literature in the late 1980s, but only become prominent in safety research in the early 1990s; 'agents' rise to prominence in AI research in the late 1990s but only peak in safety research in the early 2000s. This suggests a five-year delay (approximately) between when a technique or artefact becomes popular within general AI research, and when researchers begin to seriously consider the safety issues associated with it. This suggests that safety issues are considered reactively in response to dominant research patterns, and are only considered once a paradigm has been prominent for several years (rather than safety issues related to a given technology being considered at the outset of research, as is more often the case in domains like engineering). In other words, we find that there is a delay between the emergence of AI paradigms and safety research into those paradigms, and safety research neglects non-dominant paradigms. Looking at the plots for non-research documents (comparing documents filtered for safety (Figure  3 , Media), with all documents (Figure  2 , Media), we see that the plots are almost identical. The only slight difference is that both 'natural language processing' techniques and 'dialoguer' artefacts are less prominent in articles filtered for safety, suggesting that the media and laypeople are perhaps less concerned about the harms of conversational systems than experts are.
 Mapping paradigms to safety issues Next we look more closely at how different AI paradigms relate to specific safety issues. To do this, we need a way to categorise different safety issues. Though existing categorisations of safety issues exist  [25, 34] , we found these were too coarse for our purposes (e.g.  [34]  uses just three categories). Building on these existing categorisations, we identified key terms in surveys, blogs and events in AI safety  [6, 2, 23, 14, 41, 37] , and clustered them into groups. Through this process we identified 22 categories, as shown in Table  3 . Our clustering process attempted to aggregate safety categories in an abstract way, independently of the subfield in which a term occurs most frequently. For instance, 'distributional shift' is the term used in machine learning, whereas 'belief revision' is the term used in the area of knowledge representation and reasoning. However, both refer to solving the same type of problem -where a system has to adapt or generalise to a new context-so we group both together under the 'problem shift' category. With this list of categories, we then analyse how related different paradigms and safety issues are, by counting the number of papers (of those filtered for safety relevance) which mention both a given paradigm component (technique or artefact) and a given safety issue, for all combinations of techniques and safety issues (and the same analysis, separately, for artefacts). Figure  4  shows these relationships, where the width of each band represents the number of papers including reference to both elements, with techniques/artefacts on the left, and safety issues on the right. The most general safety issues, such as 'trust, transparency & accountability', 'privacy & integrity', 'reliability & robustness', 'problem shift' and 'interpretability', have the widest bands linking them to different paradigm components, and tend to be linked to a wide variety of different paradigm components (shown by the 'multicolour' bands coming out from these issues). We notice several interesting insights about the relationship between safety issues and techniques or artefacts. The issue of 'problem shift' is largely associated with 'knowledge representation and reasoning', which is surprising since we might expect the broad problems of generalising to new contexts and distributions to be relevant to a wide range of techniques. This may be due to the relevance of belief revision in this kind of techniques. 'Privacy and integrity' is associated with many techniques, but not with reinforcement learning -which makes sense, given that reinforcement learning is much less likely to make use of personal data than other techniques. However, reinforcement learning is also not related to 'safe exploration and side effects', and is only very weakly associated with 'problem shift', which is more surprising, since these do seem like issues that are important for ensuring RL systems are used safely. Part of the reason for this may be that our analysis simply did not find much mention of safety issues in relation to reinforcement learning overall. While we must recognise the limitations of the database we had access to (perhaps AI topics does not capture the kinds of venues where safe reinforcement learning research is published), this does suggest that greater exploration of safety issues related to reinforcement learning is an important gap. Similarly, there is relatively little 24th European Conference on Artificial Intelligence -ECAI 2020 literature linking probabilistic and Bayesian approaches or evolutionary approaches in ML to safety issues. When we focus on less covered safety issues, we see that the associations are less multicoloured. For instance, 'scalable supervision' and 'specification & value alignment' are associated only with reinforcement learning, and 'adversarial attacks' only with 'neural networks'. More surprising are the associations of 'confinement problem', 'manipulation' and 'safe exploration & side effects'. Overall, most attention is paid to a few now prominent safety issues, but more specific issues have very limited combinations with some techniques. The right plot in Figure  4  shows the analysis for artefacts. In this case, also looking at the circle from right to left, we see multicolour bands relating some of the more prominent safety issues to a variety of different artefacts. The 'agent' paradigm component, which is more prominent in general, is associated with the widest range of safety issues by far -suggesting that more research on safety issues associated with different artefacts may be worthwhile. In general, across techniques and artefacts, we see that 'reliability and robustness' is by far the most frequently discussed safety issue -perhaps because it is more immediately and directly related to the performance of a system than the others. This is followed by 'trust, transparency and accountability', 'privacy and integrity' and 'problem shift'. More research into the less prominent safety issues and how they relate to different paradigms would be valuable.
 GENERAL DISCUSSION A next step for this work would be to combine this mapping exercise with deeper conceptual analysis of the relationship between techniques, artefacts, and safety issues. For instance, the "confinement problem" has been discussed in technical safety research relatively recently  [4] , and is already strongly associated with optimisers, organisms and extractors (Figure  4 , right): systems that are naturally thought of as being encapsulated. However, other types of systems -dialoguers, estimators, providers and extenders-might also learn to behave in ways that go beyond their original specification or constraints, and it may be worth considering a wider range of "confinement" problems across many different AI systems. More broadly, thorough case studies of how a specific AI safety issue might arise for different techniques and artefacts would be very valuable, as would more systematic explorations of the various different safety issues associated with a given paradigm broadly construed. Some recent papers do go in this direction  [41, 3] . Further analysis is needed regarding how more recent safety issues relate to the kinds of AI artefacts being deployed in society today, and the techniques those artefacts depend on. For instance, some recent accidents involving self-driving cars may be thought of as a consequence of people misunderstanding the type of artefact autonomous vehicles currently are: while many regard them as 'agents', they are really only 'extenders', and not yet meant to behave autonomously. Terms such as 'auto-pilot' only aid this confusion. Other self-driving car incidents have been caused by idiosyncratic imperfect performance of object recognition systems, failing to detect a human or other objects in rare situations. For now, we recommend that research papers make explicit which particular issues, artefacts and they are covering, and which ones they are excluding, and give some indications of why it is the case (because it does not apply or left for future work). Another avenue for further research would be to explore how different paradigm components (artefacts and techniques) relate to generality. The possibility of developing much more general systems has often been a reason for concern about AI safety  [1] ; as a possibility which raises much larger, more critical risks  [6] . Considering which AI techniques and artefacts are more likely to lead to or be associated with more advanced, general, systems can therefore enable us to think about the scale of risks they may pose. If we are primarily concerned with safety issues associated with greater generality in systems, techniques involving transfer learning, curriculum learning, meta-learning, and other approaches looking for broader task coverage -which we have included in the 'general machine learning' category-may be particularly important areas to pay attention to  [38] . When we look at the artefacts, many views of a general AI system are typically associated with the 'agent' artefact. However, there is no reason to believe that providers, optimisers, extenders, etc., cannot become more general in the future, at least if we understand generality as autonomously covering more and more tasks. It is also worth noting that the prominence of an AI paradigm in the research literature should not necessarily be the main factor used to prioritise work on safety issues. Whether a paradigm results in societal applications with associated safety issues may be more related to sociological factors than technology itself. The extent to which a paradigm raises important safety issues is then associated with widespread use rather than the number of researchers who work on it (though these two things may be correlated). For instance, personal assistants are ubiquitous today, raising various safety and ethics issues, but the underlying technological advances do not necessarily correspond to a particularly prominent paradigm in AI research (reflected in the fact that personal assistants are more popular in media articles -the right plot of figure  2 -than in research papers). On the other hand, adversarial examples may be crucial for understanding the technical limitations of deep learning, but not so indicative of real-world risks  [16] . To identify and prioritise important safety issues in future, therefore, more analysis of the techniques and artefacts most likely to result in widespread use, taking into account sociological factors, will be important. The list of techniques and artefacts introduced in this paper can help identify new safety issues, starting by identifying possible links between these paradigms and safety issues that have not been made before. We need to be able to be more anticipatory about what kinds of problems might arise from different AI systems in future, while at the same time avoiding being too speculative  [2] . We hope that by thinking more explicitly about how safety issues relate to techniques and artefacts, AI safety research can both address challenges associated with current research avenues, and prepare us for a variety of potential future challenges. Figure 3 . 3 Figure3. Evolution of the relevance proportion for the period <PHONE>, using research-oriented (Research) and non-research (Media) sources from AI topics (everything like Figure2, except that here we only include the documents related to AI safety).
 TFigure 4 . 4 Figure 4. Left: Mapping between techniques and safety issues from research papers from AI topics (2010 to 2018). The width of each band connecting two elements represents the number of papers with both elements. Right: Same for artefacts.
 Table 1 . 1 The 14 categories of AI techniques we use in this paper. Given the relevance of machine learning today, and neural networks in particular, we retained several categories of machine learning techniques (general, declarative, and parametric ML, separate from neural networks). Technique category Some example subcategories and techniques Cognitive approaches Cognitive services and architectures, affective computing Declarative machine learning Rule learning, decision trees, program induction, ILP Evolutionary & nature-inspired methods Ant colony, LCS, genetic algorithms, DNA computing General machine learning Generative models, Gaussian models, AutoML, ensembles Heuristics & combinatorial optimization SAT solver, constraint satisfaction, Monte Carlo search Information retrieval Search engine, web mining, information extraction, Knowledge representation and reasoning Semantic nets, CBR, logics, commonsense reasoning Multiagent systems & game theory Distributed problem solving, cooperation, negotiation, Natural language processing Topic segmentation, parsing, question answering Neural networks Perceptron, convolutional network, GAN, RNN Parametric machine learning Support vector machines, kmeans, mixtures, LReg Planning & scheduling Backward/ forward chaining, action description language Probabilistic & Bayesian approaches Naive Bayes, probabilistic model, random field Reinforcement learning & MDPs Q-learning, deep RL, inverse RL
 Table 2 . 2 Ten different kinds of AI artefacts, key characteristics and some exemplars. Artefact Description Interface Dynamics Location Exemplars Interac. Funct. Central. Distrib. Coupled AGENT A system in a virtual or physical environment perceiving (observations and possibly rewards) and acting sensors and actuators • • a self-driving car, an autonomous drone, a robotic cleaner, a video game NPC ESTIMATOR A system representing an injective mapping from inputs to an extrapolated or estimated output digital objects • • a medical diagnostic model, an oracle, a face recognition system, a news feeder PROVIDER A system that waits for petitions that follow a protocol and responds with a solution for them command and objects • • a proof-editing and translation cognitive service, a voice processing system DIALOGUER A system that performs a conversation with a peer to extract information, explain things or change behaviour language • • virtual tutoring system, a chatter-bot sales assistant, healthcare assistant CREATOR A system that builds new things creatively following some patterns, constraints or examples specs. and/or examples • • a GAN generating faces, personalised email replier, simulated world generator EXTRACTOR A system that searches through a structured or unstructured knowledge base to retrieve some objects conditions and objects • • an expert system, a maths pundit, a web search engine, an infor. retrieval system ORGANISM A system that takes advantage of the environment or other systems to live, hybridise/mutate and reproduce resources • • • an intelligent computer worm or virus, artificial life, von Neumann probe OPTIMISER A system that finds an optimal combination of elements or parameters given some constraints constraints and objects • • • a train scheduling system, an electricity optimising system, theorem prover SWARM A system that behaves as the coordination of independent units through cooperation and/or competition sensors, actuators, communic. • • a multiagent network router, a drone swarm, a robotic warehouse, blockchain AI EXTENDER A system that regularly augments or compensates capabilities of another system (e.g., a human) commands, sensors, responses • • a memory assistant for people with dementia, a brain implant, a smart navigator 1.00 Research 1.00 Media 1.00 Research 1.00 Media 0.75 0.75 0.75 0.75 Share 0.50 0.50 Share 0.50 0.50 0.25 0.25 0.25 0.25 0.00 0.00 <PHONE> <PHONE> 2010 2020 2016 2018 0.00 0.00 <PHONE> <PHONE> 2010 2020 2016 2018 Neural Networks estimator agent provider optimiser extender dialoguer organism swarm extractor creator Figure 2. Evolution of the relevance proportion for the period <PHONE>, using research-oriented (Research) and non-research (Media) sources from AI topics. Left: 14 categories of techniques in Table1. Right: 10 categories of artefacts in Table2.
 Table 3 . 3 AI safety issue groups and their specific problems. AI Safety Issue Category Examples of specific AI problems included in the category Adversarial attacks Adversarial examples, white/black-box attacks, poisoning, policy manipulation. AI race & power AI race, monopolies, oligopolies. Authenticity & obfuscation Impersonation, authentication problems, fake media, plagiarism, obfuscation Autonomous weapons Military drones, killer robots, robotic weapon. Confinement problem AI boxing breach, containment breach. Corrigibility & interruptibility Switch-off button problems, rogue agents, self-preservation taking control. Dependency Cognitive atrophy, lack of independence, google effect, ... Interpretability Lack of intelligibility, need for explanation. Malicious use Malign uses of AI, malicious control, hacking. Manipulation Nudging, fake news, manipulative agents. Misuse & negligence AI misuse, negligent use. Moral dilemma Moral machine issues, utilitarian ethics problems, choosing ethical preferences. Moral perception & machine rights Robot rights recognition, moral status disagreement, uncanny valley. Privacy & integrity Inconsistency, private access breach, GDPR violation. Problem shift Distributional shift, concept drift, lack of generality, distribution overfitting. Reliability & robustness Error intolerance, robustness issues, reliability problems. Reward problems Honeypot problem, reward corruption, tripwire issues, tampering, wireheading. Safe exploration & side effects Negative side effects, unsafe exploration, uncontrolled impact. Scalable supervision Supervision costs, human-in-the-loop issues, sparse rewards. Self-modification Unintended self-modification, uncontrolled self-improvement. Specification & value alignment Instrumental convergence (paperclip), resource stealing, misalignment. Trust, transparency & accountability Lack of transparency, lack of trust, untraceability.
			 Similar selections of keywords can be found in [30] , which focused on the venue keywords, and performed a cluster analysis on the AAAI2013 conference keyword set, proposing a new series of keywords which were adapted by AAAI2014; and [15] , where the authors focused on views expressed about topics linked to discussions about AI in the New York Times over a 30-year period in terms of public concerns as well as optimism. 5  The complete list of exemplars of techniques, artefacts and safety issues, as well as the source code used and high-resolution plots can be found at https://github.com/nandomp/AIParadigmsSafety.
			 https://aitopics.org/misc/about. 7  We consider research those documents from the sources: "AAAI Conferences", "AI Magazine", "arXiv.org Artificial Intelligence", "Communications of the ACM", "IEEE Computer", "IEEE Spectrum", "IEEE Spectrum Robotics Channel", "MIT Technology Review", "Nature", "New Scientist" and "Science".
			 24th European Conference on Artificial Intelligence -ECAI 2020 Santiago de Compostela, Spain
			 24th European Conference on Artificial Intelligence -ECAI 2020 Santiago de Compostela, Spain
			 24th European Conference on Artificial Intelligence -ECAI 2020 Santiago de Compostela, Spain


Abstract: It's far from clear that human values will shape an Earth-based space-colonization wave, but even if they do, it seems more likely that space colonization will increase total suffering rather than decrease it. That said, other people care a lot about humanity's survival and spread into the cosmos, so I think suffering reducers should let others pursue their spacefaring dreams in exchange for stronger safety measures against future suffering. In general, I encourage people to focus on making an intergalactic future more humane if it happens rather than making sure there will be an intergalactic future.

Risks of Astronomical Future Suffering

Epigraphs If we carry the green fire-brand from star to star, and ignite around each a conflagration of vitality, we can trigger a Universal metamorphosis. [...] Because of us [...] Slag will become soil, grass will sprout, flowers will bloom, and forests will spring up in once sterile places. 1 [...] If we deny our awesome challenge; turn our backs on the living universe, and forsake our cosmic destiny, we will commit a crime of unutterable magnitude. -Marshall T. Savage, The Millennial Project: Colonizing the Galaxy in Eight Easy  Steps, 1994  Let's pray that the human race never escapes from Earth to spread its iniquity elsewhere. -C.S. Lewis If you can't beat 'em, join 'em. -proverb 2 Humans values may not control the future Nick Bostrom's "The Future of Human Evolution"  (Bostrom, 2004)  describes a scenario in which human values of fun, leisure, and relationships may be replaced by hyper-optimized agents that can better compete in the Darwinian race to control our future light cone. The only way we could avert this competitive scenario, Bostrom suggests, would be via a "singleton"  (Bostrom, 2006) , a unified agent or governing structure that could control evolution. Of course, even a singleton may not carry on human values. Many naive AI agents that humans might build may optimize an objective function that humans find pointless. Or even if humans do maintain hands on the steering wheel, it's far from guaranteed that we can preserve our goals in a stable way across major self-modifications going forward. These factors suggest that even conditional on human technological progress continuing, the probability that human values are realized in the future may not be very large. Carrying out human values seems to require a singleton that's not a blind optimizer, that can stably preserve values, and that is shaped by designers who care about human values rather than selfish gain or something else. This is important to keep in mind when we imagine what future humans might be able to bring about with their technology. Some people believe that sufficiently advanced superintelligences will discover the moral truth and hence necessarily do the right things. Thus, it's claimed, as long as humanity survives and grows more intelligent, the right things will eventually happen. There are two problems with this view. First, Occam's razor militates against the existence of a moral truth (whatever that's supposed to mean). Second, even if such moral truth existed, why should a superintelligence care about it? There are plenty of brilliant people on Earth today who eat meat. They know perfectly well the suffering that it causes, but their motivational systems aren't sufficiently engaged by the harm they're doing to farm animals. The same can be true for superintelligences. Indeed, arbitrary intelligences in mind-space needn't have even the slightest inklings of empathy for the suffering that sentients experience.
 Some scenarios for future suffering Even if humans do preserve control over the future of Earth-based life, there are still many ways in which space colonization would multiply suffering. Following are some of them.
 Spread of wild animals Humans may colonize other planets, spreading suffering-filled animal life via terraforming. Some humans may use their resources to seed life throughout the galaxy, which some sadly consider a moral imperative.
 Sentient simulations Given astronomical  (Bostrom, 2003)  computing power, post-humans may run various kinds of simulations. These sims may include many copies of wild-animal life, most of which dies painfully shortly after being born. For example, a superintelligence aiming to explore the distribution of extraterrestrials of different sorts might run vast numbers of simulations  (Thiel, Bergmann and Grey, 2003)  of evolution on various kinds of planets. Moreover, scientists might run even larger numbers of simulations of organisms-that-might-have-been, exploring the space of minds. They may simulate decillions of reinforcement learners that are sufficiently self-aware as to feel what we consider conscious pain.
 Suffering subroutines It could be that certain algorithms (say, reinforcement agents (Tomasik, 2014)) are very useful in performing complex machine-learning computations that need to be run at massive scale by advanced AI. These subroutines might be sufficiently similar to the pain programs in our own brains that we consider them to actually suffer. But profit and power may take precedence over pity, so these subroutines may be used widely throughout the AI's Matrioshka brains.
 Black Swans The range of scenarios that we can imagine is limited, and many more possibilities may emerge that we haven't thought of or maybe can't even comprehend. 4 Even a human-controlled future is likely to increase suffering If I had to make an estimate now, I would give ~70% probability that if humans choose to colonize space, this will cause more suffering than it reduces on intrinsic grounds (ignoring compromise considerations discussed later). Think about how space colonization could plausibly reduce suffering. For most of those mechanisms, there seem to be countermechanisms that will increase suffering at least as much. The following sections parallel those above.
 Spread of wild animals David Pearce coined the phrase "cosmic rescue missions"  (Pearce, n.d.)  in referring to the possibility of sending probes to other planets to alleviate the wild extraterrestrial (ET) suffering they contain. This is a nice idea, but there are a few problems. • We haven't found any ETs yet, so it's not obvious there are vast numbers of them waiting to be saved from Darwinian misery. Contrast this with the possibilities for spreading wild-animal suffering: • Humans may spread life to many planets (e.g., Mars via terraforming, other Earth-like planets via directed panspermia). The number of planets that can support life may be appreciably bigger than the number that already have it. (See the discussion of f l in the Drake equation.) Moreover, the percentage of planets that can be converted into computers that could simulate wild-animal suffering might be close to 100%. • We already know that Earth-based life is sentient, unlike for ETs. • Spreading biological life is slow and difficult, but disbursing small life-producing capsules is easier than dispatching Hedonistic Imperative probes or berserker probes. Fortunately, humans might not support spread of life that much, though some do. For terraforming, there are survival pressures to do it in the near term, but probably directed panspermia is a bigger problem in the long term. Also, given that terraforming is estimated to require at least thousands of years, while human-level digital intelligence should take at most a few hundred years to develop, terraforming may be a moot point from the perspective of catastrophic risks, since digital intelligence doesn't need terraformed planets. While I noted that ETs are not guaranteed to be sentient, I do think it's moderately likely that consciousness is fairly convergent among intelligent civilizations. This is based on (a) suggestions of convergent consciousness among animals on Earth and (b) the general principle that consciousness seems to be useful for planning, manipulating images, self-modeling, etc. On the other hand, maybe this reflects the paucity of my human imagination in conceiving of ways to be intelligent without consciousness.
 Sentient simulations It may be that biological suffering is a drop in the bucket compared with digital suffering. The biosphere of a planet is less than Type I on the Kardashev scale; it uses a tiny sliver of all the energy of its star. Intelligent computations by a Type II civilization can be many orders of magnitude higher. So humans' sims could be even more troubling than their spreading of wild animals. Of course, maybe there are ETs running sims of nature for science or amusement, or of minds in general to study biology, psychology, and sociology. If we encountered these ETs, maybe we could persuade them to be more humane. I think it's likely that humans are more empathetic than the average civilization because 1. we seem much more empathetic than the average animal on Earth, probably in part due to parental impulses and in part due to trade, although presumably some of these factors would necessarily be true of any technologically advanced civilization 2. selection bias implies that we'll agree with our own society's morals more than those of a random other society because these are the values that we were raised with and that our biology impels us toward. Based on these considerations, it seems plausible that there would be room for improvement through interaction with ETs. Indeed, we should in general expect it to be possible for any two civilizations or factions to achieve gains from compromise if they have diminishing marginal utility with respect to amount of control exerted. In addition, there may be cheap Pareto improvements to be had purely from increased intelligence and better understanding of important considerations. That said, there are some downside risks. Posthumans themselves might create suffering simulations, and what's worse, the sims that post-humans run would be more likely to be sentient than those run by random ETs because post-humans would have a tendency to simulate things closer to themselves in mind-space. They might run nature sims for aesthetic appreciation, lab sims for science experiments, or pet sims for pets.
 Suffering subroutines Suffering subroutines may be a convergent outcome of any AI, whether human-inspired or not. They might also be run by aliens, and maybe humans could ask aliens to design them in more humane ways, but this seems speculative.
 Black Swans It seems plausible that suffering in the future will be dominated by something totally unexpected. This could be a new discovery in physics, neuroscience, or even philosophy more generally. Some make the argument that because we know so very little now, it's better for humans to stick around because of the "option value": If they later realize it's bad to spread, they can stop, but if they realize they should spread, they can proceed to reduce suffering in some novel way that we haven't anticipated. Of course, the problem with the "option value" argument is that it assumes future humans do the right things, when in fact, based on examples of speculations we can imagine now, it seems future humans would probably do the wrong things much of the time. For instance, faced with a new discovery of obscene amounts of computing power somewhere, most humans would use it to run oodles more minds, some nontrivial fraction of which might suffer terribly. In general, most sources of immense power are double-edged swords that can create more happiness and more suffering, and the typical human impulse to promote life/consciousness rather than to remove them suggests that negative and negative-leaning utilitarians are on the losing side. Still, waiting and learning more is plausibly Kaldor-Hicks efficient, and maybe there are ways it can be made Pareto efficient by granting additional concessions to suffering reducers as compensation.
 What about paperclippers? Above I was largely assuming a human-oriented civilization with values that we recognize. But what if, as seems mildly likely, Earth is taken over by a paperclip maximizer, i.e., an unconstrained automation or optimization process? Wouldn't that reduce suffering because it would eliminate wild ETs as the paperclipper spread throughout the galaxy, without causing any additional suffering? Maybe, but if the paperclip maximizer is actually generally intelligent, then it won't stop at tiling the solar system with paperclips. It will want to do science, perform lab experiments on sentient creatures, possibly run suffering subroutines, and so forth. It will require lots of intelligent and potentially sentient robots to coordinate and maintain its paperclip factories, energy harvesters, and mining operations, as well as scientists and engineers to design them. And the paperclipping scenario would entail similar black swans as a human-inspired AI. Paperclippers would presumably be less intrinsically humane than a "friendly AI", so some might cause significantly more suffering than a friendly AI, though others might cause less, especially the "minimizing" paperclippers, e.g., cancer minimizers or death minimizers. If the paperclipper is not generally intelligent, I have a hard time seeing how it could cause human extinction. In this case it would be like many other catastrophic risks -deadly and destabilizing, but not capable of wiping out the human race. 6 What if human colonization is more humane than ET colonization? If we knew for certain that ETs would colonize our region of the universe if Earth-originating intelligence did not, then the question of whether humans should try to colonize space becomes less obvious. As noted above, it's plausible that humans are more compassionate than a random ET civilization would be. On the other hand, human-inspired computations might also entail more of what we consider to count as suffering because the mind architectures of the agents involved would be more familiar. And having more agents in competition for the light cone might lead to dangerous outcomes. But for the sake of argument, suppose an Earthoriginating colonization wave would be better than the expected colonization wave of an ET civilization that would colonize later if we didn't do so. In particular, suppose that if human values colonized space, they would cause only −0.5 units of suffering, compared with −1 units if random ETs colonized space. Then it would seem that as long as the probability P of some other ETs coming later is bigger than 0.5, then it's better for humans to colonize and pre-empt the ETs from colonizing, since −0.5 > −1 • P for P > 0.5. However, this analysis forgets that even if Earthoriginating intelligence does colonize space, it's not at all guaranteed that human values will control how that colonization proceeds. Evolutionary forces might distort compassionate human values into something unrecognizable. Alternatively, a rogue AI might replace humans and optimize for arbitrary values throughout the cosmos. In these cases, humans' greater-than-average compassion doesn't make much difference, so suppose that the value of these colonization waves would be −1, just like for colonization by random ETs. Let the probability be Q that these non-compassionate forces win control of Earth's colonization. Now the expected values are −31 • Q + −0.5 • (1 − Q) for Earth-originating colonization versus −1 • P if Earth doesn't colonize and leaves open the possibility of later ET colonization. For concreteness, say that Q = 0.5. (That seems plausibly too low to me, given how many times Earth has seen overhauls of hegemons in the past.) Then Earth-originating colonization is better if and only if −1 • 0.5 + −0.5 • 0.5 > −1 • P −0.75 > −1 • P P > 0.75. Given uncertainty about the Fermi paradox and Great Filter, it seems hard to maintain a probability greater than 75% that our future light cone would contain colonizing ETs if we don't ourselves colonize, although this section presents an interesting argument for thinking that the probability of future ETs is quite high. What if rogue AIs result in a different magnitude of disvalue from arbitrary ETs? Let H be the expected harm of colonization by a rogue AI. Assume ETs are as likely to develop rogue AIs as humans are. Then the disvalue of Earth-based colonization is H • Q + (−0.5) • (1 − Q), and the harm of ET colonization is P • (H • Q + (−1) • (1 − Q)). Again taking Q = 0.5, then Earth-based colonization has better expected value if where the inequality flips around when we divide by the negative number (H − 1). Figure  1  represents a plot of these threshold values for P as a function of H. H • 0.5 + −0.5 • 0.5 > P • (H • 0.5 + −1 • 0.5) H − 0.5 > P • (H − 1) P > (H − 0.5) (H − 1) , Even if H = 0 and a rogue AI caused no suffering, it would still only be better for Earth-originating intelligence to colonize if P > 0.5, i.e., if the probability of ETs colonizing in its place was at least 50%. These calculations involve many assumptions, and it could turn out that Earth-based colonization has higher expected value given certain parameter values. This is a main reason I maintain uncertainty as to the sign of Earth-based space colonization. However, this whole section was premised on humaninspired colonization being better than ET-inspired colonization, and the reverse might also be true, since computations of the future are more likely to be closer to what we most value and disvalue if humans do the colonizing.
 Why we should remain cooperative If technological development and space colonization seem poised to cause astronomical amounts of suffering, shouldn't we do our best to stop them? Well, it is worth having a discussion about the extent to which we as a society want these outcomes, but my guess is that someone will continue them, and this would be hard to curtail without extreme measures. Eventually, those who go on developing the technologies will hold most of the world's power. These people will, if only by selection effect, have strong desires to develop AI and colonize space. Resistance might not be completely futile. There's some small chance that suffering reducers could influence society in such a way as to prevent space colonization. But it would be better for suffering reducers, rather than fighting technologists, to compromise with them: We'll let you spread into the cosmos if you give more weight to our concerns about future suffering. Rather than offering a very tiny chance of complete victory for suffering reducers, this cooperation approach offers a higher chance of getting an appreciable fraction of the total suffering reduction that we want. In addition, compromise means that suffering reducers can also win in the scenario ( 30% likely in my view) that technological development does prevent more suffering than it causes even apart from considerations of strategic compromise with other people. Ideally these compromises would take the form of robust bargaining arrangements. Some examples are possible even in the short term, such as if suffering reducers and space-colonization advocates agree to cancel opposing funding in support of some commonly agreed-upon project instead. The strategic question of where to invest resources to advance your values at any given time amounts to a prisoner's dilemma with other value systems, and because we repeatedly make choices about where to invest, what stances to adopt, and what policies to push for, these prisoner's dilemmas are iterated. In Robert Axelrod's tournaments on the iterated prisoner's dilemma, the best-performing strategies were always "nice," i.e., not the first to defect. Thus, suffering reducers should not be the first to defect against space colonizers. Of course, if it seems that space colonizers show no movement toward suffering reduction, then we should also be "provocable" to temporary defection until the other side does begin to recognize our concerns. We who are nervous about space colonization stand a lot to gain from allying with its supportersin terms of thinking about what scenarios might happen and how to shape the future in better directions. We also want to remain friends because this means pro-colonization people will take our ideas more seriously. Even if space colonization happens, there will remain many sub-questions on which suffering reducers want to have a say: e.g., not spreading wildlife, not creating suffering simulations/subroutines, etc. We want to make sure suffering reducers don't become a despised group. For example, think about how eugenics is more taboo because of the Nazi atrocities than it would have been otherwise. Antitechnology people are sometimes smeared by association with the Unabomber. Animal supporters can be tarnished by the violent tactics of a few, or even by the antics of PETA. We need to be cautious about something similar happening for suffering reduction. Most people already care a lot about preventing suffering, and we don't want people to start saying, "Oh, you care about preventing harm to powerless creatures? What are you, one of those suffering reducers?" where "suffering reducers" has become such a bad name that it evokes automatic hatred. So not only is cooperation with colonization supporters the more promising option, but it's arguably the only net-positive option for us. Taking a more confrontational stance risks hardening the opposition and turning people away from our message. Remember, preventing future suffering is something that everyone cares about, and we shouldn't erode that fact by being excessively antagonistic. 8 Possible upsides to an intelligent future
 Black swans that don't cut both ways Many speculative scenarios that would allow for vastly reducing suffering in the multiverse would also allow for vastly increasing it: When you can decrease the number of organisms that exist, you can also increase the number, and those who favor creating more happiness / life / complexity / etc. will tend to want to push for the increasing side. However, there may be some black swans that really are one-sided, in the sense that more knowledge is most likely to result in a decrease of suffering. For example: We might discover that certain routine physical operations map onto our conceptions of suffering. People might be able to develop ways to re-engineer those physical processes to reduce the suffering they contain. If this could be done without a big sacrifice to happiness or other values, most people would be on board, assuming that present-day values have some share of representation in future decisions. This may be a fairly big deal. I give nontrivial probability (maybe ~10%?) that I would, upon sufficient reflection, adopt a highly inclusive view of what counts as suffering, such that I would feel that significant portions of the whole multiverse contain suffering-dense physical processes. After all, the mechanics of suffering can be seen as really simple when you think about them a certain way, and as best I can tell, what makes animal suffering special are the bells and whistles that animal sentience involves over and above crude physics -things like complex learning, thinking, memory, etc. But why can't other physical objects in the multiverse be the bells and whistles that attend suffering by other physical processes? This is all very speculative, but what understandings of the multiverse our descendants would arrive at we can only begin to imagine right now.
 Valuing reflection If we care to some extent about moral reflection on our own values, rather than assuming that suffering reduction of a particular flavor is undoubtedly the best way to go, then we have more reason to sup-port a technologically advanced future, at least if it's reflective. In an idealized scenario like coherent extrapolated volition (CEV)  (Yudkowsky, 2004) , say, if suffering reduction was the most compelling moral view, others would see this fact. 2 Indeed, all the arguments any moral philosopher has made would be put on the table for consideration (plus many more that no philosopher has yet made), and people would have a chance to even experience extreme suffering, in a controlled way, in order to assess how bad it is compared with other things. Perhaps there would be analytic approaches for predicting what people would say about how bad torture was without actually torturing them to find out. And of course, we could read through humanity's historical record and all the writings on the Internet to learn more about what actual people have said about torture, although we'd need to correct for will-to-live bias and deficits of accuracy when remembering emotions in hindsight. But, importantly, in a CEV scenario, all of those qualifications can be taken into account by people much smarter than ourselves. Of course, this rosy picture is not a likely future outcome. Historically, forces seize control because they best exert their power. It's quite plausible that someone will take over the future by disregarding the wishes of everyone else, rather than by combining and idealizing them. Or maybe concern for the powerless will just fall by the wayside, because it's not really adaptive for powerful agents to care about weak ones, unless there are strong, stable social pressures to do so. This suggests that improving prospects for a reflective, tolerant future may be an important undertaking. Rather than focusing on whether or not the future happens, I think it's more valuable for suffering reducers to focus on making the future better if it happens -by encouraging compromise, moral reflectiveness, philosophical wisdom, and altruism, all of which make everyone better off in expectation. Figure 1 : 1 Figure 1: Plot of threshold values for P as a function of H
			 Because nature contains such vast amounts of suffering, I would strongly dislike such a project. I include this quotation for rhetorical effect and to give a sense of how others see the situation.
			 Of course, what's compelling to idealized-me would not necessarily be compelling to idealized-you. Value divergences may


Abstract: Artificial intelligence tools can help save lives in a pandemic. However, the need to implement technological solutions rapidly raises challenging ethical issues. We need new approaches for ethics with urgency, to ensure AI can be safely and beneficially used in the COVID-19 response and beyond.

Asaf Tzachor
Jess Whittlestone
Lalitha Sundaram
Seán Ó Héigeartaigh

Artificial intelligence in a crisis needs ethics with urgency

T he novel coronavirus pandemic (COVID-19) is the largest global crisis in a generation, hitting the world at a time when artificial intelligence (AI) is showing potential for widespread real-world application. We are currently seeing a rapid increase in proposals for how AI can be used in many stages of pandemic prevention and response. AI can aid in detecting, understanding and predicting the spread of disease, which can provide early warning signs and inform effective interventions  1  . AI may improve the medical response to the pandemic in several ways: supporting physicians by automating aspects of diagnosis  2  , prioritizing healthcare resources  3  , and improving vaccine and drug development  4  . AI also has potential applications beyond immediate response, such as in combating online misinformation about COVID-19  5  . The current crisis presents an unprecedented opportunity to leverage AI for societal benefit. However, the urgency with which new technologies must be deployed raises particularly challenging ethical issues and risks. There is growing concern that the use of AI and data in response to COVID-19 may compromise privacy and civil liberties by incentivizing the collection and processing of large amounts of data, which may often be private or personal  6  . More broadly, although AI clearly has a great deal to offer, we must be careful not to overestimate its potential. Its efficacy will heavily depend on the reliability and relevance of the data available. With the worldwide spread of COVID-19 occurring so quickly, obtaining sufficient data for accurate AI forecasting and diagnosis is challenging. Even where AI models are strictly speaking accurate, they may have differential impacts across subpopulations, with harmful consequences that are difficult to predict in advance  7  . A further concern is that the lack of transparency in AI systems used to aid decision-making around COVID-19 may make it near impossible for the decisions of governments and public officials to be subject to public scrutiny and legitimation  8  . Finally, the current crisis may have longer-term impacts on public trust and norms around the use of AI in society. How these develop will depend on perceptions of how successful and responsible use of AI to address COVID-19 is.
 The challenge of ethics in a crisis Robust ethics and risk assessment processes are needed to ensure AI is used responsibly in response to COVID-19. However, implementing these at a time of crisis is far from straightforward, especially where new technologies need to be deployed at unprecedented speed and scale. For example, forecasting models have to be available at the early stages of disease spread and make use of all possible data to productively inform policy interventions. Current processes for ethics and risk assessment around uses of AI are still relatively immature, and the urgency of a crisis highlights their limitations. Much work in AI ethics in recent years has focused on developing high-level principles, but these principles say nothing about what to do when principles come into conflict with one another  9  . For example, principles do not tell us how to balance the potential of AI to save lives (the principle of 'beneficence') against other important values such as privacy or fairness. One common suggestion for navigating such tensions is through engagement with diverse stakeholder groups, but this may be difficult to enact with sufficient speed at times of crisis. When new technologies may pose unknown risks, we would ordinarily try to introduce them in gradual, iterative ways, allowing time for issues to be identified and addressed. In the context of a crisis, however, there is a stark trade-off between a cautious approach and the need to deploy technological solutions at scale. For example, there may be pressure to rely on systems with less human oversight and potential for override due to staff shortages and time pressures, but this must be carefully balanced against the risk of failing to notice or override crucial failures. This does not mean that ethics should be neglected at times of crisis. It only emphasizes that we must find ways to conduct ethical review and risk assessment with the same urgency that motivates the development of AI-based solutions.
 Doing ethics with urgency We suggest that ethics with urgency must at a minimum incorporate the following components: (1) the ability to think ahead rather than dealing with problems reactively, (2) more robust procedures for assuring the behaviour and safety of AI systems, and (3) building public trust through independent oversight. First, ethics with urgency must involve thinking through possible issues and risks as thoroughly as possible before systems are developed and deployed in the world. This need to think ahead is reflected in the notion of 'ethics by design': making ethical considerations part of the process of developing new applications of AI, not an afterthought  10  . For example, questions such as 'what data do we need and what issues might this raise?' and 'how do we build this model so that it is possible to interrogate key assumptions?' need to be considered throughout the development process. This means that experts in ethics and risk assessment need to be involved in teams developing AI-based solutions from the beginning, and much clearer guidelines are needed for engineers and developers to think through these issues. An ethics by design approach should also be supplemented with more extensive foresight work, looking beyond the more obvious and immediate ethical issues, and considering a wider range of longer-term and more systemic impacts. By synthesizing diverse sources of expertise, established foresight methodologies can be used to identify new comment risks and key uncertainties likely to shape the future, and use this to make better informed decisions today  11  . Second, where applications of AI are used at scale in safety-critical domains such as healthcare, ensuring the safety and reliability of those systems across a range of scenarios is of crucial importance. Finding ways to rapidly conduct robust testing and verification of systems will therefore be central to doing ethics with urgency. We suggest that the application of AI in crisis scenarios should in particular be heavily informed by research on best practices for the verification and validation of autonomous systems  12  . It may also be worthwhile for governments to fund further work on methods for establishing the reliability of machine learning systems across a range of circumstances, particularly where those systems may be deployed in high-stakes crisis scenarios. Third, an important aspect of ethics with urgency is building public trust in how AI is being used. If governments use AI systems in ways perceived to be either mistaken or problematically value-laden, this could result in a loss of public trust severe enough to drastically reduce support for beneficial uses of AI not just in this crisis, but also in the future. Building public trust around new uses of technology may be particularly challenging in crisis times, where the need to move fast makes it easier for governments to fall back on opaque and centralized forms of decision-making. Several analyses of past pandemics have argued that transparency and public scrutiny are essential for maintaining public trust  13  . An independent oversight body, responsible for reviewing any potential risks and ethical issues associated with new technologies and producing publicly available reports, could help ensure public transparency. This oversight body could, among other approaches, make use of techniques such as 'red teaming' to rigorously challenge systems and their assumptions, unearthing any limitations and biases in the applications being proposed  14  . Red teaming is widely used in security settings, but can be applied broadly: at its core, red teaming is a way of challenging the blind spots of a team by explicitly looking for flaws from an outsider or adversarial perspective. As well as allowing developers to identify and fix issues before deployment, such processes could help assure public stakeholders that the interests and values of different groups are being thoroughly considered, and that all eventualities are prepared for.
 conclusion As the COVID-19 pandemic illustrates, times of crisis can necessitate rapid deployment of new technologies in order to save lives. However, this urgency both makes it more likely that ethical issues and risks will arise, and makes them more challenging to address. Rather than neglecting ethics, we must find ways to do ethics with urgency too. We strongly encourage technologists, ethicists, policymakers and healthcare professionals to consider how ethics can be implemented at speed in the ongoing response to the COVID-19 crisis. If ethical practices can be implemented with urgency, the current crisis could provide an opportunity to drive greater application of AI for societal benefit, and to build public trust in such applications. ❐ Asaf Tzachor 1 ✉ , Jess Whittlestone  2 , Lalitha Sundaram 1 and Seán Ó hÉigeartaigh 2 1 Centre for the Study of Existential Risk, University of Cambridge, Cambridge, UK. 2 Leverhulme Centre for the Future of Intelligence, University of Cambridge, Cambridge, UK. ✉ e-mail: <EMAIL> Published online: 22 June 2020 https://doi.org/10.1038/s42256-<PHONE>-0
			 NaTure MachiNe iNTelligeNce | VOL 2 | JuLy 2020 | 365-366 | www.nature.com/natmachintell


/Users/janhendrikkirchner/code/2022/10/alignment-research-dataset/align_data/common/../../data/raw/nonarxiv_teis/Feasibility of Training an AGI using Deep Reinforcement Learning, A Very Rough Estimate.tei.xml