Abstract: We propose a method for identifying early warning signs of transformative progress in artificial intelligence (AI), and discuss how these can support the anticipatory and democratic governance of AI. We call these early warning signs 'canaries', based on the use of canaries to provide early warnings of unsafe air pollution in coal mines. Our method combines expert elicitation and collaborative causal graphs to identify key milestones and identify the relationships between them. We present two illustrations of how this method could be used: to identify early warnings of harmful impacts of language models; and of progress towards high-level machine intelligence. Identifying early warning signs of transformative applications can support more efficient monitoring and timely regulation of progress in AI: as AI advances, its impacts on society may be too great to be governed retrospectively. It is essential that those impacted by AI have a say in how it is governed. Early warnings can give the public time and focus to influence emerging technologies using democratic, participatory technology assessments. We discuss the challenges in identifying early warning signals and propose directions for future work.

Artificial Canaries: Early Warning Signs for Anticipatory and Democratic Governance of AI

I. Introduction P rogress in artificial intelligence (AI) research has accelerated in recent years. Applications are already changing society  [1]  and some researchers warn that continued progress could precipitate transformative impacts  [2] -  [5] . We use the term "transformative AI" to describe a range of possible advances with potential to impact society in significant and hard-to-reverse ways  [6] . For example, future machine learning systems could be used to optimise management of safety-critical infrastructure  [7] . Advanced language models could be used in ways that corrupt our online information ecosystem  [8]  and future advances in AI systems could trigger widespread labour automation  [9] . There is an urgent need to develop anticipatory governance approaches to AI development and deployment. As AI advances, its impacts on society will become more profound, and some harms may be too great to rely on purely 'reactive' or retrospective governance. Anticipating future impacts is a challenging task. Experts show substantial disagreement about when different advances in AI capabilities should be expected  [10] ,  [11] . Policy-makers face challenges in keeping pace with technological progress: it is difficult to foresee impacts before a technology is deployed, but after deployment it may already be too late to shape impacts, and some harm may already have been done  [12] . Ideally, we would focus preventative, anticipatory efforts on applications which are close enough to deployment to be meaningfully influenced today, but whose impacts we are not already seeing. Finding 'early warning signs' of transformative AI applications can help us to do this. Early warning signs can also help democratise AI development and governance. They can provide time and direction for much-needed public discourse about what we want and do not want from AI. It is not enough for anticipatory governance to look out for supposedly 'inevitable' future impacts. We are not mere bystanders in this AI revolution: the futures we occupy will be futures of our own making, driven by the actions of technology developers, policymakers, civil society and the public. In order to prevent foreseeable harms towards those people who bear the effects of AI deployments, we must find ways for AI developers to be held accountable to the society which they are embedded in. If we want AI to benefit society broadly, we must urgently find ways to give democratic control to those who will be impacted. Our aim with identifying early warning signs is to develop anticipatory methods which can prompt a focussed civic discourse around significant developments and provide a wider range of people with the information they need to contribute to conversations about the future of AI. We present a methodology for identifying early warning signs of potentially transformative impacts of AI and discuss how these can feed into more anticipatory and democratic governance processes. We call these early warning signs 'canaries' based on the practice of using canaries to provide early warnings of unsafe air pollution in coal mines in the industrial revolution. Others before us have used this term in the context of AI to stress the importance of early warning signs  [13] ,  [14] , but this is the first attempt to outline in detail how such 'artificial canaries' might be identified and used. Our methodology is a prototype but we believe it provides an important first step towards assessing and then trialling the feasibility of identifying canaries. We first present the approach and then illustrate it on two high-level examples, in which we identify preliminary warning signs of AI applications that could undermine democracy, and warning signs of progress towards high-level machine intelligence (HLMI). We explain why early warning signs are needed by drawing on the literature of participatory technology assessments, and we discuss the advantages and practical challenges of this method in the hope of preparing future research that might attempt to put this method into practise. Our theoretical exploration of a method to identify early warning signs of transformative applications provides a foundation towards more anticipatory, accountable and democratic governance of AI in practice.
 II. Related Work We rely on two main bodies of work. Our methodology for identifying canaries relies on the literature on forecasting and monitoring AI. Our suggestions for how canaries might be used once identified build on work on participatory technology assessments, which stresses a more inclusive approach to technology governance. While substantial research exists in both these areas, we believe this is the first piece of work that shows how they could feed into each other.
 A. AI Forecasting and Monitoring Over the past decade, an increasing number of studies have attempted to forecast AI progress. They commonly use expert elicitations to generate probabilistic estimates for when different AI advances and milestones will be achieved  [10] ,  [15] -  [17] . For example,  [16]  ask experts about when specific milestones in AI will be achieved, including passing the Turing Test or passing third grade. Both  [15]  and  [10]  ask experts to predict the arrival of high-level machine intelligence (HLMI), which the latter define as when "unaided machines can accomplish every task better and more cheaply than human workers". However, we should be cautious about giving results from these surveys too much weight. These studies have several limitations, including the fact that the questions asked are often ambiguous, that expertise is narrowly defined, and that respondents do not receive training in quantitative forecasting  [11] ,  [18] . Experts disagree substantially about when crucial capabilities will be achieved  [10] , but these surveys cannot tell us who (if anyone) is more accurate in their predictions. Issues of accuracy and reliability aside, forecasts focused solely on timelines for specific events are limited in how much they can inform our decisions about AI today. While it is interesting to know how much experts disagree on AI progress via these probabilistic estimates, they cannot tell us why experts disagree or what would change their minds. Surveys tell us little about what early warning signs to look out for or where we should place our focus today to shape the future development and impact of AI. At the same time, several projects, e.g.  [19] -  [22] , have begun to track and measure progress in AI. These projects focus on a range of indicators relevant to AI progress, but do not make any systematic attempt to identify which markers of progress are more important than others for the preparation of transformative applications. Time and attention for tracking progress is limited and it would be helpful if we were able to prioritise and monitor those research areas that are most relevant to mitigating risks. Recognising some of the limitations of existing work,  [23]  aims for a more holistic approach to AI forecasting. This framework emphasises the use of the Delphi technique  [24]  to aggregate different perspectives of a group of experts, and cognitive mapping methods to study how different milestones relate to one another, rather than to simply forecast milestones in isolation. We agree that such methods might address some limitations of previous work in both AI forecasting and monitoring. AI forecasting has focused on timelines for particularly extreme events, but these timelines are subject to enormous uncertainty and do not indicate near-term warning signs. AI measurement initiatives have the opposite limitation: they focus on near-term progress, but with little systematic reflection on which avenues of progress are, from a governance perspective, more important to monitor than others. What is needed are attempts to identify areas of progress today that may be particularly important to pay attention to, given concerns about the kinds of transformative AI systems that may be possible in future.
 B. Participatory Technology Assessments Presently, the impacts of AI are largely shaped by a small group of powerful people with a narrow perspective which can be at odds with public interest  [25] . Only a few powerful actors, such as governments, defence agencies, and firms the size of Google or Amazon, have the resources to conduct ambitious research projects. Democratic control over these research projects is limited. Governments retain discretion over what gets regulated, large technology firms can distort and avoid policies via intensive lobbying  [26]  and defence agencies may classify ongoing research. Recognising these problems, a number of initiatives over the past few years have emphasised the need for wider participation in the development and governance of AI  [27] -  [29] . In considering how best to achieve this, it is helpful to look to the field of science and technology studies (STS) which has long considered the value of democratising research progress  [30] ,  [31] . Several publications refer to the 'participatory turn'  [32]  in STS and an increasing interest in the role of the non-expert in technology development and assessment  [27] . More recently, in the spirit of "democratic experimentation"  [33] , various methods for civic participation have been developed and trialled, including deliberative polls, citizen juries and scenario exercises  [33] . With a widening conception of expertise, a large body of research on "participatory technology assessment" (PTA) has emerged, aiming to examine how we might increase civic participation in how technology is developed, assessed and rolled out. We cannot summarise this wideranging and complex body of work fully here. But we point towards some relevant pieces for interested readers to begin with.  [34]  and  [35]  present a typology of the methods and goals of participating, which now come in many forms. This means that assessments of the success of PTAs are challenging  [33]  and ongoing because different studies evaluate different PTA processes against different goals  [34] . Yet while scholars recognise remaining limitations of PTAs  [31] , several arguments for their advantages have been brought forward, ranging from citizen agency to consensus identification and justice. There are good reasons to believe that non-experts possess relevant end-user expertise. They often quickly develop the relevant subjectmatter understanding to contribute meaningfully, leading to better epistemic outcomes due to a greater diversity of views which result in a cancellation of errors  [36] ,  [37] . To assess the performance of PTAs scholars draw from case studies and identify best practices  [38] -  [40] . There is an important difference between truly participatory, democratically minded, technology assessments, and consultations that use the public to help legitimise a preconceived technology  [41] . The question of how to make PTAs count in established representational democracies is an ongoing challenge to the field  [31] ,  [33] . But  [42] , who present a recent example of collective technology policy-making, show that success and impact with PTAs is possible.  [40]  draw from 38 international case studies to extract best practices, building on  [38] , who showcase great diversity of possible ways in which to draw on the public. Comparing different approaches is difficult, but has been done  [39] ,  [43] .  [41]  present a conceptual framework with which to design and assess PTAs,  [44]  compares online versus offline methodologies and in  [35]  we find a typology of various design choices for public engagement mechanisms. See also  [45]  for a helpful discussion on how to determine the diversity of participants,  [46]  on what counts as expertise in foresight and  [30] ,  [32] ,  [47]  for challenges to be aware of in implementing PTAs. Many before us have noted that we need wider participation in the development and governance of AI, including by calling for the use of PTAs in designing algorithms  [48] ,  [49] . We see a need to go beyond greater participation in addressing existing problems with algorithms and propose that wider participation should also be considered in conversations about future AI impacts. Experts and citizens each have a role to play in ensuring that AI governance is informed by and inclusive of a wide range of knowledge, concerns and perspectives. However, the question of how best to marry expert foresight and citizen engagement is a challenging one. While a full answer to this question is beyond the scope of this paper, what we do offer is a first step: a proposal for how expert elicitation can be used to identify important warnings which can later be used to facilitate timely democratic debate. For such debates to be useful, we first need an idea of which developments on the horizon can be meaningfully assessed and influenced, for which it makes sense to draw on public expertise and limited attention. This is precisely what our method aims to provide.
 III. Identifying Early Warning Signs We believe that identifying canaries for transformative AI is a tractable problem and worth investing research effort in today. Engineering and cognitive development present a proof of principle: capabilities are achieved sequentially, meaning that there are often key underlying capabilities which, if attained, unlock progress in many other areas. For example, musical protolanguage is thought to have enabled grammatical competence in the development of language in homo sapiens  [50] . AI progress so far has also seen such amplifiers: the use of multi-layered non-linear learning or stochastic gradient descent arguably laid the foundation for unexpectedly fast progress on image recognition, translation and speech recognition  [51] . By mapping out the dependencies between different capabilities needed to reach some notion of transformative AI, therefore, we should be able to identify milestones which are particularly important for enabling many others -these are our canaries. The proposed methodology is intended to be highly adaptable and can be used to identify canaries for a number of important potentially transformative events, such as foundational research breakthroughs or the automation of tasks that affect a wide range of jobs. Many types of indicators could be of interest and classed as canaries, including: algorithmic innovation that supports key cognitive faculties (e.g., natural language understanding); overcoming known technical challenges (such as improving the data efficiency of deep learning algorithms); or improved applicability of AI to economically-relevant tasks (e.g. text summarization). Given an event for which we wish to identify canaries, our methodology has three essential steps:  (1)  identifying key milestones towards the event; (2) identifying dependency relations between these milestones; and (3) identifying milestones which underpin many others as canaries. See Fig.  1  for an illustration. We here deliberately refrain from describing the method with too much specificity, because we want to stress the flexibility of our approach, and recognise that there is currently no one-fits-all approach in forecasting. The method will require adaptation to the particular transformative event in question, but each step of this method is suited for such specifications. We outline example adaptations of the method to particular cases. e. method presented in this paper is that it requires one to have already identi ed a particular transformative event of concer
 A. Identifying Milestones Via Expert Elicitation The first step of our methodology involves using traditional approaches in expert elicitation to identify milestones that may be relevant to the transformative event in question. Which experts are selected is crucial to the outcome and reliability of studies in AI forecasting. There are unavoidable limitations of using any form of subjective judgement in forecasting, but these limitations can be minimised by carefully thinking through the group selection. Both the direct expertise of individuals, and how they contribute to the diversity of the overall group, must be considered. See  [46]  for a discussion of who counts as an expert in forecasting. Researchers should decide in advance what kinds of expertise are most relevant and must be combined to study the milestones that relate to the transformative event. Milestones might include technical limitations of current methods (e.g. adversarial attacks) and informed speculation about future capabilities (e.g. common sense) that may be important prerequisites to the transformative event. Consulting across a wide range of academic disciplines to order such diverse milestones is important. For example, a cohort of experts identifying and ordering milestones towards HLMI should include not only experts in machine learning and computer science but also cognitive scientists, philosophers, developmental psychologists, evolutionary biologists, or animal cognition experts. Such a group combines expertise on current capabilities in AI, with expertise on key pillars of cognitive development and the order in which cognitive faculties develop in animals. Groups which are diverse (on multiple dimensions) are expected to produce better epistemic outcomes  [37] ,  [52] . We encourage the careful design and phrasing of questions to enable participants to make use of their expertise, but refrain from demanding answers that lie outside their area of expertise. For example, asking machine learning researchers directly for milestones towards HLMI does not draw on their expertise. But asking machine learning researchers about the limitations of the methods they use every day; or asking psychologists what human capacities they see lacking in machines today, draws directly on their day-to-day experience. Perceived limitations can be then be transformed into milestones. There are several different methods available for expert elicitation including surveys, interviews, workshops and focus groups, each with advantages and disadvantages. Interviews provide greater opportunity to tailor questions to the specific expert, but can be time-intensive compared to surveys and reduce the sample size of experts. If possible, some combination of the two may be ideal: using carefully selected semi-structured interviews to elicit initial milestones, followed-up with surveys with a much broader group to validate which milestones are widely accepted as being key.
 B. Mapping Causal Relations Between Milestones The second step of our methodology involves convening experts to identify causal relations between identified milestones: that is, how milestones may underpin, depend on, or affect progress towards other milestones. Experts should be guided in generating directed causal graphs, a type of cognitive map that elicits a person's perceived causal relations between components. Causal graphs use arrows to represent perceived causal relations between nodes, which in this case are milestones  [53] . This process primarily focuses on finding out whether or not a relationship exists at all; how precisely this relationship is specified can be adapted to the goals of the study. An arrow from A to B at minimum indicates that progress on A will allow for further progress on B. But this relationship can also be made more precise: in some cases indicating that progress on AI is necessary for progress on B, for example. The relationship between nodes may be either linear or nonlinear; again this can be specified more precisely if needed or known. Constructing and debating causal graphs can "help groups to convert tacit knowledge into explicit knowledge"  [53] . Causal graphs are used as decision support for individuals or groups, and are often used to solve problems in policy and management involving complex relationships between components in a system by tapping into experts' mental models and intuitions. We therefore suggest that causal graphs are particularly well-suited to eliciting experts' models and assumptions about the relationship between different milestones in AI development. As a method, causal graphs are highly flexible and can be adapted to the preferred level of detail for a given study: they can be varied in complexity and can be analysed both quantitatively and qualitatively  [54] ,  [55] . We neither exclude nor favour quantitative approaches here, due to the complexity and uncertainty of the questions around transformative events. Particularly for very high-level questions, quantitative approaches might not offer much advantage and might communicate a false sense of certainty. In narrower domains where there is more existing evidence, however, quantitative approaches may help to represent differences in the strength of relationships between milestones. [56] notes that there are no ready-made designs that will fit all studies: design and analysis of causal mapping procedures must be matched to a clear theoretical context and the goal of the study. We highlight a number of different design choices which can be used to adapt the process. As more studies use causal graphs in expert elicitations about AI developments, we can learn from the success of different design choices over time and identify best practices.  [53]  stress that interviews or collective brainstorming are the most accepted method for generating the data upon which to analyse causal relations.  [57]  list heuristics on how to manage the procedure of combining graphs by different participants, or see  [58]  for a discussion on evaluating different options presented by experts.  [59]  suggest visual, interactive tools to aid the process.  [56]  and  [60]  discuss approaches to analysing graphs and extracting the emergent properties, significant 'core' nodes as well as hierarchical clusters. Core or "potent" nodes are those that relate to many clusters in the graphs and thus have implications for connected nodes. In our proposed methodology, such potent nodes play a central role in pointing to canary milestones. For more detail on the many options on how to generate, analyse and use causal graphs we refer the reader to the volume of  [57] , or reviews such as  [53] ,  [59] . See  [55]  for an example of applying cognitive mapping to expert views on UK public policies; and  [61]  for group problem solving with causal graphs. We propose that identified experts be given instruction in generating either an individual causal graph, after which a mediated discussion between experts generates a shared graph; or that the groups of experts as a whole generates the causal graph via argumentation, visualisations and voting procedures if necessary. As  [62]  emphasises, any group of experts will have both shared and conflicting assumptions, which causal graphs aim to integrate in a way that approaches greater accuracy than that contained in any single expert viewpoint. The researchers are free to add as much detail to the final maps as required or desired. Each node can be broken into subcomponents or justified with extensive literature reviews.
 C. Identifying Canaries Finally, the resulting causal graphs can be used to identify nodes of particular relevance for progress towards the transformative event in question. This can be a node with a high number of outgoing arrows, i.e. milestones which unlock many others that are prerequisites for the event in question. It can also be a node which functions as a bottleneck -a single dependency node that restricts access to a subsequent highly significant milestone. See Fig.  2  for an illustration. Progress on these milestones can thus represent a 'canary', indicating that further advances in subsequent milestones will become possible and more likely. These canaries can act as early warning signs for potentially rapid and discontinuous progress, or may signal that applications are becoming ready for deployment. Experts identify nodes which unlock or provide a bottleneck for a significant number of other nodes (some amount of discretion from the experts/conveners will be needed to determine what counts as 'significant'). Of course, in some cases generating these causal graphs and using them to identify canaries may be as complicated as a full scientific research project. The difficulty of estimating causal relationships between future technological advances must not be underestimated. However, we believe it to be the case that each individual researcher already does this to some extent, when they chose to prioritise a research project, idea or method over another within a research paradigm. Scientists also debate the most fruitful and promising research avenues and arguably place bets on implicit maps of milestones as they pick a research agenda. The idea is not to generate maps that provide a perfectly accurate indication of warning signs, but to use the wisdom of crowds to make implicit assumptions explicit, creating the best possible estimate of which milestones may provide important indications of future transformative progress.
 IV. Using Early Warning Signs Once identified, canary milestones can immediately help to focus existing efforts in forecasting and anticipatory governance. Given limited resources, early warning signs can direct governance attention to areas of AI progress which are soon likely to impact society and which can be influenced now. For example, if progress in a specific area of NLP (e.g. sentiment analysis) serves as a warning sign for the deployment of more engaging social bots to manipulate voters, policymakers and regulators can monitor or regulate access and research on this research area within NLP. We can also establish research and policy initiatives to monitor and forecast progress towards canaries. Initiatives might automate the collection, tracking and flagging of new publications relevant to canary capabilities, and build a database of relevant publications. They might use prediction platforms to enable collective forecasting of progress towards canary capabilities. Foundational research can try to validate hypothesised relationships between milestones or illuminate the societal implications of different milestones. These forecasting and tracking initiatives can be used to improve policy prioritisation more broadly. For example, if we begin to see substantial progress in an area of AI likely to impact jobs in a particular domain, policymakers can begin preparing for potential unemployment in that sector with greater urgency. However, we believe the value of early warning signs can go further and support us in democratising the development and deployment of AI. Providing opportunities for participation and control over policy is a fundamental part of living in a democratic society. It may be especially important in the case of AI, since its deployment might indeed transform society across many sectors. If AI applications are to bring benefits across such wide-ranging contexts, AI deployment strategies must consider and be directed by the diverse interests found across those sectors. Interests which are underrepresented at technology firms are otherwise likely to bear the negative impacts. There is currently an information asymmetry between those developing AI and those impacted by it. Citizens need better information about specific developments and impacts which might affect them. Public attention and funding for deliberation processes is not unlimited, so we need to think carefully about which technologies to direct public attention and funding towards. Identifying early warning signs can help address this issue, by focusing the attention of public debate and directing funding towards deliberation practises that centre around technological advancements on the horizon. We believe early warning signs may be particularly well-suited to feed into participatory technology assessments (PTAs), as introduced earlier. Early warning signs can provide a concrete focal point for citizens and domain experts to collectively discuss concerns. Having identified a specific warning sign, various PTA formats could be suited to consult citizens who are especially likely to be impacted. PTAs come in many forms and a full analysis of which design is best suited to assessing particular AI applications is beyond the scope of this article. But the options are plenty and PTAs show much potential (see section 2). For example, Taiwan has had remarkable success and engagement with an open consultation of citizens on complex technology policy questions  [42] . An impact assessment of PTA is not a simple task, but we hypothesise that carefully designed, inclusive PTAs would present a great improvement over how AI is currently developed, deployed and governed. Our suggestion is not limited to governmental bodies. PTAs or other deliberative processes can be run by research groups and private institutions such as AI labs, technology companies and think tanks who are concerned with ensuring AI benefits all of humanity.
 V. Method Illustrations We outline two examples of how this methodology could be adapted and implemented: one focused on identifying warning signs of a particular societal impact, the other on warning signs of progress towards particular technical capabilities. Both these examples pertain to high-level, complex questions about the future development and impacts of AI, meaning our discussion can only begin to illustrate what the process of identifying canaries would look like, and what questions such a process might raise. Since the results are only the suggestions of the authors of this paper, we do not show a full implementation of the method whose value lies in letting a group of experts deliberate. As mentioned previously, the work of generating these causal maps will often be a research project of its own, and we will return later to the question of what level of detail and certainty is needed to make the resulting graphs useful.
 A. First Illustration: AI Applications in Voter Manipulation We show how our method could identify warning signs of the kind of algorithmic progress which could improve the effectiveness of, or reduce the cost of, algorithmic election manipulation. The use of algorithms in attempts to manipulate election results incur great risk for the epistemic resilience of democratic countries  [63] -  [65] . Manipulations of public opinion by national and commercial actors are not a new phenomenon.  [66]  details the history of how newly emerging technologies are often used for this purpose. But recent advances in deep learning techniques, as well as the widespread use of social media, have introduced easy and more effective mechanisms for influencing opinions and behaviour.  [8]  and  [67]  detail the various ways in which political and commercial actors incur harm to the information ecosystem via the use of algorithms. Manipulators profile voters to identify susceptible targets on social media, distribute micro-targeted advertising, spread misinformation about policies of the opposing candidate and try to convince unwanted voters not to vote. Automation plays a large role in influencing online public discourse. Publications like  [68] ,  [69]  note that manipulators use both human-run accounts and bots  [70]  or a combination of the two  [71] . Misinformation  [72]  and targeted messaging  [73]  can have transformative implications for the resilience of democracies and very possibility of collective action  [74] ,  [75] . Despite attempts by national and sub-national actors to apply algorithms to influence elections, their impact so far has been contested  [76] . Yet, foreign actors and national political campaigns will continue to have incentives and substantial resources to invest in such campaigns, suggesting their efforts are unlikely to wane in future. We may thus inquire what kinds of technological progress would increase the risk that elections can be successfully manipulated. We can begin this inquiry by identifying what technological barriers currently prevent full-scale election manipulation. We would identify those technological limitations by drawing on the expertise of actors who are directly affected by these bottlenecks. Those might be managers of online political campaigns and foreign consulting firms (as described in  [8] ), who specialise in influencing public opinion via social media, or governmental organisations across the world who comment on posts, target individual influencers and operate fake accounts to uphold and spread particular beliefs. People who run such political cyber campaigns have knowledge of what technological bottlenecks still constrain their influence on voter decisions. We recommend running a series of interviews to collect a list of limitations. This list might include, for example, that the natural language functionality of social bots is a major bottleneck for effective online influence (for the plausibility of this being an important technical factor see  [8] ). Targeted users often disengage from a chat conversation after detecting that they are exchanging messages with social bots. Low retention time is presumably a bottleneck for further manipulation, which suggests that improvements in natural language processing (NLP) would significantly reduce the cost of manipulation as social bots become more effective. We will assume, for the purpose of this illustration that NLP were to be identified as a key bottleneck. We would then seek to gather experts (e.g. in a workshop) who can identify and map milestones (or current limitations) in NLP likely to be relevant to improving the functionality of social bots. This will include machine learning experts who specialise in NLP and understand the technical barriers to developing more convincing social bots; as well as experts in developmental linguistics and evolutionary biology, who can determine suitable benchmarks and the required skills, and who understand the order in which linguistic skills are usually developed in animals. From these expert elicitation processes we would acquire a list of milestones in NLP which, if achieved, would likely lower the cost and increase the effectiveness of online manipulation. Experts would then order milestones into a causal graph of dependencies. Given the interdisciplinary nature of the question at hand, we suggest in this case that the graph should be directly developed by the whole group. A mediated discussion in a workshop context can help to draw out different connections between milestones and the reasoning behind them, ensuring participants do not make judgements outside their range of expertise. A voting procedure such as majority voting should be used if no consensus can be reached. In a final step, experts can highlight milestone nodes in the final graph which are either marked by many outgoing nodes or are bottlenecks for a series of subsequent nodes that are not accessed by an alternative pathway. These (e.g. sentiment analysis) are our canaries: areas of progress which serve as a warning sign of NLP being applied more effectively in voter manipulation. Having looked at how this methodology can be used to identify warning signs of a specific societal impact, we next illustrate a different application of the method in which we aim to identify warning signs of a research breakthrough.
 B. Second Illustration: High-level Machine intelligence We use this second example to illustrate in more detail what the process of developing a causal map might look like once initial milestones have been identified, and how canary capabilities can be identified from the map. We define high-level machine intelligence (HLMI) as an AI system (or collection of AI systems) that performs at the level of an average human adult on key cognitive measures required for economically relevant tasks. We choose to focus on HLMI since it is a milestone which has been the focus of previous forecasting studies  [10] ,  [15] , and which, despite the ambiguity and uncertain nature of the concepts, is interesting to attempt to examine, because it is likely to precipitate widely transformative societal impacts. To trial this method, we used interview results from  [11] . 25 experts from a diverse set of disciplines (including computer science, cognitive science and neuroscience) were interviewed and asked what they believed to be the main limitations preventing current machine learning methods from achieving the capabilities of HLMI. These limitations can be translated into 'milestones': capabilities experts believe machine learning methods need to achieve on the path to HLMI, i.e. the output of step 1 of our methodology. Having identified key milestones, step 2 of our methodology involves exploring dependencies between them using causal graphs. We use the software VenSim to illustrate hypothesised relationships between milestones (see Fig.  2 ). For example, we hypothesise that the ability to formulate, comprehend and manipulate abstract concepts may be an important prerequisite to the ability to account for unobservable phenomena, which is in turn important for reasoning about causality. This map of causal relations and dependencies was constructed by the authors alone, and is therefore far from definitive, but provides a useful illustration of the kind of output this methodology can produce. Based on this causal map, we can identify three candidates for canary capabilities:
 Representations that allow variable-binding and disentanglement: the ability to construct abstract, discrete and disentangled representations of inputs, to allow for efficiency and variable-binding. We hypothesise that this capability underpins several others, including grammar, mathematical reasoning, concept formation, and flexible memory. Flexible memory: the ability to store, recognise, and re-use memory and knowledge representations. We hypothesise that this ability would unlock many others, including the ability to learn from dynamic data, to learn in a continual fashion, and to update old interpretations of data as new information is acquired. Positing unobservables: the ability to recognise and use unobservable concepts that are not represented in the visual features of a scene, including numerosity or intentionality. We might tentatively suggest that these are important capabilities to track progress on from the perspective of anticipating HLMI.
 VI. Discussion and Future Directions As the two illustrative examples show, there are many complexities and challenges involved in putting this method into practice. One particular challenge is that there is likely to be substantial uncertainty in the causal graphs developed. This uncertainty can come in many forms. Milestones that are not well understood are likely to be composed of several sub-milestones. As more research is produced, the graph will be in need of revision. Some such revisions may include the addition of connections between milestones that were previously not foreseen, which in turn might alter the number of outgoing connections from nodes and turn them into potent nodes, i.e. 'canaries'. The process of involving a diversity of experts in a multi-stage, collaborative process is designed to reduce this uncertainty by allowing for the identification of nodes and relationships that are widely agreed upon and so more likely to be robust. However, considerable uncertainty will inevitably remain due to the nature of forecasting. The higher the level of abstraction and ambiguity in the events studied (like events such as HLMI, which we use for our illustration) the greater the uncertainty inherent in the map and the less reliable the forecasts will likely be. It will be important to find ways to acknowledge and represent this uncertainty in the maps developed and conclusions drawn from them. This might include marking uncertainties in the graph and taking this into account when identifying and communicating 'canary' nodes. Given the uncertainty inherent in forecasting, we must consider what kinds of inevitable misjudgements are most important to try to avoid. A precautionary perspective would suggest it is better to slightly overspend resources on monitoring canaries that turn out to be false positives, rather than to miss an opportunity to anticipate significant technological impacts. This suggests we may want to set a low threshold for what should be considered a 'canary' in the final stage of the method. The uncertainty raises an important question: will it on average be better to have an imperfect, uncertain mapping of milestones rather than none at all? There is some chance that incorrect estimates of 'canaries' could be harmful. An incorrect mapping could focus undue attention on some avenue of AI progress, waste resources or distract from more important issues. Our view is that it is nonetheless preferable to attempt a prioritisation. The realistic alternative is that anticipatory governance is not attempted or informed by scholars' individual estimates in an ad-hoc manner, which we should expect to be incorrect more often than our collective and structured expert elicitation. How accurate our method is can only be studied by trialling it and tracking its predictions as AI research progresses to confirm or refute the forecasts. Future studies are likely to face several trade-offs in managing the uncertainty. For example, a large and cognitively diverse expert group may be better placed to develop robust maps eventually, but this may be a much more challenging process than doing it with a smaller, less diverse group --making the latter a tempting choice (see  [45]  for a discussion of this trade-off). The study of broad and high-level questions (such as when we might attain HLMI or automate a large percentage of jobs) may be more societally relevant or intellectually motivating, but narrower studies focused on nearer-term, well-defined applications or impacts may be easier to reach certainty on. A further risk is that this method, intended to identify warning signs so as to give time to debate transformative applications, may inadvertently speed up progress towards AI capabilities and applications. By fostering expert deliberation and mapping milestones, it is likely that important research projects and goals are highlighted and the field's research roadmap is improved. This means our method must be used with caution. However, we do not believe this is a reason to abandon the approach, since these concerns must be balanced against the benefits of being able to deliberate upon and shape the impacts of AI in advance. In particular, we believe that the process of distilling information from experts in a way that can be communicated to wider society, including those currently underrepresented in debates about the future of AI, is likely to have many more benefits than costs. The idea that we can identify 'warning signs' for progress assumes that there will be some time lag between progress on milestones, during which anticipatory governance work can take place. Of course, the extent to which this is possible will vary, and in some cases, unlocking a 'canary' capability could lead to very rapid progress on subsequent milestones. Future work could consider how to incorporate assessment of timescales into the causal graphs developed, so that it is easier to identify canaries which warn of future progress while allowing time to prepare. Future work should also critically consider what constitutes relevant 'expertise' for the task of identifying canaries, and further explore ways to effectively integrate expert knowledge with the values and perspectives of diverse publics. Our method finds a role for the expert situated in a larger democratic process of anticipating and regulating emerging technologies. Expert judgement can thereby be beneficial to wider participation. However, processes that allow more interaction between experts and citizens could be even more effective. One limitation of the method presented in this paper is that it requires one to have already identified a particular transformative event of concern, but does not provide guidance on how to identify and prioritise between events. It may be valuable to consider how citizens that are impacted by technology can play a role in identifying initial areas of concern, which can then feed into this process of expert elicitation to address the concerns.
 VII. Conclusion We have presented a flexible method for identifying early warning signs, or 'canaries' in AI progress. Once identified, these canaries can provide focal points for anticipatory governance efforts, and can form the basis for meaningful participatory processes enabling citizens to steer AI developments and their impacts. Future work must now test this method by putting it into practice, which will more clearly reveal both benefits and limitations. Our artificial canaries offer a chance for forward-looking, democratic assessments of transformative technologies.
 Appendix It is worth noting there are apparent similarities and relationships between many of these milestones. For example, representation: the ability to learn abstract representations of the environment, seems closely related to variable binding: the ability to formulate place-holder concepts. The ability to apply learning from one task to another, crossdomain generalisation, seems closely related to analogical reasoning. Further progress in research will tell which of these are clearly separate milestones or more closely related notions. Flexible memory, as described by experts in our sample, is the ability to recognize and store reusable information, in a format that is flexible so that it can be retrieved and updated when new knowledge is gained. We explain the reasoning behind the labelled arrows in Fig.  2  (see Fig. • (B): the ability to reinterpret data in light of new information likely requires flexible memory, since it requires the ability to retrieve and alter previously stored information. • (C) and (E): to make use of dynamic and changing data input, and to learn continuously over time, an agent must be able to store, correctly retrieve and modify previous data as new data comes in. • (D): in order to plan and execute strategies in brittle environments with long delays between actions and rewards, an agent must be able to store memories of past actions and rewards, but easily retrieve this information and continually update its best guess about how to obtain rewards in the environment. • (F): analogical reasoning involves comparing abstract representations, which requires forming, recognising, and retrieving representations of earlier observations. Progress in flexible memory therefore seems likely to unlock or enable many other capabilities important for HLMI, especially those crucial for applying AI systems in real environments and more complex tasks. These initial hypotheses should be validated and explored in more depth by a wider range of experts. as well as the attendees of the workshop Evaluating Progress in AI at the European Conference on AI (Aug 2020) for recognizing the potential of this work. We particularly thank Carolyn Ashurst and Luke Kemp for their efforts and commentary on our drafts. FFig. 3 . 3 Fig.3. Extract of Fig.2, showing one candidate canary capability.
 Cognitive map of dependencies between milestones collected in expert elicitations. Arrows coloured in green signify those milestones that have most outgoing arrows. See appendix for description of each milestone and dependency relations between one 'canary' node and subsequent nodes. Object Permanence Uncertainty Estimation Meta-Learning, Architecture-Search Visual Answering estion Causality Theorising, Hypothesising Common Sense Reading Comprehension Grammer Hierarchical Decomposition Posit Unobservables Adverserial A acks Cross-Domain Generalisation Scalability Mathematical Reasoning Representation, Variable-Binding, Disentanglement Concept Formation Analogical Reasoning, Overfi ing E icient Learning Catastrophic Forge ing Flexible Memory Continual Learning Active Learning Environmental Pressure Reinterpretations Dynamic Data Bri le Environments Misguided Data Collection Context-Dependent Decisions Fig. 2.


Abstract: A new book by Robin Hanson, The Age of Em: Work, Love, and Life When Robots Rule the Earth, is reviewed. The Age of Em describes a future scenario in which human minds are uploaded into computers, becoming emulations or "ems". In the scenario, ems take over the global economy by running on fast computers and copying themselves to multitask. The book's core methodology is the application of current social science to this future scenario, a welcome change from the usual perspectives from physical science, computer science, and philosophy. However, in giving a wide tour of the em world, the book sometimes gets bogged down in details. Furthermore, while the book claims that the em takeover scenario would be a good thing for the world and thus should be pursued, its argument is unpersuasive. That said, the book offers by far the most detailed description of the em world available, and its scenario offers a rich baseline for future study of this important topic.

The Social Science of Computerized Brains -Review of The Age of Em: Work, Love, and Life When Robots Rule the Earth by Robin Hanson (Oxford University Press, 2016)

Introduction The Age of Em: Work, Love, and Life When Robots Rule the Earth by Robin Hanson discusses the possibility of "uploading" human minds into computers. An uploaded mind or "em" (short for "emulation) is able to think more or less like a regular biological human, but with properties of computer programs and data. Ems can be copied ad infinitum and sped up or slowed down by being placed on faster or slower hardware. These capabilities create radical possibilities, which is the book's focus. The book is by far the most detailed description of the em world available. Hanson, an economist with a background in physics and artificial intelligence (AI), brings a distinctive and valuable perspective to this important topic.  The Age of Em describes one em world scenario in detail, giving only brief attention towards the end of the book (Ch. 28) to variations of this scenario. The chosen scenario is rated as the most likely one, given that uploading occurs (p.34-37). The scenario is intended as a baseline that is representative of many possible scenarios; interested analysts can introduce variations to produce still other scenarios. The book states that "Such straightforward scenario construction seems taboo among many professional futurists today", who instead favor creating "a modest number of scenarios to cover a wide range of possibilities across key axes of uncertainty and disagreement, with each scenario being story-like, archetypical in representing clusters of relevant driving forces, and describing equilibrium rather than transitory situations" (p.35), citing  Schoemaker (1995) . It is certainly true that some futures studies feature sets of story-like equilibrium scenarios, and it may well be true that some futurists disprefer presenting a single scenario, covering transition periods. However, in my experience, futurists are open to a wide range of approaches, and I see no reason why they should reject that of The Age of Em. The bulk of the book consists of descriptions of the world in which the ems The scope is encyclopedic, with attention to everything from the social relations among ems to the resistance of em buildings to earthquakes. Indeed, the book may be best treated as an encyclopedia of a particular em scenario. Reading it straight through, it can get bogged down in details. Topics are also often presented in an unintuitive order. For example, the initial creation of the em world appears in subsections of chapters 12 and 28, instead of in its logical place at the beginning of the book. A more story-like structure would make it more readable and may reduce certain cognitive biases  (Steinhardt & Shapiro, 2015) . However, if you are wondering about what some aspect of an em world might look like, odds are good that a description can be found somewhere in the book. A central premise of the book's scenario is that the em world is organized around market activity. Ems exist because they provide valuable labor. Ems with different capabilities are assigned to different tasks; groups of ems are organized into functional teams; cities are built to collocate ems for enhanced productivity. Ems are paid for their services and can spend their earnings on leisure, though competition between ems drives wages down. While these basics are broadly similar to much of today's society, the nature of ems creates some radical differences. For starters, while some ems are housed in robotic bodies to perform physical-world tasks, most live and work disembodied in virtual reality worlds. This enables them to travel at great speeds via telecommunications lines. Additionally, ems can be sped up or slowed down as needed, causing them to experience the world (and act on it) thousands or millions of times faster or slower than biological humans. Finally, ems can be copied repeatedly, which enables em teams to perform extreme multitasking. The book even proposes the concept of "spurs", which are em copies that perform a brief task and are then terminated or retired. These novel capacities enable ems to vastly outperform biological human workers. Em labor thus displaces humans while creating rapid economic growth. The book proposes that those humans who own parts of the em economy "could retain substantial wealth", but everyone else is "likely to starve" (p.336). To my eyes, this seems overly optimistic: if ems are so much smarter than humans, and if they have such a profit motive, then surely they could figure out how to trick, threaten, or otherwise entice humans into giving up their wealth. Humans would likely die out or, at best, scrape by in whatever meager existence the ems leave them with. Given these dire prospects for humans, one might question whether it would be good to create ems in the first place. Unfortunately, the book does not consider this topic in any detail. The book is pro-em, even proposing to "subsidize the development of related technologies to speed the arrival of this transition [to the em era], and to subsidize its smoothness, equality, or transparency to reduce disruptions and inequalities in that transition" (p.375). But this position is tenuous at best and quite possibly dangerous. The book's methodology is notable for its emphasis on social science. Many people who write about topics like uploading are trained in physical science, computer science, or philosophy; they thus "don't appreciate that social scientists do in fact know many useful things" (p.34). Hanson's defense of social science in this context is welcome. The book's methodology is rooted in extrapolation of current social science to em world conditions. At times, the extrapolations seem strained. For example, a claim that economic benefits of urban agglomeration would continue in an em world (p.215) cites  Morgan (2014) , which is a popular media description of Amazon's large data centers. It is true that Amazon data centers are large, and it may well be true that em cities are large for similar reasons, but the latter does not necessarily follow from the former. One can sympathize with Hanson, who is trying to present the social science of a topic that virtually no social scientists have previously touched. Indeed, the breadth of research covered is impressive; the book quietly doubles as a good literature review. However, some of the language relating the literature to the em world could have been better finessed. In other stretches, Hanson's personal tastes are apparent. This is seen, for example, in discussions of combinatorial auctions and prediction markets (p.184-188), two schemes for adapting market mechanisms for social decision making. Prediction markets in particular are a longstanding interest of Hanson's. The book's discussion of these topics says little about the em world and seems mainly oriented towards promoting them for society today. The reader gets the impression that Hanson wishes society today was more economically efficient and rational in a certain sense, and that he has embedded his hopes for a better world into his vision of ems. In conclusion, The Age of Em is a thoughtful and original work of futures studies, bringing an insightful social science perspective to the topic of mind uploading. The book covers wide ground about the nature of the em world, offering a valuable resource for anyone interested in the topic, or about futures studies more generally. The book is accessibly written and could be read by undergraduates or even advanced high school students, though experts from any discipline will also find much of interest. The book is especially worthwhile for anyone who could benefit from an overview of contemporary social science research, which the book provides in abundance. The book is not flawless. One can disagree with some of the book's arguments, including some of the important ones. The author's personal perspective is apparent; different authors would often reach different conclusions about the same topic. The organization is often scattered, and some literature is misused. However, none of these flaws take away from the significant accomplishment of describing the em world in great detail. The book offers a starting point-indeed, a baselinefor the serious study of future uploading scenarios. One can only hope that additional scholars, especially in the social sciences, follow the book's lead in taking on this important future scenario. Futures j o u r n a l h o m e p a g e : w w w . e l s e v i e r . c o m / l o c a t e / f u t u r e s