Abstract: In this paper we introduce a methodology for identifying early warning signs of transformative progress in AI, to aid anticipatory governance and research prioritisation. We propose using expert elicitation methods to identify milestones in AI progress, followed by collaborative causal mapping to identify key milestones which underpin several others. We call these key milestones 'canaries' based on the colloquial phrase 'canary in a coal mine' to describe advance warning of an extreme event: in this case, advance warning of transformative AI. After describing and motivating our proposed methodology, we present results from an initial implementation to identify canaries for progress towards high-level machine intelligence (HLMI). We conclude by discussing the limitations of this method, possible future improvements, and how we hope it can be used to improve monitoring of future risks from AI progress.

Canaries in Technology Mines: Warning Signs of Transformative Progress in AI

INTRODUCTION Progress in artificial intelligence (AI) research has accelerated in recent years, and applications are already beginning to impact society  [9] [43]. Some researchers warn that continued progress could lead to much more advanced AI systems, with potential to precipitate transformative societal changes  [13]   [19]   [21] [27]  [39] . For example, advanced machine learning systems could be used to optimise management of safety-critical infrastructure  [33] ; advanced language models could be used to corrupt our online information ecosystem  [31] ; and AI systems could even gradually begin to replace a large portion of economically useful work  [17] . We use the term "transformative AI" to describe a range of possible advances in AI with potential to impact society in large and hard-to-reverse ways  [22] . Preparing for the future impacts of AI is challenging given considerable uncertainty about how AI systems will develop. There is substantial expert disagreement around when different advances in AI capabilities should be expected  [11]   [19]   [34] . Policy and regulation will likely struggle to keep up with the fast pace of technological progress  [12]   [15] [42], resulting in either stale, outdated regulation or policy paralysis  [3] . It would therefore be valuable to be able to identify 'early warning signs' of transformative AI progress, to enable more anticipatory governance, as well as better prioritisation of research and allocation of resources. We call these early warning signs 'canaries', based on the colloquial use of the phrase 'canary in a coal mine' to indicate advance warning of an extreme event. Our use of this term takes its inspiration from an article by Etzioni  [16] , in which he stresses the importance of identifying such canaries. We want to take this suggestion seriously and propose a systematic methodology for identifying such canaries. Many types of indicators could be of interest and classed as canaries, including: research progress towards key cognitive faculties (e.g., natural language understanding); overcoming known technical challenges (such as improving the data efficiency of deep learning algorithms); or improved applicability of AI to economically-relevant tasks (e.g. text summarization). What distinguishes canaries from general markers of AI progress (like those discussed in  [32]  or  [36] ) is that they indicate that particularly transformative impacts of AI may be on the horizon. Given that our definition of "transformative AI" is currently very broad, canaries are therefore defined relative to a specific form of transformative AI or impact. For example, we might identify canaries for human-level artificial intelligence; canaries for automation of a specific sector of work; or canaries for specific types of societal risks from AI. From a governance perspective, we are particularly interested in canaries which indicate that faster or discontinuous progress may be on the horizon, since the impacts of rapid progress would be especially difficult to manage once they begin to manifest. These therefore particularly warrant advanced preparation. In what follows, we describe and discuss a methodology for identifying canaries of progress in technological research. We focus on AI progress, but believe this method, once trialled and tested, could be applied to other areas of technological development. We motivate and describe the methodological approach, which combines expert elicitation and causal mapping, before presenting one implementation of this methodology to identify canaries for progress towards high-level machine intelligence (HLMI). After discussing potential canaries for HLMI specifically, we discuss how to make use of canaries in monitoring of AI progress and suggest how the limitations of this methodology might be addressed in future iterations of this work. advanced AI will be achieved  [5][19] [21]  [34] . For example, Baum et al.  [5]  survey experts on when four specific milestones in AI will be achieved: passing the Turing Test, performing Nobel quality work, passing third grade, and becoming superhuman. Both Müller and Bostrom  [34] , and Grace et al.  [19]  focus their survey questions around predicting the arrival of high-level machine intelligence (HLMI), which the latter define as being achieved when "unaided machines can accomplish every task better and more cheaply than human workers".
 METHODOLOGY
 Background and Motivation However, these studies have several limitations  [6]  that should make us cautious about giving their results too much weight. The experts surveyed in these studies have no experience in quantitative forecasting and receive no training before being surveyed, which likely renders their predictions unreliable  [10][41] . Issues of reliability aside, these forecasting studies are also limited in what they can tell us about future AI progress. They have little to say about impactful milestones on the path to advanced AI, let alone early-warning signs. Experts disagree substantially about when capabilities will be achieved  [11] [19] and without knowing who (if anyone) is more accurate in their predictions, these forecasts cannot easily inform decisions and prioritisation around AI today. Quantitative expert elicitations like these also do not tell us why different experts disagree, what kinds of progress might cause them to change their judgements, or what aspects they in fact do agree upon. While broad probability estimates for when advanced AI might be achieved are interesting, they tell us little about the path from where we are now, or what could be done today to shape the future development and impact of AI. At the same time, several research projects have begun to track and measure progress in AI  [7] [23]  [36] . These projects focus on a range of indicators relevant to AI progress, but do not make any systematic attempt to identify which markers of progress are most important for anticipating potentially transformative AI. Given limited time and resources for tracking progress in AI, it is crucial that we find ways to prioritise those measures that are most relevant to ensuring societal benefits and mitigating risks of AI. The approach we propose in this paper aims to address the limitations of both work on AI forecasting and on measuring progress in AI. In a sense, the limitations of these two bodies of work are complementary. The AI forecasting literature focuses on anticipating the most extreme impacts and advanced progress in AI, but is unable to say much about the warning signs or kinds of progress that will be important in the near future. AI measurement initiatives, conversely, monitor near-future progress in AI, but with little systematic prioritisation or reflection on what progress in different areas might mean for society and governance. What is needed, building on work in both these areas, are attempts to identify areas of progress today that may be particularly important to pay attention to, given concerns about the kinds of transformative AI systems that may be possible in future. Progress in these areas would serve as crucial warning signs -canaries, as well call them -suggesting more advance preparation for future AI systems and their impacts is needed. We believe that identifying canaries for transformative AI is a tractable problem and therefore worth investing considerable research effort in today. In both engineering and cognitive development, capabilities are achieved sequentially, meaning that there are often key underlying capabilities which, if attained, unlock progress in many other areas. For example, musical protolanguage is thought to have enabled grammatical competence in the development of language in homo sapiens  [8] . AI progress so far has also arguably seen such amplifiers: the use of multilayered non-linear learning or stochastic gradient descent arguably laid the foundation for unexpectedly fast progress on image recognition, translation and speech recognition  [29] . By mapping out the dependencies between different capabilities or milestones needed to reach some notion of transformative AI, therefore, we should be able to identify milestones which are particularly important for enabling many others -these are our canaries. This is the general idea behind our approach to identifying canaries, outlined in more detail in the following sections.
 Proposed Methodology The proposed methodology can be used to identify 'canaries' for any transformative event. In the case of AI, the focus might be on a transformative technology such as HLMI or AGI, a transformative application such as flexible robotics, or a transformative impact such as the automation of at least 50% of jobs. Given a transformative event, our methodology has three main steps:  (1)  identifying key milestones towards the event;  (2)  identifying dependency relations between these milestones; and (3) identifying milestones which underpin many others as 'canaries'.
 Identifying key milestones using expert elicitation Like other studies of AI progress, we rely on expert elicitation throughout this process. However, the reliability of expert elicitation studies depends on the proper selection and use of expertise. Though there are inevitable limitations of using any form of subjective judgement, no matter how expert, these limitations can be minimised with careful selection of experts and questions. We suggest carefully selecting experts with varied expertise relevant to the chosen question. For example, for identifying milestones towards human-level AI, the cohort should include experts in machine learning and computer science but also cognitive scientists, philosophers, developmental psychologists, evolutionary biologists, and animal cognition experts. This diverse group would bring together expertise on the current capabilities and limitations of AI, with expertise on key milestones in human cognitive development and the order in which cognitive faculties develop. We also encourage careful design and phrasing of questions to enable participants to make best use of their expertise. For example, rather than asking experts to identify specific milestones towards human-level AI, which is a question for which they are not trained, we might ask machine learning researchers about the limitations of the methods they use every day, or ask psychologists what important human capacities they see lacking in machines. There are several different methods available for expert elicitation: including surveys, interviews, workshops and focus groups, each with advantages and disadvantages  [2] . Interviews provide greater opportunity to tailor questions to the specific expert, but can be extremely time-intensive compared to surveys, making it difficult to consult a large number of experts. If possible, some combination of the two may be ideal: using carefully selected semi-structured interviews to elicit initial milestones, followed-up with surveys with a much broader group to validate which milestones are widely accepted as being key.
 Mapping dependencies between milestones using causal graphs The second step of our methodology involves convening experts to identify dependency relations between identified milestones: that is, which milestones may underpin, lead to, or depend on which others. Experts should be guided in generating directed causal graphs to represent perceived causal relations between milestones  [35] . Causal graphs show causal links between elements of a system, represented as nodes (elements) and arrows (causal links). A directed positive arrow from A and B indicates that A has a positive causal influence on B. Such causal maps have been used to support decision-making, structure knowledge, and improve visualisation of complex scenarios  [20] [25][26]  [28]  and are particularly useful for exploring and understanding possible futures, rather than aiming to predict a single future  [26] . They are easily modified and constructed collaboratively, and therefore are well-suited to helping us structure expert knowledge on dependencies between different technological milestones. Fuzzy cognitive maps (FCMs), a specific type of causal graph, may be a particularly useful method for our purposes. FCMs capture all benefits of causal mapping but can be extended into a quantitative model, and thus lend themselves to computerised simulations  [25] . This will not always be necessary, but given that our proposed method is applicable to many contexts, a flexible model is desirable. FCMs are well able to document non-linear interactions and experts' mental models of causal interactions because they can handle imprecise causal links. The variables (nodes) can take any state between 0 and 1 (hence 'fuzzy'), indicating the extent to which the variable is 'present'. When a variable changes its state, it affects all concepts that are causally dependent on it. FCMs have been used successfully in environmental science  [20]   [38] , strategic planning  [30] , and other areas  [25] , and have been recommended for use in futures studies, forecasting, and technology road mapping  [1]   [26] . In a workshop format, experts should be given brief training in causal graph methods or FCMs, and then break into groups to discuss dependencies between milestones. Each group should then collaboratively construct a directed causal graph or FCM to represent these relationships. Groups should be formed so as to maximise the variation of expertise in each group.
 2.2.3 Identifying canaries from causal graphs. Finally, the resulting causal graphs can be aggregated and analysed to identify canaries, by identifying the nodes with the highest number of outgoing arrows. The aggregation process should first focus on identifying commonalities between all graphs which can be shared in the final graph. Substantive disagreements may remain, which can be the subject of mediated discussion, with a voting process to decide on final aspects of the graph. Experts then identify nodes which they agree have significantly more outgoing nodes (some amount of discretion from the experts/conveners will be needed to determine what counts as 'significant'). Since nodes with a high density of outgoing arrows represent milestones which underpin many others, progress on these milestones can act as 'canaries', indicating that we may see further advances in many other areas in the near future. These canaries can therefore act as early warning signs for more rapid and potentially discontinuous progress, as well as for new applications becoming ready for deployment (depending on which exact capabilities they are likely to unlock).
 IMPLEMENTATION: CANDIDATE CANARIES FOR HLMI We describe a partial implementation of the proposed method to identify canaries for achieving high-level machine intelligence (HLMI). We define HLMI here as an AI system (or collection of AI systems) that performs at the level of an average human adult on key cognitive measures required for economically relevant tasks.  2  We interviewed experts about the limitations of current deep learning methods from the perspective of achieving HLMI, and used the findings to construct a causal graph of milestones. This allowed us to identify candidate canary capabilities. The results must be understood as preliminary, because the causal graphs were developed just by the authors, not a cohort of experts, and so have limited validity. However, this initial demonstration and preliminary findings will form the basis for a full study with a broader range of experts in future.
 Expert elicitation to identify milestones To identify key milestones for achieving HLMI, we interviewed 25 experts (using both a non-probabilistic, purposive sampling method and stratified sampling method, as described by  [12]  in chapter six). The sample covered experts in machine learning  (9) , computer science with specialisation in AI (  5 ), cognitive psychology (2), animal cognition (1), philosophy of mind and AI (3), mathematics , neuroscience (1), neuro-informatics (1), engineering (1). Interviewees came from both academia and industry, and were deliberately selected to be at a variety of career stages. We conducted individual, semi-structured interviews, with a set of core questions and themes to guide more open-ended discussion. Semi-structured interviews use an interview guide with core questions and themes to be explored in response to open-ended questions to allow interviewees to explain their position freely  [24] . Initial questions included: what do you believe deep learning will never be able to do? Do you see limitations of deep learning that others seem not to notice? In response to these and similar questions tailored to the interviewee's specific expertise, they were asked to name what they thought were the biggest limitations of current deep learning methods, from the perspective of achieving HLMI. All named limitations were collated, with shortened explanations, and translated into 'milestones', i.e. capabilities experts believe deep learning is yet to achieve on the path to HLMI. Table  1 . shows all milestones based on limitations, named by interviewees. Because we have maintained each interviewee's preferred terminology, several of the milestones listed may turn out to refer to the same or highly similar problems.
 Table 1. Limitations of deep learning as perceived and named by experts Causal reasoning: the ability to detect and generalise from causal relations in data. Common sense: having a set of background beliefs or assumptions which are useful across domains and tasks.
 Meta-learning: the ability to learn how to best learn in each domain. Architecture search: the ability to automatically choose the best architecture of a neural network for a task. Hierarchical decomposition: the ability to decompose tasks and objects into smaller and hierarchical sub-components.
 Cross-domain generalization: the ability to apply learning from one task or domain to another. Representation: the ability to learn abstract representations of the environment for efficient learning and generalisation. Variable binding: the ability to attach symbols to learned representations, enabling generalisation and re-use. Disentanglement: the ability to understand the components and composition of observations, and recombine and recognise them in different contexts. Analogical reasoning: the ability to detect abstract similarity across domains, enabling learning and generalisation. Concept formation: the ability to formulate, manipulate and comprehend abstract concepts. Object permanence: the ability to represent objects as consistently existing even when out of sight. Grammar: the ability to construct and decompose sentences according to correct grammatical rules. Reading comprehension: the ability to detect narratives, semantic context, themes and relations between characters in long texts or stories. Mathematical reasoning: the ability to develop, identify and search mathematical proofs and follow logical deduction in reasoning.
 Visual question answering: the ability to answer open-ended questions about the content and interpretation of an image. Uncertainty estimation: the ability to represent and consider different types of uncertainty. Positing unobservables: the ability to account for unobservable phenomena, particularly in representing and navigating environments. Reinterpretation: the ability to partially re-categorise, re-assign or reinterpret data in light of new information without retraining from scratch.
 Theorising and hypothesising: the ability to propose theories and testable hypotheses, understand the difference between theory and reality, and the impact of data on theories. Flexible memory: the ability to store, recognise and retrieve knowledge so that it can be used in new environments and tasks. Efficient learning: the ability to learn efficiently from small amounts of data. Interpretability: the ability for humans to interpret internal network dynamics so that researchers can manipulate network dynamics. Continual learning: the ability to learn continuously as new data is acquired. Active learning: the ability to learn and explore in self-directed ways. Learning from inaccessible data: the ability to learn in domains where data is missing, difficult or expensive to acquire. Learning from dynamic data: the ability to learn from a continually changing stream of data.
 Navigating brittle environments: the ability to navigate irregular, and complex environments which lack clear reward signals and short feedback loops. Generating valuation functions: the ability to generate new valuation functions immediately from scratch to follow newly-given rules. Scalability: the ability to scale up learning to deal with new features without needing disproportionately more data, model parameters, and computational power. Learning in simulation: the ability to learn all relevant experience from a simulated environment. Metric identification: the ability to identify appropriate metrics of success for complex tasks, such that optimising for the measured quantity accomplishes the task in the way intended. Conscious perception: the ability to experience the world from a firstperson perspective.
 Context-sensitive decision making: the ability to adapt decision-making strategies to the needs and constraints of a given time or context. It is worth noting there are apparent similarities and relationships between many of these milestones. For example, representation: the ability to learn abstract representations of the environment, seems closely related to variable binding: the ability to formulate place-holder concepts. The ability to apply learning from one task to another, cross-domain generalisation, seems closely related to analogical reasoning. Further progress in research will tell which of these are clearly separate milestones or more closely related notions.
 Causal graphs to identify dependencies between milestones Having identified key milestones, we explore dependencies between them using fuzzy cognitive maps (FCM). We focus on how capabilities enable, not inhibit, other capabilities, which means we use only positive influence arrows. FCMs are particularly well-suited to representing the uncertainty inherent in this analysis, as it assumes that each arrow could have a weight to represent varying levels of strength. In this analysis we have not specified the weights on connections, but adding these weights could be trialled with experts in the future. A previous survey  [5]  suggests that this endeavour is a highly uncertain one, finding that many different relationships between AI milestones seem plausible to experts. Our analysis does not claim nor aim to resolve this disagreement, but instead shows only one out of many possible mappings, to illustrate the use and value of FCMs in AI progress monitoring. We use the software VenSim (vensim.com) to illustrate the hypothesised relationships between perceived milestones in Figure  1 . For example, we hypothesise that the ability to formulate, comprehend and manipulate abstract concepts may be an important prerequisite for the ability to account for unobservable phenomena, which is in turn important for reasoning about causality. A positive influence arrow does not mean that achieving one milestone necessarily leads to another, but rather that progress on the first  This map was constructed by the authors, and is therefore far from definitive or the only possible way of representing dependencies between capabilities. However, this initial map does provide an important illustration of the kind of output this methodology should aim to achieve, and generates some initial hypotheses for relationships between milestones.
 Candidate Canary Capabilities Based on this causal map, we can identify two candidates for canary capabilities. The capabilities with the most outgoing arrows are: Symbol-like representations: the ability to construct abstract, discrete and disentangled representations of inputs, to allow for efficiency and variable-binding. We hypothesise that this capability underpins several others, including grammar, mathematical reasoning, concept formation, and flexible memory. Flexible memory: the ability to store, recognise, and re-use knowledge. We hypothesise that this ability would unlock many others, including the ability to learn from dynamic data, the ability to learn in a continual fashion, and the ability to learn how to learn. We therefore tentatively suggest that these are two important capabilities to track progress on from the perspective of anticipating HLMI. We discuss one such capability, flexible memory, in more detail below. Flexible memory, as described by experts in our sample, is the ability to recognize and store reusable information, in a format that is flexible so that it can be retrieved and updated when new knowledge is gained. We explain the reasoning behind the labelled arrows in figure  2 : • (a): compact representations are a prerequisite for flexible memory since storing high-dimensional input in memory requires compressed, efficient and thus abstract representations. • (B): the ability to reinterpret data in light of new information likely requires flexible memory, since it requires the ability to retrieve and alter previously stored information. • (C) and (E): to make use of dynamic and changing data input, and to learn continuously over time, an agent must be able to store, correctly retrieve and modify previous data as new data comes in. • (D): in order to plan and execute strategies in brittle environments with long delays between actions and rewards, an agent must be able to store memories of past actions and rewards, but easily retrieve this information and continually update its best guess about how to obtain rewards in the environment. • (F): analogical reasoning involves comparing abstract representations, which requires forming, recognising, and retrieving representations of earlier observations. Progress in flexible memory therefore seems likely to unlock or enable many other capabilities important for HLMI, especially those crucial for applying AI systems in real environments and more complex tasks. These initial hypotheses should be validated and explored in more depth by a wider range of experts.
 DISCUSSION
 Advantages We believe the proposed method for identifying canaries has many strengths and could be applied to a broad range of important questions about transformative AI systems and impacts. The general methodology of using expert elicitation to identify milestones and then causal mapping to elucidate dependencies between those milestones is extremely flexible, meaning it could be applied beyond AI to other fields of science and technology progress. The method can also be adapted to the preferred level of detail for a given study: causal graphs can be made arbitrarily complex  [18]  and can be analysed both quantitatively and qualitatively. With this method, it is possible to combine different types of expertise relating to milestones: including well-understood technical limitations of current methods, with informed speculation about unknown capabilities that may be important prerequisites to some transformative event. With early warning signs we can track progress towards canary milestones, or directly prepare for the transformative events that follow after it.
 Uses We envision that this methodology could be used to identify warning signs for a number of important potentially transformative events in AI progress, such as foundational research breakthroughs, the use of AI to automate scientific research, or the automation of tasks that affect a wide range of jobs. Once canaries have been identified for some transformative event, there are numerous ways we might use them to improve preparation for its impact, including by: • Automating the collection, tracking and flagging of new publications relevant to canary capabilities, and building a database of relevant publications (perhaps similar to that described by  [40] );
 Limitations and future directions This methodology nonetheless has some limitations which further iterations could seek to improve on. There may be a fundamental trade-off between the benefits of consulting a large, diverse group of experts -enabling more thorough and robust identification of relevant milestones -and the feasibility of reaching agreement upon a single causal map, and therefore agreeing upon canaries. Relatedly, if uncertainty about milestones is too high, it may be difficult for experts to agree on a single causal map or candidates for canaries: finding questions where there is enough uncertainty for this process to be useful, but not so much uncertainty that no agreement can be reached, may be a challenge in some cases. It will also be important to recognise any potential limitations of the specific sample of experts involved in the process: recognising that machine learning researchers may be biased towards emphasising the importance of areas they themselves work on, for example, or that non-computer scientists may often lack a full understanding of what current systems can and cannot do. In using FCMs to generate causal maps, it is not clear what level of detail and quantitative analysis will be most useful. In the implementation described here, we hypothesised relationships at a high level of abstraction and without quantitative analysis, due to the high level at which experts highlighted limitations in the first stage. The higher the level of abstraction, the more uncertain the mapping will be and the less useful it may be to indicate weights. It would be valuable for future work to explore various levels of abstractions, including a more detailed and quantitative analysis using more clearly-defined technical milestones, which could result in more precise forecasts and hypotheses. Finally, it is important to note that attempts to anticipate and understand progress in AI (or any other technology) are not independent of that progress itself. Better understanding of key milestones towards AGI, HLMI, or some other notion of transformative AI, does not just improve our ability to anticipate that progress, but may also improve our ability to make progress towards transformative AI. We must therefore be cautious in identifying 'canary' capabilities, to consider the potential risks of making progress on these capabilities, and to communicate and encourage consideration of these risks to those researchers driving forward AI development. likelihood of progress on other arrows it points to.
 Figure 1 . 1 Figure 1. Cognitive map of dependencies between milestones collected in expert elicitations. Arrows coloured in pink and green indicate capabilities that have significantly more outgoing arrows.
 Figure 2 . 2 Figure 2. Extract of Figure 1, showing one candidate canary capability.
 This work builds on two main bodies of existing literature: research on AI forecasting, and an emerging body of work on measuring AI progress.The AI forecasting literature generally uses expert elicitation to generate probabilistic estimates for when different types of 1st International Workshop on Evaluating Progress in Artificial Intelligence -EPAI 2020In conjunction with the 24th European Conference on Artificial Intelligence -ECAI 2020 Santiago de Compostela, Spain
 Conducting more in-depth research on the societal and governance implications of achieving canary milestones, and preparing governance responses for these milestones ahead of time. • Generating metrics and benchmarks for evaluating progress on canary capabilities; • Using prediction platforms such as Metaculus (ai.metaculus.com) to track and forecast progress on canary capabilities; • Conducting more focused expert elicitation, for example periodically consult experts on their updated forecasts (in the form of cumulative probability estimates) for when different milestones are achieved, or when they are presented with updated progress metrics on canary capabilities; • Conducting more in-depth research to empirically and theoretically investigate hypothesised relationships between milestones: for example, to what extent do improvements in memory structures lead to empirical improvements in performance in brittle environments? •
			 We use this definition, adapted from Grace et al., to highlight that what is key for saying HLMI has been reached is that AI has the cognitive ability to perform every task better than humans workers, not that it is in practice deployed to do so.


Abstract: There have been two major lines of research aimed at capturing resource-bounded players in game theory. The first, initiated by  Rubinstein (1986) , charges an agent for doing costly computation; the second, initiated by  Neyman (1985) , does not charge for computation, but limits the computation that agents can do, typically by modeling agents as finite automata. We review recent work on applying both approaches in the context of decision theory. For the first approach, we take the objects of choice in a decision problem to be Turing machines, and charge players for the "complexity" of the Turing machine chosen (e.g., its running time). This approach can be used to explain well-known phenomena like first-impression-matters biases (i.e., people tend to put more weight on evidence they hear early on) and belief polarization (two people with different prior beliefs, hearing the same evidence, can end up with diametrically opposed conclusions) as the outcomes of quite rational decisions. For the second approach, we model people as finite automata, and provide a simple algorithm that, on a problem that captures a number of settings of interest, provably performs optimally as the number of states in the automaton increases.

Joseph Y Halpern
Rafael Pass
Lior Seeman

Decision Theory with Resource-Bounded Agents

Introduction The standard approach to decision making, going back to  Savage (1954) , suggests that an agent should maximize expected utility. But computing the relevant probabilities might be difficult, as might computing the relevant utilities. And even in cases where the probabilities and utilities are not hard to compute, finding the action that maximizes expected utilities can be difficult. In this paper, we consider approaches to decision making that explicitly take computation into account. The idea of taking computation into account goes back at least to the work of  Good (1952)  and  Simon (1955) . It has been a prominent theme in the AI literature (see, e.g.,  Horvitz, 1987; Russell & Subramanian, 1995) . Our work has been inspired by two major lines of research aimed at capturing resource-bounded (i.e., computationally-bounded) agents in the game theory literature. The first, initiated by  Rubinstein (1986) , charges an agent for doing costly computation; the second, initiated by  Neyman (1985) , does not charge for computation, but limits the computation that agents can do, typically by modeling agents as finite automata. We review recent work on applying both approaches in the context of decision theory. We consider the first approach in the context of the framework of  Halpern and Pass (2010b) , which in turn is a specialization of the framework of  Halpern and Pass (2010c)  to the case of a single agent. The idea is the following: We assume that the agent can be viewed as choosing an algorithm (i.e., a Turing machine); with each Turing machine (TM) M and input, we associate its complexity. The complexity can represent, for example, the running time of M on that input, the space used, the complexity of M (e.g., how many states it has), or the difficulty of finding M (some algorithms are easier to think of than others). We deliberately keep the complexity function abstract, to allow for the possibility of representing a number of different intuitions. The agent's utility can then depend, not just on the payoff, but on the complexity. Thus, we can "charge" for the complexity of computation. Although this approach seems quite straightforward, we show that it can be used to explain well-known phenomena like first-impression-matters biases (i.e., people tend to put more weight on evidence they hear early on) and belief polarization (people with different prior beliefs, hearing the same evidence, can end up with diametrically opposed conclusions). We then consider the second approach: modeling agents as finite automata. Finite automata are a well-known basic model of computation. An automaton receives a sequence of inputs from the environment, and changes state in response to the input. We capture the fact that an agent is resource-bounded by limiting the number of states in the automaton. In the context of decision theory, this approach was first used by  Wilson (2002) . She examined a decision problem where an agent needs to make a single decision, whose payoff depends on the state of nature (which does not change over time). Nature is in one of two possible states, G (good) and B (bad). The agent gets signals, which are correlated with the true state, until the game ends, which happens at each step with probability g > 0. At this point, the agent must make a decision. For each N, Wilson characterizes an N-state optimal finite automaton for making a decision in this setting, under the assumption that g is small (so that the agent gets information for many rounds). (See Section 3 for further details.) She then uses this characterization to argue that an optimal N-state automaton also exhibits behavior such as belief polarization (again, see Section 3). Thus, some observed human behavior can be explained by viewing people as resource-bounded, but rationally making the best use of their resources (in this case, the limited number of states). Wilson's model assumes that nature is static. But, in many important problems, ranging from investing in the stock market to deciding which route to take when driving to work, the world is dynamic. Moreover, people do not make decisions just once, but must make them often. For example, when investing in stock markets, people get signals about the market, and need to decide after each signal whether to invest more money, take out money that they have already invested, or to stick with their current position. In a recent work  (Halpern, Pass, and Seeman 2012) , we consider the problem of constructing an optimal N-state automaton for this setting. We construct a family of quite simple automata, indexed by N, the number of states, and a few other parameters. We show that as N grows large, this family approaches optimal behavior. More precisely, for all e, there is a sufficiently large N and member of this family with N states whose expected payoff is within e of optimal, provided the probability of a state transition is sufficiently small. More importantly, the members of this family reproduce observed human behavior in a series of tasks conducted by  Erev, Ert and Roth (2010)  (see Section 3 for further discussion). Again, these results emphasize the fact that some observed human behavior can be explained by viewing people as rational, resource-bounded agents.
 Charging for the complexity of computation In this section, we review and discuss the approach that is used by  Halpern and Pass (2010b)  to take cost of computation into account in decision theory. Much of the material below is taken from  (Halpern and Pass 2010b) , which the reader is encouraged to consult for further details and intuition. The framework that we use is essentially a single-agent version of what  Halpern and Pass (2010a, c)  called Bayesian machine games. In a standard Bayesian game, each player has a type in some set T, and makes a single move. Player i's type can be viewed as describing i's initial information; some facts that i knows about the world. We assume that an agent's move consists of choosing a Turing machine. As we said in the introduction, associated with each Turing machine and type is its complexity. Given as input a type, the Turing machine outputs an action. The utility of a player depends on the type profile (i.e., the types of all the players), the action profile, and the complexity profile (i.e., each player's complexity).
 Example 2.1 Suppose that an agent is given an input n, and is asked whether it is prime. The agent gets a payoff of $1,000 if he gives the correct answer, and loses $1,000 if he gives the wrong answer. However, he also has the option of playing safe, and saying "pass," in which case he gets a payoff of $1. Clearly, many agents would say "pass" on all but simple inputs, where the answer is obvious, although what counts as a "simple" input may depend on the agent. 1 The agent's type can then be taken to be the input. The set T of possible types could be, for example, all integers, or all integers that can be written in binary using at most 40 digits (i.e., a number that is less than 2 40 ). The agent can choose among a set of TMs, all of which output either "prime," "not prime," or "pass" (which can be encoded as 0, 1, and 2). One natural choice for the complexity of a pair (M, n) consisting of TM M and an input n is the running time of M on input n. As the agent's utility takes the complexity into account, this would justify the agent using a "quick-and-dirty" algorithm, which is typically right to compute whether the input is prime; if the algorithm does not return an answer in a relatively small amount of time, the agent can just "pass." The agent might prefer using such a TM rather than one that gives the correct answer, but takes a long time doing so. We can capture this tradeoff in the agent's utility function. ▪ There are some issues that we need to deal with to finish modeling Example 2.1 in our framework. Note that if the agent chooses the TM after being given the number n, then one of two TMs is clearly optimal: if n is in fact prime, then the TM which just says "prime" and halts; if n is not prime, then the TM which says "not prime" and halts is clearly optimal. The problem, of course, is that the agent does not know n. One approach to dealing with this problem is to assume that the agent chooses the TM before knowing n; this was the approach implicitly taken by  Halpern and Pass (2010c) . But there is a second problem: we have implicitly assumed that the agent knows what the complexity of each pair (M, n) is; otherwise, the agent could not maximize expected utility. Of course, in practice, an agent will not know how long it will take a Turing machine to compute an answer on a given input. We deal with both of these problems by adding one more parameter to the model: a state of nature. In Example 2.1, we allow the TM's running time and whether n is prime to depend on the state; this means that the "goodness" (i.e., accuracy) of the TM can depend on the state. We model the agent's beliefs about the running time and accuracy of the TM using a probability distribution on these states.  2  We capture these ideas formally by taking a computational decision problem with types to be a tuple D ¼ ðS; T; A; Pr; M; C; O; uÞ. We explain each of these components in turn. The first four components are fairly standard. S is a state space, T is a set of types, A is a set of actions, and Pr is a probability distribution on S9T (there may be correlation between states and types). In a standard decision-theoretic setting, there is a probability on states (not on states and types), and the utility function associates a utility with each (state, action) pair (intuitively, u(s, a) is the utility of performing action a in state s). Here things are more complicated because we want the utility to also depend on the TM chosen. This is where the remaining components of the tuple come in. The fifth component of the tuple, C, is the complexity function. Formally, C : M Â S Â T ! N, so CðM; s; tÞ is the complexity of running TM M on input t in state s. The complexity can be, for example, the running time of M on input t, the space used by M on input t, the number of states in M (this is the measure of complexity used by  Rubinstein (1986) ), and so on. The sixth component of the tuple, O, is the output function. It captures the agent's uncertainty about the TM's output. Formally, O : M Â S Â T ! N; OðM; s; tÞ is the output of M on input t in state s. Finally, an agent's utility depends on the state s, his type t, and the action OðM; s; tÞ, as is standard, and the complexity. As we describe the complexity by a natural number, we take the utility function u maps S Â T Â A Â N to R (the reals). Thus, the expected utility of choosing a TM M in the decision problem D is P s2S;t2T Prðs; tÞuðs; t; OðM; s; tÞ; CðM; s; tÞÞ. Note that now the utility function gets the complexity of M as an argument. The next example should clarify the role of all these components. Example 2.1 (cont'd): We now have the machinery to formalize Example 2.1. We take T, the type space, to consist of all natural numbers < 2 40 ; the agent must determine whether the type is prime. The agent can choose either 0 (the number is not prime), 1 (the number is prime), or 2 (pass); Thus, A = {0, 1, 2}. Let M be some set of TMs that can be used to test for primality. As suggested above, the state space S is used to capture the agent's uncertainty about the output of a TM M and the complexity of M. Thus, for example, if the agent believes that the TM will output pass with probability 2/3, then the set of states such that OðM; s; tÞ ¼ 2 has probability 2/3. We take CðM; s; tÞ to be 0 if M computes the answer within 2 20 steps on input t, and 10 otherwise. (Think of 2 20 steps as representing a hard deadline.) If, for example, the agent does not know the running time of a TM M, but ascribes probability 2/3 to M finishing in less than 2 20 steps on input t, then the set of states s such that Cðs; t; MÞ ¼ 0 has probability 2/3. We assume that there is a function Prime that captures the agent's uncertainty regarding primality; Prime(s,t) = 1 if t is prime in state s, and 0 otherwise. 3 Thus, if the agent believes that TM M gives the right answer with probability 2/3, then the set of states s where Primeðs; tÞ ¼ OðM; s; tÞ has probability 2/3. Finally, let u(s, t, a, c) = 10 À c if a is either 0 or 1, and this is the correct answer in state s (i.e., OðM; s; tÞ ¼ Primeðs; tÞ) and u(s, t, 2, c) = 1 À c. Thus, if the agent is sure that M always gives the correct output, then u(s, t, a, c) = 10 À c for all states s and a 2 {0, 1}. ▪
 Example 2.2 Biases in information processing Psychologists have observed many systematic biases in the way that individuals update their beliefs as new information is received (see  Rabin (1998)  for a survey). In particular, a first-impressions bias has been observed: individuals put too much weight on initial signals and less weight on later signals. As they become more convinced that their beliefs are correct, many individuals even seem to simply ignore all information once they reach a confidence threshold. Several papers in behavioral economics have focused on identifying and modeling some of these biases (see, e.g.,  Rabin, 1998 and references therein; Mullainathan, 2002; Rabin & Schrag, 1999) . In particular,  Mullainathan (2002)  makes a potential connection between memory and biased information processing, using a model that makes several explicit (psychology-based) assumptions on the memory process (e.g., that the agent's ability to recall a past event depends on how often he has recalled the event in the past). More recently,  Wilson (2002)  demonstrated a similar connection when modeling agents as finite automata, but her analysis is complex (and holds only in the limit). As we now show, the first-impression-matters bias can be easily explained if we assume that there is a small cost for "absorbing" new information. Consider the following simple game, which is very similar to the one studied by  Mullainathan (2002)  and  Wilson (2002) . The state of nature is a bit b that is 1 with probability 1/2. For simplicity, we assume that the agent has no uncertainty about the "goodness" or output of a TM; the only uncertainty involves whether b is 0 or 1). An agent receives as his type a sequence s 1 ,s 2 ,. . .,s n of independent samples, where s i = b with probability q > 1/2. The samples corresponds to signals the agents receive about b. An agent is supposed to output a guess b 0 for the bit b. If the guess is correct, he receives 1 À mc as utility, and Àmc otherwise, where m is the number of bits of the type he read, and c is the cost of reading a single bit (c should be thought of the cost of absorbing/interpreting information). It seems reasonable to assume that c > 0; signals usually require some effort to decode (such as reading a newspaper article, or attentively watching a movie). If c > 0, it easily follows by the Chernoff bound (see  Alon & Spencer, 2004 ) that after reading a certain (fixed) number of signals s 1 ,. . .,s i , the agents will have a sufficiently good estimate of b that the marginal cost of reading one extra signal s i+1 is higher than the expected gain of finding out the value of s i+1 . That is, after processing a certain number of signals, agents will eventually disregard all future signals and base their output guess only on the initial sequence. We omit the straightforward details. Essentially the same approach allows us to capture belief polarization. Suppose for simplicity that two agents start out with slightly different beliefs regarding the value of some random variable X (think of X as representing something like "O.J. Simpson is guilty"), and get the same sequence s 1 ,s 2 ,. . .,s n of evidence regarding the value of X. (Thus, now the type consists of the initial belief, which can, for example, be modeled as a probability or a sequence of evidence received earlier, and the new sequence of evidence.) Both agents update their beliefs by conditioning. As before, there is a cost of processing a piece of evidence, so once an agent gets sufficient evidence for either X = 0 or X = 1, he will stop processing any further evidence. If the initial evidence supports X = 0, but the later evidence supports X = 1 even more strongly, the agent that was initially inclined toward X = 0 may raise his beliefs to be above threshold, and thus stop processing, believing that X = 0, while the agent initially inclined toward X = 1 will continue processing and eventually believe that X = 1. ▪ As shown by  Halpern and Pass (2010b) , we can also use this approach to explain the status quo bias (people are much more likely to stick with what they already have;  Samuelson & Zeckhauser, 1998) .
 Value of computational information and value of conversation Having a computational model of decision making also allows us to reconsider a standard notion from decision theory, value of information, and extend it in a natural way so as to take computation into account. Value of information is meant to be a measure of how much an agent should be willing to pay to receive new information. The idea is that, before receiving the information, the agent has a probability on a set of relevant events and chooses the action that maximizes his expected utility, given that probability. If he receives new information, he can update his probabilities (by conditioning on the information) and again choose the action that maximizes his expected utility. The difference between the expected utility before and after receiving the information is the value of the information. In many cases, an agent seems to be receiving valuable information that is not about what seem to be the relevant events. This means that we cannot do a value of information calculation, at least not in the obvious way. For example, suppose that the agent is interested in learning a secret, which we assume for simplicity is a number between 1 and 1,000. A priori, suppose that the agent takes each number to be equally likely, and so has probability 0.001. Learning the secret has utility, say, $1,000,000; not learning it has utility 0. The number is locked in a safe, whose combination is a 40-digit binary number. Intuitively, learning the first 20 digits of the safe's combination gives the agent some valuable information. But this is not captured when we do a standard value-of-information calculation; learning this information has no impact at all on the agent's beliefs regarding the secret. Although this example is clearly contrived, there are many far more realistic situations where people are clearly willing to pay for information to improve computation. For example, companies pay to learn about a manufacturing process that will speed up production; people buy books on speedreading; and faster algorithms for search are clearly considered valuable. Once we bring computation into decision making, the standard definition of value of information can be applied to show that there is indeed a value to learning the first 20 digits of the combination, and to buying a more powerful computer; expected utility can increase (see  Halpern & Pass, 2010a for details) . But now we can define a new notion: value of conversation. The value of information considers the impact of learning the value of a random variable; by taking computation into account, we can extend this to consider the impact of learning a better algorithm. We can further extend to consider the impact of having a conversation. The point of a conversation is that it allows the agent to ask questions based on history. For example, if the agent is trying to guess a number chosen uniformly at random between 1 and 100, and receives utility of 100 if he guesses it right, having a conversation with a helpful TM that will correctly answer seven yes/no questions is quite valuable: as is well known, with seven questions, the agent can completely determine the number using binary search. The computational framework allows us to make this intuition precise. Again, we encourage the reader to consult (Halpern and Pass 2010a) for further details.
 Modeling people as rational finite automata We now consider the second approach discussed in the introduction, that of modeling the fact that an agent can do only bounded computation. Perhaps the first to do this in the context of decision theory was  Wilson (2002) , whose work we briefly mentioned earlier. Recall that Wilson considers a decision problem where an agent needs to make a single decision. Nature is in one of two possible states, G (good) and B (bad), which does not change over time; the agent's payoff depends on the action she chooses and the state of nature. The agent gets one of k signals, which are correlated with nature's state; signal i has probability p G i of appearing when the state of nature is G, and probability p B i of appearing when the state is B. We assume that the agent gets exactly one signal at each time step, so that P k i¼1 p G i ¼ P k i¼1 p B i ¼ 1 . This is a quite standard situation. For example, an agent may be on a jury, trying to decide guilt or innocence, or a scientist trying to determine the truth of a theory. Clearly, the agent should try to learn nature's state, so as to make an optimal decision. With no computational bounds, an agent should just keep track of all the evidence it has seen (i.e., the number of signals of each type), in order to make the best possible decision. However, a finite automaton cannot do this. Wilson characterizes the optimal Nstate automaton for making a decision in this setting, under the assumption that g (the probability that the agent has to make the decision in any given round) is small. Specifically, she shows that an optimal N-state automaton ignores all but two signals (the "best" signal for each of nature's states). The automaton's states can be laid out "linearly," as states 0, . . ., N À 1, and the automaton moves left (with some probability) only if it gets a strong signal for state G (and it is not in state 0), and moves right (with some probability) only if it gets a strong signal for state B (and is not in state N À 1). Roughly speaking, the lower the current state of the automaton, the more likely from the automaton's viewpoint that nature's state is G. The probability of moving left of right, conditional on receiving the appropriate signal, may vary from state to state. In particular, if the automaton is in state 0, the probability that it moves right is very low. Intuitively, in state 0, the automaton is "convinced" that nature's state is G, and it is very reluctant to give up on that belief. Similarly, if the automaton is in state N À 1, the probability that it will move left is very low. Wilson argues that these results can be used to explain observed biases in information processing, such as belief polarization. For suppose that the true state of nature is B. Consider two 5-state automata, A 1 and A 2 . Suppose that automaton A 1 starts in state 1, while automaton A 2 starts in state 2. (We can think of the starting state as reflecting some initial bias, for example.) They both receive the same information. Initially, the information is biased toward G, so both automata move left; A 1 moves to state 0, and A 2 moves to state 1. Now the evidence starts to shift toward the true state of the world, B. But since it is harder to "escape" from state 0, A 1 stays in state 0, while A 2 moves to state 4. Thus, when the automata are called upon to decide, A 1 makes the decision appropriate for B, while A 2 makes the decision appropriate for G. A similar argument shows how this approach can be used to explain the first-impression bias. The key point is that the order in which evidence is received can make a big difference to an optimal finite automaton, although it should make no difference to an unbounded agent. Wilson's model assumes that the state of nature never changes. In recent work  (Halpern et al. 2012) , we consider what happens if we allow nature's state to change. We con-sider a model that is intended to capture the most significant features of such a dynamic situation. As in Wilson's model, we allow nature to be in one of a number of different states, and assume that the agent gets signals correlated with nature's state. But now we allow nature's state to change, although we assume that the probability of a change is low. (Without this assumption, the signals are not of great interest.) For definiteness, assume that nature is in one of two states, which we again denote G and B. Let p be the probability of transitioning from B to G or from G to B in any given round. Thus, we assume for simplicity that these probabilities are history-independent, and the same for each of the two transitions. (Allowing different probabilities in each direction does not impact the results.) The agent has two possible actions S (safe) and R (risky). If he plays S, he gets a payoff of 0; if he plays R he gets a payoff x G > 0 when nature's state is G, and a payoff x B < 0 when nature's state is B. The agent does not learn his payoff, but, as in Wilson's model, gets one of k signals, whose probability is correlated with nature's state. However, unlike Wilson's model, the agent gets a signal only if he plays the risky action R; he does not get a signal if he plays the safe action S. We denote this setting S½p G 1 ; p B 1 ; . . .; p G k ; p B k ; x G ; x B . A setting is nontrivial if there exists some signal i such that p B i 6 ¼ p G i . If a setting is trivial, then no signal enables the agent to distinguish whether nature is in state G or B; the agent does not learn anything from the signals. For a given setting S½p G 1 ; p B 1 ; . . .; p G k ; p B k ; x G ; x B , we are interested in finding an automaton A that has high average utility when we let the number of rounds go to infinity. Unlike Wilson, we were not able to find a characterization of the optimal N-state automaton. However, we were able to find a family of quite simple automata that do very well in practice and, in the limit, approach the optimal payoff. We denote a typical member of this family A[N, p exp , Pos, Neg, r u , r d ]. The automaton A[N, p exp , Pos, Neg, r u , r d ] has N + 1 states, again denoted 0,. . .,N. State 0 is dedicated to playing S. In all other states R is played. As in Wilson's optimal automaton, only "strong" signals are considered; the rest are ignored. More precisely, the k signals are partitioned into three sets, Pos (for "positive"), Neg (for "negative"), and I (for "ignore" or "indifferent"), with Pos and Neg nonempty. The signals in Pos make it likely that nature's state is G, and the signals in Neg make it likely that the state of nature is B. The agent chooses to ignore the signals in I; they are viewed as not being sufficiently informative as to the true state of nature. (Note that I is determined by Pos and Neg.) In each round while in state 0, the agent moves to state 1 with probability p exp . In a state i > 0, if the agent receives a signal in Pos, the agent moves to i+1 with probability r u (unless he is already in state N, in which case he stays in state N if he receives a signal in Pos); thus, we can think of r u as the probability that the agent moves up if he gets a positive signal. If the agent receives a signal in Neg, the agent moves to state i À 1 with probability r d (so r d is the probability of moving down if he gets a signal in Neg); if he receives a signal in I, the agent does not change states. Clearly, this automaton is easy for a human to implement (at least, if it does not have too many states). Because the state of nature can change, it is clearly not optimal to make the states 0 and N "sticky." In particular, an optimal agent has to be able to react reasonably quickly to a change from G to B, so as to recognize that he should play S.  Erev et al. (2010)  describe contests that attempt to test various models of human decision making under uncertain conditions. In their scenarios, people were given a choice between making a safe move (that had a guaranteed constant payoff) and a "risky" move (which had a payoff that changed according to an unobserved action of other players), in the spirit of our S and R moves. They challenged researchers to present models that would predict behavior in these settings. The model that did best was called BI-Saw (bounded memory, inertia, sampling and weighting) model, suggested by  Chen, Liu, Chen, and Lee (2011) , itself a refinement of a model called I-Saw suggested by  Erev et al. (2010) . This model has three types of response mode: exploration, exploitation, and inertia. An I-Saw agent proceeds as follows: The agent tosses a coin. If it lands heads, the agent plays the action other than the one he played in the previous step (exploration); if it lands tails, he continues to do what he did in the previous step (inertia), unless the signal received in the previous round crosses a probabilistic "surprise" trigger (the lower the probability of the signal to be observed in the current state, the more likely the trigger is to be crossed); if the surprise trigger is crossed, then the agent plays the action with the best estimated subjective value, based on some sampling of the observations seen so far (exploitation). The major refinement suggested by BI-Saw involves adding a bounded memory assumption, whose main effect is a greater reliance on a small sample of past observations. The suggested family of automata incorporates all three behavior modes described by the I-Saw model. When the automaton is in state 0, the agent explores with constant probability by moving to state 1. In state i > 0, the agent continues to do what he did before (in particular, he stays in state i) unless he gets a "meaningful" signal (one in Neg or Pos), and even then he reacts only with some probability, so we have inertia-like behavior. If he does react, then he exploits the information that he has, which is carried by his current state; that is, he performs the action most appropriate according to his state. The state can be viewed as representing a sample of the last few signals (each state represents remembering seeing one more "good" signal), as in the BI-Saw model. Thus, our family of automata can be viewed as an implementation of the BI-Saw model using small, simple finite automata. These automata do quite well, both theoretically and in practice. Note that even if the agent had an oracle that told him exactly what nature's state would be at every round, if he performs optimally, he can get only x G in the rounds when nature is in state G, and 0 when it is in state B. In expectation, nature is in state G only half the time, so the optimal expected payoff is x G /2. The following result shows that if p goes to 0 sufficiently quickly, then the agent can achieve arbitrarily close to the theoretical optimum using an automaton of the form A[N, p exp , Pos, Neg, r u , r d ], even without the benefit of an oracle, by choosing the parameters appropriately. Let E p [A] denote the expected average utility of using the automaton A if the state of natures changes with probability p. Theorem 3.1  (Halpern et al. 2012)  Let Π and P exp be functions from N to (0,1] such that lim n?∞ nΠ(n) = lim n?∞ Π(n)/ P exp (n) = 0. Then for all settings S½p G 1 ; p While Theorem 3.1 gives theoretical support to the claim that automata of the form A[N, p exp , Pos, Neg, r u , r d ] are reasonable choices for a resource-bounded agent, it is also interesting to see how they do in practice, for relatively small values of N. The experimental evidence  (Halpern et al. 2012)  suggests that they will do well. For example, suppose for definiteness that p = 0.001, x G = 1, x B = À1 and there are four signals, 1,. . ., 4, which have probabilities 0.4, 0.3, 0.2, and 0.1, respectively, when the state of nature is G, and probabilities 0.1, 0.2, 0.3, and 0.4, respectively, when the state of nature is bad. Further suppose that we take signal 1 to be the "good" signal (i.e., we take Pos = {1}), take signal 4 to be the "bad" signal (i.e., we take Neg = {4}), and take r u = r d = 1. Experiments showed that using the optimal value of p exp (which is dependent on the number of states), an automaton with 5 states already gets an expected payoff of more than 0.4; even with 2 states, it gets an expected payoff of more than 0.15. Recall that even with access to an oracle that reveals when nature changes state, the best the agent can hope to get is 0.5. On the other hand, an agent that just plays randomly, or always plays G or always plays B, will get 0. These results are quite robust. For example, for N ≥ 5 the payoff does not vary much if we just use the optimum p exp for N = 5, instead of choosing the optimum value of p exp for each value of N. The bottom line here is that, by thinking in terms of the algorithms used by bounded agents, actions that may seem irrational can be viewed as quite rational responses, given resource limitations.
 Discussion and conclusion We have discussed two approaches for capturing resource-bounded agents. The first allows them to choose a TM to play for them, but charges them for the "complexity" of the choice; the second models agents as finite automata, and captures resource-boundedness by restricting the number of states of the automaton. In both cases, agents are assumed to maximize utility. Both approaches can be used to show that some systematic deviations from rationality (e.g., belief polarization) can be viewed as the result of resource-bounded agents making quite rational choices. We believe that the general idea of not viewing behavior as "irrational," but rather the outcome of resource-bounded agents making rational choices, will turn out to be useful for explaining other systematic biases in decision making and, more generally, behavior in decision problems. We would encourage work to find appropriate cost models, and simple, easy-to-implement strategies with low computational cost that perform well in real scenarios. Note that the two approaches are closely related. For example, we can easily come up with a cost model and class of TMs that result in agents choosing a particular automaton with N states to play for them, simply by charging appropriately for the number of states. Which approach is used in a particular analysis depends on whether the interesting feature is the choice of the complexity function (which can presumably be tested experimentally) or specific details of the algorithm. We are currently exploring both approaches in the context of the behavior of agents in financial markets. In particular, we are looking for simple, easy-toimplement strategies that will explain human behavior in this setting. Pos, Neg, I of the signals, and constants r d and r u such that lim N!1 E pðNÞ ½A½N; P exp ðNÞ; Pos; Neg; r u ; r d ¼ x G 2 : B 1 ; . . .; p G k ; p B k ; x G ; x B , there exists a partition
			 J. Y. Halpern, R. Pass, L. Seeman / Topics in Cognitive Science 6 (2014)

/Users/janhendrikkirchner/code/2022/10/alignment-research-dataset/align_data/common/../../data/raw/nonarxiv_teis/Topics in Cognitive Science - 2014 - Halpern - Decision Theory with Resource‐Bounded Agents.tei.xml


Abstract: The rhetoric of the race for strategic advantage is increasingly being used with regard to the development of artificial intelligence (AI), sometimes in a military context, but also more broadly. This rhetoric also reflects real shifts in strategy, as industry research groups compete for a limited pool of talented researchers, and nation states such as China announce ambitious goals for global leadership in AI. This paper assesses the potential risks of the AI race narrative and of an actual competitive race to develop AI, such as incentivising corner-cutting on safety and governance, or increasing the risk of conflict. It explores the role of the research community in responding to these risks. And it briefly explores alternative ways in which the rush to develop powerful AI could be framed so as instead to foster collaboration and responsible progress.

An AI Race for Strategic Advantage: Rhetoric and Risks

Introduction Global leadership in different areas of fundamental and applied research in artificial intelligence has emerged as a strategic priority, both for major companies and nation states. The last two years have seen heightened competition between industry research groups for talented researchers and startups  (Metz 2017 , CB Insights 2017), and the release of strategic plans aimed at establishing research leadership in AI by the United States (NSTC, 2016), China  (Fa, 2017) , and other research-leading nations  (Hall and Prescenti 2017 , NEDO 2017 , CIFAR 2017 . As the applications of AI research become more lucrative, and the impacts of AI on economic, scientific and military applications more transformative, this competition can be expected to intensify. A narrative has begun to emerge, in both policy and public contexts (Simonite 2017, Allen and Husain 2017, Stewart 2017, Allen and Kania 2017, Allen and Chan 2017), framing future trajectories relating to the development of AI in terms typically associated with a race for technological superiority. The China State Council's 2017 "A Next Generation Artificial Intelligence Development Plan" includes in its aims to seize the major strategic opportunity for the development of AI, to build China's first-mover advantage in the development of AI. The United States National Science and Technology Council's Research and Development Strategic Plan highlights that the US no longer leads the world in numbers of deep learning-related publications. Strong public statements have been made by political and technology leaders: notable examples include Russia's President Vladimir Putin stating that "whoever becomes the leader in this sphere will become the ruler of the world" (RT 2017) and Open-AI cofounder Elon Musk tweeting that "competition for AI superiority at national level [is the] most likely cause of WW3"  (Musk 2017 ). These developments reflect a perception that global leadership in AI could confer scientific, infrastructural and economic advantages to frontrunners. Such comparative advantage could also be self-perpetuating, if it helps to recruit the best research talent, or leads to greater wealth and spending power, or to the accumulation of better computing and data resources, or if the application of AI to the process of scientific research itself leads to further breakthroughs. Similarly, although many breakthroughs are likely to be local in application, the perception that some fundamental advances can be applied to a broad range of research challenges further fuels the idea that a winner could take it all. Such notions of 'winner takes all' or the importance of technological leadership, foster the framing of AI development as a race. We will refer to this framing as that of the 'race for technological superiority' in AI. This kind of race is the primary concern of this paper. But at the same time, it is important to note that there are also concerns about a specifically military AI arms race. Although this is not the primary concern of this paper, there are some noteworthy interrelations with the broader race for technological superiority: most significantly, if AI fulfils its promise as a widely-applicable and transformational technology, then general superiority in AI will to a significant degree imply also military superiority. Many aspects of AI progress are likely to have dual-use relevance to both civilian and military applications (for example, advances enabling greater autonomy and a wider range of capabilities in autonomous vehicles) not least given the deep connection between AI research and the defence community  (Geist 2016) . And if systems approach general intel-ligence or superintelligence, they are likely to confer a very significant strategic advantage that could encompass, but go well beyond, conventional weaponry, with radical implications for security and geopolitics (Allen and Chan 2017). The central concern of this paper is what might happen if the rhetoric noted above around first mover advantage or the pursuit of AI superiority becomes a widespread or even dominant framing of AI development. Our concerns arise both from use of the language of a race for technological superiority (independently of whether such a race is happening) and from such a race becoming a reality.
 The Dangers of an AI Race for Technological Advantage: In Rhetoric and in Reality What is so bad about framing the development of AI in terms of a race for technological advantage? After all, it is widely agreed that AI brings enormous potential benefits across many sectors. One recent report estimated that it could add £232 billion by 2030 to the UK economy alone, with healthcare one of the sectors most enhanced, potentially bringing faster, better service to consumers (PwC 2017). There is a widespread belief that competition and other market forces are central to such innovation. So in as much as a race to develop AI technology means these kinds of benefit come sooner, we have reason to view it positively. But at the same time, the development of such a potentially powerful new technology will need to be steered if it is to be as beneficial as possible while minimising the risks it might pose (Crawford and Calo 2016). In the words of the Future of Life Institute's open letter on 'Research Priorities for Robust and Beneficial Artificial Intelligence', signed by over 8,000 people including many leading figures in AI, work should focus "not only on making AI more capable, but also on maximizing the societal benefit of AI." (Future of Life Institute, 2017a). The danger of an AI race is that it makes exactly this thoughtful steering towards broadly beneficial outcomes more difficult.
 Three Sets of Risks We want to distinguish between three sets of risks: i) The dangers of an AI 'race for technological advantage' framing, regardless of whether the race is seriously pursued; ii) The dangers of an AI 'race for technological advantage' framing and an actual AI race for technological advantage, regardless of whether the race is won; iii) The dangers of an AI race for technological advantage being won.
 (i) Risks Posed by a Race Rhetoric Alone It is possible that the trend towards 'race for technological advantage' terminology in AI, including suggestions such as Vladimir Putin's that the winner of such a race "will become the ruler of the world," could pose risks even if the race is not pursued in earnest, let alone won. We perceive two main risks here: (i.a) The kind of thoughtful consideration of how to achieve broadly beneficial AI, as mentioned above, will require subtle, inclusive, multi-stakeholder deliberation over a prolonged peri-od. The rhetoric of the race for technological advantage, with its implicit or explicit threat that dire consequences will follow if some other group wins that race, is not likely to be conducive to such deliberation. Indeed, rhetoric around technological superiority (such as the 'arms race' rhetoric used in the Cold War in the US), played into what has been called a politics of fear, or a politics of insecurity --that is, a political climate that discourages debate in favour of unquestioning support for a prescribed agenda  (Griffith 1987 ). In The Politics of Insecurity, Jef Huysmans argues that use of the language of security (by which he means militarised language, which would include 'arms race' and related rhetoric) "is a particular technique of framing policy questions in logics of survival with a capacity to mobilize politics of fear in which social relations are structured on the basis of distrust"  (Huysmans 2006 ). (i.b) Second, if the rhetoric of a competitive, 'winner takes all' AI race is used in the absence of an actual race, it could contribute to sparking such a race. (ii) Risks Posed by a Race Emerging If the rhetoric of a race for technological advantage became an actual race to develop sophisticated AI, the risks increase further: (ii.a) First, there is the risk that racing to achieve powerful AI would not be conducive to taking the proper safety precautions that such technology will require (Armstrong, Bostrom, and Shulman 2016). We mentioned above the need for broad consultation about the role AI should play in the life of a community. This might help address important considerations such as avoiding biased systems, or maximising fairness. But in addition to these goals, serious attention must also be given to ensuring humans do not lose control of systems. Such considerations become particularly important if AI approaches general intelligence or superintelligence  (Bostrom 2014 ), but also long before, particularly when AI systems are performing critical functions. The risk is that as the perceived benefit to winning the race increases, so correspondingly does the incentive to cut corners on these safety considerations. (ii.b) Second, a 'race for technological advantage' could increase the risk of competition in AI causing real conflict (overt or covert). Huysmans argues that militarised language such as this has "a specific capacity for fabricating and sustaining antagonistic relations between groups. In the case of the race for technological advantage, it encourages us to see competitors as threats or even enemies. The belief that a country intends in earnest to win an AI race, and that this would result in technological dominance, could, for example, prompt other countries to use aggression to prevent this (akin to the cyberattacks made against Iranian nuclear facilities attributed to the US and Israel) (Nakashima 2012), or motivate the targeting of key personnel (precedents --though specific to their historical context --might include Operation Paperclip, during which over 1,600 German scientists and engineers who had worked on military technology were taken to the US (Jacobsen 2014), or the apparently ongoing operations to encourage the defection of nuclear scientists between nations)  (Golden 2017) . Such scenarios would also in-crease the risk that a general race for technological superiority became increasingly a military AI arms race.
 (iii) Risks Posed by Race Victory The third category of risks of an AI race for technological superiority are those that would arise if a race were won. We will not explore these in detail here --and the forms they take will anyway depend on the precise nature of the technology in question. But as an example, these risks include the concentration of power in the hands of whatever group possesses this transformative technology. If we survey the current international landscape, and consider the number of countries demonstrably willing to use force against others, as well as the speed with which political direction within a country can change, and the persistence of non-state actors such as terrorist groups, we might conclude that the number of groups we would not trust to responsibly manage an overwhelming technological advantage exceeds the number we would.
 Choices for the Research Community Given the dangers noted above of talking about AI development as a competitive race, one could argue that it would be better for the community of researchers considering AI and its impacts to avoid this terminology altogether. In this way, the language of the race for dominance would be seen as (in Nick Bostrom's terms) an information hazard --perhaps either an idea hazard (a general idea whose dissemination could increase risk) or an attention hazard (if we consider that the dangerous idea already exists, but is as yet largely unnoticed)  (Bostrom 2011 ). But if we believe that the idea is already being disseminated by influential actors, such as states, including major powers, then the information hazard argument is weakened. It might still apply to particularly influential researchers --those whose thoughts on the future of AI can become headline news around the world. But even in their case, and particularly in the case of lesser-known figures, there is a strong countervailing argument: that if the potentially dangerous idea of an AI race is already gaining currency, then researchers could make a positive impact by publicly drawing attention to these dangers, as well as by pursuing research dedicated to mitigating the risks of such a framing. Of course, many leading researchers are already speaking out against an AI arms race in the sense of a race to develop autonomous weapons --see for example the Future of Life Institute's open letter on this, signed by over three thousand AI researchers and over fifteen thousand others (Future of Life Institute 2015b). We believe this community could also usefully direct its attention to speaking out against an AI race in this other sense of a competitive rush to develop powerful general-purpose AI as fast as possible. Both media and governments are currently giving considerable attention to AI, yet are still exploring ways of framing it and its impacts. We believe that there is therefore an opportunity now for researchers to influence this framing 1 . One of the principles on AI agreed at the 2017 Asilomar conference offers a precedent on which to build --it states: Race Avoidance: Teams developing AI systems should actively cooperate to avoid corner-cutting on safety standards (Future of Life Institute 2017).
 Alternatives to a Race Approach If we are not to pursue (and talk about pursuing) AI development as a race, then how should we pursue (and talk about pursuing) such development? This is too big a topic to consider thoroughly here. But we note these promising directions for alternative narratives around progress in, and benefits of, AI:
 AI Development as a Shared Priority for Global Good As advances in AI find application to an ever-wider range of scientific and societal challenges, there is a burgeoning discussion around harnessing the benefits of AI for global benefit. This reflects a widespread view among AI scientists and a growing number of policymakers that AI presents tremendous opportunities for making progress on global societal ills, and aiding in tackling some of the biggest challenges we face in the coming century --among them climate change and clean energy production, biodiversity loss, healthcare, global poverty and education. Emphasising these benefits could counteract a race approach in a number of ways: First, if there is global scientific consensus that some of the key aims of AI should be to benefit humanity in these ways, then it becomes less important in which companies or countries key breakthroughs occur. Second, it makes clear that cooperation on the development of AI stands to result in faster progress on these pressing challenges. Lastly, if the aims of the field are to benefit humanity worldwide, then the global community represent stakeholders in the process of AI development; this narrative therefore promotes inclusive and collaborative development and deployment of AI.
 Cooperation on AI as it is Applied to Increasingly Safety-Critical Settings Globally The next decade will see AI applied in an increasingly integral way to safety-critical systems; healthcare, transport, infrastructure to name a few. In order to realise these benefits as quickly and safely as possible, sharing of research, datasets, and best practices will be critical. For example, to ensure the safety of autonomous cars, pooling expertise and datasets on vehicle performances across as wide as possible a range of environments and conditions (including accidents and near-accidents) would provide substantial benefits for all involved. This is particularly so given that the research, data, and testing needed to refine and ensure the safety of such systems before deployment may be considerably more costly and time-consuming than the research needed to develop the initial technological capability. Promoting recognition that deep cooperation of this nature is needed to deliver the benefits of AI robustly may be a powerful tool in dispelling a 'technological race' narrative; and a 'cooperation for safe AI' framing is likely to become increasingly important as more powerful and broadly capable AI systems are developed and deployed.
 Responsible Development of AI and Public Perception AI is the focus for a growing range of public concerns as well as optimism (Ipsos MORI 2017, Fast and Horvitz 2017). Many stakeholders, including in industry, recognise the importance of public trust in the safety and benefits offered by AI if it is to be deployed successfully  (Banavar 2017) . It is the view of the authors that a narrative focused on global cooperation and safe, responsible development of AI is likely to inspire greater public confidence than a narrative focused more on technological dominance or leadership. Other powerful new technologies, such as genetically modified organisms and nuclear power, have in the past proven controversial, with significant communities arguing for a cautious, safety-first approach, to which the rhetoric of the race is antithetical.
 Recent Narratives There have been encouraging developments promoting the above narratives in recent years. 'AI for global benefit' is perhaps best exemplified by the 2017's ITU summit on AI for Global Good (Butler 2017), although it also features prominently in narratives being put forward by the IEEE's Ethically Aligned Design process (IEEE 2016), the Partnership on AI, and programmes and materials put forward by Microsoft, DeepMind and other leading companies. Collaboration on AI in safetycritical settings is also a thematic pillar for the Partnership on AI 2 . Even more ambitious cooperative projects have been proposed by others, for example the call for a 'CERN for AI' from Professor Gary Marcus, through which participants "share their results with the world, rather than restricting them to a single country or corporation" (Marcus 2017). Finally, the overall narrative of cooperation was clearly expressed in a statement issued by a Beneficial AI conference in Tokyo 3 : The challenges of ensuring that [AI] is beneficial are challenges for us all. We urge that these challenges be addressed in a spirit of cooperation, not competition.  (Beneficial AI Tokyo, 2017) .
 Conclusion Although artificial intelligence has been discussed for decades, only recently has it received serious and sustained attention from governments, industry and the media. Among the various ways in which this technology is framed, we have highlighted one that we consider to be potentially dangerous: that of the race for technological advantage. Partly, we believe that a general race to develop AI would be dangerous because it would also encompass --given the dual use of this technology --a literal, military arms race. But even if this were not the case, we believe there would be risks -e.g. from corner-cutting in safety and consultation. Although some might argue that the research community should altogether avoid using AI 'race' terminology for fear of giving it currency, we believe that the idea is already current enough to justify interventions that draw attention to the dangers. Much work remains to be done in understanding the dynamics of a possible race, and in developing alternative 			 For example, few countries have published formal strategies or legislation on AI, but a number have commissioned reviews that have sought expert opinion, eg, the UK Government-commissioned report on AI(Hall and Prescenti, 2017)  and the White House report (NSTC, 2016).
			 See https://www.partnershiponai.org/thematic-pillars/ 3  In which both authors were involved. framings for AI development --but there are encouraging examples on which to build.