Abstract:
Decision theorists disagree about how instrumentally rational agents, i.e., agents trying to achieve some goal, should behave in so-called Newcomb-like problems, with the main contenders being causal and evidential decision theory. Since the main goal of artificial intelligence research is to create machines that make instrumentally rational decisions, the disagreement pertains to this field. In addition to the more philosophical question of what the right decision theory is, the goal of AI poses the question of how to implement any given decision theory in an AI. For example, how would one go about building an AI whose behavior matches evidential decision theory's recommendations? Conversely, we can ask which decision theories (if any) describe the behavior of any existing AI design. In this paper, we study what decision theory an approval-directed agent, i.e., an agent whose goal it is to maximize the score it receives from an overseer, implements. If we assume that the overseer rewards the agent based on the expected value of some von Neumann-Morgenstern utility function, then such an approval-directed agent is guided by two decision theories: the one used by the agent to decide which action to choose in order to maximize the reward and the one used by the overseer to compute the expected utility of a chosen action. We show which of these two decision theories describes the agent's behavior in which situations.
 Keywords Reinforcement learning • Causal decision theory • Evidential decision theory • Newcomb's problem • AI safety • Philosophical foundations of AI B Caspar Oesterheld

Approval-directed agency and the decision theory of Newcomb-like problems

Introduction In decision theory, there is a large debate about how an instrumentally rational agent, i.e., an agent trying to achieve some goal or maximize some utility function, should decide in Newcomb's problem (introduced by  Nozick 1969 ) and variations thereof (a list is given by  Ledwig 2000, pp. 80-87) . Consequently, different normative theories of instrumental rationality have been developed. The best known ones are evidential (sometimes also called Bayesian) decision theory (EDT)  (Ahmed 2014; Almond 2010; Price 1986; Horgan 1981 ) and causal decision theory (CDT)  (Gibbard and Harper 1981; Joyce 1999; Lewis 1981; Skyrms 1982; Weirich 2016 ), but many have attempted to remediate what they view as failures of the two theories by proposing further alternatives  (Spohn 2003 (Spohn , 2012 Poellinger 2013; Arntzenius 2008; Gustafsson 2011; Wedgwood 2013; Dohrn 2015; Price 2012; Soares and Levinstein 2017) . Because the main goal of artificial intelligence is to build machines that make instrumentally rational decisions  (Russell and Norvig 2010, Sects. 1.1.4, 2.2; Legg and Hutter 2007; Doyle 1992) , this normative disagreement has some bearing on how to build these machines (cf. Soares and Fallenstein 2014a, Sect. 2.2;  Soares and Fallenstein 2014b, Sect. 1; Bostrom 2014b, Chap. 13, Sect. "Decision theory") . The differences between these decision theories are probably inconsequential in most situations  (Ahmed 2014, Sect. 0.5, Chap. 4; Briggs 2017 ), 1 but still matter in some  (Ahmed 2014, Chap. 4-6; Soares 2014a; Bostrom 2014a) . In fact, AI may expose the differences more often. For example, Newcomb's problem and the prisoner's dilemma with a replica  (Kuhn 2017, Sect. 7 ) are easy to implement for agents with copyable source code (cf.  Yudkowsky 2010 pp. 85ff .  Soares and Fallenstein 2014b, Sect. 2; Soares 2014b; Cavalcanti 2010; Sect. 5) . Indeed, the existence of many copies is the norm for (successful) software, including AI-based software. While copies of present-day software systems may only interact with each other in rigid, explicitly pre-programmed ways, future AI-based systems will make decisions in a more autonomous, flexible and goal-driven way. Overall, the decision theory of Newcomb-like scenarios is a central foundational issue which will plausibly become practically important in the longer term. The problem for AI research posed by the disagreement among decision theorists can be divided into two questions: 1. What decision theory do we want an AI to follow? 2. How could we implement such a decision theory in an AI? Or: How do decision theories and AI frameworks or architectures map onto each other? Although it certainly requires further discussion, there already is a large literature related to the first question. 2 In this paper, I would thus like to draw attention to the second question. Specifically, I would like to investigate how approval-directed agents behave in Newcomb-like problems. By an approval-directed agent, I mean an agent that is coupled with an overseer. After the agent has chosen an action, the overseer scores the agent for that action. Rather than, say, trying to bring about particular states in the environment, the agent chooses actions so as to maximize the score it receives from the overseer (cf.  Christiano 2014) . A model of approval-directed agency that allows us to describe Newcomb-like situations is described and discussed in Sect. 2. Approval-directed agency is intended as a model of reinforcement learning agents (see  Sutton and Barto 1998; Russell and Norvig 2010; Chaps. 17, 21 , for introductions to reinforcement learning), for whom the reward function is analogous to the approval-directed agent's overseer. Since reinforcement learning is such a general and commonly studied problem in artificial intelligence  (Hutter 2005, e.g. Chap. 4.1.3; Russell and Norvig 2010, p. 831; Sutton and Barto 1998, Chap. 1) , it is an especially attractive target for modeling. 3 However, because decision theories are usually defined only for single decisions, we will only discuss single decisions whereas reinforcement learning is usually concerned with sequential interactions of agent and environment. However, this decision can also be a policy choice to model sequential decision problems.  4  In addition to limiting our analysis to single decisions, we will not discuss the learning process and simply assume that the agent has already formed some model of the world. If we assume that, after an action has been taken, the overseer rewards the agent based on the expected value of some von Neumann-Morgenstern utility function, the agent is implicitly driven by two decision theories: The overseer can use the regular conditional expectation or the causal expectation to estimate the value of its utility function; and the agent itself can follow CDT or EDT when maximizing the score it receives from the overseer (Sect. 3). We then show how the overall decision theory depends on these two potentially conflicting decision theories. If the overseer bases its expected value calculations on looking only at the world, then the agent's decision theory is decisive. If the overseer Footnote 2 continued  (Meacham 2010; Soares and Fallenstein 2014b, Sect. 3; Yudkowsky 2010, Sect. 2; Greene 2018)  . The same arguments imply that even if one is convinced of CDT or EDT one would not want the AI to use CDT and EDT. That said, one could also leave the self-modification to the AI.  3  Reinforcement learning and approval-directed agency are also common outside of artificial intelligence. For example,  Achen and Bartels (2016, Chap. 4 ) review evidence which shows that electorates often vote retrospectively to punish or reward incumbents.  1  A causal model of an approval-directed agent in a Newcomb-like decision problem. A denotes the agent's action, H the environment history, O the observation on which the overseer bases the reward, R is that reward, and E r is information about the way the reward is computed that is only available to the overseer. The box is used to indicate that H includes the two random variables H p and H f . All of H may have a causal influence on O bases its estimates only on the agent's action, then the overseer's decision (or perhaps rather action evaluation) theory is decisive.
 A H p H f R O E r H Fig.
 Approval-directed agency We first describe a model of approval-directed agency. To be able to apply both CDT and EDT, we will use causal models in  Pearl's (2009)  sense. Consequently, we use Pearl's do-calculus-based version of CDT  (Pearl 2009, Chap. 4 ). We will, throughout this paper, assume that the agent has already formed a (potentially implicit) model of the world 5 -e.g., based on past interactions with the environment. Also, we will only consider single decisions rather than sequential problems of iterative interaction between agent and environment. A causal model of such a one-shot Newcomb problem from the perspective of the approval-directed agent is given in Fig.  1 . In this model, the agent decides to take some action A, which may causally affect some part of the environment history, i.e., the history of states, H . We will call that part of the history the agent's causal future H f . Furthermore, the agent may be causally influenced by some other part of the environment history, which we will call the agent's causal past H p . H may contain information other than H f and H p , which we will assume to be independent of A.  6  The overseer, physically realized by, e.g., some module physically attached to the agent or a human supervisor, observes the agent's action and partially, via some percept O, the state of the world 7 . The overseer then calculates the reward R. To set proper incentives to the agent, we will assume the overseer to know not only the action and observation, but also everything that the agent knows (cf.  Christiano 2016) . The overseer may also have access to some additional piece of information E r about the way the reward is to be calculated. 8 Lastly, we assume that the sets of possible values of A, O and E r are finite. In principle, the overseer could reward the agent in all kinds of ways. E.g., it could reward the agent "deontologically" (Alexander and Moore 2016) for taking a particular action independently of the consequences of taking that action. In this paper, we will assume that the reward estimates the value of some von Neumann-Morgenstern utility function U that only depends on states of the world. I use the capital U to indicate that the utility function, too, is a random variable (in the Bayesian sense). For simplicity's sake, we will, again, assume that the set of possible values of U is finite. We will view U as representing the system designer's preferences over world states. 9 While other ways of assigning the reward are possible, this is certainly an attractive way of getting an approval-directed agent to achieve goals that we want it to achieve. After all, in real-world applications, we will usually care about the outcomes of the agent's decisions, such as whether a car has reached its destination in time or whether a human has been hurt. The standard way of estimating U (H ) (or any quantity for that matter) is the familiar conditional expectation. Thus, the overseer may compute the reward as r = E [U (H ) | e r , a, o] , (1) Footnote 6 continued know your common source code), the dependence persists. We exclude these dependences because such situations cannot be modeled by standard causal graphs. However, we could adapt causal graphs to accomodate for these kinds of dependences. First, we could modify our definition of causality in such a way that dependence does imply causation, as has been proposed by  Spohn (2003 Spohn ( , 2012 ,  Yudkowsky (2010)  and others. For instance, we could model the dependence between the outputs of two instances of an algorithm by introducing a logical node as a common cause of the two. This logical node would then represent the output of the abstract algorithm that the two copies implement. While changes to the concept of causation may affect CDT's implied behavior, the results from this paper can be directly transfered to such modifications. Alternatively, we could extend causal graphs to also include non-causal dependences (cf.  Poellinger 2013) . Such extension necessitates a new CDT formalism, so the proofs from this paper do not directly transfer to this case. That said, I expect our results to generalize given that both EDT and CDT would probably treat non-causal dependences on the action just like they treat causal arrows directed toward the action.  7  Christiano (2014) does not define approval-directed agency formally, but judging from a comment he made at https://medium.com/paulfchristiano/i-agree-that-the-key-feature-of-approval-directed-agents-isthat-the-causal-picture-is-736b4474910e, he considers it crucial to his conception that the overseer only looks at the agent's action and does not observe the action's consequences (cf. the distinction introduced in Sect. 3).  8  One reason for the overseer to have access to such additional information is that some of the human supervisor's values may not be expressible in a way that the approval-directed agent's algorithm can utilize (cf.  Muehlhauser and Helm 2012, Sects. 3, 4, 5.3 ). 9 Some have tried to modify the reward relative to the designer's preferences to make the reinforcement learning problem easier to solve  (Sorg 2011) , although  Sutton and Barto (1998, Sect. 3 .2) explicitly discourage such tricks in their reinforcement learning textbook. where r , a, e r , and o are values of R, A, E r , and O, respectively. 10 A causal decision theorist overseer agrees that after an action a is taken the righthand side of Eq. 1 most accurately estimates how much utility is achieved. She merely thinks that this term should not be used to decide which action a to take in the first place. 11 However, this puts a causal decision theorist overseer in a peculiar situation. Whatever formula she uses to compute the reward will also be used by the rewardmaximizing agent to decide which action to take. A causal decision theorist overseer might therefore worry (rightfully, as we will see) that providing rewards according to Eq. 1 will make the agent EDT-ish. Hence, she either has to incorrectly estimate how much utility was achieved; or live with the agent using an-in her mind-incorrect way of weighing her options. If she prefers the latter, she would reward according to Eq. 1. But arguably getting the agent to choose correctly is the overseer's primary goal. Thus, she might prefer to compute the reward according to r = E [U (H ) | e r , do(a), o] . (2) Here, do(a) refers to Pearl's do-calculus, where conditioning on do(a) roughly means intervening from outside the causal model to set A to a. For an introduction to the do-calculus, see  Pearl (2009) . Although a causal decision theorist overseer may prefer computing rewards according to Eq. 1, we will from now on say "the overseer uses CDT" if rewards are computed according to Eq. 2 and "the overseer uses EDT" if rewards are calculated according to Eq. 1. An approval-directed agent is characterized by maximizing the reward it receives from the overseer. 12 However, decision theory offers us, again, (at least) two different expected values, the regular expected value of EDT E [R | a] , (3) and CDT's causal expected value E [R | do(a)] . ( 4 ) 10 At first sight this may be confusing to some readers, because in reinforcement learning, utility sometimes refers to expected cumulative reward  (Russell and Norvig 2010, Chap. 17, 21) , although others use the term value function instead  (Sutton and Barto 1998, Sect. 3.7 ). Here, U does not refer to utility in that sense but in the decision-theoretical sense of representing intrinsic values. So, in the present case, we have two "layers" of goals: first, the agent maximizes the reward r . Second, the agent as incentivized by the overseer's way of calculating rewards maximizes utility U (H ). One cause of confusion is that in model applications of reinforcement learning, the reward function possesses full knowledge of the world state and thus does not require the use of the expectation operator. 11 If the disagreement in Newcomb's problem is to be about different theories of rational choice (EDT, CDT and so forth) rather than the predictive abilities of "the being", Omega or the psychologist, then after requesting both boxes a proponent of two-boxing has to believe that she will probably receive only $1000. Causal and evidential decision theorists agree that regular conditional expectation is the correct way of updating one's beliefs about the state of the world after an action has been taken (cf. the distinction between "acts" and "actions" in  Pearl 2009 Sect. 4.1.1) . We leave the interesting question of which (if any) decision theory describes the behavior of current reinforcement learning algorithms to future research 13 and in the following assume that the agent is known to implement either CDT or EDT.
 The conflict of the decision theories of agent and overseer When viewed together with the overseer, our agent may now be seen as containing two decision theories, one for computing the reward and one in the algorithm that tries to find the action to maximize that reward. These decision theories may not always be the same. Given this potential discrepancy, the question is which of the two decision theories prevails, i.e., for which configurations of the two decision theories the overall agent acts like a CDT agent and for which it acts like an EDT agent w.r.t. U . As it turns out, the answer to this question depends on the decision problem in question. In particular, it depends on whether the overseer updates its estimate of U (H ) primarily based on the action taken by the agent or on its observation of the environment. For illustration, consider two versions of Newcomb's problem. In both versions, the predictor is equally reliable-e.g., correct with 90% probability-and the potential box contents are the same-e.g., the standard $1K and $1M. As usual, the content of the opaque box cannot be causally influenced by one's decision. In the first version, the overseer eventually sees the payoff, i.e., how much money the agent has made. In this case, as soon as the money is observed, the overseer's estimate of U (H ) becomes independent of the agent's action. More generally, O may tell the overseer so much about U (H ) that it becomes independent of A even if U (H ) is not yet fully observed. That is, E [U (H ) | e r , a, o] = E [U (H ) | e r , o] (5) and E [U (H ) | e r , do(a), o] = E [U (H ) | e r , o] (6) for all e r , a and o. Note that neither of these two implies the other.  14  Intuitively speaking, these two mean that the reward is ultimately determined by U (H ). In the second version of Newcomb's problem, the monetary payoff is not observed but covertly invested into increasing the agent's utility function. Only the agent's choice can then inform the overseer about U (H ). Formally, it is both   E [U (H ) | e r , a, o] = E [U (H ) | e r , a] (7) and E [U (H ) | e r , do(a), o] = E [U (H ) | e r , do(a)] . (8) Intuitively speaking, these two equations mean that the reward is not determined by U (H ) but by what the overseer believes U (H ) will be given a or do(a). Again, we assume that this is known to the agent. An example class of cases is that in which the agent's decisions are correlated with those of agents in far-away parts of the environment (cf.  Treutlein and Oesterheld 2017; Oesterheld 2018b ). The two versions are depicted in Fig.  2 . Of course, these are only the two extremes from the set of all possible situations. In real-world Newcomb-like scenarios, the overseer may also draw some information from both sources. Nonetheless, it seems useful to understand the extreme cases, as this may also help us understand mixed ones. In the following subsections, we will show that in the first type, the decision theory of the agent is decisive, whereas in the second type, the overseer's decision theory is 15 . Roughly, the reason for that is the following: As noted earlier, the reward in the first type depends directly on U (H ). Thus, the agent will try to maximize U (H ) according to its own decision theory. In the second type, the overseer takes the agent's action a and then considers what either a or do(a) says about U (H ). Thus, the agent has to pay careful attention to whether the overseer uses EDT's or CDT's expected value. We prove this formally by considering all possible configurations of the type of the problem, the overseer's decision theory and the agent's decision theory. While we will limit our analysis to EDT and CDT, the results can easily be generalized to variants of these that arise from modifying the causal model or conditional credence distribution (e.g. Yudkowsky 2010; "Disposition-based decision theory"; Spohn 2012;  Dohrn 2015 ). The analysis is summarized in Table  1 .
 First type
 The EDT agent The EDT agent judges its action by E [R | a] . (9) If the overseer calculates regular conditional expectation, then it is E [R | a] = E [E [U (H ) | E r , O, a] | a] (10) = E [U (H ) | a] , ( 11 ) where the last line is due to what is sometimes called the law of total expectation (LTE) or the tower rule (see, e.g.,  Ross 2007, Sect. 3.4; Billingsley 1995, Theorem 34.4 ). Intuitively, you cannot expect that gaining more evidence (i.e., E r and O in addition to a) moves your expectation of U (H ) into any particular direction. Because the overseer knows more than the agent, we will need this rule in all of the following derivations. Its application makes it hard to generalize these results to other decision theories, since LTE does not apply if the two decision theories do not both compute a form of expected utility. Equations 10 and 11 show that if the overseer computes regular expected value and the agent maximizes the reward according to EDT, then the agent as a whole maximizes U according to EDT. If the overseer computes CDT's expected value, it is  E [R | a] = E [E [U (H ) | E r , do(a), O] | a] ( 12 = E [E [U (H ) | E r , a, O] | a] (15) = LTE E [U (H ) | a] (14) (16)
 The CDT agent The CDT agent judges its action by E [R | do(a)] . ( 17 ) If the overseer uses regular expected value (EDT), then E [R | do(a)] = E [E [U (H ) | a, O, E r ] | do(a)] (18) = e r ,o P(e r , o | do(a)) • E [U (H ) | a, o, e r ] (19) = eq. 5 and 6 e r ,o P(e r , o | do(a)) • E [U (H ) | do(a), o, e r ] (20) = E [E [U (H ) | do(a), O, E r ] | do(a)] (21) = LTE E [U (H ) | do(a)] (22) Learning about an intervention do(a) cannot always be treated in the same way as learning about other events. Hence, the application of the law of total expectation is not straightforward. However, P(• | do(x)) is always a probability distribution. Because the law of total expectation applies to all probability distributions, it also applies to ones resulting from the application of the do-calculus. If the overseer uses CDT's expected value, then E [R | do(a)] = E [E [U (H ) | E r , O, do(a)] | do(a)] (23) = LTE E [U (H ) | do(a)] . (24)
 Second type
 The EDT agent The EDT agent judges its actions by E [R | a] . ( 25 ) If the overseer is based on regular conditional expectation (EDT), then it is again E [R | a] = E [E [U (H ) | E r , a] | a] (26) = LTE E [U (H ) | a] . ( 27 ) If the overseer is based on CDT-type expectation, then (33) E [R | a] = E [E [U (H ) | E r , do(a)] | a] ( 28
 The CDT agent The CDT agent judges actions by E [R | do(a)] . ( 34 ) Because of Rule 2 in Theorem 3.4.1 of  Pearl (2009, Sect. 3.4. 2) applied to the causal graph of Fig.  2b , it is E [R | do(a)] = E [R | a] . (35) Thus, the analysis of the CDT agent is equivalent to that of the EDT agent.
 Conclusion In this paper, we have taken a step to map reinforcement learning architectures onto decision theories. We found that in Newcomb-like problems, if the overseer rewards the agent purely on the basis of the agent's action, then the overall system's behavior is determined by the decision theory implicit in the overseer's reward function. If the overseer judges the agent based on looking at the world, however, then the agent's decision theory is decisive. This has implications for how we should design approval-directed agents. For instance, if we would like to leave decision-theoretical judgements to the overseer, we must ensure that the overseer assigns rewards before making new observations about the world state (cf.  Christiano 2014, Sect. "Avoid lock-in") . Of course, this makes the reward less accurate and may thus slow down the agent's learning process. If we want the overseer to look at both the world and the agent's action, then we need to align both the overseer's and the agent's decision theory. Much more research is left to be done at the intersection of decision theory and artificial intelligence. For instance, what (if any) decision theories describe the way modern reinforcement learning algorithms maximize reward? Do the results of this paper generalize to sequential decision problems? Moving away from the reinforcement learning framework, what decision theories do other frameworks in AI implement? What about decision theories other than CDT and EDT? The reward is computed based on ob-(b) The reward is computed based on observing the agent's action.
 Fig. 2 2 Fig. 2 Two different ways in which the overseer can calculate the reward
 r , o | a) • E [U (H ) | e r , do(a), o] (13) = eq. 5 and 6 e r ,o P(e r , o | a) • E [U (H ) | e r , a, o]
 r | a) • E [U (H ) | do(a), e r ] r ) • E [U (H ) | do(a), e r ] r | do(a)) • E [U (H ) | do(a), e r ] (31) = E [E [U (H ) | E r , do(a)] | do(a)] (32) = LTE E [U (H ) | do(a)] .
 Table 1 1 An overview of the results of the calculations in Sect. 3 Type of Newcomb problem Agent's DT Overseer's DT Resulting DT First type CDT EDT CDT CDT CDT EDT EDT EDT CDT EDT Second type CDT EDT EDT CDT CDT EDT EDT EDT CDT CDT
			 In fact, Eells (1981)  has argued that EDT and CDT always behave in the same way, but I disagree with this assessment based on the reasons given byAhmed (2014, Sect. 4.3-4.6) and Price (1986) .2  Of course, the existing literature asks about the right decision theory proper. The answer to that question might differ from the answer to the AI-specific question (cf. Kumar 2017;  Treutlein 2018). After all, even if we have identified the right decision theory for ourselves, we may want to implement a different decision theory in an AI. One reason could be that the main contenders are not self-recommending-it has been pointed out that EDT and CDT both recommend to self-modify into slightly different decision theories
			 This is consistent with what reinforcement learning algorithms usually do-they choose policies rather than individual actions. This is because the utility of a single action usually cannot be evaluated without knowing how the agent will deal with situations that might arise as a result of taking that action. When individual actions can be evaluated in isolation, the ex ante policy choice sometimes differs from the choice of individual actions (see the absent-minded driver, introduced by Piccione and Rubinstein 1997; cf. Aumann et al. 1997 ; the Newcomb-like scenarios discussed by, e.g., Hintze 2014; Soares and Levinstein 2017, Sect. 2; and the problems in anthropics discussed by Armstrong 2011) . While it is rarely discussed in the debate between evidential and causal decision theorists, a few authors regard this discrepancy as crucial and have argued that a proper decision theory should be about optimal policy choices (e.g. Hintze 2014;   Soares and Fallenstein 2014b, Sect. 2.1; Soares and Levinstein 2017, Sect. 2). However, this issue is beyond the scope of the present paper. Further issues in sequential Newcomb-like problems are discussed by Everitt et al. (2015) .
			 There is a broad philosophical literature on whether causal relationships exist and whether they can be inferred in cases where the agent is part of the environment. See, e.g., the edited volume byPrice and Corry  (2007).6  For simplicity, we will ignore dependences not resulting from causation (Arntzenius 2010) . For example, if you play against a copy, there is a logical dependence between your and your copy's decision. Even if you know a set of nodes in the causal graph that d-separates your and your copy's decision (e.g., if you
			 In reinforcement learning, some have proposed alternative optimization targets that incorporate, e.g., risk aversion(García and Fernández 2015, Sect. 3).
			 For preliminary work on this question, see Mayer et al. (2016) , Oesterheld (2018a)  and perhaps Albert and Heiner (2001) .14  We give a brief justification of this claim. If all of a's causal influence on H can be discerned from O, then, of course, a could still be diagnostically relevant for one's estimate of U (H ). The other direction is more complicated. The idea is that Eq. 5 can be true if the causal and non-causal implications of a exactly cancel each other out. An example is a version of Newcomb's problem in which one-boxing ensures with certainty that both boxes contain the same amount of money. Then if O and E r do not contain any information, the expected value of two-boxing and one-boxing is the same and so learning of the action is irrelevant for estimating U (H ). However, two-boxing is causally better than one-boxing, so Eq. 6 is violated.
			 The dominance of the overseer's decision theory in the second type of Newcomb's problem is mentioned (though not proven) byChristiano (2014, Sect. "Avoid lock-in").
			 Synthese (2021) 198 (Suppl 27):S6491-S6504


Abstract: Sustainability is widely considered a good thing and is therefore a matter of ethical significance. This paper analyzes the ethical dimensions of existing work on AI and sustainability, finding that most of it is focused on sustaining the environment for human benefit. The paper calls for sustainability that is not human-centric and that extends into the distant future, especially for advanced future AI as a technology that can advance expansion beyond Earth.

The Ethics of Sustainability for Artificial Intelligence

Introduction A basic attribute of modern human civilization is that the stock of natural resources steadily decreases, whereas the stock of artificial resources steadily increases. For example, artificial intelligence (AI) research is commonly powered by the burning of fossil fuels, and in the process produces new technologies that civilization can benefit from. Will the increases in artificial resources be sufficient to offset the loss of natural resources, such that civilization can be sustained into the future? That is one important perspective on the ethics of sustainability as it relates to AI, though, as this paper discusses, it is not the only one. Sustainability is not an inherently ethical concept. In its essence, "sustainability" refers to a particular characteristic of systems as they change over time. The term can be used in many ways that do not have any particular ethical significance. For example, sprinters run at an unsustainable speed; eventually, their muscles will fatigue and they will be unable to continue. This is a basic characteristic of human physiology and not a matter of ethical significance. In common usage, however, sustainability takes on ethical significance. Sustainability is widely treated as a good thing and something worth pursuing  [1] . It is in that spirit that there have been initiatives on AI and sustainability, including conferences such as "Sustainable AI",  1  "Towards Sustainable AI",  2  and "AI for the Planet",  3  as well as groups such as AI4Good that work on AI in support of the United Nations Sustainable Development Goals (SDGs).  4  It is also in that spirit that sustainability is one of the principles found in some AI ethics guidelines  [2] . The ethical dimensions of sustainability warrant ethical analysis. Important ethics questions include: What exactly should be sustained, or rather, be able to be sustained? Why should it be able to be sustained? For how long? How much emphasis should be placed on sustainability relative to other goals? These ethics questions can be answered in a variety of ways. How they are answered has important implications for ongoing human activity, including for the development, use, and governance of AI technology. Prior literature on AI and sustainability (reviewed below) has not considered the ethical dimensions to any significant extent. Therefore, this paper analyzes the ethics of sustainability as it relates to AI. The paper proceeds in three parts. First, we explain the ethics of sustainability as a general concept (Section 2). Second, we describe the current usage of the concept of sustainability within work on AI (Section 3). Third, we present an argument for a long-term, non-anthropocentric conception of sustainability and explain the implications of this for AI (Section 4). Section 5 concludes. Much of the discussion in this paper covers ethical dimensions of sustainability that are not specific to AI. This is a feature, not a bug. To a large extent, the ethical dimensions of AI and sustainability are the same as those for sustainability in general. Furthermore, aspects of the topic that are specific to AI build on more general sustainability concepts. Therefore, to understand the ethics of sustainability for AI, it is essential to first understand the ethics of sustainability. The paper contributes to the growing literature on AI and sustainability. Most of this literature is on AI in relation to the sustainability of human civilization and its environmental underpinnings; this includes reviews by Nishant et al.  [3]  and Liao and Wang  [4] , the environmental politics of AI  [5] , AI in relation to systemic risk and sustainability  [6] , the environmental footprint of AI systems  [7] , and the role of AI in meeting the SDGs  [8] [9] [10] . Some literature focuses on the sustainability of the AI systems themselves  [11] , including for consumer autonomy  [12] , in global health initiatives  [13] , in certain economic mechanisms  [14] , and in decision-making applications  [15] . Additionally, there is a broader field of computational sustainability that applies computer science methods to advance environmental and social sustainability  [16] [17] [18] . Overall, the literature on AI and sustainability offers a variety of important contributions, but it provides limited discussion of the ethics of sustainability. The paper additionally contributes to some adjacent literatures. Prior studies have considered the application of AI for environmental protection  [19] [20] , for climate change mitigation  [21] [22] , and the energy consumption of AI systems  [23] ; these are relevant for environmental conceptions of sustainability. Also relevant are debates on the relative importance of near-term, medium-term, and long-term AI  [24] [25] [26] [27] ; as this paper discusses, the future-orientation of sustainability can imply an emphasis on long-term AI. Finally, of more general relevance is prior work on the ethics of sustainability  [1] , especially sustainability over 1 https://www.uni-bonn.de/en/news/<PHONE> 2 http://www.wikicfp.com/cfp/servlet/event.showcfp?eventid=140164&copyownerid=158619 3 https://aifortheplanet.org/ 4 https://ai4good.org/ long time scales  [28] , and the ethics of AI  [2, [29] [30] , especially regarding AI and the future  [31]  and AI and nonhumans  [32] .
 The Ethics of Sustainability The word "sustainability" is commonly traced to the German word Nachhaltigkeit, and specifically to Hans Carl von Carlowitz's 1713 treatise on sustainable yield forestry  [33] . The tension between using resources now and preserving or cultivating them for future use is of course much older. Modern analysis traces to the environmental economics work of Hotelling  [34] , which remains relevant today  [35] . Hotelling's work did not use the term "sustainability". Use of the term primarily traces to the 1987 report Our Common Future, which was led by former Norwegian Prime Minister Gro Harlem Brundtland and is commonly known as the Brundtland Report. Like von Carlowitz's treatise, the Brundtland Report specifically conceptualizes sustainability in socio-environmental terms. The report's definition of sustainable development, "Development that meets the needs of the present without compromising the ability of future generations to meet their needs", has been widely influential and serves as a foundation for the UN SDGs. Since Our Common Future, there has been a proliferation of definitions of sustainability and sustainable development  [36] [37] [38] . Widespread and imprecise usage of the term "sustainability" has watered it down. Critics argue that "sustainability" has become "a concept that is equivalent to 'good' and thus devoid of any specific meaning-a blanket concept to assure stakeholders of the policy's good intentions"  [39, p.3439 ]. Likewise, the term is said to have been appropriated by self-interested actors to continue "business as usual" activities that drive environmental destruction and social inequity  [37] . One notable point of criticism is the so-called "three pillars" of sustainability: the social, the economic, and the environmental-or "people, profit, and planet". The pillars construct is an attempt to represent the major components of socio-environmental sustainability. However, these categories have been criticized for being fuzzy and overlapping, for excluding of other relevant categories such as the cultural and the political, and for not being essential to the core matter of whether civilization can be sustained over time  [39] . Indeed, economic transactions are an inherently social activity, and all human activity is inherently environmental because humans are part of nature. Furthermore, sustainability is commonly associated with environmental protection, but some environmental disturbances, such as many types of air or water pollution, rapidly dissipate and have a negligible effect on future generations' ability to sustain themselves. Likewise, some matters that could classify as social and/or economic, such as social justice within the contemporary population, are often of limited relevance to the ability of future generations to sustain themselves, even if these matters may be important for other reasons. Other socio-economic matters, such as education and investment in future economic growth, may be of greater importance to the ability of future generations to sustain themselves. For these reasons, the "three pillars" provide a weak foundation for sustainability. Work on AI and sustainability that incorporates the "three pillars", such as the "AI for People: Towards Sustainable AI" conference, 5 should take note. We now turn to three fundamental questions for the ethics of sustainability (Sections 2.1-2.3) followed by a comparison between sustainability and optimization (Section 2.4). The reasons something should be able to be sustained typically derive from what are, in moral philosophy, referred to as intrinsic value and instrumental value. Roughly speaking, something is intrinsically valuable if it is valuable for its own sake, or valuable as an ultimate end in itself; something is instrumentally valuable if it is valuable because it promotes something else that is valuable  [40] . For example, we might suppose that sunlight is valuable because it can (among other things) be converted into electricity, and that electricity is valuable because it can (among other things) be used to power AI systems, and that AI systems are valuable because they can (among other things) enable humans to have enjoyable lives, and that enjoyable human lives are just good things on their own. In that case, sunlight, electricity, and AI systems are instrumentally valuable, while enjoyable human lives are intrinsically valuable. In many conceptions of ethics, what should be done-including what should be sustained-is defined with reference to some conception of what is intrinsically valuable.
 Conceptions of sustainability can vary in terms of what they intrinsically value and what sorts of instrumental values they focus on. One can seek to sustain X either because X is intrinsically valuable or because X is instrumentally valuable. For example, one can seek to sustain natural ecosystems either because one considers them to be intrinsically valuable or because one considers them to be instrumentally valuable for other things, such as for human welfare. Common conceptions of sustainability are anthropocentric, meaning that they only intrinsically value humans  [41] . For example, although the Brundtland Report's emphasis on future generations could conceivably be interpreted to mean future generations of something other than humans, the Report clearly focuses on humans. Likewise, sustainable management of natural resources generally treats the resources as instrumental values for human benefit. In contrast, some conceptions of sustainability are ecocentric, meaning that they intrinsically value ecosystems. Examples include the Earth Charter 6 and the Earth Manifesto  [42] . Though less common, the concept of sustainability can also be used with other notions of intrinsic value, such as the idea that there is intrinsic value in the welfare of sentient nonhuman animals or (if possible) sentient AI systems. The distinction between intrinsic and instrumental value does not always matter, but it often is important. For example, reducing greenhouse gas emissions is generally good on both anthropocentric and ecocentric grounds. However, some biodiversity protection is worth pursuing mainly on ecocentric grounds, because, for better or worse, certain species could go extinct with little impact on human welfare. Therefore, it is important for discussions of sustainability to explicitly specify what they intrinsically value.
 For how long should something be able to be sustained? There is a big difference between sustaining something for a few days and sustaining it for decades, centuries, or even indefinitely into the distant future. Unfortunately, discussions of sustainability are often not precise in their consideration of time scales. For example, the 1987 Brundtland Report's emphasis on future generations implies a time scale of at least decades, assuming the generations are of humans. But how many future generations? The actions needed to enable the next few generations to be sustained often differ significantly from the actions needed to enable the same for every generation that could ever exist.
 How much effort should be made for sustainability? The sustainability of intrinsic or instrumental values may be a good thing, but how good? The world has many competing values and opportunities. Indeed, the Brundtland definition was specifically crafted to acknowledge the competing values of present and future generations; while the report aspires to promote actions that support both the present and future generations, many choices involve tradeoffs between them. For example, depleting natural resources often benefits present generations at the expense of future generations. Basic research often benefits future generations at the expense of present generations, especially where the same resources could instead be used for applied research. The relative importance of the present and future can be operationalized in a variety of ways, such as through discount rates or other weighting functions  [43] [44] . This sort of intergenerational evaluation is generally made within the context of anthropocentric conceptions of sustainability, but similar approaches can be taken with other conceptions. One way or another, moral guidance about sustainability must consider how to evaluate tradeoffs between sustainability and other moral goals. This point fits within the broader issue of tensions between different AI ethics principles  [45] . Also relevant are fundamental questions about the appropriate degree of effort to take to achieve ethical goals. In moral philosophy, the term "supererogation" refers to actions that "go beyond the call of duty", meaning that they are good but not strictly required  [46] . One common question in moral philosophy is whether some moral frameworks are too demanding. This question is especially acute for consequentialist moral frameworks that call for moral agents to maximize some conception of intrinsic value, because maximization is a demanding task  [47] . These are important questions for any moral debate, certainly including those involving sustainability. Setting aside tradeoffs between sustainability and other moral goals, one can ask: How much effort should a person or an organization make to advance sustainability? Should they "give it everything they've got"? Or would just a little effort be acceptable? Conversely, is it enough to work to advance sustainability? Or is it important to also work toward more ambitious goals, such as intertemporal optimization of intrinsic value?
 Sustainability vs. Optimization Sustainability can be an optimization criterion-that would mean seeking to optimize the ability for something to be sustained over time. However, this is distinct from optimizing intrinsic value. Sustainability means enabling something to be sustained in at least some minimal form; optimization means making something be the best that it can be. Ensuring sustainability is perhaps best understood as a basic minimum standard of intertemporal conduct, whereas the intertemporal optimization of intrinsic value may be understood as a loftier ideal to aspire for. This distinction can be seen, for example, in the Brundtland Report call for the present generation to act "without compromising the ability of future generations to meet their needs". The basic needs of human life are, to an approximation, food, clothing, and shelter. Following the Report's call could result in "a society living forever at a minimum subsistence level of consumption"  [48, p.327] . Future society would be able to meet its needs, but it may not be able to do anything more. If the present generation does not act so as to enable future generations to do much better than meeting their needs, then the present generation will, quite arguably, have squandered a massive opportunity. Of course, if the present generation fails to enable future generations to meet their needs, that would be, quite arguably, a massive loss. AI is advanced technology. AI research and development is often oriented toward enabling higher standards of living instead of enabling basic future needs. Human lives do not strictly need, for example, AI systems to steer vehicles or search the internet. Such work generally falls outside the scope of sustainability, but could fall within the scope of optimizing intrinsic value.
 Prior Work on AI and Sustainability With the ethics of sustainability in mind, we now survey prior work on AI and sustainability. Sections 3.1 and 3.2 present quantitative analysis of trends in the ethics of sustainability found in AI ethics principles and AI research. Both analyses characterize sustainability in terms of the three ethics dimensions presented in Section 2: intrinsic value, time scale, and degree of effort. Section 3.3 presents overarching trends across Sections 3.1-3.2.
 AI Ethics Principles Jobin et al.  [2]  compiles 84 sets of AI ethics principles. 11 of these sets of principles include some reference to sustainability.  7  This indicates that sustainability is a small but nonzero priority in AI ethics. We examined the ethical basis of the 11 sets of principles. We found that 7 refer to some form of environmental sustainability, 3 refer to sustainability of the AI system itself, and 1 refers to sustainable social development. We further found that 3 intrinsically value humans, 5 intrinsically value humans and nonhumans including ecosystems, all life, biodiversity, and the planet, and 5 are ambiguous in terms of what is intrinsically valued. Regarding time scales, 2 refer to "future generations" and 9 do not specify time scales. Finally, none of the principles specify degree of effort. Some examples of the AI ethics principles are presented in Appendix A.
 AI Sustainability Research We performed a systematic mapping review  [49]  of research at the intersection of AI and sustainability. Our review maps this literature in terms of the ethical attributes presented in Section 2 and also used in Section 3.1, with some additional nuances to catch the diversity of the research literature. Specifically, we analyzed results from a Google Scholar search for ["artificial intelligence" "sustainability"] and ["AI" "sustainability"] conducted between Sept 16 and 21, 2021. The Google Scholar search engine was selected over other academic databases due to its inclusivity. Whereas databases such as Web of Science concentrate on 7 Table  3  of Jobin et al.  [2]  states that 14 sets if principles include sustainability, but only 12 are referenced in the text and we were unable to identify the other 2. Of the 12 referenced sets of principles, we found that 1 did not cover sustainability, leaving a total of 11 for our analysis. peer-reviewed journals, artificial intelligence research is often published in other spaces such as arXiv. The searches returned 229,000 total results for ["artificial intelligence" "sustainability"] and 1,490,000 total results for ["AI" "sustainability"], respectively. We examined the first ten pages of each of the two searches. We observed that after ten pages of each search, the search results became repetitive and less relevant. These two sets of ten pages contained 200 total results, or 153 results after duplicates were extracted. Of these 153 publications, we were unable to access 11. For the remaining 142 publications, we examined the text in the degree of detail needed to categorize its treatment of sustainability. For most of the publications, this involved looking at the abstract and introduction, and skimming the text for discussion of sustainability. In some cases, we examined the entire publication in more detail. Out of the 142 publications, 60 were found not to be relevant because they were not sufficiently on the nexus of AI and sustainability, leaving a data set of 82 publications. 66 publications were on environmental sustainability, 7 were on the sustainability of the AI system itself, and 9 were on the sustainability of something else, including 2 on the sustainability of organizations, 2 on social sustainability in human-robot/AI interactions, 1 on the sustainability of group decision-making processes, 1 on sustainable curriculum planning, 1 on sustainable healthcare systems, 1 on the social sustainability of AI, and 1 on sustainable industrial development. Of the 66 environmental sustainability publications, 43 were on environmental and social sustainability and 23 were exclusively on environmental sustainability. 29 of the 66 environmental sustainability publications referred to the Brundtland definition and/or the SDGs. 56 publications intrinsically valued humans only, 10 intrinsically valued humans and nonhumans including nonhuman species, life on Earth, biodiversity, ecosystems, the biosphere, and the planet. 16 were too ambiguous to interpret any notion of intrinsic value. Regarding time scales, 7 refer to "future generations" and 1 refers to a time frame from 1990 to 2028. The other 74 do not specify time scales. None of the publications specify degree of effort. Some examples of the AI sustainability publications are presented in Appendix B.
 Overarching Trends In consideration of the work presented in the two preceding subsections, the following overarching trends in the ethics of existing work on AI and sustainability can be identified. First, most work on AI and sustainability is focused on some form of environmental sustainability, with substantial minorities focused on the sustainability of AI systems or on the sustainability of miscellaneous other things. The environmental sustainability work mainly intrinsically values humans and sometimes intrinsically values nonhumans. These trends are consistent with wider usage of sustainability outside the context of AI. Indeed, work on AI and sustainability often explicitly links to broader treatments of sustainability, such as the Brundtland Report and the UN SDGs. Work on the sustainability of AI systems treats AI systems as instrumentally valuable, such as in decision-making applications  [15]  and in global health initiatives  [13] . Outside the context of sustainability, some research considers that AI systems could be intrinsically valuable  [50] [51] [52] . We find that the sustainability of intrinsically valuable AI systems has not yet been addressed. Second, work on AI and sustainability is imprecise on its ethical dimensions. As illustrated in Appendices A-B, our classification of AI ethics principles and AI sustainability research involved frequent parsing of ambiguous phrasings. Indeed, outside references to the SDGs or the Brundtland definition, few publications explicitly define sustainability. Within treatments of environmental sustainability, the term "sustainability" was commonly equated with environmental protection, especially efforts to minimize energy and resource consumption, even though environmental issues do not necessarily have sustainability implications. Some work equated "sustainability" with "good for people/and or society" or even just "good", which further drains "sustainability" of its meaning. As discussed in Section 2, these are common problems with sustainability discourse. Our analysis finds that these problems have been reproduced in the AI literature.
 The Moral Case for Long-Term, Non-Anthropocentric Sustainability and Optimization In this section, we present our own views on the ethics of sustainability as it relates to AI. Specifically, we make the case for sustainability that is non-anthropocentric and long-term oriented. We further argue for substantial effort for this conception of sustainability, or, preferably, for the optimization of long-term intrinsic value. Finally, we explain the implications of such sustainability for AI. Each of our arguments depends on certain positions on underlying ethical principles. As with all ethical principles, there is no universal consensus on which position to take. One can disagree with the positions we take, but doing so requires taking a different position on the underlying ethical principles. This is important to bear in mind, especially when considering the implications for AI.
 Non-Anthropocentric Sustainability First, we call for non-anthropocentric conceptions of sustainability. By non-anthropocentric, we mean that humans are not the only entities that are intrinsically valued. Modern science unambiguously shows that humans are members of the animal kingdom and part of nature. Morally significant attributes such as the innate drive to live and flourish or to experience pleasure and pain are not unique to the human species.  8  For purposes of this paper, we set aside important debates about which nonhuman entities to intrinsically value, but some potential examples include sentient nonhuman animals or (if possible) sentient AI systems, natural ecosystems, and biodiversity. We, the authors of this paper, also happen to disagree among ourselves as to which nonhumans are intrinsically valuable, but we agree that some are. There can be legitimate reasons to sometimes intrinsically value humans more than other entities-for example, a human can have a longer and richer life than a spider. However, we see no morally sound reasons to refuse to intrinsically value any nonhuman lives or entities. This means that sustainability should be defined as to also sustain some nonhumans for their own sake: it is not enough to sustain nonhumans for their instrumental role for humans.  9  8 Additionally, some conceptions of intrinsic value are rooted in the attributes of systems, such as interdependencies of biotic and abiotic entities within ecosystems. These holistic conceptions of intrinsic value are also not specific to humans  [53] . 9 For a more detailed argument for non-anthropocentrism advanced within AI ethics, see  [32] . This non-anthropocentrism is at odds with common conceptions of sustainability, including that of the Brundtland Report. In failing to intrinsically value anything other than humans, we believe these conceptions of sustainability are in moral error. It is unfortunate but not surprising that similar anthropocentric tendencies are found within existing work on AI and sustainability (Section 3). Future work on AI and sustainability should be more inclusive of the intrinsic value of nonhumans.
 Long-Term Sustainability Second, we call for sustainability over long time scales. Our motivation for this is an ethical principle of equality across time. In essence, this means that no one or no thing of moral significance should be disadvantaged because of the time in which they happen to exist. A person (for example) is of the same intrinsic value regardless of whether they live in the year 2021 or 2051 or 2151 or even 22021 or any other future time  [54] [55] . This perspective can be justified, for example, by a "veil of ignorance" thought experiment in which one does not know in advance which time period one would exist in  [56] . Under such hypothetical circumstances, it would only be fair to value each time period equally. Taking temporal equality seriously means including attention to all future time periods, including the astronomically distant future. Combined with our call for non-anthropocentrism, this means sustainability should aim to sustain that which is intrinsically valuable into the distant future. This long-termism is broadly consistent with common conceptions of sustainability, though it is different in emphasis. Common conceptions do not specify precise time scales; this is seen, for example, in the Brundtland Report's emphasis on an unspecified number of future generations. With no clear time limit, these conceptions could include the astronomically distant future, though in practice, they focus on matters that are short-term in comparison. We believe this is a moral error, an unjustified exclusion of distant-future generations and distant-future instances of anyone and anything else of intrinsic value.
 Substantial Effort for Sustainability or Long-Term Optimization Third, we call for a high degree of effort toward sustainability, or, preferably, for the optimization of long-term intrinsic value. The astronomically distant future offers astronomically large opportunities for advancing intrinsic value. These opportunities are vastly larger than those available for the present time and the near-term future. This point suggests a high degree of priority for actions oriented toward the long-term. That does not mean ignoring the present. As members of the present time period, we have special opportunities to help with present circumstances. The present also sets the stage for the future. Nonetheless, if the principle of equality across time is to be taken seriously, it requires a major focus on long-term outcomes. We further believe that people should make great efforts to advancing moral progress of all types, including sustainability, balanced mainly by the need for reasonable selfcare, and that organizations and institutions should likewise be oriented accordingly. An important perspective comes from the physics of the long-term future. Earth will become uninhabitable in roughly one billion years due to the gradual warming and expanding of the Sun  [57] . Survival beyond this time can only occur in outer space. For Earth-originating entities, this will require an advanced technological civilization capable of settling in outer space. Human civilization is already positioned to accomplish this task, given its ongoing space missions and general technological progress. As long as human civilization remains intact, the ability to sustain Earth-originating entities will persist. Long-term sustainability requires resettling in outer space  [28] . For the present generation, that means keeping human civilization intact. In many contexts, there is no significant distinction between sustainability and intertemporal optimization for the distant future. Both goals require maintaining the basic functionality of civilization, including by sustaining sufficient resources and by handling major threats such as global warming, pandemics, and nuclear warfare. They likewise entail evaluating environmental threats in terms of their implications for the continuity of human civilization and not in terms of biogeophysical disturbances or smaller-scale human consequences  [58] [59] . However, looking ahead, the goals point in different directions. Sustaining Earth-originating entities into the distant future only requires some minimal space settlement over very long timescales. In contrast, optimization of long-term intrinsic value entails space expansion sooner and at larger scales, in order to fill the universe with whatever is intrinsically valuable. The distinction between sustainability and optimization of long-term intrinsic value is also important in terms of what is intrinsically valuable. Long-term sustainability can entail the same course of action for both anthropocentric and non-anthropocentric conceptions of intrinsic value: If humanity fails to settle in outer space, then other Earth-originating entities would also die out in a billion or so years, if not sooner  [28, 60] . However, long-term optimization generally entails expansion into outer space-but expansion in what way? Anthropocentrism would entail expansion of human populations, whereas nonanthropocentrism would entail expansion of something else.
 Implications for AI AI has several important roles to play in the story outlined above. First, current and near-term forms of AI can be applied to addressing certain immediate threats to global civilization. For example, AI is in active use for addressing global warming and environmental protection in a variety of ways, and additional ways have been identified and called for  [21] . AI is also in active use for addressing the ongoing COVID-19 pandemic by supporting tasks such as medical analysis  [61]  and robotics to support social distancing  [62] . Further work along these lines could be of value for improving the resilience pf human civilization to COVID-19 and future pandemics. Whereas global warming is a traditional environmental sustainability topic, pandemics are not, though pandemics can derive from environmental activities, in particular those that put humans in contact with novel zoonotic pathogens. Nonetheless, both issues threaten the ability of global human civilization to be sustained into the long-term future. Second, future forms of AI could be particularly consequential. The field of AI has long entertained notions of extreme future AI that could be "the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control"  [63, p.33] . Recently, there has been debate on the extent to which people in AI should focus on near-term or long-term AI. To some extent, this debate may be unnecessary, due to the existence of activities that are good to do for both near-term and long-term AI  [24, [26] [27] . Nonetheless, to the extent that each type of AI merits some distinct attention, a case can be made for attention to long-term AI due to its potential importance for long-term sustainability. Long-term AI can play three roles of relevance to long-term sustainability. First, it could bolster efforts to address threats such as global warming and pandemics. Second, it could pose a threat of its own, especially for runaway AI scenarios in which the AI effectively takes over the world. Third, it could play an instrumental role in space expansion. A significant dilemma exists for the dual status of long-term AI as both threat and tool for addressing other threats. Ideally, long-term AI would be designed slowly and carefully to ensure a high standard for safety and ethics. However, delaying the deployment of long-term AI reduces the potential for its use to address other threats. One implication of this is that other work to address other threats can be of value for "buying time" to safely and ethically develop long-term AI  [64] . This includes work using near-term AI to address these other threats. If there is extended time available to slowly and carefully design long-term AI, then that also buys time to reflect on what should be done with respect to space expansion and related opportunities  [65] . On the other hand, there is no guarantee that an extended time will be available. Indeed, AI research and development is proceeding at brisk pace, prompting concerns about a race to develop long-term AI  [66] . One proposed means of buying some time is to deploy a moderately powerful AI "nanny" who can protect and support humanity while it reflects on what to do next  [67] . That possibility is not without its own risks, such as the risk of a poorly designed nanny AI that steers the world in a bad or even catastrophic direction. These are all among the AI issues that can be of profound importance for long-term sustainability.
 Conclusion In this paper, we have surveyed the ethics of sustainability, analyzed the ethical basis of existing work on AI and sustainability, and presented an argument for a non-anthropocentric, long-term conception of sustainability and an accompanying argument for favoring optimization over sustainability. Taken together, the paper provides some guidance on how ongoing work on AI and sustainability can and should proceed. First, work on AI and sustainability should precisely specify its ethical basis, in particular on what it seeks to sustain, for how long, and how much effort should be made on sustainability. Second, work on AI and sustainability should consider adopting the non-anthropocentric, long-term conception of sustainability and optimization that this paper argues for. In practice, that entails a focus on applying AI to addressing major global threats such as global warming and pandemics, to ensuring the long-term sustainability of the resource base needed for civilization, and for pursuing opportunities to expand human civilization into outer space. In closing, we wish to emphasize the ethical principle of equality across time. A fundamental aspect of sustainability is its future-orientation. The principle of equality across time means that all future times should be treated equally, or rather that there should be no bias against something just because of when it exists. This is a compelling moral principle. If it is to be taken seriously, it demands an attention to the big-picture distant future of the universe and to the ways in which near-term actions can affect it. Actions involving AI are among the most significant ways to affect the distant future. The field of AI has special opportunities to make an astronomically large positive difference-to make the universe a better place. It should make pursuit of these opportunities a major priority. strongly indicate that the natural environment is valued instrumentally to advance human wellbeing. Nothing in the statement clarifies the time scales of sustainability or degree of effort. The UNI Global Top 10 Principles for Ethical AI includes the principle, "Make AI serve people and planet: This includes codes of ethics for the development, application and use of AI so that throughout their entire operational process, AI systems remain compatible and increase the principles of human dignity, integrity, freedom, privacy and cultural and gender diversity, as well as with fundamental human rights. In addition, AI systems must protect and even improve our planet's ecosystems and biodiversity." This principle is classified as intrinsically valuing humans, ecosystems and biodiversity as the final line calls for not only protection but improvement of nonhuman entities, and the phrasing "make AI serve people and planet" strongly suggest moral consideration for also nonhumans.
 Appendix B: Examples of AI Sustainability Publications Section 3.2 presents data on publications on the nexus of AI and sustainability. This appendix presents some illustrative examples of these data. Theodorou et al.  [68]  developed a single-player game designed to simulate a sustainable world based on accurate ecological models and behavior economics principles. In the game, individual people must aim to act toward a sustainable society. The paper describes the game in terms implying that natural resources play an instrumental role to advance individual people's survival and wellbeing. Other indications of what is meant by a sustainable world is discussed in social and economic terms. This publication was, therefore, classified as implying intrinsic value of humans and instrumental value of nonhumans. Yigitcanlar & Cugurullo  [69]  define smart and sustainable cities as "an urban locality functioning as a robust system of systems with sustainable practices, supported by community, technology, and policy, to generate desired outcomes and futures for all humans and nonhumans". This publication is classified as intrinsically valuing humans and nonhumans as it explicitly refers to their benefits. van Wysnberghe  [7]  calls for a "third wave" in AI ethics and states that "This third wave must place sustainable development at its core" (emphasis original). Sustainable development is in turn defined as in the Brundtland report. This suggests an anthropocentric conception of sustainability, in which only humans are intrinsically valuable. It further suggests placing a high degree of effort on sustainability, though the emphasis on sustainable development leaves open the question of the relative importance of present vs. future generations. Gomes et al.  [70]  argue that "computational sustainability harnesses computing and artificial intelligence for human well-being and the protection of our planet" and that "planning for sustainable development encompasses complex interdisciplinary decisions spanning a range of questions concerning human well-being, infrastructure (…) and the environmental protection of the Earth and its species". All of this is strongly suggestive of intrinsic value of nonhumans as well as humans, so this publication is classified as such. Zhang et al.  [71]  studies the contributions of big data analytics capability and artificial intelligence capability to the sustainability of organizational development. The publication does not define sustainability and apply the following uses of "sustainability during the abstract and introduction only: "sustainable innovation and performance", "sustainability development projects", "sustainability design and commercialization processes", "sustainable growth and performance", "sustainable organizational growth", "sustainable competitive advantages", "sustainable investment", "sustainable development goals", "sustainable positional advantages", and big data as "sustainable resources". This publication is therefore classified as ambiguous. Larsson et al.  [72]  by the AI Sustainability Center defines sustainable AI as follows: "The AI Sustainability Center supports an approach in which the positive and negative impacts of AI on people and society are as important as the commercial benefits or efficiency gains. We call it Sustainable AI." This publication is classified as intrinsically valuing humans as it discusses and defines sustainability as pertaining to human and social aspects only, including sustainable AI which is defined by accountability, bias, malicious use, and transparency. The publication is further noteworthy for presenting an extensive literature review of AI ethics as a review of literature on sustainable AI, thus equating AI ethics or ethical AI with sustainable AI, which arguably drains "sustainability" of meaning. 5 2 . 1 21 https://aiforpeople.org/conference/ What should be able to be sustained, and why?
			 https://earthcharter.org/