Abstract: Newcomb's problem has spawned a debate about which variant of expected utility maximization (if any) should guide rational choice. In this paper, we provide a new argument against what is probably the most popular variant: causal decision theory (CDT). In particular, we provide two scenarios in which CDT voluntarily loses money. In the first, an agent faces a single choice and following CDT's recommendation yields a loss of money in expectation. The second scenario extends the first to a diachronic Dutch book against CDT.

Extracting Money from Causal Decision Theorists

Introduction In Newcomb's problem  [Nozick, 1969; Ahmed, 2014] , a "being" offers two boxes, A and B. Box A is transparent and contains $1,000. Box B is opaque and may contain either $1,000,000 or nothing. An agent is asked to choose between receiving the contents of both boxes, or of box B only. However, the being has put $1,000,000 in box B if and only if the being predicted that the agent would choose box B only. The being's predictions are uncannily accurate. What should the agent do? Causal decision theory (CDT) recommends that the agent reason as follows: I cannot causally affect the content of the boxes -whatever is in the boxes is already there. Thus, if I choose both boxes, regardless of what is in box B, I will end up with $1,000 more than if I choose one box. Hence, I should choose both boxes. Evidential decision theory (EDT), on the other hand, recommends that the agent reason as follows: if I choose one box, then in all likelihood the being predicted that I would choose one box, so I can expect to walk away with $1,000,000. (Even if the being is wrong some small percentage of the time, the expected value will remain at least close to $1,000,000.) If I choose both, then I can expect to walk away with (close to) $1,000. Hence, I should choose one box. While Newcomb's problem itself is far-fetched, it has been argued that the difference between CDT and EDT matters in various game-theoretic settings. First, it has been pointed out * Contact Author that Newcomb's problem closely resembles playing a Prisoner's Dilemma against a similar opponent  [Brams, 1975; Lewis, 1979] . Whereas CDT recommends defecting even in a Prisoner's Dilemma against an exact copy, EDT recommends cooperating when facing sufficiently similar opponents. What degree of similarity is required and whether EDT and CDT ever come apart in this way in practice has been the subject of much discussion  [Ahmed, 2014, Section 4.6] . A common view is that CDT and EDT come apart only under fairly specific circumstances that do not include most human interactions.  Still, Hofstadter [1983] , for example, argues that "superrational" humans should cooperate with each other in a real-world one-shot Prisoner's Dilemma, reasoning in a way that resembles the reasoning done under EDT. Economists have usually been somewhat dismissive of such ideas, sometimes referring to them as "(quasi-)magical thinking" when trying to explain observed human behavior  [Shafir and Tversky, 1992; Masel, 2007; Daley and Sadowski, 2017] . Indeed, standard game-theoretic solution concepts are closely related to ratificationism  [Joyce and Gibbard, 1998, Section 5 ], a variant of CDT which we will revisit later in this paper (see Part 3 of Section 4). Second, even if the players of a game are dissimilar, it is in the nature of strategic interactions that each player's behavior is predicted by the other players. Newcomb's problem itself (as well as the ADVERSARIAL OFFER discussed in this paper) differs from strategic interactions as they are usually considered in game theory in its asymmetry. However, one might still expect that CDT and EDT offer different perspectives on how rational agents should deal with the mutual prediction inherent in strategic interactions  [Gauthier, 1989, Section XI] . Third, CDT and EDT differ in their treatment of situations with imperfect recall. Seminal discussions of such games are due to  Piccione and Rubinstein [1997 ] and Aumann et al. [1997 ] [cf. Bostrom, 2010 , for an overview from a philosopher's perspective]. While these problems originally were not associated with Newcomb's problem, the relevance of different decision theories in this context has been pointed out by  Briggs [2010 ] [cf. Armstrong, 2011 Schwarz, 2015; Conitzer, 2015] . The importance of the differences in decision theory is amplified if, instead of humans, we consider artificial agents. After all, it is common that multiple copies of a software sys-tem are deployed and other parties are often able to obtain the source code of a system to analyze or predict its behavior. As some software systems choose more autonomously, we might expect their behavior will be (approximately) describable by CDT or EDT (or yet some other theory). If either of these theories has serious flaws, we might worry that if a system implements the wrong theory, it will make unexpected, suboptimal choices in some scenarios. Such scenarios might arise naturally, e.g., as many copies of a system are deployed. We might also worry about adversarial problems like the one in this paper. One argument against CDT is that causal decision theorists (tend to) walk away with less money than evidential decision theorists, but this argument has not proved decisive in the debate. For instance, one influential response has been that CDT makes the best out of the situation -fixing whether the money is in box B -which EDT does not  [Joyce, 1999, Section 5.1] . It would be more convincing if there were Newcomb-like scenarios in which a causal decision theorist volunteers to lose money (in expectation or with certainty). 1 Constructing such a scenario from Newcomb's problem is non-trivial. For example, in Newcomb's problem, a causal decision theorist may realize that box B will be empty. Hence, he would be unwilling to pay more than $1,000 for the opportunity to play the game. In this paper, we provide Newcomb-like decision problems in which the causal decision theorist voluntarily loses money to another agent. We first give a single-decision scenario in which this is true only in expectation (Section 2). We then extend the scenario to create a diachronic Dutch book against CDT -a two-step scenario in which the causal decision theorist is sure to lose money (Section 3). Finally, we discuss the implications of the existence of such scenarios (Section 4). 1 Walking away with the maximum possible (expected) payoff under any circumstances is not a realistic desideratum for a decision theory: any decision theory X has a lower expected payoff than some other decision theory Y in a decision problem that rewards agents simply for using decision theory Y [cf.  Skalse, 2018 , for a harderto-defuse version of this point]. However, such a setup does not allow one to devise a generic scenario in which an agent voluntarily loses money, i.e. loses money in spite of having the option to walk away losing nothing. Furthermore, scenarios with voluntary loss appear significantly more problematic for pragmatic reasons. Regardless of what you think is the right option in Newcomb's problem, you might not view Newcomb's problem as relevant ground for decision-theoretical argument because it is so unlikely that one would ever face Newcomb's problem in the real world. For instance, even if you thought that one-boxing is rational (and two-boxing is not), you might stick with CDT nonetheless because your real-world expected opportunity costs from two-boxing in Newcomb's problem are negligible. [For some discussion of this deflationary argument, see, e.g.,  Gauthier, 1989, Section XI; Ahmed, 2014, Section 7.1.iv; Oesterheld, 2019 , Section 1, and references therein.] However, if there is a Newcomb-like problem in which the causal decision theorist voluntarily loses money to some other agent, this generates a significant incentive to place him in such a situation.
 Extracting a profit in expectation from causal decision theorists Consider the following scenario: ADVERSARIAL OFFER: Two boxes, B 1 and B 2 , are on offer. A (risk-neutral) buyer may purchase one or none of the boxes but not both. Each of the two boxes costs $1. Yesterday, the seller put $3 in each box that she predicted the buyer not to acquire. Both the seller and the buyer believe the seller's prediction to be accurate with probability 0.75. No randomization device is available to the buyer (or at least no randomization device that is not predictable to the seller). 2 If the buyer takes either box B i , then the expected money gained by the seller is $1 − P ($3 in B i | buyer chooses B i ) • $3 = $1 − 0.25 • $3 = $0.25. Hence, the buyer suffers an expected loss of $0.25 (if he buys a box). The best action for the buyer therefore appears to be to not purchase either box. Indeed, this is the course of action prescribed by EDT as well as other decision theories that recommend one-boxing in Newcomb's problem [e.g., those proposed by  Spohn, 2012; Poellinger, 2013; Soares and Levinstein, 2017] . In contrast, CDT prescribes that the buyer buy one of the two boxes. Because the agent cannot causally affect yesterday's prediction, CDT prescribes to calculate the expected utility of buying box B i as P ($3 in box B i ) • $3 − $1, (1) where P ($3 in box B i ) is the buyer's subjective probability that the seller has put money in box B i , prior to updating this belief based on his own decision. For i = 1, 2, let p i be the probability that the buyer assigns to the seller having predicted him to buy B i . Similarly, let p 0 be the probability the buyer assigns to the seller having predicted him to buy nothing. These beliefs should satisfy p 0 + p 1 + p 2 = 1. Because p 0 ≥ 0, we have that (p 0 +p 1 )+(p 0 +p 2 ) = 2p 0 +p 1 +p 2 ≥ 1. Hence, it must be the case that p 0 + p 1 ≥ 1 2 or p 0 + p 2 ≥ 1 2 (or both). Because P ($3 in box B i ) = p 0 + p 3−i for i = 1, 2, it is P ($3 in box B i ) ≥ 1 2 for at least one i ∈ {1, 2}. Thus, the expected utility in eq. 1 of at least one of the two possible purchases is at least 1 2 • $3 − $1 = $0.50, which is positive. Any seller capable of predicting the causal decision theorist sufficiently well will thus have an incentive to use this scheme to exploit CDT agents. (It does not matter whether the seller subscribes to CDT or EDT.) It should be noted that even if the buyer uses CDT, his view of the deal matches the seller's as soon as the dollar is paid. That is, after observing his action, he will realize that the box he bought is empty with probability 0.75 and thus worth less than a dollar. CDT knows that it will regret its choice [see  Joyce, 2012; Weirich, 1985  for discussions of the phenomenon of anticipated regret a.k.a. decision instability in CDT].
 A diachronic Dutch book against causal decision theory ADVERSARIAL OFFER results in a loss in expectation for the causal decision theorist. It is natural to ask whether it is possible to set up the scenario so that the causal decision theorist ends up with a sure loss; effectively, a Dutch book. Arguably, Dutch books are more convincing than scenarios with expected losses since the very meaning of "expectations" is the subject of the debate about EDT and CDT. Of course, if the seller could perfectly predict the buyer in ADVERSARIAL OFFER (instead of being right only 75% of the time), then ADVERSARIAL OFFER would become a Dutch book. But can we construct a Dutch book without perfect prediction? We have already observed that in ADVERSARIAL OFFER the causal decision theorist always regrets his decision after observing its execution. This suggests the following simple approach to constructing a Dutch book. After the box is sold, the seller allows the buyer to reverse his decision for a small fee (ending up without any box and having lost only the fee). However, a CDT buyer may then anticipate eventually undoing his choice and therefore not buy a box in the first place  [Ahmed, 2014, Section 3.2; though cf. Skyrms, 1993; Rabinowicz, 2000 ].  3  To get our Dutch book to work, we add another choice before ADVERSARIAL OFFER. ADVERSARIAL OFFER WITH OPT-OUT: It is Monday. The buyer is scheduled to face the AD-VERSARIAL OFFER on Tuesday. He also knows that the seller's prediction was already made on Sunday. As a courtesy to her customer, the seller approaches the buyer on Monday. She offers to not offer the boxes on Tuesday if the buyer pays her $0.20. Note that the seller does not attempt to predict whether the buyer will pay to opt out. Also, we assume that the buyer cannot, on Monday, commit himself to a course of action to follow on Tuesday. It seems that a rational agent should never feel compelled to accept the Monday offer. After all, doing so loses him money with certainty, whereas simply refusing both offers (on Monday and on Tuesday) guarantees that he loses no money. CDT, however, recommends opting out on Monday, for the following reasons. A CDT buyer knows on Monday that if he does not opt out, he will buy a box on Tuesday (though he may not yet know which one). Further, he believes that whatever box he will take on Tuesday will contain $3 with only 25% probability, thus implying an overall expected payoff of 0.25 • $3 − $1 = −$0.25. This is because, on Monday, CDT treats the decision on Tuesday in the same way as it treats any other random variable in the environment. So the causal expected utility of not opting out is just what an outside observer would expect the payoff of a CDT agent facing ADVERSARIAL OFFER to be. Because this expected payoff of −$0.25 is less than the certain payoff of −$0.20 that can be obtained by opting out, CDT recommends opting out. In fact, for the argument in the previous paragraph to succeed, it is only necessary that CDT is used on Tuesday; other decision theories would also recommend accepting the Monday offer, if they anticipate that the agent will use CDT on Tuesday. For instance, if the agent followed EDT on Monday and CDT on Tuesday (and is aware on Monday that he will use CDT on Tuesday), then he would still accept the Monday offer. Similarly, if the seller believes that the buyer will pick one of the boxes on Tuesday, then she will hope that he rejects the Monday offer. Thus, it seems that what creates the opportunity for a Dutch book is the prospect of buying a box on Tuesday (as CDT recommends), not the use of CDT on Monday.
 Discussion We differentiate four types of responses to these scenarios available to supporters of causal decision theory: 1. They could claim that these scenarios are irrelevant for evaluating decision theories, in the sense that they are impossible to set up or otherwise out of scope, and therefore unpersuasive. 2. They could concede that these scenarios are relevant for evaluating decision theories, but claim that CDT's recommendations in them are acceptable. 3. They could concede that our analysis obliges them to give up on certain specific formulations of CDT, but try to modify CDT to get these scenarios right while maintaining some of its essence, in particular two-boxing and the causal dominance principle. 4. They could concede that these scenarios show that the very core of CDT (two-boxing and the causal dominance principle) is implausible. We will discuss these options in turn. 1 Surely, if one could show that a CDT agent will or can never face these scenarios -despite the seller having an obvious incentive to set them up -that would be the most convincing defense of CDT. In particular, a causal decision theorist might claim that sufficiently accurate prediction of a CDT agent is simply impossible. 4 However, not much accuracy is required, for the following reasons. The CDT agent will take one of the two boxes. Even if the seller picks the box to fill with money uniformly at random, she would therefore be right half of the time. If she can do any better than that, predicting correctly with probability 1/2+ , then she can extract money from the CDT agent by putting (instead of $3) some amount between $2/(1 − 2 ) and $2 in the box predicted not to be taken. Thus, the CDT agent needs to be completely unpredictable in order to avoid being taken advantage of in these examples. Most human beings are, generally speaking, at least somewhat predictable in their actions even when such predictability can be used against them. For example, in rock-paperscissors -which structurally resembles the ADVERSARIAL OFFER -most people follow exploitable patterns in what moves they select [see, e.g.,  Farber, 2015, and references therein] .  5  Consider such a somewhat predictable person who aims to be a causal decision theorist. It seems that he would indeed be vulnerable to the examples discussed earlier. The only defense for the supporter of causal decision theory would seem to then be that if so, the person in question is not truly acting in the way that CDT describes. That is, acting according to CDT also requires being unpredictable to the seller, either by succeeding at out-thinking the seller sufficiently often, or by acting sufficiently randomly. Is it reasonable to consider this a requirement of acting according to CDT? CDT does not suggest any strict preference for choosing randomly across options, as opposed to just deterministically choosing one of the options that is best according to the buyer's beliefs. Hence, the unpredictability would have to emerge from the buyer attempting to out-think the seller. But it does not seem that this is always an attainable goal. For example, imagine that the buyer is a deterministic computer program whose source code is known to the seller. Then regardless of how exactly the agent works, the seller can predict the buyer's behavior perfectly [cf. Soares and Fallenstein, 2014, Section 2; Cavalcanti, 2010, Section 5]. We would thus be forced to conclude that such a program cannot possibly follow CDT, which to us is an unsatisfactory conclusion. Plausibly any other physically realized agent that chooses deterministically can at least in principle (if not with current technology) be predicted by creating or emulating an atom-by-atom copy of that agent [cf.  Yudkowsky, 2010, pp. 85ff.] . Even if the supporter of CDT acknowledges that these scenarios are possible, he might nevertheless argue that they are irrelevant, in the sense that the decision theory is not intended to be used for such scenarios and hence nothing that one could show about its performance in such a scenario is of significance for evaluating the theory. "It is as if one evaluated a car by testing how it performs underwater." There is little we can say about this response. Still, we expect it to be unattractive to most decision theorists. After all, our scenarios (in particular the ADVERSARIAL OFFER) resemble Newcomb's problem -the problem that has led to the development of CDT in the first place. Further, if our scenarios were out of CDT's scope, then we (and presumably most other decision theorists) would still be interested in identifying a decision theory that does make good recommendations for predictable agents (such as artificial intelligent agents whose behavior is 5 There are multiple rock-paper-scissors bots available online which attempt to predict their opponent's future moves based on past moves (using data from other players). As of July 2019, the bot at http://www.essentially.net/rsp/ has reportedly played about 2 million rounds and won 57% more often than it lost. determined by a computer program) facing a wide range of scenarios including the ones given in this paper. 2 If our scenarios are within the scope of causal decision theory, then the supporter of causal decision theory has to contend with the fact that one can extract expected money from, and even Dutch-book, CDT agents in them. But he might question the significance of Dutch-book arguments and other money extraction schemes, either in general or in this particular context. For some general discussion of whether (diachronic) Dutch books are conclusive decision-theoretic arguments, see, e.g.,  Vineberg [2016]  or  Hájek [2009] . Note, though, that some of the most influential arguments in favor of expected utility maximization (EUM) -of which CDT is a refinement -are Dutch books. Of course, one may use different arguments to justify EUM. But it would seem odd to follow Dutch-book arguments to EUM but no further. Instead of rehashing some of the more generic reasons for and against the persuasiveness of Dutch books and loss of money in expectation, we here discuss a response that is specific to CDT and ADVERSARIAL OFFER WITH OPT-OUT.  6  A causal decision theorist may argue that it is not generally fair to expect any kind of coherence from CDT's recommendations when multiple decisions are to be made across time, due to the different perspectives that the decision maker adopts (and, arguably, has to adopt) at different points in time. Consider Newcomb's problem. Let t 0 be the time at which the predictor observes the agent (perhaps using fMRI or the like) in order to make a prediction. Then, before t 0 , CDT recommends committing -and if needed paying money to commit -to one-boxing [cf.  Barnes, 1997; Joyce, 1999, pp. 153f.; Meacham, 2010] . After t 0 , CDT recommends two-boxing. However, most decision theorists do not consider this to be a compelling argument against CDT. The causal decision theorist can easily justify the difference in the decision made by the fact that, before t 0 , the commitment decision has a causal effect on what is in the boxes, and after t 0 , it does not. It would be hypocritical for an evidential decision theorist to disagree, since EDT is dynamically inconsistent in analogous ways. For instance, consider a version of Newcomb's problem in which both boxes are transparent  [Gibbard and Harper, 1981, Section 10 ; also discussed by  Gauthier, 1989; Drescher, 2006, Section 6.2; Arntzenius, 2008, Section 7; Meacham, 2010, Section 3.2.2] . Let t 0 be the time at which the EDT agent sees the content of both boxes. Then before t 0 , EDT recommends committing -and if needed paying money to commit -to one-boxing. After t 0 , EDT recommends twoboxing.  7  The evidential decision theorist can easily justify this along similar lines: before t 0 , her commitment is evidence about what is in the boxes, and after t 0 it no longer is. Thus, at least some types of dynamic inconsistency do not constitute strong arguments against a decision theory. However, in our opinion, the dynamic inconsistency displayed by CDT in the ADVERSARIAL OFFER WITH OPT-OUT is much more problematic. For one, it leads to a Dutch book. Often, the main argument that is given for why a particular inconsistency is problematic is precisely that it allows for a Dutch book. Conversely, defenses of dynamic inconsistencies  [Ahmed, 2014, Section 3.2 , for an example in a Newcomb-like scenario] often focus on arguing that they do not allow for Dutch Books. Further, it seems that some of the reasons for (or defenses of) dynamic inconsistency in the above decision problems do not apply to CDT's dynamic inconsistency in ADVERSARIAL OFFER WITH OPT-OUT. For CDT in Newcomb's problem, there is a particular event at time t 0 that splits the decision perspectives: the loss of causal control at t 0 over the content of box B. Similarly, for EDT in the Newcomb's problem with transparent boxes, that event is the loss of evidential control [cf. Almond, 2010, Section 4.5] at t 0 over the content of box B. It is thus easy to argue for defenders of the respective theories that the perspectives from before and after t 0 or t 0 should diverge  [Ahmed and Price, 2012, pp. 22-23, Section 4] . In sharp contrast, the ADVERSARIAL OFFER WITH OPT-OUT lacks any such event between the decision points. The difference in perspectives for CDT appears to be purely a result of CDT viewing its current choice differently than it views past and future decisions. All that being said, we agree that caution should be taken when evaluating a decision theory based on scenarios with multiple decisions across time. In general, more research on what conclusions can be drawn from such scenarios is needed  [Steele and Stefánsson, 2016, Section 6 ]. Nevertheless, we do not see any clear path by which such research would justify CDT's recommendations in the ADVERSARIAL OFFER WITH OPT-OUT. In any case, even if one is at this point unwilling to consider scenarios with multiple decision points at all for the purpose of evaluating decision theories, one would still have to contend with the simpler ADVERSARIAL OFFER scenario, in which there is only one decision point. 3 If a straightforward interpretation of CDT cannot be defended against our scenarios, one may look to modify it to avoid expected or sure loss while preserving some of CDT's core tenets. In particular, in response to other alleged counterexamples, some authors have tried to modify CDT while maintaining the causal dominance  [Joyce, 1999, Section 5 .1] a.k.a. sure thing  [Gibbard and Harper, 1981, Section 7]  principle [though see  Ahmed, 2012 , for an argument against the motivation behind some of these approaches]. For example, one may turn to the concept of ratifiability. In Newcomb-like scenarios such as those under discussion here, for any choice a, we can consider the beliefs about what is in the boxes that would result from knowing that one will choose a. Then, a choice a is ratifiable if it is an optimal choice -as judged by CDT -under those beliefs. For example, in Newcomb's problem only two-boxing is ratifiable, precisely because it is causally dominant. For an overview of ratification and its relation to CDT, see  Weirich [2016, Section 3.6 ]. Unfortu-nately, this concept is of no help in the ADVERSARIAL OF-FER, because none of the three options (buying B 1 , buying B 2 or declining) is ratifiable. For instance, under the beliefs that would result from knowing that you will take box B i , it would be better to buy the other box B 3−i . The ratificationist may respond by claiming that unpredictable randomization should always be possible. If that were true, then the only ratifiable option would be to take each box with probability 50%, thus gaining money in expectation. But again, we would like to have a decision theory that works in a broad variety of scenarios, including ones where the agent expects to be somewhat predictable. Furthermore, even if a true random number generator (TRNG) (e.g., one based on nuclear decay) is in fact available, this does not settle the issue. For example, consider a variant of the AD-VERSARIAL OFFER in which the seller refrains from putting money in any box if she predicts the buyer to make different choices depending on the output of the TRNG. In this ANTI-RANDOMIZATION ADVERSARIAL OFFER, again no option is ratifiable: under the beliefs that would result from knowing that you will make different choices depending on the TRNG's output (and therefore choose a box with some positive probability), you would rather not pick any box. To circumvent this example, the ratificationist could argue that the decision maker should be able to randomize in such a way that whether he is randomizing is unpredictable. However, at this point, one might just as well assert the impossibility (or irrelevance) of Newcomb-type scenarios altogether, which we have addressed in 1. A different strategy for modifying CDT to avoid the Dutch book in the ADVERSARIAL OFFER WITH OPT-OUT is the following. The Dutch book arises from a disagreement between CDT on Monday and CDT on Tuesday (cf. the discussion under 2). A tempting possibility is to modify CDT so that it considers all decisions to be made at once. That is, such a version of CDT -let us refer to it as policy-CDT -prescribes that one decide on one's general policy all at once.  8  In the ADVERSARIAL OFFER WITH OPT-OUT, there are four possible policies: opt out, buy B 1 , buy B 2 , and buy nothing (where the last three possibilities include declining the opt-out offer). When considering these policies, buy nothing dominates opt out. Hence, policy-CDT will decline the opt-out offer and thereby avoid the Dutch book. (Note, however, that such a modification of CDT will make no difference to the choices it prescribes in ADVERSARIAL OFFER, which has only one decision point. Hence, it will still lose money in expectation.) While this appears to be a promising approach, it is nontrivial to flesh out, because on other examples it is less clear what policy-CDT should prescribe. For illustration, consider the following interpretation of policy-CDT: follow the policy to which CDT would like to commit ex ante, where "ex ante" refers to some point in time before the first decision of the scenario. Now, let us consider a version of Newcomb's problem which is supplemented by another trivial and unrelated decision -say, whether to eat a peppermint -that takes place when the agent still has a causal influence over the prediction. Then the ex-ante-commitment interpretation of policy-CDT would recommend one-boxing. To the causal decision theorist, this may be unacceptable, especially given that adding the peppermint decision is such a minor modification of Newcomb's problem. Perhaps there is a way to define policy-CDT that avoids such dependence on irrelevant decisions while also prescribing two-boxing, but it is not immediately obvious how to do so. Many other ways of modifying CDT are worth considering. For instance, in the ADVERSARIAL OFFER, it may be unrealistic for the buyer to form a single probability distribution over box contents. Instead, he may consider multiple different probability distributions, including one under which box B 1 is probably empty and one under which box B 2 is probably empty. He could then evaluate each option pessimistically, i.e., w.r.t. the probability distribution that is worst under that option. Such a version of CDT would prescribe declining to buy a box. At the same time, it would recommend twoboxing in Newcomb's problem and more generally obey the causal dominance principle. For a discussion of this maxmin criterion for choice under multiple probability distributions, see, e.g.,  Gilboa and Schmeidler [1989]  and in particular game-theoretic interpretations such as that of  Grünwald and Halpern [2011] . A more general discussion of how using sets of probability distributions (while potentially decision rules other than the maxmin criterion) is offered by  Bradley [2012] . In our setting, B 1 are B 2 are, roughly, complementary bets in the causalist's beliefs. In all worlds in which B i is empty, B 3−i is full. As discussed by Bradley, it has been argued that a rational agent should accept one of a pair of complementary bets. Indeed, expected utility maximization for a single probability distribution satisfies this complementarity criterionto the causalist's detriment in the Adversarial Offer.  Bradley [2012]  argues that in general, an agent with imprecise probabilities should not satisfy the complementarity criterion and that this allows him to avoid Dutch books -though, of course, he considers Dutch books of a very different type. 4 Finally, one may view at least one of the scenarios in this paper as supporting a persuasive argument against the very core of CDT. EDT is the obvious alternative. However, depending on how problematic we find EDT's prescriptions in other cases -such as the Smoking lesion  [Ahmed, 2014, Section 4.1-4.3]  or cases of dynamic inconsistency like Newomb's problem with transparent boxes (and the problems listed in footnote 7) -we may also look to various other decision theories that have been proposed  [Gauthier, 1989; Spohn, 2012; Poellinger, 2013; Soares and Levinstein, 2017] . 			 This decision problem resembles the widely discussed Death in Damascus scenario [introduced to the decision theory literature byGibbard and Harper, 1981, Section 11]  and even more closely the Frustrater case proposed by Spencer and Wells[2017], though these are not set up to result in an expected financial loss.Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
			 This, of course, requires that the reversal offer does not come as a surprise. Throughout, we insist that the buyer knows all the rules of the game.
			 For a general discussion of such unpredictability claims in defense of CDT, seeAhmed [2014, Chapter 8].Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
			 For a discussion of similar arguments about other diachronic Dutch books, see, e.g.,Rabinowicz [2008].7 Parfit's (1984)  hitchhiker [Barnes, 1997] , XOR Blackmail [Soares and Levinstein, 2017, Section 2] and Yankees vs. Red Sox[Arntzenius, 2008, pp. 22-23; Ahmed and Price, 2012]  similarly expose dynamic inconsistencies in EDT. Conitzer [2015]  gives a somewhat different type of scenario -based on the Sleeping Beauty problem -in which EDT is dynamically inconsistent.Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
			  Policy-CDT resembles Fisher's [nd]  disposition-based decision theory. Compare Meacham [2010]  for a discussion of explicit precommitment. Similarly, Gauthier [1989]  has argued for evaluating "plans" not decisions in Newcomb-like problems (without basing this argument on any particular theory like CDT or EDT). A few authors have also proposed policy versions of other, more EDT-like decision theories[Drescher, 2006, Section 6.2; Yudkowsky and Soares,  2018, Section 4].Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
			 Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).


Abstract: This dataset collects a list of discoveries essential to the development of the current generation of artificial intelligence. We trace this development back as early as we have records of relevant intellectual progress-in this case, to Ancient Greek philosophy-looking for the foundations of complex new approaches. Our goal in this project is to present an alternative to an achievement-based timeline of AI development. The development of new capabilities in machine intelligence frequently produces exciting tangible results, which dominate public conceptions of progress and are well-remembered in histories of the field, but such results are not always the product of novel technical insight. Some new abilities rely more on recombination of existing tools or on additional computational capacity-sufficient to solve complex problems efficiently, but not necessarily the most direct precursors of new tools for further recombination, or of new directions for future work without additional increases in computational efficiency. Our aim is to augment the current discourse of historically grounded forecasting by adding a specific discursive thread that tracks theoretical discoveries alone, complementing the more capability-focused accounts. We've collected metadata about each discovery (and its discoverers) and looked for trends in their production. Our own findings are described below, but we encourage extension and additional evaluation of the data, and will publish our own ongoing work here.

AI Insights Dataset Analysis

Methods & Assumptions Our metric for what constitutes an insight is an estimate of how novel or surprising the results of that insight are, using a comparison of LSTM and RNNs as a baseline-that is, given that RNNs exist, how surprising is the development of LSTM? All discoveries that seem equally or more surprising were included in our set. This is one of many possible thresholds for significance we could have chosen, and the rationale behind ours is subjective: Our focus on theoretical progress means that the data do not include progress captured in implementing an existing theory, or applying it more directly. For example, punched-card programming of early computers had clear impacts on the progress of computing, but we attribute the insight behind that technology to the 18th-century textile worker who developed a punchedcard method for programming looms. From this perspective, the application of punched cards to electrical computers is a combination of existing theories, not a new development in itself. We consulted two existing chronicles of the history of AI in an effort to flesh out parts of our data set that may have fallen in our blind spots: -Nilsson, N. J.  (2009) . The quest for artificial intelligence. Cambridge University  Press. -Crevier, D. (1993) . AI: the tumultuous history of the search for artificial intelligence. Basic Books. A few additional notes on the data: • First to discover, not first to publish: We credit an insight to the first person known to have described it, as best we can determine through reputable sources, not the first person to publish that description. This practice is motivated in part by our understanding that not all discoverers published work in the academic tradition (e.g. William Newcomb, credited with the conception of Newcomb's problem), and in part by some discoveries' having been kept confidential, pre-publication, in corporate and state departments. • Specific vs. general discoveries: Sometimes an early discovery constitutes a special case of a later, more general discovery; in this case, we credit the more general discovery. If the special case itself seems to constitute an influential insight, we'll credit it as a separate line item; but if it was largely ignored, or was unavailable to discoverers of the more general version, we omit it from this list. For example, ideas from evolutionary biology have been rediscovered independently by AI researchers through simulationist and purely rationalistic investigation that were previously discovered empirically through biological field research before being formalized in such a manner that they could be used by AI researchers. We count the usable formalization, rather than the case studies that could be used as raw material for specifying the usable formalization. • Hardware agnosticism: Our focus here is on information theoretical insights, not on insights that aid in the development of new hardware, the rationale being that artificial intelligence is a substrate independent pursuit, provided that the Church-Turing thesis holds. Futher, charting the history of today's electronic computers alone would constitute a project of roughly equal magnitude, and one that aligns less closely with our area of expertise.
 Data & Parameters The full dataset (as json) and its schema are available at mediangroup.org/research; this section provides a brief overview of the data for readers of this report. The set includes 193 total insights. For each insight, we record the following information (plus sources for each fact): • Name • Year of discovery • First work in which the insight was referenced or published • Discoverer(s) • Institutional affiliation of discoverer(s) at time of discovery (if applicable) • Institution(s) sector and type (see below) • Nation of origin of discoverer(s)
 • Additional notes Institutions are assigned rough sectors and types from the following lists: Sectors: • Academic • Military • Industrial • State • Other R&D • Religious • N/A Types: • Public university • Private university • For-profit entity • Non-profit entity • N/A
 Analysis For the purposes of this first analysis section, we will discuss the data as we have been able to make it available to us. Reflections on the limitations of the data and the extent to which they are relevant to readership follow in the Methodological Reflections section.
 Temporal distribution At a glance, the distribution of insights over time shows an expected uptick in progress contemporaneous with the European Enlightenment, followed by a sharp increase in the pace of discovery at the beginning of the 20th century. A view of only the most recent centuries provides more detail on the arc of recent progress, showing a significant increase in production with introduction of digital computers available to major institutions in the late 1940s. A breakdown of insight discoveries by decade clarifies the changing rates of discovery.
 Historical context The uneven distribution of insights correlates clearly with the trajectory of Western intellectual and economic history. The theological and philosophical conservatism of the middle ages matches the flat section of the first curve above from c. 500 -1500 AD. During the middle ages, there was no sense of how economic or political advantage might be gained from the use of power not derived from human, animal, or natural sources (e.g. waterwheels applied to rivers). The media of the middle ages, too, was meant for interpretation by individual human minds, as opposed to instantiation on larger groups or in artificial substrates. For example, medieval alchemical texts were written with the expectation that the text is exclusively human-readable, and so the informational payload of the text is found not within the structure of the text as an object itself, but in the interpretation and implementation of the text by its human reader; this can be contrasted with the periodic table, which describes a structure already present in nonhuman substrate. This norm continued throughout most of the Enlightenment, perhaps reaching its height during the efforts of the Encyclopedists led by Denis Diderot. The philosophical ideas throughout the 18th and the first half of the 19th century provided the foundation for a scientific community able to take action during the industrial revolution, but the industrial revolution itself provided the necessary material tools for developing automation systems capable of directly realizing political, economic, or scientific goals. The technologies of the industrial revolution, including the social technologies of the factory and the joint stock company developed immediately prior during the late Enlightenment, differed from previous technologies in that they contianed implicit instructions for their operation that were not simply transformations of a source of physical power such as a horse's muscles. A joint-stock company's organization can be thought of as a primative attempt at a superhuman intelligence implemented on a collection of individual human beings. The transition to automated systems that lacked a human in their control loop was the advent of the Jaquard loom (captured in our data as an earlier insight by Basile Bouchon). Earlier automata in the West, the Muslim World, and China, were limited to toys, and their effects limited to the thought that they inspired in their viewers. The Digesting Duck, an automaton from eighteenth century France that crudely pretended the metabolism of a waterfowl, is as inert in its physical effect on the world as a child's toy truck or Michaelangelo's David except by inspiring the behavior of human agents. The automata of the industrial period became actors in their own right. The Jaquard Loom or a modern 3D printer takes a machine-readable instruction and produces an effect that is as physically relevant as if it had been accomplished by human labor. Though automation systems capable of acting as intelligent agents in their own right, rather than replacements for physical labor, could have been developed in the 19th century through the work of Babbage and Lovelace, the uses of cryptography and automatic aiming systems for artillery during the Second World War provided the industrial basis for their maturation. Blechley Park's work in decoding the German Enigma code was perhaps the first instance when computation at a scale not accomplishable in an individual human mind was required to counter an existential threat to a major power. Previous efforts in state computation by actors such as the US Government during the censuses of the 1870s and 1880s provided valuable technological precursors, but these calculating and tabulating machines were not true computers in the modern sense. Cryptographic and cryptoanalytic efforts during the wars provided the centralization of capital required to produce an abstract automation system in the form of Colossus, the first digital programmable computing device. Such an automation system that processs and transforms information rather than simply taking an instruction to produce a physical effect is obviously required for the development of modern artificial intelligence. We must further conjecture that the invention of programmable digital computing devices was required for the psychological conditions necessary for individual scientists and engineeers to experiment in the fields that would eventually birth artificial intelligence. There was only one Turing, publishing his basic principles of computability in On Computable Numbers  (1936) , but there are an uncountable number of individuals able to experiment with the machines derived from these basic insights. The massive upward trend in insights following the end of the Second World War can thus be attributed to access to modern computing machines following their invention, as well as the incentivization of computer science and artificial intelligence research instrumentally by the states locked in conflict during the Cold War for purposes of military advantage. Artificial intelligence, like any intellectual pursuit undertaken as more than a recreation by those with leisure time and intrinsic interest, advances at pace with its support by institutions tasked with the necessities security and prosperity. The distribution of insights from the beginning of the Cold War to the present shows only a marginal deceleration after the fall of the Soviet Union, compared to the roughly linear rate of increase at the beginning of the Cold War. The shallower slopes of curves in the mid-1970s and the late 80s through early 90s correspond to the two periods now described as AI winters, when progress slowed. The former of these two periods appears to be a more substantial decline in insights we measured. We expect this effect is a result of the 1970s being a period of application and combination of existing insights to new problem areas, without substantial generation of additional insights, in addition to the effect of decreased AI-related research during this period. As Nilsson puts it: "Until about the early 1970s, most AI research dealt with what Seymour Papert called 'toy' problems. . . .However, soon after, AI efforts began a definite shift towards applications work, confronting problems of real-world importance. . . .One reason for the increasing interest in applications was that the power of AI methods had increased to the point where realistic applications seemed within reach." (Nilsson, 207)
 Geocultural distribution (Note: data for "Russia" includes insights produced by Soviet scientists born within Russian borders and working at historically Russian institutions.) The distribution of insights by nationality of discover reflects a number of well-understood trends in intellectual history. Ancient Greece provided the intellectual foundations that England and France then built upon during the Enlightenment, before the rise of Germany as it indutrialized and centralized politically. During the twentieth century, the rise of both the Russian and American superpowers correlated to substantial advances, but the United States dramatically outcompeted the Soviet Union. Some of the gains that could be credited to the Soviet Union if one assumes a model based on spheres of influence are marked as Hungarian in origin, as well as a few other countries such as Moldova that did not make it into the top ten. The United States' position as an aggregator of immigrants and refugees from Europe during and after the Second World War substantially increases the number of entries counted as American. The circumstances that made such devices available were not only economic but also politically strategic, as the political aims of military supremacy by both the United States and the Soviet Union during the Cold War enabled their local markets to distribute said computing machines. Additionally, given that the state of Israel was not established until 1949, entries there only date to that time, though many of the scientists who immigrated to Israel and established the scholarly communities that contributed to AI efforts originated in Soviet or American territory.
 Institutional distribution The vast majority of insights in our dataset, unsurprisingly, were discovered and developed by individuals at academic institutions. Early progress in mathematics and the sciences was due mostly to unaffiliated individuals-including most researchers from antiquity and the Renaissance-but as academic institutions gained prestige and resulting brilliance, they began to outcompete other institutional types. Two trends stand out in the latter part of the 20th century: the increase in output from nonacademic R&D institutions, and the slight increase in insights derived within military institutions. Nonacademic R&D includes institutions such as Bell Laboratories and the RAND Corporation, and appears in our data first in 1947 (an attribution to Bell Labs's John Tukey). Bell Laboratories (or Bell Labs) was an unusual institution, in historical context, and in many ways the first of its kind: a result of a step-function increase in the engineering progress needed for telecommunications infrastructure, Bell Labs was incubated within the monopolistic Bell Telephone Company before its 1984 dissolution after a successful antitrust suit. Within it, researchers such as Shannon, Shockley, and Hamming developed some of the core advances of their fields, and it has continued to produce novel research for decades: the most recent insight to come out of Bell Labs, in our data, was from 1995. The RAND Corporation, in contrast, was developed in response to state officials suggesting potential benefits from private involvement in relevant research areas, and RAND itself developed out of an aircraft company, becoming a separate, nonprofit institution in 1948. While insights from RAND represent a shorter timespan in our dataset, limited to just the decade between 1950 and 1960, they outnumber those from any institution besides Stanford and Princeton universities, well-known powerhouses of AI development, by the present day. Dynamic programming (Bellman) and symbolic AI (Newell, Shaw, and Simon) are notable among its body of production. The relative scarcity of of military institutions in our data suggests a more indirect role due to military involvement. The development of artificial intelligence is deeply entangled with military concerns, but miliary institutions seem to have been sources of demand for progress, not sources of progress itself. DARPA, for example, often contracted independent entities such as SRI or teams of collaborating academics to execute on finding solutions in its areas of interest. We note when such entities were affiliated with military institutions, though few such instances occur in our data. A final note on the breakdown of insights among institutional entities: while the credit owed to academic institutions is clear, nonacademic institutions have been contributing an increasing portion of total progress since the middle of the 20th century. We were surprised to see a near even distribution of insights between for-profit and non-profit entities-though RAND Corporation constitutes the bulk of the latter-in recent decades, and we suppose that this supports the importance of ongoing invesment in independent organizations focused on AI whose mission is explicitly aligned with public good.
 Methodological reflections Our methods generate a view of the past through a particular cultural lens-specifically, that of young American computer scientists. This project is not an attempt at the impossible task of aggregating all insights into artificial intelligence that exist, rather it is an attempt from our position to aggregate the ones interpretable to ourselves. As we are roughly in the same position socially as other artificial intelligence resarchers in the Anglosphere, we expect that the amount of insights accessible to Anglosphere researchers will be analagous to those accessible to ourselves. There may be other paths to artificial intelligence development that are only accessible to individuals with differing social position, technical knowledge, or linguistic and cultural proficiency. Though it is certainly likely that such paths exist, we can make no comment on their rate of advancement. Additionally, it seems prudent to make note of the likely presence of survivorship bias: we are limited to only the information that has survived to the point of being accessible to ourselves. It is possible that many of the insights of the past that we have analyzed-particularly the distant past-were developed with the aid of information sources that no longer exist, either dead with the minds that generated them or lost with the documents that once preserved them. As with any field, this lost information may compromise the ability of individuals in the present to correctly interpret and use past insights. This case of survivorship bias is unavoiable, and mitigatable to the degree that present researchers remain aware of it. Related work on this topic is available from Samo Burja of Bismarck Analysis, in his essay Intellectual Dark Matter.