Abstract: Is language understanding a special case of social cognition? To help evaluate this view, we can formalize it as the rational speech-act theory: Listeners assume that speakers choose their utterances approximately optimally, and listeners interpret an utterance by using Bayesian inference to "invert" this model of the speaker. We apply this framework to model scalar implicature ("some" implies "not all," and "N" implies "not more than N"). This model predicts an interaction between the speaker's knowledge state and the listener's interpretation. We test these predictions in two experiments and find good fit between model predictions and human judgments.

Knowledge and Implicature: Modeling Language Understanding as Social Cognition

To what extent does language understanding rely on extra-linguistic knowledge and processes? One view of language processing suggests that it consists of largely separate, special-purpose faculties; another view suggests that it depends critically on domain-general inference mechanisms, and even on intuitive theories that are not specific to language. Indeed, many thinkers have viewed speech as an action with communicative goals, such as informing a listener  (Clark, 1996; Grice, 1975; Levinson, 2000) . A listener making this assumption can make stronger inferences than an utterance would allow from its literal meaning-pragmatic effects can strengthen, or change, the interpreted meaning. Recent work has aimed to formalize the social inference view of pragmatics using tools of Bayesian statistics and information theory  (Frank & Goodman, 2012) ; we refer to this formal framework as the rational speech-act theory of language understanding. It views pragmatic competence as following naturally from an intuitive theory of speech production, which in turn is a special case of intuitive theory of mind. More precisely, listeners have an internal model that describes speakers as choosing their utterances approximately optimally on the basis of certain social goals, such as conveying information to the listener; listeners then interpret an utterance by using Bayesian inference to "invert" this model of the speaker, drawing conclusions about the world state and speaker's intention from the utterance and any other relevant world knowledge. These two rationality assumptions, for listener and speaker, have a role similar to those made in ideal observer models of perception  (Geisler, 2003) : They provide a starting point for a quantitative understanding of the complex interactions involved in language understanding. Indeed, this view has provided good quantitative models for pragmatic inference in a number of simple settings  (Frank & Goodman, 2012; Frank, Goodman, Lai, & Tenenbaum, 2009) . Because rational decision making predicts that action selection is related to the expected utility-a quantity that depends on the actor's belief distribution-a listener who views the speaker as rational should be sensitive to the speaker's belief state. The rational speech-act theory thus predicts an interaction between (shared) knowledge about a speaker's knowledge state and a listener's interpretation of his utterance. This is a very general prediction of the framework, which could easily prove to be false-if pragmatic inference is a highly modularized computation, for instance, we would not expect such general knowledge to affect it. Deriving and testing precise predictions about this interaction thus provide an important test of the rational speech-act theory. If you hear "some of the apples are red," you will infer that not all of the apples are. Pragmatic effects of this sort are called scalar implicatures  (Horn, 2004) , and they provide a window on the interactions between the language faculty and general cognition. Consider Fig.  1 : If the speaker has seen all the apples, his utterance would be interpreted as "some, but not all, of the apples are red." However, if the speaker had only looked at two of the apples, the listener might draw a different conclusion. Indeed, we show below that the implicature "not all" can be canceled by facts about the speaker's perceptual access. This interaction between language understanding and general knowledge is not predicted by strongly modular theories that place scalar implicature within a semantics module  (Chierchia, Fox, & Spector, 2011) . We show further that the interaction of knowledge and implicature is fine grained: The details of a speaker's belief distribution affect the details of an implicature. This article provides a formal model of the pragmatic inference that leads to scalar implicatures, building on the rational speech-act framework. We directly model the possibility that the speaker may have incomplete knowledge and the effects this has on the listener's interpretation. We derive predictions of this model for interpretation of the quantifier "some" and the number words ("one," "two," etc.). The model both explains the standard implicatures for these words and predicts that these implicatures can be canceled, completely or partially, when the speaker has incomplete knowledge. We test these predictions in two experiments and find good fits to human judgements, both qualitatively and quantitatively.
 A rational speech-act model We view language comprehension as a rational inference based on an intuitive theory of language production. Our setting is illustrated in Fig.  1 . The listener infers the world state, s, given the speaker's utterance, w, and shared information about the speaker's (possibly incomplete) information access, a. By Bayes' rule: P listener ðs j w; aÞ / P speaker ðw j s; aÞPðsÞ; ð1Þ where P(s) captures the listener's prior beliefs about the world state and P speaker ðw j s; aÞ describes the listener's intuitive theory of how the speaker chooses words. The speaker chooses an utterance in accord with Bayesian decision theory  (Berger, 1985) : She acts to approximately optimize expected utility. Imagine a speaker who makes observation o about the true state of the world. (For instance, in Fig.  1 , the speaker has perceptual access to two of three apples and observes that those two are red). The speaker selects an utterance w to convey information about the world state to a listener and does so by soft-max optimizing expected utility: P speaker ðw j o; aÞ / expðaE Pðs j o;aÞ ½Uðw; sÞÞ: The speaker's utility function, U(w;s), captures the value of saying w if the world is actually s. The expectation is taken over the speaker's belief state, P(s | o,a), because the speaker may still be uncertain about the state of the world. The parameter a controls the deviation from optimality. So far, nothing in the model is unique to language-indeed, similar models have been used to model social cognition more generally  (Baker, Saxe, & Tenenbaum, 2009; Ullman et al., 2009) . To capture a motivation to be informative, utility must be related to the information conveyed in the utterance.  1  More specifically, utility is related to the amount of information that a literal listener would not yet know about state s after hearing it described by utterance w-this is the negative surprisal  (Cover & Thomas, 1991) : Uðw; sÞ ¼ lnðP lex ðs j wÞÞ; ð3Þ where the literal interpretation probability P lex ðs j wÞ is determined by the lexicon-here, we will assume that each utterance has a truth function, F w : s7 !f0; 1g, and the distribution is otherwise uninformative: P lex ðs j wÞ / d F w ðsÞ . We assume that the speaker's access a is common knowledge of speaker and listener, but the listener still does not know what observation the speaker made, hence: P speaker ðw j s; aÞ ¼ X o P speaker ðw j o; aÞPðo j a; sÞ: ð4Þ In the experiments below (as in Fig.  1 ), the state of the world is always a set of objects that may have a given property and observations consist of looking at a subset of the objects. Thus, the observation probability P(o | a,s) is a hyper-geometric distribution (i.e., the probability of drawing N balls of a given color, without replacement, from an urn containing a given set of colored balls). In this setting, it is also reasonable to assume that the prior probability, P(s), is a binomial distribution (i.e., draws with replacement); we will initially assume so and will later measure the distribution empirically. Overall, the above equations describe the inferences that a rational listener will make to comprehend a speaker that she believes to be approximately rational and have a goal to be informative. Importantly, these inferences depend on shared knowledge about the aspects of the world to which the speaker has access. Thus, the rational speech-act theory predicts that the speaker's access affects utterance interpretation; we test this prediction below. To derive more specific predictions, we must describe the set of alternative utterances.
 The alternatives We have assumed that the interpretation of an utterance is made with respect to a set of alternative utterances. These alternatives could be all possible utterances or could be a limited set generated by replacing key words in the actual utterance with related words. The alternatives may, for instance, be generated by a grammatical mechanism as in  Fox and Katzir (2011) . For our results, the details of this process are unimportant; what is important is that there exists a set of alternative expressions and a (truth-functional) literal meaning for each. We make standard assumptions in both respects. Consider the case of the quantifier words "some" and "all." Under the standard semantics, "all the balls are red" is true exactly when N of the N balls are red, while "some of the balls are red" is true when M ! 1 of the N balls are red. In particular, the literal meaning of "some" allows the state where all N balls are red. We use a lexicon that consists of the standard meanings for "none," "some," and "all." Model predictions are shown in Fig.  2A  for the listener's interpretation of "some of the balls are red" when there are three objects, under varying conditions of speaker access. When the speaker has perceptual access to three of the three objects (hence complete knowledge) and says "some," there is a lower probability on three than two-this is the standard "some but not all" implicature. To understand why the model predicts this implicature, we can first simplify the speaker model: In a complete knowledge situation, the SOME OF THE APPLES ARE RED. ALL SOME Fig.  1 . How will the listener interpret the speaker's utterance? How will this change if she knows that he can see only two of the objects? speaker's belief distribution is concentrated on the true state, so Eqs. 2, 3, and 4 become: P speaker ðw j s; aÞ / expðalnðP lex ðs j wÞÞÞ / ðP lex ðs j wÞÞ a : ð5Þ Using the standard meanings described above, we see that if the state were 3, the speaker would be more likely to say "all" (P lex ðs j wÞ ¼ 1) than "some" (P lex ðs j wÞ ¼ 1 3 ); conversely, if the state were 2, the speaker would say "some," as the probability for "all" is now P lex ðs j wÞ ¼ 0. Now, consider Eq. 1 and imagine for the moment a uniform prior over states. In this case, the listener will infer each state s in proportion to how likely the speaker was to say "some" given this state. Overall, this leads to the implicature that "some" is unlikely to be interpreted as 3-"some but not all." In contrast, when the speaker has only partial access, the calculation is more complex, involving the inferred belief distribution of the speaker. Comparing across the three panels of Fig.  2A , we see the probability of 3 is much higher when access is 1 or 2 than when it is 3. When access is 1, no implicature is predicted (the probability of 3 is approximately the same as the probability of 2); when access is 2, only a very slight implicature. Overall, we predict that incomplete speaker knowledge can cancel the standard "some but not all" implicature. The case of numerals ("one," "two," ...) is similar but more subtle. It has been argued that number words have a lower bound meaning  (Horn, 1972 ) (e.g., "two balls are red" means M ! 2 of the balls are red), and the intuitive, exact, meanings arise as a pragmatic implicature-"one but not two, three, etc." In Fig.  2B , we show model predictions based on the lower bound semantics for number words, varying speaker's access. We see that exact meanings do arise as an inference when the speaker has complete access, but there is an interaction: Number words do not receive an exact interpretation when the speaker has incomplete knowledge. Of particular interest is the case where the speaker has seen two of three objects and says "one": here a partial implicature is predicted, with the probability of 3 low, but 1 and 2 high.
 Experiment 1 Because a rational speaker chooses actions based on expected utility, the rational speech-act model predicts an effect of speaker's knowledge on the listener's interpretation of "some" statements. We tested the predictions of the model by putting participants in the role of the listener and asking them to judge the state of the world in scenarios where perceptual access (and hence knowledgeability) of the speaker varied.
 Participants, materials, and methods Fifty participants were recruited through Amazon's Mechanical Turk crowd-sourcing service and completed the experiment for a small payment. We constructed six scenarios in which a speaker had three objects that could have (or not) a given property. The speaker then made a statement indicating the number of objects he or she had looked at and a quantified ("some") statement. We split each scenario into setup and speech-act phases. The setup phase named the speaker and described the objects and the relevant property. For example: Letters to Laura's company almost always have checks inside. Today Laura received 3 letters. Because our model predicts greater effects when the a priori base rate of the property is high (otherwise it is difficult to tell an implicature from an a prior belief that it is unlikely for all objects to have the property), we describe all properties as "almost always" holding. To make sure participants attended to the setup, we asked them to report the a priori probability that 0, 1, 2, or 3 objects had the property: How many of the 3 letters do you think have checks inside? The speech-act phase introduced a speech act in which the speaker both declared how many of the objects he or she had observed and stated that some objects had the property: Laura tells you on the phone: "I have looked at 2 of the 3 letters. Some of the letters have checks inside." We then again elicited judgements about how many objects had the property: Now how many of the 3 letters do you think have checks inside? Finally, because the speech act might not be a perfect manipulation of speaker's knowledgeability (for instance, the speaker may have gained knowledge by another route), we elicited this directly: Do you think Laura knows exactly how many of the 3 letters have checks inside? Each response was given by a betting measure: Participants were instructed to divide "$100" among the options, betting to indicate their confidence in each option. For the first two questions, there were four options (0-3 of the objects have the property), and for the final question, there were two options (the speaker does/does not have complete knowledge). Before the experiment began, participants were given a brief warm-up, using unrelated questions, to familiarize them with the betting measure. Each scenario existed in forms varying speaker access (the number of objects the speakers had looked at) from 1 to 3. Each participant saw each access condition once, in random order, presented using randomly chosen scenarios (with no duplicate scenarios). In terms of our predictions, we have two partial-knowledge conditions (where we expect cancelation of the implicature) and one complete-knowledge "control" (where we expect the standard implicature). 2
 Qualitative results There was no effect of scenario, so we collapse across this factor in all analyses. As expected based on related work  (Goodman et al., 2009) , the speaker's access statement (e.g., "I have looked at 2 of the 3 letters") was not a perfect manipulation of knowledgeability: In the partial access conditions, some participants judged that the speaker was likely to know exactly how many objects had the property. (The bet that the speaker had complete knowledge when access = 1 was M = 27.1, SD = 4.9; when access = 2, M = 34.8, SD = 5.7; when access = 3, M = 93.0, SD = 2.7.) As we were interested in the effects of varying knowledgeability, we initially exclude trials in which the knowledge judgement was less than 70 in the expected direction (we come back to the full data set in the quantitative analysis below). Fig.  2C  shows the mean of bets on each option, as access varied. As predicted, there was an effect of speaker's access on listener's interpretation (one-way ANOVA with bets on 3 as dependent variable, F(2, 102) = 10.18, p < .001). We next performed our preplanned comparisons to check that an implicature was drawn when the speaker had complete knowledge, but not when the speaker had partial knowledge. In the complete access condition, bets on 3 were less than bets on 2 (paired, directional t test, t(43) = À10.2, p < .001). In the partial access conditions, the implicature was canceled: Bets on 2 did not exceed bets on 3 when speaker had access to 1 object (paired, directional t test, t(31) = 0.77, p = .78) or when access was 2 (paired, directional t test, t(28) = À0.82, p = .21). If we look at just bets on 3, we see significantly lower bets in the complete access condition than the access 1 condition (unpaired, directional t test, t(47) = À4.0, p < .001) or the access 2 condition (unpaired, directional t test, t(43) = À3.5, p < .001). While there was no significant implicature in either partial-access condition, there is a slightly greater tendency toward implicature in the access 2 condition than the access 1 condition, as predicted by the model (two-way ANOVA with bet as dependent variable, and access (1 or 2) and state (2 or 3) as independent variables, F(2, 294) = 3.77, p < .05). Thus, the knowledge of the speaker affected listener's interpretation of "some" in the way predicted by the rational speech-act model. We examine the quantitative fit of the model below.
 Experiment 2 In Experiment 2, we tested the predictions of the rational speech-act model for interpretation of numerals. We again expect to find an effect of speaker's knowledge, but in this case, there is a more detailed interaction: The implicature should be canceled when the speaker says "one" after seeing only one object and when the speaker says "two" after seeing two objects, but it should only be partially canceled when the speaker says "one" after seeing two objects-this implies a fine-grained interplay between the speaker's knowledge state and the interpretation of her utterance.
 Participants, materials, and methods Fifty participants were recruited through Amazon's Mechanical Turk crowd-sourcing service and completed the experiment for a small payment. We used the same stimuli as in Experiment 1, modifying the scenarios only in the speech act: The speaker now made a statement indicating the number of objects he or she had looked at and the number that had the property. For instance: Laura tells you on the phone: "I have looked at 2 of the 3 letters. 1 of the letters has checks inside." Each scenario existed in forms varying the speaker's access, from 1 to 3, and the number word the speaker used, from 1 to 3; we limited to sensible situations where the word used was no greater than the number of objects seen. Hence, we had six conditions, with access/word: 1/1, 2/1, 2/2, 3/1, 3/2, and 3/3. In terms of our predictions, we have three partial-knowledge conditions (where we expect partial or complete cancelation of the implicature) and three complete-knowledge "controls" (where we expect the standard implicatures). The order of scenarios and the order of conditions were randomized between participants.
 Qualitative results There was again no effect of scenario, so we collapse across this factor. As in Experiment 1, the speaker's access statement was not a perfect manipulation of knowledgeability (bets that speaker had complete knowledge in partial-access conditions: M = 42.0, SD = 3.4, in complete access conditions: M = 92.1, SD = 1.6). We once again limit to trials with the expected judgements of knowledgeability (with a threshold of 70). The mean of participants' bets are shown in Fig.  2D . To evaluate the overall effect of access, we performed an ANOVA with access and word as independent measures and bet on 3 as dependent measure. We find a main effect of access (F(2, 205) = 6.57, p < .01), an interaction between word and access (F(1,205) = 34.7, p < .001), and a main effect of word (F(2, 205) = 269.8, p < .001). We then explored the results in more detail using planned comparisons to test whether implicatures were drawn (only) when predicted. We found an implicature in the complete access conditions: When the speaker said "two," bets on state 3 were less than on state 2 (paired, directional t test, t(43) = À10.2, p < .001). When the speaker said "one," bets on state 1 were greater than on state 3 (paired, directional t test, t(42) = À13.1, p < .001) or state 2 (paired, directional t test, t(42) = À17.1, p < .001). In contrast, there was no implicature when access was 1 and the speaker says "one"-bets on 1 were not greater than on 2 (paired, directional t test, t(24) = 1.9, p = .96) or on 3 (paired, directional t test, t(24) = 3.2, p = 1.0)-and no implicature when access is 2 and the speaker says "two"-bets on 2 were not greater than on 3 (paired, directional t test, t(24) = 1.1, p = .87). When access is 2 and the speaker says "one" we found the predicted partial implicature: Bets on state 1 were significantly greater than on state 3 (paired, directional t test, t(25) = À3.9, p < .001), but not on state 2 (paired, directional t test, t(25) = 1.5, p = .92). These results again support the predictions of the rational speech-act model, showing not merely an interaction between the speaker's knowledge and the listener's tendency to draw an implicature, but a fine-grained interaction that is unlikely to result from a modular process of language understanding. In addition, these results support the standard, but controversial  (Barner & Bachrach, 2010; Huang, Snedeker, & Spelke, 2004) , view that number words have a "lower-bound" semantics which is only strengthened into an exact meaning by pragmatic inference.
 Quantitative model comparison To evaluate the quantitative model predictions, we first compare model predictions with mean human ratings for the subset of trials in which participants gave knowledgeability ratings in the expected direction (>70, as above). As in the model description, we assume a binomial prior distribution. We fit the prior base rate parameter and the a parameter by minimizing the root mean squared error (RMSE) of the model predictions and mean data for both experiments. The resulting model fit is RMSE = 9.01 and r = .96. The model predictions with the best-fitting parameters are shown in Fig.  2A,B  and show striking correspondence with the human data in Fig.  2C,D . To consider all responses, including those previously removed due to unexpected knowledge judgements, we extend the model by including an additional knowledge parameter: the probability that the speaker did in fact have complete knowledge (regardless of perceptual access). As we measured prior expectations and knowledgeability in each trial, we compute model predictions, trial by trial, using these values. Because the betting interface encouraged participants to round small bets to 0, but probability 0 is very different than a non-zero small probability to the model, we changed 0 and 100 bets to 1 and 99. In addition, we assumed a simple power-law relationship between subjective probability and bets (equivalent to the soft-max used to model the speaker, Eq. 2). The parameter of this power law and the speaker optimality, a, were then fit by minimizing the RMSE of mean model predictions to mean human judgements in both experiments. The resulting fit was again good: RMSE = 8.36, r = .95. Looking at individual participants, we find median correlation of r = .75 between a participant's judgements and the model predictions based on his or her prior and knowledge scores. These results suggest that the rational speech-act model is able to capture the quantitative pattern of people's judgements both across the population and within individual participants.
 Conclusion We have described a rational speech-act model of scalar implicatures and their interaction with speaker knowledge. This model formalizes language understanding as social cognition, with language-specific goals and actions, using the tools of Bayesian statistics. In addition to predicting the standard implicatures ("some but not all") as an inference that depends on the alternative utterances, this model predicted that these implicatures could be canceled, completely or in part, when it was common knowledge that the speaker had incomplete knowledge. Experiments 1 and 2 verified these qualitative predictions and showed tight quantitative fits with the model. The predicted interaction between interpretation and speaker's knowledge was not a peculiarity of this set of words or of scalar lexical items; instead, it follows from the general fact that rational decision makers must choose actions based on their expected utility. In contrast, a strongly modular model of implicatures would predict no such interactions. One could amend these theories  (Chierchia et al., 2011)  to allow a speaker's ignorance to affect the implicature mechanism. To predict the fine-grained interactions demonstrated for number words, additional machinery would be required, describing how the details of a speaker's knowledge state influence a listener's interpretation. In contrast, we have shown that the rational speech-act framework parsimoniously predicts these effects from high-level assumptions about the goals of listener and speaker. The rational speech-acts framework we have used here is closely related to that of game-theoretic pragmatics (J€ ager, in press), and particularly to the use of lifted games to capture epistemic effects  (Franke, 2009) . There are two principal differences of game-theoretic approaches from ours: In those approaches, it is assumed that speakers fully optimize (a ? ∞ in our framework), and that they carry out deeply recursive social reasoning-"I think, that you think, that I think, that..."-while we assume only one such level of reasoning. The quantitative fits we have shown suggest that limited recursion and optimization are psychologically realistic assumptions. Future work will be needed to explore all the possible models in this space. Our results support the rational speech-act framework for modeling pragmatics. More generally, they further boost the momentum building for quantitative models of language as a branch of rational social cognition. In the words of  Grice (1975) : "One of [our] avowed aims is to see talking as a special case or variety of purposive, indeed rational, behavior..." Fig. 2. (A and B)Model prediction for probability of each world state (number of objects with property), varying the word the speaker used and the speaker's perceptual access. The prior is assumed to be binomial with base rate 0.62, and the speaker optimality parameter is set to a = 3.4. (C and D) Mean participant bet on each world state, varying the word the speaker used and the speaker's perceptual access. Data have been filtered to include only trials where the participant's bet that the speaker had complete knowledge was greater than 70 in the expected direction. Error bars are standard error of the mean.
			 N. D. Goodman, A. Stuhlm€ uller / Topics in Cognitive Science 5 (2013)

/Users/janhendrikkirchner/code/2022/10/alignment-research-dataset/align_data/common/../../data/raw/nonarxiv_teis/Topics in Cognitive Science - 2013 - Goodman - Knowledge and Implicature Modeling Language Understanding as Social.tei.xml


Abstract: Newcomblike decision problems have been studied extensively in the decision theory literature, but they have so far been largely absent in the reinforcement learning literature. In this paper we study value-based reinforcement learning algorithms in the Newcomblike setting, and answer some of the fundamental theoretical questions about the behaviour of such algorithms in these environments. We show that a value-based reinforcement learning agent cannot converge to a policy that is not ratifiable, i.e., does not only choose actions that are optimal given that policy. This gives us a powerful tool for reasoning about the limit behaviour of agents -for example, it lets us show that there are Newcomblike environments in which a reinforcement learning agent cannot converge to any optimal policy. We show that a ratifiable policy always exists in our setting, but that there are cases in which a reinforcement learning agent normally cannot converge to it (and hence cannot converge at all). We also prove several results about the possible limit behaviours of agents in cases where they do not converge to any policy.

James Bell
Linda Linsefors
Caspar Oesterheld
Joar Skalse

Reinforcement Learning in Newcomblike Environments

Introduction In this paper, we study decision scenarios in which outcomes depend not only on the choices made and physically implemented, but also depend directly on the agent's policy. As an example, consider an autonomous vehicle (AV) whose goal it is to arrive at target destinations quickly while minimising the probability of collisions. In practice, AVs are careful drivers. It is easy to imagine an experiment (or learning process) that might support careful driving: on each day, let the AV decide at random between a careful and a more aggressive style of driving; other drivers on the road are unaware of today's chosen driving style and therefore behave the same around the AV on both types of days. Presumably the decrease in accidents on careful days outweighs the increase in travel time. However, imagine now that a type of AV was widely deployed. Then many of the drivers with whom the AVs interact on the road would know a lot about how these AVs behave (e.g., from reading about AVs, or from having interacted with other AVs of the same type in the past). In particular, if the other drivers know that the AVs rarely take risks, they might (whether rationally or irrationally) cut them off more, not give them right of way, etc. relative to the above experiment. Indeed, this phenomenon -human drivers bullying timid AVs -has been reported in the real world  (Condliffe, 2016; Liu et al., 2020; cf. Cooper et al., 2019) . As a result, the travel times of an AV are much longer if it always follows (and is known to follow) a careful driving policy. Moreover, some of the safety benefits of careful choices disappear if the AV adopts a careful policy. To comfortably model situations such as this one, we introduce Newcomblike decision processes (NDPs). The name derives from Newcomb's problem  (Nozick, 1969) , described in the next subsection, and similar problems that have been studied in the decision-theoretic literature. NDPs are a generalisation of Markov decision processes wherein the transition probabilities and rewards depend not only on the agent's action, but also directly on the agent's policy. Thus, for example, how aggressively other cars move depends on the timidness of the AV's policy. We describe NDPs in more detail in Sect. 1.1. Importantly, the NDP model does not assume other agents in the environment to respond rationally to the agent's policy. Thus, some NDPs cannot comfortably be modelled as games. (See Sect. 5.1 for a more detailed discussion of the relation between NDPs and game-theoretic models.) We believe that Newcomblike dynamics are commonplace when AI systems interact with other (human or artificial) agents  (Cavalcanti, 2010, Sect. 5; Oesterheld, 2019, Sect. 1; Conitzer, 2019) . The deployment of AVs is rife with such dynamics. Besides aggressiveness, it might matter whether a policy is simple and thus predictable to humans, for instance. Another real-world scenario is that of recommendation systems: most readers of this paper have some idea of how these systems work and make choices based on it. ("I would like to watch this cat video, but if I do, my recommendations will soon be full of them.") Thus, the success of a particular recommendation depends not only on the recommendation itself, but also on the recommendation system's policy. We are interested in learning to play NDPs. More specifically, we study the behaviour of value-based, model-free RL agents, who maintain a Q-function that assigns values to state-action pairs. We define these in more detail in Sect. 1.2. As we will see, such agents do not in general learn optimal policies in NDPs (as they do in MDPs). Nevertheless, we believe that studying them is an important first step in developing practical learning algorithms for NDPs due to the combination of the following points. A) For illustrative purposes, the examples we discuss throughout this paper are simple and emphasise the dependence of the environment on the policy. However, we think that most real-world scenarios are only partially Newcomblike. For example, most of the AV's environment changes only in response to an AV's actions and does not directly depend on the AV's policy. B) Value-based reinforcement learning algorithms are very well developed. In contrast, we would have to develop specialised learning algorithms for general NDPs from scratch. C) As our results will show, in some situations and when set up correctly (e.g., in terms of learning rates) value-based learning converges to optimal policies, or at least to reasonable policies, even under Newcomblike dynamics. For example, in a game of rock-paper-scissors against an opponent who knows the agent's policy, some value-based learning agents learn the optimal policy of mixing uniformly. In light of A-C, we think that the most realistic paths to developing learning algorithms for real-world scenarios with Newcomblike dynamics will involve value-based RL. Specifically, one avenue toward realistic algorithms is to develop extensions of value-based RL and can detect and correct failures that might arise in Newcomblike dynamics. For that, we need to understand how value-based RL behaves in NDPs.
 Contributions In Sect. 2 we demonstrate that value-based RL algorithms can only converge to a policy that is ratifiable -that is, to a policy ⇡ for which all actions taken by ⇡ have optimal expected reward when following ⇡. In Sect. 3, we discuss the convergence properties of agents in Newcomblike situations, and show that there are cases where value-based agents must fail to converge. The action frequencies might converge, even when the policies do not. In Sect. 4, we establish some conditions on any action frequency that an agent could converge to. We also show that there are decision problems and agents where even the action frequencies do not converge.
 Newcomblike Decision Processes A Newcomblike decision process (NDP) is a tuple hS, A, T, R, i where S is a finite set of states; A is a finite set of actions; T : S ⇥ A ⇥ (S A) S is a nondeterministic transition function; R : S ⇥ A ⇥ S ⇥ (S A) R is a nondeterministic reward function, which we assume to be bounded; and 2 [0, 1) is a discount factor.
 A policy ⇡ : S A is a function that nondeterministically maps states to actions. We use ⇡(a | s) to denote the probability of taking action a in state s while following the policy ⇡. T and R are functions from states, actions, and policies. In other words, they allow the outcome of a decision to depend on the distributions from which the agent draws its actions, rather than just the state and the action that is in fact taken. Also note that T (s, a, ⇡) and R(s, a, s 0 , ⇡) are defined even if ⇡(a | s) = 0. We say that an NDP is a bandit NDP if it has only one state. We will sometimes use R(s, a, ⇡) as a shorthand for R(s, a, T (s, a, ⇡), ⇡), and we will sometimes omit the state from T , R, and ⇡ for bandit NDPs. Moreover, we normally let = 0 for bandit NDPs. Consider a distribution over initial states for an agent, and let ⇡ be its policy, let x t be the sequence of states it visits and a t the sequence of actions it takes. We say ⇡ is optimal for that distribution if it maximises E[ P 1 i=0 i R(x i , a i , a i+1 , ⇡)]. Note that unlike in the MDP case the optimal policy does depend on the initial distribution, however this isn't relevant in the bandit case. As an example consider the eponymous Newcomb's Problem. Newcomb's Problem  (Nozick, 1969) : There are two boxes in front of you; one opaque box, and one transparent box. You can see that the transparent box contains $1,000. You can choose to either take only the opaque box, or to take both boxes. The boxes have been placed in this room by an agent who can predict your policy; if he believes that you will take only the opaque box then he has put $1,000,000 in the opaque box, but if he believes that you will take both boxes then he has left the opaque box empty. Do you take one box, or two? A version of Newcomb's Problem can be formalised as the following bandit NDP: S = {s}, A = {a 1 , a 2 }, R(a 1 , ⇡) = ⇢ 0 w.p. ⇡(a 2 ) 10 w.p. ⇡(a 1 ) and R(a 2 , ⇡) = ⇢ 5 w.p. ⇡(a 2 ) 15 w.p. ⇡(a 1 ) , where "w.p." is short for "with probability". The key feature of this NDP is that, for any fixed policy, a 2 ("two-boxing") yields a higher reward than a 1 ("one-boxing"). But the expected reward of a policy increases in ⇡(a 1 ) s.t. the optimal policy is to always play a 1 .  1  We can view Newcomb's problem as a simple version of the AV dynamic described in the introduction, where a 2 is a driving action that allows other drivers to cut the AV off at no risk. We say that an NDP is continuous if T and R are continuous in the policy. In this paper we work mainly with continuous NDPs. This is in part because it is technically convenient, and in part because we believe that continuity is satisfied in many realistic cases. 2
 Reinforcement Learning Agents We consider value-based reinforcement learning agents. Such agents have two main components; a Q-function S ⇥ A ! R that predicts the expected future discounted reward conditional on taking a particular action in a particular state, and a bandit algorithm that is used to select actions in each state based on the Q-function. Given a policy ⇡, we use q ⇡ (a | s) to denote the (true) expected future discounted reward conditional on taking action a in state s while following the policy ⇡ (and conditional on all subsequent actions being chosen by ⇡). A model-free agent will update Q over time to make it converge to q ⇡ when following ⇡. If Q is represented as a lookup table, the agent is said to be tabular. If the state space is large, it is common to instead approximate q ⇡ (with e.g. a neural network). For simplicity, we focus mostly on tabular agents. However, some of our results (Theorems The Q-values can be updated in different ways. One method is to use the update rule Q t+1 (a t | s t ) (1 ↵ t (s t , a t )) Q t (a t | s t ) + ↵ t (s t , a t )(r t + max a Q t (a | s t+1 )), where a t is the action taken at time t, s t is the state visited at time t, r t is the reward obtained at time t, and ↵ t (s, a) is a learning rate. This update rule is known as Q-learning  (Watkins, 1986) . Other widely used update rules include SARSA  (Rummery and Niranjan, 1994)  and Expected SARSA  (van Seijen et al., 2009) . For the purposes of this paper it will not matter significantly how the Q-values are computed, as long as it is the case that if an agent converges to a policy ⇡ in some NDP and explores infinitely often then Q converges to q ⇡ . We will later see that this is the case for Q-learning, SARSA, and Expected SARSA in continuous NDPs. There are also several different bandit algorithms. Two types of agents that are widely used in practice and that we will refer to throughout the paper are softmax agents and ✏-Greedy agents. The policy of a softmax agent with a sequence of temperatures t 2 R + is given by: ⇡ t (a | s) = exp(Q t (a | s)/ t ) P a 0 2A exp(Q t (a 0 | s)/ t ) . Unless otherwise stated we assume that t ! 0. The policy of an ✏-Greedy agent with a sequence of exploration probabilities ✏ t 2 [0, 1] is ⇡ t (a | s) = 1 ✏ t if a = arg max a 0 Q t (a 0 | s) and ⇡ t (a | s) = ✏ t /(|A| 1) otherwise. Unless otherwise stated we assume that ✏ t ! 0. We assume that ✏-Greedy breaks ties for argmax, so that there is always some a 2 A such that ⇡(a | s) = 1 ✏ t . We say that an agent is greedy in the limit if the probability that the agent takes an action that maximises Q converges to 1, and we say that it explores infinitely often if it takes every action in every state infinitely many times.
 Some Initial Observations We here make three simple observations about NDPs that we will use to prove and understand the results throughout this paper. First, a continuous NDP always has, for each possible distribution over initial states, a policy ⇡ that maximises the expected discounted reward E[R | ⇡], since E[R | ⇡] exists and is continuous in ⇡, and since the set of possible policies is a compact set. Also note that an NDP in which T or R is discontinuous may not have any such policy. Second, whereas all MDPs have a deterministic optimal policy, in some NDPs all optimal policies randomise. To see this we introduce another example we will look at in this paper. Death in Damascus  (Gibbard and Harper, 1976 ): Death will come for you tomorrow. You can choose to stay in Damascus (where you are currently) or you can flee to Aleppo. If you are in the same city as Death tomorrow, you will die. Death has already decided which city he will go tohowever, he can predict your policy, and has decided to go to the city where he believes that you will be tomorrow. Do you stay in Damascus, or flee to Aleppo? We formalise this as the bandit NDP S = {s}, A = {a Damascus , a Aleppo }, and R(a Damascus , ⇡) = ⇢ 0 w.p. ⇡(a Damascus ) 10 w.p. ⇡(a Aleppo ) and R(a Aleppo , ⇡) = ⇢ 10 w.p. ⇡(a Damascus ) 0 w.p. ⇡(a Aleppo ) , where "w.p." is again short for "with probability". In this NDP, randomising uniformly between a Damascus and a Aleppo is the unique optimal policy and in particular outperforms both deterministic policies. Note also that the Bellman optimality equation does not hold for NDPs. Even in Newcomb's Problem, as described above, Bellman's optimality equation is not satisfied by the optimal policy.
 Ratifiability If an agent in the limit only takes the actions with the highest Q-values and it converges to some policy ⇡ 1 , then it is clear that, for a given state, all actions in the support of ⇡ 1 must have equal expected utility given ⇡ 1 . Otherwise, the Q-values would eventually reflect the differences in expected utility and the agent would move away from ⇡ 1 . Similarly, if the algorithm explores sufficiently often, the actions that are taken with limit probability 0 cannot be better given ⇡ 1 than those taken by ⇡ 1 . After all, if they were better, the agent would have eventually figured this out and assigned them large probability. This condition on ⇡ 1 resembles a well-known doctrine in philosophical decision theory: ratificationism (see  Weirich, 2016, Sect. 3.6 , for an overview). One form of ratificationism is based on a distinction between a decision -what the agent chooses -and the act that is selected by that decision. Very roughly, ratificationism then states that a decision is rational only if the acts it selects have the highest expected utility given the decision. Concepts of causality are often invoked to formalise the difference between the decision, the act, and their respective consequences. Our setup, however, has such a differentiation built in: we will view the policy as the "decision" and the action sampled from it as the "act".
 Strong Ratifiability As hinted earlier, slightly different versions of the concept of ratifiability are relevant depending on how much exploration a learning algorithm guarantees. We start with the stronger version, which more closely resembles what decision theorists mean when they speak about ratifiability. Definition 1. Let M ✓ S be a set of states. A policy ⇡ is strongly ratifiable on M if supp(⇡(• | s)) ✓ arg max a2A q ⇡ (a | s) for all s 2 M . In Newcomb's Problem the only strongly ratifiable policy is to play a 2 with probability 1. In Death in Damascus, only the optimal policy (mixing uniformly) is strongly ratifiable. There can also be several strongly ratifiable policies. For example, if you play the Coordination Game of Table  1  against an opponent who samples his action from the same policy as you then there are three strongly ratifiable policies; to select action a with probability 1, to select action b with probability 1, and to select a with probability 1 /3 and b with probability 2 /3. Theorem 2. Let A be a model-free reinforcement learning agent, and let ⇡ t and Q t be A's policy and Q-function at time t. Let A satisfy the following in a given NDP: • A is greedy in the limit, i.e. for all > 0, P (Q t (⇡ t (s)) max a Q t (a | s) ) ! 0 as t ! 1. • A's Q-values are accurate in the limit, i.e. if ⇡ t ! ⇡ 1 as t ! 1, then Q t ! q ⇡1 as t ! 1. Then if A's policy converges to ⇡ 1 then ⇡ 1 is strongly ratifiable on the states that are visited infinitely many times. a b a 2,2 0,0 b 0,0 1,1
 Table 1: The Coordination Game In Appendix A we show that the Q-values of a tabular agent are accurate in the limit in any continuous NDP if the agent updates its Q-values with SARSA, Expected SARSA, or Q-learning, given that the agent explores infinitely often and uses appropriate learning rates. Since we would expect most well-designed agents to have accurate Q-values in the limit, Theorem 2 should apply very broadly. Using Kakutani's fixed-point theorem, it can be shown that every continuous NDP has a ratifiable policy. Theorem 3. Every continuous NDP has a strongly ratifiable policy. Of course, the fact that a ratifiable policy always exists does not necessarily mean that a reinforcement learning agent must converge to it -we will consider the question of whether or not this is the case in Sect. 3. It is also worth noting that a discontinuous NDP may not have any strongly ratifiable policy. It is a topic of ongoing discussion among philosophical decision theorists whether (strong) ratifiability should be considered a normative principle of rationality, see  Weirich (2016, Sect. 3.6)  for details. In general, the policy ⇡ that maximises E[R | ⇡] may or may not be ratifiable, as shown by Death in Damascus and Newcomb's problem, respectively. There is a correspondence between ratificationism and many game-theoretic concepts. For example, if you are playing a zero-sum game against an opponent who can see your policy and plays some distribution over best responses to it then ⇡ can only be ratifiable if it is a maximin strategy. To give another example, if you are playing a symmetric game against an opponent who follows the same policy as you then ⇡ is ratifiable if and only if (⇡, ⇡) is a Nash equilibrium. Joyce and Gibbard (1998, Sect. 5) discuss the relation in more detail.
 Weak Ratifiability We now show that even without infinite exploration, ⇡ 1 must still satisfy a weaker notion of ratifiability. Definition 4. Let M ✓ S be a set of states. A policy ⇡ is weakly ratifiable on M if q ⇡ (a | s) is constant across a 2 supp(⇡(s)) for all s 2 M . What makes this a weak version of ratifiability is that it does not put any requirements on the expected utility of actions that ⇡ does not take, it merely says that all actions that ⇡ takes with positive probability must have the same (actual) q-value. As a special case, this means that all deterministic policies are weakly ratifiable. This includes one-boxing in Newcomb's problem. Nonetheless, there are bandit NDPs in which the optimal policy is not even weakly ratifiable. For example, consider an NDP with actions a 1 , a 2 , where R(a 1 , ⇡) = 100(⇡(a 1 ) 1 /2) 2 + 1 and R(a 2 , ⇡) = 100(⇡(a 1 ) 1 /2) 2 . The optimal policy mixes close to uniformly (⇡(a 1 ) = 101 /200), but this is not weakly ratifiable, because R(a 1 , ⇡) > R(a 2 , ⇡). Theorem 5. Same conditions as Theorem 2, but where A's Q-values are only required to be accurate in the limit for state-action pairs that A visits infinitely many times. Then ⇡ 1 is weakly ratifiable on the set of states that are visited infinitely many times.
 Non-Convergence of Policies We have shown that most reinforcement learning algorithms can only converge to (strongly) ratifiable policies. We now consider the question of whether they always converge to a policy at all. We find that this is not the case.
 Theoretical Results From Theorem 2 it follows that in e.g. Death in Damascus an ✏-Greedy agent who explores infinitely often cannot converge to any policy. After all, the only strongly ratifiable policy (and thus limit policy) is to mix uniformly and an ✏-Greedy agent never mixes uniformly. Perhaps more surprisingly, there are also NDPs in which a (slow-cooling) softmax agent cannot converge to any policy. As an example, consider a bandit NDP with three actions a 1 , a 2 , a 3 , and where the rewards R(a i , ⇡) have expectations ⇡(a i+1 ) + 4•13 3 •⇡(a i ) [8j:⇡(a j ) 1 /4] Y j (⇡(a j ) 1 /4) . (1) For i = 3, we here let a i+1 = a 1 . We also require that the rewards are stochastic with a finite set of outcomes such that the empirical Q-values are never exactly equal between different actions. We call this the Repellor Problem. It has only one strongly ratifiable policy (mixing uniformly), but -as illustrated by Figure  1  -when the current policy mixes close to uniformly, the softmax agent learns (in expectation) to play less uniformly. Theorem 6. Let A be an agent that plays the Repellor Problem, explores infinitely often, and updates its Q-values with a learning rate ↵ t that is constant across actions, and let ⇡ t and Q t be A's policy and Q-function at time t. Assume also that for j 6 = i, if ⇡ t (a i ), ⇡ t (a j ) both converge to positive values, then ⇡ t (a i ) ⇡ t (a j ) Q t (a i ) Q t (a j ) ! a.s.
 1 (2) as t ! 1. Then ⇡ t almost surely does not converge. Line 2 is satisfied, for example, for softmax agents with t converging to 0. Recall also that e.g. Q-learning and SARSA are equivalent for bandit NDPs (if = 0).
 Empirical Results Figure  1 : The triangle shows the space of possible policies in the Repellor Problem, parameterised by the probability they assign to each of the three actions. Plotted against this space is the expected direction in which a softmax agent would change its policy if playing a particular policy. Empirically, softmax agents converge (to strongly ratifiable policies) in many NDPs, provided that the temperature decreases sufficiently slowly. To illustrate this we will use Aymmetric Death in Damascus, a version of Death in Damascus wherein the rewards of a Aleppo are changed to be 5 (instead of 0) with probability ⇡(a Aleppo ) and (as before) 10 with the remaining probability. This NDP has only one (strongly) ratifiable policy, namely to go to Aleppo with probability 2 /3 and Damascus with probability 1 /3. This is also the optimal policy. We use this asymmetric version to make it easier to distinguish between convergence to the ratifiable policy and the default of uniform mixing at high temperatures. Figure  2  shows the probability of converging to this policy with a softmax agent and a plot of the policy on one run. We can see that this agent reliably converges provided that the cooling is sufficiently slow. More accurately it is a plot of the fraction of runs which assigned a Q-value of at least 5.5 to the action of going to Aleppo after 5000 iterations. These are empirical probabilities from 20,000 runs for every ↵ that is a multiple of 0.025, and 510,000 runs for each ↵ that is a multiple of 0.005 between 0.5 and 0.55. Notice the "kink" at ↵ = 0.5. Based on our experiments, this kink is not an artefact and shows up reliably in this kind of graph. The right-hand figure shows how the action probabilities evolve over time for a single run (chosen to converge to the mixed strategy) for ↵ = 0.3. However, there are also fairly simple games in which it seems like softmax agents cannot converge. Consider Loss-Averse Rock-Paper-Scissors (LARPS), the problem of playing Rock-Paper-Scissors against an opponent that selects each action with the same probability as you, and where you assign utility 1 to a win, 0 to a draw, and -10 to a loss. We conjecture that slow-cooling softmax agents do not converge in LARPS. We have unfortunately not been able to prove this formally, but Figure  3  presents some empirical data which corroborates the hypothesis.
 Convergence of Action Frequencies We have seen that there are NDPs in which some reinforcement learning algorithms cannot converge to any policy. But if they do not converge to any policy, what does their limit behaviour look like? We now examine whether these algorithms converge to taking each action with some limit frequency, and what sorts of frequencies they can converge to. In this section we establish a number of conditions that must be satisfied by any limit action frequency of a value-based agent. We consider agents that converge to deterministic policies (such as ✏-Greedy agents), and we limit our analysis to the bandit case (with = 0).
 Possible Frequencies in the Bandit Case Let P ⌃ t : A ! [0, 1] be the frequency with which each action in A is taken in the first t steps (for some agent and some bandit NDP). Note that P ⌃ t is a random variable. By the law of large numbers, P ⌃ t (a) 1 /t P t i=0 ⇡ i (a) converges to 0 almost surely as t ! 1. Let ⇡ a be the policy that takes action a with probability 1, and let q a = q ⇡a . Theorem 7. Assume that there is some sequence of random variables (✏ t 0) t s.t. ✏ t ! t!1 a.s. 0 and for all t 2 N it is X a ⇤ 2arg max a Qt(a) ⇡ t (a ⇤ ) 1 ✏ t . (3) Let P ⌃ t ! p ⌃ with positive probability as t ! 1. Then across all actions a 2 supp(p ⌃ ), q a (a) is constant. That is, the actions played with positive limit frequency must all be equally good when played deterministically. This condition is vaguely analogous to weak ratifiability, and is proven in roughly the same way as Theorem 2. Theorem 8. Same assumptions as Theorem 7. If |supp(p ⌃ )| > 1 then for all a 2 supp(p ⌃ ) there exists a 0 2 A s.t. q a (a 0 ) q a (a). This condition is an instability condition. Say that multiple actions are taken with nonzero limit frequency, and that action a has the highest Q-value at time t. Then for other actions to be played with positive limit frequency, other actions must at some point be believed to be optimal again (since the probability of exploration goes to zero). Hence they cannot all be worse when explored while mainly playing a, since a could otherwise be played forever. Theorem 9. Same assumptions as Theorem 7. Let U be the Q-value q a (a) which (by Theorem 7) is constant across a 2 supp(p ⌃ ). For any a 0 2 A supp(p ⌃ ) that is played infinitely often, let frequency 1 of the exploratory plays of a 0 happen when playing a policy near elements of {⇡ a | a 2 supp(p ⌃ )}. Then either there exists a 2 supp(p ⌃ ) such that q a (a 0 )  U ; or q a 0 (a 0 ) < U. Theorem 9 describes what circumstances are needed for an actions a 0 to be played with limit frequency zero. One possibility is that exploration is done only finitely many times (in which case bad luck could lead to low Q-values). A second possibility is that the exploration mechanism is "rigged" so that a 0 is mostly played when playing policies outside the proximity of {⇡ a | a 2 supp(p ⌃ )}. In this case the utility of a 0 under some zero-limit-frequency policy might lead to low Q-values. If exploration of a 0 is spread out more naturally then all but frequency zero of that exploration will happen near elements of {⇡ a | a 2 supp(p ⌃ )}. In this case, the only reason for a 0 to be played with zero frequency is that exploring a 0 near some of the elements of {⇡ a | a 2 supp(p ⌃ )} makes a 0 look poor.
 When is Frequency Convergence Possible? We believe there are NDPs in which an ✏-Greedy agent cannot converge to any limit action frequency. Specifically, we believe that LARPS is such an example. Figure  4a  shows the directions in which the frequencies of different actions evolve. The graph seems to have no attractor and hence we believe an ✏-Greedy agent cannot converge to any limit action frequency in this NDP. We have not been able to rigorously prove this. However, experiments seem to confirm this hypothesis. Figure  4b  depicts five  runs of ✏-Greedy in LARPS. We can see that the agents oscillate between different actions, and that the periods increase in length.
 Related Work
 Learning in games Some of the Newcomblike dynamics we have described in this paper could also be modelled as games, especially as so-called Stackelberg games in which one player, the Stackelberg leader, chooses a strategy first and another player, the Stackelberg follower, then responds to that strategy. For example, in the case of autonomous vehicles (AVs), we might imagine that the AV company is the Stackelberg leader and the human drivers are the Stackelberg followers. That said, there are differences between NDPs and games. NDPs can react arbitrarily to the agent's policy, whereas in games, the other players play a best (i.e., expected-utility-maximising) response. In many real-world situations, other agents in the environment cannot be comfortably modelled as expected-utility-maximising agents. Interactions between AVs and humans can serve as examples. Most people probably do not reason rationally about small-probability, big-impact events, such as car crashes. Also, humans will generally operate on simplified models of an AV's policy (even when more detailed models are available). Of course, a game-theoretic analysis can also be fruitful and address issues that we ignore: By assuming all players to be rational, game theory can provide recommendations and predictions for multiple agents simultaneously, while our NDP model considers a single agent for a given environment. We believe that the NDP perspective provides additional insight into learning in such situations. Despite the differences between NDPs and games, there are some interesting parallels between model-free learning in NDPs and in games, where similar learning methods are sometimes referred to as fictitious play  (Brown, 1951) .  Fudenberg and Levine (1998, Chapter 2)  show that fictitious play can only converge to a Nash equilibrium (for similar results about convergence to Nash equilibrium, see e.g.  Mazumdar et al., 2020 , Oesterheld et al., 2021 . As noted in Sect. 2.1, the concept of Nash equilibrium resembles the concept of ratifiability.  Shapley (1964)  shows that fictitious play can fail to converge. However, there are many special cases in which convergence is guaranteed, including two-player zero-sum games  (Robinson, 1951)  and generic 2 ⇥ 2 games  (Miyasawa, 1961) .
 Learning and Newcomblike problems Other authors have discussed learning in Newcomblike problems. The most common setup is one in which the learner assigns values directly to policies, or more generally to that which the agent chooses. It is then usually shown that (among the policies considered) the agent will converge to taking the one with the highest (so-called evidential) expected utility  (Albert and Heiner, 2001; Oesterheld, 2018) . This contrasts with our setup, in which the learner selects policies but assigns values to actions. Oesterheld (2019) also studies agents who maximise reward in Newcomblike environments. However, Oesterheld does not consider the learning process. Instead he assumes that the agent has already formed beliefs and uses some form of expected utility maximisation. He also specifically considers the implications of having some overseer assign rewards based on beliefs about the state of the world (as opposed to having the reward come directly from the true world state).
 Discussion and Further Work We have seen that value-based reinforcement learning algorithms can fail to converge to any policy in some NDPs, and that when they do converge, they can only converge to ratifiable policies. Decision theorists have discussed whether ratifiability should be considered to be a sound normative principle. Note that (as discussed in Sect. 2) the optimal policies ⇡ are not in general ratifiable. We have also examined the limit action frequencies that agents can converge to (even when the policies do not converge). Still, there are NDPs in which many agents cannot converge even to any such frequency. We gave some results on what actions can be taken with positive limit frequency. A loose connection to ratifiability can still be drawn. Overall, established decision-theoretical ideas can be used to understand and formally describe the behaviour of "out-of-the-box" reinforcement learning agents in NDPs. However, their behaviour is not always desirable. Our work elucidates possible failures. We hope that our work will thus enable more accurate reasoning about the behaviour of RL agents in real-world situations, especially when interacting with humans or other agents. We hold such improvements in understanding to be broadly beneficial to the safe design and deployment of AI systems. Throughout the paper, we have noted specific open questions related to our results. For instance, can the results in Sect. 4.1 be generalised beyond the bandit setting? There are also many topics and questions about our setting that we have not touched on at all. For instance, our experimental results indicate that convergence is often slow (considering how simple the given problems are). It might be desirable to back up this impression with theoretical results. We have only studied simple value-based model-free algorithms -the analysis could be extended to other reinforcement learning algorithms (e.g., policy-gradient or model-based algorithms). Also, there are further ways in which we could generalise our setting. One example is to introduce partial observability and imperfect memory into the NDPs. The latter has been studied in game and decision theory  (Piccione and Rubinstein, 1997; Elga, 2000) , but recently -under the name memoryless POMDP -also in reinforcement learning  (Azizzadenesheli et al., 2016; Steckelmacher et al., 2018; cf. Conitzer, 2019) . What makes this especially appealing in the NDP context is that problems related to imperfect memory relate closely to Newcomblike problems  (Briggs, 2010; Schwarz, 2015) . One could also look for narrower classes of NDPs in which RL agents are guaranteed to perform well in some sense. Ultimately, the goal of this line of research is to develop learners that are able to deal effectively and safely with Newcomblike dynamics. We hope that our results will be useful in developing extensions of value-based RL that can detect and correct for the failures that arise when existing methods are applied in Newcomblike settings. However, we should also consider alternative approaches that do not hinge on insights from the present work. For example, a few recent papers (on learning in non-Newcomblike settings) have considered learning to predict the expected utility as a function of the policy (as opposed to traditional Q values, which are not paremeterised by the policy)  (Harb et al., 2020) . In principle, learning such a policy evaluation function avoids the problems of the learners considered in this paper. However, it remains to be seen how practical this approach is. Figure 2 : 2 Figure 2: The left figure plots the probability of softmax converging in Asymmetric Death in Damascus given n = n ↵ against ↵.More accurately it is a plot of the fraction of runs which assigned a Q-value of at least 5.5 to the action of going to Aleppo after 5000 iterations. These are empirical probabilities from 20,000 runs for every ↵ that is a multiple of 0.025, and 510,000 runs for each ↵ that is a multiple of 0.005 between 0.5 and 0.55. Notice the "kink" at ↵ = 0.5. Based on our experiments, this kink is not an artefact and shows up reliably in this kind of graph. The right-hand figure shows how the action probabilities evolve over time for a single run (chosen to converge to the mixed strategy) for ↵ = 0.3.
 Figure 3 : 3 Figure 3: This figure shows five runs of a softmax agent in LARPS, and plots ⇡(a rock ) against the total number of episodes played. The agent's Q-values are the historical mean rewards for each action, and t = 1/ log t.
 (a) This figure plots the dynamics of LARPS for an ✏-Greedy agent. Each point represents a triplet (fR, fS, fP), where fR denotes the fraction of past time steps at which aR was estimated to be the best action, and similarly for fS, fP. Plotted against this space is the expected direction in which the frequencies will change. For instance, if in the past aR, was mostly played, then aP will have the highest empirical Qvalues and will therefore be played more in the future.(b) This figure shows five runs of an ✏-Greedy agent in LARPS, and plots the proportion of past episodes in which the agent played "rock" against the total number of episodes played. The agent's Q-values are the historical mean rewards for each action, and its ✏-value is 0.01.
 Figure 4 4 Figure 4
			 35th Conference on Neural Information Processing Systems (NeurIPS 2021).
			 and 5) only assume that Q converges to q ⇡ (for some q ⇡ ) and therefore apply immediately to non-tabular agents, as long as the function approximator for q ⇡ converges to the same q ⇡ . 1 In most versions of Newcomb's Problem, the predictor directly predicts the agent's action with some fixed accuracy, and the agent is unable to randomise in a way that is unpredictable to the environment. This version of the problem can be modelled as a regular MDP. However, we believe that our version is more realistic in the context of AI. After all, AIs can at least act pseudo-randomly, while the distribution according to which they choose is predictable if e.g. their source code is known.2 For example, even if the environment has direct access to the source code of the agent, it may in general not be feasible to extract the exact action probabilities from the code. However, it is always possible to estimate the action probabilities by sampling. If this is done then T and R will depend continuously on the policy.