Reinforcement Learning in Newcomblike Environments

1 Introduction . In this paper , we study decision scenarios in which outcomes depend not only on the choices made and physically implemented , but also depend directly on the agent ’ s policy . As an example , consider an autonomous vehicle ( AV ) whose goal it is to arrive at target destinations quickly while minimising the probability of collisions . In practice , AVs are careful drivers . It is easy to imagine an experiment ( or learning process ) that might support careful driving : on each day , let the AV decide at random between a careful and a more aggressive style of driving ; other drivers on the road are unaware of today ’ s chosen driving style and therefore behave the same around the AV on both types of days . Presumably the decrease in accidents on careful days outweighs the increase in travel time . However , imagine now that a type of AV was widely deployed . Then many of the drivers with whom the AVs interact on the road would know a lot about how these AVs behave ( e.g. , from reading about AVs , or from having interacted with other AVs of the same type in the past ) . In particular , if the other drivers know that the AVs rarely take risks , they might ( whether rationally or irrationally ) cut them off more , not give them right of way , etc . relative to the above experiment . Indeed , this phenomenon – human drivers bullying timid AVs – has been reported in the real world ( Condliffe , 2016 ; Liu et al. , 2020 ; cf . Cooper et al. , 2019 ) . As a result , the travel times of an AV are much longer if it always 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . follows ( and is known to follow ) a careful driving policy . Moreover , some of the safety benefits of careful choices disappear if the AV adopts a careful policy . To comfortably model situations such as this one , we introduce Newcomblike decision processes ( NDPs ) . The name derives from Newcomb ’ s problem ( Nozick , 1969 ) , described in the next subsection , and similar problems that have been studied in the decision-theoretic literature . NDPs are a generalisation of Markov decision processes wherein the transition probabilities and rewards depend not only on the agent ’ s action , but also directly on the agent ’ s policy . Thus , for example , how aggressively other cars move depends on the timidness of the AV ’ s policy . We describe NDPs in more detail in Sect . 1.1 . Importantly , the NDP model does not assume other agents in the environment to respond rationally to the agent ’ s policy . Thus , some NDPs can not comfortably be modelled as games . ( See Sect . 5.1 for a more detailed discussion of the relation between NDPs and game-theoretic models . ) We believe that Newcomblike dynamics are commonplace when AI systems interact with other ( human or artificial ) agents ( Cavalcanti , 2010 , Sect . 5 ; Oesterheld , 2019 , Sect . 1 ; Conitzer , 2019 ) . The deployment of AVs is rife with such dynamics . Besides aggressiveness , it might matter whether a policy is simple and thus predictable to humans , for instance . Another real-world scenario is that of recommendation systems : most readers of this paper have some idea of how these systems work and make choices based on it . ( “ I would like to watch this cat video , but if I do , my recommendations will soon be full of them. ” ) Thus , the success of a particular recommendation depends not only on the recommendation itself , but also on the recommendation system ’ s policy . We are interested in learning to play NDPs . More specifically , we study the behaviour of value-based , model-free RL agents , who maintain a Q-function that assigns values to state–action pairs . We define these in more detail in Sect . 1.2 . As we will see , such agents do not in general learn optimal policies in NDPs ( as they do in MDPs ) . Nevertheless , we believe that studying them is an important first step in developing practical learning algorithms for NDPs due to the combination of the following points . A ) For illustrative purposes , the examples we discuss throughout this paper are simple and emphasise the dependence of the environment on the policy . However , we think that most real-world scenarios are only partially Newcomblike . For example , most of the AV ’ s environment changes only in response to an AV ’ s actions and does not directly depend on the AV ’ s policy . B ) Value-based reinforcement learning algorithms are very well developed . In contrast , we would have to develop specialised learning algorithms for general NDPs from scratch . C ) As our results will show , in some situations and when set up correctly ( e.g. , in terms of learning rates ) value-based learning converges to optimal policies , or at least to reasonable policies , even under Newcomblike dynamics . For example , in a game of rock-paper-scissors against an opponent who knows the agent ’ s policy , some value-based learning agents learn the optimal policy of mixing uniformly . In light of A–C , we think that the most realistic paths to developing learning algorithms for real-world scenarios with Newcomblike dynamics will involve value-based RL . Specifically , one avenue toward realistic algorithms is to develop extensions of value-based RL and can detect and correct failures that might arise in Newcomblike dynamics . For that , we need to understand how value-based RL behaves in NDPs . Contributions In Sect . 2 we demonstrate that value-based RL algorithms can only converge to a policy that is ratifiable – that is , to a policy ⇡ for which all actions taken by ⇡ have optimal expected reward when following ⇡ . In Sect . 3 , we discuss the convergence properties of agents in Newcomblike situations , and show that there are cases where value-based agents must fail to converge . The action frequencies might converge , even when the policies do not . In Sect . 4 , we establish some conditions on any action frequency that an agent could converge to . We also show that there are decision problems and agents where even the action frequencies do not converge . 1.1 Newcomblike Decision Processes . A Newcomblike decision process ( NDP ) is a tuple hS , A , T , R , i where S is a finite set of states ; A is a finite set of actions ; T : S ⇥ A ⇥ ( S A ) S is a nondeterministic transition function ; R : S ⇥ A ⇥ S ⇥ ( S A ) R is a nondeterministic reward function , which we assume to be bounded ; and 2 [ 0 , 1 ) is a discount factor . A policy ⇡ : S A is a function that nondeterministically maps states to actions . We use ⇡ ( a | s ) to denote the probability of taking action a in state s while following the policy ⇡ . T and R are functions from states , actions , and policies . In other words , they allow the outcome of a decision to depend on the distributions from which the agent draws its actions , rather than just the state and the action that is in fact taken . Also note that T ( s , a , ⇡ ) and R ( s , a , s0 , ⇡ ) are defined even if ⇡ ( a | s ) = 0 . We say that an NDP is a bandit NDP if it has only one state . We will sometimes use R ( s , a , ⇡ ) as a shorthand for R ( s , a , T ( s , a , ⇡ ) , ⇡ ) , and we will sometimes omit the state from T , R , and ⇡ for bandit NDPs . Moreover , we normally let = 0 for bandit NDPs . Consider a distribution over initial states for an agent , and let ⇡ be its policy , let xt be the sequence of states it visits and at the sequence of actions it takes . We say ⇡ is optimal for that distribution if it maximises E [ P1 i=0 i R ( xi , ai , ai+1 , ⇡ ) ] . Note that unlike in the MDP case the optimal policy does depend on the initial distribution , however this isn ’ t relevant in the bandit case . As an example consider the eponymous Newcomb ’ s Problem . Newcomb ’ s Problem ( Nozick , 1969 ) : There are two boxes in front of you ; one opaque box , and one transparent box . You can see that the transparent box contains $ 1,000 . You can choose to either take only the opaque box , or to take both boxes . The boxes have been placed in this room by an agent who can predict your policy ; if he believes that you will take only the opaque box then he has put $ 1,000,000 in the opaque box , but if he believes that you will take both boxes then he has left the opaque box empty . Do you take one box , or two ? A version of Newcomb ’ s Problem can be formalised as the following bandit NDP : S = { s } , A = { a1 , a2 } , R ( a1 , ⇡ ) = ⇢ 0 w.p . ⇡ ( a2 ) 10 w.p . ⇡ ( a1 ) and R ( a2 , ⇡ ) = ⇢ 5 w.p . ⇡ ( a2 ) 15 w.p . ⇡ ( a1 ) , where “ w.p. ” is short for “ with probability ” . The key feature of this NDP is that , for any fixed policy , a2 ( “ two-boxing ” ) yields a higher reward than a1 ( “ one-boxing ” ) . But the expected reward of a policy increases in ⇡ ( a1 ) s.t . the optimal policy is to always play a1.1 We can view Newcomb ’ s problem as a simple version of the AV dynamic described in the introduction , where a2 is a driving action that allows other drivers to cut the AV off at no risk . We say that an NDP is continuous if T and R are continuous in the policy . In this paper we work mainly with continuous NDPs . This is in part because it is technically convenient , and in part because we believe that continuity is satisfied in many realistic cases.2 1.2 Reinforcement Learning Agents . We consider value-based reinforcement learning agents . Such agents have two main components ; a Q-function S ⇥ A ! R that predicts the expected future discounted reward conditional on taking a particular action in a particular state , and a bandit algorithm that is used to select actions in each state based on the Q-function . Given a policy ⇡ , we use q⇡ ( a | s ) to denote the ( true ) expected future discounted reward conditional on taking action a in state s while following the policy ⇡ ( and conditional on all subsequent actions being chosen by ⇡ ) . A model-free agent will update Q over time to make it converge to q⇡ when following ⇡ . If Q is represented as a lookup table , the agent is said to be tabular . If the state space is large , it is common to instead approximate q⇡ ( with e.g . a neural network ) . For simplicity , we focus mostly on tabular agents . However , some of our results ( Theorems 2 and 5 ) only assume that Q converges to q⇡ ( for some q⇡ ) and therefore apply immediately to non-tabular agents , as long as the function approximator for q⇡ converges to the same q⇡ . 1In most versions of Newcomb ’ s Problem , the predictor directly predicts the agent ’ s action with some fixed accuracy , and the agent is unable to randomise in a way that is unpredictable to the environment . This version of the problem can be modelled as a regular MDP . However , we believe that our version is more realistic in the context of AI . After all , AIs can at least act pseudo-randomly , while the distribution according to which they choose is predictable if e.g . their source code is known . 2For example , even if the environment has direct access to the source code of the agent , it may in general not be feasible to extract the exact action probabilities from the code . However , it is always possible to estimate the action probabilities by sampling . If this is done then T and R will depend continuously on the policy . The Q-values can be updated in different ways . One method is to use the update rule Qt+1 ( at | st ) ( 1 ↵t ( st , at ) ) Qt ( at | st ) + ↵t ( st , at ) ( rt + max a Qt ( a | st+1 ) ) , where at is the action taken at time t , st is the state visited at time t , rt is the reward obtained at time t , and ↵t ( s , a ) is a learning rate . This update rule is known as Q-learning ( Watkins , 1986 ) . Other widely used update rules include SARSA ( Rummery and Niranjan , 1994 ) and Expected SARSA ( van Seijen et al. , 2009 ) . For the purposes of this paper it will not matter significantly how the Q-values are computed , as long as it is the case that if an agent converges to a policy ⇡ in some NDP and explores infinitely often then Q converges to q⇡ . We will later see that this is the case for Q-learning , SARSA , and Expected SARSA in continuous NDPs . There are also several different bandit algorithms . Two types of agents that are widely used in practice and that we will refer to throughout the paper are softmax agents and ✏-Greedy agents . The policy of a softmax agent with a sequence of temperatures t 2 R+ is given by : ⇡t ( a | s ) = exp ( Qt ( a | s ) / t ) P a02A exp ( Qt ( a 0 | s ) / t ) . Unless otherwise stated we assume that t ! 0 . The policy of an ✏-Greedy agent with a sequence of exploration probabilities ✏t 2 [ 0 , 1 ] is ⇡t ( a | s ) = 1 ✏t if a = argmaxa0 Qt ( a0 | s ) and ⇡t ( a | s ) = ✏t/ ( |A| 1 ) otherwise . Unless otherwise stated we assume that ✏t ! 0 . We assume that ✏-Greedy breaks ties for argmax , so that there is always some a 2 A such that ⇡ ( a | s ) = 1 ✏t . We say that an agent is greedy in the limit if the probability that the agent takes an action that maximises Q converges to 1 , and we say that it explores infinitely often if it takes every action in every state infinitely many times .

Summary: This paper introduces Newcomblike Decision Problems (NDPs), an analog of MDPs in which the environment (specifically the transitions and reward) may depend on the agent’s _policy_ in addition to the state and action. They theoretically study the performance of standard RL algorithms when applied as is to continuous NDPs. Key results include:  1. Any RL algorithm that eventually finds accurate Q-values for its policy, and is greedy with respect to those Q-values, will converge to a strongly ratifiable policy, or will not converge at all. A strongly ratifiable policy is one that never “wishes” it could switch its action while keeping its “identity” the same. (The informal description is mine; the paper presents a formal statement.) SARSA, Expected SARSA, and Q-learning all satisfy the conditions for this theorem. 2. In any continuous NDP there exists a “strongly ratifiable” policy, but it may not be the optimal policy (for example, in the classic Newcomb problem). 3. Nonetheless, there exist continuous NDPs where typical RL algorithms almost surely do not converge to the strongly ratifiable policy. 4. When there is not infinite exploration of states and actions, RL algorithms instead satisfy weak ratifiability, where the policy never “wishes” it could switch its action to some other action that is explored infinitely often (but may “wish” to switch actions to an action that isn’t explored infinitely often). 5. In the special case of bandit NDPs, similar results can be proven for algorithms with limiting _action frequencies_ (even if there is no limiting policy).