Reinforcement Learning in Newcomblike Environments

1 Introduction . In this paper , we study decision scenarios in which outcomes depend not only on the choices made and physically implemented , but also depend directly on the agent ’ s policy . As an example , consider an autonomous vehicle ( AV ) whose goal it is to arrive at target destinations quickly while minimising the probability of collisions . In practice , AVs are careful drivers . It is easy to imagine an experiment ( or learning process ) that might support careful driving : on each day , let the AV decide at random between a careful and a more aggressive style of driving ; other drivers on the road are unaware of today ’ s chosen driving style and therefore behave the same around the AV on both types of days . Presumably the decrease in accidents on careful days outweighs the increase in travel time . However , imagine now that a type of AV was widely deployed . Then many of the drivers with whom the AVs interact on the road would know a lot about how these AVs behave ( e.g. , from reading about AVs , or from having interacted with other AVs of the same type in the past ) . In particular , if the other drivers know that the AVs rarely take risks , they might ( whether rationally or irrationally ) cut them off more , not give them right of way , etc . relative to the above experiment . Indeed , this phenomenon – human drivers bullying timid AVs – has been reported in the real world ( Condliffe , 2016 ; Liu et al. , 2020 ; cf . Cooper et al. , 2019 ) . As a result , the travel times of an AV are much longer if it always 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . follows ( and is known to follow ) a careful driving policy . Moreover , some of the safety benefits of careful choices disappear if the AV adopts a careful policy . To comfortably model situations such as this one , we introduce Newcomblike decision processes ( NDPs ) . The name derives from Newcomb ’ s problem ( Nozick , 1969 ) , described in the next subsection , and similar problems that have been studied in the decision-theoretic literature . NDPs are a generalisation of Markov decision processes wherein the transition probabilities and rewards depend not only on the agent ’ s action , but also directly on the agent ’ s policy . Thus , for example , how aggressively other cars move depends on the timidness of the AV ’ s policy . We describe NDPs in more detail in Sect . 1.1 . Importantly , the NDP model does not assume other agents in the environment to respond rationally to the agent ’ s policy . Thus , some NDPs can not comfortably be modelled as games . ( See Sect . 5.1 for a more detailed discussion of the relation between NDPs and game-theoretic models . ) We believe that Newcomblike dynamics are commonplace when AI systems interact with other ( human or artificial ) agents ( Cavalcanti , 2010 , Sect . 5 ; Oesterheld , 2019 , Sect . 1 ; Conitzer , 2019 ) . The deployment of AVs is rife with such dynamics . Besides aggressiveness , it might matter whether a policy is simple and thus predictable to humans , for instance . Another real-world scenario is that of recommendation systems : most readers of this paper have some idea of how these systems work and make choices based on it . ( “ I would like to watch this cat video , but if I do , my recommendations will soon be full of them. ” ) Thus , the success of a particular recommendation depends not only on the recommendation itself , but also on the recommendation system ’ s policy . We are interested in learning to play NDPs . More specifically , we study the behaviour of value-based , model-free RL agents , who maintain a Q-function that assigns values to state–action pairs . We define these in more detail in Sect . 1.2 . As we will see , such agents do not in general learn optimal policies in NDPs ( as they do in MDPs ) . Nevertheless , we believe that studying them is an important first step in developing practical learning algorithms for NDPs due to the combination of the following points . A ) For illustrative purposes , the examples we discuss throughout this paper are simple and emphasise the dependence of the environment on the policy . However , we think that most real-world scenarios are only partially Newcomblike . For example , most of the AV ’ s environment changes only in response to an AV ’ s actions and does not directly depend on the AV ’ s policy . B ) Value-based reinforcement learning algorithms are very well developed . In contrast , we would have to develop specialised learning algorithms for general NDPs from scratch . C ) As our results will show , in some situations and when set up correctly ( e.g. , in terms of learning rates ) value-based learning converges to optimal policies , or at least to reasonable policies , even under Newcomblike dynamics . For example , in a game of rock-paper-scissors against an opponent who knows the agent ’ s policy , some value-based learning agents learn the optimal policy of mixing uniformly . In light of A–C , we think that the most realistic paths to developing learning algorithms for real-world scenarios with Newcomblike dynamics will involve value-based RL . Specifically , one avenue toward realistic algorithms is to develop extensions of value-based RL and can detect and correct failures that might arise in Newcomblike dynamics . For that , we need to understand how value-based RL behaves in NDPs . Contributions In Sect . 2 we demonstrate that value-based RL algorithms can only converge to a policy that is ratifiable – that is , to a policy ⇡ for which all actions taken by ⇡ have optimal expected reward when following ⇡ . In Sect . 3 , we discuss the convergence properties of agents in Newcomblike situations , and show that there are cases where value-based agents must fail to converge . The action frequencies might converge , even when the policies do not . In Sect . 4 , we establish some conditions on any action frequency that an agent could converge to . We also show that there are decision problems and agents where even the action frequencies do not converge . 1.1 Newcomblike Decision Processes . A Newcomblike decision process ( NDP ) is a tuple hS , A , T , R , i where S is a finite set of states ; A is a finite set of actions ; T : S ⇥ A ⇥ ( S A ) S is a nondeterministic transition function ; R : S ⇥ A ⇥ S ⇥ ( S A ) R is a nondeterministic reward function , which we assume to be bounded ; and 2 [ 0 , 1 ) is a discount factor . A policy ⇡ : S A is a function that nondeterministically maps states to actions . We use ⇡ ( a | s ) to denote the probability of taking action a in state s while following the policy ⇡ . T and R are functions from states , actions , and policies . In other words , they allow the outcome of a decision to depend on the distributions from which the agent draws its actions , rather than just the state and the action that is in fact taken . Also note that T ( s , a , ⇡ ) and R ( s , a , s0 , ⇡ ) are defined even if ⇡ ( a | s ) = 0 . We say that an NDP is a bandit NDP if it has only one state . We will sometimes use R ( s , a , ⇡ ) as a shorthand for R ( s , a , T ( s , a , ⇡ ) , ⇡ ) , and we will sometimes omit the state from T , R , and ⇡ for bandit NDPs . Moreover , we normally let = 0 for bandit NDPs . Consider a distribution over initial states for an agent , and let ⇡ be its policy , let xt be the sequence of states it visits and at the sequence of actions it takes . We say ⇡ is optimal for that distribution if it maximises E [ P1 i=0 i R ( xi , ai , ai+1 , ⇡ ) ] . Note that unlike in the MDP case the optimal policy does depend on the initial distribution , however this isn ’ t relevant in the bandit case . As an example consider the eponymous Newcomb ’ s Problem . Newcomb ’ s Problem ( Nozick , 1969 ) : There are two boxes in front of you ; one opaque box , and one transparent box . You can see that the transparent box contains $ 1,000 . You can choose to either take only the opaque box , or to take both boxes . The boxes have been placed in this room by an agent who can predict your policy ; if he believes that you will take only the opaque box then he has put $ 1,000,000 in the opaque box , but if he believes that you will take both boxes then he has left the opaque box empty . Do you take one box , or two ? A version of Newcomb ’ s Problem can be formalised as the following bandit NDP : S = { s } , A = { a1 , a2 } , R ( a1 , ⇡ ) = ⇢ 0 w.p . ⇡ ( a2 ) 10 w.p . ⇡ ( a1 ) and R ( a2 , ⇡ ) = ⇢ 5 w.p . ⇡ ( a2 ) 15 w.p . ⇡ ( a1 ) , where “ w.p. ” is short for “ with probability ” . The key feature of this NDP is that , for any fixed policy , a2 ( “ two-boxing ” ) yields a higher reward than a1 ( “ one-boxing ” ) . But the expected reward of a policy increases in ⇡ ( a1 ) s.t . the optimal policy is to always play a1.1 We can view Newcomb ’ s problem as a simple version of the AV dynamic described in the introduction , where a2 is a driving action that allows other drivers to cut the AV off at no risk . We say that an NDP is continuous if T and R are continuous in the policy . In this paper we work mainly with continuous NDPs . This is in part because it is technically convenient , and in part because we believe that continuity is satisfied in many realistic cases.2 1.2 Reinforcement Learning Agents . We consider value-based reinforcement learning agents . Such agents have two main components ; a Q-function S ⇥ A ! R that predicts the expected future discounted reward conditional on taking a particular action in a particular state , and a bandit algorithm that is used to select actions in each state based on the Q-function . Given a policy ⇡ , we use q⇡ ( a | s ) to denote the ( true ) expected future discounted reward conditional on taking action a in state s while following the policy ⇡ ( and conditional on all subsequent actions being chosen by ⇡ ) . A model-free agent will update Q over time to make it converge to q⇡ when following ⇡ . If Q is represented as a lookup table , the agent is said to be tabular . If the state space is large , it is common to instead approximate q⇡ ( with e.g . a neural network ) . For simplicity , we focus mostly on tabular agents . However , some of our results ( Theorems 2 and 5 ) only assume that Q converges to q⇡ ( for some q⇡ ) and therefore apply immediately to non-tabular agents , as long as the function approximator for q⇡ converges to the same q⇡ . 1In most versions of Newcomb ’ s Problem , the predictor directly predicts the agent ’ s action with some fixed accuracy , and the agent is unable to randomise in a way that is unpredictable to the environment . This version of the problem can be modelled as a regular MDP . However , we believe that our version is more realistic in the context of AI . After all , AIs can at least act pseudo-randomly , while the distribution according to which they choose is predictable if e.g . their source code is known . 2For example , even if the environment has direct access to the source code of the agent , it may in general not be feasible to extract the exact action probabilities from the code . However , it is always possible to estimate the action probabilities by sampling . If this is done then T and R will depend continuously on the policy . The Q-values can be updated in different ways . One method is to use the update rule Qt+1 ( at | st ) ( 1 ↵t ( st , at ) ) Qt ( at | st ) + ↵t ( st , at ) ( rt + max a Qt ( a | st+1 ) ) , where at is the action taken at time t , st is the state visited at time t , rt is the reward obtained at time t , and ↵t ( s , a ) is a learning rate . This update rule is known as Q-learning ( Watkins , 1986 ) . Other widely used update rules include SARSA ( Rummery and Niranjan , 1994 ) and Expected SARSA ( van Seijen et al. , 2009 ) . For the purposes of this paper it will not matter significantly how the Q-values are computed , as long as it is the case that if an agent converges to a policy ⇡ in some NDP and explores infinitely often then Q converges to q⇡ . We will later see that this is the case for Q-learning , SARSA , and Expected SARSA in continuous NDPs . There are also several different bandit algorithms . Two types of agents that are widely used in practice and that we will refer to throughout the paper are softmax agents and ✏-Greedy agents . The policy of a softmax agent with a sequence of temperatures t 2 R+ is given by : ⇡t ( a | s ) = exp ( Qt ( a | s ) / t ) P a02A exp ( Qt ( a 0 | s ) / t ) . Unless otherwise stated we assume that t ! 0 . The policy of an ✏-Greedy agent with a sequence of exploration probabilities ✏t 2 [ 0 , 1 ] is ⇡t ( a | s ) = 1 ✏t if a = argmaxa0 Qt ( a0 | s ) and ⇡t ( a | s ) = ✏t/ ( |A| 1 ) otherwise . Unless otherwise stated we assume that ✏t ! 0 . We assume that ✏-Greedy breaks ties for argmax , so that there is always some a 2 A such that ⇡ ( a | s ) = 1 ✏t . We say that an agent is greedy in the limit if the probability that the agent takes an action that maximises Q converges to 1 , and we say that it explores infinitely often if it takes every action in every state infinitely many times .

Summary: The paper begins by defining a Newcomblike decision process (NDP), a natural extension of MDP in which both the reward and/or dynamics are allowed to depend on the learner's policy. The main theoretical claims of the paper are largely three-fold: 1. define two notions of ratifiability (strong and weak). strong ratifiability can be seen as a policy that always selects actions among those that are optimal w.r.t. the q-values of the policy, while weak can be seen as q-values for a policy being constant among those actions sampled by the policy 2. show that q-values, when converge, must converge to a strong ratifiable policy; this can relaxed to weak ratifiability if q-values are accurate only for state-action pairs sampled infinitely often 3. with the help of several bandit examples, show that standard RL such as QL and SARSA is not required to converge at all.