Accidental exploration through value predictors

1 INTRODUCTION . In practice all reinforcement learning environments produce only trajectories of finite length . In the case of computer simulations this is sometimes hard-coded in the environment , but even when it is not we have to introduce a cutoff to apply most interesting algorithms . Due to this cutoff we are forced to estimate the value function using the time-bounded Every-visit Monte Carlo estimator . As we will show this estimator is significantly biased . The time bound bias we will investigate stems directly from omitting the suffix of an infinite trajectory in the calculation of the value of a state . Because of that the influence of this bias is stronger in states that tend to appear close to the end of a trajectory . We use this asymmetry in experiments to investigate the bias by examining its influence on exploration for agents that use value predictors . 1.1 INFLUENCE ON EXPLORATION . To find the optimal solution an agent needs to explore , that is , visit a significant portion of the state space , even if locally the reward does not encourage this behaviour . Exploration plays a fundamental role in reinforcement learning , and ensuring that algorithms explore sufficiently remains a pressing research problem . Exploration can either be undirected or directed ( Thrun , 1992 ) . The former refers to methods like -greedy or softmax strategies ( Tijsma et al. , 2016 ) and its effectiveness is very limited . The latter attempts to implement some form of intrinsic motivation , usually based on success rate when attempting to predict future states , with methods ranging from visit counts ( Tang et al. , 2017 ) to prediction networks ( Anonymous , 2019 ) . To show the connection of the time bound bias with exploration consider the influence of the bias on a given state s. If , on average , the missing suffix of the total discounted reward in s is positive ( resp . negative ) , then the bias lowers ( resp . increases ) the learned value of s. The later the state s appears in trajectories , the stronger the bias . Thus , if in a region of the state space all the missing suffixes have similar magnitude and are positive ( resp . negative ) , then the bias discourages ( resp . encourages ) the agent to visit states that appear at later points in trajectories . It would seem natural that negative rewards should therefore encourage exploration – states that are , on average , visited later , usually lie in previously unvisited parts of the latent space . Reaching them quicker might allow the agent to go even further . Perhaps the simplest environment in which this is the case is a constant trajectory length variant of the MountainCar environment ( Moore , 1990 ) from the OpenAI gym ( Brockman et al. , 2016 ) . In this environment all rewards received before reaching the target – the single absorbing state – are equal to −1 , and zero afterwards . Exploration is a crucial component of solving the problem . We will also show that positive rewards strongly discourage exploration and in some cases the agent simply learns to stay in proximity to the starting state . However , negative rewards are not a solution to the exploration problem – we will construct an environment where rewards are all equal to −1 , but the preference on states introduced by the bias actually discourages exploration . We also propose some ways of mitigating the bias in general . 1.2 RELATED WORK . This paper investigates issues related to the problem of learning good policies for general Markov Decision Processes with access to only finite experience . A general algorithm for finding a nearoptimal solution of a MDP in polynomial time has been given in Kearns & Singh ( 2002 ) . This algorithm is model-based , so it can generate constant length simulations of the environment starting at any state . In our model-free setting , in contrast , the length of future experience available for a given state depends on the time it is visited . As described in the previous section this introduces some undesirable behaviour when using the Every-visit Monte Carlo estimator and our goal is to analyse these issues . 2 THE TIME-BOUND BIAS . We will use the following notation . When an agent using policy π experiences a trajectory ( s1 , a1 , r1 , s2 , . . . ) ∼ π the total discounted reward is Gt : = ∞∑ k=t γk−trk . The value function is defined as Vπ ( s ) : = Eπ [ Gt | st = s ] . ( 1 ) In the Every-visit Monte Carlo estimator the expected value in equation 1 is replaced by an average over all occurrences of the state s within N sampled trajectories . Denoting the number of times the state s is visited in the j-th trajectory by Mj we have : V̂π , N ( s ) : = ∑N j=1 ∑ t : sj , t=s Gj , t∑N j=1Mj . ( 2 ) The average is ill defined for states that occur an infinite number of times in any of theN trajectories , but the common assumption is that the only such state is a single absorbing state . We assume that this state has constant reward ra and its discounted value is ra1−γ . In Sutton et al . ( 1998 ) the authors have shown that the Every-visit Monte Carlo estimator has a bias which tends to 0 as N tends to infinity . This is in contrast to the unbiased First-visit Monte Carlo estimator . In this paper we make the more realistic assumption that every trajectory ’ s length is equal to a constant value T . In such a case we use the time-bounded version of the total discounted reward : GTt : = T∑ k=t γk−trk . The time-bounded Every-visit Monte Carlo estimator is therefore V̂ Tπ , N ( s ) : = ∑N j=1 ∑ t : sj , t=s GTj , t∑N j=1M T j , ( 3 ) where MTj counts the occurrences of state s in the truncated trajectory . Finally , we define the time bound bias as the difference between the expected value of the timebounded Every-visit Monte Carlo estimator and the value function in equation 1 T-biasπ , N ( s ) : = Eπ [ V̂ Tπ , N ( s ) ] − Vπ ( s ) . ( 4 ) Unlike the case of the unbounded Every-visit Monte Carlo estimator this bias does not disappear when N tends to infinity . To show the reason let us define pπ ( st = s ) as the probability that for a random trajectory sampled using policy π the t-th state is equal to s. Then , by dividing the numerator and denominator of equation equation 3 by N and changing the order of summation we get V̂ Tπ , N ( s ) = ∑T t=1 1 N ∑ j : sj , t=s GTj , t 1 N ∑N j=1M T j . By plugging this into equation equation 4 and taking the limit with respect to N , we define T-biasπ ( s ) : = lim N→∞ T-biasπ , N ( s ) = ∑T t=1 p ( st = s ) Eπ [ GTt | st = s ] ∑T t=1 p ( st = s ) − Vπ ( s ) . ( 5 ) The last part of equation equation 5 would be equal to 0 if we replaced all Eπ [ GTt | st = s ] with Eπ [ Gt | st = s ] . Therefore this bias stems directly from the missing suffixes of the trajectories . We can reach similar conclusions for the First-visit Monte Carlo estimator – both MTj and pπ ( st = s ) would have to be redefined to be the number of first visits and probability that st is the first visit respectively . This estimator will also from suffer from a variant of the time bound bias , as the problem with missing suffixes of trajectories persists . In this paper we focus only on the Every-visit variant , as it is more popular . Finally , let us define V Tπ ( s ) : = ∑T t=1 p ( st = s ) Eπ [ GTt | st = s ] ∑T t=1 p ( st = s ) , ( 6 ) so that we simply have T-biasπ ( s ) = V Tπ ( s ) − Vπ ( s ) . If we try to learn the value function , we encounter three obstacles – the time bound bias , the bias described in ( Sutton et al. , 1998 ) , and limitations in how well the model learns . In our paper we assume the model succeeds at approximating the value estimator . We limit the approximation error by using simple environments with low dimensional state spaces and training the value predictor until it converges . We ensure the second bias is low by using a large N . Thus , the effects we describe are mainly caused by T-biasπ . 3 RANDOM WALK . In this section we investigate a simple theoretical model , namely a time-bounded one-dimensional random walk with constant reward r. Let s ∈ { ⊥ } ∪ ( [ −B , B ] ∩ Z ) be the state , and t ∈ { 1 , . . . , T } – the time . The agent ’ s policy π is constant and defined as ( p−1 , p0 , p+1 ) , where pa is the probability of moving from state s to state s+a . Attempting to visit the integersB+1 or−B−1 causes instant transmission to the absorbing state ⊥ . Then the agent receives an infinite stream of rewards equal to r. In the case of an infinite random walk the value function is constant and equal to Vπ ( s ) = r1−γ . A T -step random walk illustrates how an agent experiences truncated trajectories of the infinite case . We will analyse the function V Tπ , which here is equal to the T-biasπ up to a constant factor . The walk starts at s = 0 , thus |s| ≤ T . We also assume that T < B . As a consequence , the time-bounded agent will never reach the boundaries or the absorbing state , so we do not have to treat them in any special way in the calculations below . We define the probability distribution pπ ( s , t ) to be the probability of the state being s and the time being t. We may assume that the marginal distribution pπ ( t ) is uniform and equal to 1T – this is convenient , and consistent with the definition of V Tπ if we say that pπ ( st = s ) = pπ ( s | t ) pπ ( s , t ) = pπ ( s | t ) · pπ ( t ) = pπ ( s | t ) T . It is worth noting that in our experiments this distribution can be interpreted as the distribution of data from the point of view of the value predictor . In this case the marginal distribution on time is also uniform , but only because we use Every-visit Monte Carlo as our value estimator . We can calculate pπ ( s , t ) using the recurrent formula for pπ ( s | t ) : pπ ( s | t = 1 ) = { 1 if s = 0 , 0 otherwise , pπ ( s | t = k + 1 ) = p+1 · pπ ( s− 1 | t = k ) + p0 · pπ ( s | t = k ) + p−1 · pπ ( s+ 1 | t = k ) , and now we have : GTt = r γT−t+1 − 1 γ − 1 , V Tπ ( s ) = ∑T t=1 pπ ( s , t ) ·GTt∑T t=1 pπ ( s , t ) . For examples of the value function with r = 1 see Figure 1 . As it turns out , the value function is an even function if only p−1 , p+1 > 0 . To see this note that every partial trajectory that ends at time t in state s can be bijectively coupled with a mirror partial trajectory , where every step right is replaced with step left and step left replaced with step right . Thus if p−1 , p+1 > 0 , then p ( s , t ) = ( p+1 p−1 ) sp ( −s , t ) , and we get V Tπ ( −s ) = ∑T t=1 pπ ( −s , t ) ·GTt∑T t=1 pπ ( −s , t ) = ∑T t=1 ( p+1 p−1 ) −spπ ( s , t ) ·GTt∑T t=1 ( p+1 p−1 ) −spπ ( s , t ) = ∑T t=1 pπ ( s , t ) ·GTt∑T t=1 pπ ( s , t ) = V Tπ ( s ) . The value function is linear with respect to the constant reward r. For r = 1 we will observe that the value function has a sharp peak at 0 . The optimal policy that maximises the average value of a visited state is to always stay at 0 . Thus , generalised policy iteration will be strongly biased towards this particular policy . Learning using an advantage function instead of the reward will have a similar effect , as in the case of constant reward the preference stems only from the value function . If we set r = −1 the effects will be symmetric and in turn encourage moving away from the beginning of the walk . In Appendix A we perform a detailed analysis of the even simpler case of the Wiener process with no discount factor . This might provide the reader with a better intuition regarding the bias , as it is a good approximation of the discrete random walk .

Summary: I have read the authors’ response and the other reviews.  While the authors have made some improvements, my core criticism remains – the paper does not produce concrete theoretical or empirical results that definitively address the problems described.  In addition, there are many confusing statements throughout the paper.  For instance, the discussion of positive and negative rewards in the introduction does not conform with the rest of the literature on exploration in RL.