Adapting Behaviour for Learning Progress

1 INTRODUCTION . Reinforcement learning ( RL ) is a general formalism modelling sequential decision making . It aspires to be broadly applicable , making minimal assumptions about the task at hand and reducing the need for prior knowledge . By learning behaviour from scratch , it has the potential to surpass human expertise or tackle complex domains where human intuition is not applicable . In practice , however , generality is often traded for performance and efficiency , with RL practitioners tuning algorithms , architectures and hyper-parameters to the task at hand ( Hessel et al. , 2019 ) . A side-effect of this is that the resulting methods can be brittle , or difficult to reliably reproduce ( Nagarajan et al. , 2018 ) . Exploration is one of the main aspects commonly designed or tuned specifically for the task being solved . Previous work has shown that large sample-efficiency gains are possible , for example , when the exploratory behaviour ’ s level of stochasticity is adjusted to the environment ’ s hazard rate ( Garcı́a & Fernández , 2015 ) , or when an appropriate prior is used in large action spaces ( DulacArnold et al. , 2015 ; Czarnecki et al. , 2018 ; Vinyals et al. , 2019 ) . Ideal exploration in the presence of function approximation should be agent-centred . It ought to focus more on generating data that supports the learning of agent at its current parameters θ , rather than making progress on objective measurements of information gathering . A useful notion here is learning progress ( LP ) , defined as the improvement of the learned policy πθ ( Section 3 ) . The agent ’ s source of data is its behaviour policy . Beyond the conventional RL setting of a single stream of experience , distributed agents that interact with parallel copies of the environment can have multiple such data sources ( Horgan et al. , 2018 ) . In this paper , we restrict ourselves to the setting where all behaviour policies are derived from a single set of learned parameters θ , for example when θ parameterises an action-value function Qθ . Consequently the behaviour policies are given by π ( Qθ , z ) , where each modulation z leads to meaningfully different behaviour . This can be guaranteed if z is semantic ( e.g . degree of stochasticity ) and consistent across multiple time-steps . The latter is achieved by holding z fixed throughout each episode ( Section 2 ) . We propose to estimate a proxy that is indicative of future learning progress , f ( z ) ( Section 3 ) , separately for each modulation z , and to adapt the distribution over modulations to maximize f , using a non-stationary multi-armed bandit that can exploit the factored structure of the modulations ( Section 4 ) . Figure 1 shows a diagram of all these components . This results in an autonomous adaptation of behaviour to the agent ’ s stage of learning ( Section 5 ) , varying across tasks and across time , and reducing the need for hyper-parameter tuning . 2 MODULATED BEHAVIOUR . As usual in RL , the objective of the agent is to find a policy π that maximises the γ-discounted expected return Gt . = ∑∞ i=0 γ iRt+i , where Rt is the reward obtained during the transition from time t to t + 1 . A common way to address this problem is to use methods that compute the actionvalue function Qπ given by Qπ ( s , a ) .= E [ Gt|s , a ] , i.e . the expected return when starting from state s with action a and then following π ( Puterman , 1994 ) . A richer representation of Qπ that aims to capture more information about the underlying distribution of Gt has been proposed by Bellemare et al . ( 2017 ) , and extended by Dabney et al . ( 2018 ) . Instead of approximating only the mean of the return distribution , we approximate a discrete set of n quantile values qν ( where ν ∈ { 12n , 3 2n , . . . , 2n−1 2n } ) such that P ( Q π ≤ qν ) = ν . Outside the benefits in performance and representation learning ( Such et al. , 2018 ) , these quantile estimates provide a way of inducing risk-sensitive behaviour . We approximate all qν using a single deep neural network with parameters θ , and define the evaluation policy as the greedy one with respect to the mean estimate : πθ ( ·|s ) ∈ arg max a 1 n ∑ ν qν ( s , a ) . The behaviour policy is the central element of exploration : it generates exploratory behaviour ( and experience therefrom ) which is used to learn πθ ; ideally in such a way as to reduce the total amount of experience required to achieve good performance . Instead of a single monolithic behaviour policy , we propose to use a modulated policy to support parameterized variation . Its modulations z should satisfy the following criteria : they need to ( i ) be impactful , having a direct and meaningful effect on generated behaviour ; ( ii ) have small dimensionality , as to quickly adapt to the needs of the learning algorithm , and interpretable semantics to ease the choice of viable ranges and initialisation ; and ( iii ) be frugal , in the sense that they are relatively simple and computationally inexpensive to apply . In this work , we consider five concrete types of such modulations : Temperature : a Boltzmann softmax policy based on action-logits , modulated by temperature , T . Flat stochasticity : with probability the agent ignores the action distribution produced by the softmax , and samples an action uniformly at random ( -greedy ) . Per-action biases : action-logit offsets , b , to bias the agent to prefer some actions . Action-repeat probability : with probability ρ , the previous action is repeated ( Machado et al. , 2017 ) . This produces chains of repeated actions with expected length 11−ρ . Optimism : as the value function is represented by quantiles qν , the aggregate estimate Qω can be parameterised by an optimism exponent ω , such that ω = 0 recovers the default flat average , while positive values of ω imply optimism and negative ones pessimism . When near risk-neutral , our simple risk measure produces qualitatively similar transforms to those of Wang ( 2000 ) . We combine the above modulations to produce the overall z-modulated policy π ( a|s , z ) .= ( 1− ) ( 1− ρ ) e 1 T ( Qω ( s , a ) +ba ) ∑ a′∈A e 1 T ( Qω ( s , a ′ ) +ba′ ) + ( 1− ρ ) |A| + ρIa=at−1 , where z .= ( T , , b , ρ , ω ) , Ix is the indicator function , and the optimism-aggregated value is Qω . = ∑ ν e −ωνqν∑ ν e −ων . Now that the behaviour policy can be modulated , the following two sections discuss the criteria and mechanisms for choosing modulations z . 3 EXPLORATION & THE EFFECTIVE ACQUISITION OF INFORMATION . A key component of a successful reinforcement learning algorithm is the ability to acquire experience ( information ) that allows it to make expeditious progress towards its objective of learning to act in the environment in such a way as to optimise returns over the relevant ( potentially discounted ) horizon . The types of experience that most benefit an agent ’ s ultimate performance may differ qualitatively throughout the course of learning — a behaviour modulation that is beneficial in the beginning of training often enough does not carry over to the end , as illustrated by the analysis in Figure 5 . However , this analysis was conducted in hindsight , and in general how to generate such experience optimally — optimal exploration in any environment — remains an open problem . One approach is to require exploration to be in service of the agent ’ s future learning progress ( LP ) , and to optimise this quantity during learning . Although there are multiple ways of defining learning progress , in this work we opted for a task-related measure , namely the improvement of the policy in terms of expected return . This choice of measure corresponds to the local steepness of the learning curve of the evaluation policy πθ , LPt ( ∆θ ) . = E s0 [ V πθt+∆θ ( s0 ) − V πθt ( s0 ) ] , ( 1 ) where the expectation is over start states s0 , the value V π ( s ) = Eπ [ ∑ γiRi|s0 = s ] is the γdiscounted return one would expected to obtain , starting in state s and following policy π afterwards , and ∆θ is the change in the agent ’ s parameters . Note that this is still a limited criterion , as it is myopic and might be prone to local optima . As prefaced in the last section , our goal here is to define a mechanism that can switch between different behaviour modulations depending on which of them seems most promising at this point in the training process . Thus in order to adapt the distribution over modulations z , we want to assess the expected LP when learning from data generated according to z-modulated behaviour : LPt ( z ) . = E τ∼πθt ( z ) [ LPt ( ∆θ ( τ , t ) ) ] , with ∆θ ( τ , t ) the weight-change of learning from trajectory τ at time t. This is a subjective utility measure , quantifying how useful τ is for a particular learning algorithm , at this stage in training . Proxies for learning progress : WhileLP ( z ) is a simple and clear progress metric , it is not readily available during training , so that in practice , a proxy fitness ft ( z ) ≈ LPt ( z ) needs to be used . A key practical challenge is to construct ft from inexpensively measurable proxies , in a way that is sufficiently informative to effectively adapt the distribution over z , while being robust to noise , approximation error , state distribution shift and mismatch between the proxies and learning progress . The ideal choice of f ( z ) is a matter of empirical study , and this paper only scratches the surface on this topic . After some initial experimentation , we opted for the simple proxy of empirical ( undiscounted ) episodic return : ft ( z ) = ∑ ai∼π ( Qθt , z ) Ri . This is trivial to estimate , but it departs from LP ( z ) in a number of ways . First , it does not contain learner-subjective information , but this is partly mitigated through the joint use of with prioritized replay ( see Section 5.1 ) that over-samples high error experience . Another potential mechanism by which the episodic return can be indicative of future learning is because an improved policy tends to be preceded by some higher-return episodes – in general , there is a lag between best-seen performance and reliably reproducing it . Second , the fitness is based on absolute returns not differences in returns as suggested by Equation 1 ; this makes no difference to the relative orderings of z ( and the resulting probabilities induced by the bandit ) , but it has the benefit that the non-stationarity takes a different form : a difference-based metric will appear stationary if the policy performance keeps increasing at a steady rate , but such a policy must be changing significantly to achieve that progress , and therefore the selection mechanism should keep revisiting other modulations . In contrast , our absolute fitness naturally has this effect when paired with a non-stationary bandit , as described in the next section .

Summary: This paper develops a multi-arm bandit-based algorithm to dynamically adapt the exploration policy for reinforcement learning. The arms of the bandit are parameters of the policy such as exploration noise, per-action biases etc. A proxy fitness metric is defined that measures the return of the trajectories upon perturbations of the policy z; the bandit then samples perturbations z that are better than the average fitness of the past few perturbations.