Occupy & Specify: Investigations into a Maximum Credit Assignment Occupancy Objective for Data-efficient Reinforcement Learning

1 PROBLEM STATEMENT . Learning in the real world implies dealing with very large , potentially unlimited environments , over which the data to collect is seemingly infinite . Efficient exploration is thus one of the key aspects of open-ended learning Santucci et al . ( 2020 ) , when no final model of the environment can feasibly be expected to be engineered or trained . On the one side , having access to unlimited data is very beneficial for the training of complex multi-layered perceptrons , for they are known to rely on large datasets to improve their performance . On the other side , the circular dependence between the learning algorithm and the data on which it operates renders the learning very tricky , at high risk of data overfitting and trapping in local optima . The open-ended learning problem is generally addressed through the lens of the reinforcement learning framework ( Sutton et al. , 1998 ) , where rewards are collected during the interaction , and the selection of action is fit so as to maximize the total number of positive rewards , and prevent the encounter of negative ones . Fitting behaviour to rewards is however at the risk of ignoring important data from the rest of the environment , where putatively more rewarding regions may be neglected . The agreement of reward-seeking ( that is exploitation ) with data collecting ( that is exploration ) , is still one of the fundamental issues of modern artificial intelligence . An important effort has recently been put on reframing the reinforcement learning setup into a more general probabilistic inference framework , allowing to link rewards seeking and data modelling under a single perspective ( Furmston & Barber , 2010 ; Levine , 2018 ; Haarnoja et al. , 2018 ; Abdolmaleki et al. , 2018 ; Fellows et al. , 2019 ) . This greater focus over the data collection problem is linked to an important set of training algorithms , that contain some forms of exploration bonuses , including “ curiosity ” drives ( Schmidhuber et al. , 2009 ; Pathak et al. , 2017 ) , intrinsic rewards ( Oudeyer et al. , 2007 ) and pseudo-counts ( Bellemare et al. , 2016 ; Tang et al. , 2017 ) . However , at the difference of the classic optimization on rewards alone , where the Bellman optimum is well defined , there is still no consensus about the objective followed when optimizing both on rewards and data collection under a variational perspective ( Eysenbach & Levine , 2019 ) . The data collection problem is effectively shadowed by the reward maximization objective , under which it is still considered as a incidental component . An important body of work has recently been devoted to addressing the data collection problem as such , with the notable design of the MaxEnt algorithm ( Hazan et al. , 2019 ) , State Marginal Matching ( Lee et al. , 2019 ) and E3D ( Daucé , 2020 ) , that aim at fitting the distribution of the states encountered to a uniform distribution , in the absence of definite rewards . This is here referred as a MaxEnt-on-state principle ( or MaxEnt to be short ) , not to be confounded with the MaxEnt-on-actions principle implemented in the soft actor critic ( Haarnoja et al. , 2018 ) for instance . Following a MaxEnt objective means optimizing the policy so as the states visited are maximally variable , ideally uniformly visiting all possible states . We develop in the following a possible extension of the MaxEnt principle , that brings a considerable simplification in the expression of the evidence lower bound ( ELBO ) with regards with the existing literature ( Furmston & Barber , 2010 ; Abdolmaleki et al. , 2018 ; Fellows et al. , 2019 ) . In contrast to pure MaxEnt , our approach provides a way to combine the MaxEnt objective with a reward maximization objective , under a variational inference perspective . An intriguing property of the resulting ELBO formula is that the future states ( the ones that are visited after the current observation ) play the role of a model for the current data , participating in the elaboration of the returns collected under the current policy . This gives ways toward optimizing the policy with respect to the distribution of the data , and provides a principled justification to the use of intrinsic rewards in the design of reinforcement learning algorithms . 2 PRINCIPLES . 2.1 PROBABILITY MATCHING RL . We assume an agent acting in a fully observable environment . The state of the environment is provided by an observation s ∈ S , with S the set of all possible states . The agent can act on the environment through its actuators . Such a motor command is described by a ∈ A , with A the set of all motor commands . In the following , the capital letters S and A will reflect random variables on S andA , and the lower cases s and a will either reflect observations or random draw realizations . The decision of which action to choose relies on a policy , that maps the current observation to the action space , generally expressed in a conditional probabilistic form π ( a|s ) . A reinforcement learning problem consists in finding a policy π∗ that maximizes a certain objective function , without knowing the physical or mechanical properties of the environment . It is supposed here , for simplicity , that the dynamics of the environment is Markovian ( no hidden states ) . Moreover , the environment is providing an auxiliary signal called the reward . Sending an action to the environment makes it possible to access to a new state s′ , and to obtain a reward r ∈ R. A classic objective in learning is to maximize the global return , generally described as a discounted sum of future rewards over all possible trajectories . Let us now denote by st the state visited at time t and τ ( st ) = ( st+1 , ... , st+T , ... ) a certain pathway that is visited after observing st. During this visit , a certain number of rewards can be collected , and R ( τ ) is the ( discounted ) return obtained over τ , i.e . R ( τ ) = ∑ t γ trt , with γ ∈ [ 0 , 1 [ a discounting factor that sums up the rewards up to an “ horizon ” of the order of 11−γ . This said , the dynamic programming objective ( Bellman , 1966 ) , is the result of π∗ = maxΠ Es∼p ( S0 ) , τ∼pπ ( τ |s ) R ( τ ) , with p ( S0 ) the distribution of initial states , and Π the set of all conditional policies . When a state transition model p ( S′|s , a ) is provided , the unique solution is given by the dynamic programming recurrent equation in the discrete case ( Bellman , 1966 ) . On the contrary , a large panel of reinforcement learning techniques allow to approach the solution in the model-free setup , assuming an effective sampling of all state-action pairs ( Sutton et al. , 1998 ) . We are here interested in a different class of objective function , that rely on fitting the rewards toward probabilities of state occupancies . A reward should indicate in which proportion the different states ( and actions ) should be visited ( and selected ) during trials ( with the idea that the states providing high return should be visited more often than the ones providing low returns ) . Solving the reinforcement learning problem then means to match the external cue to an actual distribution of visit over states and actions , where a differential in rewards only indicates a difference in the number of visits , allowing to seek rewards in a flexible way ( so it is also referred as to “ soft ” reinforcement learning ( Haarnoja et al. , 2018 ) ) . This idea stems back from empirical observations on human and animal behaviors , and was coined the “ matching law ” in the operant conditioning literature ( Herrnstein , 1961 ; Eysenbach & Levine , 2019 ) . 2.2 STATE OCCUPANCY AND CONDITIONAL STATE OCCUPANCY . Matching rewards to probabilities can be done in many different ways . We frame here the probability matching reinforcement learning problem into a state occupancy matching problem . It relies on the use of an occupancy distribution , that is a density of state visit under a certain policy . Importantly , it ignores the time order at which the different states are visited , still conserving some aspects of causality between states in the form of conditional probabilities as we see later . Dating back from Dayan ( 1993 ) , an occupancy distribution is a distribution on states , designed so as to match with the distribution measured over the trajectories of the MDP . Following the definitions of ( Puterman , 2014 ; Ho & Ermon , 2016 ; Hazan et al. , 2019 ) , a gammaabsorbing state occupancy of a Markov Decision process ( with a policy π ) is the ( discounted ) density of visit of the states — or ( state , action ) pairs — of the environment when starting from the initial distribution p ( S0 ) . It is defined , as : { ρπ ( s ) = ( 1− γ ) p0 ( s ) + γ ∑ s′ , a′ p ( s|s′ , a′ ) π ( a′|s′ ) ρπ ( s′ ) ρπ ( s , a ) = π ( a|s ) ρπ ( s ) ( 1 ) so that any policy π settled on an MDP defines an occupancy on the states of that MDP . It comes that , inversely , any valid ( state , action ) occupancy ( meaning that this occupancy is effectively feasible in a given agent/environment setup ) , defines a unique corresponding policy : π ( a|s ) = ρ ( s , a ) ρ ( s ) ( 2 ) that is a softmax ( stochastic ) conditional policy over the states . Following the same reasoning , let ρπ ( S+|s ... ) the conditional occupancy be defined recursively . Let Tπ ( s ) the set of trajectories starting from s : ∀s+ ∈ Tπ ( s ) , ρπ ( s+|s ... ) = pπ ( s+|s ) + γ ∑ s′∈Tπ ( s ) pπ ( s +|s′ ) ρπ ( s′|s ... ) The triple dots ( ... ) are intended to help distinguish the one-step distribution pπ ( S′|s ) from the longterm distribution ρπ ( S+|s ... ) . This conditional distribution provides a description of the “ future ” of s , that is the distribution of states that will most probably follow s. It can be seen as an instance of the “ successor ” representation of states initially proposed by Dayan ( 1993 ) . Those future states will generally be noted s+ , with the ’ + ’ exponent meaning the state being measured “ further away in time ” . 2.3 MATCHING REWARDS TO OCCUPANCIES . Those definitions provide a way toward interpreting rewards as occupancy templates , allowing to implement the “ matching law ” in a principled way . The mapping of rewards toward probabilities relies on using exponentiated returns in the parameters of a stochastic policy , such as in the softmax ( or Boltzmann ) decision rule case . Let π ( a|s ) = exp βQ ( s , a ) K ( s ) with K ( s ) = ∑ a expβQ ( s , a ) , with β the “ inverse temperature ” , and the state-action value Q ( s , a ) representing the total return estimated at ( s , a ) . Let τ = ( s0 , s1 , ... , st , ... ) a certain trajectory observed on the MDP under the policy π . The set of all possible trajectories is noted T , pπ ( τ ) is a measure over the trajectories for a certain policy π , and ρπ is the corresponding occupancy on states . Consider for instance the series of rewards encountered when following τ . It comes that : Es0∼p0V ( s0 ) = Eτ∼pπ ( T ) ∑ t γtr ( st , at ) ≈ E s∼ρπ ( S ) a∼π ( A|s ) r ( s , a ) ∑ t γt = Es , a∼ρπ ( S , A ) r ( s , a ) 1− γ so that ∀t , r ( st , at ) 1−γ is interpreted as an estimator of V ( s0 ) . Next , for any t , it comes that ∀t′ > t , r ( st′ , at′ ) 1−γ is an estimator of the state-action value Q ( st , at ) , i.e : Q ( st , at ) = Eτ∼pπ ( T ) st∈τ ∑ t′ > t γ ( t ′−t ) r ( st′ , at′ ) ≈ Es+∼ρπ ( S+|st , at ... ) a+∼π ( A|s+ ) r ( s+ , a+ ) 1− γ ( 3 ) In that setup , the rewards are interpreted as value samples . This means , in short , that each future reward r ( s+ , a+ ) takes the role of a “ model ” for the total returnQ ( st , at ) . The models are weighted according to the conditional occupancy ρπ ( S+ , A+|s , a ... ) , that takes the role of the “ mixture ” . Then , noting that log π ( a|s ) = βQ ( s , a ) −K ( s ) , we define : R̄ ( s , s+ , a+ ) , r ( s+ , a+ ) 1− γ − 1 β ( K ( s ) − log ρπ ( s ) ) ( 4 ) said the “ extended ” return composed of the return estimator plus a virtual baseline . Then , because the policy and the occupancy are exchangeable from eq . ( 2 ) , each reward collected after ( s , a ) may also take the role of a “ model ” for the occupancy , sampled from the conditional occupancy , i.e . : log ρπ ( s , a ) ≈ Es+∼ρπ ( S+|s , a ... ) a+∼π ( A|s+ ) βR̄ ( s , s+ , a+ ) ( 5 )

Summary: # Summary & Contributions * This paper examines a variational approach to reinforcement learning, leveraging occupancy measures over previously visited and future state-action pairs in order to address the exploration challenge. * The author propose a variational approximation to the so-called "conditional occupancy" of the current behavior policy denoting the distribution over future state-action pairs. The approximation is used to induce broader state visitation from the behavior policy, coupled with a standard actor-critic algorithm. * Empirical results show considerable performance gains in simpler continuous control problems and matching performance against baseline methods in more high-dimensional control tasks.