Occupy & Specify: Investigations into a Maximum Credit Assignment Occupancy Objective for Data-efficient Reinforcement Learning

1 PROBLEM STATEMENT . Learning in the real world implies dealing with very large , potentially unlimited environments , over which the data to collect is seemingly infinite . Efficient exploration is thus one of the key aspects of open-ended learning Santucci et al . ( 2020 ) , when no final model of the environment can feasibly be expected to be engineered or trained . On the one side , having access to unlimited data is very beneficial for the training of complex multi-layered perceptrons , for they are known to rely on large datasets to improve their performance . On the other side , the circular dependence between the learning algorithm and the data on which it operates renders the learning very tricky , at high risk of data overfitting and trapping in local optima . The open-ended learning problem is generally addressed through the lens of the reinforcement learning framework ( Sutton et al. , 1998 ) , where rewards are collected during the interaction , and the selection of action is fit so as to maximize the total number of positive rewards , and prevent the encounter of negative ones . Fitting behaviour to rewards is however at the risk of ignoring important data from the rest of the environment , where putatively more rewarding regions may be neglected . The agreement of reward-seeking ( that is exploitation ) with data collecting ( that is exploration ) , is still one of the fundamental issues of modern artificial intelligence . An important effort has recently been put on reframing the reinforcement learning setup into a more general probabilistic inference framework , allowing to link rewards seeking and data modelling under a single perspective ( Furmston & Barber , 2010 ; Levine , 2018 ; Haarnoja et al. , 2018 ; Abdolmaleki et al. , 2018 ; Fellows et al. , 2019 ) . This greater focus over the data collection problem is linked to an important set of training algorithms , that contain some forms of exploration bonuses , including “ curiosity ” drives ( Schmidhuber et al. , 2009 ; Pathak et al. , 2017 ) , intrinsic rewards ( Oudeyer et al. , 2007 ) and pseudo-counts ( Bellemare et al. , 2016 ; Tang et al. , 2017 ) . However , at the difference of the classic optimization on rewards alone , where the Bellman optimum is well defined , there is still no consensus about the objective followed when optimizing both on rewards and data collection under a variational perspective ( Eysenbach & Levine , 2019 ) . The data collection problem is effectively shadowed by the reward maximization objective , under which it is still considered as a incidental component . An important body of work has recently been devoted to addressing the data collection problem as such , with the notable design of the MaxEnt algorithm ( Hazan et al. , 2019 ) , State Marginal Matching ( Lee et al. , 2019 ) and E3D ( Daucé , 2020 ) , that aim at fitting the distribution of the states encountered to a uniform distribution , in the absence of definite rewards . This is here referred as a MaxEnt-on-state principle ( or MaxEnt to be short ) , not to be confounded with the MaxEnt-on-actions principle implemented in the soft actor critic ( Haarnoja et al. , 2018 ) for instance . Following a MaxEnt objective means optimizing the policy so as the states visited are maximally variable , ideally uniformly visiting all possible states . We develop in the following a possible extension of the MaxEnt principle , that brings a considerable simplification in the expression of the evidence lower bound ( ELBO ) with regards with the existing literature ( Furmston & Barber , 2010 ; Abdolmaleki et al. , 2018 ; Fellows et al. , 2019 ) . In contrast to pure MaxEnt , our approach provides a way to combine the MaxEnt objective with a reward maximization objective , under a variational inference perspective . An intriguing property of the resulting ELBO formula is that the future states ( the ones that are visited after the current observation ) play the role of a model for the current data , participating in the elaboration of the returns collected under the current policy . This gives ways toward optimizing the policy with respect to the distribution of the data , and provides a principled justification to the use of intrinsic rewards in the design of reinforcement learning algorithms . 2 PRINCIPLES . 2.1 PROBABILITY MATCHING RL . We assume an agent acting in a fully observable environment . The state of the environment is provided by an observation s ∈ S , with S the set of all possible states . The agent can act on the environment through its actuators . Such a motor command is described by a ∈ A , with A the set of all motor commands . In the following , the capital letters S and A will reflect random variables on S andA , and the lower cases s and a will either reflect observations or random draw realizations . The decision of which action to choose relies on a policy , that maps the current observation to the action space , generally expressed in a conditional probabilistic form π ( a|s ) . A reinforcement learning problem consists in finding a policy π∗ that maximizes a certain objective function , without knowing the physical or mechanical properties of the environment . It is supposed here , for simplicity , that the dynamics of the environment is Markovian ( no hidden states ) . Moreover , the environment is providing an auxiliary signal called the reward . Sending an action to the environment makes it possible to access to a new state s′ , and to obtain a reward r ∈ R. A classic objective in learning is to maximize the global return , generally described as a discounted sum of future rewards over all possible trajectories . Let us now denote by st the state visited at time t and τ ( st ) = ( st+1 , ... , st+T , ... ) a certain pathway that is visited after observing st. During this visit , a certain number of rewards can be collected , and R ( τ ) is the ( discounted ) return obtained over τ , i.e . R ( τ ) = ∑ t γ trt , with γ ∈ [ 0 , 1 [ a discounting factor that sums up the rewards up to an “ horizon ” of the order of 11−γ . This said , the dynamic programming objective ( Bellman , 1966 ) , is the result of π∗ = maxΠ Es∼p ( S0 ) , τ∼pπ ( τ |s ) R ( τ ) , with p ( S0 ) the distribution of initial states , and Π the set of all conditional policies . When a state transition model p ( S′|s , a ) is provided , the unique solution is given by the dynamic programming recurrent equation in the discrete case ( Bellman , 1966 ) . On the contrary , a large panel of reinforcement learning techniques allow to approach the solution in the model-free setup , assuming an effective sampling of all state-action pairs ( Sutton et al. , 1998 ) . We are here interested in a different class of objective function , that rely on fitting the rewards toward probabilities of state occupancies . A reward should indicate in which proportion the different states ( and actions ) should be visited ( and selected ) during trials ( with the idea that the states providing high return should be visited more often than the ones providing low returns ) . Solving the reinforcement learning problem then means to match the external cue to an actual distribution of visit over states and actions , where a differential in rewards only indicates a difference in the number of visits , allowing to seek rewards in a flexible way ( so it is also referred as to “ soft ” reinforcement learning ( Haarnoja et al. , 2018 ) ) . This idea stems back from empirical observations on human and animal behaviors , and was coined the “ matching law ” in the operant conditioning literature ( Herrnstein , 1961 ; Eysenbach & Levine , 2019 ) . 2.2 STATE OCCUPANCY AND CONDITIONAL STATE OCCUPANCY . Matching rewards to probabilities can be done in many different ways . We frame here the probability matching reinforcement learning problem into a state occupancy matching problem . It relies on the use of an occupancy distribution , that is a density of state visit under a certain policy . Importantly , it ignores the time order at which the different states are visited , still conserving some aspects of causality between states in the form of conditional probabilities as we see later . Dating back from Dayan ( 1993 ) , an occupancy distribution is a distribution on states , designed so as to match with the distribution measured over the trajectories of the MDP . Following the definitions of ( Puterman , 2014 ; Ho & Ermon , 2016 ; Hazan et al. , 2019 ) , a gammaabsorbing state occupancy of a Markov Decision process ( with a policy π ) is the ( discounted ) density of visit of the states — or ( state , action ) pairs — of the environment when starting from the initial distribution p ( S0 ) . It is defined , as : { ρπ ( s ) = ( 1− γ ) p0 ( s ) + γ ∑ s′ , a′ p ( s|s′ , a′ ) π ( a′|s′ ) ρπ ( s′ ) ρπ ( s , a ) = π ( a|s ) ρπ ( s ) ( 1 ) so that any policy π settled on an MDP defines an occupancy on the states of that MDP . It comes that , inversely , any valid ( state , action ) occupancy ( meaning that this occupancy is effectively feasible in a given agent/environment setup ) , defines a unique corresponding policy : π ( a|s ) = ρ ( s , a ) ρ ( s ) ( 2 ) that is a softmax ( stochastic ) conditional policy over the states . Following the same reasoning , let ρπ ( S+|s ... ) the conditional occupancy be defined recursively . Let Tπ ( s ) the set of trajectories starting from s : ∀s+ ∈ Tπ ( s ) , ρπ ( s+|s ... ) = pπ ( s+|s ) + γ ∑ s′∈Tπ ( s ) pπ ( s +|s′ ) ρπ ( s′|s ... ) The triple dots ( ... ) are intended to help distinguish the one-step distribution pπ ( S′|s ) from the longterm distribution ρπ ( S+|s ... ) . This conditional distribution provides a description of the “ future ” of s , that is the distribution of states that will most probably follow s. It can be seen as an instance of the “ successor ” representation of states initially proposed by Dayan ( 1993 ) . Those future states will generally be noted s+ , with the ’ + ’ exponent meaning the state being measured “ further away in time ” . 2.3 MATCHING REWARDS TO OCCUPANCIES . Those definitions provide a way toward interpreting rewards as occupancy templates , allowing to implement the “ matching law ” in a principled way . The mapping of rewards toward probabilities relies on using exponentiated returns in the parameters of a stochastic policy , such as in the softmax ( or Boltzmann ) decision rule case . Let π ( a|s ) = exp βQ ( s , a ) K ( s ) with K ( s ) = ∑ a expβQ ( s , a ) , with β the “ inverse temperature ” , and the state-action value Q ( s , a ) representing the total return estimated at ( s , a ) . Let τ = ( s0 , s1 , ... , st , ... ) a certain trajectory observed on the MDP under the policy π . The set of all possible trajectories is noted T , pπ ( τ ) is a measure over the trajectories for a certain policy π , and ρπ is the corresponding occupancy on states . Consider for instance the series of rewards encountered when following τ . It comes that : Es0∼p0V ( s0 ) = Eτ∼pπ ( T ) ∑ t γtr ( st , at ) ≈ E s∼ρπ ( S ) a∼π ( A|s ) r ( s , a ) ∑ t γt = Es , a∼ρπ ( S , A ) r ( s , a ) 1− γ so that ∀t , r ( st , at ) 1−γ is interpreted as an estimator of V ( s0 ) . Next , for any t , it comes that ∀t′ > t , r ( st′ , at′ ) 1−γ is an estimator of the state-action value Q ( st , at ) , i.e : Q ( st , at ) = Eτ∼pπ ( T ) st∈τ ∑ t′ > t γ ( t ′−t ) r ( st′ , at′ ) ≈ Es+∼ρπ ( S+|st , at ... ) a+∼π ( A|s+ ) r ( s+ , a+ ) 1− γ ( 3 ) In that setup , the rewards are interpreted as value samples . This means , in short , that each future reward r ( s+ , a+ ) takes the role of a “ model ” for the total returnQ ( st , at ) . The models are weighted according to the conditional occupancy ρπ ( S+ , A+|s , a ... ) , that takes the role of the “ mixture ” . Then , noting that log π ( a|s ) = βQ ( s , a ) −K ( s ) , we define : R̄ ( s , s+ , a+ ) , r ( s+ , a+ ) 1− γ − 1 β ( K ( s ) − log ρπ ( s ) ) ( 4 ) said the “ extended ” return composed of the return estimator plus a virtual baseline . Then , because the policy and the occupancy are exchangeable from eq . ( 2 ) , each reward collected after ( s , a ) may also take the role of a “ model ” for the occupancy , sampled from the conditional occupancy , i.e . : log ρπ ( s , a ) ≈ Es+∼ρπ ( S+|s , a ... ) a+∼π ( A|s+ ) βR̄ ( s , s+ , a+ ) ( 5 )

Summary: The authors address the problem of exploration in reinforcement learning. They suggest applying the maximum entropy principle over the state space and try to maximize the make the occupancy of the policy as entropic as possible while still optimizing the original goal of maximizing cumulative reward. They claims include an algorithm that does this using a replay buffer and off-policy learning, which improves sampling efficiency, in sparse and dense reward settings, in classic motor control benchmarks.