Occupy & Specify: Investigations into a Maximum Credit Assignment Occupancy Objective for Data-efficient Reinforcement Learning
Authors: Emmanuel Daucé
Venue: ICLR 2022 Submitted (2022)
Keywords: Reinforcement Learning
Intrinsic reward
MaxEnt
Probability matching
Motor control
Variational inference

Abstract: The capability to widely sample the state and action spaces is a key ingredient toward building effective reinforcement learning algorithms. The trade-off between exploration and exploitation generally requires the use of a data model, from which novelty bonuses are estimated and used to bias the return toward wider exploration. Surprisingly, little is known about the optimization objective followed when novelty (or entropy) bonuses are considered. Following the ``probability matching'' principle, we interpret here returns (cumulative rewards) as set points that fixate the occupancy of the state space, that is the frequency at which the different states are expected to be visited during trials. The circular dependence of the rewards sampling on the occupancy/policy makes it difficult to evaluate. We provide here a variational formulation for the matching objective, named MaCAO (Maximal Credit Assignment Occupancy) that interprets rewards as a log-likelihood on occupancy, that operates anticausally from the effects toward the causes. It is, broadly speaking, an estimation of the contribution of a state toward reaching a (future) goal. It is constructed so as to provide better convergence guaranties, with a complementary term serving as a regularizer, that, in principle, may reduce the greediness. In the absence of an explicit target occupancy, a uniform prior is used, making the regularizer consistent with a MaxEnt (Maximum Entropy) objective on states. Optimizing the entropy on states in known to be more tricky than optimizing the entropy on actions, because of an external sampling through the (unknown) environment, that prevents the propagation of a gradient. In our practical implementations, the MaxEnt regularizer is interpreted as a TD-error rather than a reward, making it possible to define an update in both the discrete and continuous cases. It is implemented on an actor-critic off-policy setup with a replay buffer, using gradient descent on a multi-layered neural network, and shown to provide significant increase in the sampling efficacy, that reflects in a reduced training time and higher returns on a set of classical motor learning benchmarks, in both the dense and the sparse rewards cases. 

Under review as a conference paper at ICLR 2022
OCCUPY & SPECIFY: INVESTIGATIONS INTO A MAX- IMUM CREDIT ASSIGNMENT OCCUPANCY OBJECTIVE FOR DATA-EFFICIENT REINFORCEMENT LEARNING
Anonymous authors Paper under double-blee blind review
ABSTRACT
The capability to widely sample the state and action spaces is a key ingredient toward building effective reinforcement learning algorithmss. The trade-off be- tween exploration and exploitation generally requires the use of a data model, from which novelty bonuses estimated and used to bias the return toward wider exploration. Surprisingly, little is known about the optimization objec- tive followed when novelty (or entropy) bonuses are considered. Following the "probability matching" principle, we interpret here returns (cumulative rewards) as set points that fixate the occupancy of the state space, that is the frequency which the different states are expected to be visited during trials. The circular dependence of the rewards sampling on the occupancy/policy makes it difficult to evaluate. We provide here variational formulation for the matching objective, named MaCAO (Maximal Credit Assignment Occupancy) that interprets rewards a log- lieeekkl on occupancy, that operate anticausally from the effects toward the causes. It is, broadly speaking, an estimation of the contribution of a state to- ward reaching (future) goal. is constructed as to provide better convergence guaranties, with a complementary term serving as a regularizer, that, in principle, may reduce the greediness. In the absence of an explicit objective occupancy, a uniform prior is used, making the regularizer consistent with a MaxEnt (Maxi- mum Entropy) objective on states. Optimizing the entropy on states in known to be more tricky than optimizing the entropy on actions, because of an exter- nal sampling through the (unknown) environment, that prevents the propagation of gradient. In our practical implementations, the MaxEnt regularizer is inter- preted as TD-error rather than a reward, making it possible to define an update in both the discrete and continuous cases. It is implemented on an actor-critic off- policy setup with a replay buffer, using gradient descent on a multi-layered neural network, and shown provide significant increase in the sampling efficacy, that reflects in a reduced training time and higher returns on a set of classical motor learning benchmarks, in both the dense and the sparse rewards cases.
PROBLEM STATEMENT
Learning in the real world implies dealing with very large, potentially unlimited environments, over which the data to collect is seemingly infinite. Efficient exploration is thus one of the key aspects of open-ended learning (Santucii et al. 2020)), when no final model of the environment can feasibly be expected to be engineered or trained. On the one side, having access unlimited data is very beneficial for the training of complex multi-layered perceptrons, for they are known to rely on large datasets improve their performance. On the other side, the circular dependence between the learn- ing algorithm and the data on which it operates renders the learning very tricky, at high risk of data local optima. The open-ended learning problem generally addressed
overfitting and trapping through the lens of the reinforcement learning framework (Sutton et al.) 1998), where rewards are collected during the interaction, and the selection of action is fit so as to maximize the total number of positive rewards, and prevent the encounter of negative ones. Fitting behaviour to rewards is however at the risk of ignoring important data from the rest of the environment, where putatively more rewarding regions may be neglected. The agreement of reward-seeking (that is exploitation) with data collecting (that is exploration), is still one of the fundamental issues of modern artificial intelligence.

---

Under review as a conference paper at ICLR 2022
An important effort has recently been put on reframing the reinforcement learning setup into more general probabilistic inference framework, allowing to link rewards seeking and data modelling under single perspective (Furmston & Barber 2010 Levine 2018 Haarnoja et al.] 2018 Ab- dolmaleki et al. 2018 Fellows et al. 2019). This greater focus over the data collection problem is linked to an important set of training algorithms, that contain some forms of exploration bonuses, in- cluding "curiosity" drives (Schmidhuber et al. | Pathak et al., 2017). intrinsic rewards (Oudeyer) ct al., 2007) and pseudo-counts (Bellemare et 2016 Tang et al.) (017). However, at the differ- ence of the classic optimization on rewards alone, where the Bellman optimum is well define, there still no consensus about the objective followed when optimizing both on rewards and data col- lection under a variational perspective (Eysenbach & Levine 2019). The data collection problem is effectively shadowed by the reward maximization objective, under which it is still considered as a incidental component. An important body of work has recently been devoted to addressing the data collection problem as such, with the notable design of the MaxEnt algorithm (Hazan et al. 2019), State Marginal Matching (Lee et al. 2019) and E3D (Daucé, 2020), that aim at fitting the distribu- tion of the states encountered to a uniform distribution, in the absence of definite rewards. This is here referred as a MaxEnt-on-stiane principle (or MaxEnt to be short), not to be confounded with the MaxEnt-on-actionn
principle implemented in the soft actor critic (Haarnoja et al. 2018) for instance. Following MaxEnt objective means optimizing the policy so the states visited are maximally variable, ideally uniformly visiting all possible states. We develop in the following a possible ex- tension of the MaxEnt principle, that brings a considerable simplification in the expression of the evidence lower bound (ELBO) with regards with the existing literature (Furmston & Barber 2010)) Abdolmaleki et al. 2018, Fellows et al., 2019). In contrast to pure MaxEnt, our approach provides a way to combine the MaxEnt objective with reward maximization objective, under a variational inference perspective. An intriguing property of the resulting ELBO formula is that the future states (the ones that are visited after the current observation) play the role of model for the current data, participating in the elaboration of the returns collected under the current policy. This gives ways toward optimizing the policy with respect to the distribution of the data, and provides principled justification to the use of intrinsic rewards in the design of reinforcement learning algorithms.
2 PRINCIPLES
2.1 PROBABILITY MATCHING RL
We assume an agent acting in a fully observable environment. The state of the environment is provided by an observation s E S, with S the set of all possible states. The agent can act on the environment through its actuators. Such a motor command is described by E A, with A the set of all motor commands. In the following, the capital letters and A will reflect random variables on S and A. and the lower cases and will either reflect observations or random draw realizations. The decision of which action to choose relies on a policy, that maps the current observation to the action space, generally expressed in conditional probabilistic form T(als).
A reinforcement learning problem consists in finding a policy n- that maximizes a certain objective function, without knowing the physical or mechanical properties of the environment. It is supposed here, for simplicity, that the dynamics of the environment is Markovian (no hidden states). More- over, the environment is providing an auxiliary signal called the reward. Sending action to the environment makes it possible to access to new state s', and to obtain a reward r E R. A classic discounted sum of
objective in learning is maximize the global return, generally described future rewards over all possible trajectories. Let us now denote by the state visited at time t and r($:) = (S.41..... St+T..... a certain pathway that is visited after observing st. During this visit, a certain number of rewards can be collected, and R(T) is the (discounted) return obtained over r, i.c R(T) = E. rr,, with 7 E [0, 1[ a discounting factor that sums up the rewards up to an "horizon" 1. This said, the dynamic programming objective (Bellman 1966), is the result of π = maxn Eg-m(So) (r)= R((), with p(So) the distribution of initial states, and II the set of all conditional policies. When a state transition model p(S'|s. s, a) is provided, the unique solution is given by the dynamic programming recurrent equation in the discrete case (Bellman 1966). On the contrary, a large panel of reinforcement learning techniques allow to approach the solution in the model-free setup, assuming an effective sampling of all state-action pairs (Sutton et al. 1998).
of the order of I--
2

---

Under review as a conference paper at ICLR 2022
We are here interested in a different class of objective function, that rely on fitting the rewards toward probabilities of state occupancies. A reward should indicate in which proportion the different states (and actions) should be visited (and selected) during trials (with the idea that the states providing high return should be visited more often than the ones providing low returns). Solving the reinforce- ment learning problem then means to match the external states and actions, where a differential in rewards only indicates a difference in the number of visits, allowing to seek rewards in flexible way (so is also referred as "soft" reinforcement learning HHarmoja et al. 2018)). This idea stems back from empirical observations on human and animal behaviors, and was coined the "matching law" in the operant conditioning literature (Hermstein Eysenbach & Levine 2019).
to actual distribution of visit over
1961
2.2 STATE OCCUPANCY AND CONDITIONAL STATE OCCUPANCY
Matching rewards to probabilities can be done in many different ways. We frame here the probability matching reinforeement learning problem into state occupancy matching problem. It relies on the use of an occupancy distribution, that is a density of state visit under a certain policy. Importantly, it ignores the time order which the different states are visited, still conserving some aspects of causality between states in the form of conditional probabilities as we see later. Dating back from Dayan (1993), an occupancy distribution a distribution on states, designed as match with the distribution measured over the trajectories of the MDP.
Following the definitions of (Puterman 2014 Ho Ermon 2016 Hazan et al. 2019). a gamma-
absorbing state occupancy of a Markov Decision process (with policy ক is the (discounted) density of visit of the states or (state, action) pairs of the environment when starting from the initial distribution p(So). It is defined, as:
P (s) = - -p)pp(s)+ +7 p(sls, "al,,, (d)) Pr (s) p(a)s)p=(s)
(I)
so that any policy settled on an MDP defines occupancy on the states of that MDP. It comes that, inversely, any valid (state, action) occupancy (meaning that this occupancy is effectively feasible in given agent/environment setup), defines unique corresponding policy:
p(s.a) a) p(s)
T(a|s)
that is a softmax (stochastic) conditional policy over the states.
Following the same reasoning, let Px (S+ (s...) the conditional occupancy be defined recursively. Let Ta (s) the set of trajectories starting from s:
Vs+ Ta (s), (x) (s...) Pa (s+(s)) +7
Σ S'ET((s)
The triple dots (...) are intended to help distinguish the one-step distribution Ps (S' from the long- term distribution Par (S* |S*+s...).. |s...). This conditional distribution provides description of the "future" of s. that is the distribution of states that will most probably follow s. It can be seen as an instance of the "successor" representation of states initially proposed by Dayan (1993). Those future states will generally be noted st, with the '+' exponent meaning the state being measured "further away in time".
2.3 MATCHING REWARDS OCCUPANCIES
Those definitions provide a way toward interpreting rewards as occupancy templates, allowing to implement the "matching law" in a principled way. The mapping of rewards toward probabilities relies on using exponentiated returns in the parameters of stochastic policy, such as in the softmax
(or Boltzmann) decision rule case. Let (als) exp (Q(s.o) with K(s) Σ. exp (Q(s, a), with
K(*) the "inverse temperature", and the state-action value Q a) representing the total return estimated at a).
Let T = (Sp. $I...., st.) a certain trajectory observed on the MDP under the policy т. The set of all possible trajectories is noted T. (+) measure the trajectories for a certain policy
3

---

Under review as a conference paper at ICLR 2022
and Pr is the corresponding occupancy on states. Consider for instance the series of rewards encountered when following т. It comes that:
r(s,a)
Esorpo V(su) = Erop, (T) E Ir(r(s a1) xo E (s) r(s, a) E, Es (5.A)
Er
avT((AA)
i
so that Vt, r(s,.a)) is interpreted as an estimator of V(so). I-y
Next, for any 1, it comes that AIl > L, r(sg.ay) is an estimator of the state-action value Q(st, at),
Le:
r(staa) a
(++, 0z) = (T E
(T)
Σ US1
700 (r(err ar) v Eg (st at atmm(Allet)
(3)
1 y
In that setup, the rewards are interpreted as value samples. This means, in short, that each future reward r(st, aat) takes the role of "model" for the total return (sx; az). The models are weighted
according to the conditional occupancy Ps (St AAt|s, a...). that takes the role of the "mixture".
Then, noting that log (als) BQ(s, a) (s),, we define
R(s, st at r(staa" a+)
1
(K(())
- log Ps (s))
(4)
B
said the "extended" return composed of the return estimator plus virtual baseline. Then, because the policy and the occupancy are exchangeable from eq. (2. each reward collected after (s, a) may also take the role of "model" for the occupancy, sampled from the conditional occupancy, i.e.:
log E, (s BR(s, 8 ,aa
2.4 DENSITY MATCHING OPTIMIZATION
Assuming a current policy, Psr a corresponding occupancy, and taking (logp(s, a-.s)))).t as shorthand for Eyt (sp ls..)....) (R st ,a) the sampling- optimization writes:
(Als*)
argmax Es,arp (log p(s, a)) Att
which is cross-entropy objective that aims at fitting Pr (S, A) with (PS)))), The optimization can be done, for instance, by optimizing a current policy by stochastic gradient ascent on the objective, which conducts to maximizing the return, under softmax policy, like in classic policy gradient.
we can make a step further by trying estimate how far is (log p(s, a)) from the opti- mum log p (s, a). Interestingly, for any distribution on the future states ((St, A+), the following inequality holds:
However,
E,1 .a ((()) .A1 A1s aa") 2 E, la rg(S1 .A 1 R[ss, st statt=++++++++) (a AAt))lP" (Att,,,...))))
R aagttt , log (s, ( A+
providing variational (log-) evidence lower bound (ELBO) interpretation of the maximization on RR(s, s* ..*). satt.. At the convergence of the gradient ascent, the distribution is expected to match the posterior p², , and the inequality would becomes an equality, i.e.
pog."(s, a) Eg+
(........)) 3R( (s(ss -*.
al......)llp(( Is* ""ll="ssttt"i a...))
This formula ( not directly usable in optimization, because the optimal posterior p² is not spec- ified, but provides new hints into interpreting the current occupancy at the light of its future effects. The actual optimization is indeed done on R(s st (5s.statt aa) solely, irrespective of the divergence bias. This bias is expected to fade away with the progress of the training, making the sampling of the rewards more and more accurate at estimating the parameters of the policyloccupancy. This how- ever, is not guaranteed, and the optimization high divergence throughout the optimization, hindering the convergence toward the optimum. This illustrates more general problem that is the lack of efficacy in sampling the data (the posterior occupancy). a more robust upper bound guarantee would be preferred, even at the risk of a lesser final optimality with regard to the Bellman optimum.
the (pseudo) cross entropy is at risk of keeping
4

---

Under review as a conference paper at ICLR 2022
2.5 MAXIMAL CREDIT ASSIGNMENT OCCUPANCY
We now introduce the main insight of our Maximal Credit Assignment Occupancy (MaCAO) model. It appears that the formulas (5) and (6) are not entirely satisfactory: the average over the rewards translates into an average over log-density templates, that actually depend on (st, aa), though is not explicit in the formula. Expressing this dependeney is possible by reshaping the ELBO formula like:
(8) This introduces a new conditional distribution, namely p(S, A..s" ..t) that is the frequency at which (s, a) may precede (st at) in the iteration of the dynamics. This distribution expresses an "anti-causal" relationship between the future states and the current observations, that is the exact measure of how much (s, is instrumental in reaching (st a+). This is also said the "credit signment" in the reinforcement learning literature (Sutton) 1988, Harutyunyan et al., 2019).. Here p [s, al...st a represents the target credit assignment. From the Bayesian perspective, it inter- preted as log-likelihood of the "data" (s, a), given the "model" (s*,a**), that is way to say that (st ".a"" exerts a control on (s,a).
logp"(s, 2 "attttttg(S*...... ,At) log al...st ,."... (xI,( A')llp"(Ss A*))
expresses
This conducts to reconsider the rewards as log-likelihood templates rather than occupancy tem- plates. It comes from lemma 4 (see appendix)) that: Egg, (ool(s log p"(ss, (s,aa...." st.....staa aat N Ext, Bat,atng(St wat.gggsttttee (s) st ta, ,a)), so that:
p"g a) 2 tattgstttttt R(s, st ,a*) Dx - (+))
(9)
This new objective provides a variational Bayesian perspective on the density matching optimiza- tion, with q (the conditional occupancy) taking the role of the variational distribution. This is equiv- alent to augmenting the return with a supplementary divergence term on the conditional occupaney. Then, matching log, p (s, a) with the returns becomes identical to maximize the ELBO ( from variational inference.
The loss is composed of two complementary terms, first term being the occupancy matching on cumulative rewards (that is consistent with the softmax optimization), while the second term is the explicit density matching of a posterior with prior. The prior takes the role of supervision signal, that aims at putting a constraint on the conditional occupancy (that represents the exploration pattern). By construction of the loss, the right term is made shape the posterior occupancy, that conducts the evaluation of the return. This is formally analog to the case of Bayesian inference where the prior serves as regularizer that tends counteract the overfitting of the data. This sort of regularizer is highly expected in reinforcement learning that is known to be prone to overfitting. The analogy with Bayesian inference suggests for instance to consider a Gaussian or a uniform distribution. This has important consequences though. Taking a prior that is not the target occupancy strongly modifies the interpretation of the loss, that is now composed of two concurrent terms. The likelihood part aims at fitting the occupancy with the rewards collected, while the divergence part aims at fitting the posterior occupancy with an arbitrary prior. This breaks the original symmetry, for the implies concurrently follow two different objectives. For instance, the role of regularizer that would be devoted to a uniform prior introduces a bias in the probability matching, for the target occupancy is not anymore the noftmax-Bellman optimum, but rather intermediary occupancy that combines the Bellman optimum and a uniform occupancy. This is expected to be beneficial for a wider exploration, helping avoid overfitting, but this is at the cost of a relaxed constraint on fitting to the optimum.
This expression of a variational upper bound is reminiscent of the standard variational bounds con- sidered in the reinforcement learning literature (Furmston & Barber] 2010 Abdolmaleki et al. 2018), from which it could be interpreted as an occupancy occupancy-oriented variant. By breaking the tem- poral chain dependence, however, it surprisingly provides justification for considering an entropy on states in the optimization of the policy. Indeed, the classic variational optimization operates on chained sequences of observations, for which the state transitions vanish during the optimization of т (Haarnoja et al. 2018). In our case, ignoring the sequential time order establishes both the future states and the future actions parameters of the policy, over which it should be optimized. This allows to address the data efficiency through explicitly optimizing on the exploration path. It more- over provides room to the maxent-on-stae ot-ste term (Hazan et al. 2019). as regularization incentive in a more general expression. This conceptual shift has however an important consequence on the objective followed, for the target occupancy is not anymore equal to the softmax Bellman optimum.
5

---

Under review as a conference paper at ICLR 2022
3 METHOD
The main ingredients for an efficient implementation is the access to a wide variety of samples (s, (s,a,r, a, s', aa) over which optimization tor") and a parameterized action value function Q₂ (said the "critic"). It is here implicitly assumed that both the actor and the critic consist of multi-layered perceptrons, containing many parameters and organized in layered weights, over which a gradient descent is operated on losses expressed as negative objectives.
be carried out on parameterized policy no (said the "ac-
Assuming an off-policy approach, we consider a replay buffer containing many samples of states, actions and rewards observed from interacting with the environment. In a variational setup, one can assume an alternation between two complementary steps. A first step, said the "estimation" step, consists in evaluating a distribution over the parameters of the model, that here assumed to be a distribution over future states, identified as the conditional occupancy under the current policy (r- This distribution is then exploited in second step, said the "maximization" step, where the parameters of the policy/Q-function are updated so to maximize the proximal objective (that is fitting a policy to a distribution of returns obtained from the current occupancy).
We assume here the estimation of yr being obtained from a parametric or non-parametric method over a sufficiently large sample of recent states (or state, action pairs), and concentrate on the optimization of the actor under the guidance of the critic. Knowing 9n, and given sample (s, a, r, r's,, a'), an element of optimization is given by the log-difference log qu (s', a')-log pa (s', a') that is point estimate of the Kullback-Leibler divergence, that is expected be minimized during the optimization of the policy. This term is supposed to be differentiated with regards to π, providing a first gradient direction that should contribute improving the policy toward wider occupancy on future states. A second and independent element of optimization is the parameterized action-value mapping Q(s, a), relying on the TD-error construct Sutton (1988). based on approximating the fu- ture rewards with the best current estimate at t 1, defining gamma discounttdd proximal objective value ((s, a) = r + (Q('s, a'). The squared difference ((Q(s, a) Qu a))², with A a precision hyperparameter, is known as the mean-square -quured Bellman error (MSBE), providing a second gradient that aims at maximizing the return with regards to the policy parameters. The concurrence of both gradients contains the necessary elements to combine exploration and exploitation in an principled way.
Building a full parametric model of the occupancy is however a difficult task that should be under- taken with care. The building of such a probabilistic model is indeed at a non negligible cost of regressing parametric distributions from samples, that inherently contain design choices and a spe- cific optimization of simplicity, we consider here the case of a non- non-parametric estimator of the occupancy distribution 4r. This could appear counter-intuitive at first glance, for the update of the policy is supposed to rely on backpropagating gradients through the estimator. As a workaround, we provide here a method allowing to directly inject the gradient information in the design of the Q-function.
set of latent parameters (like in the case, c.g., of auto-encoders). For the sake
Assume parameterize Q-function Qu that should undergo dual optimization under two concur- rent objectives. This implies in short that the value settled in the Q-function may arbitrate between the reward-secking and the occupancy-seckinn only may be equivalent to the previous concurrent optimization. Considering (s, a, r, s', a') sam- ple, simple way to retain the divergence information is to simply add the log difference information log 9z (s', .a' log p" (s', aa) to the current Q Qufuntiion, making it possible formula. This sketch of idea implies the use of two concurrent update rules in the Q-function itself. first term L a, r, r,,', a,)) would be the traditional MSBE loss on rewards. The second loss needs to consider the lof-difference itself as an error, for it to be "contained" (so to say) in Q after the update. Consider the implicit reward:
objectives, so that maximizing π with respect to Q
recover the original
[Qu(esa)- _ (log an (s', log log ()))
(s, 2 (1 7))
(10)
and let Q(s.a)) = rk)(s, + (s', Then a complementary MSBE loss is:
(s, (raa a) + 2Q a()(s))))))))))
a))

---

Under review as a conference paper at ICLR 2022
Algorithm Maximum Credit Assignment Occupancy (MaCAO)
Require: Tee (actor), Q+ (critic), B (replay buffer), 8, A (hyperparameters) while number trials not exceeded initialize the environment
while trial not terminated do
observe
choose (T()As))
read s'.r
store (s, s') in B
end while if B is full enough then
randomly sample batch of (next) states {s* ,..} from B.
estimate g with a nonparametric method. while number of batch updates not exceeded do
randomly sample hatch of transitions b = a(,, a, 8'), from B.
for all (s, a, r, 8') b do
estimate log q(s') and set rki. (eq. 10
sample a' ~ Ta (Als')
calculate (MacAo(s, a, r, 8', a') (eq.
end for
update the critic Qu by gradient descent over all losses.
for all s E
sample a ~ (AIs) estimate (s, a) calculate Lact Qu as)
log (Als) (including the reparameterization trick)
end for update the actor by gradient descent over all losses.
end while end end while
So that the final loss expression is:
a,, => (r +Q00s(s'''a) Qu(s,a))²
(II)
[1-
(1-7)
log'''lopppp (ogg"(s',aa")) (s' + (Qu(d'))) - 2u(s,))))
The additional precision hyperparameter À plays here the role of an extra regularizer, helping to adjust between both terms in of highly sparse rewards. The main lines of our implementation are provided in algorithm[] [ (see Appendix), that fits the pursuit of the MaCAO objective in an actor- critic setup. relies on a wide use replay buffers (Mnih al. 2013) to regularize the gradient over batches that mix the samples from many different trials. From this perspective, important shortcut our mn-the-fly calculation of the log- og-occupancc, with the help of kernel-based density estimation method (Pedregosa al., 2011) from an initial sampling of (future) states from the buffer at each start of an update sequence (line 10). This occupancy sample remains quite limited number (about 1000) in order to avoid unnecessary computer overload.
4 RESULTS
In order to reach state-of-the art efficacy, many algorithmic improvements need to be included in supplement the baseline algorithm") This concerns particular the use of target Q-networks up- dated at slower pace (Mnih et al. (2013), and the clipped double-Q trick (Fujimoto al. 2018). Our implementation moreover drawn over the "spinning-up" open source framework (Achiam) 2018), allowing for direct comparison with the state of the art. We consider here for comparison the soft actor-critic method (SAC) (Haarnoja et al.] 2018)), proximal policy optimization (Schulman et al.) 2017) and TD3 (Fujimoto et al. 2018). Our method ments, as provided by the "Gym" suite (Brockman et al. 2016). We concentrate here on the contin- states and actions case, that is the most challenging one with regards function approximation.
tested over several benchmark environ-
code freely available http://gibuub.oom/xx/yyyy (to come).

---

Under review as a conference paper at ICLR 2022
The different setups compared on the basis of the returns col- lected during training. This is expressed as average return (that the total sum of rewards gath- ered at the end of an episode), the average reward (total rewards collected divided by the episode length) and cumulative rewards (the total sum of rewards collected a given stage of the training). The width of the occupancy over the state space is not compared here, for the other frameworks are not designed to optimize it. The different environments differ in scale, difficulty and rewards den- sity. All continuous problems pro- posed in the library provide dense rewards, that are a compound of negatively and positively weighted extrinsic informations, like the en- ergy consumption, the speed of the agent or its clevation. The prob- lems separate in two broad cate- gories. first class of problems provides only dense rewards. A second class of problems have, in addition, supplementary sparse reward taking the form of an "end- of-episode" bonus or penalty. In that case, the dense rewards may may not) contain relevant in- formation with regards to the task hand.
1.2444
722491
5/5400 186.0
Plestt/
Figure 1: Methods comparison Average episode rewards, average rewards and cumulative rewards compared in the course of Iearn- ing for the MaCAO, SAC, PPO and TD3 frameworks, on contin- uous state/continuous control problems. Row 1: Gym Continuous Mountain Car problem. & = 10, A = 0.1, 7 = 0.99, hidden lay- with N 32 neurons. 5 seeds. Row 2: Gym/MuloCo Swim- mer problem.
From that prospect, the most un- favorable problem is the Contin- uous Mountain Car problem (first row of figure 4). Here the dense rewards only refer to the energy consumption, at the exception of +100 end-of-episode bonus ob- tained at the hilltop. This in- evitably conducts baseline algo- rithms to remain stucked at the bottom of the hill, where the en- ergy consumption is low. Only our approach, that contains an explicit incentive for widening the occupancy of the state space, has the capability to reach the most rewarding states, finally providing policy that solves the task.
0.995, hidden layers with N 32 neurons. seeds. Row 3: Gym/Box2D Bipedal Walker. 8 = 30, = 1, = 0.99. hidden layers with N 64 neurons. seeds. Row 4: Gym/MuJoCo Ant. 3 10, A 0.3. 7 = 0.99. 2 hidden layers with N 256 neurons. 5 seeds. Row 5: Gym/MuloCo Humanoid. 10, 3, 0.98, hidden layers with 256 neurons. seed.
30, A 0.3,
The Bipedal Walker (second row of figure is also problem that combines dense and sparse rewards. A negative (-100) reward is undergone when the agent falls down, and a positive (+ (+100) reward is gained when the agent reaches the end an incentive for staying upright and increase the velocity. This task reveals more tricky to train than expected, and contains enough variability for the agent to develop various gaits and locomotion patterns over the course of learning. Our approach shows here clear advantage, that is maybe more obvious when comparing with the SAC. Like the Mountain car, the problem is about reaching a final (distal) end-of-par path objective, from which strong bonus allows to "freeze" the behavior in favorable locomotion pattern. The (S-shaped) discontinuity in the MaCAO and the SAC learning
track. The continual dense rewards provide
8

---

Under review as a conference paper at ICLR 2022
curves reflect the reaching of the distal objective, after which close-by policies are followed. The difference in the two curves is the time at which it is attained, that is less than 10 iterations in the first case, and more than 10t in the second. This one order difference illustrates the disadvantage of occupancy-agnostie optimization methods.
The swimmer task (third row) is concerned with the development of locomotion pattern that is swimming in a liquid medium. The reward is only the speed at which an eel-like agent manage to swim over the place (that is coordinating segments in periodic manner). This tasks contain local optimum that corresponds to rower pattern that coordinates the extremal segments, and global optimum that corresponds a classic swimming ripple from the head toward the tail. Despite its apparent simplicity. only an extensive exploration such as the one provided by our approach allows to reach the optimum.
The fourth task, known as the "ant" aims at controlling the locomotion of a 4-legged agent. The state space contains a detailed account of joint angle and torque moments plus contact sensors in a 111 dimension observation vector (Schulman et al., 2015), but the control space is more reduced (8 DOFS). Here again the displacement speed is the main incentive, with a survival bonus, and an energy cost penalty, All 3 actor critic frameworks (namely MaCAO, SAC and E3D) are here capable to reach a decent locomotion pattern in about 400000 iterations of the dynamics, which can be considered data efficient here. No clear advantage is found here for our approach.
Last, the humanoid task shows a large number of degrees of freedom, and the unlimited number of possible locomotion patterns often result in strange-looking final gaits. Only the SAC and the MaCAO methods allow here to reach decent locomotion patterns in the limited number of steps considered. When looking in detail, the light advantage observed for the SAC algorithm on the average episode return is reversed when considering the average return. This apparent contradiction explained when looking at the detailed behavior. Here, the high-speed risky locomotion patterns developed in the MaCAO framework result in a higher number of early failures. This is not related to a risk seeking incentive, but rather explained by a tendency to maintain high diversity of behavior while pursuing the reward-guide objective, which reveals to be more risky when the balance of the body needs to be maintained over time.
DISCUSSION
This work participates to general trend toward the development of data models in reinforce- ment learning, that provide ways to help the agent toward better exploring the world. This is known of practical use and has been largely exploited so far in the large family of curiosity-driven and mekimum--nntropy algorithms. Our contribution here is to provide a more detailed appraisal of the benefits and putative risks of such a construct. Bayesian/variations optimization where the future data plays the role of model, and where an evidence lower bound is maximized through gradient ascent over the policy parameters. The gen- eral principles exposed point to the importance of an occupancy model that synthesizes the general distribution the agent's environmental states over which it can act (defining a virtual "territory"). Those occupancy models are the subject of frequent updates as the exploration progresses and that new states are undisclosed during the course of the training. By making an additional uniform prior assumption the occupancy, the resulting loss expresses balance between two concurrent tenden- cies, namely the widening of the occupancy space and the maximization of the rewards, reminding of the classical exploration/exploiiatoon trade-off. The consequence is a shift in the target occupancy pursued, that relaxes the constraint on fitting the initial Bellman objective. Both are embodied in MSBE Loss operating on single Q-function in our implementation (though this is not necessary the case). Computer simulations illustrate the benefit of our conceptual developments, both in the case of sparse and dense rewards.
is shown here to frame into a larger
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Mar- tin Riedmiller. Maximum a posteriori policy optimisation. arXiv preprint arXiv: 1806.06966, 2018.
Joshua Achiam. Spinning Up in Deep Reinforcement Learning. 2018.

---

Under review as a conference paper at ICLR 2022
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information pro- cessing systems, 29:1471-1479, 2016.
Richard Bellman. Dynamic programming. Science, 153(3731):34-37, 1966.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint 160.::666600054// 2016.
Emmanuel Daucé. End-effect exploration drive for effective motor learning. In International Work- shop on Active Inference, pp. 114-124. Springer, 2020.
Peter Dayan. Improving generalization for temporal difference learning: The successor representa- tion. Neural Computation, 5(4):613 624, 1993.
Benjamin Eysenbach and Sergey Levine. If maxent the answer, what the question? arXiv preprint arXiv:1910.01900 2019.
Matthew Fellows, Anuj Mahajan, Tim GJ Rudner, and Shimon Whiteson. Virel: A variational inference framework for reinforcement learning. Advances in Neural Information Processing Systems, 32:7122-7111 2019.
Scott Fujimoto. Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, pp. 1587-1596. PMLR, 2018.
Thomas Furmston and David Barber. Variational methods for reinforcement learning. In Pro- ceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp- 241-248 JMLR Workshop and Conference Proceedings, 2010.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- on machine learning. pp. 1861-1870. PMLR, 2018
Anna Harutyunyan, Will Dabney, Thomas Mesnard, Mohammad Gheshlaghi Azar, Bilal Piot, Nico- las Heess, Hado P van Hasselt, Gregory Wayne, Satinder Singh, Doina Precup. et al. Hindsight credit assignment. Advances neural information processing systems, 32:12488-12497, 2019.
Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy PMLR, 2019.
exploration. International Conference on Machine Learning, pp.
Richard Hermstein. Relative and absolute strength of response as a function of frequency of reinforcement. Journal of the experimental analysis of behavior, 4(3):267, 1961.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural
information processing systems, 14:4565-4573 2016
Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdi- nov. Efficient exploration via state marginal matching. arXiv preprint arXiv:1906.05274 2019.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv: 08059 2018
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Joannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 1rXiv:1312.5602, 2013.
Pierre-Yve Oudeyer, Friric Kaplan, and Verena V Hafner. Intrinsic motivation systems for au- tonomous mental development. IEEE transactions on evolutionary compulation, 11(2):265-866. 2007.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. 2778-2787 PMLR, 2017.
10

---

Under review as a conference paper at ICLR 2022
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Seikit-learn Machine learning in python. the Journal of machine Learning research, 12:2825-2830. 2011.
Martin L. Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Vieri Giuliano Santucci, Pierre-Yvv Oudeyer, Andrew Barto, and Gianluca Baldassarre. Editorial: Intrinsically motivated open-ended learning in autonomous robots. Frontiers in Neurorobotics, 13:115, 2020. ISSN 1662-5218.
Jurgen Schmidhuber al. Simple algorithmic theory of subjective beauty, novelty, surprise, in- terestingness, attention, curiosity, creativity, art, science, music, jokes. Journal of SICE, 48(1), 2009.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Picter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.024553 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: arXiv:1707.063477 2017.
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, (1):9-44, 1988.
Richard S Sutton, Andrew G Barto, et al. Introduction to reinforcement learning, volume 135, MIT press Cambridge, 1998.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. exploration: study of count-based exploration for deep reinforcement learning. In 31st Conference on Neural Information Processing Systems (NIPS), volume 30, pp. 1-18, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5033.. IEEE, 2012.
A
MATHEMATICAL COMPLEMENTS
A.1 LOWER BOUNDS
Consider triplet of random variables AS, Z) defined on the domains S x A Z and obeying to the mixture probability p(S, A, 2). Les S be "state", a E AA be an "action" and 2 "successor". Define p(S). p( A), p( [)), p(S, A), p(S, 2) and p(A, 2) the corresponding marginals p(X,Y)
and define any conditional p (X p(Y)
Lemma 1. Take a probability distribution on A, then S,
log 2 Ba-z(A) logp(s, a) - log (a)
Proof. Noting that log p(s, a) = logp(s) p(alss))::
EarT(A) log ((s,(s,o a) log π (a) = logp(s) - Ee-r(A) (log (()) log ((ass)
and (A) log (a) log p(als) 2 0 (Kullback-Leible divergence positivity).
Lemma 2. Take probability distribution on 2, then V(s, E S x A.
logp(8, a) 2 Rz)z(z) log p(s, a|2) log (() + log)(z)
Proof. Noting that log a/2) + (2) = log a) + p(2|s, a) (Bayes Theorem) Ezzg(Z) logp(s, a/2) log ((2) + p(z) log (s) - Ez~g(Z) (log q(2) - log p(2))
and Beng(2) log (=) (ogp(=) 2 0 (Kullback-Leibler divergence positivity).
11

---

Under review as a conference paper at ICLR 2022
Lemma 3. Take probability distribution A and 9 probability distribution on Z, then E,
logp(s) 2 logts, log (a) log ((2) (2) log p(2)
Proof. From lemmas Tand use the transitivity of the 2 operator
Remark:
the random variable Z has been used here for readability. Taking Z = .A*) with (S+ ,,,)* S A and p( = p(S ,A+), the previous properties still hold.
A.2 THE REWARD CONSTRAINT
Take B E IRt, E|0, 1], and 2 + R reward function.
r(*) Note R(2) 1-y"
Assume that p(A(s) (said the "optimal policy") is such that
exp B1E p(a R(z)
p(a|s) ENeE exp 5T / (l,,,, R(z')
and define K(s) log Enes exp 3E p( R(2) Lemma 4. (xxp(((),,) log (ogp(s,a122
E Benp 20 2 K(s) + log(p(s))
Proof.
logp(s, a) = p(s) logp(als)
logp(s) E K(s)
From Bayes rule, we know that: vz: logp(s, + log log p(2) 2) + p(g,((, alz). Then:
Egap(Z])sa) logp(s, a|z) logp(=/s; logp(=) E 3R(3) K(s) log)(s)
Noting that:
E log logg(pp)) 20
it comes that:
Eznp(2)na) loggoooopp(s(s 22n ((s) ))+ + logp(s)
in the main text, R(s, 2) = Ezso(Z)a R(2) (K()) log)(ss) said the "extended return".
Remark:
Define (A[s), conditional distribution actions, said the "current policy". Then: Lemma 5.
(a)x(A)s) log + (als) E a>a(A AR(s, 2) loggop)/ss.00 a) + logp(z) 2~p(2)ss.a)
Proof. Identify from lemma (AA) = (Alss) and g(Z) p(Z]s, Then from lemma 4
comes:
log p(s) E (Als) s~p(Z(a,a)
BR(s 2) log m(ls) log a) logp(2)
which proves the formula.
The right term of the formula is said the (log)-evidence lower bound (ELBO) in the main text. It is constructed such that RR(s 2) is interpreted as the log likelihood of (s, a) (even it is, more strictly speaking, lower bound of the log likelihood).
12

---

Under review as a conference paper at ICLR 2022
A.3
OPTIMIZATION
The ELBO being bounded from above by log p(s), one can now define (local) optimization crite- rion. Say II be family of policies. Then the optimization criterion is:
max E amm((A s) R(s, z) log π (a]ss) log p())s.a) + log
TEll 2mp(77|8.0)
This criterion contains by construction two optimization objectives.
A first objective said the reward Ceenstt-matcchine objective:
max E ave((Al)) R(s, 2) log (als)
2>p(2]s;.)
It is constructed so that the log- org-robbabylity of choosing action proportional to the aver- age reward observed further stages of the dynamics.
A second objective is said the regularization objective:
max (Als) log p(z) logp(z|s,a)
mell
gropl ((,,a)
Taking uniform prior distribution p(Z) imposes for the policy to make the successor Z fit, on average, uniform distribution.
13


Reviews:
[{"review_id": "y79vqGzgdOF", "reviewer": "Reviewer_yV1o", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Quality\n* #### Weaknesses\n    * ##### Major\n        * While numerous mathematical statements are made throughout the paper, some are already well-established in the reinforcement learning literature (the chain of statements ending Section 2.2 should all hold with equality and dates back to the successor representation[1]) and others are simply stated as fact without proof, although surrounding text would suggest that the authors are claiming them as technical contributions of this work. While the introduction of variational approximations leads to lower bounds that resemble those shown in the paper, the various notations and distributions introduced make it difficult to confirm their validity right away. The authors should (at a minimum) include proofs for novel technical contributions of this work and further include proofs for supporting results that may help the reader understand the nature of their contributions.\n        * In Equations 9 and 10, (my understanding is that) the included log-likelihood ratio is between the variational approximation of the behavior policy visitation and that of the optimal policy. It seems rather likely that this ratio may suffer from numerical errors (namely, division by zero) when the exploratory behavior policy visits a region of the state-action space not covered by the optimal policy. Could the authors comment on why this is not an issue? Or are the authors making some kind of absolute continuity assumption?\n        * Ultimately, this paper is deeply concerned with addressing the challenge of exploration. Can the authors comment on why this formulation under probabilistic inference is reasonable given recent work which shows how the approximations needed for these methods to be applied in practice fail to pass even basic sanity checks? [2]\n\t* ##### Minor\n\t\t* \n\n### Clarity\n* #### Strengths\n    * The clearest element in the paper is Algorithm 1, which at least conveys the generic structure of what the authors are trying to achieve. Unfortunately, everything leading up to Section 3 is incredibly unclear stemming from a variety of sources including non-standard mathematical notation, undefined jargon/terminology, and (to a lesser extent) grammatical errors. The authors have wasted a page describing details of widely-known Mujoco environments that could instead be allocated to adding clarity to Sections 2 and 3, which seem to articulate the bulk of the contribution. \n* #### Weaknesses\n    * ##### Major\n        * The mathematical notation used throughout the paper is terribly difficult to parse and inconsistent with the broader reinforcement-learning literature. As a concrete example, Section 2.3 contains undefined notation (\\mathcal{T}\\pi(s)), ambiguous notation (s... notation, s+ appears to denote a state and yet is later said to denote a sequence of future states but then used as a normal state in Equation 3), and nonsensical notation (the second paragraph of Section 2.5 has Q(s,a) \\sim \\frac{r(s+,a+)}{(1-\\gamma)}, which is entirely vacuous without being defined beforehand). The authors seem to jump back and forth between states indexed by time vs. not. \n        * The authors make a bad habit of introducing assumptions unnecessarily. In Section 2.4, if the first assumption on Boltzmann policies is made, then why is the second assumption on the existence of K(s) needed as well? The statement should hold under the first assumption and K(s) can be written out explicitly. Why is the first assumption of Section 2.5 needed as if it re-defines the optimal policy? Unless the policy class over which the argmax is taken is a subset of all policies, this is the definition of the optimal policy for a MDP. \n        * The authors introduce their own terminology without any formal definitions or visualizations to assist the reader in understanding what the terms are meant to convey. As a result, the first paragraph of Section 2.7 reads vacuously without knowing what is meant by \"pathways\", \"conducts\", \"adverse\", through\", or \"width.\"\n        * The paper has several grammatical errors throughout. While a small handful of these would not constitute a major weakness (and, in fact, could be easily enumerated and corrected), there are enough of them to impact readability of the paper.\n\t* ##### Minor\n\t\t* The use of the phrase \"probability matching\" in this paper is misleading since it is widely associated with Thompson sampling. Distribution matching would be a more accurate phrase for what the author is trying to convey.\n\t\t\n \n### Originality\n* #### Weaknesses\n    * ##### Major\n        * The paper lacks a section dedicated to providing readers with an overview of related prior work. Consequently, it is difficult to precisely identify how the authors' variational formulation extends or expands beyond some of the prior work mentioned in brief at the top of page 2.  \n\t* ##### Minor\n\t\t* \n\n### Significance\n* #### Weaknesses\n    * ##### Major\n        * The authors haven't included details of how many random seeds are used to generate the results in Figure 1; consequently, I'm skeptical of their significance and reproducibility. That said, just examining the average return column (which is the standard metric), it seems that the proposed approach only leads to meaningful performance improvements in simpler control problems, while results in Swimmer and Humanoid are only on par with SAC.\n        * Simpler experiments in domains where things can be concisely and cleanly visualized would go a long way towards driving home the central claim that this approach facilitates improved exploration in practice.\n\t* ##### Minor\n\t\t* \n\n# References\n1. Dayan, Peter. \"Improving generalization for temporal difference learning: The successor representation.\" Neural Computation 5, no. 4 (1993): 613-624.\n2. O'Donoghue, Brendan, Ian Osband, and Catalin Ionescu. \"Making Sense of Reinforcement Learning and Probabilistic Inference.\" In International Conference on Learning Representations. 2019.\n", "summary_of_the_paper": "# Summary & Contributions\n* This paper examines a variational approach to reinforcement learning, leveraging occupancy measures over previously visited and future state-action pairs in order to address the exploration challenge.\n* The author propose a variational approximation to the so-called \"conditional occupancy\" of the current behavior policy denoting the distribution over future state-action pairs. The approximation is used to induce broader state visitation from the behavior policy, coupled with a standard actor-critic algorithm.\n* Empirical results show considerable performance gains in simpler continuous control problems and matching performance against baseline methods in more high-dimensional control tasks.", "summary_of_the_review": "# Final Remarks\n* As evidenced by my main review, I have struggled to identify strengths of this paper, in large part because the precise nature of the technical contribution and how it improves upon prior work is unclear to me. \n* Still, the empirical results are at the level of a preliminary workshop paper and more experimentation seems warranted across a broader set of environments to confirm the utility of the proposed approach. \n* The clarity issues raised above suggest that the paper requires substantial revision in order to more clearly convey the contributions and algorithms to readers.\n* For now, my score is quite low though I would be happy to raise it in light of suitable clarifications from the authors.\n\n======= Post Rebuttal =======\n\nI thank the authors for their response but the lack of clarity around the approach itself and originality as well as issues with the writing style point to substantial revisions that are needed before the submission is ready for publication.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XXsOfrnfHB-", "reviewer": "Reviewer_6SWm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I found the organization and clarity of the paper to be poor. Consequently, it wasn't clear to me what the theoretical contributions of the paper were. As far as I could tell, 2.1-2.6 were background and a refactoring of notation/existing ideas, and it wasn't clear to me how novel 2.7 is compared to similar methods in the literature. I provide some more detailed comments on how the clarity can be improved at the bottom.\n\nThe empirical evaluation is also lacking. In short:\n- Missing baselines. The authors don't compare against other methods which aim to maximize state entropy.\n- Poor reproducibility. Missing # of trials/seeds. Visually it seems there is some shading which suggests multiple seeds, but this information is absent. Weirdly, this error term disappears over average reward and cumulative rewards. There's per environment hyperparameters and the graphs seem to finish at arbitrary time steps, suggesting these results have been overly tuned to outperform the baselines. Code is not provided (although promised), and the method is not reproducible from only the details given in the paper. \n- Missing analysis. Does the method improve state entropy or add to exploration? There's also no ablation study or any addition insight about the empirical properties of the algorithm.\n\nI've scored my confidence as 4 because I am confident in my score, given the current state of the paper, however, I am not confident I have fully understood all the details of the paper. \n\nSuggestions for clarity:\n- $q_\\pi$ notation is never defined. $K$ in section 2.4 is never defined. \n- Section 2 should be split into background and a section which defines contributions. I'm not sure Section 2 needs so many subsections, and there is no explanation besides from the title, of what each subsection is meant to demonstrate.\n- Too many recursive definitions. For example Eqn (3) is used solely to define $\\beta \\bar{R(s^+,a^+)}$, which is then replaced with $\\tilde{log} p(s,a)$.\n- It would be easier if any unconventional notation was defined in one place rather than scattered throughout section 2. \n- ~ is used weirdly, for example Q(s,a) is \"sampled\" from the reward function above equation 4. Presumably this is meant to replace expectation but should be defined. \n- Inconsistent notation. S,A are often used but are never defined. Are these meant to be the set of all possible state/actions, as defined by $\\mathcal{S}$ and $\\mathcal{A}$ in 2.1? \n- Problem statement really needs to be split into at least 2, if not more, paragraphs.\n- The full algorithm description should be moved from the appendix into section 3. The current algorithm description is also unclear in details. For example, \"update the critic over all losses\" is unclear which losses are being referred to. \n\nMinor comments on clarity\n- Given average return is undiscounted, I'm not sure average reward is an interesting metric to look at. \n- Ordering environments by difficulty is subjective. Describing the environments in detail can be delayed to the appendix. \n- Typo in 2.1 [0,1[ (should also be [0,1) if we're considering a discounted objective). \n- It would be easier to follow if $S^+$ notation was introduced above the equation at the end of page 3.\n", "summary_of_the_paper": "The authors introduce a RL method for optimizing entropy over states, as opposed to actions. The algorithm is tested on three MuJoCo domains and one classic control environment. ", "summary_of_the_review": "Poor clarity makes the contributions hard to follow, along with weak empirical analysis puts the paper below the bar for acceptance. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2HFCEuomMAy", "reviewer": "Reviewer_FQij", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "First sentence \"to deal\" -> \"dealing\" . \"Open-ended learning\" is not defined, I don't know what this means. What does it mean to \"carry out\" a model?  Page  \" including the curiosity drives\" -> \"including curiosity drives\". Some citations are missing parentheses. The next sentence doesn't make sense. The authors here cite the \"E3D\" algorithm and \"MaxEnt\" with which they compare in the empirical section, but for somebody not familiar with these algorithms, the paper should at least briefly describe how these are different from their proposal for these comparisons to make sense. \"the dynamics of the environment is Markovian\" -> \"the dynamics of the environment are Markovian\", \"The dynamic programming objective is described like:\", is that a definition rather? Page 3, what is a \"set point\"?  \"to seek for\" -> \"to seek\"? \"so as to match\" -> \"to match\".  The paper uses its own language which is not formal or universal, and quite confusing, and introduces many shorthands on top of shorthands. It uses capital letters and lower case letters arbitrary at times, although I assume the regular rule of capital meaning random variables is the intention. What is a \"value sample\"? Waht does the \"future of s_t\" mean? What is K? it is not introduced, I assume the log sum exp/partition/normalization function? In section 2.4 equations do not easily from one from the other, leaving the readers confused at times, and having to do these substitutions themselves.  Why is the $\\rho(s)$ appearing in equation 3. Where does it come from? When you say something contains \"a guess\" about something else, what does that mean in a standard concrete mathematical way? What does $\\bar$ over a function mean? What does $\\tilde$ over a function mean? At the end of page 4., the author say the next objective function is the one considered by Soft-actor critic, but that is not obvious, SAC works in policy space, not over the space of occupancy maps. What is \"a complement term\". Using again $q_\\pi$ not to mean the action-value function, but a stationary distribution, is very confusing. The paper jumps to the variational perspective, but does not explain this interpretation and why it is helpful or provably useful.  Above section 2.7, \"at the light\" -> \"in light\", \"the progress of the training\" -> \"training progress\". Section 2.7, \"goal pathways as modes of the objective occupancy\", what does that mean mathematically, what is the \\emph{objective} occupancy. What is \"a through\" (noun)? What does the \"width\" at which it is influential on the number of pathways\" mean? Next sentence confuses upper case S, A with lower case s,a and reverses them.  What is the \"objective credit assignment\"? it's not clear what this means. On page 6, there is a claim that exploration in this way of entropy maximization helps the policy converge faster to the optimum, but it is not supported by anything (citation, or results inside the paper), \"for the implies to concurrently\" -> \"for this implies to concurrently\". What does \"overfitting\" mean here? we have results in policy optimization explaining slow convergence due to flat regions of the optimization landscape. \"state transition vanish during the optimization...\", what vanishes? Page 7, Sutton(1988) is missing citation parentheses. The last sentence of that paragraph, is probably a hypothesis explored in this paper, but do we know for certain this is the case? The exact algorithm is not described in the paper, the authors pointing to the appendix for the most important aspect of the paper, such as how does one compute the log-occupancy? It also mentions the use of a \"kernel-based density estimation method\" which appears to be fundamental, but this is not explained or discussed. The algorithm implementation or pseudo-code is also not given in the paper. \"Results\" section, \"several benchmark environments\", which? and why those? \"expressed as average return\" -> \"\"expressed as average return\". Page 8, \"comparing with the SAC\" -> \"\"comparing with SAC\". Page 9, contains an interesting description of the games, showing insight. It ends with \"this ripple pattern is not contain in the reward signal\", what does that mean? ", "summary_of_the_paper": "The authors address the problem of exploration in reinforcement learning. They suggest applying the maximum entropy principle over the state space and try to maximize the make the occupancy of the policy as entropic as possible while still optimizing the original goal of maximizing cumulative reward. They claims include an algorithm that does this using a replay buffer and off-policy learning, which improves sampling efficiency, in sparse and dense reward settings, in classic motor control benchmarks.", "summary_of_the_review": "The paper addresses the exploration-exploitation trade-off proposing the maximum entropy principle over the state-space as a solution to this problem. Although a promising avenue, many parts of the paper are confusing, using inside jargon, and decreasing the overall clarity. Some parts of the paper are not supported by results, and the main methods used are referred to the appendix. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: While the main idea of the paper (using a Max-Ent objective on the states of an MDP) was considered interesting, all reviewers raised the problem of clarity of the paper which needs to be drastically improved. While the writing could be improved by the revsion, these concerns could also not be fully alleviated by the rebuttal of the authors. The reviewers agreed that the paper needs rewritting in order to clarify the contribution before the paper can be published.
Decision: Reject