Predictable Reinforcement Learning Dynamics through Entropy Rate Minimization
Authors: Daniel Jarne Ornia
Giannis Delimpaltadakis
Jens Kober
Javier Alonso-Mora
Venue: Accepted by TMLR ()

Abstract: In Reinforcement Learning (RL), agents have no incentive to exhibit predictable trajectories, and are often pushed (through e.g. policy entropy regularisation) to randomise their actions in favor of exploration. This lack of predictability awareness often makes it challenging for other agents and humans to predict an agent's trajectories, possibly triggering unsafe scenarios (e.g. in human-robot interaction). We propose a novel method to induce predictable trajectories in RL agents, termed Predictability-Aware RL (PARL), employing the agent's trajectory entropy rate to quantify predictability. Our method maximizes a linear combination of a standard discounted reward and the negative entropy rate, thus trading off optimality with predictability. We show how the entropy rate can be formally cast as an average reward, how entropy-rate value functions can be estimated from a learned model and incorporate this in policy-gradient algorithms, and demonstrate how this approach produces predictable (near-optimal) policies in tasks inspired by human-robot use-cases.

Published in Transactions on Machine Learning Research (06/2025)
Predictable Reinforcement Learning Dynamics through Entropy Rate Minimization
Daniel Jarne Ornia* University of Oxford
danieljonneemiieoocs.aaakk
Giannis Delimpaltadakis*
Eindloven Universily of Technology
Jens Kober
Delft University of Technology
Javier Alonso-Mora
Delft University of Technology
Reviewed on OpenReview: https: /ttps://openpeee opt.r/vee....ee...om net/ forum? DDUsc ID27
Abstract
In Reinforcement Learning (RL), agents have no incentive to exhibit predictable trajectories, and often pushed (through e-g. policy entropy regularisation) randomise their actions in favor of exploration. This lack of predictability awareness often makes it challenging for other agents and humans to predict an agent's trajectories, possibly triggering unsafe scenarios (e.g. in human-robot interaction). We propose a novel method to induce pre- dictable trajectories in RL agents, termed Predictablity-Awarr RL (PARL), employing the agent's trajectory entropy rate to quantify predictability, Our method maximizes linear combination of a standard discounted reward and the negative entropy rate, thus trading off optimality with predictability. We show how the entropy rate can be formally cast an av- erage reward, how entropy-rate value functions incorporate this in policy-gradienl algorithms, and demonstrate how this approach produces predictable (near-optimal) policies in tasks inspired by human-robot use-cases.
be estimated from learned model and
1 Introduction
As Reinforcement Learning (RL) (Sutton & Barto 2018) agents deployed to interact with humans, it becomes crucial to ensure that their behaviour! predictable. A robot trained under general RL algorithms operating in a human environment has no incentive to follow trajectories that are casy to predict. This makes it challenging for other robots or humans to forecast the robot's 's behaviour, affecting coordination and interactions, and possibly triggering unsafe scenarios. RL algorithms are oblivious to the predictability of behaviours they induce in agents: one aims to maximize an expected reward, regardless of how unpredictable trajectorics taken by the agents may be. In fact, many algorithms propose some form of regularisation in
"Equal contribution.
Work done while at Delf University of Technology.
We agent behaviour refer throughout this work, the state trajectorics agents exhibit (and observers may perceive). We consider 'agent bchaviour' 'agent trajectory' interchangeably, but that mainly focus state prodictability. We make the case that agent predictable if their next fixed palicy) easy predict given their past state(s) for standard inference hms..

---

Published in Transactions on Machine Learning Research (06/2025)
value functions (Pitis et al. 2020 Zhao et al, 2019 Kim et. al. 2023 for better exploration, inducing higher aleatoric uncertainty in agents' trajectories.
action complexity (Schulman et al. 2017 Han & Sung 2021
We quantify predictability of an RL agent's trajectories by employing the notion of entropy rate: the infinite-horizon time-average entropy of the agent's trajectories, which measures the complexity of the trajectory distributions induced RI. agents. Higher entropy rate implies more complex and less predictable trajectories, and vice-versa. Similar information- ries have been widely used to quantify (un)prodictability of stochastic processes Shannon 1948
met-
Savas et al. 2022 Biondi et al. (2014) Duan et al. (019)) Stefansson & Johansson
2021
Motivation In general, RL agents are oblivious to the infor- mation theoretic loads they generate with their behaviour. In a world where agents do not exist in a vacuum (even if we train RL agents in single-agent settings, these agents will rarely be deployod in isolation), one could argue there is an advantage to inducing complerity awareness in RL agents; an agent can solve task generating lower information rates, it should do Lower information rates correspond (intuitively and formally, through forms of entropy) with lower uncertainty. However, we do not argue that this is a necessary feature in all agents (or even always desireable). We simply argue that an interest- ing feature to consider for general RL agents that can benefit the deployment of RL agents, propose formal approach to tar- get this, and evaluate how this impacts such agents in different settings.
PA-RE
F1
Figure 1: Qualitative representation of PARL. Trajectories represented symbol- ically connecting initial with final set of states. PARL shifts the policies to- wards smaller trajectory entropy.
Entropy and Predictability The established notion for predictability in information theory is entropy: a lower-entropy stochastic process is easier to predict by standard inference al- gorithms. For example, assuming the trajectory distribution is Gaussian, lower entropy is equivalent to lower variance. Then it is clear that (inside the family of normal distributions) inference algorithms would be able to predict more accurately a lower variance distribution. The same applies to other distribution families, including discrete distributions. In the limit, minimum entropy implies that the agent follows the same deterministic trajectory over and over, which means it can be predicted with prediction crror from inference on past data (since it is delerministic).
Contributions
We propose novel approach to model-based RL that induces more predictable behaviour in RI. agents, termed Predictabilily-Aue Aware RT (PART.). We maximize the linear combination of standard discounted cumulative reward and the negative entropy rate, thus trading-off optimality with predictability. Towards this, we cast entropy-rate minimization as an expected average reward minimization problem, with a policy-dependent reward function, called local entropy. To circumvent local entyo'y's 's policy-dependency and enable the use of on- and off-policy RL algorithms, we introduce a state-action- depeddent surrogate entropy, and show that deterministic policies minimizing the average surrogate entropy exist and also minimize the entropy rate. Further, show how, employing learned model and the surrogate reward, entropy-rate value functions, and incorporate this in -grediggradient schemes. Finally, we showcase how PARL produces much more predictable agents while achieving near-optimal rewards in several robotics and autonomous driving lasks
estimate
1.1 Related work
The idea of introducing some form of entropy objectives in policy gradient algorithms has been extensively
explored Williams & Peng [991]: Fox et al. (2015 Peters et al. (2010); Zimin & Neul (2013); ; Neu et al.
28e the project repository https:/ hagpostoryyttpps/gtttub...m
for details.
2

---

Published in Transactions on Machine Learning Research (06/2025)
(2177]: |: Tiapkin et al. 2023)). In most instances, these regularization terms are designed to either help policy randomization and exploration Mutti et al. (2021] 2022), or to stabilize RL algorithms. However, these works focus on policy (state-action) entropy maximization, and do not focus on trajectory entropy and how
it affects prodictability of RL agents.
Instead, Guo et al. (2023b ap: Biondi et al. 2014); George et al. (2018): : Duan et al. (2019): Stefansson &
Johansson (2021); Savas et al. (2020) 2022) consider entropy (rate) maximization in (PO)MDPs, to yield umpredictable behaviours. However, these works require full knowledge of the model, and entropy (rate)
maximization is cast non-linear program. Instead, in our work, the model is not known, and we resort to learning entropy-rate value functions.
Recent work Lu et al. (2020); : Eysenbach el al. 2021]: :: Park & Levine (2023 has tackled robustness and
generalization via introducing Information-Theone Theoretic penalty terms in the reward function. In particular,
Eysenbach et al. 202T makes the explicit connection from such information-thcorttic penaltics to the emer- gence of predictable behaviour in RL agents, and uses mutual information penalties to restrict the bits of information that the agents are allowed to use, resulting in simpler, less complex policies. We address di- rectly this predictability problem by the minimisation of entropy rates in RL agents' stochastic dynamics. This allows us to provide theoretical results regarding existence and convergence of optimal (predictable) policies towards minimum entropy-male agents, and make our scheme generalizeable to any RI. algorithm (on and off policy). In this line, Berseth et al. (2021) propose Surprise Minimizing Reinforcement Learning, where an estimate of the state stationary distribution of visited states is kept, and the agents are penalised for visiting states with low probabilities in this stationary distribution. While having a similar flavour, our work addresses state trajectory entropy, contrary to stationary distribution entropy, since we aim to address the predictability of the agent's ss data generating process (and not simply avoid unknown and changing en- vironment regions]? 3 Finally, our work is tangentially related to alignment and interpretability in RL Shah et al. 2019 Carroll et al.] 22019). where human-agent interaction requires exhibiting nuuma--nterppettblle behaviour. Further, work on legibility of robot motion Dragan et al. (2013) Liu et al. (2023): Busch ct al. (2017) shares our motivation; to make robotic systems behaviour more legible by humans.
2 Background
We, first, introduce preliminary concepts employed throughout this work. For more detail on Markov chains and decision processes, the reader is referred to Putermann (014)). Given set A, ((A) denotes the probability simplex over A, and A denotes the k-times Cartesian product A x x ... x A. If A is finite, [A] denotes its cardinality. Given two probability distributions J, v, we use Drv(ullu) as the total variation distance between two distributions. We use supp(n) to denote the support of jr. Given two vectors &, we write & 2 n, if each i-th entry of & is bigger or equal to the i-th entry of n)
2.1 Markov processes and Rewards
A Markov Chain (MC) is a tuple C - (X', P. no) where X' is a finite set of states, P: x - [0, is transition probability measure and μa0 E ((x) is a probability distribution over initial states. Specifically, P(z,g) y) is the probability of transitioning from state x to state y. P'(z, y) is the probability of landing in y after t time-steps, starting from x. The limit transition function is P* == lim pe, We use uppercase X, to refer to the random variable that is the state of the random process governed by the MC at time 1, and lower case to indicate specific states, e.g- E X. Similarly, trajectory or path is sequence of states x = {Io, ), where I; E X, and Pk denotes the set of all (k + 1)-legtth paths. We also denote Xr. = [X,, Xe+1, X.). Further, we define p: Prx + [0, 1] as a probability measure over the Borel o-algebra ((xo,))) of infinite-enggth paths of a MCH conditioned to initial distribution po. For example, p(X, = x) = no(x) is the probability of the initial state being x; p( (Xxx: X3) is the probability that the MC's state follows the path X3. Markov Decision Process (MDP) is a tuple M = (X, P, R, Ho) where X is a finite set of states, u is a finite set of actions, P X x U x X + |0, 1] is probability measure
3 simple counterexample can show that an agent can have (almost) zero trajectory entropy and have high stationary distribution entropy; think of two connected states B where A has one self loop with near to zero probability. Thhi measure well defined by the Tonescu-Tull Theorem, see e.g. Dudley (011))
3

---

Published in Transactions on Machine Learning Research (06/2025)
of the transitions between states given action, R X' x U x X -R is bounded reward function and M40 E ((2) is the probability distribution of initial states. A stationary Markov policy is a stochastic ic kernel т : + A(U). With abuse of notation, we use T(u z) as the probability of taking action u at state under policy т. Let II be the set of all stationary Markov policics, and IT S II the set of deterministic policies. The composition of an MDP M and a policy x generales a MC with transition probabilities (x( == Eueu i(u | P(r, u, y)- If said MC admits a unique stationary distribution, we denote it by u², where μ² -X- + [0, 1]. We, also, use the shorthand Ro Buwn(w) [R(X,, U, X We will assumee any fixed policy x in MDP M induces aperiodic and irreducible MC.
In discounted cumulative reward problems the goal is to find a policy x' that maximizes the discounted sum of rewards for discount factor ry E [0, 1): i.e. x' E arg maxxen B| 7 RT Xo = z),, for all E X. In this case Puterman (2014) given policy म, the value function under n, V X' + R. is V E RR, Xo = 5.. The aclion-value function (or Q-function) under is given by (z, = Eyex P(x, u. ()( u, (z))
Discounted Reward MDPs
Average Reward MDPs In average reward maximization problems, we aim at maximizing the reward rale (or gain) gr (2)) defined as, together with the bias 5
(x) = E
lim TH
R
No = x (x) E lim Σ (R - (0-(g(X))) | No x T
(1)
Note that the bias is the expected difference between the stationary rate and the rewards obtained by initialising the system at a given state. For π E II, the average-reward (action) value-functions Vayg X + R is defined by Abounadi et al. (2001 avg (I) == Eu-z.y-P(,.a.) [R(x, u, u.y) - g" + Varg (v) + V avg
and @ (2, u) == SymP((m,,) [RIz, v), g"
[v)]. The optimal (action) value functions (which ex- ists for ergodic MDPs) satisfy V Vag(()) max P(zaa))) [R(æ,, u, v) -g" Vaig(v]] and Qavg Fang(2, u)
EymP(() [R(x, u,y) - g" + Vavg(1)) (v)] where g is the optimal reward rate.
2.2 Shannon Entropy and MDPs
For a discrete random variable A with finite support A, Shannon entropy Shannon 1948 is a mcasure of uncertainty induced by its distribution, and it defined as 1(AA) == - -mne Pr (A = log(Pr (A a)). Shamnon entropy measures the amount of information encoded in a random variable: a uniform distribution maximizes entropy (minimal information). and Dirac distribution minimizes it (maximal information). Recall p: : Po [0, 1] is a probability measure over the Borel o-algebra of infinite-length paths of MC. Let us denote the dependency of on fixed policy as p" (a fixed policy in a MDP induces MC For an
MDP under policy т, recall Section 2.1 Then we define the conditional entropy (Cover 1999
Biondi et al.
2014 of XT+1 given X as:
A (X Xo.T) :- Σ D" (XT+1 No. xT) (XT+1 Xo. = XT)),
VERXPERT
and the joint entropy F of the path Xo.r is hTt (XuT) h(Xo) Σt 1x Xoa-1)-
Whenever the limit erists, the entropy rate of an MDP M under policy и defined by h" limm th" (Xo.).
Definition 1 (Entropy Rate Shannon 1948
The entropy rate represents the rate of diversity in the information generated by the induced MC's paths. Smaller entropy rates imply more predictable trajectories of the induced MC.
TTee bias can also be written in vector form as b b= (1 P--P++P'- P*) |
P*) where R E B| is the vector of state
rewards, R3 = -Ea-x[As [R(r, u. w)]. See Chapter 8 in Puterman (2014).
"Observe that, for argodic MDPs, 5" (x) vw [(y
Thee second equality obtained by applying the general product rule the joint probabilities of Dr-
4

---

Published in Transactions on Machine Learning Research (06/2025)
3 Entropy Rates:
Estimation and Learning
Problem Statement The problem considered in this work is the following, Consider an unknown MDP M (X, P. R. no). Further, assume that we can sample transitions (x, u, R(x, u, applying any action u E U and letting M evolve according to P(x,u, u, We want
T E arg max E TEll
> RRR - kh"
(2)
t=
where &: > 0 is a tuncable parameter S In words, we are looking for policies that maximize a tuncable weighted linear combination of the negative entropy -h and standard expected discounted cumulative reward. As such, we establish a trade-off, which is tuned via the parameter k, between enlropy rale minimization (i.e. predictability) and optimality w.r.t. the cumulative reward.
Proposed approach We first show how the entropy rate h" can be treated as average reward criterion, with the so-called local entropy 1" as its corresponding local reward. Then, because 1a is policy-dependent, we introduce a surrogate reward, that solely depends on states and actions and can be learned in-the-loop. We show that deterministic policies minimizing the expected average surrogate reward exist. and also minimize the actual entropy rate. Moreover, we prove that, given a learned model of the MDP, we are able to (locally optimally) approximate the value function associated to the entropy rate, via learning the surrogate's 's value functions. Based on these results, we propose a (model-basex D RL algorithm with its maximization objective being the combination of the cumulative reward and an average reward involving the surrogate local entropy.
Estimating Entropy Rates Towards writing the entropy rate hT as an expected average reward, let us define the local entropy, under policy æ,, for state E X' as
1s (x) := - Σ Px v) log P=(x,y).
(3)
BEX
Now, making use of the Markov property, the entropy rate for an MDP reduces to
--= lim h(Xo) + Σ (Xe Xxx--))
1 ((Xo)) + Σ 1" Xx-i)
lim T-x T
(4)
T To0 T
Observe now, from the definition of nd and again the Markov properly,
h"t h² Xx-1)= = Σ
p" = y, X, x)) log (X+ x
xE
=- Σ ((X (X, = Xx-1 = =
Σ VEX
(x log (z
Therefore, subslituting in equation 4
T
T Σ FX.)]. T 1=0
1 T
h-
lim +00
Σ
E (X+)
E lim
(5)
T
We can treat the local entropy I under policy π as a policy-dependent reward (or cost) function, since ! is stationary, history independent and bounded. Thus, h is treated expected average reward, with reward function 1"
WW choose to cast the problem as maximization of linear combination of objectives to allow agents to find efficient trade-offs. This problem could be solved through e.g. multi-objective optimization methods Skalse al. 2002 Hayes et al.) 2022). We leave this an application-deeeeeeett choice.
We the term model-based, since require learning (approximated) representation of the dynamics of the MDP to estimate the entropy. However, the choice whether to improve the policy using pure model free algorithms versus using the learned model left a design choice, beyond the scope of this work.
5

---

Published in Transactions on Machine Learning Research (06/2025)
3.1 A Surrogate for Local Entropy
In conventional RL seltings, one is able to sample rewards (and transitions, e.g. from a simulator). However, here, part of the expected reward to be maximized in equation 2 is the negative entropy -h", which, as aforementionod, can be scen as an expected average reward with a state- and policy-dependent local reward 1" (x). It is not reasonable to assume that one can directly sample local entropies 1" (x)): one would have to estimate 1a (z) through estimating transition probabilities (= (by sampling transitions) and using equation 3 However, a new challenge arises: 1 depends on the action distribution and to apply average reward MDP theory need the rewards to be state-action dependent. To address this, we consider surrogate for 1st that is poliey-independent:
s(z, u) = Σ
P(x, u, y. log (P(x, u)))
NEX
Define (x) = -i Ei (xx,
(X.)) = =d]. The following relationships hold.
The sollowing statements hold. a) (x)x()) u)] S (z),, for all II b) (x) hg h- for some bg R, for all x E II. c) (a) =)]=", (x) and h =", for all x IID.
Lemma 1. Consider MDP M
P, R.
Observe that, by considering the surrogate entropy rate, we effectively decouple the influence of the policy entropy in the entropy rate estimations. Policy stochasticity directly influences the true entropy rate h", but does not affect the surrogate entropy rate hg; in a fully deterministic environment, Ag = 0 for all æ E II, but he would not. We make the case that, given the formal results in Lemma (and the results to be presented in coming sections) this effect does not degrade the effectiveness of our method; fact, it allows agents find less uncertain environment regions while not directly discouraging exploration (and can still render minimum entropy rate policics, discussed in Theorem 1 below).
3.2 Minimum Entropy Policies
Based on Lemma 1
derive one of our main results. For the proof, we make use of fundamental results existance of average reward optimal policies of MDPs. In particular Theorem 3 included in the Appendix, applied directly from Puterman (014)), which states that under mild assumptions the gain and bias exist in average reward MDPs for any policy, and that an optimal policy exists that maximises the reward rate.
Theorem 1. Consider MDP M = (X,U, R. po)- The following hold: a) There erists a deterministic policy ã E _1 ménimizing the surrogate entropy rate i.e. â E arg min, hg. b) Any mizing the surrogate entropy rate also mindmizes the true entropy rate: â E arg min h and tt II" î arg min h²" Additionally, deterministic policies locally minimizing h also locally minimize h c) There crists a deterministic policy at E ITP such that ft E aargmnn min, ht
11 mini-
Proof of Theorem 1 The first statement follows directly from Theorem in the Appendix, which guarantees that there is at least one deterministic policy & that minimizes the surrogate entropy rate h5. Then, since f E I1,, from Lemma statements and we have that the following holds for all π E II:
-
hg hM
Thus, A minimizes ha and it follows that # arg min h and E II" = Å E arg min h². The same argument also applies locally, thereby yileding that deterministic local minimizers of hy are also local minimizers of hT Finally, the third statement follows as combination of the other two.
Theorem 1 is an utterly relevant result for our work. First, it guarantees that minimizing policies both for hy and hMd exist. More importantly, it tells us that, to minimize the entropy rate of an RL agent, it is sufficient to minimize the surrogale entropy rale. Since (globally) minimizing hy implies minimizing he and since s is bbiey-indepennden, in contrast to 1", in what follows, our RL algorithm uses estimates of s to minimize hg. instcad of estimates of 1xt to minimize h 10
10 Observe that we cannot employ the same method for entropy rate maximization, since the maximizer hg is not necessarily maximizer ht.

---

Published in Transactions on Machine Learning Research (06/2025)
4 Learning to Act Predictably
In the following, we show how predictability of the agent's behaviour can be cast as an RL objective and combined with a primary discounted reward goal. To do this, we rely on Theorem [and employ the surrogate entropy s(x, u) as a local reward along with its corresponding value function. We prove that, given a learned model of the MDP, we are able to approximate the true entropy rate value functions. In the next section, we combine this section's results with conventional discounted rewards and standard PG results, to address the problem mentioned in the Problem Statement and derive PG algorithm that maximizes the combined reward objective. We define the predictability objective to be minimized:
E xx))) T-000
(x =
hg
lim
T
Motivated by Theorem we have employed the surrogate entropy as a local reward and consider the corresponding average-reward problem. As commonly done in average reward problems, we define the (surrogate) entropy value function for a policy π, W x R to be equal the bias, i.e.:
(T) =E Σ (xx, hT Xu = x = Ey- (a,) [s(z, - h2 + (w)],
(6)
Additionally, define the (surrogate) entropy action-value function S X x u + R by ST ST == Ey-P, (x,u) [s(2, u) - h + (w)]. However, recall that we do not know the local reward s. To estimate 8, one needs to have an estimate of the transition function P of the MDP. We use
= =- Σ Po(x, log (Po v,
DEX
(and hy correspondingly, for its associated rate) to denote the parameterised by á estimate of s, which results from corresponding estimate Pa of P (i.e. Pe is the learned model). Similarly, we will use Jag. W and Sg to denote value functions computed with the model estimates $$- Now, it is crucial to know that by using the model estimates 86 we are still able to approximate well the objective J, and the value functions W Let us first show that for small error between Pe and P (i.e. small modeling error), the error between &o and and the objectives (x(T) = hig and Jsg (x) = hig is also small.
Proposition 1. Consider MDP M (X,U, P. Ro).). Consider Pe X x U x X - [0, 1], parameterised by o E E. Assume that the total variation error between Po and P is bounded as, V x E X and u E U,
mEE Drv (Po(Pa u, I(x,, u, .) S e, for some with O S e 1. Let K(E) = e log (1x1) - e log Then, [|5g(x, u) - s(x, =0 S K(e), and the surrogate entropy rate error for any policy n,
E
lim toc
1
Σ
solXn, ((((x, Xo po] -al-hh-- hg s K(e).
T
The proof of proposition (1973); [Addenarrt (2007). where we simply assume the density matrices are diagonal matrices with the transition probability densitics cigenvalues. Observe that as c- 0, i.. the learned model approaches the real one, then the surrogate entropy rate converges to the actual ond (since K(+) + 0). This result indicates that we can indeed use 8a, obtained by learned model Pg, instead of the unknown 8, as the error between the objectives JT(T) = hg and Jay (T) = hT, is small, for small model errors. Assume now, without loss of generality that we have parameterised entropy value function (crilic) S., with parameters E Ω. We show that a standard on-policy algorithm, with policy म, with value function approximation S,, using the approximated model Ps, learns entropy value functions that are in o(ch-noighbourhood of the truc entropy value functions STh and 8(e) vanishes with e.
0 hinges on the Fannes-Audenaee, inequality for Von Neumann entropies Fannes
This result echoes the Simulation Lemma in Kearns Singh 2002.. but with bound derived in infinite horizon by using the entropy properties.

---

Published in Transactions on Machine Learning Research (06/2025)
Assumption 1. Any learning rate au 1) satisfies Ei=o a4 00, Ei=0 of < 00.
Assumption 2. The model Ps satisfies max (rv(Po u, (I,,, u) c for a small c E [0, Proposition 2. Consider on MDP M, a policy v, a learned model Po of the MDP and critic Sw linear on w, and w C R", where Q2 is compuct. Let Assumption 2 collect & trujectories Tk of length T, and construct (unbiosed) estimates 12 Sz. Let the critic parameters w E Ω be updated as Wk-1 = wk=- - Bk Awkk with wa E 52, 5x being a learning rate and
hold. At every step k of parameter iteration, let
DWn = (S un) - (1k, ( uk)
dw
Then,
W converge
5(c) 8(c)-neighhourtood of one of the (local) minimizers of where 5(c) is vamishing with
E 200p (2)
1 S5 (x.
S ux))
w))²|
In other words, for small model errors, the value function approximator converges to locally optimal value function approximation of the true value function S
4.1 Predictability-Aarre Policy Gradient
Now, we are ready to address the Problem Statement, combining the entropy rate objective with discounted reward objective. In what follows, assume that we have a parameterised policy тe with parameters 0 E O. Let J(TO) = E 7t 1. We use Qş with parameters Ç E for the parameterised critic of the discounted reward objective (when using a form of actor-critic algorithm)..
Theorem 2. Consider an MDP M, parameterised policy ne, learned model P& of the MDP and (linear) 2 hold. Let a given PG algorithm marimize (locally) the discounted reward objective J(To) B| R, Let the value function QE (or Ve)) be parameterised by & E, and the entropy value function S., (or Wa) have the same parameterisation class. Then, the same PG algorithm with updates
critic S.. Let Assumption
4 proje (0 (VoJ(Te) kV kV (VoJ)((TT)) (TT)
converges to a local mazimum of the combined objective J (TTe kJso (Te).
Proof of Theorem (Sketch). By standard PG arguments Sutton et al. 1999). if PG algorithm converges to local maximum of the objective J(Te) then the updates (Te( are in the direction of the gradient (up to stochastic approximation noise). By the same arguments, given Proposition 2 the same algorithm converges to a local minimum of the entropy value function W., through updates - (Te) and these are in the direction of the true gradient (again, up to stochastic approximation noise). Then, the linear combination of gradient updates (me( - (Ta) is in the direction of the gradient of the combined objective J(TT)= k.J, (To). Finally, since both objectives are locally concave (necessary condition following from existence of gradient schemes that locally maximize them), their linear combination is also locally concave. This concludes the proof.
]
Remark 1. Regarding pure entropy rate minémization, f.e. without the discounted reward objective, already proven by Theorem 1. policy that is globally optimal for the surrogate entropy rate J.(T) is also optimal for the actual entropy rale h" The same holds for locally optimal deterministic policies. However, in general, this is not the case for stochastic local minimizers.
Following a vanilla policy gradient structure, in Algorithm | we first sample trajectory 7 of length T, under policy To, and store it in buffer D (for training the approximate model P&). Then, use D to train Psi update the estimated entropy rate; compute estimated objective gradients (oJ(T))) VoJe, (ne) from trajectory [ and finally update the policy and critics S2, Qt-
Remark 2. If we were to consider average reward MDPs instead of discounted reward MDPs, the formulation of the optimization problem solved in. Theorem results in a more natural interpretation when adding entropy rate objectives. See Appendir for details.
12 e.g. TD(0) value estimation Sutton & Barto 2018
13this can be done through any policy gradient algorithm at choice.
8

---

Published in Transactions on Machine Learning Research (06/2025)
4.2 Implementation
Policy Learning We implement our predictability- aware scheme as first, an on-policy version based on average-reward PPO algorithu Ma et al. 22021 we call this PAPPO. In particular, for every collected tra- jectory T, we update the estimate hr, and (surrogate) entropy value function Wa parameterised by w E from the collected samples, and we compule entropy advantages for all (z, u, y, s2(2, u)) in the collected tra- jectories as: A7 = s2 -) + (w) (w) ( (2). Then we apply the gradient steps as PPO SShhlmmum et al.] (2015) 2017) by clipping the policy updates. Second, we implement it in an off-policy fashion to compare with recent results
Algorithm 1 Predictability Aware Policy Gradi-
ent.
Require: Pa. Te, critics Ww, VE
Require: 0u, k k> 0
for E epochs do
D t Trajectory T of length T.
Train Ps from D.
h + EoNNEr so(2, u).
Compute VoJ(TO), VoJas (TT) from T. 0 proje [ + or (VoJ(TT) kVoJ VoJ. (Ta)
Update S (and Qg if used)
entemation-theorete RL, based Soft Actor-Critic (Haarnoja et al., 2018) implementa- tion; we call this PASAC. The only modification necessary is the learning of a second Q function (5) for the surrogate trajectory entropy, and the policy loss is computed as a weighted sum of Q and S. See Appendix for details on PASAC.
end for
Model Learning To learn the approximated model Pa. we assunne the transitions to follow Gaussian distributions, similarly to Janner al. (2019 14 Following the definition of differential entropy of a con- tinuous Gaussian distribution, sa(z, u) = logio an) + K, where K 1/2 [log(2x) + 1). Therefore, we estimate the entropy directly by the variance output of our model. Furthermore, since we only need to estimate the variance per transition (xr, u) + y, it is sufficient to construct a model So : X' x + X that approximates the mean fo(2, u) se yP(z, u. y)dy, and we do this through minimizing a mean-squared er- ror loss of transition samples our model. Then, we estimate the entropy of an observed transition as y)
(1 u) log(Ey--P(nnn)) (ff (o
Entropy Rate Computation Given trained agent, the entropy rates are estimated equally for all algorithms. At inference (when rolling out the trained agent), given the trained model fo, the entropy rate for a trajectory {(X,, (Xe))losssTT of length T is estimated as hT IT= (og -X44))))
5 Experiments
We implemented PARL on set of robotics and autonomous driving tasks, evaluated the obtained rewards and entropy rates, and compared against different baselines. For the MuJoCo hyperparameters, we took pre- tuned values from Raffin et al. (2019). For the experiments using PASAC and explicit comparison against other SAC-based baselines including RPC (Eysenbach al. 2021 see Appendix C.I
15
Rewards, Entropies and Ablation To evaluate the influence of the trade-off parameter k, test PAPPO on MuJoCo tasks and compare to on-policy baselines. We train all agents using the same hyper- parameters, and we only vary the trade-off k in the PARL agents to evaluate the influence. Additionally, we run both the deterministic and stochastic resulting policies (deterministic chooses the mean action that comes out of the policy, stochastic samples from it). The results are reported in Figure Note that, as Mujoco tasks have continuous state and action spaces, entropy rates may be negative
1hss is strong assumption, and in many multi-modal problems, it may not be sufficient to capture the dynamics. Note, however, that our method is compatible with any representation of learned model P.. We have also designed two additional representative robotic tasks where agents PARL to avoid unnecossary stochusticity in the enviroument. See the Appendix for these.
Recaall that, for our implementation, predictability for continuous random variables, which may indeed negative. A different metric one may use is relative entropy (the KT-ddivergence from the uniform distribution).
address contimuuus state/action tasks, we differential entropy quantify
9

---

Published in Transactions on Machine Learning Research (06/2025)
Un Determ. | Stoch.
D Determ. Stoch.
[ Determ. I Stoch.
Rewards (k's)
Rewards (k's)
Rewards (k's)
PPO
4: = 0.05
k ==0, 0.1
== 0.25
k = 0.5
PPO
k 0.05
k 0.1
0.25 0.25
k 0.5
0.5
PPO
k 0.05
0.05
k 0.1
k= 0.25
k == 0.5
k
Determ.
Stoch.
Determ. I Stoch.
5
2
Entropy Rate
Entropy Rate
Entropy Rate
0
0
5
-2
-2
10
1 Determ. In Stoch.
PPO
==0000 0.05
0.1
0.25
0.5
PPO
k ==0.0
0.05
0.1
0.1
0.25
==0
0.5
PPO
k 0.05
k ==111
k 0.25
==
k=
k=
k=
(a) Walker
(b) Aut
(c) HallCheetali
Figure 2: Rewards and Entropy rates for PAPPO as a function of k. Leftmost is PPO baseline.
The effect of entropy-rate mininization and PAPPO trajectory distributions is showcased in Figure 3 We evaluated trained agents over 10 full episodes, and plotted the observed trajectories in task space to compare trajectory distributions. We plot as representative variables the z-position and angle of the front tip of the robot. In both cases, we observe that PAPPO policies induce more regular, clustered trajectories, which suggests smaller distributional complexity. Generally, PAPPO trajectories have considerably smaller variance, and visit a smaller portion of the state-space. In the Walker and Ant environments this difference is very pronounced, as PAPPO trajectories are concentrated in tiny region of the state-space, and especially in terms of the angles observed, are limited to much smaller range.
Trajectory Complexity in MuJoCo
Predictable Driving We test PAPPO in the Highway Environment (Leurent] 2018), where an agent learns to drive a desired speed while navigating a crowded road with other autonomous agents. The agent gets rewarded for tracking the desired speed and penalised for collisions. We consider highway and roundabout scenario (see Figure 4. We compare against PPO (Schulman et al. 2017) and DQN Mnih ct al. (2015)) agents, and take the hyperparameters directly from Leurent (2018). The results are presented in Table 1 In the highway enviromment, agents slow down their speed and stop overtaking (arguably, a more predictable driving pattern). This results in longer episode lengths (and larger episodic rewards), but lower rewards per time-step (since reward is given for driving faster). In the roundabout scenario, a different behaviour emerges: Agents keep a constant high speed to traverse the roundabout as fast as possible, as the roundabout is the main source of complexity.
10

---

Published in Transactions on Machine Learning Research (06/2025)
122
0,315
6.33
12.
0,725
61-
0.02
patton
6.2
60-
-01-
-62-
6.35
-63-
0.003
16
2.4
1/
6.
da
13
14
-54
-1.75
600
cás
0.50
t.5s
1ede 1011
regle
3.2
6.3
6.-
3.2
6.2-
2.-
61-
60
-11
-6.2
-62-
6.
-4.5
7.6
5.5
1.0
3.2
16
1.8
02 3.6
3.0
1.0
12
-5.
00
-1.75
-0.3
-2.2
0,00
0.25
030
5.7
17
ande mod)
zegle mdd)
angle 7o0
(a) Walker
(b) Ant
(c) HalfCheetah
Figure 3: 10 trajectory plots for agents with seed 0. Blue is PPO, orange is PAPPO with = 0.5.
Table 1: Results for autonomous driving environments.
Highway
Rewards
Ep. Length
Avg.
Speed
Entropy
Rate
DQN
17.51
7.73
20.92 + 9.08
6.51
0.34
-0.37 + 0.16
PPO
18.88 1 6.74
24.41
1 8.49
5.62
1 0.10
-0.88
1 0.05
PAPPOR=0.1
21.05 4.03
28.56 + 5.27
5.11
0.05
-1.43
+ 0.09
PAPPO
21.03 = 2.05
29.60
+ 2.64
5.06
0.01
-1.51
= 0.09
PAPPO
20.89 + 2.05
29.61
+ 2.58
5.05
0.00
-1.51
+ 0.06
Roundabout
Rewards
Ep. Length
Avg.
Speed
Entropy
Rate
DQN
22.53 + 11.93
26.92 + 14.41
3.09
+ 0.47
-0.55 + 0.88
PPO
29.26 +
32.92
11.67
2.80
0.12
-0.23
0.27
PAPPO
28.86 1 10.96
32.98
1 12.23
2.65
1 0.50
-0.28
1 0.77
PAPPO
17.99 + 13.34
23.45
+ 17.60
3.75 + 0.05
-2.13
0.12
PAPPOk=1
16.83
1 13.28
22.05
1 17.67
3.79
1 0.04
-2.21
0.14
6 Average Reward Formulation
Through the work we argue that the entropy rate can be efficiently cast and implemented as an aver- age reward criterion, to be combined with other primary reward objectives. A natural question to ask is how does the formulation of our work adapt to the case where the primary objective is average reward objective over the MDP rewards. Consider the case where we are interested in optimizing a lin- ear combination of average reward and entropy rate objectives. Then, we can write the reward objec- tive as J(T) are taken in expectation over the arg max ((x) - (T)) = [imm T E R ks? where we already included the surrogate entropy rell
limr- 1 Et= R. and since both the average rewards and the average entropies probability space, the trade-off objective in equation ? becomes
sT = s(x, - (x))) Further, recall that s(x, u) = P log P(x, u.y)]. Then, assuming knowlodge of P (or approximation of it), one can define an adjusted reward R(x, u, v) R(x, y) + klog P(x, y) and a modified value function Vir(()) EurT,ggePe...)) [R(r,, v) + A log P(r, y) 3m + V (v)], where 6" E lim 1 Eto Ry is the modified reward rate under policy (which is independent of the initial set of states for ergodic MDPs). Observe that, even though R(æ, u, u. # R(x, u, y) ks(z, u), they are equal in expectation. Then, for any fixed policy, the expected average reward can be computed through R, and
11

---

Published in Transactions on Machine Learning Research (06/2025)
policy n" solving the optimization problem x² E arg max V (z) vx X is guaranteed to solve the average
*ell reward objective. This allows for more compact formulation, and for the learning of a single value function (instead of two), which can be desirable in some cases. also implics that transitions with low probabilities of being observed are penalized. Computationally, this is also advanlageous since we do not need to compute the entropy of the learned model, but instead evaluate the likelihood of the observed transition and adjust the rewards accordingly.
7 Discussion
Summary of Results We proposed a novel method, namely PARL, that induces more predictable be- haviour in RL agents by maximizing tuncable lincar combination of standard expected reward and the negative entropy rale, thus trading-off oplimality with predictabilily. In the experimental results, we see how PARL greatly reduces the entropy rates of the RL agents while achieving near-optimal rewards, depending on the trade-off parameter. In the autonomous driving setting, agents learn to be more predictable while driving around stochastic agents. In the MuJoCo experiments, PARL obtains policies that yield more clus- tered, less complex trajectory distributions, allowing models to predict better the dynamics. This results in, for example, a smaller range of values of orientation angles as seen in the trajectory representations on Figure 3] Additionally, following our method, the entropy rate can be directly interpreted as the average complexity necessary to correctly predict the trajectory of the agents, and if assuming Gaussian predictions, this is proportional to the log of the prediction variance observed by the agent's internal prediction model, which shows how lower entropy rates yield predictability.
Shortcomings Our scheme results in a setting where agents maximize trade-off between two different objectives. This, combined with learning dynamic model (which is notoriously difficult), introduces imple- mentation challenges related to learning multiple coupled models simultancously. We make an attempt at discussing systematic way of addressing these in the Appendix. Additionally, for many applications, avoid- ing ripy policies may restrict the ability of RL agents to learn optimal behaviours (see, e-g-, results in Eysenbach et al. 2021). And although in human-robot interaction and human- aligned Al predictability is intuitively benoficial, we cannot claim that minimizing entropy-rates is desirable for all RL applications. Ad- ditionally, question remains regarding how would our method change in partially observable environments. The entropy rate of a POMDP (if measured on the state sequence, not observations) is influenced by the policy given an observation, observation probability given state, and the state transition, resulting in a much more difficult object to estimate (sce e.g. (Savas et al. (2022)). We believe the theoretical and practical considerations of minimising entropy rates in POMDPs (in model-free settings) will be a very interesting future direction.
Entropy and Exploration One might wonder if minimizing entropy rates of RL agents may hinder exploration. From a theoretical perspective, this is not problem due to ergodicity. From practical perspective, this undesired effect is highly mitigated by the fact that for the first (many) steps, the agent is still learning an adequate model of the dynamics, and therefore the entropy signal is very noisy which favors exploration.
Acknowledgements
Authors want to thank Alvaro Serra, Lasse Peters, Khaled A. Mustafa and Elia Trevisan for the useful discussions on this topic. This research was supported by funding from the Dutch Research Council NWO- NWA, within the " "Acting under uncertainty" (ACT) project (Grant No. NWA. 1tt..29...9.22
References
J. Abounadi, D. Bertsekas, and V. S. Borkar. Learning Algorithms for Markov Decision Processes with Average Cost. STAM Journal on Control and Oplimization, 40(3):681-668) January 2001. ISSN
12

---

Published in Transactions on Machine Learning Research (06/2025)
0363-0129, 1095-7138. doi: 00.1337/80330119993817777 URL http:://pppb....aa...o..o//////o htt.:pppbbb.ssa.aoroo..eeeoeeeee http://s......m.r////////1.1.3.. S03680129993619744
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next- generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD Interna- tional Conference Knowledge Discovery and Data Mining, 2019.
Koenraad M Audenaert. sharp continuity estimate for the von neumann entropy. Journal of Physics A: Mathematical and Theoretical, 40(28):8127, jun 2007. doi: 10.1088/1751-8113 4///133//4////////// URL https: //dx.ddi.oog... /1tp://000000011111111111/////10
Glen Berseth, Daniel Geng, Coline Manon Devin, Nicholas Rhinehart, Chelsea Finn, Dinesh Jayaraman, and Sergey Levine. Smirl: Surprise minimizing reinforcement learning in unstable environments. International Conference on Learning Representations, 2021.
Fabrizio Biondi, Axel Legay, Bo Friis Nielsen, and Andrzej Wasowski. Maximizing entropy over Markov processes. Journal of Logical and Algebraic Methods in Programming, 83(5-6):384 399, September 2014. ISSN 23522208. doi: (..01//..aam..200. 201.....00....11...
Vivek S Borkar. Stochastic approzimation: dynamical systems viewpoint, volume 48. Springer, 2009.
Baptiste Busch, Jonathan Grizou, Manuel Lopes, and Freek Stulp. Learning legible motion from human robot interactions. International Journal of Social Robotics, 9(5):765-77) 2017.
Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. In H. Wallach, H. Larochelle, A. Beygelz- imer, F. d' A'Achhe Buc, E. Fox, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL http:tttpp/oscoeddaeeeeeo neurips....p...... cc/paper sacper_files/peeeee happe// 0t5ss//DD//bb118889997226666666
wwwp:/ffb1122b9aa-aaap-e pdf
Maxime Sheevalier-oiivert Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement lcarning environments for goal-oriented tasks. CoRR, abs/2306.13388 2023.
Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
Anca D. Dragan, Kenton C.T. Lee, and Siddhartha S. Srinivasa. Legibility and predictability of robot motion. In 2013 8th ACM/IEEN International Conference Human-Robot Interaction (HRI), pp. 301-308, 2013. doi: R..1009/HRL. 201...00136488600
Xiaoming Duan, Mishel George, and Francesco Bullo. Markov chains with maximum return time entropy for robotic surveillance. IEEE Transactions on Automatic Control, 65(1):72-55) 86, 2019.
Richard M Dudley. Real analysis and probability. CRC Press, 2018.
Ben Eysenbach, Russ R Salakhuidinov, and Sergey Levine. Robust predictable control. Advances in Newral Information Processing Systems, 34:27813 27822, 2021.
Mark Fannes, contimuity property of the entropy density for spin lattice systems. Communications in Mathematical Physics, 31:291-2941 1973.
Roy Fox, Ari Pakman, and Naftali Tishby. Taming the noise in reinforcement learning via soft updates. 08566, 2015.
arXiv preprint
Mishel George, Saber Jafarpour, and Francesco Bullo. Markov chains with maximum entropy for robotic surveillance. IEEE Transactions Automatic Control, 64(4):1566 1580, 2018.
Hongliang Guo, Qi Kang, Wui Yau, Marcelo H. Ang, and Danicla Rus. Em-patroller: Entropy maximized multi-robot patrolling with steady state distribution approximation. IEEE Robotics and Automation Letters, 8(9):5712 5719, 2023a. doi: 10. A......0..3300245,
13

---

Published in Transactions on Machine Learning Research (06/2025)
Lingxiao Guo, Haoxuan Pan, Xiaoming Duan, and Jianping He. Balancing efficiency and umpredietailiity in multi-robot patrolling: A marl-based approach. In 2023 IEFE International Conference on Robotics and Automation (ICRA), pp. 3504 3509, 2023b. doi: 10.1009/ICRA44881..000..000000
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018.
Seungyul Han and Youngchul Sung. A max-min entropy framework for reinforcement learn- ing. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vanghan (eds.). Advances in Neural Information Processing Systems, volume 34, pp. 25732-25745 Curran Associates, Inc., 2021. URL https:// newr.ss wwwrips.cc/papet..eeeeeeeeeeeepe ftless/paper20221/1ee PTD766ff00023311777eee8881
pdf
Conor F Hayes, Roxana Rädulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfarlane, Mathieu Reymond, Timothy Verstraeten, Luisa M Zintgral, Richard Dazeley, Fredrik Heintz, et al. A practical guide to multi-objective reinforcement learning and planning. Autonomous Agents and Multi-Agent Systems, 36 (1):26, 2022.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-basod policy optimization. Advances in neural information processing systems, 32, 2019.
Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polynomial time. Machine learning, 49:209-205 232, 2002.
Dongyoung Kim, Jinwoo Shin, Pieter Abbeel, and Younggyo Seo. Accelerating reinforcement learning with valuo-conditional state entropy exploration, 2023.
Edouard Lourent. An environment for autonomous driving docision-making, https://github. com/
eleurent/highway-enn 2018.
Yanyu Liu, Yifeng Zeng, Biyang Ma, Yinghui Pan, Huifan Gao, and Xiaohan Huang. Improvement and evaluation of the policy legibility in reinforcement learning. In Proceedings of the 2023 International
Conference Autonomous Agents and Multiagent Systems, pp. 3044 3046, 2023.
Xingyu Lu, Kimin Lee, Picter Abbeel, and Stas Tiomkin. Dynamics generalization via information bottleneck in deep reinforcement learning. arxio preprint arXim:2008.000 00614, 2020.
Xiaoteng Ma, Xiaohang Tang, Li Xia, Jun Yang, and Qianchuan Zhao. Average-reward reinforcement learning with trust region methods. arXiv preprint arXiv:2106.03/ 2021.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusn, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):559-53)) 2015.
Mirco Mutti, Lorenzo Pratissoli, and Marcello Restelli. Task-agnostic exploration via policy gradient of a non-parametric state entropy estimate. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 9028-9036, 2021.
Mirco Mutti, Riccardo De Santi, and Marcello Restelli. The importance of non-markovianity in maximum state entropy exploration. In International Conference Machine Learning, pp. (6223-16222 PMLR, 2022.
Gergely Neu, Anders Jonsson, and Vicenç Gómez. A unified view of entropy-rogularized markov decision processes. arXiv preprint urXi::1705.07799, 2017.
Scohong Park and Sergey Levine. Predictable mdp abstraction for unsupervised model basod rl. arXiv preprinl ar Xixi:2200..0 039221 2023.
14

---

Published in Transactions on Machine Learning Research (06/2025)
Jan Peters, Katharina Mulling, and Yasemin Altun. Relative entropy policy search. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 24, pp. 1607-1612, 2010.
Silviu Pitis, Harris Chan, Stephen Zhao, Bradly Stadie, and Jimmy Ba. Maximum entropy gain exploration for long horizon multi-goal reinforcement learning, In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learn- ing Research, pp. 7750-7761. PMLR, 13-18 Jul 2020. URL https://proceedings.. mlr wtt.press//vi9// pitis20a. html
mmr
Martin L. Puterman. Markow decision processes: discrete stochastic dynamic programmiing. John Wiley Sons, 2014.
Antonin Raffin, Ashley Hill, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, and Noah Dormann. Stable baselines3, 2019.
Yagiz Savas, Melkior Ornik, Murat Cubuktepe, Mustafa O. Karabag, and Ufuk Topcu. Entropy Maximization for Markov Decision Processes Under Temporal Logic Constraints. IEEE Transactions Automatic doi: N0.1109/TAC.2019.
Control, 65(4))1552-1567, April 2020. ISSN 0018-9886, 1558-2523, 2334-3303. 2922583. URL https://iesexxplors ieeeeeppwpreeeeeeeeeeeee.eeeeeee wwt.:eieeeeeeooouuonreeeeoe orgg/oocument//73581111
Yagiz Savas, Michael Hibbard, Bo Wu, Takashi Tanaka, and Ufuk Topeu. Entropy maximization for partially observable markov decision processes. JEEE transactions on aulomalic control, 67(12):6948 6955, 2022.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithmss arXiv preprint ar TXX::1107.0000 7707003377 2017.
Rohin Shah, Noah Gundotra, Pieter Abbeel, and Anca Dragan. On the feasibility of learning, rather than assuming, human biases for reward inference. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Reseorch, pp. 5670-5679. PMLR, 09-15 Jun 2019. URL https://ppoceeddnng: mlr
Claude E Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3): 079-423, 1948.
Joar Skalse, Lewis Hammond, Charlic Griffin, and Alessandro Abate. Lexicographic multi-objective reinforce- ment learning. In Lud De Raedt (ed.), Proceedings of the Thirly-Firsl International Joinl Conference on Artificial Intelligence, IJCAI-22, pp. 3430-344 3436. International Joint Conferences on Artificial Intelligence Organization, 7 2022. doi: Main Track.
476. URL http::::/aoi..rrg//.0444..53 http:///.....//.ccaaa........
Elis Stefansson and Karl H Johansson. Computing complexity-aware plans using kolmogorov complexity. In 2021 60th IEEE Conference on Decision and Coutrol (CDC), pp. 3420 3427. IEEE, 2021.
Richard S Sutton and Andrew G Barto. Reinforcement leurning: An introduction. MIT press, 2018.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Daniil Tiapkin, Denis Belomestuy, Daniele Calandriello, Eric Moulines, Remi Munos, Alexey Naumov, Pierre Perrault, Yunhao Tang, Michal Valko, and Pierre Menard. Fast rates for maximum entropy exploration. In International Conference on Machine Learning, pp. 34161-34221. PMLR, 2023.
Ronald J Williams and Jing Peng. Function optimization using connectionist reinforcement learning algo- rithms. Connection Science, 3(3):241-25) 2-88, 1991.
15

---

Published in Transactions on Machine Learning Research (06/2025)
Rui Zhao, Xudong Sun, and Volker Tresp. Maximum entropy- regulaazee multi-goal reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 7553 7562. PMLR, 09-15 Jun 2019. URL httpps//pproceeddngss mlr prwss/v97/s/ https/////22a119..ncommm zhw.l9d.
Alexander Zimin and Gergely Neu. Online learning in episodic markovian decision processes by relative entropy policy search. Advances in neural informalion processing systems, 26, 2013.
16

---

Published in Transactions on Machine Learning Research (06/2025)
A Auxiliary Results
We include in this Appendix some existing results used throughout our work. The following Theorem is a combination results presented by Puterman (0144) regarding the existence of optimal average reward policies.
Theorem (Average Reward Policies Puterman (2014)). Given an MDP M the following hold: g (x)) and bat (x() exist, 9 (z) (=))="" for all (t.e. (x(x) is constant), and there ezists deterministic, stationary policy A E argmax 9r that mazimizes the expected average reward. Additionally, the same holds if is compact and R and P are continuous functions of
Theorem (Stochastic Recursive Inclusions Borkar 2009)). Let In E Rd be vector following a sequence:
In+1 = 2n + an (n(h((n)) + Mn+1 + nn).
(7)
where sup, ||اد|| < oo a.s., On is a learning rate, : RR + Rd is a Lipschitz map, Mn. is a Martingale difference sequence with respect to the alggger Fn == o M1, M,,..., Mn) fand square integrable), and Th an error term bounded by cn. Define H == {a E E h(x)) 0}- Then, for any 0, there erists an O such that Ven (0, the sequence { converyes almost surely to 6-neighbourhood of H.
Theorem 5 (Policy Gradient with Function Approximation Sutton et al. (1999)). Let be parameterised policy and fw : X x U - I be parameterised (approzimation of) action value function in an MDP M. Let the parameterisation be compatible, i.c. satisfy:
(z, (TT(I, u)
1
dw
00
TO( u)
Let the parameters and 0 be updated at each step such that:
(m, O,
Σ (x) Σ To(ar, u)((""(((( - Sw (2r, u))
Owk
ak+1 = 04 + 1100(1 Σ @mol u)
fun (x, w).
00
Then, limg->o
0p(Ox] 8e
0, where p(0) is either the discounted or average reward in the MDP.
B
Technical Proofs
Proof of Lemma
For statement a) observe 17 (x) can be expressed as
1th (x) E (1x, u. y)lu Txx)] log (E [P(x, u, g)lu (x))),
and recall
E (x [5(2, u)] Eu Σ
P(x, u, v) log (P(x, u, v))]
MEX
Then, from Jensen's inequality, E, Ea-r(n)ls(E,, u)] 1
In Statement b) the fact that h (x:) is constant follows from Theorem 3 by considering R(I, u, v) s(x, Now, note that we can write hg Emex (s)m u))e and ha E 17 (x)) (see Puterman
(2014h.. Thus:
hg
E u))" z
Σ (7)2) (x) ne xEX
xeX
where we employed Statement a)
For Statement
take w E ITP Then, (('l x:) and al x) = for an action a' 2 and all u + d'
Then
Eusn(z)ls(E, w)l Σ
P(x, , v) log (P(z, v', x)) 1" (z).
DEX
The fact that hg Te follows, then, trivially.
17

---

Published in Transactions on Machine Learning Research (06/2025)
Proof of Proposition Observe that s(z, u) and sø(2, u) are the entropies of probability distributions P(r, .. and Po(ar, U, respectively. Thus, the Mannes-Audenaerr inequality Audenaeri (0077) for two probability distributions p and g states:
H(p) - H(g)) & 27 log(d) - 27 log(27);
where le- and d is the dimensions of the support of the distributions. Then, taking P(a, u, and Po (x, .) as distributions, for any action u we obtain:
| u) - s(x, u) S <elog (1x1) log K(t);
Finally:
1
E
lim
So (X,, (Xi)) Xo [Mo] h2
T
0
T
E
lim
S6 (X,, T (Xi)) s(Xe, (XX))] z
+00
T
T Σ 1=0
E
lim
Σ
| X1 π (x)) - s(X,,
SE lim
k(c) = k(().
T
T
T-
T
1=0
Proof of Proposition Since S. is linear w'. Now, from equation and Theorem 8.2.6 Puterman 2014), S and S be written in vector form (over the stales X) as:
Q and Q is compact, there exists at least minimizer
S (T P. PT) P*)s"
Sg (T - + (T
where S" is the vector representation of s(; (()) (and analogously for sz)- Therefore, from Proposition [[ST(x) - (t)) s K()). Then, we can write without loss of generality
STH (T S7. u) + n(c,,),
with ((c,a)) .) being (c) for all x since is bounded by a function K(e which O(c). Finally, we can write the parameter iteration as
We+1 = wi + Be VoLL + M41 +n)()]
with Lg =- zm1" (S* (I u) - Sz u))? and the term M41 = (5, ()) (I,,
(8.(a.)) du
is
(=)
Martingale with bounded variance (since is bounded).
Therefore, by Theorem 6 in (Brrkar 2009 the iterates converge to some point w/ - (7( almost surely
as f + 0o, with (70))) being the ( neighbourhood of the stationary points satisfying V 0. ]
C Experimental Results and Methodology
We present here the extended experimental results, training curves and additional details corresponding the experimental framework. All experiments were run a single CPU, running Ubuntu 20.04, and all libraries and requirements are properly listed in the paper code.
18

---

Published in Transactions on Machine Learning Research (06/2025)
Figure 4: Self-Driving Environments.
C.1 Soft Actor-Critée Experiments
As discussed in Section
4.2 we implemented a version of PARL based on a Soft Actor- implementation
(Haarnoja al. 2018). For this, we simply learn in parallel an average reward Q function to optimize the entropy rate (Sw), and combine the Q functions to compute the actor objective as:
[T) = EE-D [Euwn [a log T(u x)) - (2(2, u)+
(8)
(K, u)]]
As a bascline for our SAC implementation, we use RPC Eysenbach et al. 2021 which is another SAC basod algorithm aiming to maximize rewards while compressing policies to a maximurn complexity, to achieve more simple, robust and predictable behaviours. Please note, the comparison is merely qualitative: RPC does not optimize for entropy rate in the agent's behaviour, however simpler policics do induce smaller entropy rates (as seen in the reported results) and thus it is still useful as baseline to evaluate the impact of entropy rate objectives in SAC agents. We trained all agents with the same parameters, 5 agents per parameter combination and evaluated over 50 independent episodes. For RPC, we trained agents with different policy compression rates (2 bits and 5 bits), and for PASAC with two different trade-off parameters (k = and = 0.5). We train both PASAC and RPC with the same architecture for the prediction models (decoder in RPC), and we use an identity encoder for RPC make the prediction models equivalent, and the entropy to be estimated in task space (not in a latent space). The results presented in table 2 DDPG baseline 17 which is an inherently deterministic policy.
We also include
Table 2: Results for MuJoCo environments, Off Policy algorithms.
Lengsh
Batropy
DDPG
1871.23 1 757.25
889.60 2: 170 73
4:14 1: 0.91
PASACE-1
6178.43 4 442.87
897 12 4 48.45
-0.70
2 0.12
PASAC
$285.99
1266.45
967.72
168 66
-0.27
0.44
RPC2bat
2064.51
074.88 30
958.44
108 76
1.81 0.56
RPC564
5640.18
486.67
996.15
60.78
1.92 0.12
HalCheetth
Rewards
Ep. Leagth
Eatropy Rate
04
1816.95
PASAC
11184.26
1808.
0.21
PASACUEO.S
RPCaba
RPC
Hopper
Ep. Length
Entropy
DDPG
1080.05 2: 840.22
313.40
193.
0.89
0.23
PASAC
1856.82
943. 95
479.17
300
-4.26
0.30
PASAC,-0.5
2845. 24
1140.
739.70
852 04
-3.69
0.24
RPC26it
2667.34 34
1091.18
732.44
318. 25
0.96
1 0.15
RPO360
2458.70 1 1236 72
655.78
333. 75
1.19
1 0.08
17e Ant-v4 DDPG obtain very high variance rewards and entropies. agents and picked the best
To get more conservative baseline, we trained
mean reward obtained.
19

---

Published in Transactions on Machine Learning Research (06/2025)
Figure 5: Interactive Tasks, Left is a moving obstacle navigation task where the agent has the option to de-activate the obstacles. Right is a navigation problem where the ground is slippery, resulting in random motion.
C.2 Interactive Robot Tasks
We created two tasks inspired by real human-robot use-cases, where it is beneficial for agents to avoid high entropy state-space regions. These are based on Minigrid environments Chevalier- Boisvvrrt et al. (0023) see Figure 5 The agents can execute actions U = {forward, turn-left, turn-right, toggle), and the observation space is = I²" such that the observation includes the robot's position and orientation, the position of obstacles and the state of environment features (e.g. the switch). In both tasks, the agent gets reward of 1 for reaching the goal, or negative reward of -1 for colliding or falling in the lava. Both grid environments were wrappod in normalizing vectorized wrapper, to normalize observations, but the model data was kept un-normalized.
Turning Off Obstacles This task is designed a dynamic obstacle navigation task, where the motion of the obstacles be stopped (the obstacles can be switched off) by the agent toggling switch small cost of rewards. The environment is depicted in Figure (left). The switch is to the left of the agent, shown as green bar (orange if off). The agent gets reward of 0.95 if it turns the obstacle off. The intuition behind this task is that agents do not learn to turn off the obstacles, and attempt to navigate the environment. This, however, induces less predictable dynamics since the obstacle keeps adding noise to the observation, and the agent is forced to take high variance trajectories to avoid it. Our Predictability-Awarr algorithm converges to policies that consistently disable the stochasticity of the obstacles, navigating the environment freely afterwards, while staying near-optimal.
Task 1:
Task 2: Slippery Navigation The second task is inspired by a cliff navigation environment, where a large portion of the ground is slippery, but a path around is not. In this problem, the slippery part has uncertain transitions (i.e. a given action does not yield always the same result, because the robot might slip), but the non-slippery path induces deterministic behaviour (the robot follows the direction dictated by the action). The agent needs to navigate to the green square avoiding the lava. If it enters the slippery region, it has a probability of p 0.35 of spinning and changing direction randomly. The intuition behind this enviromment is that PPO agents do not learn to avoid the slippery regions, resulting in higher entropy rates and less predictable behaviours. On the contrary, PAPPO agents consistently avoid the slippery regions. This can be seen Figure
C.3 MuJoCo Experiments
We include here the full set trajectory plots task space, both for position of the front tip versus angle of the front tip, and x-y velocities of the front tip (See Figures 7 include in tables and the full numerical results for the simulated agents. Each result reported computed for 10 independently trained agents, and each agent evaluated over 50 independent trajectories.
10 I1 and 12). Additionally, we
20

---

Published in Transactions on Machine Learning Research (06/2025)
(a) Dyn Obstacle Env. (b) Slippery Nav. Env.
Figure 6: 2D trajectory projections. Blue
PPO policy, orange is PAPPO policy.
Table 3: Results for MuJoCo environments, Stochastic Policies.
Walker
Rewards
Length
PPO
2466.27
1329.71
838.20 + 296.82
1.19
0.40
PAPPOR20.05
2496.12
1190-43 45
826.77
263.34
0.90
0.74
2037.30
561-40
253
0.05
0.61
1315-47
910
550.18
296
-1.28
1.79
PAPPOR=0.5
1072.88
-11
848.38
1 358-74
-1.90
1.16
Ant
Rewards
Pip Length
Retropy Rate
PAPPO
PAPPO =
2107.21
23
756.58
330.00
1.64
PAPPO
1497.67
854.70
860.96
196.30
3.04
PAPPO
1204.47
087.53
55.00
1.88
HalCCheetah
Rewards
Ep. Longth
Ealcopy Rate
PPO
4150.53 + 1646.16
1000.0 # 0.0
1.01 # 0.34
PAPPO
3575.56 2: 1743.61
1000.0 4 0.0
0:46 :: 0:40
PAPPO
4482.48 2. 1679.52
1000.0
4 0.0
0.31 + 0.26
PAPPO
4427.07
1996.80
1000.0
0.0
-0.27 0.74
PA
3982.70
1822.80
1000.0
+ 0.0
-0.54 0.99
Trajectory Representations As expected, the observed trajectories for the case of PARL agents present much less complex (lower entropy) distribution. In particular, for the slippery navigation task where agents have the choice of taking fully deterministic paths, it is even more obvious that the PARL agent chooses to execute the same trajectory over and over, where PPO agents result in a more complex distribution due to the traversing of the stochastic regions.
C.4 Learning Results
We include the learning curves for the trained agents all the environments included in the paper.
C.5 Model Learning
Our proposed predictable RI. scheme consists of a model-based architecture where the agent learns simulta- neously a model P& for the transition function and a policy mo and value functions Ve, Ww, for the discounted rewards and entropy rates. Simultancously to a policy and a value function, we learn model P& to ap- proximate the transitions (means) in the environment. For this, we train neural network with inputs (x, u) X x U and outputs the mean next state Tau- The model is trained using the MSE loss for stored data D = {1x, ::
C, =
Σ D
(Jeu - y)²
2/D
We do this by considering D to be a replay buffer (Lo reduce bias lowards current policy parameters), and at each iteration we perform K mini-batel updates of the model sampling uniformly from the buffer. Additionally, we pre-train the model a set number of steps before beginning to update the agents, by running a fixed number of environment steps with a randomly initialised policy, and training the model on this preliminary data. All models are implemented as forward networks with ReLU activations.
21

---

Published in Transactions on Machine Learning Research (06/2025)
Figure 7: Walker Trajectory Plots, z-axis is torso angle in radians, y-axis is coordinate position of the torso. Blue are PPO agents, orange are PARL agents.
Figure 8: Walker Trajectory Plots, axis is velocity, y axis is velocity. Blue are PPO agents, orange are PARL agents.
22

---

Published in Transactions on Machine Learning Research (06/2025)
Figure 9: Ant Trajectory Plots, z-axis is torso angle in radians, y-axis is coordinate position of the torso. Blue are PPO agents, orange are PARL agents.
Figure 10: Ant Trajectory Plots, axis is x velocity, axis is velocity. Blue are PPO agents, orange are PARL agents.
23

---

Published in Transactions on Machine Learning Research (06/2025)
Figure 11: HalfCheetah Trajectory Plots, z-axis is torso angle in radians, y-axis is coordinate position of the torso. Blue are PPO agents, orange are PARL agents.
Figure 12: HallCheetah Trajectory Plots, axis is x velocity, axis is y velocity. Blue are PPO agents,
orange are PARL agents.
24

---

Published in Transactions on Machine Learning Research (06/2025)
Table Results for MuJoCo environments, Deterministic policies.
Walker
Rewards
Ep. Lengih
Entropy Rate
3144.86
1426.220
769.80
PAPPO
25
1794.17
1331.01
062.40
1.64
PAPPOR=O.
1212.20
1201.21
672.42
305.88
Rewards
Ep. Length
Entropy
Rate
PPO
3013.54
1524.71
808.31 à: 313 55
0.52 0 38
PAPPO
9186.00
1490.50
$83.08
+ 200.07 07
0.27
0.54
PAPPO
2633.00
1836.18
$20.35
+ 801 95
-0.41
2.17
1011.57
902 54
964.98
150 04
.4.77
3.63
1216.27
1129:12
956.30
4: 90.72
-0.44 2.33
HalfChretah
Rewards
Length
Rntropy
Rtate
PPO
6192.40
1191.34
1000.0 + 0.0
0.52
0.31
PAPPO 20000 200.0
5115.65
1378-23
1000.0
0.0
PAPPO PAPPO k00
b3522.42
1215.20
0.0
0.0
0.40
k00
1740.05
Stpp.
Stex
Saps.
PPO
PARL
Figure 13: Training results for Slippery Navigation task.
Entropy estimation We found that it more numerically stable to use the variance estimations as the surrogate entropy (we do this since the log function is monotonically increasing, and thus maximizing the variance maximizes the logarithm of the variance). This prevented entropy values to explode for environments where some of the transitions are deterministie, thus yielding very large (negative) entropies.
C.6
Tuning and Hyperparamters
The tuning of PARL, due to its modular structure, be done through the following steps:
Tune (adequate) parameters for vanilla RI. algorithm used (e.g. PPO).
2. Without the predictable objectives, tune the model learning parameters using the vanilla hyperpa- rameters.
3. Freezing both agent and model parameters, tune the trade-off parameter i and specific PARL parameters (e.g. entropy value function updates) to desired behaviours.
For our experiments, we took PPO and SAC parameters tuned from Gtable-Baseliness Raffin et al. (2019) and Haarnoja al. (2018), and used automatic hyperparameter tuning Akiba et al. (2019) for model and predictability parameters. In the implementation optimizing the policy for some steps without minimizing the entropy rate. For all hyperparaneters used in every environment and implementation details we refer the reader to the project repository
introduced delay parameter to allow agents to start
25

---

Published in Transactions on Machine Learning Research (06/2025)
23
18000
30000
1300
5.00
1900
2000
13094
10000
3000 3800
1990
3000
Step
Stepe
S400
PPO
PARL
Figure 14: Training results for Obstacle Navigation task.
330
20
Drowy Rutt
325
230
1300 300 2000 1300
3000
0300
7008
300 3000 3000 AR0E
5000 0300 7500
1300 3300 2000 2000 3000 3000 TOOO
PPO
PARL (6 0.1)
PARL (8 0.5)
PARL (R 1)
Figure 15: Training results for Highway environment.
350
370
Sten
Stepe
0.5)
(R
Figure 16: Training results for Roundabout environment.
3303
3333
Eeney Rad
0.-
14
10
0.2
02
0.1
05
PPO
PARL (k 0.00)
PARL (A - n1)
PABL (2 0.25)
PARL (A - 0.5)
PARL (R 3)
Figure 17: Training results for HalfChectah-v4 environment.
2.0
Exoee Rate
0.2
044
14
6.5
10
030
611
15
45
100
34
0.5
05
x11
PPO
PARI. (8 0.05)
PARI (A 0.1)
PARI (E 0.25)
PARI. (k 0.5)
PARI (R 1)
Figure 18: Training results for Walker2d-v4 environment.
26

---

Published in Transactions on Machine Learning Research (06/2025)
300
10
1/
04
24
Days
-13
Styes
X11
PPO
PARL (K 0.05)
PARE (h- 0.1)
PARL (R 0.253
PARL (K 0.5)
PARL (R- 1)
Figure 19: Training results for Ant-v4 environment.
27


Reviews:
[{"review_id": "eN2QXJMHmU", "reviewer": "Reviewer_r3hW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe paper proposes an interesting idea which runs counter to prior work on entropy maximization in RL.\n\nThe approach overcomes some technical difficulty in formulating an entropy-minimizing objective.\n\nThe algorithm is clearly presented.\n\nThe paper evaluates on both deterministic and stochastic variants of environments, and considers multiple environments from a well-established benchmark (Mujoco).\n\nUnder a number of assumptions which, while not guaranteed to hold in practice, are a reasonable starting point for analysis, the method is shown to converge.\n\nThe visualization of the differing trajectories between the two methods provided a nice qualitative example of the behaviour induced by the algorithm. Quantitatively, the results on the entropy under different coefficients suggest that the objective is being successfully optimized.\n\n\n**Weaknesses**\n\nThe claim that the proposed method increases the predictability of the agent’s behaviour is inaccurate. As a trivial example, an agent in an environment where turning left on the first timestep takes the agent into a region where the transition dynamics become deterministic and turning right causes the agent to enter a region where the dynamics are noisy can maximize the PAPPO objective by always turning left, even if it then follows a uniform random policy in every other state. \n\nIt seems to me that the proposed algorithm is optimizing for predictability of the environment, encouraging the agent to visit states where the environment dynamics are closer to deterministic. This type of objective seems like it would be especially useful in partially observable environments, which to the best of my knowledge are not studied in this paper. I would be surprised if the mujoco tasks evaluated were really the best domain to showcase the benefits of entropy minimization. The multi-agent driving environments get closer to this property (indeed, I could imagine that this type of entropy penalty might induce more “defensive” driving compared to the baseline agent), but not consistently, as evidenced by the performance degradation in roundabout.\n\nThe method generally appears to at best provide a very modest improvement in performance (which is still within the confidence interval of the baseline PPO agent) and at worst significantly degrade performance for larger entropy penalties. \n\nThe method requires a world model, and I would guess that its success depends heavily on properties of this model. In particular, in rich-observation settings I would expect that the generalization of the learned model would heavily influence the agent’s exploratory behaviour. I did not see any additional investigation into the role of the world model choice on performance, which seems important to understanding the method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes using an information-theoretic objective to minimize the entropy of the state visitation distribution in RL agents to make their behaviour more predictable, evaluating the approach on mujoco and driving environments.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address my concern regarding the accuracy of referring to this approach as maximizing the predictability of the *agent's* behaviour.\n\nI think the paper's empirical contribution would be significantly stronger if the authors were able to demonstrate an environment where the method unambiguously helps to improve performance, and where this improvement can be well-explained by the nature of the source of transition-entropy in the environment (for example, noisy states being harder to learn an optimal policy on or less controllable).", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IB88WvI7JD", "reviewer": "Reviewer_myg1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: The use of entropic functional to enforce predictable behaviors is rather new, additionally the derivations and results are rather on standard (to me).\n\nWeakness: while being a positive as well, the rather non-standard derivations are rather hard to follow, often based on some results not properly introduced. \n\nSome high-level concerns about the main motivation and context:\n- It is still just at an intuitive level to me how a low-entropic distribution over trajectories should be inherently more predictable. I think a broader discussion of what to do with almost deterministic trajectories (once reached) should be added.\n- The authors argue that \"[randomized actions] often makes it challenging for other agents and humans to predict an agent’s behavior, triggering unsafe scenarios\" and suggest to enforce deterministic distributions over trajectories. This is clearly different from enforcing deterministic policies only, but I think justifying why a simple DDPG architecture (for instance) would not suffice is essential. Also, testing against less deterministic environments would be needed, as in (almost) deterministic environments deterministic policies induce low-entropy distributions over trajectories.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a method to enforce predictability of RL behaviours, defined as polices inducing low-entropy distributions over trajectories. In order to do so, the authors propose a (negative) reward shaping term based on two fundamental steps: first,  a reformulation of the entropy over trajectory distributions as an average reward problem, then an objective surrogate that upper-bounds the true one in order to guarantee non-degradation over the true objective. The derivation is accompanied with a series of theoretical result to guarantee that the proposed method is well-founded and a series experiments confirming that the proposed method does induce trajectories with low entropy rate.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Major Concerns and Changes:\n- The main theoretical contributions are often not presented in a clear and self-explanatory way. Every step and/or result needed for understanding the derivations should be reported.Too often, results from other works are cited but not reported, making the overall understanding hard. For instance, \n  - Lemma 1: Theorem 3 is left undefined in the main paper, even thought it seems rather central; \n  - Proposition 1, \"Thus, from the Fannes–Audenaert inequality Fannes (1973), for the special case of diagonal density matrices (representing conventional probability distributions), we obtain:\" is obscure to me, also because the inequality is usually applied to Von Neumann entropies, as far as I know; \n  - Proposition 2: I am not sure how a bound on infinity norm could turn into an equality with no dependency on x as the first passage. Also, \"Therefore, by Theorem 6 in (Borkar, 2009), the iterates converge to some point\" is a rather obscure sentence to the average reader which might need way more discussion.\n- I am not sure to have understood how the all discussion of Section 7 is relevant at that point of the discussion.\n- Experiments: \n  - Deterministic policies in PPO are used as comparison, but algorithms that learn deterministic policies directly (DDPG) were not. I would expect to see them as a comparison.\n  - How were entropy rates computed in practice should be discussed.\n  - In the autonomous driving task, a negative reward is usually given in case of collision and lower rewards would actually result in unsafe behaviours, how this is related to the whole narrative should be discussed. \n\n\n\nSome Minor Concerns and Changes:\n\n- (Page 1) Explain why \"Higher entropy rate implies more complex and less predictable trajectories, and vice-versa.\" as it is not obvious to all the readers.\n- (P2) In figure 1 the needed policy seems to be deterministic (rather than the distribution over trajectories), if this is the case I would comment on why.\n- (P3) Delta notation for distributions not introduced.\n- (P4) typo--> 0:T-1 should be 0:t-1 in the joint entropy, I would also merge section 3 with the next one\n- (P4) I believe the definition of conditional entropy can be made more rigorous by defining what p stands for and where the dependency on the policy in maintained.\n- (P5) the derivation of Eq 4 can be made more structured and formal, being of central interest. For instance, how P_\\pi relates to the p of the previous definition, how the dependency on the joint probability over trajectories of the conditional entropy is implied to the Markov kernel in Eq 3.\n- (P8) typo --> I think S^\\pi_\\phi should be dependent on P_\\phi. And the Cited Corollary 1 is actually Proposition 1.\n- (P10) Remark 3 is hard to read as it is.", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vHC3dilEKq", "reviewer": "Reviewer_HXvg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\\\nThe discussion of predictable RL is novel and interesting. As far as I can recall, there is no work focusing on forcing agents produce simpler and more predictable trajectories. The authors have done a good job on telling the story in the introduction. I especially like how the authors defined the scope: \"RL agents are oblivious to the information theoretic loads they generate with their behavior...we do not argue that this is a necessary feature in all agents. We simply argue that it is an interesting feature to consider for general RL agents that can benefit the deployment of RL agents\". It is also nice to see that after theoretical proofs, the authors conducted experiments on Mujoco which are not overly simple, showing the efficacy of the proposed method.\n\n**Weaknesses**:\\\n - I believe the paper can benefit from moving technical details to the appendix to make the paper less than 12 pages (TMLR short paper). The current draft contains too much technical details that may distract the reader. For example, Proposition 2 is fairly standard where (Borkar, 2009) was used. Its proof can be left to the appendix. \n - Though it is nice to see that the proposed trajectory-entropy minimization led to simpler and clustered trajectories. It is somewhat expected since entropy is a measure of randomness. My concern is that the predictability was not achieved through increasing prediction capability of the agent, but rather through decreasing the randomness of trajectories. This may risk the exploring behavior of the agent in scenarios like multiple reward modes like that in the SAC paper. \n - Another concern is that though the authors critized the entropy-regularization methods that induce higher aleatoric uncertainty, the proposed method falls also into the regularization class. Therefore, the optimal policy is necessarily biased. Does this regularized predictable RL framework \"benefit the deployment of RL agents\" to problem that requires maximizing task rewards, see ref [1]?\n\nReference:\\\n[1] Greedy actor critic, ICLR 2023", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed a new RL paradigm to study the complexity of trajectories produced by agents. Simpler trajectories are more predictable and therefore less prone to danger, yet have the potential to achieve the goal same as the complex ones. The authors formulated the predictable RL problem as a trajectory-level entropy regularization problem and presented theoretical proofs justifying the use of surrogate local entropy and approximated dynamics model. Experiments on Mujoco showed that the proposed algorithm led to compact and clustered trajectories.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weaknesses section for requested changes.", "broader_impact_concerns": "not applicable", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper deals with a problem formulation, regularized RL with the negative entropy rate, that is, to the best of my knowledge, original. I believe this formulation is worth studying and the contribution has potential, a sentiment that is shared by the reviewers. The paper is solid and does not present major flaws, although some area of improvements have been extensively reported in the reviews.

I am proposing acceptance of the paper as is, although I am advising the authors to read carefully the reviews when drafting their final version. Especially,
- to take seriously the comment from Reviewer r3hW and adjust the narrative as "trajectory predictability" rather than "action predictability";
- expand the discussion on why this formulation may be particularly valuable in POMDPs (as commented in response to Reviewer r3hW).
Most of the concerns of other reviewers seem to have been incorporated in the manuscript already.

Finally, the authors may want to discuss the relation between their work and "SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environments" (Berseth et al., 2021), which is studying a formulation with some similarities (and important differences, mainly the entropy of the state distribution instead of trajectories and surprise minimization as an unsupervised objective instead of regularization).
Decision: Accept (poster)