Decoupling regularization from the action space
Authors: Sobhan Mohammadpour
Emma Frejinger
Pierre-Luc Bacon
Venue: ICLR 2024 poster (2024)
Keywords: reinforcement learning
regularized markov decision process
soft actor-critique
Primary area: reinforcement learning
TL;DR: Regularized MDP regularize states with more actions more, we fix this.

Abstract: Regularized reinforcement learning (RL), particularly the entropy-regularized kind, has gained traction in optimal control and inverse RL. While standard unregularized RL methods remain unaffected by changes in the number of actions, we show that it can severely impact their regularized counterparts. This paper demonstrates the importance of decoupling the regularizer from the action space: that is, to maintain a consistent level of regularization regardless of how many actions are involved to avoid over-regularization. Whereas the problem can be avoided by introducing a task-specific temperature parameter, it is often undesirable and cannot solve the problem when action spaces are state-dependent. In the state-dependent action context, different states with varying action spaces are regularized inconsistently. We introduce two solutions: a static temperature selection approach and a dynamic counterpart, universally applicable where this problem arises. Implementing these changes improves performance on the DeepMind control suite in static and dynamic temperature regimes and a biological design task.

Under review as a conference paper at ICLR 2023
DECOUPLING REGULARIZATION FROM THE ACTION SPACE
Anonymous authors Paper under Houble-blind review
ABSTRACT
Regularized reinforcement learning (RL), particularly the entropy-regularized kind, has gained traction optimal control and inverse RL. While standard unregularized RL methods remain unaffected by changes in the number of actions, show that it can severely impact their regularized counterparts. This paper demonstrates the importance of decoupling the regularizer from the action space: that is, maintain a consistent level of regularization regardless of how many actions are involved to avoid ver-regularization. Whereas the problem can be avoided by introducing a task-specific temperature parameter, it is often undesirable and cannot solve the problem when action spaces are dete-deeendent. In the stteedependent action context, different states with varying action spaces are regularized inconsistently. We introduce two solutions: a static temperature selection approach and dynamic counterpart, universally applicable where this problem arises. Implementing these changes improves performance on the DeepMind control suite static and dynamic temperature regimes and biological sequence design task.
INTRODUCTION
Regularized reinforcement learning (RL) (Geist et al., 2019) has gained prominence as widely- used framework for inverse RI. (Rust, 1987; Ziebart et al., 2008; Fosgerau al., 2013) and control (Todorov, 2006; Peters et al., 2010; Rawlik et al., 2012; Van Hoof et al., 2015; Fox et al., 2016; Nachum et al., 2017; Haarnoja et al., 2017: 2018; Garg et al., 2023). The added regularization can help with robustness (Derman et al., 2021), having policy that has full support (Rust, 1987), and inducing a specific behavior (Todorov, 2006). However, we show that these methods are not robust to changes in the action space. We argue that changing the action space should not change the optimal regularized policy under the same change. For instance, changing the robot's acceleration unit from meters per second squared to feet per minute squared should not lead to different optimal policy. While Haarnoja et al. (2018)'s heuristic is a step in the right direction, we argue that the heuristic does not reflect the structure of the action space, just the number of actions, and does not generalize to other regularizers MDPs.
The key idea proposed here to control the range of the regularizer by changing the temperature. Indeed, by not changing the temperature, we demonstrate that we inadvertently regularize states with different action spaces differently. We show that for regularizers that we call standard, which include entropy, states with more actions are always regularized more than states with fewer actions. We introduce decoupled regularizers, class of regularizers that fit Geist et (2019)'s 's formalism and have constant range. We show that we can convert any non-decoupled regularizer into decoupled one.
Our contribution is follows. First, we propose static temperature selection scheme that works for a broad class regularized Markov Decision Processes (MDPs), including entropy. Secondly, we introduce an easy-to-i casy-to-implemen dynamic temperature heuristic applicable all regularized MDPs. Finally, we show that our approach improves the performance on benchmarks such as the DeepMind control suite (Tassa al., 2018) and the drug design MDP of Bengio et al. (2021).

---

Under review as a conference paper at ICLR 2023
2 PRELIMINARIES
discounted MDP is tuple (S, R,, P.?) where S represents the set of states, A is the collection
of all possible actions and (s) represents the set of valid actions at state If [A(s) {)] is not constant
for all 8 E S, we say that the MDP has state-dependent actions. The reward function, denoted by
R: 8 x determines the probability of transitioning to the next state, where ASS) indicates the probability simplex over the of states S. Additionally, the discount factor, represented by Y E (0, 1], is included in our problem formulation.
IR maps state-action pairs to real numbers. The transition function, P S A (S).
When solving Markov Decision problem under the infinite horizon discounted setting, the aim to find a policy T(s) + (A((s) that maximizes the expected discounted return (s) & E I y [[s, sp = s] for all states. A fundamental result in dynamic programming states that the value function v for any stationary optimal policy " must satisfy the Bellman equations (Bellman, 1954):
max Ea-x, 1Q(s,)) V(s) x,EA((()))
Vs e S,
where ((s, a) is defined as R(s, a) + +. [V+']] Regularized MDPs (Geist et al., 2019) introduce strictly convex regularizer Ω with temperature regularize the policy
V(s)
max EaNa, [Q(s, - - = (Q((s(s)))) EA((())
s]. The optimal policy equals the gradient of Ω2 (Geist et al., 2019). Replacing Ω by the negative entropy yields soft Q-learning (SQL) as 22; is the log-sum-exp function at temperature (7 log E. ssp(Q(s,a)/a) and VΩ; the softmax temperature т. π (a|s) exp(Q(s,a)) a)/T).
such that (s) A EI (
Whereas the proposed approach applies to all regularized MDPs, we focus on the case where -Q2 the entropy. There are two main reasons for this choice. First, it is widely used (e.g. Ziebart et al., 2008; Haarnoja et al., 2017). Second, it allows us to derive analytical bounds. Other alternatives include Tsallis entropy (Lee al., 2019).
3 GRAVITATION TOWARDS REGULARIZATION
To quantify the impact of a change in action space regularization, we first define the range of regularizer.
Definition 1. The range the regularizer Q over the action space L(2, A) SuPrEA(A) Q(T) min ((T)..
The range is sometimes used for analyzing regularized sellow-the-leade algorithms (e.g.. Theorem 5.2 Hazan et al., 2016), and its square is referred to as the diameter (Hazan et al., 2016). If the range depends on the action space of the state, the propagation of the regularization by the Bellman equation can have compounding effect. Thus, the change in regularization affects not only the state itself but all states that can reach it. Thus, balaneing regularization and reward maximization in MDPs in sequential decision decision-making processes is crucial. We show this using two small illustrative examples. Example 1. (Bias due to (s))) In the MDP shown in Figure la, with reward on all transitions starting at $1 and zero otherwise, the probability of taking the action ag is a+1 (where n + 1 the number of paths) with no discounting.
Proof. The result follows from the definition of V. The value of s2 is log n, thus the probability of expr/r taking the action ag is expr/r+exp (r+T log n)/r temperature 7.
Li
Example 2. (Bias due to loops) In the MDP shown in Figure 1b, with reward r on all transitions, the probability of taking the action an is exp(rrT) at temperature 7 and no discounting.
Proof. The value of $1, V. equals 7 log In exp(r/T + V/T) exp(r/T) or T log(1 n exp(r/t)). Thus, the probability of taking ag is exp(r/r)/ exp/v//) = n exx((r/T). The MDP diverges if S exx((r/7)
2

---

Under review as a conference paper at ICLR 2023
a0
02
$1
01
$3
ag
$1
82
02
Gn
On
(a) An MDP with + paths.
(b) An MDP with n loops.
Figure 1: Toy MDPs
These examples show gravitation towards regularization. Concretely, with negative entropy, the regularization at the states with larger action spaces is greater, resulting in higher regularization and higher reward to pass through those states. Thus, when we increase n, the probability of passing through the state with n consistently, and how much we regularize state should not depend on its action space. One way of measuring this quantity is the range defined in Definition 1. Indeed, argue that the range should not depend on the action space. This motivates our solution, which we call decoupled regularizers.
+ 1 actions increases. However, the states should be regularized
Despite the specificity of these examples, the same behavior can be observed more broadly with other regularizers, including those in stochastic MDPs (Mai and Jaillet, 2020). To this end, we introduce general class of regularized MDPs that show similar problem in Section 5. It is also important to note that the discount factor was set to one for mathematical clarity, and including discounting alleviates the risk of divergence but does not completely eliminate the problematic behavior. In the following, we introduce our approach to address inconsistent regularization across action spaces.
4 DECOUPLED REGULARIZERS
We note that in the following we can replace the sum with an integral in the continuous actions space. We look at differential entropy and continuous actions in Section 7.
Definition 2. We call regularizer Ω decoupled if the range of is constant over all action spaces A (s) for all valid states For any non-decoupled regularizer ft at state s, ((T),, defined as ((x) (s)) is the decoupled version of ②
Concretely, the value of regularized MDP at state & is given by (ss) (Q(ss))
which we propose to replace with
(ss) Ω (1EE
We give the range of some commonly used regularizers on discrete actions in Table 1, Note that the Tsallis case, is often set to 2. While there are no known analytical solutions for the convex conjugate of Tsallis entropy, when = it can be solved efficiently (Michelot, 1986; Hazan et al., 2016; Duchi et al., 2008). We further note that the convex conjugate of KL with the uniform distribution (denoted U' is sometimes called mellowmax (Asadi and Littman, 2017). The relationship between mellowmax and KL divergence was first shown in Geist et al. (2019). The range for the negative entropy regularizer log (s)). which equals the logarithm of the number of actions. Thus the effective temperature is T/ log [s() as the minimum discrete entropy is zero. Entropy divided by maximum entropy is called efficiency (Alencar, 2014).
STANDARD REGULARIZERS AND THE DRIFT IN RANGE
We now look a general class of regularizers over discrete action spaces.

---

Under review as a conference paper at ICLR 2023
H(m)
KL(x||U>
negative Tsallis entropy
Q(m)
EmEAC)) m[(as) logm(a)s)
EnEACo a(cls) log (T(a|+)/ (ator)
(Enaac m[alo]e
(*(((s)))
log Enea exp(Q(s,a)/r)
r Flog (xoo) EmEr(h) exp(Q(s,a)/t))
exp(((s,a)/t) Emere exp(Q(s.a)/r)
exp(Q(s.
v0;((s.-))
EnEAc exp(Q(s.o)/T)
((PTEA(4(4)) ((x))
log [A(s)]
min)Ca()6)) 9(m)
log [A[+]]
g-t
(())
log (+)]
log [s)]
Table I: Different values at state s. Empty cells indicate no known analytical solution.
Definition We call a regularizer in the form ((x.) = 9(E. (a)) for a strictly convex function f and strictly monotonically increasing function standard regularizer. We assume that (()) is strictly convex to be compatible with regularized MDPs.
We chose this form because it is easy to reason about yet general enough to encapsulate many regularizers, including entropy and Tsallis entropy. The regularizer is invariant to permutation and naturally extends itself to higher dimensions.
In addition, we include regularity assumption.
Assumption 1. (Symmetry) we assume that f(0) and f11) equal to O.
We now show that under Assumption I the supremum is constant.
Lemma 1. Under Assumption 1, the supremum of the regularizer is equal to the limit the regularizer at deterministic distribution (i.e., only one action has non-zero probability, and the others have zero probability).
Proof. The supremum of is at O and 1. By convexity, at any other point between 0 and 1, is smaller than O. By strict convexity Yo<p<if(p) pf(1) (1 p))
Assumption helps identify the supremum of the regularizer and show that it is constant. However, is possible that Lemma holds even Assumption does not. For instance, the maximum negative Tsallis entropy is always zero.
With the supremum being g(0), we now calculate the minimum regularizer. Lemma 2. The minimum regularization is achieved under the uniform policy.
Proof. We proceed with a proof by contradiction. Suppose that the minimum a nonuniform policy. Since the uniform policy is in the convex hull of all permutations of that policy, by strict convexity and symmetry, it has lower value than the supposed minimum and thus contradictory.
We now show that the minimum of the regularizer, attained the uniform policy, is decreasing in the number actions.
Lemma 3. The minimum regularization decreases with the number of actions.
Proof. By strict convexity, we have that that æ. (xr) f(x) and as a consequence f'(x)/x- /2² the gradient of (xx)/x, is always positive. This implies that minimum regularization decreases as x 1/n decreases.
z). By Assumption 1, we have
show that the range grows with the number of actions. Theorem 1. The range of standard regularizers grows with the number of actions.
Equipped with these three lemmas,
4

---

Under review as a conference paper at ICLR 2023
Proof. While the minimum grows per Lemma 3, the supermum stays the same per Lemma 1, and the range hence grows with the number of actions.
Remark 1. This result holds for any regularizer invariant to permutation, and its supremum is constant. The standard form guarantees permutation invariance. For instance, the range of negative Tsallis entropy also grows with the number of actions.
Remark We have made no comment on the rate which the range grows. For instance, the range of Tsallis entropy grows to 1, and thus, the range of negative Tsallis entropy does not grow as fast as negative entropy with respect to the number of action spaces.
VISITING DECOUPLED MAXIMUM ENTROPY RL
In this section, review decoupled maximum entropy RL, revisit the examples provided in Section 3, and show that decoupling improves the convergence of undiscounted entropy regularized MDPs.
Algorithm Decoupled SQL
Sample s from p0 if decoupled then
+ T/ log (s))
else
T' T
end
while true do
Sample action a E A (s) with probability exp(Q(s, EweRrs)) exp(Q(s,'')/7) Play action and observe s',r decoupled then end
T' / log [s())
Q(s, a) r 7' log En'Erse) exp(Q(s),d)/)
s
end while
First, we provide an example of a tabular implementation in Algorithm 1. The conditional shows the changes needed to decouple SQL.
Next, we revisit the MDP in Figure la. Using decoupled SQL, we get that the probability of action ag constant as the value of 82 T/ 1/log log(n exp(Ologn/7) decoupled entropy. The value of 81 is 7/ log 2 log [exp(r log 2/7) + exp(r log 2/T + log 2/T) or r + T/ log 3. The probability of taking action dg is equal exp(r 2/T ((81) log 2/T) or 1/3.
т when regularizing by
The state 8 of the MDP Figure will be at temperature 7/ log(n + 1); thus, if is less than -1, it will not diverge. The probability of taking the action do is exp(r log 1//T), which is strictly decreasing and has a root at r equal to -1; thus, the regularized Bellman equation converges below that threshold. The improved convergence of the MDP in Figure 1b using decoupled entropy motivates the following more general result.
Proposition 1. In maximum entropy undiscounted inverse reinforcement learning with determin- istic dynamics like Ziebart et al. (2008) or Fosgerau et al. (2013), decoupled entropy guarantees convergence if the maximun reward is less than T.
Proof. If EeEM(s)) exp R(s, a)/7 < 1, a solution always exists (Mai and Frejinger, 2022, Remark 2). Since (R < 1/n, the model always has solution.
7 AUTOMATIC TEMPERATURE FOR REGULARIZED MDPS
Haarojj et al. (2018) proposed adding a lower bound on the entropy the policy to find the right temperature. Concretely, they propose using the constraint H=*(+ss) 2 (A((s)) for some function
5

---

Under review as a conference paper at ICLR 2023
H the Bellman equation. They propose using the dual of the aforementioned constraint as the temperature leading to Algorithm 2. We note that we parametrize т, the dual variable and temperature, in terms of its logarithm so that it stays positive. While this change deviates from Haarnoja et al. (2018)'s notation, it better reflects their actual implementation.
Algorithm 2 Soft actor-critic's update
is sampled from p0
while do
is sampled from (-1s)
is sampled by playing a
0 0 XV)
Update critic (3a)
á + Ø AV a ():T)
b. Update policy (3b)
log t log + AV log (l (A(s))
Update temperature (3c)
s'
end while
Jq(0) Ea (Qo(ss))) - +r+ d') (a'(s')
(3a)
(0) = Fa log To (s)sa))
(3b)
Joog (log (; A(s)) =Ea log H({(s)))
(3c)
Haarnoja et al. (2018) propose using the negative dimensions of the actions as the target entropy. if the action is vector in R", H is -n. Their proposed solution has two downsides: First, there is no reason that the same heuristic would be meaningful if another regularizer, for instance, if Tsallis entropy, used. Second, Haarnoja et al. (2018)'s heuristic does not reflect the action space. Both of these points are easy to illustrate: if the action space is a real number from -5e-3 to 5e-3, the maximum entropy is -2, which would be lower than Haarnoja al. (2018)'s heuristic, and make the problem infeasible. It is important to stress that 7 will grow to infinity if the target entropy H is infeasible.
e So
To remedy these, we propose a H inspired by the range of the regularizer Ω. Concretely, we argue that
((T(-1s))
sup x'E((A()))
2(T') < - AI:;
should hold for some constant a between 0 and 1. Setting a to 0 is equivalent to disabling the constraint, and setting to one results in becoming the minimum regularized (or maximum entropy) policy. Translating (4) back to entropy yields
H (TT(-ss) aH(U) +1 (1- ()()).(V).
(5)
where U is the uniform and V is the minimum reasonable entropy policy policy should have. We need define V as the minimum entropy policy as is not defined for differential entropy. We note again that setting a to one yields the uniform and a to zero yields the minimum entropy policy. Note that (U) the logarithm of the volume of the action space, i.e., the logarithm of the integral the unit function over the action space. We discuss choosing a in the next section.
EXPERIMENTS
In this section, we provide three sets of experiments: toy MDP where the number of actions is parameter, a set of experiments on the DeepMind Control suite (Tassa et al., 2018), and lastly, the drug design MDP of Bengio et al. (2021).
8.1 A TOY MDP
To illustrate the importance of temperature normalization, we propose a toy MDP where the number of actions is parameter. The state dimension vector in the natural non-zero numbers

---

Under review as a conference paper at ICLR 2023
such that 8; I m for all i [I....,,.] for some Each action increases decreases one of the elements of s. Doing an action that would invalidate the state (for instance, make elements of zero) does not change the state. The agent starts at state [1.1,...,, 1]. The agent receives - reward for every time step that it has not reached the goal point [m. m,... m]. The episode terminates after reaching the goal point. When n = 2, the MDP is grid where the agent starts the bottom left corner and receives negative reward as long as it has not reached the top right corner. The agent can only move to neighboring states but not diagonal ones. We display the expected time exit with y 0.99 and r 0.4 in Figure 2. The time to exit of SQL becomes very large for > 6. important to stress that if the temperature less than 1, decoupled SQL cannot diverge by Proposition 1. We also note that we have to set the temperature very low so that the SQL does not diverge at five dimensions, and thus, the decoupled version is fairly close to the shortest path. This example illustrates two main points: First, it highlights the importance of decoupling regularizers across benchmarks. Indeed, setting a unique temperature for all n yields suboptimal behavior as the agent gains more regularization in higher dimension spaces, and the balance between reward and regularization is broken. Second, it highlights the improved convergence properties of decoupled entropy.
8.2 DEEPMIND CONTROL
The maximum entropy in the DeepMind control (DMC)
benchmark is n l2g(2 where n is the number of actions and / is chosen such that the actions are in the -8 to 8 range. In the first experiment, we fix the temperature to 0.25. The test rewards over the training, shown in Figure 3, do not worsen and can, in many instances, improve com- pared to the non-decoupled version. We provide the full experiments in Appendix B. While the model is sensitive to changes in action space, the gain in performance can still be observed across different values of B3. In addition to analyzing how changes in reward scale change the per- formance as Henderson et al. (2018) suggests, we argue that it is also important to analyze how the performance changes in response changes to the action space and range. We note that we do not any scale invariant optimizer or loss function and that reaching full invariance to changes in action scale is beyond the scope of this work.
Foguszel spooee lmgga
SQL Decoupted SQL
130
Shurtest path
100
80
Dimerabaniity
Figure 2: Episode length at different di-
mensions of the hypergrid problem.
We now focus on the dynamic temperature setting. We chose a x 0.77 to get similar results as Haarnoja et al. (2018) when the actions are in the [-1, range, this is our recommended default. Oth- erwise, the alternative is finding the optimal a as one would with the temperature as the interpretation similar, the higher a, the higher the final temperature will be.
As shown in Figure 4b, Haarnoja et al. (2017)'s heuristic becomes infeasible, leading to very high temperatures. High temperatures, in turn, lead learning failure. Figure 4c shows similar performance as the temperature is extremely low for both models.
8.3 COMPARISON WITH GENERATIVE FLOW NETWORKS
Our final experiment involves the drug design by fragments of Bengio et al. (2021). In this MDP, an agent adds fragments. collections of atoms, to other fragments build molecule (we refer to Jin et al.. 2018, for a more detailed description of the representation). The agent can end the episode when the state is a valid molecule, making the horizon finite but random. Each molecule is represented as tree, and each fragment is a node in this tree. Each tree corresponds to a unique and valid molecule. GFlowNets (GFN) aims to sample molecules proportionally to some proxy that predicts reactivity with some material (Bengio et al., 2021). The goal is not to find only one molecule with high reward but a diverse set of molecules with high rewards. As such, our main metric, other than high reward, is the number of modes or molecules that have low similarity to other modes. We find the set of modes by iterating over the list of generated molecules and adding molecules that are not similar any existing mode to that set.
7

---

Under review as a conference paper at ICLR 2023
detaul:
Easy
Firghp
HopperStand
HnmancidStand
Reacherllard
Walker Run
1000
750
500
30
5000
500
0
0
0
(a) Actions
1-1,1]
F Fingerr Fasy
Upriggh
HopperStand
HumancidStand
Reacher Hard
WallorRon
1000
750
750
300
250
250
0.5
(b) Actions in
-2,5, .25.
Finger EasyeTyy
Uright
HopperStand
HumanoidStand
ReacherHard
Walker Run
1000
750
200
1000
500
500
600
500
250
0
0.01
(c) Actions in [-4,4].
Figure 3: Test reward on the DMC benchmark with = 0.25. The X axis is the number iterations divided by le6.
defsult
CallluC
Balancee
Furaerrd
HopperStand
PointMase Easy
RescherHard
1000
1000
1000
500
500
500
500
0
0
(a) Actions
[-1, 1].
BallinCupCatch
CartpoleBalaaee
Faraer
HopperStand
PointMas
Reacherllad
1000
1000
500
500
250
500
0
0
(b) Actions [-.1, .].
BallhoCupCateh
Balance
FingerTuroBa
HopperStand
Mosot Fasy
ReacheeHard
1000
1000
500
500
500
500
(c) Actions in [-1,4] 1].
Figure 4: Test reward on the DMC benchmark with automatic temperature. The x-axis is the number of iterations divided by le6.
It is beyond the scope of this work to properly introduce GFNs; we therefore simply state that they train policies to sample terminal states in proportion to an unnormalized distribution. Bengio et al. (2021) imposes four constraints on the MDP: there should only be one initial state, each state reachable from the initial states, no state is reachable from itself, and the transition function is deterministic. Lastly, Bengio et al. (2021) assumes knowledge about the inverse dynamics. Concretely, for every state s' they assume they have the list states s that can reach s', i.e. (s Ba E (s)s.. a) 0). These assumptions are not always easy satisfy. For instance,
8

---

Under review as a conference paper at ICLR 2023
0.8
1250
Decoupled SQL
0.6
730
0.4
300
0.2
GFN
250
Decoupled SQL
0.0
0.0
0.5
1.0
1.5
20
0
5000
10000
15000
20000
25000
30000
X10
Figure 5: The left plot is the median reward of each batch. The right is the number of modes found. The shaded area shows the interquartile range, and the heavy line shows the interquartile mean.
undoing actions is trivially possible with these assumptions. We use trajectory balance for the GFN loss (Malkin et al., 2022). For algorithm parity, we use path consistency learning (Nachum al., 2017) as our SQL loss. We note that we use static temperature.
The results in Figure show the median reward and number of modes found. Indeed, the median reward of decoupled SQL is higher than SQL and GFN through training. The left subplot shows that decoupled SQL. finds many high-quality modes. While SQL over-regularizes states with more actions, leading to a policy that prefers to pass through these hub states with many actions, decoupled SQL does not have this problem. This result alone highlights the need for decoupling in the state-dependent action setting.
CONCLUSION
In this paper, we argued that the amount we regularize should not depend on the action space. For example, we should not have to change the temperature of our regularized MDPs when we change the units of our robots. To illustrate our point, we introduced standard regularizers, which include entropy. We showed that standard regularizers increase how much they regularize with the number of actions. We proposed that the range should not depend on the action space and introduced decoupled regularizers regularizers whose range is constant. We showed that we can obtain decoupled regularizers from normal regularizers by dividing them by their range. While instead of decoupling, change the temperature manually, argue that it is often not desirable for benchmarks and cannot solve the problem in the state-dependent action setting. We emphasize the broad applicability of our findings; both the static and dynamic temperature schemes work for all regularized MDPs.
Perhaps most notably, our research has achieved unprecedented results the domain of drug design. This is especially significant Bengio al. (2021) did not include SQL in their results they mentioned it was too unstable and inherently prefers larger molecules. However, we found that our decoupled regularizers with PCL resolved both issues, serving the critical, missing component. The innate simplicity of SQL, adaptability in environments characterized by cycles, and independence from inverse dynamics, the need to know which states can reach another state that is fundamental to GFNs, accentuate its appeal, and underscore its suitability for MDPs.
Our proposed method works regardless of the chosen regularizers but we only justified its use for standard regularizers; of course, not all regularizers are standard. For instance, π ArT for strictly positive definite matrix only standard if A is the identity matrix. We posit that the same approach we took here might be insightful in that there may exist function similar to the range that should be kept constant by transforming the regularizer. Lastly, while we moved closer to regularized icale-endependent RL by introducing regularized RL models that are not sensitive to changes in action space, we believe there more work to be done on the optimization side of the problem to enhance the scale invariance properties further.

---

Under review as a conference paper at ICLR 2023
REFERENCES
M. S. Alencar. Information theory. Momentum Press, 2014.
K. Asadi and M. L. Littman. An alternative softmax operator for reinforcement learning. In International Conference on Machine Learning, pages 243 252.252. PMLR, 2017.
R. Bellman, The theory of dynamic programming. Bulletin of the American Mathematical Society, 60(6):503-515, 1954.
E. Bengio, M. Jain, M. Korablyov, D. Precup, and Y. Bengio. Flow network based generative models for non-iterative diverse candidate generation. Advances in Neural Information Processing Systems, 34:27381-27399 2021.
E. Derman, M. Geist, S. Mannor. Twice regularized mdps and the equivalence hetween robustness and regularization. Advances in Neural Information Processing Systems, 34:22274-22287, 2021.
J. Duchi, S. Shalev-Shwartz, Y. Singer, and T. Chandra. Efficient projections onto the 1-ball for learning in high dimensions. Proceedings of the 25th international conference on Machine learning, pages 272-279, 2008.
M. Fosgerau, E. Frejinger, and A. Karlstrom. A link based network route choice model with unrestricted choice set. Transportation Research Part B: Methodological, 56:70-80, 2013.
R. Fox, A. Pakman, and N. Tishby. Taming the noise in reinforcement learning via soft updates. In Proceedings of the Sheron Conference Uncertainty in Artificial Intelligence, UAI' 16, page 202-211. AUAI Press, 2016.
D. Garg, J. Hejna, M. Geist, and S. Ermon. Extreme q-learning: Maxent rl without entropy. In The Eleventh International Conference on Learning Representations, 2023,
M. Geist, Scherrer, and Pietquin. A theory of regularized markov decision processes. In International Conference Machine Learning, pages 2160-2169. PMLR, 2019.
T. Haarnoja, H. Tang, P. Abbeel, and S. Levine. Reinforcement learning with deep energy- policies. In International conference on machine learning, pages 1352-1361. PMLR, 2017.
Haarnoja, A. Zhou, P. Abbeel, and Levine. Soft actor-critic: Off Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference machine learning. pages 1861-1870. PMLR, 2018.
E. Hazan et al. Introduction to online convex optimization. Foundations and Trends@ in Optimization, 235-4):1773325 2016.
Henderson, Islam, Bachman, J. Pineau, D. Precup, and D. Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
W. Jin, R. Barzilay, and T. Jaakkola. Junction tree variational autoencoder for molecular graph generation. In International conference machine learning, pages 2323 22332. PMLR, 2018.
K. Lee, S. Kim, Lim, S. Choi, and S. Oh. Tsallis reinfoncement learning: unified framework for maximum entropy reinforcement learning. arXiv preprint arXiv: 1902.00137 2019.
T. Mai and E. Frejinger. Undiscounted recursive path choice models: Convergence properties and algorithms. Transportation Science, 56(6):1469-144 2022.
Mai and P. Jaillet. A relation analysis of markov decision process frameworks. arXiv preprint arXiv:2008.00 078200 2020.
N. Malkin, M. Jain, E. Bengio, C. Sun, and Y. Bengio. Trajectory balance: Improved credit assignment in gilownets. Advances in Neural Information Processing Systems, 2022.
C. Michelot. A finite algorithm for finding the projection of a point onto the canonical simplex of n. Journal of Optimization Theory and Applications, 50:195-200 1986.
10

---

Under review as a conference paper at ICLR 2023
O. Nachum, M. Norouzi, K. Xu, and D. Schuurmans. Bridging the gap between value and policy based reinforcement learning. Advances in neural information processing systems, 30, 2017.
J. Peters, K. Mulling, and Y. Altun. Relative entropy policy search. Proceedings of the AAAI Conference on Artificial Intelligence, 24(1):1607-1612, Jul. 2010.
K. Rawlik, M. Toussaint, and S. Vijayakumar. On stochastic optimal control and reinforcement learning by approximate inference. Proceedings of Robotics: Science and Systems VIII, 2012.
J. Rust. Optimal replacement of gme bus engines: An empirical model of harold zurcher. Economer- rica: Journal of the Econometric Society, pages 999-1033, 1987.
Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. d. L. Casas, D. Budden, A. Abdolmaleki, J. Merel,
A. Lefrancq, et al. Deepmind control suite. arXiv preprint 180110060000 2018.
E. Todorov. Linearly-solvabee markov decision problems. Advances in neural information processing systems, 19, 2006.
H. Van Hoof, J. Peters, and G. Neumann. Learning of non-paramctric control policies with high- dimensional state features. In Artificial Intelligence and Statistics, pages 995-1003, PMLR, 2015.
B. D. Ziebart, A. Maas, J. A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforcement learning. In Proceedings of the Twenty-Thing AAAI Conference on Artificial Intelligence, volume 8, pages 1433-1438, 2008.

---

Under review as a conference paper at ICLR 2023
detaul:
ArrobotSwingup
CallhaCopCatch
CarlpeleHlalance
CartpoleSwingop
CheetahRum
FingerSpin
1000
5.0
500
500
500
2.3
0
0
innger FugerTurnFess Fasy
F Finger HurdHard
FlahSwim
Fish Upright
HopperHop
HopperStand
300
750
750
500
500
200
500
250
HumanoidRan
H HumanoidStand
HumanoidWalk
MasippulstoeBinngBll
PeudulumsSwingup
Mosseyy
1010
7.5
1.25
500
5.0
0
0.0
0.5
0.0
0.5
0.0
0.5
0
Rasy
Reacher Hand
SwittmeerSimmmee immee11
WalberRon
WalkerWalk
1000
1000
1000
300
000
200
0
Figure 6: DMC test reward with the action scale set to 0.25, Static temperature.
defsuls
oILEs
AcrobotSwingup
Caplla
Carlnele
CartpoleSwingup
CheetahRun
FingerSpin
1000
5.0
500
500
500
2.5
0
0
FingeTTurnEasy TurnEasy
Finger
F FingerTune TurnHlad
FiskSwim
Fish Upright
HopperHlop
HopperStund
300
750
750
500
500
200
500
250
250
RumanoidRun
H HumanoidStand
HumenoidWalk
ManippclatoeBrrngBBa.
PouduiummSwnggu
Mosnt
1.25
1.00
1.0
500
0.75
0
0.0
0.5
0.0
0.5
0.0
0.5
0
Reacherlasy
Herd
SwiimeeeSwimmeel
SwinneerSSwnnmmer
WalkerRan
WalkerWalk
1000
1000
300
1000
500
200
300
200
500
0
Figure 7: DMC test reward with the action scale set to 0.5. Static temperature.
A
REPRODUCIBLITY
All code is hosted htt.:p/e/.aaaoooooo..oooo.eeaeae and https://anonymonnn wwwp:/ommmmuuuuup......oom htgpen.cccaamre///eeeuullleeeers
5CAB/
B EXTENDED DEEPMIND CONTROL EXPERIMENTS
We define our minimum entropy distribution V as a uniform distribution over a le - range. We argue that for practical purposes, any distribution with such low is deterministic for all intents and purposes.
12

---

Under review as a conference paper at ICLR 2023
defanls
Swiggot
CalleaCuCaaehh
Ballncee
CactpoleSwingup
CheetahRun
FingerSpin
1000
1000
1000
5.0
500
2.5
500
0
FingerTurnEess
F IngerTuratinn
FishSwim
Uirhgp
Hoppertllo
HopperStand
1000
300
50
500
200
300
250
0
HamanoidRan
HumanoidStand
Humanoid Walk
PendulunSSwaguu
PointMaesBBa
1.5
100
50
500
1.0
0.0
0.5
0.0
0.5
0.0
0.5
ReucherEasy
Reacherllad
StimmeerSwimmer1
SwiimmerSSrimmarr
WallcerRan
Walk
1000
300
500
300
500
200
0
Figure 8: DMC test reward with the action scale set to 1. Static temperature.
defanit
AcrobotSwingup
BellioCupCateh
CarlpoleBalance
CartpoleSwingup
CheetahRum
FinporSpin
1000
1000
1000
5.0
2.5
500
0
Finger TurnEasy
F VingerTuraHann TurnHaed
FistSwim
Fish Fish Upright
HoppezHop
HopperStand
1000
50
200
500
500
500
250
HomanoidRRm
HomanoidSSamnl
HumanoidWal
Manipulttod BringggB
PeudulunSwingen
Point Massyy
1.5
50
100
100
500
1.0
25
0.0
0.5
0.0
0.5
0.0
0.5
0
ReacherEasy
Reacher
SwiimmerSSimmeet
SwinmeerrSWinnme
WalleerRun
WalkerWalk
1000
300
500
200
300
200
500
0
Figure 9: DMC test reward with the action scale set to 2. Static temperature.
13

---

Under review as a conference paper at ICLR 2023
defanls
Swinget
Callh
CaclpeleBalance
CactpoleSwingup
Cheetah:Rm
FpngerSpin
1000
1000
1000
5.0
500
2.5
D
F FingerTurrEEas
F Vinger Turaltad
FishSwim
Fish Upright
HopperkHop
HopperStand
1000
750
500
500
500
250
0
HamanoidRan
HumanoidStamd
Humanoid Walk
Manipelator BringBall
Pendulun.Swragee
PoentMa
1.25
200
1.5
100
500
0.75
0.0
0.5
0.0
0.5
0.0
0.5
ReucherEasy
Readherlladd
StrimeerSSinmer1155
Swimmmertttinmmarr
WalkerRan
WalkerWalk
1000
500
200
300
500
200
0
Figure 10: DMC test reward with the action scale set to 4. Static temperature.
defanit
SerobotSwingup
CollloCopCateh
BadapoleBalance
CartpoleSwingup
CheetahRum
FinporSpin
1000
1000
1000
7.5
500
2.5
Finger TurnEasy
F Finger TuraHaed
FistSwim
Fish U'right
HoppezHop
HopperStand
500
500
200
250
HomanoidRam
HamanoidStamn
HummnoidWall
Manipclator Bringgg
PeudulumSwinguu
Point Massay
1.5
250
500
0
ReacherEaay
HeadherHard
S5eimmerSwiimee115
SwinmeerSSrnnmae
WalkerrRnn
WalkerWalk
1000
300
300
500
200
100
0
Figure 11: DMC test reward with the action scale set to .1. Dynamic temperature.
14

---

Under review as a conference paper at ICLR 2023
defanls
Swiggot
CalleaCupCaehh
Ballnele
CactpoleSwingup
CheetahRun
FingerSpin
7,5
1000
1000
1000
5.0
500
2.5
0
Finger TurnEasy
F Turataaddd
FishSwim
Uiriht
HopperHlop
HopperStand
300
100
500
500
500
250
100
250
H HumanoidRan
HumanoidStand
Humamoid
SannipclatoeBringB.a.
Pendulun-Swiguup
Paesosy
0.50
0.75
250
500
0.25
ReacherEasy
Reacherlladd
StrimmerSrimme115
SwimmmerSSrimmaer
WallcerRan
Walk
1000
300
300
500
200
0
Figure 12: DMC test reward with the action scale set to .25, Dynamic temperature.
defanlt
AcrobotSwingup
BellloCupCateh
BarlpoleBalance
CartpoleSwingup
CheetahRum
FinporSpin
1000
1000
1000
7.5
5.0
500
2.5
500
0
Finger TurnEasy
F VingerTuratee TurnHaed
FistSwim
Fish Upright
HoppezHop
HopperStand
1000
100
500
200
500
HomanoidRRu
HumanoiddStand
HumanoidWaa
Manipulatod BringgBBeeBBigBlll
PeudulumSSwnggn
Point Mass
500
0.75
5000
50
50
0.50
500
0.25
0
ReacherEaay
Reacher
SwimmeerSwinmel11
Swinmnett
WalleerRon
WalkerWalk
1000
300
300
500
500
200
200
0
Figure 13: DMC test reward with the action scale set to 1. Dynamic temperature.
15

---

Under review as a conference paper at ICLR 2023
default
AcrobotSwingup
Caphe
Carlpele
CactpolsSwingup
CheetahRun
FingerSpin
1000
1000
1000
7.5
500
5.0
2.5
500
0
0
F Finger TurnEasy
F Finger Turnllad
FishSwiin
Fish Upright
Hoppertllp
HopperStand
1000
200
800
500
500
100
250
0
Hom
HumanoidStand
HumanoldWall
Manipolator BringBall
PendulumSuutngnn
Massayy
10
250
100
0.50
500
500
0.25
Rasy
Reacher Hard
Sw SwemmeerSwinmen iimmer15
SwiimmerrSwmmmmee
WalkerRon
WalkerWald
1000
500
200
300
500
150
0
Figure 14: DMC test reward with the action scale set to 4. Dynamic temperature.
16


Reviews:
[{"review_id": "1XJI0CBIk9", "reviewer": "Reviewer_nhQG", "summary": "The paper proposes a new method of regularization of common RL algorithms. First, the paper proves that standard regularization techniques used in RL algorithms such as SQL and SAC suffer from regularization that behaves differently with the size of the action space. The authors propose a new family of regularization techniques where state-dependent temperature parameters are learned by normalizing by the range of regularization function. The authors also show that such normalization can also be used on regularization techniques that dynamically learn the temperature parameter. Finally, the authors demonstrate their results on a variety of domains, including DM env and drug discovery.", "questions": "1. The empirical results swap between dynamic and static learning of the temperature parameter. Is there a reason why one of the method works better for a particular domain. Specifically, will dynamic temperature learning at least reproduce the results of using a static temperature in the drug discovery experiment?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper motivates the problem with standard regularization techniques well in Section 2-5. \n\n2. The empirical results are quite compelling. The proposed method, while simple, succeeds in various DM env tasks where standard methods fail. More convincingly, the proposed method allows SQL to succeed on a drug discovery task, whereas prior attempts were known to be too unstable.", "weaknesses": "1. The method proposed in the paper is very simple, and involves normalizing the standard temperature by the range the regularization objective can take (e.g. minimum possible entropy). While I do not think that the simplicity of the approach should detract from the novelty of the paper, I am not convinced that the better experimental results are actually due to the proposed range-normalization, and not simply that the regularization can now be state-dependent. To my knowledge, state-dependent temperatures is not itself a novel concept, as other works have tried to dynamically set temperature based on state [1, 2]. The results would be more convincing if the authors showed that prior efforts in state-dependent regularization fail whereas theirs succeeds.\n\n2. The algorithm does not actually circumvent the extensive hyperparameter tuning required by existing RL algorithms. It is unclear how sensitive the proposed approach is to various settings of temperature and \\alpha. \n\n\n[1] https://proceedings.mlr.press/v70/asadi17a/asadi17a.pdf\n\n[2] https://arxiv.org/pdf/2111.14204.pdf", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V1vtouSzQr", "reviewer": "Reviewer_aWwr", "summary": "This paper focuses on regularized reinforcement learning, where the objective is to find a policy in an MDP which maximizes the reward minus a weighted regularization term—most commonly the negative entropy. The authors argue that the weight of the regularization term should vary depending on the environment and even sometimes based on the specific state, since the range of values the regularizer can take might depend heavily on the action space. To solve this problem, the authors introduce a method for dynamically setting the regularization coefficient based on the maximum and minimum values the regularizer can take at a particular state. They show that this can improve RL performance compared to using the same coefficient across environments.", "questions": "* Why should the ideas in this paper apply to SAC when the entropy as measured in SAC is actually unbounded?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "As the authors note, regularized RL is widely applicable in both control and IRL, so improving its sensitivity to hyperparameters could be helpful for a variety of applications. The experimental results show that the proposed method for setting the regularization coefficient seems to work quite well in practice, particularly when the action space does not have a standard scale.", "weaknesses": "While the paper is promising, I worry that in it's current form it is not ready for publication at a top conference. First, the contribution is relatively small, since it is already known how to choose the entropy coefficient for many environments (e.g., via the SAC rule of using $\\bar{H}=-n$) and it often needs to be tuned regardless depending on the scale of the reward function. While I still think the ideas here are useful, the writing, theory, and experiments should be of very high quality to justify publication.\n\nHowever, I found the paper to be often unclear and imprecise, making it difficult to read. See the list of issues below for a partial list of the problems I noticed. The fact that many claims are unprecise and thus actually incorrect as stated makes it difficult to know which contributions to trust.\n\nOne possible extension to this work that could increase the contribution of the paper is to also include regularization to a base policy, as is often done in RLHF for LLMs via a KL divergence regularization term (e.g., see Bai et al., \"Training a Helpful and Harmless Assistant with\nReinforcement Learning from Human Feedback\"). There have been various papers attempting to find a systematic way of setting the regularization coefficient (e.g., the above paper and Gao et al., \"Scaling Laws for Reward Model Overparameterization\"), but none have considered possible setting the coefficient differently at different states, so this could be an interesting direction to apply the ideas presented in this paper.\n\nAlso, a relevant paper is \"LESS is More: Rethinking Probabilistic Models of Human Behavior\" by Bobu et al. They approach the problem from the motivation of modeling human behavior, which under the \"Boltzmann rational\" model is equivalent to solving a regularized MDP. They have a different approach to solving the issue of how the number of actions or number of trajectories that can achieve a particular reward affects the optimal regularized solution. It would be good to compare the approach in this paper with theirs.\n\n**Issues with clarity and precision:**\n * The first paragraph of the intro argues \"changing the action space should not change the optimal policy.\" This is true in most cases—the issue is that \"changing the action space should not change the optimal *entropy-regularized* policy.\" I think it's important to make this distinction since otherwise it sounds wrong.\n * In Section 2, it is not specified that the MDPs considered have discrete actions, although it later appears this is an unspoken assumption. Then later, the paper switches to continuous MDPs without any explicit distinction.\n * In Section 2, according to the formalism in Geist et al., $\\Omega$ should really be a function of only $\\pi(s)$, not of $\\pi$ in general.\n * In Section 2, the first time regularization is introduced it's with $\\tau \\Omega(\\pi)$ subtracted from the value function; then immediately below it's instead added to the value function.\n * The definition of $\\Omega^*_\\tau$ is unclear.\n * Proof of Lemma 1: this proof seems insufficient; why exactly does convexity imply any other point must be smaller?\n * In SAC, the entropy is actually calculated before applying $\\tanh$ to squash the actions into the action space. This means that the entropy is unbounded, and thus the idea in this paper does not apply directly.\n\n**Typos:**\n * Proof of Lemma 1: \"supermum\" -> \"supremum\"\n * Section 3 title: \"graviation\" -> \"gravitation\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TEpsDLMiup", "reviewer": "Reviewer_g8g2", "summary": "This work reveals the problem that the regularization effect can be severely affected by the number of actions (especially when action spaces are state-dependent) in regularized reinforcement learning. Starting with two motivating examples, this paper introduces decoupled regularizers whose range is constant over action spaces. The authors further prove that the range of a general class of standard regularizers grows with the number of actions. Two solutions of temperature selection, static v.s., dynamic, are proposed based on which practical algorithms can be implemented. The proposed solutions are evaluated in toy examples, DMC and drug design environments with SQL, SAC and GFN as backbones and baselines. The results demonstrating the effectiveness of the proposed solutions.", "questions": "1) Is it possible to have some case-analysis for the results of DMC (e.g., in Figure 3,4)? For example, the proposed method achieves significantly better performance in tasks like BallInCupCatch, FingerTurnHard, HopperStand while the baseline fails totally.\n\n2) Which is the type of temperature setting adopted for the experiments in Section 8.3, i.e., fixed temperature or dynamic temperature?\n\n&nbsp;\n\nMinor:\n- It seems that the coefficients $\\gamma$ and $\\tau$ are missing for the entropy term in Equation 3a.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is overall well organized and written. Illustrative examples and discussions are used for smooth reading.\n- The proposed solutions to decoupling regularization and temperature adjustment are of generality to a family of commonly seen regularizations.\n- The experiments cover toy examples, popular DMC environments and drug design problem. Noticeably, unprecedented results in the domain of drug design are achieved.", "weaknesses": "- I think the key hyperparameter $\\alpha$ needs more discussion. It will be helpful to analyze the (empirical) effects of different choices of the value of $\\alpha$ and recommend the values or the strategy of value selection.\n- The content in the experiment part lacks of sufficient details, which is also missing in the appendix. I recommend the authors to add the key details of experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)