$q$-exponential family for policy optimization
Authors: Lingwei Zhu
Haseeb Shah
Han Wang
Yukie Nagai
Martha White
Venue: ICLR 2025 Poster (2025)
Keywords: reinforcement learning
policy optimization
$q$-exponential family
heavy-tailed policies
sparse policies
Primary area: reinforcement learning
TL;DR: we explored a family of policy distributions that are promising alternatives to the standard Gaussian policy in policy optimization

Abstract: Policy optimization methods benefit from a simple and tractable policy parametrization, usually the Gaussian for continuous action spaces. In this paper, we consider a broader policy family  that remains tractable: the $q$-exponential family. 
This family of policies is flexible, allowing the specification of both heavy-tailed policies ($q>1$) and light-tailed policies ($q<1$). This paper examines the interplay between $q$-exponential policies for several actor-critic algorithms conducted on both online and offline problems. We find that heavy-tailed policies are more effective in general and can consistently improve on Gaussian. 
In particular, we find the Student's t-distribution to be more stable than the Gaussian across settings and that a heavy-tailed $q$-Gaussian for Tsallis Advantage Weighted Actor-Critic consistently performs well in offline benchmark problems.
In summary, we find that the Student's t policy a strong candidate for drop-in replacement to the Gaussian.
Our code is available at \url{https://github.com/lingweizhu/qexp}.

Published as a conference paper at ICLR 2025
q- EXPONETIIA FAMILY FOR POLICY OPTIMIZATION
Lingwei Zhu'
Haseeb Shah
Han Wang"
University of Tokyo lingwei uuua berta.ca
University of Alberta hshahl @ua be rt
University of Alberta han8@ua wwerta.ercoomm
Yukie Nagai
Martha White University of Alberta
University of Tokyo
ABSTRACT
Policy optimization methods benefit from a simple and tractable policy parametriza- tion, usually the Gaussian for continuous action spaces. In this paper, consider a broader policy family that remains tractable: the q-exponential family. This fam- ily of policies is flexible, allowing the specification of both heavy-tailed policies (q > 1) and light-tailed policies (q < 1). This paper examines the interplay be- tween g-exponential policies for several actor-critic algorithms conducted on both online and offline problems. We find that heavy-tailed policies are more effective in general and can consistently improve on Gaussian. In particular, we find the Student's t-distribution to be more stable than the Gaussian across settings and that a heavy-tailed q- I-Gaussian for Tsallis Advantage Weighted Actor-Critic consis- tently performs well in offline benchmark problems. In summary, we find that the Student's 's policy a strong candidate for drop-in replacement to the Gaussian. Our code is available at https: : //gttttiin gighub...b... com h http//lingweiizhi gexp.
INTRODUCTION
Policy optimization methods optimize the parameters of a stochastic policy towards maximizing some performance measure (Sutton et al., 1999). These methods benefit from a simple and tractable policy functional. For discrete action spaces, the Boltzmann- (Mei et al., 2020; Cen al., 2022); while the Gaussian policy is standard for the continuous case. For continuous action spaces, sampling the BG policy is computationally expensive due to the normalizing log- iag-patttionn function. A Gaussian policy is often used as a tractable approximation. While there are other candidates such as the Beta policy (Chou et al., 2017), the Gaussian remains the most common choice for both online and offline policy optimization methods (Haarnoja et al., 2018; Neumann et al., 2023; Xiao et al., 2023).
(BG) policy is often preferred
In this paper, we consider a broader policy family that remains tractable called the g-exponential family. The geexponential family was proposed to study non-extensive system behaviors in the statistical physics (Naudts, 2010; Matsuzoe & Ohara, 2011) I), and has recently been exploited in transformers (Peters et al., 2019; Martins et al., 2022), By setting 9 = 1, it recovers the standard exponential family. With g > tails than the Gaussian, such as the Student's 's t-distribution (Kobayashi, 2019) the Lévy Process distribution (Sim- sekli et al., 2019; Bedi et al., 2024). Heavy-tailed dis- tributions can preferable as they are more robust (Lange et al., 1989), can facilitate exploration and help escape local optima in the sparse reward context (Chakraborty et al., 2023). When 9 1, light-tailed (sparse) policies such as the G-Gaussian distribution can recovered. The sparse q- y-Gaussian has finite support and
Gomeian
ş Ganssia q 2 Studknss
Goneean g
obtain policies with heavier
Figure 1: The policy parametrizations in this paper. Student's t q-exponential
with q + 2/(v + 1) z 1.67.
iidiicttes joint first authors.

---

Published as a conference paper at ICLR 2025
Reference
Scope of exP,
Explicit?
Hcavy & Sparse?
Continuous?
RL?
Naudts (2010); Matsuzoe & Ohara (2011) Martins et al. (2022)
g R
< 1
x
x
Loe et al. (2018); Chow et al. (2018a)
0
/
X
X
Lee et al. (2020); Zhu et al. (2023; 2024)
9 <
X
Li et al. (2023)
q=
X
This paper
9 R
Table I: Existing works and their scopes. We are the first to consider the general g-exponential family for the parameterized policy in reinforcement learning. The family includes continuous heavy-tailed and sparse policies. Prior works in RL considered only the discrete case continuous policy with specific entropic index q. Further, in many cases they still used Gaussian policy parameterization approximate implicit target distribution that is g-exponential, rather than explicitly using the q- G-Gusssian as the policy parameterization.
can serve as a continuous generalization of the discrete sparsemax. As result, q- Gaussian helps alleviate safety concerns incurred by the infinite support Gaussian (Xu et al., 2023; Li et al., 2023).
Such g-exponential families have been considered in reinforcement learning, with the existing work summarized in Table 1. Lee et al. (2018); Chow et al. (2018b) studied the discrete setting with q = 0, called the sparsemax. Li et al. (2023) similarly considered q = 0 policy parameterization for the continuous action setting. All other works, however, used Gaussian policy parameterization to (implicitly) approximate an idealized target distribution that is q- g-Gaussian, and specifically for q Lee et al. (2020); Zhu et al. (2024). Such choice is suboptimal, approximate light-tailed (sparse) target policies. And in fact that choice was not strictly necessary as the policy parameterization need not have been chosen to be Gaussian: it could also have been a q- g-Gaussian. Though obvious in hindsight, this gap was likely due to simply not considering the use of general continuous g-exponential family for the policy parameterization, which is what we introduce in this work.
Gaussians are used to
In this paper, we empirically investigate the g-exponential family as replacement for the Gaussian inside several existing policy optimization algorithms. Our contribu- tions include the following.
Overall Performance Across TAWAC, IQL
Heavy-Tailed g-Gaussian
(1) We show how to use g-exponential family policy parameterizations inside a va- riety of existing actor-critic algorithms. (2) We provide comprehensive experiments on both online and offline problems showing that g-exponential family policies can improve on the Gaussian by a large margin. In partic- ular, find that the Student's policy is more stable, performing well across algorithms and problems, shown in Figure 2. (3) We provide empirical evidence supporting the assumption that algorithms may prefer specific policies depending on the actor loss objective. particular, we find that by replacing the Gaussian with heavy-tailed q- Gaussian, Tsallis Advantage Weighted Chtti-Criiic (Zhu et al., 2024) consistently performs better across offline benchmark problems. This outcome makes sense; as men- tioned above, this algorithm implicitly has a target policy that is a g-Gaussian, so using matching q- Gaussian parameterization should perform better.
Student's
Beta
100% of Squashed Gaussian
Gaussian
80
90
100
110
Performance relative to Squashed Gaussian (%)
Figure 2: Performance relative to the Squashed Gaussian on the offline D4RL MuJoCo task, averaged across the se- lected algorithms and environments.
2 BACKGROUND
We focus on discounted Markov Decision Processes (MDPs) expressed by the tuple (S, A, A. P,u,r, r.).. where S and A denote state space and action space, respectively. Let A(X) denote the set of probability distributions over X. and μ denote the transition probability and initial state distribution, respectively. r(s, a) defines the reward associated with that transition. 7 (0, 1) is the discount factor.
2

---

Published as a conference paper at ICLR 2025
A policy S - A(A) is mapping from the state space to distributions over actions. To assess the quality of a policy, we define the expected return as ((T) = p"(ss S4 (())s)((s, d)dads, where /" (s) = Ei= ? P(s+ s) is the unnormalized state visitation frequency. The goal is to learn policy that maximizes J(r).. We also define the action value and state value Q" (s, a)
En 12==0 7'r(st, (s) ~ ao = a). (s) = Ex [Q (s. a)}. For the ease of later notations, we write the dependence on state subscript, e.g. Q(s a) will be written as (a).
In practice, the policy is often parametrized by vector of parameters 0 E R". The policy can then be optimized by adjusting its parameters the high reward region utilizing its gradient information.
The Policy Gradient Theorem (Sutton et al., 1999) featured by many policy gradient methods states that the gradient can be computed by:
V 8) = Egpi' a 12 (a) Vo In (a;
In practice, the expectation is approximated by sampling. When the state space is large, the action value function also parametrized, leading to the Actor-Critic methods (Degris et al., 2012).
In contrast to the study of policy gradient algorithms, the impact of specific policy parametrizations on performance remains a less studied topic. Researchers typically consider policy parametrizations that can be written the following:
(a; 8) Z, exp (0 Φa(a)) = exp (0T (a()) -Z,)
(1)
Here, (a(a) is a vector of statistics and R" is a vector of parameters, Z, the normalizing constant ensuring the policy is a valid distribution and Z, == exp (2"). One immediate instance is the Boltzmann- (BG) policy (a; 0) = exp (Q) - Z.). where Z = In exp (Q, da the log-partition function. In the discrete case, also called the softmax transformation (Cover & Thomas, 2006). BG policy has been studied extensively in RL for encouraging exploration and smoothing the optimization landscape, name a few applications (Haarnoja al., 2018; Ahmed et al., 2019; Cen et al., 2022). However, evaluating the log- poratiiiooon function is in general intractable.
3 EXPONENTIAL AND q- g-EXPONENTIAL FAMILIES
We first review the commonly used policy parametrizations. They permit an expression using the exponential function. We arrive at the more general g-exponential family by deforming the exponential. In Table 2, we summarize all policies presented in the paper.
3.1 THE EXPONENTIAL FAMILY POLICIES
The Gaussian policy one of the simplest distributions one can consider due its omnipresence in statistics and parametric estimation as well as its widely available sampling procedure implemen- tations. Since evaluating the parppatition function of BG is intractable, due to the aforementioned
advantages many researchers consider the Gaussian policy instead: n. (a) V2na. exp (aa-a.))
20
For simplicity we drop the dependence on state s. To see it is a member of the exponential family, in Eq. (1) let = [ 2
1 for μ (-00, 00), a > 0; (a) = [a, a²]T, and Z, = In (V2To)
This amounts to setting Qa(a) = (a-1)² in the BG (Gu et al., 2016). We write Gaussian policy as (a) = Na: μ4, o2). The gradients of the Gaussian are Vp In (T,(aa (a=p) and
-
V, (a) (a-ي)² 1. On one hand, the Gaussian policy is simple to implement. On the other hand, when becomes small, Gaussian can be unstable due to overly large gradients and can prematurely concentrate on a suboptimal action. As a result, susceptible to noise/outliers and does not encourage sufficient exploration due to its thin tails. This paper investigates location-scale alternatives within the generalized g g-exponential family.
Another interesting member is the Beta distribution (Chou et al., 2017): Tetaa (a) T(a+a) [[]] a -0-1 a)--1, a E (0, 1), where I'(-) is the gamma function. It can be retrieved from equation by letting 0 = [a, "T. (a) = [ln a, In(1 - -a] Z, = [(a)(( T(o+B) Since Beta distri- bution's support is bounded between (0, 1), Chou et al. (2017) argued that it might alleviate the
3

---

Published as a conference paper at ICLR 2025
Family
Policy
Parameters 0
Statistics (a(a)
Normalization Z, (l
Gaussian
[a, a²]
V2Ta
Eq. (13)
exp
Beta
[a, []
Ina, In(1 - -a]
[(a)( [(++8)
v yowa ((+)
Student's
-2p
la, lo,a²]
Eq. (14)
r(
g-exp
4-Gaussian (g 1)
0 20
a0,a9)
Eq. (15)
Q G-Gaussian (1 < <(
Table 2: Policy parametrizations from the exp and q-exp families studied in this paper. We primarily interested in the location-scale family. Their multivariate forms are shown in Appendix A.
3.0
2.5
0.0
4,00
-2.
-5.0
Figure 3: exp, x and Ing x for q < 1 and g > 1. When = 1 they respectively recover their standard counterpart. For 1 the g-exp return zero values and hence q-exp policies may achieve sparsity. For 1, g-exp decays more slowly towards 0, resulting in heavy-tailed behaviors. The rightmost shows the q- g-Gaussian with different q-
bias introduced by truncating Gaussian densities outside the action space bounds. The beta policy is the only non-location-scale family distribution in this paper. However, we will show in the experiments, the Beta policy generally does not perform favourably against the Gaussian.
3.2 THE g-EXPONENTIAL FAMILY, HEAVY-TAILED AND LIGHT-TAILED POLICIES
Generalizing the exponential family using the geexponential function has been extensively discussed in statistical physics (Naudts, 2002; Tsaliis, 2009; Naudts, 2010; Amari & Ohara, 201 10. In the machine learning literature, the g-exponential generalization has attracted some attention since allows for tuning the tail behavior by adjusting the value of (Sears, 2008; Ding & Vishwanathan, 2010; Amid et al., 2019). The g-exponential and its unique inverse function q-logarithm are:
expxr x,
==
In xt-e-1
exp, x
Ing
(2)
(1- *
q x
where [[+ = m. 0}. q-exp/log generalize exp/log since limg-+1 exp, x = expx and limg-+1 Ing x In Similar exp. q-exp an increasing and convex function for g > 0. satisfying exp, (0) = 1. However, an important difference of q-exp is that exp,(a + b) # exp, (a) exp, (b)
unless q = We visualize g-exp/log in Figure
We now define the g-exponential family as:
Ta 0)
exp, (0 (a))) = exp, (0T (s 2g,0)
(3)
Zn
where 9, (.(a), Zg.s have similar meanings to equation 1. Note that Zy.e + exp, (Z/rs) unless 1. The g-exponential family includes the q Gaussian and Student's distributions described in the next subsections.
4

---

Published as a conference paper at ICLR 2025
3.2.1 9- P-GAUSSIAN
As the counterpart of Gaussian in the q-exp family, q-Gaussian unifies both light-tailed and heavy- tailed policies by varying the entropic index (Matsuzoe & Ohara, 2011):
(a - 11)2
(N = Z 9,8
exp1
20
+ 8
if 00 <9<1,
(4)
where Zg.s
0
if <
is heavy-tailed when < q-90 3 and light-tailed when q < 1. TN,, (a) is no longer integrable for q 2 3 (Naudts, 2010). We visualize these g-Gaussians in Figure 3.
Since popular libraries like the PyTorch (Paszke et al., 2019) do not have implementations of q- Gaussians available, discuss their sampling methods. It was shown by (Martins et al., 2022) that a sparse q q-Gaussian (q < 1) random variable permits a stochastic representation μ + rAu, where u Unif (8 is random sample from the N N-1 dimensional unit sphere. A is the scaled
7 Σd. r the radius of the distribution, and the ratio follows the Beta distribution r² R Beta ((2 g)/(1 9), N/22, where R is radius of the supporting sphere of the standard q- Gaussian 10,
matrix [
N
2-4 +
T-g
2
I
R
(5)
π
Notice that R depends only on the dimensionality N and the entropic index This method provides low-variance samples, but unfortunately it does not extend to q > 1. Therefore, for 1 < q < 3 we adopt the Generalized Box-Müller Method (GBMM) (Thistleton et al., 2007) to transform uniform random variables u1, t42 Uni f(0, 1) by the following:
21
2 2ng (u1)- - cos (2muz))
22
Ing (u:) sin (2Tr22)
Then each 21, Z2 is a standard G-Gassiian variable with new entropic index g' = (34 1) + 1). Often we know the desired g' in advance, in this simply let the q-log take the index The desired random vector is given by μa + Σiz.
q
1)/(3 d').
3.2.2 STUDENT'S S T
Heavy-tailed distributions like the Student's is are pop- ular for robust modelling (Lange et al., 1989). The Student's 's distribution is
Algorithm 1: GaGusssin sampling
Input: 9. N.N.
if < 1 then
(1+
11)3
sample u ~ Uni ((SN)
(a) VIVVO 1 (5)
TV
sample Beta
2-g N
I-g' 2
(7)
compute R per Eq. (5)
where > 0 is the degree of freedom. As V D0, Student's distribution approaches the Gaussian. Numerically, Student's is t with v 2 30 is considered to closely match the Gaussian. Therefore, v can be an important learnable parameter in addition to its location ju and scale o. It allows the policy to adaptively balance the exploration-explen exploitation trade- off by interpolating the Gaussian and heavy-tailed
+
compute A = Σ
2N+
return μa V 1RR Au
else if q > 1 then sample uA, U2 Uni Unif(00 1) compute by GBMM Eq. (6) return + Σ
policies. Now let q = 1 2 and define
D+I
TVO I (%)
Z (a 11) 0 (T).a))
2g,a
r (
(1 4) av
(8)
(s) exp, (0), (TT (a)) - -1n-- Zg,z)
5

---

Published as a conference paper at ICLR 2025
which we see is indeed q-exp policy and Z,.. -m2-- 2g.s. Student's t policy has been used in (Kobayashi, 2019) to encourage exploration and to escape local optima. Another related case the Cauchy's 's distribution recovered when q = 2 (or be used as the starting point for learning Student's 's t. Note that Cauchy's distribution does not have valid mean, variance any higher moments.
1 from Student's t). Cauchy's distribution can
4 USING G-EXPONENTIAL FAMILIES FOR ACTOR-CRITIC ALGORITHMS
In this section, we outline three key actor-critic algorithms we use in our study and the nuances of incorporating q-exp policies into them. For example, the q-exp policies may not have closed-form Shannon entropy. Therefore, approximations are needed for algorithms like SAC and GreedyAC. Moreover, though for the Gaussian evaluating the log- oggliiklihood for off off-polic/ooflinee actions causes no problem, it raises new issue for the light-tailed g- gaGusssian, since these actions can fall outside of its support.
Soft Actor-Critic SAC (Haarnoja et al., 2018) encourages exploration by adding to reward the Shannon entropy. The actor minimizes the following KL loss
exp (T (Q)) )
(0) = -EgB [Dx (To.s TBG.,) = EA-B DKL TØ.s
Z,
where states are sampled from replay buffer B. The parametrized policy Te. is projected to be close to the BG policy. By default mg is chosen be the Gaussian policy, but potentially a more exploring policy like the Student's 's could be better. Depending on action values, BG can have multiple modes and heavy tails. The Gaussian may not be able fully capture these characteristics.
Greedy Actor-Critic. GreedyAC (Neumann et al., 2023) maintains an additional proposal policy for exploration by maximizing Shannon entropy augmented rewards. Its actor policy maximizes unbiased reward and learns from the high-quality actions generated by the proposal policy, To simplify notations, we use (s) to denote the set of high quality actions given s.
poup (p) = a LGreedyCC peup
[- In To.. H (TT
aE((s)
LereedyAC, actor ( = E B
In T..
(sl(s)
GreedyAC maximizes log- ligllikeihhod of the actor and entropy augmented likelihood for the proposal policy. Note that the when Td,a q-exp policy, it may not have closed-form Shannon entropy expression. Therefore, we can use oog-robbabilities as surrogate just like in SAC.
Tsallis Advantage Weighted Actor-Critic. TAWAC (Zhu et al., 2024) proposed use a light- g-exp policy for offline learning. However, the light-tailed distribution was approximated with the Gaussian which an infinite-support policy. Let ap denote the empirical behavior policy and DD the offline dataset. TAWAC minimizes the following actor loss:
exp, ((olu)-Ve In mosalaa
-V8 Vo
(())= E [DEL (TTKL., [o.s)] = E aND a~Ep
where (a) (a) (r (Q. V.)) denotes the Tsaliis KL regularized policy. Tø.s mimics a TKL policy which can be sparse depending on q'. In this case, it is natural to expect that a sparse policy parametrization may lead to better performance.
Algorithms like TAWAC that sample from behavior policy RD needs extra caution when using the g-exp policies. When the light-tailed q- g-Gaussian is used, numerical issues can be incurred since the action sampled may fall outside the support of T&. leading undefined log-likeliine likelihood. To resolve this problem, we propose sample from To batch of K actions and replace the oft-of-support action with the in-support one with least T, distance, see Alg. 2.
Algorithm 2: Out-of-support action han- dling for the light-tailed q- Gaussian
Input:
out-of-support action a sample in-support actions {b, bb>k
solve = arg min, |b, - al12 return b,.

---

Published as a conference paper at ICLR 2025
200
3.0
400
15000
SAC
reaard
-600
-200
-000
-900
-3000
-1000
10.00
-000
2000
-500
Grendy-AC
500
-400-
-10000
revord
-2700
300
-900
-3500
-1000
10000
TAWACII
800
400-
1.000
C00
0000
-2000
1000
10000
200
-000
TAWACRS
reaar
400
1000
-1200
Rata
2700
Light-tailed Gaussian
Student-t
tailed Gaussian
1000
-1000
20001 1000 10000 3000 2000 30000
10000 10000
10000
30000 20000 30000
20000 103000 1:300
200000 250000 200000
Figure 4: Learning curves on the classic control environments. Only the Gaussian and the best policy parametrization for each setting were shown with full opacity. The best policy is picked based on the total area under the curve (AUC). TAWAC(() refers to TAWAC with entropic index o' = 0 in Eq. (9). Despite tuning hyperparameters separately for each policy, Gaussian is the best policy in only 1/12 settings. most other settings, the Gaussian policy performs significantly worse than the best.
2001
Perertage of time batter than Gausiin
10
00
60
3
Haany-
Studert44
Belt
Ligh- sindd
Haenty 201ee Studer14
Deta
Light-Tt led
Gausesian
Figure 5: (Left) The percentage of times that each policy parametrization is better than the Gaussian across all line, then means that the policy parametrization better than Gaussian on average. Student's i and Light-tailed Gaussians are better than the Gaussian in 75% and 66% of the settings, respectively. (Right) Count of times where policy parametrization performed the best across all snborithm-euvironmen combinations based on AUC. We observe that the student-t policy performed the best in 5/ 12 settings, whereas the Gaussian policy performed the best only once.
combinations based on total AUC. If the bar is above the 50%
see that
EXPERIMENTS
Our empirical study's 's primary goal is to understand better the performance differences under this broader class of policy parameterizations in both online and offfine settings. We experiments
7

---

Published as a conference paper at ICLR 2025
with different algorithms, to get a better sense of how conclusions about policy parameterization vary across different actor-critic algorithms.
We parametrize Student-t's DOF parameter in addition its location and scale. By contrast, the heavy-tailed q- g-Gaussiam is fixed at q = 2. since its allowable range is < q < 3. For the light-tailed g-Gaussian, opt for the standard choice of q 0. Since Student's heavy-tailed q- g-Gaussian, and Gaussian have unbounded support, we clipped the sampled action to fit the task's s action space without modifying the density. We swept the hyperparameters using five random seeds, then increased the number of seeds to 10 for the best parameter setting. The hyperparameter sweep ranges and the best values are provided in Appendix D.2 and D.3.
5.1 ONLINE CLASSIC CONTROL
Domains and Baselines. We used three classical control environments in the continuous action setting: Mountain Car (Sutton & Barto, 2018), Pendulum (Degris et al., 2012) and Acrobot (Sutton & Barto, 2018). We chose the cost-to-gon -oga version of Mountain Car, which outputs - 1 reward per time step to encourage reaching the goal early. We compared SAC, GreedyAC and two versions of TAWAC, o' = 0 and
Results. Figure shows the learning curves of all algorithm- -nviromment combinations. Only the Gaussian and the environment specific best policy are shown with full opacity, computed based on area under curve (AUC). One immediate observation is that, though all three algorithms by default choose the Gaussian policy, it was seldom the best policy parametrization. Environment-wise on Mountain Car the Gaussian did not rank the best for any of the algorithms. By contrast, the Beta policy attained the first place with SAC, as was the light-tailed q- Gaussiia with TAWAC. The same trend for the Gaussian holds in Acrobot and Pendulum as well, with exception only on TAWAC(0) Acrobot, where its curve closely resembled that of the light-tailed q-Gaussian.
Algorithm-wise three observations are to be made: (i) on Mountain Car the Beta policy performed significantly better than others. This could be due its flexibility in maintaining skewed distribution shape that matches the BG policy more closely in contrast to the other location scale family members. (ii) The Gaussiias in general outperformed the Gaussian on TAWAC(0) and TAWAC(2) whose actor explicitly mimics a q-exp policy. (iii) Student's 's has ranked the top involving all three algorithms. Figure LHS summarizes the percentage of each policy parametrization outperforming the Gaussian. The Student's i t and light-tailed Gaussian went above 50%, suggesting potentially greater applicability. The RHS shows out 12 total combinations, how many times each policy parametrization has ranked the top. The result shows that the Student's is attained 5 times, contrasting the 1 time of the Gaussian.
In Figure 6 we visualized the evolution of Gaus- sian and G-aaussian policies the starting state over the first 4 X 10a steps (10% the entire learning horizon). Note that the allowed action range is [] but the plot shows [- for better visualization. Gaussian tends quickly concentrate like a delta policy. This can be detri- mental algorithms like SAC and Greedy AC which demand stochasticity to generate diverse samples. By contrast, both light- and heavy- tailed Gaussiaass tend to be more stochastic.
Greedy-AC Policy Evolution on Mountain Car
Gaussian
Heavy-talled q-Gaussian Light-tailed q. G-Gaussian
1
Density
25 1 2
In Figure 11 we show the Manhattan plot of SAC with all swept hyperparameters on all envi- ronments. Though there no definitive winner for all cases, it is visible that the Student's ss and Gaussian have a similar behavior to hyperpa- rameter changes. Therefore, if we are tackling problem where the Gaussian works, the Student- is very likely work. And judging from Fig, 5, we know that Student's t is 75% more likely to perform better than Gaussian given the same hyperparameter sweeping range.
Steps (xint)
Action
Figure 6: Policy evolution of Greedy on Moun- tain Car. The Gaussian collapsed into a delta-like policy after only 10% of the learning horizon.
8

---

Published as a conference paper at ICLR 2025
Squashed Gaussian
Student's
Heavy-Talled q-Gaussian
Repiuu Data
HalfCheetah
Hopper
Walker2d
Normalized alized Retum
100
50
25
0
TAWAC AWAC
IQL
InAC
TAWAC AWAC
IQL
InAC
TAWAC AWAC
IQL
InAC
Figure 7: Normalized scores on Medium-Replay level datasets from the MuloCo suite. The black bar shows the median. Boxes and whiskers are 1 and 1.5x interquartile ranges, respectively. See Figure 15 for full comparison. Environment-wing TAWAC with heavy-tailed q G-Gasssian is often the top performer. Algorithm-wii Student's consistently outperforms Squashed Gaussian.
5.2 OFFLINE D4RL MuJoCo
Gaussian
Beta
Student's t Medium Data
Heavy-Tailed G-Gaussian Medium-Replay Data
Medium- Medium-Expert Data
Squa Squashed Gaussian
120%
100%
Proportion
Performanc
60%
TAWAC AWAC
TAWAC AWAC
TAWAC AWAC
Figure 8: Relative improvement to the Squashed Gaussian policy, averaged over multiple environ- ments in the MuJoCo suite. The Student's consistently outperforms the Gaussian with all the chosen algorithms. The heavy-tailed g-Gaussian with TAWAC and IQL also achieved significant improvement. The improvement can reach up 20%. Black vertical lines at the top indicate one standard error.
Domains and Baselines. We used the standard benchmark MuJoCo suite from D4RL to evaluate aggorttmm-policy combinations (Fu et al., 2020). The following algorithms are compared: TAWAC, Advantage Weighetd Actor-Critic (AWAC) (Nair al., 2021), Implicit Q-Learnng (IQL) (Kostrikov et al., 2022), In-sample Actor-Critic (InAC) (Xiao et al., 2023). For TAWAC, we fixed its leading deeep with o' O. In Appendix C.2 detailed the compared algorithms. We also included popular variant of the Gaussian known as the Squashed Gaussian for comparison. Being able to evaluate the offline tog-probability is critical the tested algorithms, found that light-tailed g-Gaussian leads to poor performance even with random online sampling, hence we do not show them here.
Results. Figure 7 compared the normalized scores on the Medium Replly datasets. It can be seen that environment-wise TAWAC heavy-tailed q- g-Gaussian was the top performer, and could improve on the Squashed Gaussian by non-negligible margin. On HalfCheetah, heavy-tailed g-Gaussian attained the best score with every algorithm. Algorithm-wise, the heavy-tailed q- g-Gaussian or/and Student's t were better or equivalent to the Squashed Gaussian, except with AWAC on Hopper. Student's was stable across algorithms, including these with which heavy-tailed G-ausssi performed poorly (c.g.. InAC). This demonstrates the value of the learnable DOF parameter that allows interpolates the Gaussian. In Appendix E we provided comparison on other policies and datasets.
Figure 8 summarized the relative improvement over the Squashed Gaussian across environments. Several observations can made: (i) though the Squashed Gaussian outperformed the Gaussian in general, it seldom the best performer. (ii) the Student's could consistently perform better

---

Published as a conference paper at ICLR 2025
Heavy-Talled a-Gaussian
Student's t
Gaussian
Dimmensio
Dimension
Dut-t By Syrtes Biff- Bohw7.47
Dot-a Bet.sepe Stf 9g 9ghwa.5
Eg Sy Rd
M 1m
Dof-3 Fgr
Deokk
Drtaky
oth Dimenston
5th Dimensson on
0th Dimension
22.80
Datus Sgriup Ritme Re Sof
Sprius
Denur
Anten
Actes
Adee
Figure Policy evolution of all actions dimensions of TAWAC on Walker2d Medium Replay. Student's t was flexible in that on some dimensions it had lighter tails like the Gaussian by having large DOF (e.g. 4th), and with heavier tails on the others by having smaller DOF (e.g. 3rd, 6th). The peaks at the edges were caused by clipping actions into the allowed range.
than the Gaussian, the improvement can sometimes reach up to ~ 20%. The same holds for the heavy-tailed q- g-Gaussian with TAWAC and IQL. (iii) though there was no single winner for all cases, choosing the Student's for the actors with exponential loss functions (AWAC, IQL, InAC), or the heavy-tailed y-Gaussian for q-exponential actor losses (e.g. TAWAC) are generally effective.
Figure 9 visualized the policy evolution of the Squashed Gaussian and the two heavy-tailed policies, learned with TAWAC on Medium Replay Walker2D. Squashed Gaussian tended to converge slower here. Since the offline MuJoCo environments are fully deterministie, a wide distribution indicates failure of finding the mode of the optimal action and therefore can be detrimental to learning performance. The Squashed Gaussian converged slower than the heavy-tailed (performed the best) and the Student's t. Student's is t was flexible in that it beared lighter tails like the Gaussian in some dimensions by having a large DOF, for example in the 4th and 5th dimensions. On the other hand, could take heavy tails by having small DOF like in the 3rd and 6th dimensions.
6 CONCLUSION
The Gaussian policy is standard for policy optimization algorithms continuous action spaces. In this paper we considered a broader family of policies that remains tractable, called the g-exponential family. We empirically investigated their utility promising alternative the Gaussian. Specifically, we looked at the Student's 's t, light- and heavy-tailed g-Gaussian policies. Extensive experiments on both online and offline tasks with various actor-critic methods showed that heavy-tailed policies are in general effective. In summary, we found the Student's 's t policy to be generally more performant and stable than the Gaussian and could be used drop-in replacement. By contrast, the Heavy- tailed y-Gaussian seemed to favor especially T'allis regularization and outperformed the baselines.
We acknowledge that the paper has limitations. Perhaps the greatest is the inherent dilemma of the light-tailed g- Gaussian evaluating out-of-support actions. Off-policy/oflline algorithms require evaluating actions from some behavior policy and the actions can fall outside the support of the sparse q- g-aaussian. Naively discarding these samples results extremely slow or no learning. In this paper we proposed to alleviate this issue by replacing them with the in-support sampled action with the least Lg distance. Nonetheless, this method did not help much in offline experiments. We envision potential solution that is left to future investigation: projecting the out-of -u-ppor actions precisely to the boundary of the g- Gaussian.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans. Understanding the impact of entropy on policy optimization. In Proceedings of 36th International Conference on Machine Learning, volume 97, pp. 151-160, 2019.
Shun-ichi Amari and Atsumi Ohara. Geometry of q-exponential family of probability distributions. Entropy, 13(6):1 100011877-11,, 2011.
Ehsan Amid, Manfred K. Warmuth, and Sriram Srinivasan. Two-temperature logistic regression based on the tsallis divergence. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89, pp. 2388 2396, 2019.
Amrit Singh Bedi, Anjaly Parayil, Junyu Zhang, Mengdi Wang, and Alec Koppel. On the sample complexity and metastability of heavy-tailed policy search in continuous control. Journal of Machine Learning Research, 25(39):1 2024.
Boris Belousov and Jan Peters, Entropic regularization of markov decision processes. Entropy, 21(7). 2019.
Shicong Cen, Chen Cheng, Yuxin Chen, Yuting Wei, and Yuejie Chi. Fast global convergence of natural policy gradient methods with entropy regularization. Operations Research, 70(4): 2563-2578, 2022.
Souradip Chakraborty, Amrit Singh Bedi, Kasun Weerakoon, Prithvi Poddar, Alec Koppel, Pratap Tokekar, and Dinesh Manocha. Dealing with sparse rewards in continuous control robotics via heavy-tailed policy optimization. In 2023 IEEE International Conference Robotics and Automation (ICRA), pp. 989-995, 2023.
Po-Wei Chou, Daniel Maturana, and Sebastian Scherer. Improving stochastic policy gradients in continuous control with deep reinforcement learning using the beta distribution. In Proceedings of the 34th International Conference on Machine Learning, pp. 334-843, 2017.
Yinlam Chow, Ofir Nachum, and Mohammad Ghavamzadeh. Path consistency learning in Tsallis entropy regularized MDPs. In International Conference on Machine Learning, pp. 979-988, 2018a.
Yinlam Chow, Nachum Ofir, Edgar Duenez-guzman, and Mohammad Ghavamzadeh. A Lyapunov- based Approach to Safe Reinforcement Learning. In Annual Conference Neural Information Processing Systems (NIPS), pp. 1-10, 2018b.
Thomas M. Cover and Joy A. Thomas. Elements of Information Theory (Wiley Series in Telecommu- nications and Signal Processing). Wiley- eiley-Interscience, USA, 2006.
Thomas Degris, Martha White, and Richard S. Sutton. Off-policy actor-critic. In Proceedings of the 29th International Coference on International Conference on Machine Learning, pp. 179-186, 2012.
Nan Ding and S.v.n. Vishwanathan. t-logistic regression. In Advances in Neural Information Processing Systems, volume 23, 2010.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach offline reinforcement learning, In Thirty-Fifth Conference on Neural information Processing Systems, 2021.
S. Furuichi, K. Yanagi, and K. Kuriyama. Fundamental properties of tsallis relative entropy. Journal of Mathematical Physics, 45(12):4868-44 4877, 2004.
Shigeru Furuichi. On the maximum entropy principle and the minimization of the fisher information in tsallis statistics. Journal of Mathematical Physics, 50:013303, 01 2010.
Peter Grünwald and Alexander Dawid. Game theory, maximum entropy, minimum discrepancy and robust bayesian decision theory. Annaly of Statistics, 32, 2004.

---

Published as a conference paper at ICLR 2025
Shixiang Gu, Timothy Lillicrap, lly Sutskever, and Sergey Levine. Continuous deep q-learning with model-based acceleration, In Proceedings of The 33rd Interattonaal Conference on Machine Learning, pp. 2829 2829,2838, 2016.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, pp. 1861-1870, 2018.
E. T. Jaynes. Information theory and statistical mechanics. Phys. Rev., 106:620-6255 630, 1957.
Taisuke Kobayashi. Student-t policy in reinforcement learning to acquire global optimum of robot control. Applied Intelligence, 114335--3347 2019.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2022.
Kenneth L. Lange, Roderick J. A. Little, and Jeremy M. G. Taylor. Robust statistical modeling using the distribution. Journal of the American Statistical Association, 84:881-896, 1989.
Kyungjae Lee, Sungjoon Choi, and Songhwai Oh. Sparse markov decision processes with causal sparse tsallis entropy regularization for reinforcement learning. IEEE Robotics and Automation Letters, 3:1466-1473, 2018.
Kyungjae Lee, Sungyub Kim, Sungbin Lim, Sungjoon Choi, Mineui Hong, Jae In Kim, Yong-Lae Park, and Songhwai Oh. Generalized tsallis entropy reinforcement learning and its application to soft mobile robots. In Robotics: Science and Systems XVI, pp. 1-10, 2020.
Yuhan Li, Wenzhuo Zhou, and Ruoqing Zhu. Quasi-optimal reinforcement learning with continuous actions. In The Eleventh International Conference on Learning Representations, 2023.
Andrā@ F. T. Martins, Marcos Treviso, Antãônio Farinhas, Pedro M. Q. Aguiar, MAGíio A. T. Figueiredo, Mathieu Blondel, and Vlad Niculae. Sparse continuous distributions and fenchel- young losses. Journal of Machine Learning Research, 23(257):1-74, 2022.
Hiroshi Matsuzoe and Atsumi Ohara. Geometry of e-exponential families. Recent Progress in Differential Geometry and Its Related Fields, pp. 55-71, 2011.
Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the global convergence rates of softmax policy gradient methods. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pp. 6820-6829, 2020.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. (AWAC): Accelerating online reinforcement learning with offline datasets, 2021.
Jan Naudts. Deformed exponentials and logarithms in generalized thermostatistics. Physica A- statistical Mechanics and Its Applications, 316:323-334. 2002.
Jan Naudts. The g-exponential family statistical physics. Journal of Physics: Conference Series, pp. 012003, 2010.
Samuel Neumann, Sungsu Lim, Ajin George Joseph, Yangchen Pan, Adam White, and Martha White. Greedy actor-critic: A new conditional cross-entropy method for policy improvement. In The Eleventh International Conference on Learning Representations, 2023.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Detito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, rerforrrformncce deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024-8035 2019,
Ben Peters, Vlad Niculae, and André F. T. Martins. Sparse sequence-to-sequeee models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1504-1519, 2019.
12

---

Published as a conference paper at ICLR 2025
Kaare Brandt Petersen and Michael Syskind Pedersen. The matrix cookbook. 2012.
Timothy Sears. Generalized Maximum Entropy, Convexity and Machine Learning. PhD thesis, The Australian National University and Computer Science Laboratory, Research School of Information Sciences and Engineering, 2008.
Umut Simsekli, Levent Sagun, and Mert Gurbuzbalaban. A tail-index analysis of stochastic gradient noise in deep neural networks. In Proceedings of the 36th International Conference on Machine Learning, Proceedings of Machine Learning Research, pp. 5827-5827757 2019.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. Bradford Book, Cambridge, MA, USA, 2018.
Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems (NIPS), pp. 1057-1063, 1999.
H. Suyari and M. Tsukada. Law of error tsallis statistics. IEEE Transactions on Information Theory, 51(2):553-757 2005.
William J. Thistleton, John A. Marsh, Kenric Nelson, and Constantino Tsallis. Generalized box müller method for generating q q-gaussian random deviates. IEEE Transactions on Infor- mation The 53:4803-4481111 2007.
C. Tsallis. Introduction to Nonextensive Statistical Mechanics: Approaching Complex World. Springer New York, 2009. ISBN 9780387853581.
Chenjun Xiao, Han Wang, Yangchen Pan, Adam White, and Martha White. The in-sample soft- max for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023.
Haoran Xu, Li Jiang, Jianxiong Li, and Xianyuan Zhan. A policy-guided imitation approach for offline reinforcement learning. Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang. Victor Wai Kin Chan, and Xianyuan Zhan. Offline RL with no OOD actions: In-sample learning via implicit value regularization. In The Eleventh International Conference on Learning Representations, 2023.
Lingwei Zhu, Zheng Chen, Matthew Schlegel, and Martha White. Generalized munchausen rein- forcement learning using tsallis kl divergence. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Lingwei Zhu, Matthew Schlegel, Han Wang, and Martha White. Offline reinforcement learning with tsallis regularization. Transactions Machine Learning Research, 2024.
Brian D. Ziebart. Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy. PhD thesis, Carnegie Mellon University. Carnegie Mellon University, 2010.
13

---

Published as a conference paper at ICLR 2025
APPENDIX
The Appendix is organized into the following sections. section A we summarize the multivariate form of q-exp policies and derive gradients of their log-likelihood. In section C we discuss the connection between the q-exp family and the entropy regularization literature. Based on this, we further discuss how different algorithms may prefer specific policies depending on its actor loss. We then provide implementation details including hyperparameters and how to sample from g Gaussian in section D. Lastly provide additional experimental results in section E.
A Multivariate q-exp Policies and Log-likelihood
Connection to Entropy Regularization
C Actor Losses
D Implementation Details
E Additional Results
A MULTIVARIATE DENSITY OF Q-EXP POLICIES
Policy
Density
V (In
Gaussian
exp
2 (a (a)=)) E-1 (a 11)
(13)
(2x) x 121/
I( 1(
1 + v (a - 11) Σ- (a --) Nir
Student's 's t
Eq.
T(3) Σ
(1-g) 4 1(
exp2 ( //4 (a n1) 1 E-1 (a -1i)
Gaaussin (g 1)
r( ) 4 トエ
Eq. (15)
(o- + T( (x
g-Gaussian (1 < 3)
exp, (-1 (a u4) Eー
T (a u))
T( (( 3/2 ز39
Table 3: Multivariate q-exp policies and gradients of log-likelihood.
In Table we show multivariate density of the q-cxp policies introduced in the main text. Note that multivariate Student's is constructed based on the assumption that diagonal E leads to independent action dimensions, same as the Gaussian policy. On the other hand, for g-Gaussian this is no longer true, since a diagonal Σ does lead to product of univariate densities.
In the main text we showed their one-dimensional cases for simplicity. For experiments the multivari- ate densities were used for experiments. We now derive their gradients of log-likelihood with respect to parameters. The following equations will be used frequently (Petersen & Pedersen, 2012):
V (a - -1) T
(a - μ) = -2 -4). (p)+
(10)
Vx In ΣΣ] = (ΣT) -1
(11)
Vg(a - 10) - (a p1) = - ---'(a
(a - (a)(a 11) T-
(12)
With these tools in hand, the following gradient expressions be readily derived.
14

---

Published as a conference paper at ICLR 2025
A.1 GAUSSIAN
Being member of the exponential family, the gradient of Gaussian log-likelihood allows straightfor- ward derivation by using Eq. (10)-Eq. (12):
N
In Tu (a) = 2
In 2T
|Σ| 2 (a - 11) x-> -4)
2
V (a) (a 1t),
(13)
VylnT,(()) - ( E (a 1)( 1-1--
A.2 STUDENT'S T
In addition to μ.. Σ. Student's policy has additional learnable parameter degree of freedom
Recall that v = 1 corresponds to the Cauchy's distribution, while numerically with v 2 30 can be seen Gaussian distribution.
In n. (a) In
N +
In
(")
N
In VT
In |
N + In 1
1 (a (a-+) 1 14)
2
2
2
N +
E (a 11)
V (a)
V
1 1 (a 11) T 1 - 11)
1
(N + 1 (a 11) 11) T 1
Vy.laT(()) =
Σー1
v (a 11) E-1 14)
(o)n = =
N + 1/
1/
1/
N
N
In
1 - aa- 11) Σ-
= (a 11)
2
2
2v
2
N + } n1) 2 (a 11)
+
2 =サ (a 11) T -11)
(14)
where (b) is the digamma function. For μa and E. we again leveraged Eq. (10)-Eq. (12).
A.3 q- G-GUSSSINN
Since we do not parametrize the entropic index g, the gradients of log-likelihood with respect to μa, are the same for both heavy- and light-tailed g-Gaussian. Therefore, we focus on the light-tailed case q < 1 and absorb into the constant C the terms only related to q.
In (a) In C
In Σ
In
1-
(a 11) 11) T 11)
2
1
(1 - 9)2 1a
(a 1))
2 D-1 11)
Vn (a)
1-9 1-2( (a 1)) TT 1a-((a- --)]+ expy
(- (() T 11)) 11))1-9²
2 Σ-1 Σ- - -1)(a- -11) 11-
VyInT, =
exp, (- (/ TE (1)= 1-9
(15)
is interesting to see that the gradients of g-Gaussian log-likelihood can be seen as the Gaussian counterparts scaled by the reciprocal of (.) 9, Since exp, can take zero values when < 1, the gradients as well as the log- likeelioo function may be undefined outside the support. However, this does not happen for heavy-tailed q- Gaussiia 1 q 3.
To make these policies suitable for deep reinforcement learning, we discuss in Appendix how to parametrize the policies using neural networks.
B CONNECTION TO ENTROPY REGULARIZATION
The e-exp family provides a general class of stochastic policies. But perhaps more importantly, they can be derived solutions to the maximum Tsallis entropy principle (Suyari & Tsukada, 2005;
15

---

Published as a conference paper at ICLR 2025
Furuichi, 2010), generalizing the maximum Shannon entropy principle (Jaynes, 1957; Grünwald & Dawid, 2004; Ziebart, 2010). We discuss both principles in equation 16.
For notational convenience, we define the inner product for any two functions F1, F2 E IR S|x|A| over actions as (Fi, F2) R We write F, express the function's dependency F on state 8. Often F. E Iili whenever its component is of concern, we denote it by Fa(a).
B.1 BAIIZMANN-GIBBE REGULARIZATION
Consider regularized policy as the solution to the following regularization problem:
#2,s arg max (T Qs) [((r))),
(16)
T.EAA
where Ω is a proper, lower semi-continuous, strietly convex function. We can absorb the regularization
coefficient r > 0 into by $2 rΩ.. is classic result that the limit - 0 the unregularized optimal action is recovered: lim,-0 - = 1[a a" i.e., a" that maximizes Q..
One of the most well-studied regularizers is the negative Shannon entropy l((T.) (T.. ln which leads to the Boltzmann- Gibbs policy (a) = exp (Q.(a) Z.). Another popular choice is the KL divergence (T(T.) = (T,, In n. In (ua) for some reference policy Msa The regularized policy (a) = (a(a) exp (Q) Z.). Notice that it is also a member of the exponential
family by writing TKI., (a) = exp (Q.(a) 2. + (mm()aa)))
B.2 TSALLIS REGULARIZATION
Originally, the deformed logarithm function introduced in the statistical physies to generalize the Shannon entropy by deforming the logarithm contained in it (Naudts, 2010). Consider replacing
Shannon entropy in equation 16 with the negative Tsallis entropy (Tx) - ((1, T3) 1).
has been shown that 22y (T,) leads to following regularized policy:
T (a) exp2-9 q (Q.(a) Zn,s).
(17)
We see that when q = recovers the sparsemax policy introduced in Section 3.2. As indicated by (Zhu et al., 2023), the effect of different 9 E (-o0, 1) lies in the extent of thresholding. One
can also consider regularization by the Tsallis KL divergence Dir. (T& Iss) = (Te, Ing (Furuichi et al., 2004). Likewise to the KL case, 4l typically taken to be the last policy, in which the regularized policy is the product of two g-exp functions.
is worth noting that there are other regularization functionals that can induce q-exp policies. One of the prominent examples is the c-entropy/divergeneee which can be defined by simply letting p q in 22y (xx)) (Peters et al., 2019; Belousov & Peters, 2019). It is shown in (Xu al., 2022; 2023) that when a = -1 it induces the sparsemax policy. Therefore, q-exp policies can also be viewed as solutions to the a regularization.
B.3 TSALLIS ADVANTAGE WEIGHTED ACTOR CRITIC
An advantage of g-exp (resp. exp) policies is it may improve the consistency of algorithms that explicitly mimics q-exp (resp. exp) policy. For example, Tsallis Advantage Weighted Actor Critic (TAWAC) proposed to use a light-tailed q-exp policy for offline learning (Zhu et al., 2024). However, TAWAC was implemented with Gaussian, which amounts approximating light-tailed distribution using one with infinite support. Let denote the empirical behavior policy and D the offline dataset. TAWAC minimizes the following actor loss, where we ignore the parametrization of value functions:
(Q,(a) -Va In (a)
((Φ) = EE-D (TTKL.. x) = E AND o~E
exp,
(18)
where TTKL., (a) x TD.=(a exp, (Q.(a) V=)) denotes the Tsallis KL regularized policy, We can generalize TAWAC to online learning by simply changing the expectation to be w.r.t. arbitrary behavior policy. is clear that depending on q'. choosing Gaussian as T& may incur inconsistency with the theory. A q-exp policy would be more suitable and could improve the performance. As evidenced by our experimental results, heavy tailed policies indeed further improve the performance of TAWAC by large margin.
16

---

Published as a conference paper at ICLR 2025
C ACTOR LOSSES
To help understand when exp-family policies (resp. q-exp) may be more preferable, we compare the actor loss functions of the algorithms in the experiment section.
C.I ONLINE ALGORITHMS
Soft Actor-Critic. SAC minimizes the following KL loss for the actor
exp (T- (r-1Q(s,)
(oAc( == 1B (Dk((T(()ss)) TBG(-|s)) = 1B DKL
TT(-(s)
Z.
where states sampled from replay buffer B. The parametrized policy T is projected to be close to the BG policy, therefore is reasonable to expect that choosing Te from the exp-family may be more preferable. Depending action values, BG can be skewed, multi-modal Therefore, the symmetric, unimodal Gaussian may not be able to fully capture these characteristics.
Greedy Actor-Critic. GreedyAC maintains additional proposal policy besides the actor. The proposal policy is responsible for producing actions from which the top k% of actions are used to update the actor. The proposal policy itself updated similarly but with entropy bonus encouraging exploration. To simplify notations, we use ((s) to denote the set containing top k% actions given s.
GroodyAC, pe) == E2 s In (als)) - H (Ts (-1)))
aE](s)
actor LGreedyAC, actor (Ø) == E e~B
|- In (alss)
aEi(a)
GreedyAC maximizes log-likeli of the actor and proposal policy. These policies impose no constraints on the functional form of π.
Online Tsallis AWAC. Online TAWAC extended to condition on the behavior policy that collects experiences Ttheory (a|s) x Abchavior (als) expa ((s.a)-V(s)
(TAWAc(O) EgNB DKL (Tiheory (-18) (-(-(s))
((s
- Vss) In (als)
s a~73
exp,
where the condition Fg is because the target policy is used to sample actions. Since Tsallis AWAC explicitly minimizes KL loss to q-exp policy, which can be hight-taile//haayyttiill depending on Therefore, choosing g-exp my could lead to better performance.
C.2 OFFLINE ALGORITHMM
AWAC. Advantage Weighted Actor-Critic (AWAC) is the basis of many algorithms. AWAC minimizes the following actor loss:
(Q(s.a) (())
In Inmollls))
LAWAC = AND
exp
awED
T
which is derived the result of minimizing KL loss DKI (TD Ra) and applying the trick in Eq. 18, ((s.a)-V(s) - In (n However, the exp
i.e, Tiheory (a|s) x (als)) exp
(Q())))v(s))
shape of this policy can be multi-modal and skewed depending on the values and .т. It is visible from experimental results that Beta and Squashed Gaussian have similar performance.
IQL. In contrast to AWAC, Implicit Q-Learning (IQL) does not have an explicit actor learning procedure and uses (awac(ó) means for policy extraction from the learned value functions. The exponential advantage function acts simply as weights. Therefore, IQL does not assume the functional form of πT.
InAC. In-Sample Actor-Critic (InAC) proposed to impose an in-sample constraint on the entropy- regularized BG policy. As such, the dependence the behavior policy is moved into the exponential- advantage weighting function:
(((s.a)
V(s) In ( In mol(als)
LIaAC = E anD
exp
17

---

Published as a conference paper at ICLR 2025
Beta, distribution B(a, (8)
B(5,5)
((5
B)2,5)
3,05,
Figure 10: Beta distribution with 1, < takes bowl shape rather than bell shape. The shape can also be skewed as well as symmetric.
As result, InAC is not as sensitive the advantage weighting AWAC does, which implies that InAC may favor exp Ty but less than AWAC.
Offline Tsallis AWAC. The offline case of Tsallis AWAC same the online case except the change of expectation:
(TW == EE anD awoD
exp,
In (als)
Same with the online case, offline Tsallis AWAC may theoretically prefer q-exp Te-
TD3BC. In Appendix E we include additional results of TD3BC (Fujimoto & Gu, 2021), whose actor loss obtained by simply augmenting the TD3 loss with behavior cloning term:
LIDBBC ()) == sND
XQ(s, (s)) (π(() a)²]
The behavior cloning term is simply minimizing the La distance to actions in the dataset. Though another interpretation by (Xiao et al., 2023) is that this term can understood as applying KL regularization to Gaussian policy.
D IMPLEMENTATION DETAILS
Details of our implementation is provided in this section. Specifically, detail our design choices hyperparameters and network architectures.
D.1 POLICIES
We discuss how to parametrize Beta, Student's and q- g-Gaussian policies. Specifically, we parametrize a, 3 for Beta policy; M for g-Gaussian. In additional to location and scale, Student's 's has an additional learnable parameter v.
For Student's policy, we initialized base DOF = 1 and leam v/ by the softplus function. The Student's policy therefore always has DOF 1, which is equivalent starting as the Cauchy's distribution. For Beta policy, we similarly constrain a, 8 to be the output of softplus function plus 1. Beta policy takes on bowl shape rather than bell shape, Figure 10. For Gaussian and q- g-auussian policies, we follow the standard practice to parametrize mean by the tanh activation and scale by the log-std transform.
This is because when a < 1, 3 <
In the tested off-policy/oflline algorithms, it is necessary to evaluate log-probability for off- policy/offline actions stored in the buffer. For light-tailed g-Gaussian this can cause numerical issues since the evaluated actions may fall outside the support, incurring -oo for log probabbibbll.y.. To avoid this issue, we sample a batch of on on-policy actions from the g-Gaussian and replace the out-of-support actions with the nearest action in the Ta2 sense.
18

---

Published as a conference paper at ICLR 2025
In our experiments, all environments had bounded action space. Squashed Gaussian and light-tailed q q-Gaussian provide bounded output. However, Student's ' t, heavy tailed q g-Gaussian and Gaussian have unbounded support. For these distributions, clipped the sampled action to fit the action space of the task, without further modification on the density. The mean value is constrained using tanh function in distributions with unbounded support, except the standard Gaussian in offline learning.
D.2 ONLINE EXPERIMENTS
We used three classical control environments in the continuous action setting: Mountain Car (Sutton & Barto, 2018), Pendulum (Degris et al., 2012) and Acrobot (Sutton & Barto, 2018). All episodes are truncated at 1000 time steps. In Mountain Car, the action is the force applied to the car in [-1, 1, 1], and the agent receives a reward of -1 at every time step. In Pendulum, the action is the torque applied to the base of the pendulum in [-2, 2] and the reward is defined by r = -(0² +0.1 0 (=e)² where 9 denotes the angle, d0 is the derivative of time and the torque applied. Finally, in acrobot, the action is the torque applied on the joint between two links in [-1, 1] and the agent receives a reward of 1 per time step.
2 + 0.001 * a²)
When sweeping different hyperparameter configurations, we pause the training every 10,000 time steps and then evaluate the learned policy by averaging the total reward over episodes. However, when running the best hyperparameter configuration, we evaluate by freezing the policy every 1000 time steps and then computing the total reward obtained for episode.
Experiment settings:
Parameter sweeping: We sweep the hyperparameters with independent runs and then evaluate the run configuration for 30 seeds. We select the best hyperparameters based on the overall area under curve. When running the best hyperparameter configurations, we discard the original 5 seeds used for the hyperparameter sweep order to avoid the bias caused by hyperparameter selection. Details regarding the fixed and swept hyperparameters are provided in Table 4.
Agent learning: We used 2-layer network with 64 nodes on each layer and ReLU non-linearities. The batch size was 32. Agents used a target network for the critic, updated with polyak averaging with a = 0.01.
Hyperparameter
Value
Critic Learning rate
Swept in (1 x 10 10-2 1,,,111xxx 10 3 1 1 10 9)
Critic learning rate multiplier for actor
Swept in (0.1, 10}
Temperature
Swept in f0.01, 1}
Discount rate
0.99
Hidden size of Value network
64
Hidden layers of Value network
Hidden size of Policy network
64
Hidden layers of Policy network
2
Minibatch size
32
Adam.
0.9
Adam,
0.999
Number of seeds for sweeping
10
Number of sceds for the best setting
30
Table 4: Default hyperparameters and sweeping choices for online experiments.
D.3 OFFLINE EXPERIMENTS
We use the MuJoCo suite from D4RL (Apache-2/CC-BY licence) (Fu et al., 2020) for offline experiments. The D4RL offline datasets all contain I million samples generated by partially trained SAC agent. The name reflects the level of the trained agent used to collect the transitions. The Medium dataset contains samples generated by medium-level (trained halfway) SAC policy. Medium-expert mixes the trajectories from the Medium level and that produced by an expert agent. Medium-replay consists of samples in the replay buffer during training until the policy reaches the
19

---

Published as a conference paper at ICLR 2025
medium level of performance. In summary, the ranking of levels is Medium-expert Medium Medium-replay.
Experiment settings: We conducted the offline experiment using 9 datasets provided in D4RL: setiihhetaa-mmeiium--ee nexpium-expert, halfchectah-mediume halfchhettah-medimme medium-replay, hopper-medium- medium-exper walker2d-medium, and falkeerdd-medium-rellaa. We run agents: TAWAC, AWAC, IQL. InAC, and TD3BC. The results of TD3BC posted in the appendix. For each agent, tested distributions: Gaussian, Squashed Gaussian, Beta, Student's I, and Heavy-tailed g-Gaussian. As offfine learning algorithms usually require distribution covering the whole action space, Light-tailed q g-Gaussian is not considered in offline learning experiments. Each agent was trained for 1 x 10 steps. The policy was evaluated every 1000 steps. The score was averaged over rollouts in the real environment; each had 1000 steps.
halfcheetah
expert, hopper-mediumm hopper-medium-rrepe
Parameter sweeping: All results shown the paper were generated by the best parameter setting after sweeping. We list the parameter setting in Table 5. Learning rate and temperature in TAWAC + medium datasets were swept as the experiments their publication did not include the medium dataset. The best learning rates are reported in Table 6, and the temperatures are listed in Table 7.
Hyperparameter
Value
Swept in {3 10 x ,, x 10 10-4, x 10-4)
Learning rate
See the best setting in Table 6
Same the number reported
the publication of each algorithm.
Temperature
Except TAWAC + medium datasets, the value was
swept in {1.0, 0.5, 0.01}.
See the setting in Table 7
IQL Expectile
0.7
Discount rate
0.99
Hidden size of Value network
256
Hidden layers of Value network
2
Hidden size of Policy network
256
Hidden layers of Policy network
2
Minibatch size
A
0.9
Adam,B2
0.99
Number seeds for sweeping
Number of seeds for the best setting
10
Table 5: Default hyperparameters and sweeping choices for offline experiments.
We used 2-layer network with 256 nodes each layer. The batch size was 256. Agents used a target network for the critic, updated with polyak averaging with a = 0.005. The discount rate was set to 0.99.
Agent learning:
Sampling. To give an intuition for sampling time, we drew 10a samples from a randomly initialized actor two environments: Hal @Cheetah with 17-dim state and 6-dim action. The sparse q- Gaussian, heavy-tailed q- g-Gaussian and Gaussian respectively cost (107.12, 72.09, 27.94) seconds. We confirmed that the methods in Alg. were on the same magnitude to the Gaussian, but the sparse q- y-Gaussian cost more than the heavy-tailed due to more computation to produce low-variance samples. This further confirmed by Hopper with 1-- state, 3-dim action, where they costed (98.13, 65.17, 25.17) seconds.
E FURTHER RESULTS
Figure 11 shows the Manhattan plot of Soft-Aetor-Critic (SAC) with all swept hyperparameters on the online classic control environments. Student-t and Gaussian both seem to have similar behavior to hyperparameters. Although there is no definitive winner here, we can safely conclude that if we have problem where Gaussian works, Student-t is very likely work. Additionally, give the results
20

---

Published as a conference paper at ICLR 2025
Dataset
Distribution
TAWAC
AWAC
IQL
InAC
TD3BC
BetCCheetah-Mediunee Expert
Brddium-Expertt
Heavy-Taine Tailed q- Gaussian
0.001
0.001
0.001
0.001
0.0003
HalfCheetah- Bendiiim--xppet Expert
Squashed Gaussian
0.001
0.0003
0.0003
0.001
0.000S
MallCChettah-Modiie Medium-Expert
Gaussian
0.0003
0.0001
0.0003
0.0003
0.0003
HallChectah--Mesln Mediiu--xppertt Expertt
Beta
0.001
0.0003
0.001
0.001
0.001
MallCChcctab-Mddium Exxpert
Student's
0.001
0.0003
0.0003
0,0003
0.001
MallCChhctaa-Mddiu---eaaaa -Rpllay
Heavy-Tailed q-Gaussian
0.001
0.001
1.000
0.001
0.001
HallChectah-Meedimm RRpllay
Squashed Gaussian
0.001
0.0003
0,0000
100003
0.001
0.003
HallCheetah-Meddiu -eellay
Gaussian
0.001
0.0001
0,0000
0.001
0.001
MallChectab-MMdiumm Meaii----piiuuuaada----
Beta
0.001
0.0003
0,0003
0.001
0.001
HallChectab-MMdiimm Repllay
Student's
0.001
0.0003
0,0003
0.0003
0.003
HallChectah-Mccdiu
Heavy-Tailed g-Gaussian
0.001
0.001
0.001
0.001
0.0003
HallChectah-Meddiu
Squashed Gaussian
0.001
0.0003
0.001 0.001
0.001
0.0003
AllCChettah-Mediiue
Gaussian
0.0003
0.0001
0,0003
0.001
0.001
HalfCheetah-Meehe
Beta Student's 's
0.001
0.001
0.001
0.001
0.0003
HalfCheetah- Meddium
0.001
0.0003
0.001
0.001
0.001
Hopper-Medium Mediun--pett
Heavy- -ailled q- Gaussian
0.001
0.001
0.001
0.001
0.0001
Hopper-Medium- Exppe--edduun
Squashed Gaussian
0.001
0.001
0.001
0.001
0.0001
Hopper-Medium- Meppu---eeuu
Gaussian
0.0003
0.0003
0.001
0.001
0.0001
Hopper-Medium- Meppu--eediu
Beta Student's 's
0.001
0.001
0.001
0.003
0.0003
Hopper-Medium- Mediut-Expert
0.001
0.0003
0.001
0.003
0.0001
Hopper-Medium- Meplay
Heavy-Tailed Tailed q- Gaussian
0.001
0.0001
0.001
0. 0.0001
0,001
Hopper-Medium-- -edium-Reply
Squashed Gaussian
0.0001
0.0003
0.001
0 0003
0,001
Hopper-Medium-KEe Replay
Gaussian
0.0003
0.0003
0.001
0.0003
0,001
Hopper-Medium--Ke Replay
Beta Student's
0.0001
0.0003
0.0003
0.003
0,003
Hopper-MecdumnRRpplee
0.0003
0.0003
0.0003
0.0003
0,001
Hopper-Medium
Heavy-Tailed g-Gaussian
0.003
0.001
0.001
0.001
0.0001
Hopper-Mcdium
Squashed Gaussian
0.001
0.0003
0.001
0.0003
0.0001
Hopper-Medium
Gaussian
0.001
0.001
0000003
0.001
0.001
Hopper-Medium
Beta
0.001
0.001
0.003
0.001
0.001
Hopper-Medium
Student's
0.001
0.001
0.001
0.001
0.0001
Walker2ddMecdium-Eppeee
Heavy-Tailld g-ausssinn
0.0003
0.001
0.001
0.001
0.0003
Walker2d-Medium-Fii
Squashed Gaussian
0.001
0.001
0.0003
0.001
0.0003
Gaussian
0.0003
0.0001
0,00033
0.001
0.001
Milkee/dd-Medunn- Expert
Beta
0.001
0.0003
0.001
0.001
0.001
MaikerrddMddiunn Exppert
Student's
0.001
0.0003
0.0003
0.0003
0.0003
Walker2d- Walker2d-Medium-
-eill-T q- Gaussian
0.0003
0.0003
0.003
0.0003
0.001
Walker2d- Walker2d-Medium-
Squashed Gaussian
0.001
0.0003
0.0003
0.001
0.001
Replay
Gaussian
0.001
0.0003
0.0003
0.001
0.003
Walker2d-Medium Medium-Replay
Beta
0.001
0.0003
0.0003
0.001
0.0003
Walker2d-Medinn Replay
Student's
0.0003
0.0003
0.0003
0.001
0.001
Walker2d-Mes Medium
Hlaay-TTailed q- Gaussian
0.003
0.001
0.001
0.001
0.0001
halker2d-Medione
Squashed Gaussian
0.001
0.001
0.001
0.001
0.0001
Walker2d-Mediun
Gaussian
0.001
0.0001
0.001
0.001
0.0001
Walker2d-Medium
Beta
0.001
0.0003
0.003
0.001
0.0001
Walker2d-Meedum
Student's
0.001
0.0003
0.001
0.001
0.0001
Table 6: Best learning rates for offline experiments.
in the main text, Student's t more likely to perform better given the same hyperparameter sweeping range.
Our additional offline results include all algorithm-policy combination on all environments. We also include TD3BC (Fujimoto Gu, 2021) for comparison. Figure 12 shows the overall comparison with TD3. It is clear that Squashed Gaussian performs well and Beta can show slight improvements in some cases. Though it is visible that no much difference shown except on the Medium-Reple data. We conjecture that the better performance of Squashed Gaussian and Beta could be due to the TD3BC behavior cloning loss. It is encouraged that policy closely approximates the actions from the dataset. Therefore, policies like Beta that concentrate faster may be more advantageous.
Figures 13 to 15 display boxplots of the combinations on environments of each level. Consistent observations to that in the main text can be drawn from these plots, but with the exception that in Figure 14 the environment-wi wise best combination is TAWAC + Student's t. TD3BC does not exhibit strong sensitivity the choice of policy.
21

---

Published as a conference paper at ICLR 2025
Dea
Grddas
Hory filld Conuus
si Guscin
Stiders
-200
400
-000
Acrebot
-200
Roward
4000
600
-6000
-1000 2d
Figure 11: Manhattan plot of Soft-Actor- (SAC) with all swept hyperparameters the online classic control environments. The rewards on the y-axis are averaged over the final 10% of the total steps. Since different policy parameterizations have different numbers of runs in the sweep. oversampled the smaller sweeps with replacement. From the plot of Acrobot, we observe that Student- and Gaussian both respond similarly to changing hyper-parameters. Therefore, we hypothesize that if we have an environment where the Gaussian policy works, Student-t is also very likely to work. Additionally, from Figure (left), we know that student-t 75% more likely outperform the Gaussian given the same byperparameter sweeping range.
22

---

Published as a conference paper at ICLR 2025
Dataset
Distribution
TAWAC
AWAC
IQL
InAC
TD3BC
HalfCheetah- Mediaum-----i--------- Expeert
Heavy-Taike Tailed q- g-Gaussian
1.00
1.00
0.33
0.10
2.50
HalfCheetah- Mechhecch--MMduum---- Sedium-Expert
Squashed Gaussian
1.00
1.00
0.33
0.10
2.50
MetChhethh-Meiiune Exppert
Gaussian
1.00
1.00
0.33
0.10
2.50
Expert
Beta
1.00)
1.00
0.33
0.10
2.50
HallCheetah- Expert
Student's
1.00
1.00
0.33
2.50
HalfCheetah- Meeiaam--------------- Medium-Replay
Hleavy-Tailees q- g-Gaussian
0.01
1.00
0.33
0.50
2.50
BarCCheetah-Mediine Replay
Medium-Replay
Squashed Gaussian
0.01
1.00
0.33
0.50
2.50
Gaussian
0.01
1.00
0.33
0.50
2.50
HalfCheetah Baathheethh-MMiuun Replay
Beta
0.01
1.00
0.33
0.50
2.50
Satthrettth-Meiuuutt Replay
Student's
0.01
1.00
0.33
0.50
2.50
HalfCheetah- Medium
Heavy-Tailed q- Gaussian
0.01
0.50
0.33
0.33
2.50
HalfCheetah-Meeie Meddium
Squashed Gaussian
0.01
0.50
0.33
0.33
2.50
WWTCCheetah-Mdiinne
Gaussian
0.01
0.50
0.33
0.33
2.50
HalfCheetahh-Meiee Mediium
Beta
0.50
0.33
2.50
Seathheeta--MMdiinu
Student's 's
0.01
0.50
0.33
0.33
2.50
Hopper-Meclieg Mcdiim-Expertt
Hcavy-Tailed g-Gaussian
0.50
1.00
0.33
0.01
2.50
Mopper-Mcdium-- Expert
Squashed Gaussian
0.50
1.00
0.33
0.01
2.50
Hopper-Medium- Medium- Expert
Gaussian
0.50
1.00
0.33
0.01
2.50
Hopper-Mediune Exprett
Beta
0.50
1.00
0.33
0.01
2.50
Popper-Mediung Aediuut-Expert
Student's
0.50
1,00
0.33
0.01
2.50
Hopper- Medium-Mieiiee Replay
Heavy-Tailed g-Gaussian
0.50
0.50
0.33
0.50
2.50
Hopper-Meen Medium- Replly
Squashed Gaussian
0.50
0.50
0.33
0.50
2.50
Hopper-Medium Repllay
Gaussian
0.50
0.50
0.33
2.50
Hopper-Medium Medium- Replay
Beta
0.50
0.50
0.33
0.50
2.50
Peeper-Medium-ai Medium-Replay
Student's 's
0.50
0.50
0.33
0.50
2,50
Hopper- Medium
Medper-Medium
Tailed q- Gaussian
0.50
0.50
0.33
0.10
2.50
Squashed Gaussian
0.50
0.50
0.33
0.10
2.50
Hopper- Medium
Gaussian
0.50
0.50
0.33
0.10
2.50
Hopper-Medium Medium
Beta
0.50
0.50
0.33
0.10
2.50
Hopper- Medium
Student's
0.01
0.50
0.33
2.50
Hleavy-Tailed q- g-Gaussian
0.10
0.33
2.50
MaakkerrddMediuu-BBee Exppert
Squashed Gaussian
0.10
0.33
2.50
Walker2d-Medine -Meiium Expert
Gaussian
0.01
0.10
0.33
0.10
2.50
Walker2d- Miikerdd-Medium- Hxpper
Beta
0.01
0.10
0.33
0.10
2.50
Walker2d-Medium Expert
Student's
0.01
0.10
0.33
0.10
2.50
Walker2d- Meeiidd--ddiu---RRe Replay
Maille q- Gaussian
0.50
0.10
0.33
0.50
2.50
Walker2d Repllay
Squashed Gaussian
0.50
0.10
0.33
0.50
2.50
Balker2i-Medium Reppll
Gaussian
0.50
0.10
0.33
0.50
2.50
Walker2d Mereerdd-Meiiu Medium-RRppay
Beta
0.50
0.10
0.33
2.50
Student's is
0.50
0.10
0.33
2.50
Balker2dd-Medine
Heavy-Tailed q- D-Guussian
0.01
0.10
0.33
0.33
2,50
Malker2dd-Medinn
Squashed Caussian
1.00
0.10
0.33
0.33
2.50
Walker2d-Medium
Gaussian
1.00
0.10
0.33
0.33
2.50
Walker2d-Medium
Beta
1.00
0.10
0.33
0.33
2.50
Walker2d Meierrdd-Meiumm
Student's
0.01
0.10
0.33
0.33
2.50
Table 7: Temperature settings for offline experiments.
Table 8 examined the accumulated probabilities that fell on each It can be seen that the Student's is and the Gaussian tended to increasingly put more densities on the boundaries. This is in sheer contrast to the heavy-tailed q-G g-Gaussian that put the majority of probability density within the boundary. This may explain the better performance of TAWAC + heavy- q- Gaussiian.
Lastly. for all of the results shown above, their learning curves are shown in Figures 16 to 20, We smoothed the curves with window size 10 for better visualization.
23

---

Published as a conference paper at ICLR 2025
# Updates
100
200
300
400
Policy
Heavy-tailed y. G.Guussian
(24.39, 13.19)
(45.23, 2.36)
(45.49, 2.04)
(45.52, 1.98)
(45.54, 1.89)
Student's
(148.43, 71.23)
(198.89, 45.30)
(205.04, 37.04)
(207. 15. 32.96)
(207.00, 33.84)
Gaussian
(190.96, 65.89)
(206.92, 53.05)
(211.77, 39.08)
(213.57, 33.71)
(214.39. 31.26)
Table The summation of probability density accumulated on the left and the right edge in Figure 9 before clipping. Each pair indicates the left and right edge. The Student's t and the Gaussian increasing put more densities on the edges as compared to the heavy-tailed G-Guussian...
Gaussian
Beta
Student's
Heavy-Tailed q-Gaussian
Medium- Beedium--xpprttt Data
Medium Data
Medium-Rese Recillm--eppay Data
Proport on to Squashed Gaussian Performance
100%
80%
60%
40%
TAWAC AWAC
IQL
InAC TD3BC
TAWAC AWAC
IQL
InAC TD30C
TAWAC AWAC
IQL
InAC TD39C
Figure 12: Relative improvement to the Squashed Gaussian policy, averaged over environments.
Black vertical lines the top indicate one standard error. For TD3BC, Beta policy outperforms the Squashed Gaussian on Medium -exuert and Medium Medium-Replay.
Squashed
Student's
Heavy-Tailed Gausssiinn
Medium-Expert Data Hopper
HalfCheetah
Walker2d
100
Normalized Return
75
25
TAWAC
AWAC
IQL
InAC
TD3BC
TAWAC
AWAC
IOL
InAC
TD3BC
TAWAC
AWAC
IQL
InAC
TD3BC
Figure 13: Normalized scores on Medium- Expert level datasets. The black bar shows the median. Boxes and whiskers show 1 and 1.5x interquartile ranges, respectively. Fliers are plotted for uncluttered visualization. Environment-wise, InAC with heavy-tailed g-Gaussian is the top performer. Algorithm-win wise. heavy-tailed or/and Student's can improve or match the performance of the Squashed Gaussian except AWAC. With TD3BC no significant difference between policies is observed.
24

---

Published as a conference paper at ICLR 2025
Gaussian
Squashed Gaussian
Beta
Student's t
Heavy a Gaussian
Medium Data
HalfCheetah
Hopper
Walker2d
90
Normaliized Return
60
30
TAWAC
AWAC
IOL
InAC
TD3BC
TAWAC
AWAC
IQL
InAC
TD3BC
TAWAC
AWAC
IQL
InAC
TD3BC
Figure 14: Normalized scores on Medium level datasets. The black bar shows the median. Boxes and whiskers show 1x and 1.5x interquartile ranges, respectively. Fliers are not plotted for uncluttered visualization. Environment-win
InAC with heavy-tailed g-Gaussian is the top performer. Algorithm- wise, heavy-tailed g-Gaussian has observed significant performance drop with AWAC and InAC on Hopper and Walker2d. With TD3BC no significant difference between policies is observed.
Gaussian
Squashed Gaussian
Bela
Student's
Haave-Tailed q Gaussian
Medium-Replay Data
HalfCheetah
Hopper
Walker2d
100
Normalized Return
50
25
TAWAC
AWAC
IQL
InAC
TD3BC
TAWAC
AWAC
IOL
InAC
TD3BC
TAWAC
AWAC
IOL
InAC
TD3BC
Figure 15: Normalized scores on Medium-Repla level datasets. The black bar shows the median. Boxes and whiskers show 1 and 1.5x interquartile ranges, respectively. Fliers are not plotted for uncluttered visualization. Environment-wise, TAWAC + heavy-tailed G-ausssian is the best performer. Algorithm-wilt wise,, Student's 's stable and can match or improve on the performance of (Squashed) Gaussian.
25

---

Published as a conference paper at ICLR 2025
TAWAC Learning Curves
Gaussian
Squashed Geussian
Heavy-Talled g-Gaussian
Student's
Beta
HallCheetah-Meddu-EEEprrr
Hlopper-Medium-Expen
100
100
150
75
75
100
50
50
50
0
0
-25
-25
-50
25
50
100
25
50
100
25
50
100
Normalized Return
HalfCheetah-Medium
Hopper-Medium
Walker2d-Medium
60
80
100
40
60
75
40
50
20
20
25
0
-20
-20
25
25
50
100
25
50
100
25
50
100
HallCheetah-Meddum-Rppaay
Hopper-Medium-Replay
Walkerrd-MMddiumRRepla
60
100
100
40
75
75
50
50
20
25
25
0
0
0
20
25
25
25
50
100
25
50
100
25
50
100
Figure 16: TAWAC learning curves in all datasets. Columns show different environments and rows are levels the environments. x-axis denotes the number steps (x10*), and y-axis normalized score. Each curve was smoothed with window size 10.
AWAC Learning Curves
Gaussian
Squashed Gaussian
Heavy-Talled G-Gaussian
Student's
Beta
MaifCheetah-Mediunge Expert
Hopper-Medium-EExee
-iilir22--Mduuu-Expxeee
60
80
60
40
60
40
20
-20
-20
-20
0
25
50
75
100
0
25
50
75
100
0
25
50
75
100
Normalized Return
HalfCheetah-Medium
Hopper-Medium
Walker2d-Medium
60
80
100
40
60
75
50
20
20
-20
-25
25
50
75
100
0
25
75
100
0
25
50
100
HaalChheeah-MMeiiu--cco
Hopper-Medium-RRpple
Balkkerdd-Medium-eee
60
100
100
40
75
75
50
20
0
-20
-25
-25
25
50
75
100
0
25
75
100
0
25
50
100
Figure 17: AWAC learning curves in all datasets. Columns show different environments and rows are the levels of the environments. x-axis denotes the number of steps (x 10"), and y-axis is the normalized score. Each curve was smoothed with window size 10.
26

---

Published as a conference paper at ICLR 2025
IQL Learning Curves
Gaussian
Squashed Geussian
Heavy-Talled q-Gaussian
Student's
Beta
MaMlCheeta-MMdiim-EEpreee
Hopper-Medium-Exper
Walkerrd--Meiiu-EXpeee
100
80
150
75
60
100
50
40
50
0
0
-25
20
-50
25
50
75
100
25
50
75
100
25
50
75
100
Normalized Return
HalfCheetah-Medium
Hopper-Medium
Walker2d-Medium
60
80
100
40
60
20
0
0
0
-20
20
-25
25
50
75
100
25
75
100
25
50
75
100
HallChheetah-Mediu--Reeppp
Hopper-Medium-RRplla
Walker22-MeddlumRRRpaa
60
100
80
40
75
20
0
0
0
20
25
20
25
50
75
100
25
75
100
25
75
100
Figure 18: IQL learning curves in all datasets. Columns show different environments and rows are the levels of the environments. x-axis denotes the number of steps (x10*), and y-axis the normalized score. Each curve was smoothed with window size 10.
InAC Learning Curves
Gaussian
Squashed Gaussian
Heavy-Talled g-Gaussian
Student's
Beta
Hopper-Mediumy Meedium-Expert
Malker2d-Mediine Expertt
100
100
150
75
75
100
50
50
50
0-
0
-25
-25
-50
25
50
75
100
25
50
100
25
50
75
100
Normalized Return
HallCheetah-Meddimm
Hopper-Medium
Walker2d-MMdium
60
80
100
40
60
75
40
50
20
25
0
-20
-20
-25
25
50
100
25
50
100
25
50
100
Hopper-Medium-Repla.
Walker2d-Meeium-RReee
60
100
100
40
75
75
50
50
20
0-
-20
-25
-25
25
50
100
25
50
100
25
50
100
Figure 19: InAC learning curves in all datasets. Columns show different environments and rows are the levels of the environments. x-axis denotes the number of steps (x 10"), and y-axis is the normalized score. Each curve was smoothed with window size 10.
27

---

Published as a conference paper at ICLR 2025
TD3BC Learning Curves
Gaussian
Squashed Geussian
Heavy-Talled Gausssian
Student's
Beta
HallCheetah-Mediiu-EExerr
Hlgppe--Medium-Epee
Walker2d--Mediu--Exxp
80
80
150
60
60
100
50
-20
-20
-50
25
50
75
100
25
50
75
100
0
25
50
100
Return
HallCheetah-Medium
Hopper-Medium Medium
Walker2d-Mediun
60
60
100
40
40
75
50
20
20
20
-20
25
25
50
75
100
25
50
75
100
0
25
50
75
100
HallCheetah--Medlum-RRpaaaa
Hopper-Medium-RRella
Walkee2dd-Medllm-RRella
40
40
40
30
30
20
-20
-10
-10
0
25
50
75
100
0
25
50
75
100
0
25
50
100
Figure 20: TD3+BC learning curves in all datasets. Columns show different environments and rows are the levels of the environments. x-axis denotes the number of steps (x 10"), and y-axis is the normalized score. Each curve was smoothed with window size 10.
28


Reviews:
[{"review_id": "bb8OXlLNmb", "reviewer": "Reviewer_nnsC", "summary": "This paper focuses on solving reinforcement learning problems with continuous action spaces using policy optimization methods. In such cases, a typical solution for handling continuous action spaces is to impose a probability distribution (specified by finitely many parameters) on the continuous space. A common choice is the Gaussian distribution. The authors propose using a different family of probability distributions: the q-exponential family. Extensive numerical simulations are provided.", "questions": "The results are clear and I do not have questions.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The motivation for using the q-exponential family of distributions to encourage exploration seems intuitive. The numerical simulations are quite extensive and successfully demonstrate that it is not always optimal to use a Gaussian distribution to parameterize the policy in reinforcement learning.", "weaknesses": "I suspect that there is an exploration vs. exploitation trade-off in using heavy-tailed probability distributions. Is it possible to design a synthetic example to verify this? Additionally, once this is verified, there might be a way to switch probability distributions during training to balance the exploration-exploitation trade-off, which seems like an interesting direction to investigate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sy2IkZsTpB", "reviewer": "Reviewer_dEFf", "summary": "This paper conducts an empirical investigation of the use of the q-exponential family of policy parameterizations as an alternative to the commonly used Gaussian for continuous state action spaces in RL for policy optimization. In particular, these policies have been incorporated into existing Actor-Critic algorithms and experiments have been performed to compare the different policies to showcase the improved performance of some of them compared to the Gaussian policy in online and offline settings across several environments.", "questions": "**Main questions (from the most to the least important):**\n\n- Do all the policy neural networks have a similar number of parameters/weights across experiments for fairness of comparison? \n- Are the implementations mostly readily available in existing RL packages or other libraries that could be directly be used? I see that you provide some of the sampling mechanisms but these seem to be already well known in the statistics literature given their popularity. \n- Is GBMM only valid for 1 < q < 3? \n- l. 41: ‘Heavy-tailed distributions could be more preferable as they are more robust’, what do you mean by robustness here? Can you elaborate more? \n- Can you clarify the difference between the different Data settings in Fig. 8 (Medium-Expert, Medium, Medium-Replay)? \n\n**Additional questions:**\n \n- Any comments about the so-called Lévy alpha-stable distribution? It seems it has been explored in the literature to model the SGD noise. \nSee e.g. Simsekli et al. 2019. A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks. ICML 2019. \n- SAC encourages exploration. Would it be interesting to see if just changing the policy parameterization with a heavy tailed policy would already perform better than SAC? SAC seems to combine two effects: the parameterization of the policy itself and and the regularization to force the policy to be closer to a BG policy (why?)? \n- More of a possible future work direction comment: This work explores in isolation each one of the parameterized policies to showcase the advantages and shortcomings of each one of them and compare them. Can we combine them to get the best of each one of them and have more flexibility like in ensemble methods or bagging approaches in Machine Learning? For instance we might need a very exploratory policy in some settings/regions of the action space or much less once we progress in policy optimization beyond the ability of single distributions to concentrate throughout learning.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- To the best of my knowledge, there is no such study in the literature and I believe it is useful for RL in practice. Proposing more alternatives to the Gaussian policy for continuous state action space settings is interesting.  \nIt would be nice to have the proposed parameterizations and their implementations integrated to existing RL packages to make them readily usable by practitioners. \n\n- The execution of the study is solid: experiments are methodologically and rigorously conducted, figures are professional and the presentation is pleasant, this effort is much appreciated for a practical paper.", "weaknesses": "**Main Comments:**\n\n- While the paper provides some insights, in terms of conclusions, it is hard to make a claim to extrapolate the findings beyond the interesting but expected fact that Gaussian policy might not always be the best choice and some other parameterizations might perform better. The paper does perform several experiments but  it is hard to derive actionable practical advice from 4 algorithms x 3 environments regarding which parameterization to be used in general. \nI think the paper could do better to showcase the advantages and shortcomings of each one of the policies, especially to demonstrate the intuitive points mentioned in the introduction regarding each one of them. While the environments tested are standard benchmarks in RL, I think it would be interesting to design and/or consider simple environments with different levels of exploration needed for instance to show the differences between the policies. From the viewpoint of the practitioner, can we come up with a procedure to decide which one to use rather than myopically testing each one of them an decide accordingly? Can we design an index performance of exploration hardness to be exploited? Little can be said regarding this from the paper. Of course, we cannot reasonably be confident to choose the right parameterization since this would highly depend on the setting at hand as the paper shows but even heuristics would be useful. In other (older) ML areas, such procedures seem to be widely used in practice (e.g. for model selection in classification …) \n\n- Other policy parameterizations than the Gaussian have been investigated in the literature as the paper acknowledges. Some additional relevant related work discussing the use of heavy tailed policies in RL and their benefits: \n\nAmrit Singh Bedi and Anjaly Parayil and Junyu Zhang and Mengdi Wang and Alec Koppel. ‘On the Sample Complexity and Metastability of Heavy-tailed Policy Search in Continuous Control’, JMLR 2024. \n\nS. Chakraborty et al., ‘Dealing with Sparse Rewards in Continuous Control Robotics via Heavy-Tailed Policy Optimization’, 2023 IEEE International Conference on Robotics and Automation. \n\n- The paper could comment about the overhead computational effort (if any) required when going for more ‘sophisticated’ policy parameterizations than the simple Gaussian, notably in terms of sampling mechanism complexity and number of samples requirements, for instance depending on the dimensionality of the problem. Any differences regarding efficiency depending on the dimensionality of the problem? \n\n- See questions below for clarifications. \n\n**Minor comments:** \n- l. 122-123: ‘In continuous action spaces, evaluating the log-partition function is generally intractable. Therefore, many researchers consider the Gaussian policy instead.’ I find this sentence a bit confusing. Is Gaussian policy only considered because BG policy cannot be used? In principle, one could consider any policy parameterization. For the continuous setting, it turns out that the Gaussian policy is the one of the simplest one can consider perhaps due to its omnipresence in statistics and parametric estimation as well as its widely available sampling procedure implementations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gwSEDvDxub", "reviewer": "Reviewer_Thwb", "summary": "In continuous action RL many researches parameterize the policy as a mapping from states to std and mean of a Gaussian from which the actions are sampled.\nWhen also the standard deviation of the Gaussian is learned, the authors argue that phenomenon like instabilities and lack of exploration can occurs. To this end, this paper explores empirically other continuous distribution to use to parameterize policies. In particular, it seems that heavier tails distributions are beneficial in Continuous Online Control experiments such as Acrobot and Cartpole and in Offline MuJoCo experiments.", "questions": "Is it possible to understand which characteristics of the environment makes ine distribution better than the other ?\n\n\nIt would be very useful for the user to get some prior information about which distributions could work well to avoid to search over all the distributions proposed in your work and finding the best performing one.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "I think that it is useful to have a paper that clearly states that the Gaussian parameterization might not be the best performing in practice.", "weaknesses": "The paper does not have a clear explanation or conclusive experiment about which parameterization should be used in which cases.\nThe main take away is that Gaussian parameterization should be avoided but it is not clear with which other distribution it should be replaced.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)