$q$-exponential family for policy optimization

## Abstract

Abstract Policy optimization methods benefit from a simple and tractable policy parametrization, usually the Gaussian for continuous action spaces. In this paper, we consider a broader policy family that remains tractable: the q 𝑞 q italic_q -exponential family.
This family of policies is flexible, allowing the specification of both heavy-tailed policies ( q > 1 𝑞 1 q>1 italic_q > 1 ) and light-tailed policies ( q < 1 𝑞 1 q<1 italic_q < 1 ). This paper examines the interplay between q 𝑞 q italic_q -exponential policies for several actor-critic algorithms conducted on both online and offline problems. We find that heavy-tailed policies are more effective in general and can consistently improve on Gaussian.
In particular, we find the Student’s t-distribution to be more stable than the Gaussian across settings and that a heavy-tailed q 𝑞 q italic_q -Gaussian for Tsallis Advantage Weighted Actor-Critic consistently performs well in offline benchmark problems.

## 1 Introduction

Policy optimization methods optimize the parameters of a stochastic policy towards maximizing some performance measure (Sutton et al., ).
These methods benefit from a simple and tractable policy functional.
For discrete action spaces, the Boltzmann-Gibbs (BG) policy is often preferred (Mei et al., ; Cen et al., ); while the Gaussian policy is standard for the continuous case.
For continuous action spaces, sampling the BG policy is computationally expensive due to the normalizing log-partition function.
A Gaussian policy is often used as a tractable approximation.
While there are other candidates such as the Beta policy (Chou et al., ), the Gaussian remains the most common choice for both online and offline policy optimization methods (Haarnoja et al., ; Neumann et al., ; Xiao et al., ).

Figure: Figure 1: The policy parametrizations considered in this paper.
Refer to caption: extracted/5897743/img/compare_policies_new.png

In this paper, we consider a broader policy family that remains tractable called the $q$-exponential family.
The $q$-exponential family was proposed to study non-extensive system behaviors in the statistical physics (Naudts, ; Matsuzoe & Ohara, ), and has recently been exploited in the transformers (Peters et al., ; Martins et al., ).
By setting $q=1$, it recovers the standard exponential family.
With $q>1$, we can obtain policies with heavier tails than the Gaussian, such as the Student’s t-distribution.
Heavy-tailed distributions could be more preferable as they are more robust (Lange et al., ), can facilitate exploration and help escape local optima (Kobayashi, ).
When $q<1$, light-tailed (sparse) policies such as the $q$-Gaussian distribution can be recovered.
The sparse $q$-Gaussian has finite support and can serve as a continuous generalization of the discrete sparsemax.
As a result, $q$-Gaussian may help alleviate the safety concerns incurred by the infinite support Gaussian (Xu et al., ; Li et al., ).

Such $q$-exponential families have been considered in reinforcement learning, with the existing work summarized in Table 1. Lee et al. (); Chow et al. () studied the discrete setting with $q=0$, called the sparsemax. Li et al. () similarly considered $q=0$ policy parameterization for the continuous action setting. All other works, however, used a Gaussian policy parameterization to (implicitly) approximate an idealized target distribution that is $q$-Gaussian, and specifically for $q<1$ Lee et al. (); Zhu et al. (). Such a choice is suboptimal, as Gaussians are used to approximate light-tailed (sparse) target policies. And in fact that choice was not strictly necessary as the policy parameterization need not have been chosen to be Gaussian: it could also have been a $q$-Gaussian. The gap was in recognizing that we could use the general continuous $q$-exponential family for the policy parameterization.

Figure: Figure 2: Performance relative to the Squashed Gaussian on the offline D4RL MuJoCo task, averaged across the selected algorithms and environments.
Refer to caption: x1.png

In this paper, we empirically investigate the $q$-exponential family as a replacement for the Gaussian inside several existing policy optimization algorithms. Our contributions include the following.
(1) We show how to use $q$-exponential family policy parameterizations inside a variety of existing actor-critic algorithms.
(2) We provide comprehensive experiments on both online and offline problems showing that $q$-exponential family policies can improve on the Gaussian by a large margin.
In particular, we find that the Student’s t policy is more stable, performing well across algorithms and problems, shown in Figure 2.
(3) We provide empirical evidence supporting the assumption that algorithms may prefer specific policies depending on the actor loss objective. In particular, we find that by replacing the Gaussian with a heavy-tailed $q$-Gaussian, Tsallis Advantage Weighted Actor-Critic (Zhu et al., ) consistently performs better across offline benchmark problems. This outcome makes sense; as mentioned above, this algorithm implicitly has a target policy that is a $q$-Gaussian, so using a matching $q$-Gaussian parameterization should perform better.

**Table 1: Existing works and their scopes. We are the first to consider the general $q$-exponential family for the parameterized policy in reinforcement learning. The family includes continuous heavy-tailed and sparse policies. Prior works in RL considered only the discrete case or continuous policy with a specific entropic index $q$. Further, in many cases they still used a Gaussian policy parameterization to approximate an implicit target distribution that is a $q$-exponential, rather than explicitly using the $q$-Gaussian as the policy parameterization.**
| Reference | Scope of $\exp_{q}$ | Explicit? | Heavy & Sparse? | Continuous? | RL? |
| --- | --- | --- | --- | --- | --- |
| Naudts (); Matsuzoe & Ohara () | $q\in\mathbb{R}$ | - | ✓ | ✓ | ✗ |
| Martins et al. () | $q<1$ | - | ✗ | ✓ | ✗ |
| Lee et al. (); Chow et al. () | $q=0$ | ✓ | ✗ | ✗ | ✓ |
| Lee et al. (); Zhu et al. (; ) | $q<1$ | ✗ | ✗ | ✓ | ✓ |
| Li et al. () | $q=0$ | ✓ | ✗ | ✓ | ✓ |
| This paper | $q\in\mathbb{R}$ | ✓ | ✓ | ✓ | ✓ |

## 2 Background

We focus on discounted Markov Decision Processes (MDPs) expressed by the tuple $(\mathcal{S},\mathcal{A},P,\mu,r,\gamma)$, where $\mathcal{S}$ and $\mathcal{A}$ denote state space and finite action space, respectively.
Let $\Delta(\mathcal{X})$ denote the set of probability distributions over $\mathcal{X}$.
$P$ and $\mu$ denote the transition probability and initial state distribution, respectively.
$r(s,a)$ defines the reward associated with that transition.
$\gamma\in(0,1)$ is the discount factor.
A policy $\pi:\mathcal{S}\rightarrow\Delta(\mathcal{A})$ is a mapping from the state space to distributions over actions.
To assess the quality of a policy, we define the expected return as $J(\pi)=\int_{\mathcal{S}}\rho^{\pi}(s)\int_{\mathcal{A}}\pi(a|s)r(s,a)\mathrm{
d}a\mathrm{d}s$,
where $\rho^{\pi}_{s}=\sum_{t=0}^{\infty}\gamma^{t}P(s_{t}=s)$ is the unnormalized state visitation frequency.
The goal is to learn a policy that maximizes $J(\pi)$.
We also define the action value and state value as $Q^{\pi}(s,a)=\mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t})
|s_{0}\sim\mu,a_{0}=a\right]$, $V^{\pi}(s)=\mathbb{E}_{\pi}\left[Q^{\pi}(s,a)\right]$.

In practice, the policy is often parametrized by a vector of parameters $\theta\in\mathbb{R}^{n}$.
The policy can then be optimized by adjusting its parameters to the high reward region utilizing its gradient information.
The Policy Gradient Theorem (Sutton et al., ) featured by many policy gradient methods states that the gradient can be computed by:

$$ $\nabla_{\theta}J(\pi;\theta)=\mathbb{E}_{s\sim\rho^{\pi},\,a\sim\pi_{\theta}} \left[Q^{\pi}_{s}(a)\nabla_{\theta}\ln\pi_{s}(a;\theta)\right].$ $$

In practice, the expectation is approximated by sampling.
When the state space is large, the action value function is also parametrized, leading to the Actor-Critic methods (Degris et al., ).

In contrast to the study of policy gradient algorithms, the impact of specific policy parametrizations on performance remains a less studied topic.
Researchers typically consider policy parametrizations that can be written as the following:

$$ $\displaystyle\pi_{s}(a;\theta)=\frac{1}{Z_{s}}\exp\big{(}\theta^{\top}\phi_{s} (a)\big{)}=\exp\big{(}\theta^{\top}\phi_{s}(a)-Z_{s}^{\prime}\big{)}.$ (1) $$

Here, $\phi_{s}(a)$ is a vector of statistics and $\theta\in\mathbb{R}^{n}$ is a vector of parameters, $Z_{s}$ is the normalizing constant ensuring the policy is a valid distribution and $Z_{s}:=\exp\left(Z_{s}^{\prime}\right)$.
One immediate instance is the Boltzmann-Gibbs (BG) policy $\pi_{\text{BG},s}(a;\theta)=\exp\big{(}Q_{s}(a)-Z_{s}\big{)}$, where $Z^{\prime}_{s}=\ln\int\exp\big{(}Q_{s}(a)\big{)}\mathrm{d}a$ is the log-partition function.
In the discrete case, it is also called the *softmax transformation* (Cover & Thomas, ).
BG policy has been studied extensively in RL for encouraging exploration and smoothing the optimization landscape, to name a few applications (Haarnoja et al., ; Ahmed et al., ; Cen et al., ).
In continuous action spaces, evaluating the log-partition function is generally intractable.
Therefore, many researchers consider the Gaussian policy instead.

## 3 Exponential and q 𝑞 q italic_q -Exponential Families

We first review the commonly used policy parametrizations.
They permit an expression using the exponential function.
We arrive at the more general $q$-exponential family by deforming the exponential.
In Table 2, we summarize all policies presented in the paper.

### 3.1 The Exponential Family Policies

Researchers usually default to using the Gaussian policy
$\pi_{s}(a)=\frac{1}{\sqrt{2\pi}\sigma(s)}\exp\left(\frac{-\left(a-\mu(s)\right
)^{2}}{2\sigma(s)^{2}}\right)$
for continuous action spaces.
For simplicity we drop the dependence on state $s$.
To see it is a member of the exponential family, in Eq. (1) let $\theta=[\frac{\mu}{\sigma^{2}},-\frac{1}{2\sigma^{2}}]^{\top}$ for $\mu\!\in\!(-\infty,\infty),\sigma>0$; $\phi_{s}(a)=[a,a^{2}]^{\top}$, and $Z_{s}=\ln\big{(}\sqrt{2\pi}\sigma\big{)}$.
This amounts to setting $Q_{s}(a)=-\frac{(a-\mu)^{2}}{2\sigma^{2}}$ in the BG (Gu et al., ).
We write a Gaussian policy as $\pi_{\mathcal{N},s}(a)=\mathcal{N}(a;\mu,\sigma^{2})$.
The gradients of the Gaussian are $\nabla_{\mu}\ln\pi_{s}(a)=\frac{(a-\mu)}{\sigma^{2}}$ and $\nabla_{\sigma}\ln\pi_{s}(a)=\frac{(a-\mu)^{2}}{\sigma^{3}}-\frac{1}{\sigma}$.
On one hand, the Gaussian policy is simple to implement.
On the other hand, when $\sigma$ becomes small, Gaussian can be unstable due to overly large gradients and
can prematurely concentrate on a suboptimal action.
As a result, it is susceptible to noise/outliers and does not encourage sufficient exploration due to its thin tails.
This paper investigates location-scale alternatives within the generalized $q$-exponential family.

Another interesting member is the Beta distribution (Chou et al., ): $\pi_{\text{Beta},s}(a)=\frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)
}a^{\alpha-1}(1-a)^{\beta-1},\,\,a\in(0,1)$, where $\Gamma(\cdot)$ is the gamma function.
It can be retrieved from equation 1 by letting $\theta=[\alpha,\beta]^{\top},\phi_{s}(a)=[\ln a,\ln(1-a)]^{\top}$, $Z_{s}=\frac{\Gamma(\alpha)\Gamma(\beta)}{\Gamma(\alpha+\beta)}$.
Since Beta distribution’s support is bounded between $(0,1)$, Chou et al. () argued that it might alleviate the bias introduced by truncating Gaussian densities outside the action space bounds.
The beta policy is the only non-location-scale family distribution in this paper.
However, as we will show in the experiments, the Beta policy generally does not perform favourably against the Gaussian.

**Table 2: Policy parametrizations from the exp and $q$-exp families studied in this paper. We are primarily interested in the location-scale family. Their multivariate forms are shown in Appendix A.**
| Family | Policy | Parameters $\theta$ | Statistics $\phi_{s}(a)$ | Normalization $Z_{s}$ | $\nabla\ln\pi_{s}(a)$ |
| --- | --- | --- | --- | --- | --- |
| $\exp$ | Gaussian | $[\frac{\mu}{\sigma^{2}},-\frac{1}{2\sigma^{2}}]$ | $[a,a^{2}]$ | ${\sqrt{2\pi}\sigma}$ | Eq. (13) |
| Beta | $[\alpha,\beta]$ | $[\ln a,\ln(1-a)]$ | $\frac{\Gamma(\alpha)\Gamma(\beta)}{\Gamma(\alpha+\beta)}$ | - |  |
| $q$-$\exp$ | Student’s t | $\left[\frac{-2\mu}{\nu\sigma},\frac{1}{\nu\sigma}\right]$ | $[a,a^{2}]$ | $\frac{\sqrt{\pi\nu\sigma}\,\Gamma\left(\frac{\nu}{2}\right)}{\Gamma\left(\frac<br>{\nu+1}{2}\right)}$ | Eq. (14) |
| $q$-Gaussian $(q<1)$ | $[\frac{\mu}{\sigma^{2}},-\frac{1}{2\sigma^{2}}]$ | $[a,a^{2}]$ | $\sqrt{\frac{\pi}{1-q}}\frac{\Gamma\left(\frac{1}{1-q}+1\right)}{\Gamma\left(<br>\frac{1}{1-q}+\frac{3}{2}\right)}$ | Eq. (15) |  |
| $q$-Gaussian $(1<q<3)$ | $\sqrt{\frac{\pi}{q-1}}\frac{\Gamma\left(\frac{1}{q-1}-\frac{1}{2}\right)}{<br>\Gamma\left(\frac{1}{q-1}\right)}$ |  |  |  |  |

Figure: Figure 3: $\exp_{q}x$ and $\ln_{q}x$ for $q<1$ and $q>1$. When $q=1$ they respectively recover their standard counterpart. For $q<1$ the $q$-exp can return zero values and hence $q$-exp policies may achieve sparsity. For $q>1$, $q$-exp decays more slowly towards 0, resulting in heavy-tailed behaviors. The rightmost shows the $q$-Gaussian with different $q$.
Refer to caption: extracted/5897743/img/q_exp_illus.png

### 3.2 The q 𝑞 q italic_q -Exponential Family, Heavy-tailed and Light-tailed Policies

Generalizing the exponential family using the $q$-exponential function has been extensively discussed in statistical physics (Naudts, ; Tsallis, ; Naudts, ; Amari & Ohara, ).
In the machine learning literature, the $q$-exponential generalization has attracted some attention since it allows for tuning the tail behavior by adjusting the value of $q$ (Sears, ; Ding & Vishwanathan, ; Amid et al., ).
The $q$-exponential and its unique inverse function $q$-logarithm are:

$$ $\displaystyle\exp_{q}\!\,x:=\begin{cases}\exp x,&q=1\\ \left[1+(1-q)x\right]^{\frac{1}{1-q}}_{+},&q\neq 1\end{cases},\quad\quad\ln_{q }\!x:=\begin{cases}\ln x,&q=1\\ \frac{x^{1-q}-1}{1-q},&q\neq 1\end{cases}$ (2) $$

where $[\cdot]_{+}:=\max\{\cdot,0\}$.
$q$-exp/log generalize exp/log since
$\lim_{q\rightarrow 1}\exp_{q}x=\exp x$ and $\lim_{q\rightarrow 1}\ln_{q}x=\ln x$.
Similar to exp, $q$-exp is an increasing and convex function for $q>0$, satisfying $\exp_{q}(0)=1$.
However, an important difference of $q$-exp is that $\exp_{q}(a+b)\neq\exp_{q}(a)\exp_{q}(b)$ unless $q=1$.
We visualize $q$-exp/log in Figure 3.

We now define the $q$-exponential family as:

$$ $\displaystyle\pi_{q,s}(a;\theta)=\frac{1}{Z_{q,s}}\exp_{q}\!\big{(}\theta^{ \top}\phi_{s}(a)\big{)}=\exp_{q}\!\big{(}\theta^{\top}\phi_{s}(a)-Z^{\prime}_{ q,s}\big{)},$ (3) $$

where $\theta,\phi_{s}(a),Z_{q,s}$ have similar meanings to equation 1.
Note that $Z_{q,s}\neq\exp_{q}\left(Z^{\prime}_{q,s}\right)$ unless $q=1$.
The $q$-exponential family includes the $q$-Gaussian and Student’s t distributions described in the next subsections.

#### 3.2.1 q 𝑞 q italic_q -Gaussian

As the counterpart of Gaussian in the $q$-exp family,
$q$-Gaussian unifies both light-tailed and heavy-tailed policies by varying the entropic index $q$ (Matsuzoe & Ohara, ):

$$ $\displaystyle\begin{split}&\pi_{\mathcal{N}_{q},s}(a)=\frac{1}{Z_{q,s}}\exp_{q }\!\left(-\frac{(a-\mu)^{2}}{2\sigma^{2}}\right),\\ \text{where }Z_{q,s}=&\begin{cases}\sigma\sqrt{\frac{\pi}{1-q}}\,\,{\Gamma\! \left(\frac{1}{1-q}+1\right)}/\,\,{\Gamma\!\left(\frac{1}{1-q}+\frac{3}{2} \right)}&\text{if }-\infty<q<1,\\ \sigma\sqrt{\frac{\pi}{q-1}}\,\,{\Gamma\!\left(\frac{1}{q-1}-\frac{1}{2}\right )}/\,\,{\Gamma\!\left(\frac{1}{q-1}\right)}&\text{if }1<q<3.\end{cases}\end{split}$ (4) $$

It is heavy-tailed when $1<q<3$ and light-tailed when $q<1$.
$\pi_{\mathcal{N}_{q},s}(a)$ is no longer integrable for $q\geq 3$ (Naudts, ).
We visualize these $q$-Gaussians in Figure 3.

Since popular libraries like the PyTorch (Paszke et al., ) do not have implementations of $q$-Gaussians available, we discuss their sampling methods.
It was shown by (Martins et al., ) that a sparse $q$-Gaussian ($q<1$) random variable permits a stochastic representation $\bm{\mu}+rA\bm{u}$, where $\bm{u}\sim\texttt{Unif}\left(\mathbb{S}^{N}\right)$ is a random sample from the $N\!-\!1$ dimensional unit sphere.
$A$ is the scaled matrix $|\Sigma|^{-\frac{1}{2N+\frac{4}{1-q}}}\Sigma^{\frac{1}{2}}$.
$r$ is the radius of the distribution, and the ratio follows the Beta distribution ${r^{2}}/{R^{2}}\,\sim\,\texttt{Beta}\left((2-q)/(1-q),{N}/{2}\right),$
where $R$ is radius of the supporting sphere of the standard $q$-Gaussian $\mathcal{N}_{q}(0,I)$:

$$ $\displaystyle R=\left(\frac{\Gamma\left(\frac{N}{2}+\frac{2-q}{1-q}\right)}{{ \Gamma\left(\frac{2-q}{1-q}\right)\pi^{\frac{N}{2}}}}\,\cdot\,\left(\frac{2}{1 -q}\right)^{\frac{1}{1-q}}\right)^{\frac{1-q}{2+(1-q)N}}.$ (5) $$

Notice that $R$ depends only on the dimensionality $N$ and the entropic index $q$.
This method provides low-variance samples, but unfortunately it does not extend to $q>1$.

Figure: Algorithm 1 $q$-Gaussian sampling

Therefore, for $1<q<3$ we adopt the Generalized Box-Müller Method (GBMM) (Thistleton et al., )
to transform uniform random variables $\bm{u}_{1},\bm{u}_{2}\sim\texttt{Unif}(0,1)^{N}$ by the following:

$$ $\displaystyle\bm{z}_{1}=\sqrt{-2\ln_{q}\left(\bm{u}_{1}\right)}\cdot\cos\left( 2\pi\bm{u}_{2}\right),\qquad\bm{z}_{2}=\sqrt{-2\ln_{q}\left(\bm{u}_{1}\right)} \cdot\sin\left(2\pi\bm{u}_{2}\right).$ (6) $$

Then each of $\bm{z}_{1},\bm{z}_{2}$ is a standard $q$-Gaussian variable with new entropic index $q^{\prime}=(3q-1)/(q+1)$.
Often we know the desired $q^{\prime}$ in advance, in this case we simply let the $q$-log take on the index $q=(q^{\prime}-1)/(3-q^{\prime})$.
The desired random vector is given by
$\bm{\mu}+\Sigma^{\frac{1}{2}}\bm{z}$.

#### 3.2.2 Student’s t

Heavy-tailed distributions like the Student’s t are popular for robust modelling (Lange et al., ). The Student’s t distribution is

$$ $\displaystyle\pi_{\text{St},s}(a)=\frac{\Gamma\left(\frac{\nu+1}{2}\right)}{ \sqrt{\pi\nu\sigma}\,\Gamma\left(\frac{\nu}{2}\right)}\left(1+\frac{(a-\mu)^{2 }}{\sigma\nu}\right)^{-\frac{\nu+1}{2}},$ (7) $$

where $\nu>0$ is the degree of freedom.
As $\nu\rightarrow\infty$, Student’s t distribution approaches the Gaussian.
Numerically, Student’s t with $\nu\geq 30$ is considered to closely match the Gaussian.
Therefore, $\nu$ can be an important learnable parameter in addition to its location $\mu$ and scale $\sigma$.
It allows the policy to adaptively balance the exploration-exploitation trade-off by interpolating the Gaussian and heavy-tailed policies.
Now let $q=1+\frac{2}{\nu+1}$ and define

$$ $\displaystyle\begin{split}Z_{q,s}:=&\frac{\sqrt{\pi\nu\sigma}\,\Gamma\left( \frac{\nu}{2}\right)}{\Gamma\left(\frac{\nu+1}{2}\right)},\quad\theta^{\top} \phi_{s}(a):=\frac{Z_{q,s}^{q-1}}{(1-q)}\frac{(a-\mu)^{2}}{\sigma\nu},\\ \Rightarrow\qquad&\pi_{\text{St, s}}(a)=\exp_{q}\!\left(\theta^{\top}\phi_{s}( a)-\ln_{2-q}{Z_{q,s}}\right),\end{split}$ (8) $$

which we see it is indeed a $q$-exp policy and $Z^{\prime}_{q,s}=\ln_{2-q}Z_{q,s}$.
Student’s t policy has been used in (Kobayashi, ) to encourage exploration and to escape local optima.
Another related case is the Cauchy’s distribution recovered when $q=2$ (or $\nu=1$ from Student’s t).
Cauchy’s distribution can be used as the starting point for learning Student’s t.
Note that Cauchy’s distribution does not have valid mean, variance or any higher moments.

## 4 Using q 𝑞 q italic_q -exponential families for actor-critic algorithms

In this section, we outline three key actor-critic algorithms we use in our study and the nuances of incorporating $q$-exp policies into them.
For example, the $q$-exp policies may not have closed-form Shannon entropy.
Therefore, approximations are needed for algorithms like SAC and GreedyAC.
Moreover, though for the Gaussian evaluating the log-likelihood for off-policy/offline actions causes no problem, it raises a new issue for the light-tailed $q$-Gaussian, since these actions can fall outside of its support.

Soft Actor-Critic. SAC (Haarnoja et al., ) encourages exploration by adding to reward the Shannon entropy.
The actor minimizes the following KL loss

$$ $\displaystyle\mathcal{L}_{\text{SAC}}(\phi):=\mathbb{E}_{s\sim\mathcal{B}} \left[D_{\!K\!L}\!\left(\pi_{\phi,s}\left|\vphantom{{\pi}_{\text{BG},s}}\! \right|{\pi}_{\text{BG},s}\right)\right]=\mathbb{E}_{s\sim\mathcal{B}}\left[D_ {\!K\!L}\!\left(\pi_{\phi,s}\left|\vphantom{\frac{\exp\left(\tau^{-1}Q_{s} \right)}{Z_{s}}}\!\right|\frac{\exp\left(\tau^{-1}Q_{s}\right)}{Z_{s}}\right) \right],$ $$

where states are sampled from replay buffer $\mathcal{B}$.
The parametrized policy $\pi_{\phi}$ is projected to be close to the BG policy.
By default $\pi_{\phi}$ is chosen to be the Gaussian policy, but potentially a more exploring policy like the Student’s t could be better.
Depending on action values, BG can have multiple modes and heavy tails.
The Gaussian may not be able to fully capture these characteristics.

Greedy Actor-Critic.
GreedyAC (Neumann et al., ) maintains an additional proposal policy for exploration by maximizing Shannon entropy augmented rewards.
Its actor policy maximizes unbiased reward and learns from the high-quality actions generated by the proposal policy.
To simplify notations, we use $I(s)$ to denote the set of high quality actions given $s$.

$$ $\displaystyle\mathcal{L}_{\text{GreedyAC, prop}}(\phi):=\mathbb{E}_{\begin{ subarray}{c}s\sim\mathcal{B}\\ a\in I(s)\end{subarray}}\left[-\ln\pi_{\phi,s}-\mathcal{H}\left(\pi_{\phi,s} \right)\right],$ $\displaystyle\mathcal{L}_{\text{GreedyAC, actor}}(\bar{\phi}):=\mathbb{E}_{ \begin{subarray}{c}s\sim\mathcal{B}\\ a\in I(s)\end{subarray}}\left[-\ln\pi_{\bar{\phi},s}\right].$ $$

GreedyAC maximizes log-likelihood of the actor and entropy-augmented likelihood for the proposal policy.
Note that the when $\pi_{\phi,s}$ is a $q$-exp policy, it may not have a closed-form Shannon entropy expression.
Therefore, we need to approximate it with an empirical expectation of log-probabilities.

Tsallis Advantage Weighted Actor-Critic.
TAWAC (Zhu et al., ) proposed to use a light-tailed $q$-exp policy for offline learning.
However, the light-tailed distribution was approximated with the Gaussian which is an infinite-support policy.
Let $\pi_{\mathcal{D}}$ denote the empirical behavior policy and $\mathcal{D}$ the offline dataset.
TAWAC minimizes the following actor loss:

$$ $\displaystyle\mathcal{L}(\phi):=\mathbb{E}_{s\sim\mathcal{D}}\left[D_{\!K\!L} \!\left(\pi_{\text{TKL},s}\left|\vphantom{\pi_{\phi,s}}\!\right|\pi_{\phi,s} \right)\right]$ $\displaystyle=\mathbb{E}_{\begin{subarray}{c}s\sim\mathcal{D}\\ a\sim\pi_{\mathcal{D}}\end{subarray}}\left[-\exp_{q^{\prime}}\left(\frac{Q_{s} (a)-V_{s}}{\tau}\right)\ln\pi_{\phi,s}(a)\right],$ (9) $$

where $\pi_{\text{TKL},s}(a)\propto\pi_{\mathcal{D},s}(a)\exp_{q^{\prime}}\!\left(
\tau^{-1}\left(Q_{s}(a)-V_{s}\right)\right)$ denotes the Tsallis KL regularized policy.
$\pi_{\phi,s}$ mimics a TKL policy which can be sparse depending on $q^{\prime}$.
In this case, it is natural to expect that a sparse policy parametrization may lead to better performance.

Figure: Algorithm 2 Out-of-support action handling for the light-tailed $q$-Gaussian

Algorithms like TAWAC that sample from a behavior policy $\pi_{\mathcal{D}}$ needs extra caution when using the $q$-exp policies.
When the light-tailed $q$-Gaussian is used, numerical issues can be incurred since the action sampled may fall outside the support of $\pi_{\phi}$, leading to undefined log-likelihood.
To resolve this problem, we propose to sample from $\pi_{\phi}$ a batch of $K$ actions and replace the out-of-support action with the in-support one with least $L_{2}$ distance, see Alg. 2.

## 5 Experiments

Figure: Figure 4: Learning curves on the classic control environments. Only the Gaussian and the best policy parametrization for each setting were shown with full opacity. The best policy is picked based on the total area under the curve (AUC). TAWAC(0) refers to TAWAC with entropic index $q^{\prime}=0$ in Eq. (9). Despite tuning hyperparameters separately for each policy, Gaussian is the best policy in only $1/12$ settings. In most other settings, the Gaussian policy performs significantly worse than the best.
Refer to caption: extracted/5897743/img/4x3onlinefigure_colors_uglier_green.png

Figure: Figure 5: (Left) The percentage of times that each policy parametrization is better than the Gaussian across all algorithm-environment combinations based on total AUC. If the bar is above the $50\%$ line, then it means that the said policy parametrization is better than Gaussian on average. We see that Student’s t and Light-tailed Gaussians are better than the Gaussian in $75\%$ and $66\%$ of the settings, respectively. (Right) Count of times where a policy parametrization performed the best across all algorithm-environment combinations based on AUC. We observe that the student-t policy performed the best in $5/12$ settings, whereas the Gaussian policy performed the best only once.
Refer to caption: extracted/5897743/img/123b.png

Our empirical study’s primary goal is to understand better the performance differences under this broader class of policy parameterizations in both online and offline settings.
We ran experiments with different algorithms, to get a better sense of how conclusions about policy parameterization vary across different actor-critic algorithms.

We parametrize Student-t’s DOF parameter $\nu$ in addition to its location and scale.
By contrast, the heavy-tailed $q$-Gaussian is fixed at $q=2$, since its allowable range is $1<q<3$.
For the light-tailed $q$-Gaussian, we opt for the standard choice of $q=0$.
Since Student’s t, heavy-tailed q-Gaussian, and Gaussian have unbounded support, we clipped the sampled action to fit the task’s action space without modifying the density.
We swept the hyperparameters using five random seeds, then increased the number of seeds to 10 for the best parameter setting. The hyperparameter sweeping ranges and the best values are provided in Appendix D.2 and D.3.

### 5.1 Online Classic Control

Domains and Baselines.
We used three classical control environments in the continuous action setting: Mountain Car (Sutton & Barto, ), Pendulum (Degris et al., ) and Acrobot (Sutton & Barto, ).
We chose the cost-to-goal version of Mountain Car, which outputs $-1$ reward per time step to encourage reaching the goal early.
We compared SAC, GreedyAC and two versions of TAWAC, $q^{\prime}=0$ and $q^{\prime}=2$.

Results.
Figure 4 shows the learning curves of all algorithm-environment combinations.
Only the Gaussian and the environment-specific best policy are shown with full opacity, computed based on area under curve (AUC).
One immediate observation is that, though all three algorithms by default choose the Gaussian policy, it was seldom the best policy parametrization.
Environment-wise,
on Mountain Car the Gaussian did not rank the best for any of the algorithms.
By contrast, the Beta policy attained the first place with SAC, as was the light-tailed $q$-Gaussian with TAWAC.
The same trend for the Gaussian holds in Acrobot and Pendulum as well, with exception only on TAWAC(0) Acrobot, where its curve closely resembled that of the light-tailed $q$-Gaussian.

Algorithm-wise, three observations are to be made:
(i) on Mountain Car the Beta policy performed significantly better than others.
This could be due to its flexibility in maintaining a skewed distribution shape that matches the BG policy more closely in contrast to the other location scale family members.
(ii) The $q$-Gaussians in general outperformed the Gaussian on TAWAC(0) and TAWAC(2) whose actor explicitly mimics a $q$-exp policy.
(iii) Student’s t has ranked the top involving all three algorithms.

Figure: Figure 6: Policy evolution of GreedyAC on Mountain Car. The Gaussian collapsed into a delta-like policy after only $10\%$ of the learning horizon.
Refer to caption: x2.png

Figure 5 LHS summarizes the percentage of each policy parametrization outperforming the Gaussian.
The Student’s t and light-tailed Gaussian went above 50%, suggesting potentially greater applicability.
The RHS shows out of 12 total combinations, how many times each policy parametrization has ranked the top.
The result shows that the Student’s t attained 5 times, contrasting the 1 time of the Gaussian.

In Figure 6 we visualized the evolution of Gaussian and $q$-Gaussian policies on the starting state over the first $4\times 10^{4}$ steps ($10\%$ of the entire learning horizon).
Note that the allowed action range is $[-1,1]$ but the plot shows $[-2,2]$ for better visualization.
As stated, Gaussian tends to quickly concentrate like a delta policy.
This can be detrimental to algorithms like SAC and GreedyAC which demand stochasticity to generate diverse samples.
By contrast, both light- and heavy-tailed $q$-Gaussians tend to be more stochastic.

### 5.2 Offline D4RL MuJoCo

Figure: Figure 7: Normalized scores on Medium-Replay level datasets from the MuJoCo suite. The black bar shows the median. Boxes and whiskers are $1\times$ and $1.5\times$ interquartile ranges, respectively. See Figure 14 for full comparison. Environment-wise, TAWAC with heavy-tailed $q$-Gaussian is often the top performer. Algorithm-wise, Student’s t consistently outperforms Squashed Gaussian.
Refer to caption: x3.png

Figure: Figure 8: Relative improvement to the Squashed Gaussian policy, averaged over multiple environments in the MuJoCo suite. The Student’s t could consistently outperform the Gaussian with all the chosen algorithms. The heavy-tailed $q$-Gaussian with TAWAC and IQL also achieved significant improvement. The improvement can reach up to $\sim 20\%$. Black vertical lines at the top indicate one standard error.
Refer to caption: x4.png

Figure: Figure 9: Policy evolution of all actions dimensions of TAWAC on Walker2d Medium Replay. Student’s t was flexible in that on some dimensions it beared lighter tails like the Gaussian by having large DOF (e.g. 4th), and with heavier tails on the others by having smaller DOF (e.g. 3rd, 6th). The peaks at the edges were caused by clipping actions into the allowed range.
Refer to caption: x5.png

Domains and Baselines.
We used the standard benchmark MuJoCo suite from D4RL to evaluate algorithm-policy combinations (Fu et al., ).
The following algorithms are compared: TAWAC, Advantage Weighetd Actor-Critic (AWAC) (Nair et al., ),
Implicit Q-Learning (IQL) (Kostrikov et al., ), In-sample Actor-Critic (InAC) (Xiao et al., ).
For TAWAC, we fixed its leading $q^{\prime}$-exp with $q^{\prime}=0$.
In Appendix C.2 we detailed the compared algorithms.
We also included a popular variant of the Gaussian known as the Squashed Gaussian for comparison.
Being able to evaluate the offline log-probability is critical to the tested algorithms, we found that light-tailed $q$-Gaussian leads to poor performance even with random online sampling, hence we do not show them here.

Results.
Figure 7 compared the normalized scores on the Medium-Replay datasets.
It can be seen that environment-wise, TAWAC + heavy-tailed $q$-Gaussian was the top performer, and could improve on the Squashed Gaussian by a non-negligible margin.
On HalfCheetah, heavy-tailed $q$-Gaussian attained the best score with every algorithm.
Algorithm-wise, the heavy-tailed $q$-Gaussian or/and Student’s t were better or equivalent to the Squashed Gaussian, except with AWAC on Hopper.
Student’s t was stable across algorithms, including these with which heavy-tailed $q$-Gaussian performed poorly (e.g., InAC).
This demonstrates the value of the learnable DOF parameter that allows it interpolates the Gaussian.
In Appendix E we provided comparison on other policies and datasets.

Figure 8 summarized the relative improvement over the Squashed Gaussian across environments.
Several observations can be made:
(i) though the Squashed Gaussian outperformed the Gaussian in general, it was seldom the best performer.
(ii) the Student’s t could consistently perform better than the Gaussian, the improvement can sometimes reach up to $\sim 20\%$.
The same holds for the heavy-tailed $q$-Gaussian with TAWAC and IQL.
(iii) though there was no single policy performing the best for all cases, choosing the Student’s t for the actors with exponential loss functions (AWAC, IQL, InAC), or
the heavy-tailed $q$-Gaussian for $q$-exponential actor losses (e.g. TAWAC) are generally effective.

Figure 9 visualized the policy evolution of the Squashed Gaussian and the two heavy-tailed policies, learned with TAWAC on Medium-Replay Walker2D.
Squashed Gaussian tended to converge slower here.
Since the offline MuJoCo environments are fully deterministic, a wide distribution indicates failure of finding the mode of the optimal action and therefore can be detrimental to learning performance.
The Squashed Gaussian converged slower than the heavy-tailed (performed the best) and the Student’s t.
Student’s t was flexible in that it beared lighter tails like the Gaussian in some dimensions by having a large DOF, for example in the 4th and 5th dimensions.
On the other hand, it could take heavy tails by having a small DOF like in the 3rd and 6th dimensions.

Table 3 examined the accumulated probabilities that fell on each
It can be seen that the Student’s t and the Gaussian tended to increasingly put more densities on the boundaries.
This is in sheer contrast to the heavy-tailed $q$-Gaussian that put the majority of probability density within the boundary.
This may explain the better performance of TAWAC + heavy-tailed $q$-Gaussian.

**Table 3: The summation of probability density accumulated on the left and the right edge in Figure 9 before clipping. Each pair indicates the left and right edge. The Student’s t and the Gaussian increasing put more densities on the edges as compared to the heavy-tailed $q$-Gaussian.**
| Policy<br># Updates | 0 | 100 | 200 | 300 | 400 |
| --- | --- | --- | --- | --- | --- |
| Heavy-tailed $q$-Gaussian | (24.39, 13.19) | (45.23, 2.36) | (45.49, 2.04) | (45.52, 1.98) | (45.54, 1.89) |
| Student’s t | (148.43, 71.23) | (198.89, 45.30) | (205.04, 37.04) | (207.15, 32.96) | (207.00, 33.84) |
| Gaussian | (190.96, 65.89) | (206.92, 53.05) | (211.77, 39.08) | (213.57, 33.71) | (214.39, 31.26) |

## 6 Conclusion

The Gaussian policy is standard for policy optimization algorithms on continuous action spaces.
In this paper we considered a broader family of policies that remains tractable, called the $q$-exponential family.
We empirically investigated their utility as a promising alternative to the Gaussian.
Specifically, we looked at the Student’s t, light- and heavy-tailed $q$-Gaussian policies.
Extensive experiments on both online and offline tasks with various actor-critic methods showed that heavy-tailed policies are in general effective.
In particular, we found the Student’s t policy to be generally more performing and stable than the Gaussian.
By contrast, the Heavy-tailed $q$-Gaussian seemed to favor especially Tsallis regularization and combined with which outperformed the baselines.

We acknowledge that the paper has limitations.
Perhaps the greatest is the inherent dilemma of the light-tailed $q$-Gaussian evaluating out-of-support actions.
Off-policy/offline algorithms require evaluating actions from some behavior policy and the actions can fall outside the support of the sparse $q$-Gaussian.
Naïvely discarding these samples results extremely slow or no learning.
In this paper we proposed to alleviate this issue by replacing them with the in-support sampled action with the least $L_{2}$ distance.
Nonetheless, this method did not help much in offline experiments.
We envision a potential solution that is left to future investigation: projecting the out-of-support actions precisely to the boundary of the $q$-Gaussian.
However, this would result in an increasingly difficult constrained optimization problem as the dimension grows.

## Appendix

The Appendix is organized into the following sections.
In section A we summarize the multivariate form of $q$-exp policies and derive gradients of their log-likelihood.
In section C we discuss the connection between the $q$-exp family and the entropy regularization literature.
Based on this, we further discuss how different algorithms may prefer specific policies depending on its actor loss.
We then provide implementation details including hyperparameters and how to sample from $q$-Gaussian in section D.
Lastly we provide additional experimental results in section E.

- A
Multivariate $q$-exp Policies and Log-likelihood
- B
Connection to Entropy Regularization
- C
Actor Losses
- D
Implementation Details
- E
Additional Results

## Appendix A Multivariate Density of q 𝑞 q italic_q -exp Policies

**Table 4: Multivariate $q$-exp policies and gradients of log-likelihood.**
| Policy | Density | $\nabla\ln\pi_{s}(a)$ |
| --- | --- | --- |
| Gaussian | $\frac{1}{(2\pi)^{\frac{N}{2}}|\Sigma|^{\frac{1}{2}}}\exp\left(-\frac{1}{2}<br>\left(\bm{a}-\bm{\mu}\right)^{\top}{\Sigma}^{-1}\left(\bm{a}-\bm{\mu}\right)\right)$ | Eq. (13) |
| Student’s t | $\frac{\Gamma\left(\frac{N+\nu}{2}\right)}{\Gamma\left(\frac{\nu}{2}\right)(\nu<br>\pi)^{\frac{N}{2}}|\Sigma|^{\frac{1}{2}}}\left[1+\frac{1}{\nu}\left(\bm{a}-\bm<br>{\mu}\right)^{\top}{\Sigma}^{-1}\left(\bm{a}-\bm{\mu}\right)\right]^{-\frac{N+<br>\nu}{2}}$ | Eq. (14) |
| $q$-Gaussian $(q<1)$ | $\frac{(1-q)^{\frac{N}{2}}\Gamma\left(\frac{2-q}{1-q}+\frac{N}{2}\right)}{<br>\Gamma\left(\frac{2-q}{1-q}\right)\pi^{\frac{N}{2}}|\Sigma|^{\frac{1}{2}}}\exp<br>_{q}\left(-\frac{1}{2}\left(\bm{a}-\bm{\mu}\right)^{\top}{\Sigma}^{-1}\left(<br>\bm{a}-\bm{\mu}\right)\right)$ | Eq. (15) |
| $q$-Gaussian $(1<q<3)$ | $\frac{(q-1)^{\frac{N}{2}}\Gamma\left(\frac{3-q}{2(q-1)}+\frac{N}{2}\right)}{<br>\Gamma\left(\frac{3-q}{2(q-1)}\right)\pi^{\frac{N}{2}}|\Sigma|^{\frac{1}{2}}}<br>\exp_{q}\left(-\frac{1}{2}\left(\bm{a}-\bm{\mu}\right)^{\top}{\Sigma}^{-1}<br>\left(\bm{a}-\bm{\mu}\right)\right)$ |  |

In Table 4 we show multivariate density of the $q$-exp policies introduced in the main text.
Note that multivariate Student’s t is constructed based on the assumption that a diagonal $\Sigma$ leads to independent action dimensions, same as the Gaussian policy.
On the other hand, for $q$-Gaussian this is no longer true, since a diagonal $\Sigma$ does not lead to product of univariate densities.

In the main text we showed their one-dimensional cases for simplicity.
For experiments the multivariate densities were used for experiments.
We now derive their gradients of log-likelihood with respect to parameters.
The following equations will be used frequently (Petersen & Pedersen, ):

$$ $\displaystyle\nabla_{\bm{\mu}}\,(\bm{a}-\bm{\mu})^{\top}\Sigma^{-1}(\bm{a}-\bm {\mu})=-2\Sigma^{-1}(\bm{a}-\bm{\mu}),$ (10) $\displaystyle\nabla_{\Sigma}\,\ln|\Sigma|=\left(\Sigma^{\top}\right)^{-1},$ (11) $\displaystyle\nabla_{\Sigma}(\bm{a}-\bm{\mu})^{\top}\Sigma^{-1}(\bm{a}-\bm{\mu })=-\Sigma^{-1}(\bm{a}-\bm{\mu})(\bm{a}-\bm{\mu})^{\top}\Sigma^{-1}.$ (12) $$

With these tools in hand, the following gradient expressions can be readily derived.

### A.1 Gaussian

Being a member of the exponential family, the gradient of Gaussian log-likelihood allows straightforward derivation by using Eq. (10)-Eq. (12):

$$ $\displaystyle\begin{split}\ln\pi_{s}(a)&=-\frac{N}{2}\ln 2\pi-\frac{1}{2}\ln| \Sigma|-\frac{1}{2}(\bm{a}-\bm{\mu})^{\top}\Sigma^{-1}(\bm{a}-\bm{\mu})\\ \Rightarrow\quad&\nabla_{\bm{\mu}}\ln\pi_{s}(a)=-\Sigma^{-1}(\bm{a}-\bm{\mu}), \\ &\nabla_{\Sigma}\ln\pi_{s}(a)=-\frac{1}{2}\left(\Sigma^{-1}-\Sigma^{-1}(\bm{a} -\bm{\mu})(\bm{a}-\bm{\mu})^{\top}\Sigma^{-1}\right).\end{split}$ (13) $$

### A.2 Student’s t

In addition to $\bm{\mu},\Sigma$, Student’s t policy has an additional learnable parameter degree of freedom $\nu$.
Recall that $\nu=1$ corresponds to the Cauchy’s distribution, while numerically with $\nu\geq 30$ it can be seen as a Gaussian distribution.

$$ $\displaystyle\begin{split}\ln\pi_{s}(a)&=\ln\Gamma\left(\frac{N+\nu}{2}\right) -\ln\Gamma\left(\frac{\nu}{2}\right)-\frac{N}{2}\ln\nu\pi-\frac{1}{2}\ln| \Sigma|-\frac{N+\nu}{2}\ln\left(1+\frac{1}{\nu}(\bm{a}-\bm{\mu})^{\top}\Sigma^ {-1}(\bm{a}-\bm{\mu})\right)\\ \Rightarrow\quad&\nabla_{\bm{\mu}}\ln\pi_{s}(a)=\frac{N+\nu}{\nu}\,\cdot\, \frac{\Sigma^{-1}(\bm{a}-\bm{\mu})}{1+\frac{1}{\nu}(\bm{a}-\bm{\mu})^{\top} \Sigma^{-1}(\bm{a}-\bm{\mu})},\\ &\nabla_{\Sigma}\ln\pi_{s}(a)=-\frac{1}{2}\left(\Sigma^{-1}-\frac{(N+\nu) \Sigma^{-1}(\bm{a}-\bm{\mu})(\bm{a}-\bm{\mu})^{\top}\Sigma^{-1}}{\nu+(\bm{a}- \bm{\mu})^{\top}\Sigma^{-1}(\bm{a}-\bm{\mu})}\right),\\ &\nabla_{\nu}\ln\pi_{s}(a)=\psi\left(\frac{N+\nu}{2}\right)-\psi\left(\frac{ \nu}{2}\right)-\frac{N}{2\nu}-\frac{N}{2}\ln\left(1+\frac{1}{\nu}(\bm{a}-\bm{ \mu})^{\top}\Sigma^{-1}(\bm{a}-\bm{\mu})\right)\\ &\hskip 199.16928pt+\frac{N+\nu}{2}\frac{\frac{1}{\nu}(\bm{a}-\bm{\mu})^{\top} \Sigma^{-1}(\bm{a}-\bm{\mu})}{\nu+(\bm{a}-\bm{\mu})^{\top}\Sigma^{-1}(\bm{a}- \bm{\mu})},\end{split}$ (14) $$

where $\psi(\cdot)$ is the digamma function.
For $\bm{\mu}$ and $\Sigma$ we again leveraged Eq. (10)-Eq. (12).

### A.3 q 𝑞 q italic_q -Gaussian

Since we do not parametrize the entropic index $q$, the gradients of log-likelihood with respect to $\bm{\mu},\Sigma$ are the same for both heavy- and light-tailed $q$-Gaussian.
Therefore, we focus on the light-tailed case $q<1$ and absorb into the constant $C$ the terms only related to $q$.

$$ $\displaystyle\begin{split}&\ln\pi_{s}(a)=\ln C-\frac{1}{2}\ln|\Sigma|+\frac{1} {1-q}\ln\left[1-\frac{1-q}{2}(\bm{a}-\bm{\mu})^{\top}\Sigma^{-1}(\bm{a}-\bm{ \mu})\right]_{+}\\ &\Rightarrow\,\nabla_{\bm{\mu}}\ln\pi_{s}(a)=\frac{1}{1-q}\frac{(1-q)\Sigma^{- 1}(\bm{a}-\bm{\mu})}{\left[1-\frac{1-q}{2}(\bm{a}-\bm{\mu})^{\top}\Sigma^{-1}( \bm{a}-\bm{\mu})\right]_{+}}=\frac{\Sigma^{-1}(\bm{a}-\bm{\mu})}{\exp_{q}\left (-\frac{1}{2}(\bm{a}-\bm{\mu})^{\top}\Sigma^{-1}(\bm{a}-\bm{\mu})\right)^{1-q} },\\ &\quad\,\,\,\,\nabla_{\Sigma}\ln\pi_{s}(a)=-\frac{1}{2}\left(\Sigma^{-1}-\frac {\Sigma^{-1}(\bm{a}-\bm{\mu})(\bm{a}-\bm{\mu})^{\top}\Sigma^{-1}}{\exp_{q} \left(-\frac{1}{2}(\bm{a}-\bm{\mu})^{\top}\Sigma^{-1}(\bm{a}-\bm{\mu})\right)^ {1-q}}\right).\end{split}$ (15) $$

It is interesting to see that the gradients of $q$-Gaussian log-likelihood can be seen as the Gaussian counterparts scaled by the reciprocal of $\exp_{q}(\cdot)^{1-q}$.
Since $\exp_{q}$ can take on zero values when $q<1$, the gradients as well as the log-likelihood function may be undefined outside the support.
However, this does not happen for heavy-tailed $q$-Gaussian $1<q<3$.

To make these policies suitable for deep reinforcement learning, we discuss in Appendix D how to parametrize the policies using neural networks.

## Appendix B Connection to Entropy Regularization

The $q$-exp family provides a general class of stochastic policies.
But perhaps more importantly, they can be derived as solutions to the maximum Tsallis entropy principle (Suyari & Tsukada, ; Furuichi, ), generalizing the maximum Shannon entropy principle (Jaynes, ; Grünwald & Dawid, ; Ziebart, ).
We discuss both principles in equation 16.

For notational convenience, we define the inner product for any two functions $F_{1},F_{2}\in\mathbb{R}^{|\mathcal{S}|\times|\mathcal{A}|}$ over actions as $\left\langle F_{1},F_{2}\right\rangle\in\mathbb{R}^{|\mathcal{S}|}$.
We write $F_{s}$ to express the function’s dependency $F$ on state $s$.
Often $F_{s}\in\mathbb{R}^{|\mathcal{A}|}$,
whenever its component is of concern, we denote it by $F_{s}(a)$.

### B.1 Boltzmann-Gibbs Regularization

Consider a regularized policy as the solution to the following regularization problem:

$$ $\displaystyle\pi_{\Omega,s}=\operatorname*{arg\,max}_{\pi_{s}\in\Delta_{ \mathcal{A}}}\left\langle\pi_{s},Q_{s}\right\rangle-\Omega(\pi_{s}),$ (16) $$

where $\Omega$ is a proper, lower semi-continuous, strictly convex function.
We can absorb the regularization coefficient $\tau>0$ into $\Omega$ by $\Omega:=\tau\tilde{\Omega}$.
It is a classic result that at the limit $\tau\rightarrow 0$ the unregularized optimal action is recovered: $\lim_{\tau\rightarrow 0}\pi_{\tau\tilde{\Omega},s}=\mathbbm{1}\{a=a^{*}\}$, i.e., $a^{*}$ that maximizes $Q_{s}$.

One of the most well-studied regularizers is the negative Shannon entropy $\Omega(\pi_{s})=\left\langle\pi_{s},\ln\pi_{s}\right\rangle$,
which leads to the Boltzmann-Gibbs policy $\pi_{\text{BG},s}(a)=\exp\big{(}Q_{s}(a)-Z_{s}\big{)}$.
Another popular choice is the KL divergence $\Omega(\pi_{s})=\left\langle\pi_{s},\ln\pi_{s}-\ln\mu_{s}\right\rangle$ for some reference policy $\mu_{s}$.
The regularized policy is $\pi_{\text{KL},s}(a)=\mu_{s}(a)\exp\big{(}Q_{s}(a)-Z_{s}\big{)}$.
Notice that it is also a member of the exponential family by writing $\pi_{\text{KL},s}(a)=\exp\big{(}Q_{s}(a)-Z_{s}+\ln\mu_{s}(a)\big{)}$.

### B.2 Tsallis Regularization

Originally, the deformed logarithm function was introduced in the statistical physics to generalize the Shannon entropy by deforming the logarithm contained in it (Naudts, ).
Consider replacing Shannon entropy in equation 16 with the negative Tsallis entropy $\Omega_{q}(\pi_{s})=\frac{1}{q-1}\left(\left\langle\mathbf{1},\pi_{s}^{q}
\right\rangle-1\right)$.
It has been shown that $\Omega_{q}(\pi_{s})$ leads to following regularized policy:

$$ $\displaystyle\pi_{\Omega_{q},s}(a)=\exp_{2-q}\big{(}Q_{s}(a)-Z^{\prime}_{q,s} \big{)}.$ (17) $$

We see that when $q=2$, it recovers the sparsemax policy introduced in Section 3.2.
As indicated by (Zhu et al., ), the effect of different $q\in(-\infty,1)$ lies in the extent of thresholding.
One can also consider regularization by the Tsallis KL divergence
$D^{q}_{\!K\!L}\!\left(\pi_{s}\left|\!\right|\mu_{s}\right):=\left\langle\pi_{s
},-\ln_{q}\frac{\mu_{s}}{\pi_{s}}\right\rangle$ (Furuichi et al., ).
Likewise to the KL case, $\mu$ is typically taken to be the last policy, in which the regularized policy is the product of two $q$-exp functions.

It is worth noting that there are other regularization functionals that can induce $q$-exp policies.
One of the prominent examples is the $\alpha$-entropy/divergence, which can be defined by simply letting $p=\frac{1}{q}$ in $\Omega_{q}(\pi_{s})$ (Peters et al., ; Belousov & Peters, ).
It is shown in (Xu et al., ; ) that when $\alpha=-1$ it induces the sparsemax policy.
Therefore, $q$-exp policies can also be viewed as solutions to the $\alpha$ regularization.

### B.3 Tsallis Advantage Weighted Actor Critic

An advantage of $q$-exp (resp. exp) policies is it may improve the consistency of algorithms that explicitly mimics a $q$-exp (resp. exp) policy.
For example, Tsallis Advantage Weighted Actor Critic (TAWAC) proposed to use a light-tailed $q$-exp policy for offline learning (Zhu et al., ).
However, TAWAC was implemented with Gaussian, which amounts to approximating a light-tailed distribution using one with infinite support.
Let $\pi_{\mathcal{D}}$ denote the empirical behavior policy and $\mathcal{D}$ the offline dataset.
TAWAC minimizes the following actor loss, where we ignore the parametrization of value functions:

$$ $\displaystyle\begin{split}\mathcal{L}(\phi):=\mathbb{E}_{s\sim\mathcal{D}} \left[D_{\!K\!L}\!\left(\pi_{\text{TKL},s}\left|\vphantom{\pi_{\phi,s}}\! \right|\pi_{\phi,s}\right)\right]&=\mathbb{E}_{\begin{subarray}{c}s\sim \mathcal{D}\\ a\sim\pi_{\mathcal{D}}\end{subarray}}\left[-\exp_{q^{\prime}}\left(\frac{Q_{s} (a)-V_{s}}{\tau}\right)\ln\pi_{\phi,s}(a)\right],\end{split}$ (18) $$

where $\pi_{\text{TKL},s}(a)\propto\pi_{\mathcal{D},s}(a)\exp_{q^{\prime}}\!\left(
\tau^{-1}\left(Q_{s}(a)-V_{s}\right)\right)$ denotes the Tsallis KL regularized policy.
We can generalize TAWAC to online learning by simply changing the expectation to be w.r.t. arbitrary behavior policy.
It is clear that depending on $q^{\prime}$, choosing Gaussian as $\pi_{\phi}$ may incur inconsistency with the theory.
A $q$-exp policy would be more suitable and could improve the performance.
As evidenced by our experimental results, heavy tailed policies indeed further improve the performance of TAWAC by a large margin.

## Appendix C Actor Losses

To help understand when exp-family policies (resp. $q$-exp) may be more preferable, we compare the actor loss functions of the algorithms in the experiment section.

### C.1 Online Algorithms

Soft Actor-Critic. SAC minimizes the following KL loss for the actor

$$ $\displaystyle\mathcal{L}_{\text{SAC}}(\phi):=\mathbb{E}_{s\sim\mathcal{B}} \left[D_{\!K\!L}\!\left(\pi_{\phi}(\cdot|s)\left|\vphantom{{\pi}_{\text{BG}}( \cdot|s)}\!\right|{\pi}_{\text{BG}}(\cdot|s)\right)\right]=\mathbb{E}_{s\sim \mathcal{B}}\left[D_{\!K\!L}\!\left(\pi_{\phi}(\cdot|s)\left|\vphantom{\frac{ \exp\left(\tau^{-1}Q(s,\cdot)\right)}{Z_{s}}}\!\right|\frac{\exp\left(\tau^{-1 }Q(s,\cdot)\right)}{Z_{s}}\right)\right],$ $$

where states are sampled from replay buffer $\mathcal{B}$.
The parametrized policy $\pi_{\phi}$ is projected to be close to the BG policy, therefore it is reasonable to expect that choosing $\pi_{\phi}$ from the exp-family may be more preferable.
Depending on action values, BG can be skewed, multi-modal.
Therefore, the symmetric, unimodal Gaussian may not be able to fully capture these characteristics.

Greedy Actor-Critic.
GreedyAC maintains an additional proposal policy besides the actor.
The proposal policy is responsible for producing actions from which
the top $k\%$ of actions are used to update the actor.
The proposal policy itself is updated similarly but with an entropy bonus encouraging exploration.
To simplify notations, we use $I(s)$ to denote the set containing top $k\%$ actions given $s$.

$$ $\displaystyle\mathcal{L}_{\text{GreedyAC, prop}}(\phi):=\mathbb{E}_{\begin{ subarray}{c}s\sim\mathcal{B}\\ a\in I(s)\end{subarray}}\left[-\ln\pi_{\phi}(a|s)-\mathcal{H}\left(\pi_{\phi}( \cdot|s)\right)\right],$ $\displaystyle\mathcal{L}_{\text{GreedyAC, actor}}(\bar{\phi}):=\mathbb{E}_{ \begin{subarray}{c}s\sim\mathcal{B}\\ a\in I(s)\end{subarray}}\left[-\ln\pi_{\bar{\phi}}(a|s)\right].$ $$

GreedyAC maximizes log-likelihood of the actor and proposal policy.
These policies impose no constraints on the functional form of $\pi$.

Online Tsallis AWAC.
Online TAWAC is extended to condition on the behavior policy that collects experiences $\pi_{\text{theory}}(a|s)\propto\pi_{\text{behavior}}(a|s)\exp_{q}\left(\frac{Q
(s,a)-V(s)}{\tau}\right)$.

$$ $\displaystyle\mathcal{L}_{\text{TAWAC}}(\phi):$ $\displaystyle=\mathbb{E}_{s\sim\mathcal{B}}\left[D_{\!K\!L}\!\left(\pi_{\text{ theory}}(\cdot|s)\left|\vphantom{\pi_{\phi}(\cdot|s)}\!\right|\pi_{\phi}(\cdot |s)\right)\right]$ $\displaystyle=\mathbb{E}_{\begin{subarray}{c}s\sim\mathcal{B}\\ a\sim\pi_{\bar{\phi}}\end{subarray}}\left[-\exp_{q}\left(\frac{Q(s,a)-V(s)}{ \tau}\right)\ln\pi_{\phi}(a|s)\right],$ $$

where the condition $a\sim\pi_{\bar{\phi}}$ is because the target policy is used to sample actions.
Since Tsallis AWAC explicitly minimizes KL loss to a $q$-exp policy, which can be light-tailed/heavy-tailed depending on $q$.
Therefore, choosing a $q$-exp $\pi_{\phi}$ could lead to better performance.

### C.2 Offline Algorithms

AWAC.
Advantage Weighted Actor-Critic (AWAC) is the basis of many algorithms. AWAC minimizes the following actor loss:

$$ $\displaystyle\mathcal{L}_{\text{AWAC}}(\phi):=\mathbb{E}_{\begin{subarray}{c}s \sim\mathcal{D}\\ a\sim\pi_{\mathcal{D}}\end{subarray}}\left[-\exp\left(\frac{Q(s,a)-V(s)}{\tau} \right)\ln\pi_{\phi}(a|s)\right],$ $$

which is derived as the result of minimizing KL loss
$D_{\!K\!L}\!\left(\pi_{\mathcal{D}}\left|\vphantom{\pi_{\phi}}\!\right|\pi_{
\phi}\right)$ and applying the trick in Eq. 18,
i.e., $\pi_{\text{theory}}(a|s)\propto\pi_{\mathcal{D}}(a|s)\exp\left(\frac{Q(s,a)-V(
s)}{\tau}\right)=\exp\left(\frac{Q(s,a)-V(s)}{\tau}-\ln\pi_{\mathcal{D}}(a|s)\right)$.
However, the shape of this policy can be multi-modal and skewed depending on the values and $\pi_{\mathcal{D}}$.
It is visible from experimental results that Beta and Squashed Gaussian have similar performance.

IQL.
In contrast to AWAC, Implicit Q-Learning (IQL) does not have an explicit actor learning procedure and uses $\mathcal{L}_{\text{AWAC}}(\phi)$ as a means for policy extraction from the learned value functions.
The exponential advantage function acts simply as weights.
Therefore, IQL does not assume the functional form of $\pi_{\phi}$.

InAC.
In-Sample Actor-Critic (InAC) proposed to impose an in-sample constraint on the entropy-regularized BG policy.
As such, the dependence on the behavior policy is moved into the exponential-advantage weighting function:

$$ $\displaystyle\mathcal{L}_{\text{InAC}}(\phi):=\mathbb{E}_{\begin{subarray}{c}s \sim\mathcal{D}\\ a\sim\pi_{\mathcal{D}}\end{subarray}}\left[-\exp\left(\frac{Q(s,a)-V(s)}{\tau} -\ln\pi_{\mathcal{D}}(a|s)\right)\ln\pi_{\phi}(a|s)\right].$ $$

As a result, InAC is not as sensitive to the advantage weighting as AWAC does, which implies that InAC may favor an exp $\pi_{\phi}$ but less than AWAC.

Offline Tsallis AWAC.
The offline case of Tsallis AWAC is same as the online case except the change of expectation:

$$ $\displaystyle\mathcal{L}_{\text{TAWAC}}(\phi):=\mathbb{E}_{\begin{subarray}{c} s\sim\mathcal{D}\\ a\sim\pi_{\mathcal{D}}\end{subarray}}\left[-\exp_{q}\left(\frac{Q(s,a)-V(s)}{ \tau}\right)\ln\pi_{\phi}(a|s)\right].$ $$

Same with the online case, offline Tsallis AWAC may theoretically prefer a $q$-exp $\pi_{\phi}$.

TD3BC.
In Appendix E we include additional results of TD3BC (Fujimoto & Gu, ), whose actor loss is obtained by simply augmenting the TD3 loss with a behavior cloning term:

$$ $\displaystyle\mathcal{L}_{\text{TD3BC}}(\phi):=\mathbb{E}_{\begin{subarray}{c} s\sim\mathcal{D}\\ a\sim\pi_{\mathcal{D}}\end{subarray}}\left[\lambda Q(s,\pi(s))-\left(\pi(s)-a \right)^{2}\right].$ $$

The behavior cloning term is simply minimizing the $L_{2}$ distance to actions in the dataset.
Though another interpretation by (Xiao et al., ) is that this term can be understood as applying KL regularization to Gaussian policy.

## Appendix D Implementation Details

Details of our implementation is provided in this section.
Specifically, we detail our design choices hyperparameters and network architectures.

Figure: Figure 10: Beta distribution with $\alpha<1,\beta<1$ takes on a bowl shape rather than a bell shape. The shape can also be skewed as well as symmetric.
Refer to caption: extracted/5897743/img/different_beta_distributions.png

### D.1 Policies

We discuss how to parametrize Beta, Student’s t and $q$-Gaussian policies.
Specifically, we parametrize $\alpha,\beta$ for Beta policy; $\bm{\mu},\Sigma$ for $q$-Gaussian.
In additional to location and scale, Student’s t has an additional learnable parameter $\nu$.

For Student’s t policy, we initialized a base DOF $\nu_{0}=1$ and learn $\nu$ by the softplus function.
The Student’s t policy therefore always has DOF $\nu>1$,
which is equivalent to starting as the Cauchy’s distribution.
For Beta policy, we similarly constrain $\alpha,\beta$ to be the output of softplus function plus 1.
This is because when $\alpha<1,\beta<1$ the Beta policy takes on a bowl shape rather than a bell shape, see Figure 10.
For Gaussian and $q$-Gaussian policies, we follow the standard practice to parametrize mean by the tanh activation and scale by the log-std transform.

In the tested off-policy/offline algorithms, it is necessary to evaluate log-probability for off-policy/offline actions stored in the buffer.
For light-tailed $q$-Gaussian this can cause numerical issues since the evaluated actions may fall outside the support, incurring $-\infty$ for log-probability.
To avoid this issue, we sample a batch of on-policy actions from the $q$-Gaussian and replace the out-of-support actions with the nearest action in the $L_{2}$ sense.

In our experiments, all environments had bounded action space. Squashed Gaussian and light-tailed q-Gaussian provide bounded output. However, Student’s t, heavy tailed q-Gaussian and Gaussian have unbounded support. For these distributions, we clipped the sampled action to fit the action space of the task, without further modification on the density. The mean value is constrained using tanh function in distributions with unbounded support, except the standard Gaussian in offline learning.

### D.2 Online Experiments

We used three classical control environments in the continuous action setting: Mountain Car (Sutton & Barto, ), Pendulum (Degris et al., ) and Acrobot (Sutton & Barto, ). All episodes are truncated at 1000 time steps. In Mountain Car, the action is the force applied to the car in $[-1,1]$, and the agent receives a reward of -1 at every time step.
In Pendulum, the action is the torque applied to the base of the pendulum in $[-2,2]$ and the reward is defined by $r=-(\theta^{2}+0.1*(\frac{\mathrm{d}\theta}{\mathrm{d}t})^{2}+0.001*a^{2})$ where $\theta$ denotes the angle, $\frac{\mathrm{d}\theta}{\mathrm{d}t}$ is the derivative of time and $a$ the torque applied. Finally, in acrobot, the action is the torque applied on the joint between two links in $[-1,1]$ and the agent receives a reward of $-1$ per time step.

Experiment settings: When sweeping different hyperparameter configurations, we pause the training every 10,000 time steps and then evaluate the learned policy by averaging the total reward over 3 episodes. However, when running the best hyperparameter configuration, we evaluate by freezing the policy every 1000 time steps and then computing the total reward obtained for 1 episode.

Parameter sweeping: We sweep the hyperparameters with 5 independent runs and then evaluate the run configuration for 30 seeds. We select the best hyperparameters based on the overall area under curve. When running the best hyperparameter configurations, we discard the original 5 seeds used for the hyperparameter sweep in order to avoid the bias caused by hyperparameter selection. Details regarding the fixed and swept hyperparameters are provided in Table 5.

Agent learning: We used a 2-layer network with 64 nodes on each layer and ReLU non-linearities. The batch size was 32. Agents used a target network for the critic, updated with polyak averaging with $\alpha=0.01$.

**Table 5: Default hyperparameters and sweeping choices for online experiments.**
| Hyperparameter | Value |
| --- | --- |
| Critic Learning rate | Swept in $\{1\times 10^{-2},1\times 10^{-3},1\times 10^{-4},1\times 10^{-5}\}$ |
| Critic learning rate multiplier for actor | Swept in $\{0.1,1,10\}$ |
| Temperature | Swept in $\{0.01,0.1,1\}$ |
| Discount rate | 0.99 |
| Hidden size of Value network | 64 |
| Hidden layers of Value network | 2 |
| Hidden size of Policy network | 64 |
| Hidden layers of Policy network | 2 |
| Minibatch size | 32 |
| Adam.$\beta_{1}$ | 0.9 |
| Adam.$\beta_{2}$ | 0.999 |
| Number of seeds for sweeping | 10 |
| Number of seeds for the best setting | 30 |

### D.3 Offline Experiments

We use the MuJoCo suite from D4RL (Apache-2/CC-BY licence) (Fu et al., ) for offline experiments.
The offline datasets are collected by a SAC agent,
and their naming reflects the level of the trained agent used to collect the transitions.
the medium dataset contains 1 million samples generated by a partially-trained SAC policy.
Medium-expert combines the trajectories of the expert and the medium.
Medium-replay consists of samples in the replay buffer during training until the policy reaches the medium level of performance.

Experiment settings: We conducted the offline experiment using 9 datasets provided in D4RL: halfcheetah-medium-expert, halfcheetah-medium, halfcheetah-medium-replay, hopper-medium-expert, hopper-medium, hopper-medium-replay, walker2d-medium-expert, walker2d-medium, and walker2d-medium-replay. We run 5 agents: TAWAC, AWAC, IQL, InAC, and TD3BC. The results of TD3BC are posted in the appendix. For each agent, we tested 5 distributions: Gaussian, Squashed Gaussian, Beta, Student’s t, and Heavy-tailed $q$-Gaussian. As offline learning algorithms usually require a distribution covering the whole action space, Light-tailed q-Gaussian is not considered in offline learning experiments. Each agent was trained for $1\times 10^{6}$ steps. The policy was evaluated every $1000$ steps. The score was averaged over $5$ rollouts in the real environment; each had $1000$ steps.

Parameter sweeping: All results shown in the paper were generated by the best parameter setting after sweeping. We list the parameter setting in Table 6. Learning rate and temperature in TAWAC + medium datasets were swept as the experiments in their publication did not include the medium dataset. The best learning rates are reported in Table 7, and the temperatures are listed in Table 8.

**Table 6: Default hyperparameters and sweeping choices for offline experiments.**
| Hyperparameter | Value |
| --- | --- |
| Learning rate | Swept in $\{3\times 10^{-3},1\times 10^{-3},3\times 10^{-4},1\times 10^{-4}\}$<br>See the best setting in Table 7 |
| Temperature | Same as the number reported in<br>the publication of each algorithm.<br>Except in TAWAC + medium datasets, the value was<br>swept in $\{1.0,0.5,0.01\}$.<br>See the setting in Table 8 |
| IQL Expectile | 0.7 |
| Discount rate | 0.99 |
| Hidden size of Value network | 256 |
| Hidden layers of Value network | 2 |
| Hidden size of Policy network | 256 |
| Hidden layers of Policy network | 2 |
| Minibatch size | 256 |
| Adam.$\beta_{1}$ | 0.9 |
| Adam.$\beta_{2}$ | 0.99 |
| Number of seeds for sweeping | 5 |
| Number of seeds for the best setting | 10 |

**Table 7: Best learning rates for offline experiments.**
| Dataset | Distribution | TAWAC | AWAC | IQL | InAC | TD3BC |
| --- | --- | --- | --- | --- | --- | --- |
| HalfCheetah-Medium-Expert | Heavy-Tailed q-Gaussian | 0.001 | 0.001 | 0.001 | 0.001 | 0.0003 |
| HalfCheetah-Medium-Expert | Squashed Gaussian | 0.001 | 0.0003 | 0.0003 | 0.001 | 0.0003 |
| HalfCheetah-Medium-Expert | Gaussian | 0.0003 | 0.0001 | 0.0003 | 0.0003 | 0.0003 |
| HalfCheetah-Medium-Expert | Beta | 0.001 | 0.0003 | 0.001 | 0.001 | 0.001 |
| HalfCheetah-Medium-Expert | Student’s t | 0.001 | 0.0003 | 0.0003 | 0.0003 | 0.001 |
| HalfCheetah-Medium-Replay | Heavy-Tailed q-Gaussian | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 |
| HalfCheetah-Medium-Replay | Squashed Gaussian | 0.001 | 0.0003 | 0.0003 | 0.001 | 0.003 |
| HalfCheetah-Medium-Replay | Gaussian | 0.001 | 0.0001 | 0.0003 | 0.001 | 0.001 |
| HalfCheetah-Medium-Replay | Beta | 0.001 | 0.0003 | 0.0003 | 0.001 | 0.001 |
| HalfCheetah-Medium-Replay | Student’s t | 0.001 | 0.0003 | 0.0003 | 0.0003 | 0.003 |
| HalfCheetah-Medium | Heavy-Tailed q-Gaussian | 0.001 | 0.001 | 0.001 | 0.001 | 0.0003 |
| HalfCheetah-Medium | Squashed Gaussian | 0.001 | 0.0003 | 0.001 | 0.001 | 0.0003 |
| HalfCheetah-Medium | Gaussian | 0.0003 | 0.0001 | 0.0003 | 0.001 | 0.001 |
| HalfCheetah-Medium | Beta | 0.001 | 0.001 | 0.001 | 0.001 | 0.0003 |
| HalfCheetah-Medium | Student’s t | 0.001 | 0.0003 | 0.001 | 0.001 | 0.001 |
| Hopper-Medium-Expert | Heavy-Tailed q-Gaussian | 0.001 | 0.001 | 0.001 | 0.001 | 0.0001 |
| Hopper-Medium-Expert | Squashed Gaussian | 0.001 | 0.001 | 0.001 | 0.001 | 0.0001 |
| Hopper-Medium-Expert | Gaussian | 0.0003 | 0.0003 | 0.001 | 0.001 | 0.0001 |
| Hopper-Medium-Expert | Beta | 0.001 | 0.001 | 0.001 | 0.003 | 0.0003 |
| Hopper-Medium-Expert | Student’s t | 0.001 | 0.0003 | 0.001 | 0.003 | 0.0001 |
| Hopper-Medium-Replay | Heavy-Tailed q-Gaussian | 0.001 | 0.0001 | 0.001 | 0.0001 | 0.001 |
| Hopper-Medium-Replay | Squashed Gaussian | 0.0001 | 0.0003 | 0.001 | 0.0003 | 0.001 |
| Hopper-Medium-Replay | Gaussian | 0.0003 | 0.0003 | 0.001 | 0.0003 | 0.001 |
| Hopper-Medium-Replay | Beta | 0.0001 | 0.0003 | 0.0003 | 0.003 | 0.003 |
| Hopper-Medium-Replay | Student’s t | 0.0003 | 0.0003 | 0.0003 | 0.0003 | 0.001 |
| Hopper-Medium | Heavy-Tailed q-Gaussian | 0.003 | 0.001 | 0.001 | 0.001 | 0.0001 |
| Hopper-Medium | Squashed Gaussian | 0.001 | 0.0003 | 0.001 | 0.0003 | 0.0001 |
| Hopper-Medium | Gaussian | 0.001 | 0.001 | 0.0003 | 0.001 | 0.001 |
| Hopper-Medium | Beta | 0.001 | 0.001 | 0.003 | 0.001 | 0.001 |
| Hopper-Medium | Student’s t | 0.001 | 0.001 | 0.001 | 0.001 | 0.0001 |
| Walker2d-Medium-Expert | Heavy-Tailed q-Gaussian | 0.0003 | 0.001 | 0.001 | 0.001 | 0.0003 |
| Walker2d-Medium-Expert | Squashed Gaussian | 0.001 | 0.001 | 0.0003 | 0.001 | 0.0003 |
| Walker2d-Medium-Expert | Gaussian | 0.0003 | 0.0001 | 0.0003 | 0.001 | 0.001 |
| Walker2d-Medium-Expert | Beta | 0.001 | 0.0003 | 0.001 | 0.001 | 0.001 |
| Walker2d-Medium-Expert | Student’s t | 0.001 | 0.0003 | 0.0003 | 0.0003 | 0.0003 |
| Walker2d-Medium-Replay | Heavy-Tailed q-Gaussian | 0.0003 | 0.0003 | 0.003 | 0.0003 | 0.001 |
| Walker2d-Medium-Replay | Squashed Gaussian | 0.001 | 0.0003 | 0.0003 | 0.001 | 0.001 |
| Walker2d-Medium-Replay | Gaussian | 0.001 | 0.0003 | 0.0003 | 0.001 | 0.003 |
| Walker2d-Medium-Replay | Beta | 0.001 | 0.0003 | 0.0003 | 0.001 | 0.0003 |
| Walker2d-Medium-Replay | Student’s t | 0.0003 | 0.0003 | 0.0003 | 0.001 | 0.001 |
| Walker2d-Medium | Heavy-Tailed q-Gaussian | 0.003 | 0.001 | 0.001 | 0.001 | 0.0001 |
| Walker2d-Medium | Squashed Gaussian | 0.001 | 0.001 | 0.001 | 0.001 | 0.0001 |
| Walker2d-Medium | Gaussian | 0.001 | 0.0001 | 0.001 | 0.001 | 0.0001 |
| Walker2d-Medium | Beta | 0.001 | 0.0003 | 0.003 | 0.001 | 0.0001 |
| Walker2d-Medium | Student’s t | 0.001 | 0.0003 | 0.001 | 0.001 | 0.0001 |

**Table 8: Temperature settings for offline experiments.**
| Dataset | Distribution | TAWAC | AWAC | IQL | InAC | TD3BC |
| --- | --- | --- | --- | --- | --- | --- |
| HalfCheetah-Medium-Expert | Heavy-Tailed q-Gaussian | 1.00 | 1.00 | 0.33 | 0.10 | 2.50 |
| HalfCheetah-Medium-Expert | Squashed Gaussian | 1.00 | 1.00 | 0.33 | 0.10 | 2.50 |
| HalfCheetah-Medium-Expert | Gaussian | 1.00 | 1.00 | 0.33 | 0.10 | 2.50 |
| HalfCheetah-Medium-Expert | Beta | 1.00 | 1.00 | 0.33 | 0.10 | 2.50 |
| HalfCheetah-Medium-Expert | Student’s t | 1.00 | 1.00 | 0.33 | 0.10 | 2.50 |
| HalfCheetah-Medium-Replay | Heavy-Tailed q-Gaussian | 0.01 | 1.00 | 0.33 | 0.50 | 2.50 |
| HalfCheetah-Medium-Replay | Squashed Gaussian | 0.01 | 1.00 | 0.33 | 0.50 | 2.50 |
| HalfCheetah-Medium-Replay | Gaussian | 0.01 | 1.00 | 0.33 | 0.50 | 2.50 |
| HalfCheetah-Medium-Replay | Beta | 0.01 | 1.00 | 0.33 | 0.50 | 2.50 |
| HalfCheetah-Medium-Replay | Student’s t | 0.01 | 1.00 | 0.33 | 0.50 | 2.50 |
| HalfCheetah-Medium | Heavy-Tailed q-Gaussian | 0.01 | 0.50 | 0.33 | 0.33 | 2.50 |
| HalfCheetah-Medium | Squashed Gaussian | 0.01 | 0.50 | 0.33 | 0.33 | 2.50 |
| HalfCheetah-Medium | Gaussian | 0.01 | 0.50 | 0.33 | 0.33 | 2.50 |
| HalfCheetah-Medium | Beta | 0.01 | 0.50 | 0.33 | 0.33 | 2.50 |
| HalfCheetah-Medium | Student’s t | 0.01 | 0.50 | 0.33 | 0.33 | 2.50 |
| Hopper-Medium-Expert | Heavy-Tailed q-Gaussian | 0.50 | 1.00 | 0.33 | 0.01 | 2.50 |
| Hopper-Medium-Expert | Squashed Gaussian | 0.50 | 1.00 | 0.33 | 0.01 | 2.50 |
| Hopper-Medium-Expert | Gaussian | 0.50 | 1.00 | 0.33 | 0.01 | 2.50 |
| Hopper-Medium-Expert | Beta | 0.50 | 1.00 | 0.33 | 0.01 | 2.50 |
| Hopper-Medium-Expert | Student’s t | 0.50 | 1.00 | 0.33 | 0.01 | 2.50 |
| Hopper-Medium-Replay | Heavy-Tailed q-Gaussian | 0.50 | 0.50 | 0.33 | 0.50 | 2.50 |
| Hopper-Medium-Replay | Squashed Gaussian | 0.50 | 0.50 | 0.33 | 0.50 | 2.50 |
| Hopper-Medium-Replay | Gaussian | 0.50 | 0.50 | 0.33 | 0.50 | 2.50 |
| Hopper-Medium-Replay | Beta | 0.50 | 0.50 | 0.33 | 0.50 | 2.50 |
| Hopper-Medium-Replay | Student’s t | 0.50 | 0.50 | 0.33 | 0.50 | 2.50 |
| Hopper-Medium | Heavy-Tailed q-Gaussian | 0.50 | 0.50 | 0.33 | 0.10 | 2.50 |
| Hopper-Medium | Squashed Gaussian | 0.50 | 0.50 | 0.33 | 0.10 | 2.50 |
| Hopper-Medium | Gaussian | 0.50 | 0.50 | 0.33 | 0.10 | 2.50 |
| Hopper-Medium | Beta | 0.50 | 0.50 | 0.33 | 0.10 | 2.50 |
| Hopper-Medium | Student’s t | 0.01 | 0.50 | 0.33 | 0.10 | 2.50 |
| Walker2d-Medium-Expert | Heavy-Tailed q-Gaussian | 0.01 | 0.10 | 0.33 | 0.10 | 2.50 |
| Walker2d-Medium-Expert | Squashed Gaussian | 0.01 | 0.10 | 0.33 | 0.10 | 2.50 |
| Walker2d-Medium-Expert | Gaussian | 0.01 | 0.10 | 0.33 | 0.10 | 2.50 |
| Walker2d-Medium-Expert | Beta | 0.01 | 0.10 | 0.33 | 0.10 | 2.50 |
| Walker2d-Medium-Expert | Student’s t | 0.01 | 0.10 | 0.33 | 0.10 | 2.50 |
| Walker2d-Medium-Replay | Heavy-Tailed q-Gaussian | 0.50 | 0.10 | 0.33 | 0.50 | 2.50 |
| Walker2d-Medium-Replay | Squashed Gaussian | 0.50 | 0.10 | 0.33 | 0.50 | 2.50 |
| Walker2d-Medium-Replay | Gaussian | 0.50 | 0.10 | 0.33 | 0.50 | 2.50 |
| Walker2d-Medium-Replay | Beta | 0.50 | 0.10 | 0.33 | 0.50 | 2.50 |
| Walker2d-Medium-Replay | Student’s t | 0.50 | 0.10 | 0.33 | 0.50 | 2.50 |
| Walker2d-Medium | Heavy-Tailed q-Gaussian | 0.01 | 0.10 | 0.33 | 0.33 | 2.50 |
| Walker2d-Medium | Squashed Gaussian | 1.00 | 0.10 | 0.33 | 0.33 | 2.50 |
| Walker2d-Medium | Gaussian | 1.00 | 0.10 | 0.33 | 0.33 | 2.50 |
| Walker2d-Medium | Beta | 1.00 | 0.10 | 0.33 | 0.33 | 2.50 |
| Walker2d-Medium | Student’s t | 0.01 | 0.10 | 0.33 | 0.33 | 2.50 |

Agent learning: We used a 2-layer network with 256 nodes on each layer. The batch size was 256. Agents used a target network for the critic, updated with polyak averaging with $\alpha=0.005$. The discount rate was set to $0.99$.

## Appendix E Further Results

Our offline experiments comprise 3 environments, 3 datasets, 5 agents, and 5 policies.
We sweep hyperparameters with 5 seeds and run extra 5 seeds for the best hyperparameter configuration.
Running time in CPU hours for each agent is approximately:
AWAC=8, InAC=12, IQL=8, TD3BC=7, TAWAC=8.
Therefore, the total running time on is 53,175 CPU-hours. Additionally, the total running time for sweeping the hyperparameters and evaluating the best configurations is around 23,652 CPU-hours.

Here we show the results of all algorithm-policy combination on all environments.
We also include TD3BC (Fujimoto & Gu, ) for comparison.

Figure 11 shows the overall comparison with TD3.
It is clear that Squashed Gaussian performs well and Beta can show slight improvements in some cases.
Though it is visible that no much difference is shown except on the Medium-Replay data.
We conjecture that the better performance of Squashed Gaussian and Beta could be due to the TD3BC behavior cloning loss.
It is encouraged that policy closely approximates the actions from the dataset.
Therefore, policies like Beta that can concentrate faster may be more advantageous.

Figures 12 to 14 display boxplots of the combinations on environments of each level.
Consistent observations to that in the main text can be drawn from these plots, but with the exception that in Figure 13 the environment-wise best combination is TAWAC + Student’s t.
TD3BC does not exhibit strong sensitivity to the choice of policy.

Lastly, for all of the results shown above, their learning curves are shown in Figures 15 to 19. We smoothed the curves with window size 10 for better visualization.

Figure: Figure 11: Relative improvement to the Squashed Gaussian policy, averaged over environments. Black vertical lines at the top indicate one standard error. For TD3BC, Beta policy outperforms the Squashed Gaussian on Medium-Expert and Medium-Replay.
Refer to caption: x12.png

Figure: Figure 12: Normalized scores on Medium-Expert level datasets. The black bar shows the median. Boxes and whiskers show $1\times$ and $1.5\times$ interquartile ranges, respectively. Fliers are not plotted for uncluttered visualization. Environment-wise, InAC with heavy-tailed $q$-Gaussian is the top performer. Algorithm-wise, heavy-tailed or/and Student’s t can improve or match the performance of the Squashed Gaussian except AWAC. With TD3BC no significant difference between policies is observed.
Refer to caption: x13.png

Figure: Figure 13: Normalized scores on Medium level datasets. The black bar shows the median. Boxes and whiskers show $1\times$ and $1.5\times$ interquartile ranges, respectively. Fliers are not plotted for uncluttered visualization. Environment-wise, InAC with heavy-tailed $q$-Gaussian is the top performer. Algorithm-wise, heavy-tailed $q$-Gaussian has observed significant performance drop with AWAC and InAC on Hopper and Walker2d. With TD3BC no significant difference between policies is observed.
Refer to caption: x14.png

Figure: Figure 14: Normalized scores on Medium-Replay level datasets. The black bar shows the median. Boxes and whiskers show $1\times$ and $1.5\times$ interquartile ranges, respectively. Fliers are not plotted for uncluttered visualization. Environment-wise, TAWAC + heavy-tailed $q$-Gaussian is the best performer. Algorithm-wise, Student’s t is stable and can match or improve on the performance of (Squashed) Gaussian.
Refer to caption: x15.png

Figure: Figure 15: TAWAC learning curves in all datasets. Columns show different environments and rows are the levels of the environments. x-axis denotes the number of steps ($\times 10^{4}$), and y-axis is the normalized score. Each curve was smoothed with window size 10.
Refer to caption: x16.png

Figure: Figure 16: AWAC learning curves in all datasets. Columns show different environments and rows are the levels of the environments. x-axis denotes the number of steps ($\times 10^{4}$), and y-axis is the normalized score. Each curve was smoothed with window size 10.
Refer to caption: x17.png

Figure: Figure 17: IQL learning curves in all datasets. Columns show different environments and rows are the levels of the environments. x-axis denotes the number of steps ($\times 10^{4}$), and y-axis is the normalized score. Each curve was smoothed with window size 10.
Refer to caption: x18.png

Figure: Figure 18: InAC learning curves in all datasets. Columns show different environments and rows are the levels of the environments. x-axis denotes the number of steps ($\times 10^{4}$), and y-axis is the normalized score. Each curve was smoothed with window size 10.
Refer to caption: x19.png

Figure: Figure 19: TD3+BC learning curves in all datasets. Columns show different environments and rows are the levels of the environments. x-axis denotes the number of steps ($\times 10^{4}$), and y-axis is the normalized score. Each curve was smoothed with window size 10.
Refer to caption: x20.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: The results are clear and I do not have questions.
soundness: 3
strengths: The motivation for using the q-exponential family of distributions to encourage exploration seems intuitive. The numerical simulations are quite extensive and successfully demonstrate that it is not always optimal to use a Gaussian distribution to parameterize the policy in reinforcement learning.
summary: This paper focuses on solving reinforcement learning problems with continuous action spaces using policy optimization methods. In such cases, a typical solution for handling continuous action spaces is to impose a probability distribution (specified by finitely many parameters) on the continuous space. A common choice is the Gaussian distribution. The authors propose using a different family of probability distributions: the q-exponential family. Extensive numerical simulations are provided.
weaknesses: I suspect that there is an exploration vs. exploitation trade-off in using heavy-tailed probability distributions. Is it possible to design a synthetic example to verify this? Additionally, once this is verified, there might be a way to switch probability distributions during training to balance the exploration-exploitation trade-off, which seems like an interesting direction to investigate.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: **Main questions (from the most to the least important):**

- Do all the policy neural networks have a similar number of parameters/weights across experiments for fairness of comparison? 
- Are the implementations mostly readily available in existing RL packages or other libraries that could be directly be used? I see that you provide some of the sampling mechanisms but these seem to be already well known in the statistics literature given their popularity. 
- Is GBMM only valid for 1 < q < 3? 
- l. 41: ‘Heavy-tailed distributions could be more preferable as they are more robust’, what do you mean by robustness here? Can you elaborate more? 
- Can you clarify the difference between the different Data settings in Fig. 8 (Medium-Expert, Medium, Medium-Replay)? 

**Additional questions:**
 
- Any comments about the so-called Lévy alpha-stable distribution? It seems it has been explored in the literature to model the SGD noise. 
See e.g. Simsekli et al. 2019. A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks. ICML 2019. 
- SAC encourages exploration. Would it be interesting to see if just changing the policy parameterization with a heavy tailed policy would already perform better than SAC? SAC seems to combine two effects: the parameterization of the policy itself and and the regularization to force the policy to be closer to a BG policy (why?)? 
- More of a possible future work direction comment: This work explores in isolation each one of the parameterized policies to showcase the advantages and shortcomings of each one of them and compare them. Can we combine them to get the best of each one of them and have more flexibility like in ensemble methods or bagging approaches in Machine Learning? For instance we might need a very exploratory policy in some settings/regions of the action space or much less once we progress in policy optimization beyond the ability of single distributions to concentrate throughout learning.
soundness: 3
strengths: - To the best of my knowledge, there is no such study in the literature and I believe it is useful for RL in practice. Proposing more alternatives to the Gaussian policy for continuous state action space settings is interesting.  
It would be nice to have the proposed parameterizations and their implementations integrated to existing RL packages to make them readily usable by practitioners. 

- The execution of the study is solid: experiments are methodologically and rigorously conducted, figures are professional and the presentation is pleasant, this effort is much appreciated for a practical paper.
summary: This paper conducts an empirical investigation of the use of the q-exponential family of policy parameterizations as an alternative to the commonly used Gaussian for continuous state action spaces in RL for policy optimization. In particular, these policies have been incorporated into existing Actor-Critic algorithms and experiments have been performed to compare the different policies to showcase the improved performance of some of them compared to the Gaussian policy in online and offline settings across several environments.
weaknesses: **Main Comments:**

- While the paper provides some insights, in terms of conclusions, it is hard to make a claim to extrapolate the findings beyond the interesting but expected fact that Gaussian policy might not always be the best choice and some other parameterizations might perform better. The paper does perform several experiments but  it is hard to derive actionable practical advice from 4 algorithms x 3 environments regarding which parameterization to be used in general. 
I think the paper could do better to showcase the advantages and shortcomings of each one of the policies, especially to demonstrate the intuitive points mentioned in the introduction regarding each one of them. While the environments tested are standard benchmarks in RL, I think it would be interesting to design and/or consider simple environments with different levels of exploration needed for instance to show the differences between the policies. From the viewpoint of the practitioner, can we come up with a procedure to decide which one to use rather than myopically testing each one of them an decide accordingly? Can we design an index performance of exploration hardness to be exploited? Little can be said regarding this from the paper. Of course, we cannot reasonably be confident to choose the right parameterization since this would highly depend on the setting at hand as the paper shows but even heuristics would be useful. In other (older) ML areas, such procedures seem to be widely used in practice (e.g. for model selection in classification …) 

- Other policy parameterizations than the Gaussian have been investigated in the literature as the paper acknowledges. Some additional relevant related work discussing the use of heavy tailed policies in RL and their benefits: 

Amrit Singh Bedi and Anjaly Parayil and Junyu Zhang and Mengdi Wang and Alec Koppel. ‘On the Sample Complexity and Metastability of Heavy-tailed Policy Search in Continuous Control’, JMLR 2024. 

S. Chakraborty et al., ‘Dealing with Sparse Rewards in Continuous Control Robotics via Heavy-Tailed Policy Optimization’, 2023 IEEE International Conference on Robotics and Automation. 

- The paper could comment about the overhead computational effort (if any) required when going for more ‘sophisticated’ policy parameterizations than the simple Gaussian, notably in terms of sampling mechanism complexity and number of samples requirements, for instance depending on the dimensionality of the problem. Any differences regarding efficiency depending on the dimensionality of the problem? 

- See questions below for clarifications. 

**Minor comments:** 
- l. 122-123: ‘In continuous action spaces, evaluating the log-partition function is generally intractable. Therefore, many researchers consider the Gaussian policy instead.’ I find this sentence a bit confusing. Is Gaussian policy only considered because BG policy cannot be used? In principle, one could consider any policy parameterization. For the continuous setting, it turns out that the Gaussian policy is the one of the simplest one can consider perhaps due to its omnipresence in statistics and parametric estimation as well as its widely available sampling procedure implementations.

---

confidence: 2
contribution: 2
initial_rating: 6
presentation: 3
questions: Is it possible to understand which characteristics of the environment makes ine distribution better than the other ?


It would be very useful for the user to get some prior information about which distributions could work well to avoid to search over all the distributions proposed in your work and finding the best performing one.
soundness: 3
strengths: I think that it is useful to have a paper that clearly states that the Gaussian parameterization might not be the best performing in practice.
summary: In continuous action RL many researches parameterize the policy as a mapping from states to std and mean of a Gaussian from which the actions are sampled.
When also the standard deviation of the Gaussian is learned, the authors argue that phenomenon like instabilities and lack of exploration can occurs. To this end, this paper explores empirically other continuous distribution to use to parameterize policies. In particular, it seems that heavier tails distributions are beneficial in Continuous Online Control experiments such as Acrobot and Cartpole and in Offline MuJoCo experiments.
weaknesses: The paper does not have a clear explanation or conclusive experiment about which parameterization should be used in which cases.
The main take away is that Gaussian parameterization should be avoided but it is not clear with which other distribution it should be replaced.

Meta-review: This paper proposes a general class of policy optimization algorithms for continuous action spaces which replaces the Gaussian by general q-exponential family. This paper provides a discussion on the choice of q value and their effectiveness, and conclude on the general advantage of using student t-distribution as policy. The problem is important but lack study in the field. All reviewers are convinced by the importance of such study and by the contribution made by this paper to the field. We thus recommend acceptance.

Decision: Accept (Poster)