Offline Reinforcement Learning via Tsallis Regularization
Authors: Lingwei Zhu
Matthew Kyle Schlegel
Han Wang
Martha White
Venue: Accepted by TMLR ()

Abstract: Offline reinforcement learning (RL) focuses on learning a good policy from a fixed dataset. The dataset is generated by an unknown behavior policy through interactions with the environment and contains only a subset of the state-action spaces. Standard off-policy algorithms often perform poorly in this setting, suffering from errorneously optimistic values incurred by the out-of-distribution (OOD) actions not present in the dataset. The optimisim cannot be corrected as no further interaction with the environment is possible.  Imposing divergence regularization and in-sample constraints are among the most popular methods to overcoming the issue by ensuring that the learned policy stays close to the behavior policy to minimize the occurrence of OOD actions.  This paper proposes Tsallis regularization for offline RL, which aligns the induced sparsemax policies to the in-sample constraint. Sparsemax interpolates existing methods utilizing hard-max and softmax policies, in that only a subset of actions contributes non-zero action probability as compared to softmax (all actions) and hard-max (single action). We leverage this property to model the behavior policy and show that under several assumptions the learned sparsemax policies may have sparsity-conditional KL divergence to the behavior policy, making Tsallis regularization especially suitable for the Behavior Cloning methods. We propose a novel actor-critic algorithm: Tsallis Advantage Weighted Actor-Critic (Tsallis AWAC) generalizing AWAC and analyze its performance in standard Mujoco environments. Our code is available at \url{https://github.com/lingweizhu/tsallis_regularization}.

Published in Transactions on Machine Learning Research (05/2024)
Offline Reinforcement Learning via Tsallis Regularization
Lingwei Zhu t
linggueid Qualberra..aabbba...cccc
Deportment of Computing Science, University of Alberta
Alberta Machine Intelligence Institute (Amit), Conada
Matthew Schlegel
mkkchteg@ruaberrt..cd
Department of Computing Science, University of Alberta Alberta Machine Intelligence Institute (Amii), Canada
Han Wang
han&@ualberta.cca
Deportment of Computing Science, University of Alberta Alberta Machine Intelligence Institute (Amii), Canada
Martha White
whiterm@hllerrta.co
Canada CIFAR AT Chair
Deportment of Computing Science, University of Alberta
Alberta Machine Intelligence Institute (Amii), Conoda
Reviewed on OpenReview: https: // openreview. net/ form?? id= HNNGE2DDDR
Abstract
Offline reinforcement learning (RL) focuses learning good policy from fixed dataset. The dataset is generated by an unknown behavior policy through interactions with the environment and contains only a subset of the state-action spaces. Standard off-policy algorithms often perform poorly in this setting, suffering from errorneously optimistic values incurred by the out-of-distributiin (OOD) actions not present in the dataset. The optimisim cannot be corrected as no further interaction with the environment is possible. Imposing divergence regularization and in-sample constraints among the most popular methods to overcoming the issue by ensuring that the learned policy stays close to the behavior policy to minimize the occurrence of OOD actions. This paper proposes Tsallis regularization for offline RL, which aligns the induced sparsemar: policies to the in-sample constraint. Sparsemax interpolates cxisting methods utilizing hard-max and softmax policics, in that only a subset of actions contributes non-zero action probability as compared softmax (all actions) and hard-max (single action). We leverage this property to model the behavior policy and show that under several assumptions the learned sparsemax policics may have sparsity-conditiooaal KL divergence to the behavior policy, making Tsallis regularization especially suitable for the Behavior Cloning methods. We propose novel actor-critic algorithm: Tsallis Advantage Weighted Actor-Critic (Tsallis AWAC) generalizing AWAC (Nair et al.) 2021) and analyze its performance in standard Mujoco environments. Our code available at htttps://gitii http:///gggeeeinntttsaaaaaaoo latllarguuuaaaaiioorr
1 Introduction
Reinforcement learning (RL) has achieved impressive successes in various domains through learning from online interactions with the environment. (Mnih et al.] 2015 Silver et al.] 2017 Andrychowicz et al. 2020). However, online RL is often less suited to real-world domains, especially when acting unconstrained in an environment can be expensive or dangerous. Offline RL instead addresses the problem of learning good
These authors contributed equally this work.

---

Published in Transactions on Machine Learning Research (05/2024)
policies completely from a given dataset generated following unknown policies. The goal of offiine RL is to learn policies which outperform-or or at least match-the the policies used to generate the dataset.
However, many of the difficulties of offline RL stem from not using online interaction. Standard off-policy RL algorithms tend to perform poorly, due to the well-known extrapolation error or
(OOD) action problem: improving the learned policy beyond the level of behavior policy requires estimating values of state-action pairs not present in the dataset. Optimistic estimates will bias the agent into favoring the absent actions in the policy improvement stage, leading to vicious loop (Fujimoto et al.] 2019: Kostrikov et al., 2022). Since no further interactions with the environment is allowed, the agent may be increasingly prone to choose actions with delusively high values It is worth noting the extrapolation problem does not occur in the tabular case but rather is the result of function approximators (Gulcehre et al. 2021 Dadashi et al. 2021
A popular branch of offline RL is behavior cloning (BC) (Pomerleau 1988), referred to as imitation based methods in (Xu et al.] (022). BC based methods enforce the learned policy to stay close to or reproduce
the behavior policy (Dadashi et al.
2021
Ghasemipour et al. 2021 Nair et al.] 2021 Siegel et al.] 2020
Wang et al.) 2020 Wu et al.
2022
2020 This is often achieved in two ways: (1) by using divergence
regularization D [1+ TDC
())
when updating policy to penalize large deviation from the learned policy
We to the behavior policy Tp
Brandfonbrener et al. 2021 Jaques et al. 2020 Kostrikov et al.) 2021 Wu
let al.
2020 Osa et. al.
2023). , usually D is chosen to be the KI. divergence but other divergences such as
MMD (Kumar et al. 2019), Fisher's divergence (Kostrikov et al. 2021) or Shannon entropy (Xiao et al.
2023
have also been used; (2) imposing an in-sample constraint to the updates (Fujimoto et al. 2019
Kostrikov et al., 2022 Xiao et al., 2023) where the target hard max operator max, Q(s,a) in Q-learning is
replaced to the in-sample maximum (axa:xp Q(s, a). This scheme has been recently extended to the
in-sample log-sun-exp In Eamp(els)>> eQ(s,a) (Xiao et al. 2023).
In this paper, we propose use general but less studied class of regularizers that interpolates softmax and hard-max--the Tsallis regularizers, specifically, Tsallis entropy and Tsallis KL divergence- as the choice of D. While Tsallis entropy and Tsallis KL divergence have recently been investigated in online RL (Lee) et al. 2018 2020: Pacchiano et al.] 2021 Zhan et al. 2023 Zhu et al., 2023), they have never been utilized in the offline RL context. Tsallis entropy (resp. Tsallis KL) is a strict generalization of Shannon entropy (resp. KL). Different from the softmax policy induced by Shannon entropy and KL that has full support (Azar et al. 2012, Kozuno et al., 2019 Vieillard et al. 2020), Tsallis regularization induces the sparsemax policy that truncates actions with low values, i.e. setting their probability to zero. We link action truncation to the in-sample constraint (als)) > 0, arriving at the assumption that the behavior policy is within the sparsemax policy class and collects in the offline dataset a subset of actions with high values. Intuitively, the assumption suggests to learn a policy that shrinks the support of the behavior policy, which renders Tsallis regularization especially suited to BC or imitation-based methods. We formalize this intuition in Section 1 by showing that the KL divergence between TD and the learned policy Tt, all within the sparsemax class, may be upper bounded depending on the Tsallis entropic index controlling the sparsity.
By combining in-sample constraint and Tsallis regularization, we propose two actor-critie algorithms: Tsallis advantage weighted actor-critic (Tsallis AWAC) based Tsallis KT. divergence; and Tsallis In-sample Actor-Critic (Tsallis InAC), based on Tsallis entropy. Both methods extend their base algorithms to the >> 1 domain. With the introduced sparsity by q-exp, Tsallis AWAC is among the best performers among the compared methods, while Tsallis InAC is less stable and only competes favorably for expert datasets.
2 Background
We model our problem as Markov decision process (MDP) expressed by the tuple (S,A,P,,,,)) S is the set of states, AA is the set of actions. P(-Is, denotes transition probability over the state space given state-action pair (s,a), and r(s, a) defines the reward associaled with that transition. E [0, 1) is the discount factor. A policy 2(-1s) is a mapping from the state space to distributions over actions. The state-action value function starting from (s, a) following policy π is defined as (s. = E, 12> ( at)s0 = -, s,a0 = a]. In this paper we consider entropy- regularzee formulation E,, [Q, a) [(T(-Is))], where n(T(-Is)) E || is the regularizer convex in л. Popular choices for Ω include the negative Shannon entropy - (=(+1)) =
2

---

Published in Transactions on Machine Learning Research (05/2024)
Entropy m
5.0
Taalis KL m iin
2.5
0.5
0.0
-2.5
-5.0
0.0
2(2,1)
[2.75 1), 2(325
Figure 1: (From left to right) g-logarithm, g-exponential, Tsallis entropy and Tsallis KL divergence Gaussian policies. Note that q= 0 is shown here only for an illustration purpose. We consider > this paper for theoretically sound regularizers. When q = 1, the above functions recover their standard counterparts.
En (als)) which encourages the policy to be uniform with weighting the effect. KT divergence DRL (T(15) ll ((-)s)) [=))==)===> = >. (alss In s(ale]
penalizing large deviation from the reference policy is another
plass]
popular choice AAzr al. 2012 Vieilard ct al. 2020: Chan et al. 2022. Zhu & Matsubaral 2023
It is worth noting that offline RL problems often have continuous action spaces, while many existing methods were derived based on a discrete footing. Approximation is usually necessary, e-g., in evaluating policies (Fujimoto et al. 2019 Xiao et al.) 2023). Similar to (Xiao et al. 2023), our method is positioned in the cutropy-regularizer literature and established based on the discrete action setting (Vieillard et al. 2020 When applied to continuous action problems, our method also necessitates approximation in policy evaluation, as detailed in Section While such approximation may affect the performance, its use is justified in the following sense: (1) evaluating the continuous policy exactly is generally intractable; (2) the approximate policy is still an exp/q-exp function which retains some desired properties. Developing theories in the continuous action setting is left as an interesting future work.
2.1 Tsallis Regularization and Sparsemax Policies
We consider a broad class of less studied entropic regularizers as Ω: Tsallis entropy and Tsallis KL divergence. We can define these regularizers using g-logarithm in similar manner to the standard logarithm. For q E R4, we define q-logarithm as Ing -=-1__ and its unique inverse function g-xxponential exp, q-1
[1 (1- where {++ = max( 0}, see Figure for an illustration. As q gets larger, exp, z becomes more flat (second plot). Notice that exp, is only invertible when r > - On the other hand, Ing x is more peaked than the standard logarithm for input x>> 1 and more flat otherwise (Ding & Vishwanathan, 2010 Therefore, for 0 < π < 1, the Tsallis entropy component - In, π is more flat than the Shannon entropy (third plot). Note that when S 0, Tsallis entropy becomes a convex function and Tsallis KL a concave function, therefore they are not valid regularizers (Geist et al. 2019). We consider exclusively q > 0 this paper, and q = 0 is shown in Figure onnly for an illustration purpose. We can define the Tsallis entropy using R. When y - 1, the q-logarithm (resp. g-exponential) recovers the standard logarithm (resp. exponential) and hence Tsallis entropy degenerates to Shannon entropy. When = co, the regularizer vanishes. When k = 2. 2, we arrive at the most important
g-logarithm (Tallis 2009):
-- E. Tta ((),)
non-trivial case: Tsallis sparse entropy S2(T(-1s)) = E. (als)) (1 (als)) Chow et al. 2018 Lee et al.
2018). max, Σ. (a a) + -S2(=()1)) attains its maximum at EmEK())
Q(s.a)
Q(a.2
+
The name sparse entropy comes from the fact that the regularizer leads to sparse support of the resulting sparsemar policy (Blondel et al.) 2020 Martins & Astudillo 2016). We compare sparsemax against two other commonly used policies argmax and softmax in Figure]
For * 1,2, x0, the resulting policy does not have a closed-form expression. But we can apply Taylor's first-order expansion on the resulting g-exp function to obtain an approximate policy form: T[als) =
E Eackin) Q(a.a)-
$ (hu et al. 2023). In Table we provide a comparison on related literature to further clarify the notations used this paper. We call this policy sporsemar for all g R2, (1), since they truncate actions by definition of q-exp. K(s) is the set. of
exp, ((a.aa v ((a..) )), th Q(a.))
K(a)
3

---

Published in Transactions on Machine Learning Research (05/2024)
0,00 1J
argmax
softmax
sparsemax
[0. 1,0
[1,0,0]
((s)
Figure 2: (Left) Comparison between argmax, softmax and sparsemax on the probability simplex. Argmax produces a deterministic policy residing on the vertices, while a softmax policy lies inside the simplex. By contrast, sparsemax policy lives on the border. (Right) Sparsemax acting on a Gaussian policy by truncating actions with value below the threshold vh. Actions with value larger than & are collected in the set K(s).
highest-value actions satisfying 1 1+i i O(s.ama Ej _j=1 Q(s,a() with agy denotes the action with j-th largest
value. Zhu et al. (2023) recently proved that Tsallis KL divergence regularization leads to a similar policy functional (aass) p(als) exp, (((..)) - w/ ((x.)) )). hence also a member the sparsemax policy.
2.2 Offline Reinforcement Learning
We consider the problem of offline RL, where the agent cannot interact with the environment and instead learn from a fixed dataset D = {(s, a, a,r,s") collected by some unknown behavior policy Tp. The datasct D typically contains only a small subset of the S x A space. Standard off-policy algorithms are known to suffer from extrapolation error referring due generalization capibility of function approximators. Unlike online RL, where the OOD actions can lead to more sampling around the low sample region and eventually correction of the values, in offiine RL the correction is impossible since no further interaction with the environment is allowed.
erroneously optimistic acation values for out-of-distribution actions
We position our paper in the popular BC/imitationn-asse literature, where the goal of learning is to reproduce the near-expert behavior policy To (Dadashi et al.) 2021 Fujimoto et al., 2019 Fujimoto & Gu 2021 Nair et al. 2021). In this conext, explicit or implicit constraints are usually used to enforce the proximity between learned policics and TD to minimize the effoct of OOD actions. Explicit constraints can be implemented via density models Ghaseemipour et al.) 20211 Wu et al. 2022) or in-sample constraints (Fujimoto et al. 2019 Kostrikov et al. 2022 Xiao et al. 2023) avoid querying OOD actions. Implicit constraints are often achieved via a divergence regularizer D that is added to the objective of policy improvement: where D is typically chosen as KL divergence. The regularization leads to the policy form ([ass) (als)) exp (r-IQ(s, a)) where the learned policy conditions the support of the bchavior policy and weighted by exponential of action values (advantage) (Peng et al. 2020: Siegel el al. 2020 Nair et al. 2021). However, as shown by (Rudner el. al. 2021), KL regularization can lead to pathologies of the learned policies such as vanishing variances and exploding gradients. In this paper, we propose to use the Tsallis KI. regularization which generalizes KL divergence and offers more ficxibility over the standard KL choice (e.g., AWAC). However, investigation of pathologies is beyond the scope and we leave it future work.
max, E.-D [Bn- [Q(s, a)] - [
4

---

Published in Transactions on Machine Learning Research (05/2024)
3 Matching In-sample Constraint By Sparsemax Truncation
To alleviate the OOD error, Fujimoto et al. 2019 proposed the in-sample Bellman optimality equation that modilies the Q-learning updale targel to only for the actions present in the dataset:
(8,7((,0)) = r(s, a) + ( P(Hea))
aafmp (a))s)) (6.
max
(1)
This scheme was extended to the distributional RL setting by Kostrikov al. 2022 Recently, Xiao el. al. 2023 proposed the in-sample softmax Bellman optimality equation:
Qs - r(s,a) +7BB P(yH)))
TIn Σ exp (T (o,T,, a'stp(e')s')20
(2)
In-sample softmax policy takes the form (a)t) x TD(aas) exp
(emp(saa In (aaas))) the subtracted
term In uD is to make sure not tightly follow the behavior policy when it is not good.
Let us consider regularization with (=((s))))). This change of regularizer embodies fundamentally different assumption: we can link the fact that the dataset contains only subset of actions to sparsemax behavior policies. Indeed, if we assume the dataset was generated by the behavior policy in a manner such that an action is sampled with probability proportional to its value (Kostrikov et al., 2021), then we can model the behavior policy by a sparsemax policy. This assumption mild since softmax policy a member of sparsemax when q 1. To formalize the idea, let us assume the behavior policy is within the sparsemax policy class induced by Tsallis entropy (als) expy (-=
Eeekp(s) ( = 1, where to is an unknown coefficient and Kp(s) denotes the set of actions present in the dataset. Therefore, we can replace the in-sample constraint a (a(a))) > 0 to the truncation criterion a E KD(s):
(Rpp!sa))
( (s. = a() + Y. T ~((pn)
max Σ
(Qu +
a'EKo(e)
T+1ss((ass) exp, (Qmmp(s.a)
T4+1.0 (a|s)= 1.
(3)
(EK
This replacement hints at two potential benefits of applying in-sample Tsallis regularization: (i) suppose the action values are fixed, then a sparsemax policy could extract a new subset of allowable actions from Kp, and this procedure could continue until there is only the highest-valued action in the set. (ii) Sparsemax policies generated recursively from (i) may have finite KL divergence, which in turn implies that the KL between the n, and TD may be bounded. We discuss (i) here and leave (ii) to next section.
Sparsemax interpolates softmax and hard-max. Continuing the discussion of Eq. (3), let us define (s) := Ks(s) as our starting point. Then during learning, a new subset of allowable actions Ke,((1 depending g and iteration & can be extracted from (s). Recursively, every Krt1, contains only a subset of actions from the last set Krg, or mathematically speaking, Ku,g ż K1,g 2 K2 > ... 2 Ke,g 2 K where we used ž B denote A a subset. of B. To be more concrete, let us consider q = 2 where the q- g-maximum has an analytic solution. From Section 2.1 it clear that:
QLIT (2 (QQrop55)))
2
(s,1) r(s,a) (EswP()a))
Σ
+
(4)
2
T
e'EK.a(o)
Notice the constraint under the summation was replaced from Ks(s) in Eq. (3) to a' (s), due to the aforementioned recursive extraction. However, it is worth noting that the recursive extraction may hold true as action values change during updates.
5

---

Published in Transactions on Machine Learning Research (05/2024)
The above scheme regularizes both policy improvement and policy evaluation. Another possibility is to regularize only policy improvement by choosing Tsallis KT. divergence as the regularizer. We use the following policy iteration schemes to help clarify the difference:
(a) T++1 = arg max Ex [Qi(s, a) + + [[a|s)]
(6) [ arg max, Ex (Q((,,a) - -T In, m[(ass C1+1 r(s, a) + PEp [Q + T Ing Te+ (a'l6'),
{ [0m===((ss) r(s, + 7Bpp, 1Qe(s',
s(als)
(5)
where (a) corresponds to Tsallis entropy + regularize both and (b) Tsallis KL + policy improvement only. It is worth noting that both schemes fall into the mirror descent policy iteration framework (Vieillard ct al. 2020 Eq)( In the offline RL context, the first scheme (regularizing both policy improvement and eva ua Lion) is discussed in (Wu et al. 2020); and the second scheme is the basis of advantage weighted actor-critic (AWAC) (Nair et al. (021)) We present in Section implementation for both schemes. Tsallis KL chosen for the second scheme with the following reason: we notice that augmenting the value function with divergence often results unstable learning possibly due to the existence of policy ratios.
4 In-sample Sparsemax Policies Have q-bounded KL Divergence
A sparsemax policy can be expressed as q-exponential policy. Therefore, existing theoretical results on g-statistics (Yamano 2002 Ding & Vishwanathan, 2010)) may provide clue to characterizing the similarity between two sparsemax policies. We are especially interested in characterizing the KL divergence between learned policy and the behavior policy, which is particularly important for BC/imitation-
methods (Fjiioooo et al. 2019 Fujimoto & Gul 2021 Wu et al. 2020 2022b. However, bounding potentially unbounded KL divergence in general a difficult task, if not impossible (McAllester & Stratos 2020). Instead, we do not seek to provide a general upper bound but rather aim at quantilying the dependeney of KL divergence on the intermediate variable q.
The following thcorem states that the upper bound on KL divergence between cearned/bchaving policies can be flexibly adjusted by changing q. Therefore, our prior knowledge regarding the quality of the dataset may impact our choice of q: when we are confident that sufficient near-expert trajectories exist, it may be preferable to choose a large q to encourage more similarity to nD; on the other hand, choosing q = may be more robust to a non-expert dataset.
Theorem 1. Suppose the dalasel D is generaled by sparsemar behavior policies such that for each state there is at most one behavior policy of arbitrary entropic index q. Let (t.g(s) I KD(s) denote the set of allowable actions of the learned sporsemax policy at t-th iteration. Then the following q q-conditional upper bound of KL divergence can be derived for sparsemax policies ae and nD:
DKs ( [ - (Ken()) ID
if S Tp,
1 <<9 80
(6)
00.
otherwise
9-3 where II..1 := =2 (als) (((as)
1 + Tb
(als)
T (als).
Proof. See Appendix for the proof.
We may interpret the term (e(ass) as a bascline having a fixed power o. (als). When q= 1, the upper bound may be oc. On the other hand, when q - 20, the learned policy n. approaches the arg max (Lee et al. 2020), and the upper bound approaches zero. Intuitively, when q = 00, (s)(s) has only action, which corresponds to assuming the behavior policy is an arg max and it is identified by Tı(als) = 1, therefore the KL is zero when the their supports agree, or do otherwise.
We can replace the reference policy TD in equation 0 to a learned policy कi-1 if the assumption of fixed action values holds, since n, N πT--- Therefore, the bound provides a mcans to understanding the distance between sparsemax policies. For g = (the in-sample softmax case) the bound is not useful and simply states the KT. divergence may be unbounded. On the other hand, choosing any q > 1 brings an upper

---

Published in Transactions on Machine Learning Research (05/2024)
bound of most (5)). When q = 2, the in-sample sparsemax has KL divergence to the behavior policy bounded by (x) (T)((s)) ( + 2). In general, there is a trade-off between œ and (Kt,(()) (s) tends to collect less actions when + 20. Again, we should note that the theorem holds only when K2 ž K1,g = x2.9= ... ž= kg.g as per the discussion for equation I It should also be noted that the bound is only applicable to discrete action spaces. Moreover, slaying close to the behavior policy may not always be preferable, especially when the dataset contains too many poor trajectorics.
5 In-sample Sparsemax Actor-Critic
We propose Tsallis Advantage Weighted Actor-Critic (Tsallis AWAC) based on the Tsallis KL divergence regularization. We also discuss another new algorithm Tsallis In-Sample Actor-Critie (Tsallis InAC) based on Tsallis entropy. However, as will be analyzed shortly, Tsallis TnAC may not necessarily introduce sparsity and does not enjoy straightforward policy evaluation. Nonetheless, we include it in analysis and experiments for completeness.
To fulfill the in-sample constraint we leverage only those actions present in the offline dataset but not sampled actions to learn the actor. The architecture is similar to soft actor-critic (Haarnoja et al. 2018): let 0 denote the parametrization of the critic Qo, the actor Ts and Ç the state value function Vg. Another network is trained by maximum likelihood (s [log Tw (a[s)] to imitate the behavior policy Tw mD-
Before deriving the Tsallis AWAC loss functions, recall the actor loss for AWAC:
Lactor (6) = - -E exp (o((s a) -V((s) In (ala(l
which is a common actor loss function for advantage regression methods (Wang et al. 2020 Siegel
let al. 2020 Nair et al.j 2021 Xu et al. 2023 Garg et al. 2023). It can also be derived from the (b) scheme in Eq. (5) by replacing Tsallis KL standard KL. Intuitively, minimizing C (Ø) results in maximizing the loeliikl of actions in the dataset
D and implicitly minimizing the likelihood for these not in the dataset, as the probabilities of all actions should sum to one. The degree of maximization is controlled by the exponential advantage of an action exp (Q(s.a)-V((s)) when an action has high Qo Vc(+), the policy gets updated increase its probability more, which may be problematic when the value estimates are poor. By contrast, in-sample softmax (Xiao al., 2023) compensates for this fact by subtracting In aD term inside the exponential. For the proposed Tsallis methods, crucial difference lics in how this exponential advantage function is modified.
Tsallis AWAC (TAWAC). This scheme regularizes only the policy improvement step as can be seen from (als) x (al(as) exp, ( (s), Repeating the KL Eq. (5) Given Qo(s,a), we write the policy as "Qo.a loss step, we have:
TAWAC
DKL. (
(-1s) (-(+)))
Ron sacc Co.r
(
(-1e) [In & @ (als) - In [a]s))
Em
exp,
Q0(s,a) vb (oo(s.))
(2 T
(In + #2u. ln In IImm((as)))
The normalization function s poses challenge to contimuous-action problems. Inspired by (Xiao al. 2023) propose to replace it with the state value function Vç. We write the loss functions as:
(Qo(s,a) Ve(+)) In Inmo(als)
IANAC (())
(...,) exp,
7 (r+>Ve(s')) - a)(s)a)) ( B(E.
critic (0)
TAWAC
(8)
TANAC
baseeliee (S) EEnDD [VV(()) Qo(s, a))
Remark. Comparing Mater (Ø) and Lavee (Ø), it is clear that even though Tsallis AWAC is only different AWAC in the g-exponential function, it brings sparsity to the policy. To see this, we can write out actor (+)
7

---

Published in Transactions on Machine Learning Research (05/2024)
the following:
[
[ 1 + (q (1-1)
(oo(3, V3 [oo(5,0)-K)]
((++0--))
(0-()-1))
( (Qllsaa-VVic --
- Ve(s)
In (als)
T
7
Since the root does not affect the sign, can be seen that actions with values Qo(s.e)
9-1 will be truncated: the actor loss for these actions becomes zero and TAWAC does not maximize its likelihood. Another interesting observation is that by setting q 2, Caetor (ő) recovers the sparse Q-learning objective in (Xu et al. 2023). which was derived from the a-divergence perspective. However, our loss is more general since for all g > 0 (6) has a truncation effect. By contrast, all exponential advantage weighted methods do not have sparsity and can maximize likelihood.
Tsallis In-Sample Actor-Critic (Tsallis InAC). Different from Tsallis AWAC that has clean policy form, Tsallis entropy regularization induces a much more complicated policy (see Appendix D
(als) = (alaas) expy (= + in <b(als) ) Qo.
1 9-1 + (q (9- (0e-1)-00(s,00 a) Ing to(als)
)
TINAC
It can be seen that the Tsallis InAC policy cannot contain everything inside the g-exp function. This can lead to numerical issues and more importantly, the loss of sparsity. We found that Tsallis InAC in general underperforms Tsallis AWAC. Two potential reasons are identified. Firstly, the Tsallis InAC actor loss in general does not lead sparsity, since there Qo(s.a) swietel term outside the g-exponential function.
This term very sensitive to the sign of Q and magnifies its value by the term In2 1 which is typically large. Directly from this, the second point is that Tsallis InAC may be drastic in dociding good /badd actions simply by its sign and can be minimizing likelihood of bad actions. This stands in sheer contrast to InAC and Tsallis AWAC, which do not explicitly minimize likelihood. The drastic behavior of Tsallis InAC in general leads to undesired performance, but with exception for expert datasets, see Appendix D
6 Experiments
Below, evaluate Tsallis AWAC against several oflline reinforcement learning methods. The goal of the experiments is to find when is teh proposed method best applied in offline RL, as well as gain insight into the effect the t and q parameters (especially as we enter regimes where there aren't closed form expressions for the policies). Finally, we evaluate if the upper bound holds the continuous action setting.
Domain Details. We compare the proposed method against number of existing algorithms on standard benchmark D4RL environments (Fu et al. 2020). Specifically, use three datascts from the Mujoco suite in D4RL. Trajectories in the offline datasets expert/ expert/medium medium replay reflects the level of the trained agent used to collect the transitions. The expert dataset contains trajectories collected by fully trained SAC agent; the medium dataset contains 1M samples generated from an partially-trained SAC policy. Medium-expert combines the trajectories of the expert and the medium. Medium-replay consists of samples the replay buffer during training until the policy reaches the medium level of performance.
collected by SAC agent. The naming expert/mediun /eddium
Compared algorithms. We compare against number of existing algorithms: InAC: in-sample softmax actor-critic (iiao et al.] 2023). It is worth noting InAC can be seen as the specical case q 1 of Tsallis InAC. TD3 + BC (Fujimoto & Gu 2021 augments the policy improvement step of TD3 with an additional behavior cloning (BC) term (1()) a²a as indicated by (Xiao et al. 2023). This term can be seen as a KL regularization under the Gaussian policy parametrization. IQL: implicit Q-learning (Kostrikov et al. 2022 employed in-sample hard max Eq. [ AWAC: advantage-weighted actor-critic (Nair et al. 2021). ). can be as special case of Tsallis AWAC when g = For all the compared algorithms except InAC, we followed the published settings of the compared agents. Implementation details can be found in Appendix
8

---

Published in Transactions on Machine Learning Research (05/2024)
Ant
HalfCheetah
Hopper
Walker 2D
Expert
1.25
1.0
1.00
10
0.8
Normalizat
0.75
0.8
Retarn
0.50
0.6
2.
0.4
25
50
75
0
25
50
75
50
75
25
75
Medium Expert
1.25
0.8
1.0
1.00
Normalized
0,75
0.6
Return
0,50
0.2
0.25
0.0
Medium
1.0
Replay
0.8
40
Normalized
35
0.6
0.6
0.6
Return
30
0.4
0.4
0.4
0.2
Steps (Ie5)
Steps (1e5)
Steps (Ie5)
Steps (1e5)
Tsallis AWAC
AWAC
Figure 3: Comparison between Tsallis AWAC and AWAC. Normalized policy evaluation scores Mujoco Medium Replay dataset over one million steps. Results are the average over 5 runs with ribbon denoting the standard error. y-axis: scores; r-axis: number of iterations.
Dataset
BC
UAWAC
AWAC
TD3BC
IQL
InAC
Tkalis InAC
Tsalis AWAC
walker2d-E E
106.7
1.5
110.2 0.03
79.4 1.0
110.6 +
108.7
113.2
walker2dl ME
96.
1.
89.7 0.6
96.0 0.4
109.0
0.80
101.1 0.02
walker2d-MRR
23.6
+
17.2
66.3 0.5
69.8
0.74
halfcheetah-E
91.8
9..940.6
1..0
92.1 0.03
91.5
93.6
93.1 13.12000
92.7 = 0.00
halfcheetah-MM
44.0
42.7
38.5
54.1 0..14...
83.4 0.5
$3.5
50.2
84.5 = 0.01
halkchoctah-MMR
37.6
35.9
44.8 0.011
35.0 0.2
45.0
0.03
44.3 0.02
37.9
44.9 0.00
hopper-E.
107.7 0.7
110.5 0.5
45.0
78.9 0.9
89.4 0.7
103.4 0.4
110.3 0.00
109.9 0.00
53.9++4. 4.7
44.9 8.1
2..3
50.8 0.3
0.3
61.8 + 1.0
93.8 0.7
63.4 0.03
101. 0.02
hopper-MR
16.6 + 4.8
25.3 + 1.7
71.2
20.3 +: 0.2
60.3 +: 0.5
92.1 +
26.7 + 0.00
86.4 + 0.02
ant-E
100.6 0.9
93.3 1.0
125.4 128.4 0.5
125.2 0.01
aut-ME
07.8 1.6
88.7 0.9
121.0 0.6
120.9 0.6
114.2 + 0.00
130.2 0.01
aut-MR
65.0 1.2
5..2
89.3 1 0.4
8.6
68.7 - 0.00 0.00
96.0 0.01
Table 1: Final performance compared algorithms, expert; ME: medium expert; MR: medium replay.
Scores are reproduced from (Xiao et al. (023) and (Lyu et al.) 2022). Tsallis AWAC and InAC perform favorably across various datasets, while Tsallis TnAC is mostly competitive in expert-level datasets only. Notice that both Tsallis InAC and Tsallis AWAC tend to be stable by having negligible standard deviation.
6.1 Comparison Against the Existing Methods
Since Tsallis AWAC generalizes AWAC, first compare the generalizations in Figure 3 and leave the comparison against all methods to Figure q and Table 1
Tsallis AWAC against AWAC. From Figure it is visible Tsallis AWAC outperforms AWAC by a large margin on Expert and Medium- Medium-Expeer datasets and the performance remains stable across all datasets, The poor performance of AWAC has been discussed a lot in the literature (Xiao et al.) 2023 Xu al. 2023): the exponential term in Seeee can cause unstable gradients and is also more vulnerable to hyperparameter choices. On the other hand, the favorable performance of Tsallis AWAC confirms the discussion given by

---

Published in Transactions on Machine Learning Research (05/2024)
Ant
HalfCheetah
Hopper
Walker 2D
Expert
1.0
10
Normalizad
1.0
Retarn
0.8
0.4
0.6
0.2
0.2
0.4
25
50
75
0
25
50
75
25
50
75
25
75
Medium Expert
1.2
0.8
0.8
Normalized
1.0
Return
0.6
0.6
0.4
Medium
1.0
0.8
Replay
0.9
0
0.8
0.6
Normalized
Retaru
0.6
0 30-
0.5
0
Steps (1e5)
Steps (1e5)
Steps (Ie5)
Steps (Ie5)
Tsallis InAC
Tsallis AWAC
TD3BC
IQL
Figure 4: Additional comparison against TD3BC and Implicit Q-learning. Normalized policy evaluation scores over one million steps and are the average over 5 runs, with ribbon denoting the standard error. y-axis: scores; w-axis: number of iterations.
the remark after equation 7 the sparsity-inducing learning objective Lactor ( (Ø) is more robust against both caansition/suoptiiaan noises and numerical errors than the exponential function (Xu et al. 2023
Against all methods. From Figure 4 it is visible that Tsallis InAC and TD3BC are the best performers expert level datasets-an in terms of convergence speed and the final score. By the analysis for Tsallis InAC, the term In xp may dominate the actor and render Tsallis InAC behave like a BC method even without an explicit BC term. This is especially apparent on Half eeeeeeeeeheeepertt and Hopper-expert where Tsallis InAC learned even faster than TD3BC. Both TD3BC and Tsallis InAC drastically degrade for Medium Expert and Medium Replay datasets. Tsallis AWAC outperforms IQL on almost all non-expert datasets, which matches the observation of (Xiao et al. 2023 IQL may perform poorly when the data distribution is skewed towards smbaptimal actions in some states, pulling down the expectile regression targets.
To confirm our conclusion, we show in Table I the final performance of all the compared algorithms. We also add behavior cloning (BC) and Uncertainty AWAC (UAWAC) (Wu et al. 2021) for further comparison. Scores of the BC and UAWAC are quoted from (Lyu et al. 2022). is visible that Tsallis AWAC and TnAC are the top performers across datasets, with Tsallis AWAC achieving the best scores on 8 out of 12 datasets. This result encouraging and lestifies to the power of sparsity brought by the g-exp formulation, since Tsallis AWAC differs from other AWAC-based methods mostly in the exponential function. By contrast, Tsallis InAC is only competitive expert-level datasets. Upon examining Table 1 it is visible that in medium datasets BC, TD3BC and Tsallis InAC perform poorly, which echoes the findings in (Kumar et al. (022). On the other hand, Tsallis AWAC performs stably across datasets thanks to the introduced sparsity. Investigating Tsallis AWAC beyond the current setting is therefore an interesting future direction.
6.2 The Importance of and
Entropic index q and regularization coefficient determine the degree of sparsity. Therefore, they are related to performance and the y-conditional distance to the behavior policy shown by equation 0 We evaluate
10

---

Published in Transactions on Machine Learning Research (05/2024)
Hopper Expert
Walker2d - Expert
0.01
1
0
0
01
0
33
0
2
1.04
1.02
0.00
1.00
10
0.75
10
T
t
T
0.01
0.1
0.33
T 0.5
1.0
Hopper
1.0
1.0
Normuttire
Herter
04
0.8
06
20
40
90
40)
90
90
90
20
80
Walker 2D
1.0
1.0
Normultind
Rerorn
Stops
Stops
T-InAC T-AWAC(Q=2) T-AWAC(Q=3) T-AWAC(Q=S) T-AWAC(Q=10)
Figure 5: (Upper) Heatmap of average normalized score of Tsallis AWAC with different g. evaluated over first 500K steps of training. (Lower) Normalized score learning curves of Tsallis AWAC with different q.
different (q, r) combinations of Tsallis AWAC in Figure 5 Performance is evaluated after 5 x 10s steps of training. It is visible from the color maps that for the first half of learning, Tsallis AWAC on both environments exhibited preference to larger q: they tend to learn quicker and are relatively insensitive to t; while smaller such as = 2 prefers larger т. Consider fixed T, by definition of the Tsallis AWAC policy
(p(als) (Qo(s,))V<(s)) <a(as)) 1 + (q 1) VV(())
the difference Qo a) Ve(s) becomes less significant with under power function with larger power (imagine softmax with a large r) and the truncation threshold - Comparing q = 2 where exp, is linear in its argument, higher 7 Iruncates more, especially for actions that Qa(s, a) (s) < 0. Let us in turn consider fixed Larger T results in Kr Similar to the softmax case (Haarnoja et al. 2017). it is advocated that choosing a larger at the beginning and gradually annealed towards zero. Therelore, when the dataset contains sufficient (near) expert trajectories, choosing large 7q, may accelerate learning the early stage. In Figure it is visible that on the expert level environments all Tsallis agents seem be robust and converged to policies of similar level.
--I-
6.3
KL divergence to the Behavior Policy
Though higher similarity to the behavior policy does not necessarily imply better performance, we quanti- tatively evaluate it in support of our theory Eq. (6) We plot in Figure DKr (+)(s))) || TT(-Is)) for Tsallis AWAC with different q and Tsallis InAC throughout training, Since the KL divergence (1)) [In (a)s) In (a]ss)) and The is imitating TD, we replace the sampling part to ~ Tp(+ls) to allow for random sampling actions from the dataset to compute the log-policy difference. Though in the contimuuus action setting Ke,y uncountable and the upper bound on KL is no longer valid, we can empirically investigate On Hopper-expert Tsallis AWAC for all converged to 0, and Tsallis InAC remains stable around 1; different q did not seem affect the KL divergence. On the other hand, on Walker2d-expers clear stratification displayed: larger q resulted in lower divergence value. Theoretically, the RHS of
be written
11

---

Published in Transactions on Machine Learning Research (05/2024)
Hopper Expert
Walker2d Expert
1.10
1.13
Average Normalized
1.08
1.12
Returns
1.09
1.00
t
1-InAC- - T-AWAC(Q-2) T-AWAC(Q-3) T-AWAC(Q-5)
T-AWAC((=10)
Figure 6: Sensitivity of Tsallss AWAC to Normalized scores averaged the final 500k steps of training.
Hopper- - Expert
Walker2d Expert
1.50
1.50
1.25
1.00-
1.00
Divergence
KL
0.50-
0.25
0.00-
0.25
0.25
20
60
80
0.00
0
20
40
80
steps (1e5)
steps (Ie5)
T-InAC- T-AWAC(Q=2) T-AWAC(Q=3)
T-AWAC(Q=5)
T-AWAC((10)
Figure 7: DKz (-1s) [[]((())))) throughout training for different with best 7.
Eq. (6 approaches [s) ([([ss) - -- as gets larger, which is indeed tighter than q = 2. Therefore, the upperbound may still be in effect even in the continuous action setting.
7 Limitations of The Proposed Method
We recognize that the paper has linitations. Firstly, the upper bound in Theorem only q- conditional, and it necessitated the assumption that the ordering between action values remained unchanged. Deriving a generally applicable upper bound on KL divergence is difficult, if not impossible (McAllester & Stratos 2020). Utilizing other tools instead of q-statistics for derivation is an interesting future direction.
Secondly, as seen in Section 5 Tsallis InAC induced by Tsallis entropy had a very complicated policy form that failed to contain everything inside the q-exp function. As result, Tsallis InAC could have numerical issues and lose the sparsity by q-exp, as compared to Tsallis AWAC. The experimental results confirmed that Tsallis InAC indoed performed poorly especially non-expert datasets. How to leverage q-statistics literature to address these problems remains an open question.
8 Discussion and Conclusion
Tsallis regularizers have been less popular in RL due to the action truncation of its regularized optimal policy. The truncation often leads to underperformance in online problems resulting from limited exploration. This paper is the first work that introduces Tsallis regularizers to offline RL, where no exploration is required. Tsallis regularizers bring close two popular offline RL methods avoiding producing crronesouly optimistic distr--distribttion actions: divergence regularization and in-sample constraint. Tsallis regularization induces sparsemax policies that truncate actions with low action values, which exploited to link to the fact. that.
12

---

Published in Transactions on Machine Learning Research (05/2024)
offiine datasets contain only subset of actions: we assumed the dataset was generated by sparsemax behavior policy. As such, the in-sample constraint can be replaced by the truncation criterion. We showed two interesting facts given the assumption: (1) sparsemax policies interpolate hardmax and softmax, and when action values are fixed, consecutive sparsemax policies are within or equal to the support of the last sparsemax policy. (2) the KL, divergence bet ween a two sparsemax policies has a q-conditional upper bound.
We proposed Tsallis Advantage- Advantage-Weighhted Actor-Critic (Tsallis AWAC) that generalizes AWAC to the domain of q. AWAC is important offline RL algorithm: its actor loss has been used in many state-of-the-art methods and can be derived from the perspective of KL regularization. This generalization is non-trivial: unlike AWAC that only considers maximizing log- ligeeikd of actions in the datasct, Tsallis AWAC introduced sparsity: it can set the loss for bad actions to zero thanks to property of the g-exponential formulation. Sparsity has been very recently investigated to be crucial for superior performance. Leveraging the properties of g-statistics, we established g-conditional upper bound on the KI. divergence between learned policies and the behavior policy. We also discussed another possibility of generalization by Tsallis entropy, leading to another new algorithm Tsallis InAC. We evaluated Tsallis AWAC and Tsallis InAC the standard D4RL benchmark problem. We found that Tsallis InAC was sensitive to expert datasets which it was among the best performer but quickly degraded on non-expert datascts. By contrast, Tsallis AWAC was more stable and outperformed AWAC by large margin on almost every dataset, thanks to the sparsity introduced by the geexponential policy. In fact, Tsallis AWAC was among the best performers in the compared algorithms.
Several interesting future directions concerning Tsallis regularization for offline RL exist. Theoretically, a probabilistic upper bound for the KL divergence of sparsemax policies and guarantees of policy improvement should be derived to deeply investigate the benefit of sparsity. Practically, it is important to improve Tsallis InAC on non-expert datasets. This could be potentially achieved by referring to Tsallis statistics literature to transform g-exponential. Moreover, examining the pattern of environment/ paapnt/datttttttte ddatsset-specife optimal entropic index is another interesting topic.
References
OpenAl: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, and Others. Learning dexterous in-hand manipulation. The International Journal of Robolics Research, 39(1) 20, 2020.
Mohammad Gheshlaghi Azar, Vicenç Gómez, and Hilbert J Kappen. Dynamic policy programming. Journal of Machine Learning Research, 13(1): 13(1)3300733255, 2012.
Boris Belousov and Jan Peters. Entropic regularization of markov decision processes. Entropy, 21(7), 2019.
Mathieu Blondel, AndrÅ F.T. Martins, and Vlad Niculae. Learning with fenchel-young losses. Journal of Machine Learning Research, 21 (35):1 69, 2020.
David Brandfonbrener, William F Whitney, Rajesh Ranganath, and Joan Bruna. Offline RL without off-policy evaluation. In A. Beygekimmer, Y. Dauphin, Liang, and J. Wortman Vaughan (eds.), Advances Neural Information Processing Systems, 2021. URL http://///peeeeeeeeeeee/trrrtrne
Alan Chan, Hugo Silva, Sungsu Lim, Tadashi Kozuno, A. Rupam Mahmood, and Martha White. Greedification operators for policy optimization: Investigating forward and reverse kl divergences. Journal of Machine Learning Research, 23 2022.
Yinlam Chow, Ofir Nachum, and Mohammad Ghavamzadeh. Path consistency learning in Tsallis entropy regularized MDPs. In International Conference on Machine Learning, pp. 979 988, 2018.
Robert. Dadashi, Shideh Rezaeifar, Nino Vieillard, Léonard Hussenot, Olivier Pietquin, and Matthieu Geist. Offline reinforcement learning with pseudometric learning. In Proceedings of the 38th International Conference on Machine Learning, pp. 2307-2318, 2021.
Nan Ding and S.v.n. Vishwanathan. t-logistic regression. In Advances in Neural Information Processing Systems, volume 23, 2010.
13

---

Published in Transactions on Machine Learning Research (05/2024)
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach offline reinforcement learning. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration.
In Proceedings of the 36th International Conference on Machine Learning, pp. 2052-2062, 2019.
Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent RL without entropy. In The Eleventh International Conference on Learning Representations, 2023. URL https:
//openreview wwwpreevveeeeetefffeeeeeeeeeee wewt/eorumidd-SSOo.........
Matthicu Gcist, Bruno Scherrer, and Olivier Pictquin. A theory of regularized Markov decission processes. In 36th International Conference on Machine Learning, volume 97, pp. 2160 2169, 2019.
Seyed Kamyar Seyed Chasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected-max q-learning operator for simple yet elfective oflline and online rl. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pp. 3682 3691, 2021.
Caglar Gulcehre, Sergio Gómez Colmenarejo, Ziyu Wang, Jakub Sygnowski, Thomas Paine, Konrad Zolna, Yutian Chen, Matthew Hoffman, Razvan Pascanu, and Nando de Freitas. Regularized behavior value estimation, 2021.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In 94th International Conference on Machine Learning, pp. 1352-1361, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. Proceedings of the 35th International Conference Machine Learning, pp. 1861-1870, 2018.
Natasha Jaques, Asma Ghandeharioum, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of human preferences in dialog, 2020. URL httpps://openrevier http://paarreiiiee.eeeooooorddds rett/foruumidxxx115=vvvi
Liyiming Ke, Sanjiban Choudhury, Matt Barnes, Wen Sun, Gilwoo Lee, and Siddhartha Srinivasa. Imitation learning as S-divergence minimization. arviv, 2019. URL atttp::/arrxiv.or//bbb//// 12888
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofr Nachum. Offline reinforeement learning with fisher divergence critic regularization. In Proceedings of the 38th International Conference on Machine Learning, pp. 5774-5783, 2021.
Tla Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement. learning with implicit q-learning. In International Conference on Learning Representations, 2022. URL https::/openrevierr https://pppeeeriieeeeeeffforrrir net/forumiid
68n2a9ZJWFB
Tadashi Kozuno, Eiji Uchibe, and Kenji Doya. Theoretical analysis of efficiency and robustness of softmax and gap-increasing operators in reinforcement learning. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89, pp. 2995-3003 2019.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off policy q-learning via bootstrapping error reduction. In Advances Neural Information Processing Systems, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline rein- forcement learning, In Advances in Neural Information Processing Systems, volume 33, pp. 1179 1191, 2020.
Aviral Kumar, Jocy Hong, Anikait Singh, and Sergey Levine. Should run offline reinforcement learning behavioral cloning? In Intermational Conference on Learning Representations, 2022. URL https http:r/rwwieeeeeetfooomee//aaaaa
14

---

Published in Transactions on Machine Learning Research (05/2024)
Kyungiae Lee, Sungjoon Choi, and Songhwai Oh. Sparse markov decision processes with causal sparse tsallis entropy regularization for reinforcement learning. TEEE Robolics and Aulomation Letters, 22146611473) 2018.
Kyungjae Lee, Sungyub Kim, Sungbin Lim, Sungjoon Choi, Mineui Hong, Jac In Kim, Yong-Lac Park, and Songhwai Oh. Generalized tsallis entropy reinforcement learning and its application to soft mobile robots. In Robotics: Science and Systems XVI, pp. 1-10, 2020.
Yuhan Li, Wenzhuo Zhou, and Ruoqing Zhu. Quasi-optimal reinforcement learning with continuous actions.
In The Eleventh International Conference on Learning Representations, 2023. URL https://openrevieer Bet/forrm221d=DBvc5227
Jiafei Lyu, aicheng Gong, Le Wan, Zongqing Lu, and Xiu Li. State advantage weighting for offline RL. In 3rd Offline RL Workshop: Offline RL as a "Launchpad", 2022.
André F. T. Martins and Ramón F. Astudillo. From softmax to spansemax: sparse model of attention and multi-label classification. In Proceedings of the 33rd International Conference on Machine Learning, pp. 1614 1623, 2016.
David McAllester and Karl Stratos. Formal limitations on the measurement of mutual information. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108, pp. 884, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, and Others. Human-level control through deep reinforcement learning. Nature, 518(7540):529 533, 2015,
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. (AWAC}: Accelerating online reinforcement learning with offline datasets, 2021. URL httpps://openreviee. httm://oooourrm1111111111111111
Takayuki Osa, Akinobu Hayashi, Pranav Deo, Naoki Morihira, and Takahide Yoshikke. Offline reinforcement learning with mixture of deterministic policies. Transacttons on Machine Learning Reseorch, 2023. URL https://openrevieyy http://www...toomm///d/dd/dddlpp
Aldo Pacchiano, Jonathan N Lee, Peter Bartleu, and Ofir Nachum. Near optimal policy optimization via reps. In Advances in Neural Information Processing Systems, volume 34, pp. 1100 10101110,,,11000 2021.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage weighted regression: Simple and scalable off-policy reinforcement learning, 2020. URL https://opeereriiee http:///wweeeeeeeeoommmmmnontaan
Dean A. Pomerlcau. Alvinn: An autonomous land vehicle in neural network. In Aduances in Neural Information Processing Systems, 1988.
Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., New York, NY, USA, 1st edition, 1994.
Tim G. J. Rudner, Cong Lu, Michael Osborne, Yarin Gal, and Yee Whye Teh. On pathologies in KL-regularized reinforcement learning from expert demonstrations. In Advances in Neural Information Processing Systems, 2021. URL
Noah Sicgel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaloki, Michael Ncunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavior modelling priors for offline reinforcement learning. In International Conference on Learning Representations, 2020. URL http:sss///penreviee..eeettttttt wis.rrrmmmiidrrkeeeeeeeet.ee
David Silver, Julian Schrittwicser, Karen Simonyan, Ioannis Antonoglou, Aja Huang. Arthur Guez, and Others. Mastering the game of Go without human knowledge. Nalure, 550:354 359, 2017.
H. Suyari and M. Tsukada. Law of error in tsallis statistics. IEEE Transactions on Information Theory, 51 (2):753 757, 2005.
15

---

Published in Transactions on Machine Learning Research (05/2024)
C. Tsallis. Introduction to Nonertensive Statistical Mechanies: Approaching Complez World. Springer New York, 2009. ISBN 9780387853581.
Nino Vieillard, Tadashi Kozuno, Bruno Scherrer, Olivier Pietquin, Rémi Munos, and Matthieu Gcist. Leverage the average: analysis of regularization in rl. In Advances in Neural Information Processing Systems 33, pp. 1-12, 2020.
Ziyu Wang, Alexander Novikov, Konrad Zotua, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel. Josh Merel, Caglar Gulcchre, Nicolas Heess, and Nando de Freitas. Critic regularized regression. 34th Conference Neural Informalion Processing Systems, 2020.
Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long, Supported policy optimization for offline reinforcement learning. In Advances in Neural Information Processing Systems, 2022. URL
httpss//openrreview.cn http::/www.....oooooooo//oooooo.
Yifan Wu, George Tucker, and Ofr Nachum. Behavior regularized offline reinforcement learning, 2020. URL
Yue Wu, Shuangfoi Zhai, Nitish Srivastava, Joshua M Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainly weighted aclor-critic for oflline reinforcement learning. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pp. 19119-1128,, 2021.
Chenjun Xiao, Han Wang, Yangchen Pan, Adam White, and Martha White. The in-sample softmax for offline reinforcement learning. In The Eleventh International Conference Learning Representations, 2023. URL https://opperreview
htt////orrninuuuuuuue Rdu-RRuyDDQCC
Haoran Xu, Li Jiang, Jianxiong Li, and Xianyuan Zhan. policy-guided imitation approach for offline reinforcement learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.). Advances in Neural Information Processing Systems, 2022. URL https::/openreevie http://pereeviee.e.eeefffrorrooo net/forummid=
CKbqDtZnSc
Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline RL with no OOD actions: In-sample learning via implicit value regularization. In The Eleventh International Conference on Learning Representations, 2023. URL https://opeareriew..ettt wtw.:/rrnetortm idd= ueYYgo2pSSU
Takuya Yamano. Some properties of q-logarithm and q-exponential ial functions in tsallis statistics. Physica A: Statistical Mechanics and Applications, 305(3):486-455) 2002.
Wenhao Zhan, Shicong Cen, Baihe Huang, Yuxin Chen, Jason D. Lee, and Yuejie Chi. Policy mirror descent for regularized reinforcement lcarning: A generalized framework with lincar convergence. CoRR, 2023.
URL http:///aaxxvv.ooaaaaa///aaa
11066
Lingwei Zhu and Takamitsu Matsubara. Cautious policy programming: exploiting KL regularization for monotonic policy improvement in reinforcement learning. Machine Learning, 2023.
Lingwei Zhu, Zheng Chen, Matthew Schlegel, and Martha White. Generalized munchausen reinforcement. learning using tsallis kl divergence. In Advances in Neural Information Processing Systems, 2023.
16

---

Published in Transactions on Machine Learning Research (05/2024)
Appendix
The Appendix is organized into the following sections. We first discuss related work that helps position this paper; followed by the missing proof of Theorem 1 Lastly, implementation details and further results Tsallis InAC are included.
A Related Work
Proof of Theorem
C Implementation Details
D Details and further results of Tsallis InAC
A Related Work
A.1 Entropy Regularization
It is classic result that there exists a deterministic, stationary optimal policy that maximizos the cumu- lative return with a fixed point Q.(s.a) satisfying the Bellman optimality equation Q=(s. = r(s, a) + [max, y [max_ Q.(s',a') (Puterman) 1994j. The optimal policy m. can then be extracted by simply acting greedily with respect to the optimal action value function. In this case, T. = arg max, Ex (Q,(s, a)], Vs is a deterministic policy.
In this paper Er (Q(s,a) ((T(-1s))], where (2(((-1s) E || is proper, lower semi-continuous, strictly con- vex function in æ. Popular choices for Ω include the negative Shannon entropy -TH (T(-1s)) == Iss In (als), which encourages the policy to be uniform with weighting the effect. The max- imum of the problem max, E. ((a)s)((ss a) + TH (T(-ss)) is attained at the well-known log-partition function TIu E. exp (T
consider stochastic, entropy-regularized policies
solution of the problem
T E.. T(ass)
when the policy is the Boltzmann softmax distribution (aass) x exp ((,, a)), where & denotes proportional to up to a constant not depending on actions. KL. di- vergence DKI (T((s)) ((-1s) == E. (a]s) In s(als) is another popular choice where μ is some reference plais
policy (Azar et al. 2012 Vieillard et al. 2020 Chan et al.) 2022 Zhu & Matsubara 2023). Using KL divergence as Ω penalizes large deviation from μ.. By choosing to be the uniform distribution, KI, di- vergence recovers the negative Shannon entropy case. The KL-regularized optimal policy takes the form (als) p(als) exp (Q), where we overloaded the coefficient for Shannon entropy.
A.2 More on Tsallis Regularization
Sparsemax policies can be obtained as a result of Tsallis regularization. Recall the sparsemax policies are written as (als) = exp, (Q(s.)) 7 - ( (Qi=))) ss ((G-)) (Q()))== E.EK(o) Q(mma)
9-2 Intuitively, the policy 9-1"
[K(+)]
first sorts actions a(1),.... ((LA)) and then compute the threshold vb Suppose Q(s, a(++))) < < Q(s, ag)) then ((++))..... ()....., (((A)) are truncated and have zero probability of being selected. The actions a(1)...., a() are called allowable actions and collected in the set K(s). The degree of truncation can be controlled by either or 7. As gets larger, by definition of exp, the truncation becomes stronger, as all x < T will be truncated; when becomes larger, more actions are collected in K(s).
A nuance arises since the literature has considered different notations. We provide a comparison below in Table 2 to clarify the notations and standard used in this paper. It can be seen that this paper follows the convention of prior RL papers (Lee et al.) 2020) but adopts the Taylor's first-order approximate policy from (Zhu et al. 2023). The pioncering work (Lee et al. 2018 Chow et al. 2018) can be recovered from our formulation by setting q = Suyari & Tsukada (2005 Lee al. (2020) did not give computable normalization function.
17

---

Published in Transactions on Machine Learning Research (05/2024)
Literature
Policy Punctional
General Normalization
Physics/ RRL
LERRO Dipai-1
Lee al.
2018
Chow
2018
[0(s,a) - ((Q(()))))
RL
[R(a]]
Lee al.
2020
[1 [1 (g - (Q(=.a) - (QQ(s, -))
N/A
RL
Suyari & Tsukada 2005
Tsallis
2009
[1+ (1 - g) (Q(+,a) ((Q(,,))) m
N/A
Physics
Qlaa)
Zhu et al.
2023
[] +1 - 4) (Q(a, 11)
RL
This paper
[] (g 1) (Q(a, 11)
V
RL
Table 2: Comparison of existing work. Physics/RL indicates the category of the cited papers. Our work follows the Taylor's first-order approximate policy of (Zhu et al. 2023), but with change of variable 2 -9 more consistently follow the RL literature.
A.3 Other Regularizers and Sparse Policies
Recently, there has seen growing body of BC/imitation-basee methods featuring a variety of divergences. Ke et al. (2019) discussed imitation learning via the lens of f-divergence minimization and proposed variational solutions. Xu et al. (2023) proposed a framework based on the a-divergence (Belousov & Peters 2019) and showed that standard KL can be recovered as a special case. In the samne context, they showed that Conservative Q-learning (Kumar et al. 2020) corresponds to the x² regularization. In this paper, we consider regularization by Tsallis KL divergence and Tsallis entropy, inducing a general class of sparsemax policies. Offline RL with sparsity has also been discussed very recently: Xu et al. 2202 2023) showed that a-divergence with a 11 also leads lo sparsity. By contrast, we consider general sparsemax policies induced by Tsallis regularizers for all > 1. Li et al. (2023) proposed to use g-Guussian distribution (Suyari & Tsukada 2005 which is a special case of the g-exponential policy. In this paper we do not specify the functional form of policies, but investigating the benefits of specific geexponential policy parametrizations is an interesting future direction.
B Proof of Theorem 1
To prove Theorem decompose KL divergence
Dkz Dez(Te(-Iss) TD(-ss) = Eawn [ln ((alss - In [a|s)]
(9)
= =aww lnms(a - In, T. (als) + In, (elas) - In, (D(als) + In, (ala)s) - In (a(ss))))
(181)
Therefore, bounding respectively the three terms allows us to provide a g-conditional bound on the KL. divergence. To bound these log/ log differences, we prove the following fact:
d
In - In. = (g -
Ing x In In,
dq
We can verify this is true by working the right-hand side:
d
d x
(g - 1)
In x - In In, x
= (q 1)
In x Ing æ
dg
dg --1
(r--1 11'gg - 1) (x -1)(4 1 -1' In In
(g 1)
(g 1)2
(g 1)x0 1 In (x -1]
(g 1)
- In In
(q vP-1) 1)²
x In In, (g 1) In In, ((g 1) In
1) Il x In, x (g 1) In In,
In Ing
18

---

Published in Transactions on Machine Learning Research (05/2024)
Now to bound the log/q-log differences, we assume that both the learned policy and the behavior policy are sparsemax:
d
( [ - In, (a(ss)) - (a- - 1)
In, Te (a(s)) 8) - Im, T. (alss) In me(as))))
dq
न ( (n In [x(ass) (1 (q (110-))mmnnn l)
1
(10)
क - (als) (als) + q-1 + In (m
2
( कं (als)) (als)
q 9-1*
where we leveraged the monotonicity of q-logarithm; and In I x 1, Iy exp,(x) = when r > 0. Considering the policy is q-exponential policy (e(ass) exp,
Q (Qu((a)) v Qi (((211))) and exp,
1 1 (q- (1-))))) we must have ni (a|s) > +0 a ()-1,(()) + --1 (0-1(1))) - ( S 0 Ding & Vishwanathan 2010). If a & (s)),((),, then (als) = -0 and the KL term is unbounded. The same fact is exploited to yield an upper bound 1 for (ii). Note that the same holds true for the 9-1
Tsallis KL. policy m=(als) = plaass) exp, (Qe-iloa)) v (Rezz(s)) Since Ing Z is monotonically increasing, we (Q-=((=))) Same
have that q-r < Ing m(ass) expy (e-1(am) vb (2-z(111) < Ing expy (--1(s.)) by monotonicity, we can bound (ii) depending on by:
(iii) : Ing (als))) - (n)s( = -q -0 1) In <b(aas)) - In
dg
(11)
(als -
(als) 11 TD(als)) (1-mp(als)) (als) x (
Putting all terms logether, we arrive at the following g-dependent upper bound:
(=(( - [Kig())] ID..
if n. 3 TD, x otherwisc
(12)
00,
where IID,1. == (a|s) (als) (a 9-3 (als) T (als), and the leading term (s)) came from upper bounding the expectation Easx((le) with all allowable actions in the set Keg(s).
9
C Implementation Details
We fine-tune Taallss InAC, Tsallis AWAC and InAC since they share same set of hyperparameters. All the algorithms used a shared set of hyperparameters found in Table
A grid search was done for Tsallis InAC, Tsallis AWAC, TnAC according to the same protocol as (iao et al., 2023). In addition, we also added a larger learning rate (0.001), which secmed to improve InAC, Tsallis AWAC, and Tsallis InAC slightly on some domains. Other methods use performance data shared by (Xiao) et al.) 2023). The best hyperparameters are reported in Table 4 of evaluation tests of the normalized return to select the best hyperparameter shown. All hyperparameter settings were evaluated across 5 independent runs.
In the grid search, we used the final 50%
The agents train from the specified datasets, and are evaluated every 10k steps on the corresponding Mujoco environments. Normalized scores are calculated according to the min and max values provided as part of the benchmarks (uu et al. 2020 Finally, results are over 5 runs unless otherwise specified.
Details and Further Results of Tsallis InAC
Tsallis InAC Loss Functions. We follow in-sample softmax Xiao al. 2023 but replaces the Shannon entropy to Tsallis entropy. In order to fulfill the in-sample constraint, step similar to (Xiao et al. 2023 is
19

---

Published in Transactions on Machine Learning Research (05/2024)
Name
Value
Number of sleps
1000000
Logging interval
10000
Hidden Units
256
Name
Swetp Values [0.00003, 0.0001, 0.0003, 0.001] [0.01, 0.1, 0.33, 0.5, 1.0)
Batch Size
256
Learning Rate
Target Network Update Rate
1
Polyak Constant
0.995
Discount (7)
0.99
Learning Rate
swept
Regularization coefficient (r)
swept
Table 3: (Left) Shared hyperparameters. (Right) Swept hyperparameters.
Environment
Dataset
Parameter
Tsallis InAC
Tsallis AWAC
Ir
0.01
0.01
Expert
T
0.5
1.0
Ir
0.0003
0.001
Ant
Medium Expert
T
0.1
1.0
Ir
0.0003
0.001
Medium Replay
0.01
0.5
Ir 0.001
0.001
Expert
0.33
1.0
1r 0.001
0.001
HalfCheetah
Medium Expert
0.5
1.0
Ir
0.001
0.001
Medium Replay
T
1.0
0.01
lr
0.001
0.001
Expert
7
0.01
1.0
1r
0.0001
0.001
Hopper
Medium Expert
0.01
0.5
Ir
0.001
0.001
Medium Replay
7
0.01
0.5
1r 0.001
0.001
Expert
0.1
1.0
Ir
0.0003
0.001
Walker
Medium Expert
T
0.01
0.01
Ir
0.001
0.001
Medium Replay
0.01
0.5
Table 4: Best learning rate (Ir) and regularization coefficient 7 for the proposed method.
made to extract up from the Tsallis entropy regularized policy for the actor loss:
1 (oo(s.a)
PInAC "Qo (als) & Lia: =p(als) > 0} exp,
= exp, 1 (ols.aa) TD(als) exp, In, (T(a
exp, - (o((,a)
(13)
(esp, = <al(a exp, (= + In Td(als) + (9- - - a) In,
1
Q0(s,
<D(als)
where we assumed that This can happen when Qo(s, a) and TD = 0. In this case, the entire
Eq. (13) should be 0 due conditional on Tp. (exp, exp,,) 2- (x + -- (g 1)2xy Yamano
2002 Following (Haarnoja al. 2018), update towards this policy by minimizing the KT. divergence
20

---

Published in Transactions on Machine Learning Research (05/2024)
between A and Fg. When used the first argument of KL loss, the leading ap allows to compute the loss using only actions from the dataset:
DeL (#
(+1s)
[ (+(ss))
Ba Rooo (110) [In At Qo.TD (als) - In
TInac
(1SP)) (Q0(s,a)
( expo (20(s)00)) Ing To (a(s)))
- (In a TInAC Qo.aD (a|s) In ITS (als))
9-1
+ (q- 1)2 Qo(s, a) Ing ma(a
= (1a
Notice the term In &TInAC RQ..T (a|s) does not depend on T and hence can be removed from the actor loss. We observed that for Tsallis entropy regularization, adding normalization tends to underperform, therefore we propose to remove the normalization. We write out our losses for 2
( exx2 (Q0(,,a)
1
Qo(s, a) Iuz Ra(coo] In (alss)
C (x) -ByganD
In2 Ta(aas))
7
C (0) E B'
(I+ 7Vc(s') - Qo(cs,)))]
(14)
baseline (C) E (aDD,
(VV(()) (Qo(s, Ing T (als))))
CTInAO (Ø) is likely to cause numerical issues. To avoid it, we clip the range of Tue by The term [a(s) actor
max(c, Th (als)), with c= 10-8 However, since Ing x is proportional to the q-th power, we are unable to sweep over larger q due to numerical problems even with clipping. Therefore, in the experiments we sweep different q for the Tsallis KL implementation only.
Remark. Let us focus on C (Ø) and define the term in the bracket before In T as C:
(Q((s,a)
)
+ Qa(s, Inz mutals]
C : = exp2
+ In2
Tes (als))
then CTInAC(O) (Ø) -Eg (olsss(
Notice that exp2 (o((sa) + Inn xu [alo]]
can be written as
{ 1+ Qo(s,a)
> o0] (1+ Qo T Tu (als)
1
1
Inz
+ Inn
Tee (als))
T
therefore, it must be non-negative and the sign of C depends on Qo(o,a) luz Tutaiet
However, since
twtals) > O, the sign actually depends on Qo(s, a) only. If this term is negative and Qa(s.)) Ins a. ia
In
exp2 (Q((..)) + Inz =..l)))):: then we will be minimizing the loss E.a-D [[C] In [a(a)))], which corresponds to thinking an action a is bad (since Qe(s, a) is highly negative) and explicitly minimizing its log- ligllikel;hood; otherwise, we will be maximizing its likelihood. By contrast, all exponential-based methods (exponential advantage weighted regression algorithms like AWAC) do not minimize likelihood for actions in the dataset explicitly, since exp is always positive. Moreover, this suggests that the behavior of Tsallis InAC may be more
1)2 1)) Qa(s,a) Inz swtap] will have a large negative value when q is large even with slightly negative Q0(s, a). As a summarry, Tsallis InAC may behave drastically: it may perform either very well very bad depending on whether the dataset contains majority of near-expert trajectories.
extreme with larger 4, since by equation 13 (g
Tsallis InAC against InAC. From Figure g it is clear that Tsallis InAC competes favorably against InAC only on the Expert dataset, and degrades significantly with the increase of non-expert trajectories in the dataset. In fact, Tsallis InAC behaves like a behavior cloning method and learns very fast on Expert HalfCheetah and Hopper. As analyzed in the remark after equation 14 the difference between InAC and AWAC policies lies in the term - In TD in the policy (als) x (als)) exp (Qo(e.a) In In This term correct the bias introduced by suboptimal data distributions: poor action with high (a(ss))
21

---

Published in Transactions on Machine Learning Research (05/2024)
Ant
HalfCheetah
Hopper
Walker 2D
Expert
133
1.2
1.2
10
Normalized
1.1
0.6
0.8
Retarn
1.0
0.8
0.4-
0.9
0.81
0.6
0.4
25
50
75
25
50
75
50
75
25
75
Medium Expert
1.2
Normalized
0.7
0.8
Return
1.
0.6
0.4
0.8
0.4
Medium
1.0
0.8
Replay
Normalized
0.8
0.40
0.6
Retarn
0.6
0.4
0.2
Steps (1e5)
Steps (1e5)
Steps (Ie5)
Steps (1e5)
Tsallis InAC
InAC
Figure 8: Comparison between Tsallis InAC and TnAC. Normalized policy evaluation scores Mujoco Expert and Medium Expert datasets over one million steps. Results are the average over 5 runs with ribbon denoting the standard error. g-axis: scores; r-axis: number of iterations.
does not necessarily lead to high (a)ss since it is weighted down inside the exponential by - In TD (als). While Tsallis InAC follows this design choice, in general In, का(া|. > In কা(āæ] In fact, since In, Rotaje is proportional to the g-th power of -p(aae] 2 1, this term likely to dominate the entire CTIAA suggesting Tsallis InAC is very sensitive to the level of behavior policy like BC method.
22


Reviews:
[{"review_id": "zLLjnFV4r5", "reviewer": "Reviewer_XgiD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper demonstrates that a seemingly simple replacement of a regularization function can have a big impact on policy behavior. \n* The paper has several interesting findings concerning sparsity, the role of regularization on various stages of the algorithm, behavior for expert and non-expert datasets, or a bound on the KL divergence between two sparsemax policies.\n* The paper connects the dots between multiple approaches existing in the literature.\n\nWeaknesses:\n* General comments:\n  * Sections 3-5 are dense and contain a lot of inline formulas. The content of these sections could be organized in a more clean way. Additionally, the paper would be much easier to read if it was (close to) self-contained. The major results used from other papers should be proved (or sketched) in the Appendix.\n  * It seems that the same results are repeated in several places. It would be nice to have the main facts gathered in one place (e.g., the solutions to the optimization problems under various regularization terms).\n  * There is no related work section; the references are scattered across the paper.\n  * The paper could be shortened by moving parts of the math to the Appendix and reducing the repetitive statements (e.g., Figs 4-6, or the form of the solution to regularized problem).\n  * The environments are simple Mujoco tasks and there are not that many offline RL baselines. \n\n* More details follow:\n  * Section 2:\n\t* In the paragraph following eq. (1), the characterization of $K(s)$ does not clearly match the one in the cited reference (Zhu et al. 2023, page 15).\n\t* Equation (1) describes approximation, but it is not clear to what and in what sense.\n  * Section 3:\n\t* In the last paragraph of page 5 it is not clear where $\\mathcal H$ is (does not appear in eqs. (2) or (3)).\n\t* In eq (4), there should either be $\\pi_*$ or $Q_{t, \\pi_\\mathcal D}$. \n\t* The discussion around eq. (5) is unclear, particularly around $K_{t,2}$. Additionally, only $K$ depends on $t$.\n\t* The last paragraph of this section discusses using regularization for policy improvement and policy evaluation. It is unclear, whether the choice of where to use regularization follows from the method or is a design choice. In the latter case, there would be the question concerning the methods' convergence.\n  * Section 4:\n\t* This section should be structured more formally, with the main result (eq. (9)) as a theorem with all the necessary assumptions and the rest of the section as proof. The proof could be placed in the Appendix.\n  * Section 5:\n\t* The last paragraph of Section 3 talks about the regularization of policy improvement and policy evaluation, and hints at more details in Section 5. This topic should be expanded upon in Section 5, with a clearer explanation of where the regularization enters the Tallis InAC and Tallis AWAC loss functions.\n\t* Eq. (11) should assume some convention when dealing with for expressions like 0/0.\n\t* The term \"baseline\" is overloaded: there are \"baseline algorithms\" and a \"baseline\" in the advantage function (used in the subscript for the losses).\n  * Section 6:\n\t* Environments are simple Mujoco tasks (Ant, HalfCheetah, Hopper, Walker 2D).\n\t* Fig 3 should be placed in the Appendix.\n\t* The datasets expert, medium expert, and medium replay should be explained in more detail (in particular, medium replay is not explained).\n\t* There is a small number of seeds (5).\n\t* Figures 4-6 duplicate a lot of information. The redundancy should be reduced. Additionally, making the colors on the plot more distinct could help with the figures' readability.\n  * Section 6.2\n\t* Why only include Hopper and Walker2d (in expert versions)?\n\t* Some statements are not very clear:\n\t\t* \"larger q tend to learn quicker and are relatively insensitive to training\" - seems rather vague and not necessarily visible in Figure 7.\n\t\t* \"shrinks the gap Q-V\".\n\t\t* \"larger set of allowable actions\".\n\t* Consider splitting Fig 8 and Fig 9: Fig 9 belongs to Sec. 6.2, Fig 8 to Sec 6.3.\n\nQuestions:\n* How does the behavior of methods for near-expert and non-expert data relate to the findings in [1]?\n\n[1] Kumar, et al. When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?, 2022", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper considers modifications of two offline RL algorithms in-sample actor-critic (InAC) and advantage-weighted actor-critic (AWAC): Tsallis InAC, based on Tsallis Entropy, and Tsallis AWAC, based on Tsallis KL divergence. The paper investigates various problem statements, compares them with previous works, and provides theoretical insights. The loss functions are stated, and the algorithms are tested for four Mujoco tasks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the sections above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "r25e5XtkNF", "reviewer": "Reviewer_pk3h", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n1. The proposed algorithm is simple and elegant as well as novel for the offline RL setting. The idea of using Tsallis regularizers that induce sparsemax instead of softmax policies (as in AWAC) is intuitively appealing for the offline RL setting. \n\n2. The experiments are thorough, providing a variety of sweeps on standard benchmark tasks as well as some diagnostic experiments to understand the behavior of the proposed algorithms.\n\n### Weaknesses\n\n1. There is a disconnect between the presentation and the actual algorithms implemented for the experiments since to my understanding, the presentation is entirely for discrete action spaces, but the algorithms are applied in continuous spaces. This major disconnect seems to be only briefly addressed in the algorithm presentation when certain terms are replaced to allow for continuous actions. Since this likely significantly affects the performance of the algorithms in the experimental results, more clarity is needed as to how the motivation connects to the actually implemented algorithms.\n\n2. The theoretical section (section 4) is somewhat confusing. First, making a more clear theorem-proof structure could help to clarify what the main assumptions are and what the takeaway is supposed to be for the reader. Second, it is unclear to me what this section is supposed to be doing. It seems that the idea is to relate the proposed sparsemax policy to the standard KL divergence in related work, but it is not clearly introduced and the final result is very difficult to interpret. \n\n3. The paper overall is a bit verbose. While I understand that the authors may want to provide detailed introduction, it could be useful to get into the novel contributions more quickly.\n\n4. I am confused by the decision to present the largely negative results for Tsallis-InAc on similar footing to the positive results for Tsallis-AWAC. It is nice to present the negative results, but it could perhaps streamline the presentation and takeaways for the reader to focus the paper on the modifications to AWAC and move the InAc-related results to an appendix for the interested reader.\n\nMinor comments:\n- The sum at the end of the first paragraph at the top of page 2 is referred to as ``softmax'', but it not in fact the standard softmax function (which is actually a soft version of argmax). I know this is not a specific problem with this paper, but with terminology in the field at large, but to reduce confusion it could be worth either referring to this quantity as sumexp (or logsumexp) where appropriate, or to add some footnote just clarifying this is not the standard soft arg max. \n- The axes labels in figure 1 are too small.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents Tsallis-regularized algorithms for offline RL. The main idea is to translate some recent ideas from Zhu et al. (2023) in the online setting that takes standard RL algorithms that use KL divergences and replace them with Tsallis divergences. This idea is applied on top of the AWAC and InAc algorithms for offline RL with promising results on top of AWAC and less promising results on top of InAc.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "To resolve the weaknesses above, I would suggest the following changes:\n\n1. Clarify the discrete vs. continuous issue.\n\n2. Improve the presentation of the theoretical section and connect it to the proposed algorithms more clearly.\n\n3. Streamline the presentation of sections 2 and 3.\n\n4. Move the InAc-related results to an appendix and focus more on the AWAC-related results. If the authors/other reviewers think this is too strong, I would at least advocate separating the paper to present these as two more clearly distinct algorithms by e.g. first presenting the AWAC algorithms and experimental results together and then presenting the InAc algorithm and experiments second.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2kWI3O9q6a", "reviewer": "Reviewer_qDjc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "=== Strengths ===\n\n- The paper is mostly well-written, and I appreciate how the authors carefully explain their motivation, supporting it with empirical evidence.\n\n- The resulting method is simple. It is also nice to see that the authors combine their method with a recent work, InAC.\n\n- detailed parameter study is conducted\n\n=== Weaknesses ===\n\n- the novelty of this paper is somewhat limited, but this is not a big issue for this venue\n\n- some expressions are confusing, I list some of them below:\n\n  - page 1, *Since the environment cannot be sampled, the bias can never be alleviated by visiting the region and correcting the value estimate*, what do you mean by the environment cannot be sampled? I also disagree that the value estimate cannot be corrected with only static data. This can be done with uncertainty based methods like UWAC [1]\n\n  - page 2, *Intuitively, this assumption invites learning an improved sparsemax policy with support within the that of the behavior policy*, this sentence is unnecessarily complex. Also, what do you mean by *within the that of the behavior policy*? Is this sentence generated by LLMs? If this is generated by LLMs, please make sure that all of the sentences are checked by humans. Meanwhile, please break down long, complex sentences into short, concise sentences in the entire paper.\n\n[1] Uncertainty weighted actor-critic for offline reinforcement learning\n\n- the quality of the figures can be significantly improved (e.g., Figure 1). Please try to export pdf with matplotlib instead of taking screenshots by convention.\n\n- Tsallis AWAC beats AWAC on most of the tasks, however Tsallis InAC underperforms InAC on most of the tasks. The authors comment that this is due to *Tsallis InAC is very sensitive to the level of behavior policy like a BC method* which is analyzed in the remark after equation 12. I do not seem to find the corresponding explanations and analysis. Please make it clearer. Also, please add discussions on the limitations of your method.\n\n- it is vital to analyze the applicability of your method. Since it downgrades the performance of InAC in numerous tasks, is it general enough to benefit a wide range of policy regularization offline RL algorithms? \n\n- no code seems to be provided\n\n- the performance reported in this paper is obtained by a very careful tuning of hyperparameters (see Table 3) . This further downgrades the applicability of the proposed method. Meanwhile, please specify detailed hyperparameter setup on each dataset.\n\n- learning rate seems to be a critical hyperparameter, but no parameter study is conducted\n\n- the experiments are only conducted on D4RL MuJoCo datasets. It is unclear how the proposed method behave on another domain like antmaze, adroit, etc. It is also unclear whether the proposed method can achieve good performance on medium-level datasets. More experiments are expected, which I believe will make this paper stronger. I know this means a large number of experiments, please at least conduct experiments on 2 antmaze datasets, 2 adroit datasets, and medium-level datasets\n\n- the compared baselines are few, please consider adding more baselines in Figure 6. If you do not have time to run them, you can create a table and directly take the reported numerical results of recent baselines from their papers.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "There are numerous methods being developed to learn policies with offline static datasets. Among them, one of the most popular methods is policy regularization method, which actively constrain the learned policy to be close to the behavior policy (aka, data-collecting policy). This paper lies in this category. The authors propose Tsallis regularization for offline reinforcement learning (RL), which realizes in-sample policy constraint via sparsemax method. The authors show that the learned sparsemax policies may have sparsity-conditional KL divergence to the behavior policy. Moreover, the authors combine their Tsallis regularization with two methods, InAC and AWAC and empirically evaluate their performance in the D4RL datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please answer my above questions and modify the manuscript accorrdingly. I also have the following additional comments:\n\n- reorganize section 4 and summarize a formal lemma/proposition/theorem. The proof details can be attached in the appendix.\n\n- the following work seems to be relevant (in-sample offline learning) and should be included in the manuscript [1]\n\n[1] State Advantage Weighting for Offline RL", "broader_impact_concerns": "I believe no broader impact concerns are needed to be specified.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an alternative way to do regularization in offline RL and two new algorithms to use the technique. The reviewers all agreed that this is a significant contribution. The listed weaknesses were almost entirely about structure, presentation, readability, and technical clarity. The reviewers gave several points of detailed feedback on how to improve the paper, which the authors have taken to apply significant changes to the paper. As a result, the paper has greatly improved. All of the reviewers made note of the significant improvements and agreed upon an acceptance recommendation after the detailed changes.
Decision: Accept (poster)