Fat-to-Thin Policy Optimization: Offline Reinforcement Learning with Sparse Policies
Authors: Lingwei Zhu
Han Wang
Yukie Nagai
Venue: ICLR 2025 Poster (2025)
Keywords: reinforcement learning
offline reinforcement learning
actor critic
sparse policies
Primary area: reinforcement learning
TL;DR: this paper for the first time addresses the issues of offline reinforcement learning with sparse policies

Abstract: Sparse continuous policies are distributions that can choose some actions at random yet keep strictly zero probability for the other actions, which are radically different from the Gaussian.
They have important real-world implications, e.g. in modeling safety-critical tasks like medicine.
The combination of offline reinforcement learning and sparse policies provides a novel paradigm that enables learning completely from logged datasets a safety-aware  sparse policy. 
However, sparse policies can cause difficulty with the existing offline algorithms which require evaluating actions that fall outside of the current support.
In this paper, we propose  the first offline policy optimization algorithm that tackles this challenge: Fat-to-Thin Policy Optimization (FtTPO).
Specifically, we maintain a fat (heavy-tailed) proposal policy that effectively learns from the dataset and injects knowledge to a thin (sparse) policy, which is responsible for interacting with the environment.
We instantiate FtTPO with the general $q$-Gaussian family that encompasses both heavy-tailed and sparse policies and verify that it performs favorably in a safety-critical treatment simulation and the standard MuJoCo suite.
Our code is available at https://github.com/lingweizhu/fat2thin.

Published as a conference paper at ICLR 2025
FAT-TO-THIN POLICY OPTIMIZATION: OFFLINE RL WITH SPARSE POLICIES
Lingwei Zhu"
Han Wang"
Yukie Nagai
University of Tokyo 1 lingwei uau Iberta.c.. ca
University of Alberta han8@ ua aalabbrrt..
University of Tokyo
ABSTRACT
Sparse continuous policies are distributions that can choose some actions at ran- dom yet keep strictly zero probability for the other actions, which radically different from the Gaussian. They have important real-world implications, c.g. modeling safety-critical tasks like medicine. The combination of offline reinforce- ment learning and sparse policies provides a novel paradigm that enables learning completely from logged datasets a safety-aware sparse policy. However, sparse policies can cause difficulty with the existing offline algorithms which require evaluating actions that fall outside of the current support. In this paper, we propose the first offline policy optimization algorithm that tackles this challenge: Fat-to- Thin Policy Optimization (FiTPO). Specifically, we maintain a fat (heavy-tailed) proposal policy that effectively learns from the dataset and injects knowledge to a thin (sparse) policy, which is responsible for interacting with the environment. We instantiate FiTPO with the general q- q-Gaussian family that encompasses both heavy-tailed and sparse policies and verify that it performs favorably in a safety- critical treatment simulation and the standard MuJoCo suite. Our code is available at https: / /i tuub. com/ /////innn wingweizhu fat2thinn
INTRODUCTION
Sparse continuous policies are distributions that can choose some actions random, yet maintain strictly zero probability for the other actions. Therefore, they present a radically different solution than the Gaussian policy which has been standard in the existing policy optimization algorithms (Haarnjja et al., 2018; Abdolmaleki et al., 2018). Sparse policies have important real-world implications, e.g. in safety-critical tasks such as treatment where dangerous actions should never be picked but some other actions should be explored (Fatemi et al., 2021; Yu et al., 2021). Infinite-support policies like the Gaussian or the heavy-tailed distributions (Kobayashi, 2019; Zhu al., 2024b) are hence not suitable in this regard. When sparse policies coupled with offline reinforcement learning (RL), an attractive and novel paradigm emerges: a safe sparse policy can be learned completely from a logged dataset without potentially harming the environment in an online manner.
However, sparse policies pose a challenge to the existing offliie deep RL algorithms: the dataset actions can fall outside of the sparse policy's 's support, leading to undefined log-like likeeiill and hence learning failure. The same issue persists for off-policy algorithms since offline learning can be seen as an extreme case of off-policy learning. Note that this issue does not occur for infinite- policies like the Gaussian but rather is inherent to all sparse policies. To the best of our knowledge, there is no systematic solution to the problem of oft-of-support actions in offline learning incurred by sparse policies. Existing methods resort to ad hoc solutions such as approximating the sparse policy with the Gaussian (Lee et al., 2020; Xu et al., 2023) replacing out-of-support actions with random in-support actions (Zhu et al., 2024b).
In this paper propose Fat-to-Thin Policy Optimization (FrTPO) that for the first time addresses the problem of offline learning with out-of-support actions induced by sparse policies. Our method consists of two steps: learning an infinite-suppor policy (either the Gaussian the heavy-tailed) from the dataset, then imparting its knowledge to a sparse policy. Since there is no prior work
indicates joint first authors.

---

Published as a conference paper at ICLR 2025
directly comparable, we compare FtTPO against the existing ad hoc tricks and popular offline algorithms. As it is commonly perceived that the sparse policies are inherently handicapped at the exploration-exploitttoon tradeoff, find it surprising that the sparse policy learned by FtTPO can outperform full-support policies: FrTPO competes favorably against the popular offline algorithms that the Gaussian by default. To summarize, investigate the out-of-support action issue with offline learning incurred by sparse policies; (2) we propose Fat-to-Thin Policy Optimization: the first deep offline RL framework for learning sparse policies; (3) we verify the sparse policy learned by FrTPO can indeed concentrate on small band of actions. Therefore, outperforms existing popular baselines on a safety- treatment simulated environment and the MuloCo suite.
contributions include: (1) we are the first to
2 BACKGROUND
We focus on discounted Markov Decision Processes (MDPs) expressed by the tuple P. 7). where S and A denote state space and action space, respectively. Let A(X) denote the set of probability distributions over X' P S x A + (SS) denotes the transition probability function, and r(s, a) defines the reward associated with that transition. E (0, 1) is the discount factor. policy : :: + ((A) is a mapping from the state space to distributions over actions. In this paper we focus on the offline setting where we learn from fixed dataset that stores transitions. We denote the behavior policy that generates the dataset by TD. The learninig goal is to search for an optimal policy that maximizes long-term accumulated rewards. We define the action value and state value as " ('r((s 0₂) (5 = s, Go = a), V (s) = E, [QT (2"(saa) a)).
Q" a) = E 15t=0
We specifically consider policies defined by the deformed g-exponential function. The q-exponential function and its unique inverse function g-logarithm are defined by (Naudts, 2002):
expx,
q ==
In a 1-9-1
9=1
exp
Ing.
[1+
-)
9 9
where [++ maxf- 0} sets negative part of the input to zero. Note that exp,zy # exp, exp,rexp,p exp,y unless = 1 and the same for my (Tsallis, 2009). When < 1, it is clear that for the q-exp truncates x < - i.e.
exP x- = (1))) 2 0)
We can use this property define sparse distributions. Prior works focused on discrete sparsemax policies (Martins & Astudillo, 2016; Lee et al., 2018; Chow et al., 2018; Lee et al., 2020). In this paper we consider continuous sparse policies, especially the q- P-Gaussian (Naudts, 2010; Furuichi, 2010; Matsuzoe & Ohara, 2011).
3 OFFLINE LEARNING WITH SPARSE POLICIES
Offline RL algorithms typically require evaluating actions produced by behavior policies using the current policy, e.g. in computing the log- ligeliool The Gaussian policy the standard choice does not incur any issue since it is an infinite-support distribution that can always yield nonzero probability for offline actions. By contrast, the actions may fall outside the support of sparse policy, resulting in numerical issues or even failed learning.
3.1 WHEN SPARSE POLICIES CAN FAIL
For simplicity, we illustrate the sparse policy issue by assuming an actor-critic framework where the actor minimizes the forward KL divergence between the desired policy and the parametrized policy. This setting is common offline RI. and has several popular variants (Jaques et al., 2020; Wu et al., 2020; Nair et al., 2021). The loss can be written as:
L (())
Eg-D DDRL. ( (Tidesired (-1s) π [-[s)))
E anD
[In Teesired (a]s) In no (a|s)),
(1)
a~Tissist
In [a]ss]
snD Toaind
2

---

Published as a conference paper at ICLR 2025
Spaces Empastaal ee
apere polcc
HalfCheetah
008 sumple
Medium- Expeett-
Artion priahiliy
80)
FUTPO
Seare
40
RAR
miniraapi la
ReverseKL
0-
50 Step (x10*)
100
w(ou(elss) -0
Figure 1: (Left) Illustration of offline learning with sparse policy. The red histogram represents an empirical behavior policy. The blue distribution indicates the sparse policy being learned. Actions outside the intersection have zero probability under the learned policy. Therefore, their log-likelihood will return -oo. (Middle) The random action replacement (RAR) trick used by (Zhu et al., 2024b). When an out-of-support (OOS) action appears as the red dot, is replaced by the nearest in-support action (purple dots) sampled from the policy in the L2 sense. This method falls short in high dimensional spaces like the MuloCo suite. (Right) Performance the proposed FrTPO against the ad hoc tricks RAR and reverse KL on the MuloCo Halfcheetah environment, averaged over 10 seeds.
Minimizing this loss amounts to maximizing the log likelihoodd of T under the actions sampled from the desired policy. The last line ignores the term not depending on the optimization variable ø Typically, desiree set to the behavior policy up (or sampling only from the dataset for in-sample learning (Fujimoto et al., 2019)). It is clear that when T is specified as infinite-supoort policy like the Gaussian, optimizing In (TD to be sparse policy, its support can be disjoint with the behavior policy, i.e. (T(TD(())s) and therefore In .T (T))s)) returns -o.. Figure left illustrates this process. The red histogram represents an empirical behavior policy. The blue policy indicates the sparse policy being learned. Behavior actions outside the intersection cannot be used in log-probability evaluation.
does not incur any issue. However, when Ty is designated
0
3.2 AD HOC SOLUTIONS
Since the issue out-of-support actions have not been studied before, there is no systematic solution to the best of our knowledge. We discuss some ad hoc tricks used by existing works. These tricks include random action replacement or reverse KL loss minimization (Zhu et al., 2024b). Random action replacement (RAR) refers to replacing the out-of-support action with the nearest in-support action sampled from the current policy, as visualized in the middle Figure 1. However, should be noted that this method becomes increasingly ineffective as dimension grows, since in high dimension spaces samples are increasingly concentrated inside the ellipsoid (Martins et al., 2022). As a result, becomes more and more difficult sample actions near the boundary of the sparse policy. In Figure I we visualize the performance of RAR which shows slow learning on the example environment.
Another potential solution is to avoid offline actions from the behavior policy. For example, this can be done by sampling from the learning policy itself. By reversing the direction of the KL divergence in Eq. (1) the loss becomes a reverse KI. divergence (Chan et al., 2022):
L R (Ø) EAND
( (+s))
E SnD [In (als) In (p(ss))
(2)
arit
Since actions are now sampled from Tg. no out-of-support issue will occur. However, in practice find this method performs poorly, be from Figure 1. We conjecture that the inability of reverse KL alone is due to the finite support of sparse policies, which cause the sampled actions concentrate around its mode and therefore no significant update can be performed, resulting in extremely slow learning.
4 BAT-TO-THIN POLICY OPTIMIZATION
In Section 3 we discussed several important considerations underlying popular offline learning methods. They can be instructive even for learning sparse policies, e.g. using samples from the
3

---

Published as a conference paper at ICLR 2025
dataset directly or from learned behavior policy for learning stability. Our proposed method builds upon these considerations and takes inspiration from recent advances two-stage actor-critic methods (Neumann et al., 2023). We discuss related work in more detail in Section 6.
4.1 TWO-STAGE LEARNING
Recent studies explore two-stage actor-critic methods that aim to efficiently learn an unbiased policy. The first policy, called the proposal policy, often maximizes biased reward (c.g. with entropy bonus). It is used to generate actions for the second stage. The second policy called the actor policy maximizes unbiased reward by learning from the actions sampled from the proposal policy.
Taking inspiration from this design, formulate Fat-to-Thin Policy Optimization by maintaining two policies for different purposes: an infinite-support policy (fat) that learns from the offline dataset, parametrized by Φ and sparse policy (thin) that learns from the actions generated by the fat policy, parametrized by 0. Specifically, we follow the common practice Eq. (1) for the first stage and minimizes the reverse KL divergence between the two policies. The loss functions of FtTPO can be expressed as the following:
Liaa, Proposal ( (6) = E AND ampp
w- a) In To (a|s)]
(3)
Chiin, Actor (0) = EeND
[RR (TO To (-[s)]
(a(a
E anD
(a In Im -11 To (a|s)
In the proposal loss, have an additional coefficient w(s, for weighting the importance of actions. By letting w(s,a) a) = 1, it is clear that LFat, Proposal recovers the forward KL case in Eq. (1), We discuss more choices that facilitate learning in Section 4.3. Computing the KL divergence involving sparse policy can be very unstable. Therefore, we propose to do the following two steps for the actor loss: (1) before every update of the actor, copy the proposal mean to the actor; (2) use an unbiased estimator of KI. divergence that has less variance (Schulmann, 2020).
While Fau Propusal (Ø) and LTiin, (f) alone - respectively corresponding to forward and reverse KL minimization cannot learn sparse policies, FrTPO can successfully learn one that competes favorably against popular existing methods. We attribute the success to the two-stage framework where the thin policy allows for further improvement of the behavior-constrained fat policy. It is also worth noting that many other alternatives are possible for learning the fat and thin policies. Our choice is based on simplicity and performance. Some more sophisticated methods like SPOT (Wu et al., 2022) do not lead better performance. We discuss them in related work and verify that the experiment section.
4.2 INSTANTIATING FTTPO WITH 9- GAASSIANN
While in principle any distribution can be used for the fat and thin policy, consider the g-Gaussian family which includes both heavy-tailed and sparse members (Naudts, 2010; Zhu et al., 2024b):
1 (s) 1
(a - p(s))2
TA TN,(als) (als)
exp,
20)
((()) o(s))
+
8
if - 00 <
(5)
where (s)
(+
if < <<<3
Here, uu(s), refer to the state-conditioned mean and standard deviation. Zy denotes the normal- ization constant that ensures the policy integrates to 1. Without confusion omit the dependence on the state. Recall that when q = 1 the q- g-aasssian recovers the standard Gaussian distribution. q < 1 corresponds to sparse distributions and 1 < q < 3 to the heavy-tailed.
We can set Ty as heave-tailed G-ausssian and Ty as a sparse q- Gaussiann in Eq. (3). We choose == 0 for the thin policy and q = 2 for the fat policy, which are standard values (Chow al., 2018; Zhu et al., 2024b). To sample from the gGauussians, we resort to the Generalized Büx-Müller Method (GBMM) to map uniform random variables g-Gaussian variables for all q < 3 (Thistleton et al.,
4

---

Published as a conference paper at ICLR 2025
2007). Specifically, sample u1 U2 Uni form(0, 1) and compute the following:
21 12 (u1) cos (2Tu2).
22 V=-- 2 2 (w1) sin (2Tu2),
(6)
then each of 21, 22 is standard g-Gaussian with new index q (3g' 11/(g' + 1). Often we know the desired q in advance (as is the case for our fat and thin policies q = 2 and q= 0), we simply Trom.NG (H, 2T) where denotes an N-dimensional mean vector and an x N covariance matrix, we sample uniform random vectors U1 U2 Uni form(0, 1)N and compute the transformed entry-wise via the GBMM. The desired random vector is given by μ + E12.
generate variables by using q' = (q 1)/(3 g). To sample from
4.3 Q-EXPONENTIAL AS WEIGHTING COEFFICIENT
Many weighting schemes have been proposed to promote "good actions" in Eq. (3). One prominent example is the exponential advantage function w(s, a) :- exp ((s.a)-V(s) where T is the tem- perature coefficient (Peng al., 2020; Nair et al., 2021). When an action has large advantage value, its log-likelihood will be emphasized and the others implicitly de-emphasized. This scheme is the basis of many ttat-oof-the-an algorithms (Kostrikov et al., 2022; Garg et al., 2023; Xu et al., 2023). However, it is worth noting that due to the non- non-negativ of exp function, all actions will receive nonzero weights regardless of how bad they are.
We propose change the exponential function to q-exp: w(s,a) := exxx,, (Qlaa)-Vis) ( < 1.
with The q-exp weights have the desirable effect of filtering out "bad actions" with low advantages thanks to the sparsity of q-exp:
of [((1+(1-0)) ((s,a) V(s)
+ 20] (I+(--)) (1((L-))(((ss)-V Q(s.a)- v(s)) +
T
T
Since the root does not affect the sign, it is clear that the q-exp weights will truncate actions with advantage ((s,a) V(s) < __-
I-g' -1--. But higher can shrink the magnitude of the advantage function. Since it has been shown by (Zhu et al., 2023) that the role of q and are interchangeable in this regard, we can safely choose q = O. The geexponential advantage weighting has been shown to achieve superior performance the single actor policy setting (Xu et al., 2023; Zhu al., 2024a). The experiments verify the sparse policy performs competitive against the original method.
FtTPO is listed in Algorithm 3. For simplicity, we assume at every policy update step f the action value Q+, parametrized by th2 and state value V., parametrized by Ç. are available. They are trained by the standard critic learning procedures which will be detailed in the appendix. Recall that we set the entropic index 4u O for the weighting coefficient w(s, a). but in principle any Qu will have the filtering property. We initialize by Alg. and sample from the policies by Alg. 2. The loss functions are empirical expectations Es,a over the sampled states and actions.
Algorithm 3: Pat-to-Thin Policy Optimization
Algorithm 1: G-Gasssian Initialization
Input: D, 7, > 0, fw <
Initialize policies by Alg. 1;
Input: Of >> 1 and Qs < Init. Tt. by N2, (1eg, Ei)) per Eq. (5)
while t < 7 do
sample states from dataset D;
Init. Tg by Ng. Ep)
sample actions a from behavior policy Tp:
return To: Tg
compute Qu, (s, a) and Vç, (s);
update dy to Φ++1 by minimizing
-Bxo exp2
Q (s.a) (s)
In n., (als)]
Algorithm 2: G-aussin Sampling
Input: v', N₂ E, Σ sample u1, U2 ~ Uni form(0, 1)
sample b from no, by Alg. 2:
copy More to 140,
compute == /-2 In (uz) cos (2muz) return μ + 1
update 0, to 0,11 by minimizing
E
Te. (Mss
1 - In Fu. (b|s) ne, 8]
2o, (b15)
end
5

---

Published as a conference paper at ICLR 2025
Learaing Curve
Final Policy
FETPO Policy
Actur
Proposal
10
Deadity
Soove
-10
0.6
0.0
0.4
0.4
PYTPO
XQL SQU
IQL
0.2
Step (x10*)
Action
Action
Figure 2: (Left) Scores across the learning. FrTPO attained the highest score, indicating it has learned a safety-aware treatment strategy. The sub-optimal scores the baselines suggest potential unsafe actions. The solid lines show the mean and the ribbons 95% confidence interval. (Middle) The final policy learned by each algorithm. Only FTTPO managed to learn sparse yet stochastic policy tightly concentrating around small band of actions. By contrast, SQL collapsed into delta-like policy due to approximating a sparse policy with Gaussian. Other baselines have overly large randomness. (Right) The FrTPO actor policy learns from the proposal policy by truncating its heavy tails and retaining only the crucial trunk.
Policy Exolution on Synthetic Eavironment FoTPO Proposal and
Policy Evolution on Syuthesir Environment
PtTPO Actor Baselines
XQL
SPOT
PUTPO Actir
FVTPO Proposal
JQL
SQL
Dauity
Dunity
0.5
0.5
Artian
Antin
Stages (Si7eP)
0
Stape
Figure 3:
The policy evolution plots over the first 400 updates. LHS shows that the FITPO proposal policy located high-reward region and then the actor policy concentrated by removing the heavy tails. RHS compares the FrTPO actor against other baselines using the Squashed Gaussian policy. All baselines tended have overly large randomness which can result in dangerous dosage.
EXPERIMENTS
In the experiments, we first verify that FITPO is capable of learning a sparse and safe policy on simulated medicine environment. Then the D4RL Mujoco benchmark, we demonstrate that FtTPO can perform favorably against several popular offline algorithms that by default employ the Gaussian policy. Lastly, we examine in the ablation studies that FrTPO improves on its components. Implementation details are available in Appendix A.
Baselines. We choose several state-of-the-an methods as the baselines to verify that FrTPO can competes favorably against them both in in terms of safety and performance. Specifically, we choose XQL (eXtreme Q Learning) that learns a maximum entropy policy without the entropy bias (Garg et al., 2023); SQL (Sparse Q Learning) that approximates an a-divergence induced sparse policy with the Gaussian (Xu et al., 2023), InAC (In sampee Softmax Actor-Critic) Critic) (Xiao et al., 2023) and IQL (Implicit Q Learning) (Kostrikov et al., 2022) that have been shown perform very competitively. Implementation details such as the Hyperparameter tuning are provided Appendix A.2. For the baselines their published settings used.

---

Published as a conference paper at ICLR 2025
FYTPO
IQL
InAC
TTWACC
AWAC
XQL
SQL
HallCheetah Medium- Meeeett-Expert
HaliCheetah
HallCheetah
Mediiun-eppaay
Medium
100
40
50
20
20
0-
75
Hopper
Hopper
Hopper Medium
Mediumm: Expert
Medium Replay
100
70
70
35
Score
50
75
100
30
75
100
50
75
100
Walker2d
Walker2d MMediuumReelaay
Walker2d
Medium-Expeer
Medium
100
70
50
35
35
0-
75
75
Step (x10')
Figure 4: Comparison between FITPO and the baselines on the MuJoCo. Only FITPO and the environment-specne best performing algorithm are shown with full transparency. The best algorithm picked using the final score. All algorithms are run for seeds. The solid lines show the mean and the ribbons 95% confidence interval.
5.1 SAFETY-CRITICAL TREATMENT SIMULATION
In this paper we consider the case where safety explicitly coded into reward, so higher cumulative reward suggests safety-respen reseetinn policy. However, this may not always be true and extra care required when reward and safety need to be considered separately. We defer such investigation to future study. We opt for the synthetic environment used in (Li et al., 2023) that simulates real circumstances for continuous treatment. Note that only the most challenging environment used here. The environment has 8-dimensional state space and an unbounded action space. Safety is explicitly coded into the reward function, such that danger is defined as cumulative dosage exceeding pre-specified threshold. Note that the states are also conditional on past dosage, therefore danger states can vary depending on the dosage in the beginning states, see Appendix A for detail. According to (Li al., 2023), the optimal dosage is unique, and high dosage leads to excessive toxicity while a lower dosage is ineffective (Lu et al., 2022). We expect that the sparse policy can achieve superior performance by randomly selecting dosage from a sub-interval and by strictly setting the action probability of other dosage to O. The offline dataset contains 50 trajectories each comprising 24 steps.
Figure 2 shows the performance. All the algorithms are run for 10 seeds. The solid lines show the mean and the ribbons 95% confidence interval. It is visible that FTTPO outperformed all the baselines in this safety-aware task. All baselines except the SPOT converged to sub-optimal band of scores as result of their Squashed Gaussian policies. Indeed, from the middle subplot it can seen that the basclines tended to have overly large randomness which sometimes adopt dosage that are not safe. The SQL final policy collapsed as the result of approximating a sparse policy using the Gaussian. By contrast, FrTPO managed to learn a sparse yet stochastic policy tightly concentrating around the optimal action. The last subplot visualizes the synergy in FrTPO: the proposal policy located high heward region, and the actor truncated its heavy tails and kept only the essential trunk.
To better visualize the learning process, Figure shows the policy evolution plots for the first 400 updates. The left hand side shows how the FTT proposal and actor policies worked together in locating
7

---

Published as a conference paper at ICLR 2025
high reward region and then concentrating it by removing heavy tails. The right hand side compares the FtT actor with the baselines that used the default Squashed Gaussian policy. is clear that the baselines tended to have overly large randomness spanning the action range [-2, 2] which can result in dangerous dosage.
5.2
MuJoCo
The D4RL MuloCo suite has been standard benchmark for testing various offline RI. algorithms. In this section we compare FtTPO against the baselines on datasets each corresponding to a behavior policy level and environment combination. We also include Tsallis Advantage Weighted Actor- Critic (TAWAC) and AWAC as baselines, which respectively corrresoond to the proposal policy only and the exponential advantage weighting setting. We run all algorithms for 1 x 10 steps and average over 10 seeds.
Figure shows the full result. Only FtTPO and the environment--pecciic best algorithm from the baselines are shown with full transparency. Others are shown with low transparency for uncluttered visualization. The best baseline is picked by the final score. Recall that XQL, SQL, InAC, TAWAC are very competent on the Mu- JoCo tasks.
Policy Exolution HalRCheetah FETPO Proposal and Actor
PVTPO Astor
PTTP Proposel
Dundty
0.5
Given that is commonly perceived that the sparse policies are inherently handicapped at the pelloration-eploitallie
tradeoff, and no past prior work has demonstrated competitive per- formance to the infinite-support policies, it may come surprise that FtTPO performs favor- ably to or sometimes better than those state-of- the-art algorithms even with sparse actor. This result suggests that full-support random- ness might not be necessary. Rather, learning sparse policy yet not deterministic, with randomness only in fixed region could be more preferable. This can be seen from Figure plotting evolution of the 1st action dimension. See Figure 15 for all dimensions.
Action
-G
300
Stapel (8102)
Figure 5: Policy evolution of the 1st action dimen- sion of FrTPO on HalfCheetah Medium-Expert.
5.3 ABLATION STUDY
In the ablation studies are interested in answering the following questions: (1) more sophisti- cated methods that sample from the learned policy such as SPOT (see equation 7) superior to the simple FtTPO actor KI. minimization? (2) is FtTPO inferior to its proposal only setting, which has been shown to be performant in the MuJoCo benchmark? Note that the proposal-only setting competitive baseline that learns heavy-tailed policy rather than sparse policy. (3) is the heavy-tailed g-Gaussian FiTPO proposal policy better than the Gaussian?
To answer these questions, we evaluate the final scores of the aforementioned combinations and visualize the proportion relative to the FiTPO scores in Figure 6. Here, 100% means exactly the same final score the FtTPO. can be seen that despite the simplicity of FtTPO, (1) the KL minimization actor loss is on par with sophisticated SPOT actor learning loss, with -TTOO-PPOT performed slightly better on only 3 environments. (2) even FrTPO outputs sparse policy, the performance no worse than the proposal policy which yields heavy-tailed policy. (3) overall, FrTPO + Squashed Gaussian policy (FTPP-SSG) performed significantly poorer than FLTPO, with exception only on the Medium-Repla. Hopper environment.
RELATED WORKS
FITPO draws a connection to many existing works. But perhaps the most similar work to ours the Greedy Actor-Critic (GAC) that adopts two-stage policy learning scheme for learning unbiased
8

---

Published as a conference paper at ICLR 2025
Final Performance Propotional to FtTPO (96)
120.0
HalfCheetah
Hopper
Walker2d
FUTPO-SC SG PYTPO-SPOT TAWAC-ITT
80.0
60.0
Medium- Espert
Mediuma-
Medium
Mediun- Expert
Medhum-Replay Replay
Mediun
Medium- Expert
Medium- Replay
Medium
Figure 6: Propotion of final score of FiTPO-SPOT and TAWAC-HT relative to F(TPO The similar score verifies that the simple FrTPO actor loss is robust and does lead to inferior performance compared to more complex actor learning procedure (FTTPO-SPOT) and to the FrTPO proposal only setting (TAWAC-HT). The overall better performance to FrTPO-SG (Squashed Gaussian) validates the heavy-tailed proposal policy.
reward-maximizing solutions. There are other methods allowing sampling from thin policy such as the Supported Policy OpTimization (SPOT), which can be used for learning the thin policy. We compare against it in the ablation study. Finally, the q-exponential distributions which are frequeetly used the physics literature are reviewed.
Two-stage policy optimization.
Greedy Actor-Critic (GAC) is online algorithm that considers the conditional cross-entropy method as its actor loss function (Rubinstein, 1999). GAC maintains a cntropy-regularingg proposal policy provide high-valued actions for updating the unregularized actor interacting with the environment (Neumann et al., 2023). Specifically, GAC optimizes the following actor losses:
LGAC, Proposal (x) = F
amp
[- In (a a (TT(+ss))
a ((k,5))))
CGAC, (0) = E soop
[- [a]ss))
(kkm))
where H (I) denotes the Shannon entropy and a > 0 the coefficient. T(k, nT) denotes the set of the top k% actions: the action values of the sampled actions are computed and ranked, and then the actions of the top 1% are extracted. GAC maximizes entropy-regularized log-likelihood for the proposal policy to facilitate exploration and log-likelihood for the actor. We find the set / (k: T.) does not bring noticeable improvement. Therefore, we simply all samples from the policy.
In-support sampling and updating. Unlike other methods that sample from the dataset or the behavior policy, Supported Policy OpTimization (SPOT) constrains the policy to be close to the behavior policy by sampling from the learned policy itself (Wu et al., 2022). SPOT learns an actor policy a that maximizes action value and log-likelihood of the behavior policy acting regularization term:
LSPOT (Ø) = E S-D I-Q(s, a) - ln Tp(a|ss))
SPOT samples actions from Ty and imposes constraints directly the density. While SPOT can also be used for learning the thin policy learning step. we find that empirically it is on better than the simple KL minimization. We show the results the ablation study Section 5.3.
Sparse and htavy-tailed policies for RL. The Gaussian policy has been the default choice for handling continuous action spaces. Some research works explored heavy-tailed policies such as the Student's s t t (Kobayashi, 2019) or more generally the heavy-tailed g-Gaussian (Zhu et al., 2024b). These attempts showed promising performance the heavy-tailed policies as alternatives to the Gaussian. Li et al. (2023) explored sparse g-Gaussian with & 0 implemented by the kernel embedding and fixed-length trajectories. But they did not investigate the out-of out-of-suppor issue and their implementation required manual design of basis functions specific simple environments. We provide a general framework empowered by deep networks to learn arbitrary q g-aaussians that perform favorably in high-dimensional tasks.
9

---

Published as a conference paper at ICLR 2025
7 CONCLUSION
Sparse policies combined with the offline reinforcement learning framework gave rise to a promising new paradigm. It became possible to obtain safety-aware yet exploratory sparse policy important for realistic systems completely from logged datasets. However, this combination raised a challenge to the existing offline algorithms that require evaluating dataset actions that may fall outside of the sparse policy's support, giving rise to numerical issues and learning failure.
In this paper we proposed Fat-to-Thin Policy Optimization, the first deep offinee learning algorithm addressing this issue. We demonstrated that FTTPO was indeed capable of learning a sparse policy that outperformed the popular algorithms safety-critical treatment simulation and on the standard MuloCo control tasks. Policy evolution plots verified that the ability to tightly concentrate around subset of actions was the key its superior performance. We conducted ablation studies verify that the simplicity and sparse policy did not lead to worse performance than its more complex and heavy-tailed counterparts.
Currently, there are few works that study sparse policies in RL. An open question remains that what specific effect the sparse policies bring, c.g. to quantifying the safety-awarenee in offline RL or to the exploitation-exploratio tradeoff in the online context. This paper focused offline learning where no exploration is required. But an interesting future direction is to investigate the theoretical implications brought by the sparse policies. For example, it would be interesting to analyze how exploitation capability is gained at the cost of exploration by truncating the heavy tails.
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum posteriori policy optimisation. In International Conference Learning Representations, 2018
Alan Chan, Hugo Silva, Sungsu Lim, Tadashi Kozuno, A. Rupam Mahmood, and Martha White. Greedification operators for policy optimization: Investigating forward and reverse kl divergences. Journal of Machine Learning Research, 23(253):1-79, 2022.
Yinlam Chow, Ofir Nachum, and Mohammad Ghavamzadeh. Path consistency learning in Tsallis entropy regularized MDPs. In International Conference on Machine Learning, pp. 979-988, 2018.
Mehdi Fatemi, Taylor W. Killian, Jayakumar Subramanian, and Marzyeh Ghassemi. Medical dead- ends and learning to identify high-risk states and treatments. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Proceedings of the 36th International Conference on Machine Learning, pp. 2052-2062, 2019.
Shigeru Furuichi. On the maximum entropy principle and the minimization of the fisher information in tsallis statistics. Journal of Mathematical Physics, 50:013303, 01 2010.
Divyansh Garg. Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q- q-learning: Maxent RL without entropy. In The Eleventh International Conference on Learning Representations, 2023.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference Machine Learning, pp. 1861-1870, 2018.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of human preferences in dialog, 2020.
Taisuke Kobayashi. Student-t policy in reinforcement learning to acquire global optimum of robot control. Applied Intelligence, 49:4335 2019.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference Learning Representations, 2022.
10

---

Published as a conference paper at ICLR 2025
Kyungjae Lee, Sungjoon Choi, and Songhwai Oh. Sparse markov decision processes with causal sparse tsallis entropy regularization for reinforcement learning. IEEE Robotics and Automation Letters, 3:1466-1473, 2018.
Kyungjae Lee, Sungyub Kim, Sungbin Lim, Sungjoon Choi, Mineui Hong, Jae In Kim, Yong- Lae Park, and Songhwai Oh. Generalized tsallis entropy reinforcement learning and its application to soft mobile robots. In Robotics: Science and Systems XVI, pp. 1-10, 2020.
Yuhan Li, Wenzhuo Zhou, and Ruoqing Zhu. Quasi-optimal reinforcement learning with continuous actions. In The Eleventh International Conference on Learning Representations, 2023.
Liling Lu, Victor Brodzik, Talisa, and Chung-Chou H., Chang. Safe reinforcement learning for sepsis treatment. Master's thesis, 2022.
André F. T. Martins and Ramón F. Astudillo. From softmax to sparsemax: A sparse model of attention and multi-label classification. In Proceedings of the 33rd International Conference on Machine Learning, pp. 1614-1623, 2016.
L F. T. Martins, Marcos Treviso, AntãOnio Farinhas, Pedro M. Q. Aguiar, MAfIiO A. T. Figueiredo, Mathieu Blondel, and Vlad Niculae. Sparse continuous distributions and fenchel- young losses. Journal of Machine Learning Research, 23(257):1-74, 2022.
Hiroshi Matsuzoe and Atsumi Ohara. Geometries for q-exponential families. In Recent Progress in Differential Geometry and Its Related Fields, pp. 55-71, 2011.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. (AWAC): Accelerating online reinforcement learning with offiine datasets, 2021.
Jan Naudts. Deformed exponentials and logarithms in generalized thermostatisties. Physica A- statistical Mechanics and Its Applications, 116:323-3344 2002.
Jan Naudts. The g-exponential family statistical physics. Journal of Physics: Conference Series, pp. 012003, 2010.
Samuel Neumann, Sungsu Lim, Ajin George Joseph, Yangchen Pan, Adam White, and Martha White. Greedy actor-critic: A new conditional cross-entropy method for policy improvement. In The Eleventh International Conference Learning Representations, 2023.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage weighted regression: Simple and scalable off-policy reinforcement learning, 2020.
Reuven Y. Rubinstein. The cross-entropy method for combinatorial and continuous optimization. Methodology And Computing In Applied Probability, 1:127-190, 1999.
John Schulmann. Approximating kl divergence. http:/ http:///oschh..net/lllg////
html, 2020.
Tim Seyde, Igor Gilitschenski, Wilko Schwarting, Bartolomeo Stellato, Martin Riedmiller, Markus Wulfmeier, and Daniela Rus. Is bang-bang control all you need? solving continuous control with bermoulli policies. In Advances in Neural Information Processing Systems, 2021. URL
https: / wwwnreeeeeeeeeeeeeeeeepe eew net forum? : 9BVDIW6_qx7.
William J. Thistleton, John A. Marsh, Kenric Nelson, and Constantino Tsallis. Generalized box-mo mülller method for generating q-gaussian random deviates. IEEE Transactions on Infor- mation Theory, 53:4805-4810, 2007.
C. Tsallis. Introduction Nonextensive Statistical Mechanics: Approaching Complex World. Springer New York, 2009. ISBN 9780387853581.
Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimiza- tion for offline reinforcement learning. In Advances in Neural Information Processing Systems, 2022.
Yifan Wu, George Tucker, and Ofir Nachum, Behavior regularized offline reinforcement learning, 2020.
11

---

Published as a conference paper at ICLR 2025
Chenjun Xiao, Han Wang, Yangchen Pan, Adam White, and Martha White. The in in-sampl soft- max for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023.
Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline RL with no OOD actions: In-sample learning via implicit value regularization. In The Eleventh International Conference on Learning Representations, 2023.
Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healtheare: survey, ACM Comput. Surv., 55, nov 2021.
Lingwei Zhu, Zheng Chen, Matthew Schlegel, and Martha White. Generalized munchausen rein- forcement learning using tsallis divergence. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Lingwei Zhu, Matthew Schlegel, Han Wang. and Martha White. Offline reinforcement learning with tsallis regularization. Transactions on Machine Learning Research, 2024a.
Lingwei Zhu, Haseeb Shah, Han Wang, and Martha White. q-exponential family for policy optimiza- tion, 2024b. URL htl.tt/ps//arre org /ab/////////// 2 248. 07245.
12

---

Published as a conference paper at ICLR 2025
APPENDIX
A IMPLEMENTATION DETAILS
A.1 SAFETY-CRITICAL TREATMENT SIMULATION
We followed (Li al., 2023) on reproducing this environment, see their Appendix. D.1 for detail. The simulated environment uses the following paradigm to generate data. The action space is unbounded. and actions are sampled uniformly from the range (-100, 100). But the policy can select actions on the entire real line. The transition dynamics follow aद+1 (( Σ) with lu 1 and Σ being a pre-specified covariance matrix. Specifically,
of 100
+ 14.s) - exp (- (
+ m4.s)
1+1
exp
100
if 1,2,3,4
cxp
at 100
+ (4:s) + exp
(- (-(10)
100
+ +14.5)
1+1
exp
4: 100
+ [11.5) exp
-
100
of
+ 14.
5,6,78
exp
100 m4.s) exp
af
100
of
+ p4.s)
The reward function is given by
++1
(+1
3
++1
37
1/
9:
2
+
8
1,2
2
1,3
+ 1.4
+2
5,5
2
1,6
2 ( + +t²)
According (Li al., 2023), this environment simulates high-dimension state space and well- separated reward function. The reward function causes the effecet that selecting non-optimal actions will greatly damage the rewards and increases the risk. The environment is tailored examining whether sparse policies could identify sub-regions and avoids sub-optimal actions which greatly damage the performance.
A.2 PARAMETER SELECTION
We provide parameter settings of D4RL experiments in Table and the synthetic environment in Table 2. The environment-specific best hyperparameters are listed in Table 3 and 4, respectively.
In terms of computation time, training FrTPO on average costed around 15 hours for 1 million steps. By contrast, among the baselines InAC took 8.5 hours,IQL 6 hours and TAWAC 6.5 hours. This confirms that by maintaining two policy networks FrTPO costs around double computation time. When interacting with the environment, we sample actions only from the actor policy, which costs similar time to the Gaussian.
B FURTHER RESULTS
Figure 7 visualizes how the sparse policy of F(TPO provides a principled solution to ensuring safety, The left figure compares F(TPO actor against IQL + Gaussian on the HalfCheetah. It is visible that by clipping the policy, IQL Gaussian causes excessive density falling on the clipping boundary due to normalization constraint. As a result, naive clipping can cause the policy to be a Bang- controller (Seyde et al., 2021) that picks actions almost exclusively on the boundary, leading to unsafe actions. The right figure shows the FtTPO actor and reward curves (gray) along with policy evolution. By the definition of reward function, it depends recursively the past actions. Therefore, dangerous action can lead to potential future low reward. is crucial that the agent is capable of utilizing not overly large yet effective dosage at each step to ensure safety. Since the low point of rewards does not reach zero, FtTPO does not incur danger during the evolution.
Figure shows the learning curves of ablation studies. The first shows FtTPO versus FCTPO-SPOT vs FrTPO-SG and the second FiTPO versus TAWAC-HT. Recall that FITPO-SPOR replaces the actor with SPOT for learning sparse policy. FtTPO-SG refers to replacing the proposal policy to the
13

---

Published as a conference paper at ICLR 2025
Policy Evolution on HallCheetak F-TPO Accor and Baselines
Policy Evolution on Synthelic Eavironment PtIPO Actor and Haseines
PTTPO
IQL.
FITPO
Demilt
Autian
Autlen
155
10
(220P)
4
Stags
Figure 7: (Left) FITPO actor compares against IQL + Gaussian on the HalfCheetah. It is visible that by clipping the Gaussian policy, IQL. Gaussian causes excessive density falling the clipping boundary due to normalization constraint. This can lead to choosing actions almost exclusively from the boundary and hence unsafe actions. (Right) FITPO actor reward curves (gray) along with policy evolution. Since the low point of rewards does not reach zero, by the definition of reward function FtTPO does not incur danger during the evolution.
Parameter
Value
FrT: Swept in (le - 3, 3. 1}
Learning rate
Baselines: Swept in (3 x 10 1 10 1,3x 10 4, 4,1 10
FTT: Swept in {I.0, 0.5, 0.01}
Baselines: Same as the number reported in the publication of each algorithm.
Weights
Except in TAWAC medium datasets, the value swept in (1.0, 0.5, 0.013.
Discount rate
0.99
Timeout
1000
Training Iterations
1,000,000
Hidden size of Value network
256
Hidden layers of Value network
2
Hidden size of Policy network
256
Hidden layers of Policy network
2
Minibatch size
256
Adam.
0.9
Adam.32
0.99
Target network synchronization
Polyak averaging with 0.005
Number of seeds for sweeping
5
Number of seeds for the best setting
10
STD sparse policy
Clipped the upper bound of the action space
Table 1: Default parameters and sweeping choices in D4RL.
Squashed Gaussian policy. TAWAC-HT is TAWAC equipped with the heavy-tailed g-Gaussian, which is the proposal policy only setting.
The bar plot Figure 6 was made using the final scores. But it is clear from the curves that FITPO similar to FUTPO-SPOT also in the sense area under curve (AUC), and they are both slightly better than FtTPO-SG overall. For the lower plot, it is clear that the sparse policy of FrTPO leads to roughly the same performance as the heavy-tailed policy. Therefore, performance is not concern when replacing the Gaussian/heav--tailed policies with the sparse one.
We also visualized the policy evolution in HalfCheetah medium-expert dataset in Figure 15. can be seen that the same trend
14

---

Published as a conference paper at ICLR 2025
FUTPO
PUTPO-SPOT
FCTPO-SG
HalfCheetah
HaliChrotah Medium Replay
HallCheetab
Medium Expert
Medium
100
40
20
20
Hopper
Happer
Hopper Medium
Medium- Medlunn-xxpert
Mediumm Mediuu--Repay
100
70
35
Soore
35
50
75
100
25
50
75
100
26
50
75
100
Walker2d
Walker2d Medlium Meellun-Replay
Walker2d
Medlumm Expert
Medium
70
70
35
35
75
73
Step (x10*)
FiTPO
TAWAC-HT
HalfCheetah Medium-Expert
HalfCheetah Medium-Replas
HalCChettl
Medium
100
40
40
300
20
Hopper
Hopper
Hopper Medium
Medium-EBaa Experr
Medium--eene Replay
Score
50
75
100
25
30
75
100
25
50
75
100
Walker2d
Walker2d
Walker2d Medium
Medium-Expert
Medium--Replar
100
70
33
35
Step (x10*)
Figure 8: (Upper) Learning curves of the ablation study FTTPO against FITPO-SPOT and RTPO-SG (Squashed Gaussian). (Lower) Learning curves of the ablation study FtTPO against the proposal only setting TAWAC-HT (Tsallis Advantage Weighted Actor- Citoo-Critii-ee Heatic-Heavy-Taiie
15

---

Published as a conference paper at ICLR 2025
Policy Evalution on HalfChoetah FeTPQ Proposal and Actor
Policy Faolution on HallChhrtahh
PrTPO Progoeal and Actor
PIIPO Astur
Propuse
Propoad
Deesiry
0.5
0.6
30
Steppe (CIDS))
-20
Stapur
0
Dimension
Dimension 2
Policy Evolution on Hal:Choetah FcTPO Proposal and Actor
Policy Evolution HalfCherish PrTPO Propoeal and Actor
PUTPO Astur
PCTPO Propose
FITPO Actoe
FITPO Propoad
Decality
0.5
40
40
Stager (520F)
Aetian
0
Stegps (clott
-0
Dimension
Dimension 4
Pulicy Evolution Hal:Cheetai FcTPO Proposal
Policy Evolution HallCherial
PrTPO Propoeal Actoo
P-TPO Artur
PCTPO Proposed
FYTPO Actoe
PVTPO Propousl
Desaity
40
Aerica
-2p
10
Stapel (XIOP)
40
Strpps (stotty
2
Dimension
Dimension 6
Figure 15: The policy evolution plots over the first 4000 updates on HalfCheetah medium-expert dataset. The FtTPO actor policy concentrated by removing the heavy tails.
16

---

Published as a conference paper at ICLR 2025
Parameter
Value
FtT: Swept in f3 10-4
10 3 10 10-5
Learning rate
Baselines: Swept in (3 x 10 x 10
3
10
1x ,1x
FtT: Swept in (1.0, 0.1,0.1,
Weights
IQL: temperature: f1/3, 0.7}: expectile: (0.7,0.8,0.9)
SQL, XQL: (2,5)
Discount rate
0.9
Timeout
24
Training Iterations
500,000
Hidden size of Value network
256
Hidden layers Value network
2
Hidden size of Policy network
256
Hidden layers of Policy network Minibatch
2
256
Adam, 31
0.9
Adam.32
0.99
Target network synchronization
Polyak averaging with 0.005
Number of seeds for sweeping
5
Number of seeds for the best setting
10
STD sparse policy
Clipped at the upper bound of action space
Table 2: Default parameters and sweeping choices in Synthetic environment.
Dataset
FITPO
FYTPO-SG
PITPO-SPOT
Expert
0.001
0.001
0.001
HalfCheetah-Meedh
0.001
0.001
0.001
MalfCheetah-Meding
0.001
0.001
0.001
Hopper-Medium Expert
0.001
0.001
0.001
Syaper-Medium-Rephee
0.001
0.0003
0.0003
Hopper-Mediunt
0.001
0.001
0.001
malker2d-Medium -ediin--xpprrrr
0.001
0.001
0.0003
Balker2d-MMeiumt Replay
0,001
0.001
0.0003
2alker2d-Medium
0.001
0.001
0.001
ComEnv3-Randan
0,0003
3c-05
3c-05
Table 3: Best for FtTPO and variants.
Dataset
FYTPO
FITPO-SG
RITPO-SPOT
HalfCheetah-Mediim- Expert
1.0
1.0
1.0
AalfCheetah-Mediim---
0.5
0.5
0.01
HalfCheetah-Medhe
0.01
0.01
0.01
Hopper-Medium-
1.0
Hopper-Medium-Roan
1.0
0.5
0.01
Hopper-Medium
0.01
0.01
0.01
Malker22d-Meium--BBll Expertt
1.0
0.01
1.0
0.01
Walker2d-
1.0
1.0
0.5
0.01
0.5
0.5
Table Best temperatures for RTPO and variants.
17


Reviews:
[{"review_id": "LryVp5phFi", "reviewer": "Reviewer_vniq", "summary": "This paper introduces a novel approach for learning sparse policies within the context of offline RL. Sparse policies are crucial for enhancing safety, as they prevent the policies from considering all possible actions, which can reduce potential risks. Unlike existing methods that rely on ad hoc techniques such as reverse KL divergence or random action replacement, this study proposes the Fat-to-Thin Policy Optimization (FtTPO) algorithm. This algorithm leverages the deformed q-exponential function to parameterize policies and employs a greedy two-stage actor-critic optimization approach. The result is a method that achieves desirable sparsity and outperforms existing techniques in simulated deep offline RL tasks.", "questions": "- A primary factor contributing to the success of the proposed algorithm appears to be the use of the weighted q-exponential function in the objective. According to line 241, $ q = 0 $ is used in practice. If so, then how does that affect filtering of \"bad actions\"?\n\n- In algorithm 3, won't copying $\\mu_{\\phi_t}$ to $\\mu_{\\theta_t}$ violate the original choice of the policy parametrization? Is the sampling procedure still valid after copying only parameters and not changing the associated sampling parameters? \n\n\nMinor comments: \n- How to pronounce FtTPO? \n- Typo in line 52 (algorithmsk)", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper presents an innovate approach for learning sparse policies from logged datasets that outperform current offline RL methods.\n- The work is empirical rigorous, with sufficient analysis and abalations of the proposed method on a variety of simulated tasks - including safety-critical  bencahmark and Mujoco. \n- The paper is clearly written and accessible, making it easy to understand the authors' arguments and methodology.", "weaknesses": "- It could be great if the authors can provide insight into why the combination of forward and reverse KL (with the two-stage optimization framework) helps in the first place? \n- Another element that the work is missing is to compare how the proposed method works in low-data regime. In a lot of safety-critical settings, size of the datasets are quite limited, so it would have been nice to have that analysis in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6iKicMZ3iC", "reviewer": "Reviewer_fP7N", "summary": "This paper proposes a novel approach for offline reinforcement learning with sparse policies. Sparse policies have important real-world implications but pose challenges to existing algorithms. The paper presents Fat-to-Thin Policy Optimization (FtTPO), a two-stage learning method that addresses these challenges.", "questions": "NA", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a novel FtTPO algorithm for handling sparse policies in offline reinforcement learning. This is a new combination of ideas as it builds on two-stage actor-critic methods and uses a fat (infinite-support) policy to inform a thin (sparse) policy. It addresses the previously less-studied problem of out-of-support actions in offline learning with sparse policies, providing a solution where prior works relied on ad hoc methods.\n\nThe paper is well-structured, with clear sections for introduction, background, method description, experiments, and related works.", "weaknesses": "The safety benefit of having a thin (sparse) policy isn't directly clear to readers. The authors somehow use 'performance' and 'safety' interchangeably and imply that a higher reward means higher safety.  A very common misunderstanding is that 'dangerous action' means 'higher dosage'. For example, the daily insulin dosage for T1 diabetic patients is around 0.5-1 unit per day. A 0.001 unit dosage can be considered a low dosage. However, if such a dosage is given per 2 min, the accumulative dosage will far exceed body tolerance and cause serious damage to the patient. In fact, 'dangerous states and actions' are precise in medicine. In domains such as dynamic treatment regimes, dangerous states are explicitly defined. I encourage the authors to define safety clearly before using it. Besides, the authors can visualize the occurrence of dangerous states following their policy versus baseline policies to see if safety is indeed enhanced.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o0rVxjYVGI", "reviewer": "Reviewer_NgxJ", "summary": "This paper proposes a novel architecture for learning sparse policies in an offline RL paradigm. Specifically, the authors address issues of domain alignment when training sparse policies, whereby the sparse policy may not support actions within the offline dataset. The induced sparse policy placing measure 0 probability over part of the offline dataset can result in numerical instabilities, for example, when using a log likelihood loss.\nThe proposed architecture requires first learning a “fat” (or non-sparse policy) policy i.e., a policy with at least the support of the offline dataset (the authors propose a Gaussian), which is subsequently refined to a “thin” policy using existing methods for learning sparse policies i.e., using the reverse KL divergence.", "questions": "-\tUnderstanding why the authors feel the fat-to-thin architecture is less complex than existing methods and why it is surprising the a sparse policy competes with existing methods would help in changing my opinion.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "-\tOriginality: To the best of my knowledge the method of induing a sparse policy from one with a larger support has not been explored and as such, the research is original. \n-\tClarity: The paper is well presented and written in the sense that relevant (to the best of my knowledge) existing work is contextualised, the proposed solution is well defined and the paper is generally easy to follow.\n-\tQuality and Significance: The problem is well motivated and thus assuming the solution can be demonstrated to provide significant performance improvements over existing “ad-hoc” approaches, the “fat to thin” architecture  would be of great significance to practioners aiming to learn sparse policies in an offline setting. Furthermore, the breath of evaluation benchmarks used is reasonable in the sense that a reasonable number of challenging offline RL benchmarks have been used.", "weaknesses": "The core weakness of the paper is the experimental approach. The authors consistently draw conclusions that it is “surprising” that the proposed architecture performs so well given the “sparse policy” (referenced in section 5.2) or given the relative simplicity of the proposed architecture (referenced in section 5.3). I disagree with this line of analysis for several reasons:\n-\tA sparse policy working “surprisingly well” is not a substantively validated claim as the authors do not explain why it is surprising. Without this additional analysis, the performance of the proposed method is inconsistent and only obtains notable performance on the Medium-Expert and Medium-Replay HalfCheetah datasets.\n-\tThe authors have not made clear why the proposed architecture is less complex and I would contest, that given two policies are required to be maintained, the proposed architecture is actually more complex than the existing baseline approaches.\nBroadly speaking, the proposed architecture appears to be overly complex and the experimental direction of demonstrating that the architecture performs “at least as well as” existing methods is not strong enough to support this additional complexity. \nAdditionally, it is not completely convincing that the proposed architecture is necessary. Based on Figure 6, the proposal only policy (utilising existing and simpler methods for learning sparse policies) performs on par with the proposed architecture. Furthermore, from Figure 5, the proposal policy appears to already learn a relatively concentrated policy. As such, it is unclear why using only proposal policy and preventing actions being selected outside X standard deviations would not achieve the desired result (assuming an improper distribution of the resulting policy is reasonable).\nOverall, I would encourage the authors to revisit the experiments and strive to obtain stronger results that demonstrate solid benefits of the algorithm. Deriving a novel architecture and having it reasonably converge is no mean feat however, the results presented in the paper are not yet ready for publishing. It might be worth exploring why the proposed architecture performed better on the aforementioned HalfCheetah environments to understand the types of environments where the fat-to-thin model is strongly beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "441oVXsSH7", "reviewer": "Reviewer_jcMU", "summary": "This paper studies offline reinforcement learning in continuous action settings where it is desired to learn a sparse policy (that has zero probability for some actions) that is stochastic. This occurs, for instance, when there is a safety aspect and so some parts of the action space should have zero probability. This paper creates a new algorithm for this setting based on a two-stage actor-critic approach, where a non-sparse proposal policy learns and injects knowledge into a sparse actor. There are quite a few design decisions along the way. The resulting method FtTPO is compared to a large number of existing methods on both safety-motivated and general RL (Mujoco) benchmarks. On the safety benchmarks, it is the strongest, and on general RL it is very competitive.", "questions": "1. For 5.1, how many seeds where used?\n2. What is shown by the shaded area on all graphs?\n3. How does hyperparameter tuning FtTPO compare to the other methods? (I see Tables 1 and 2—I'm interested in a qualitative statement).\n4. How do these methods compare in terms of computational time?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is excellent at contextualizing its contributions relative to existing work.\n- The description of the algorithm, the design decisions, and the motivations behind the design decisions is clear.\n- FtTPO makes design decisions that are different than previous algorithms.\n- Experimental results are strong against a wide variety of relevant benchmarks. There is enough task variation.\n- The demonstration that there are sparse, strong, policies for Mujoco tasks is interesting and provocative.\n- Hyperparameter selection is clearly explained.", "weaknesses": "- The paper is driven purely by intuition and empirical results. There are several cases where there is not a clear understanding of why a certain approach does not work (e.g., reversing the direction of the KL divergence).\n- There is no comparison of computational cost (time) of the different methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)