S$2$AC: Energy-Based Reinforcement Learning with Stein Soft Actor Critic
Authors: Safa Messaoud
Billel Mokeddem
Zhenghai Xue
Linsey Pang
Bo An
Haipeng Chen
Sanjay Chawla
Venue: ICLR 2024 poster (2024)
Keywords: Max-Entropy RL
Entropy
Energy-Based-Models
Primary area: reinforcement learning
TL;DR: We derive a closed for expression of the entropy of Energy-Based-Models and use it to learn multi-modal RL policies.

Abstract: Learning expressive stochastic policies instead of deterministic ones has been proposed to achieve better stability, sample complexity and robustness. Notably, in Maximum Entropy reinforcement learning (MaxEnt RL), the policy is modeled as an expressive energy-based model (EBM) over the Q-values. However, this formulation requires the estimation of the entropy of such EBM distributions which is an open problem. To address this, previous MaxEnt RL methods either implicitly estimate the entropy, yielding high computational complexity and variance (SQL), or follow a variational inference approach that fits simplified distributions (e.g., Gaussian) for tractability (SAC). We propose Sein Soft Actor-Critic (S$^2$AC), a MaxEnt RL algorithm that learns expressive policies without compromising efficiency. S$^2$AC uses parameterized Stein Variational Gradient Descent (SVGD) as the underlying policy. At the core of S$^2$AC is a new solution to the above open challenge of entropy computation for EBMs. Our entropy formula is computationally efficient and only depends on first-order derivatives and vector products. Empirical results show that S$^2$AC yields more optimal solutions to the MaxEnt objective than SQL and SAC in the multi-goal environment, and outperforms SAC and SQL on the MuJoCo benchmark. Our code is available at: https://anonymous.4open.science/r/Stein-Soft-Actor-Critic/

Published as a conference paper at ICLR 2024
S AC: ENERGY-BASED REINFORCEMENT LEARNING WITH STEIN SOFT ACTOR CRITIC
Safa Messaoud-1, Billel Mokeddem 1
Zhenghai Xue Linsey Pang", Bo An
Haipeng Chen Sanjay Chawla
1 Qatar Computing Research Institute, Hamad Bin Khalifa University, School of Computer Science and Engineering, Nanyang Technological University, *SalesForce, 'Skywork AI, Data Science, William & Mary (mmessacd, bmok eddem, schawla @hbku. edu qa, zhenghai00 1001 Be ntu. edu. pangl i nsse@gmaii com, boan@ntu. edu. sg, hchen2 3 wm. edu Equal contrihution Corresponding authors
ABSTRACT
Learning expressive stochastic policies instead of deterministic ones has been proposed to achieve better stability, sample complexity, and robustness. Notably, in Maximum Entropy Reinforcement Learning (MaxEnt RL). the policy is modeled an expressive Energy-Basn -Based Model (EBM) over the Q-values. However, this formulation requires the estimation of the entropy of such EBMs, which is an open problem. To address this, previous MaxEnt RL methods either implicitly estimate the entropy, resulting in high computational complexity and variance (SQL), or follow variational inference procedure that fits simplified actor distributions (e.g., Gaussian) for tractability (SAC). We propose Stein Soft Actor-Critic AC),,, a MaxEnt RL algorithm that learns expressive policies without compromising effi- ciency. Specifically, AC uses parameterized Stein Variational Gradient Descent (SVGD) as the underlying policy. We derive a closed-form expression of the en- tropy of such policies. Our formula is computationally efficient and only depends on first-order derivatives and vector products. Empirical results show that S AC yields more optimal solutions to the MaxEnt objective than SQL and SAC in the multi-goal environment, and outperforms SAC and SQL on the MuJoCo bench- mark. Our code is available at: https: / hut com wtt.////aaaaaaaaaaaaaaaoac SLAC-Energyy B RI th Ste Sof Act Actor-Critie
INTRODUCTION
MaxEnt RL Todorov 2006 Ziebart 2010 Haarnoja al.
2017
Kappen, 2005; Toussaint 2009 Thoodorou et al.
SPAC (ours)
2010
Abdolmaleki et al. 2018 Haarnoja et al. 2018a Sa
Explisi unbrsry evansation
Vieillard et al. (020) has been proposed to address chal-
lenges hampering the deployment of RL to real-world applications, including stability, sample efficiency (Gu
SQL (Haarmog al, ICML 17:
SAC (Haamaga ICML 18)
ct al. 2017), and robustness (Eysenbach & Levine 2022).
Figure 1: Comparing S AC to SQL and SAC. S AC with a parameterized pol- icy is reduced to SAC if the number of SVGD steps is O. SQL becomes equiva- lent to S AC if the entropy is evaluated explicitly with our derived formula.
Instead of learning deterministic policy, in classical
RL (Sutton et al. 1999 Sccullan et all 2017 [Siver et al.
2014; Lillcrap al. 2015). MaxEnt RL learns stochas- tie policy that captures the intricacies of the action space. This enables better exploration during training and even- tually better robustness to environmental perturbations at test time, i.e., the agent learns multimodal action space distributions which enables picking the next best action in case perturbation prevents the execution of the optimal one. To achieve this, MaxEnt RL models the policy using the expressive family of EBMs (LeCun et al., (006). This translates into learning policies that maximize the sum of expected future reward and expected future entropy. However, estimating the entropy of such complex distributions remains an open problem.
To address this, existing approaches either use tricks to go around the entropy computation make limiting assumptions on the policy. This results in either poor scalability or convergence to suboptimal solutions. For example, SQL (Haarooj et al. 2017) implicitly incorporates entropy in the Q-function computation. This requires using importance sampling, which results in high variability and hence poor training stability and limited scalability to high dimensional action spaces. SAC (Haarnoja

---

Published as a conference paper at ICLR 2024
SAC
SQL
S-AC
Figure 2: AC learns a more optimal solution to the MaxEnt RL objective than SAC and SQL. We design a multigoal environment where an agent starts from the center of the 2-d map and tries to reach one of the three goals (G1, G2. and G3). The maximum expected future reward (level curves) the same for all the goals but the expected future entropy is different (higher on the path to G2 G2)G the action distribution π (a]s) is bi-modal on the path to the left (G2 and G3) and unimodal to the right (G Hence, we expect the optimal policy for the MaxEnt RL objective to assign more weights to G2 and G3. We visualize trajectories (in blue) sampled from the policies learned using SAC, SQL, and S AC. SAC quickly commits to a single mode due to its actor being tied to a Gaussian policy. Though SQL also recovers the three modes, the trajectories are evenly distributed. S AC recovers all the modes and approaches the left two goals more frequently. This indicates that it successfully maximizes not only the expected future reward but also the expected future entropy.
et al. Sa), the other hand, follows variational inference procedure by fitting Gaussian distribution to the EBM policy. This enables a closed-form evaluation of the entropy but results in suboptimal solution. For instance, SAC fails in environments characterized by multimodal action distributions. Similar to SAC, IAPO (Marino et al. 202T) models the policy as uni-modal Gaussian. Instead of optimizing a MaxEnt objective, achieves multimodal policies by learning collection of parameter estimates (mean, variance) through different initializations for different policies. To improve the expressiveness of SAC, SSPG (Cetin & Celiktutan 2022) and SAC-NF (Mazoure et al. 2020) model the policy as a Markov chain with Gaussian transition probabilities and as a normalizing flow (Rezende Mohamed 2013), respectively. However, due to training stability issues, the reported results in Cetin & Celiktutan (202)) show that though both models learn multi-modal policies, they fail to maximize the expected future entropy in positive rewards setups.
We propose new algorithm, S AC, that yields more optimal solution to the MaxEnt RL objective. To achieve expressivity, AC models the policy as a Stein Variational Gradient Descent (SVGD) LLu 2017) sampler from an EBM over Q-values (target distribution). SVGD proceeds by first sampling a set of particles from an initial distribution, and then iteratively transforming these particles a sequence of updates fit the target distribution. To compute a closed-form estimate of the entropy of such policies, we use the thange-of-variabe formula for pdfss (Devore et al., 2012). We prove that this only possible due to the invertibility of the SVGD update rule, which does not necessarily hold for other popular samplers (e.g.. Langevin Dynamics (Welling & Teh 201T)). While normalizing flow models (Rezende Mohamed 2015) are also invertible, SVGD- policy is more expressive as it encodes the inductive bias about the unnormalized density and incorporates dispersion term to encourage multi-modallty, whereas normalizing flows encode a restrictive class of invertible transformations (with easy-to- -os--to-ettimate Jacobian determinants). Moreover, our formula is computationally efficient and only requires evaluating first-order derivatives and vector products. To improve scalability, we model the initial distribution of the SVGD sampler and learn its parameters, Lee, mean and standard deviation, end-to-end. We show that this results in faster convergence to the target distribution, i.e., fewer SVGD steps. Intuitively, the initial distribution learns to contour the high-density region of the target distribution while the SVGD updates result in better and faster convergence the modes within that region. Hence, our approach is as parameter efficient as SAC, since the SVGD updates do not introduce additional trainable parameters.
isotropic Gaussian
Note that 2 AC can be reduced to SAC when the number of SVGD steps is zero. Also, SQL becomes equivalent to S AC if the entropy is computed explicitly using our formula (the policy in SQL is an amortized SVGD sampler). Beyond RL, the backbone of AC is a new variational inference algorithm with more expressive and scalable distribution characterized by closed-fon entropy estimate. We believe that this variational distribution can have a wider range of exciting applications.
We conduct extensive empirical evaluations of S AC from three aspects. We start with a sanity check on the merit of our derived SVGD-based entropy estimate on target distributions with known entropy values (e.g., Gaussian) log- ligeliiossssds (e.g., Gaussian Mixture Models) and assess its
2

---

Published as a conference paper at ICLR 2024
sensitivity to different SVGD parameters (kemel, initial distribution, number of steps and number of particles). We observe that its performance depends on the choice of the kernel and is robust to variations of the remaining parameters. In particular, we find out that the kernel should be chosen to guarantee inter-dependencies between the particles, which turns out to be essential for invertibility. Next, we assess the performance of S AC on a multi-goal environment (Haamoja et al. 2017) where different goals are associated with the same positive (maximum) expected future reward but different (maximum) expected future entropy. We show that S AC learns multimodal policies and effectively maximizes the entropy, leading to better robustness to obstacles placed at test time. Finally, we test S AC on the MuJoCo benchmark (Duan et al., 2016). S AC yields better performances than the baselines on four out of the five environments. Moreover, S AC shows higher sample efficiency it tends to converge with fewer training steps. These results were obtained from running SVGD for only three steps, which results in small overhead compared to SAC during training. Furthermore, to maximize the run-time efficiency during testing, we train an amortized SVGD version of the policy to mimic the SVGD- based policy. Hence, this reduces inference to forward pass through the policy network without compromising the performance.
PRELIMINARIES
2.1 SAMPLERS FOR ENERGY-BASED MODELS
In this work, we study three representative methods for sampling from EBMs: (1) Stochastic Gradient Langevin Dynamics (SGLD) & Deterministic Langevin Dynamics (DLD) (Welling & Teh T01
(2) Hamiltonian Monte Carlo (HMC) (Neal et al. 201T), and (3) Stein Variational Gradient Descent (SVGD) (Liu& & Wang 2016 We review SVGD here since it is the sampler eventually use in AC, and leave the rest to Appendix [. SVGD is particle- based Bayesian inference algorithm. Compared SGLD and HMC which have single particle in their dynamics, SVGD operates set of particles. Specifically, SVGD samples a set of m particles {a, {ay)r= from an initial distribution 7º which it then transforms through a sequence of updates fit the target distribution. Formally, at every iteration I, SVGD applies a form of functional gradient descent A that minimizes the KL-divergence between the target distribution p and the proposal distribution q¹ induced by the particles, i.e., the update rule for the 5th particles is: d;
d (af) with fa() = Bo a of log logp(d)) + Vajk(a,a)))-
(1)
Here, e is the step size and k(:,)) is the kernel function, e.g., the RBF kernel: k(a; a,) = exp(lla; - a,ll /112/20"). The first term within the gradient drives the particles toward the high probability regions of p. while the second term serves as repulsive force to encourage dispersion.
2.2 MAXIMUM-ENTROPY RL.
We consider an infinite horizon Markov Decision Process (MDP) defined by a tuple (S,A, p. r). where S is the state space, A is the action space and S x A x S - [0, oc] is the state transition probability modeling the density of the next state 8t+1 E S given the current state 8, E S and action ag E A. Additionally, we assume that the environment emits a bounded reward function r E [I'in: Jmax] at every iteration. We use Pa (s:) and ps (=(s,, ae) denote the state and state-action marginals of the trajectory distribution induced by a policy т (s)) We consider the setup of
continuous action spaces Lazaric et al. (2007): Lee et al. (2018); Zhou & Lu (023). MaxEnt RL [ooorovv 2006; Ziebart 2210:] Rawlik et al. 2012) learns policy T² (aa))))). that instead of maximizing the expected future reward, maximizes the sum of the expected future reward and entropy:
x² = arg max, + Σ, 7 Ess,,, [r(st, at) + (cH(T((((ss)))
(2)
where a: is temperature parameter controlling the stochasticity of the policy and H(x((1ss)) is the entropy of the policy at state sy. The conventional RL objective can be recovered for a = 0. Note
that the MaxEnt RL objective above is equivalent to approximating the policy, modeled as an EBM over Q-values, by a variational distribution (aag(s)) (see proof of equivalence in Appendix D). i.e.
x² arg min, Σ, Es,-p= D (-((s)) (xp(Q(ss.))/a)Z))) ))])/
(3)
where DKL is the KL-divergence and Z is the normalizing constant. We now review two landmark MaxEnt RL algorithms: SAC (Haarnoja et al. 2018a) and SQL (Haaroja et al. 2017)
SAC is actor-critic algorithm that alternates between policy evaluation, i.e., evaluating the Q-values for a policy Te (aclss)):
Qo(si, az) <- r(8,, az) + 7 Eag [Qo(set1, aet1) +
(4)
3

---

Published as a conference paper at ICLR 2024
and policy improvement, L.e., using the updated Q-values to compute better policy:
W = arg m Σ, Bsraa [Qo 8,) + a=((T((-ss:)]]-
(5)
SAC models Te as isotropic Gaussian, Le., To (T(+ss) = (10,009) While this enables computing a closed-form expression of the entropy, it incurs an over-simplification of the true action distribution, and thus cannot represent complex distributions, e.g., multimodal distributions.
SQL goes around the entropy computation, by defining a soft version of the value function V a log (Sa exp Qols, d')) This enables expressing the Q-value (Eq ( independently from the entropy, i.e. Qo(st, ar) = r(s;, at) + yB [=p[V[((++1)) Hence, SQL follows a soft value iteration which alternates between the updates of the "soft" versions of Q and value functions:
Qs(sr, 01) t r(s ar) + /BeessrplVVle (o(+)) + a log (J4 exp (o d')) da)), Vst.
V(s,, ax)
(6)
(7)
Once the Qo and Ve functions converge, SQL uses amortized SVGD Wang & Liu (016 to learn stochastic sampling network fo(E, se) that maps noise samples & into the action samples from the EBM policy distribution x =a(a)())))))))))))))))){))) = exp (2 (s,, ) (*)s)) The parameters @ are obtained by minimizing the loss Js(s;) Dxz (To (-)((- (Q
V (sx)) with respect to 0. Here, Ty denotes the policy induced by fo. SVGD is designed to minimize such KL-divergence without explicitly computing Ty.- In particular, SVGD provides the most greedy direction functional A 8e) (Eq
( which can be used to approximate the gradient DJe/Oa,. Hence, the gradient of the loss Jo with respect to 9 is: ( /00 x Eg AfooE, 8: fo)s, st)/a)] Note that the integral in Eq 7 is approximated via importance sampling, which is known result in high variance estimates and hence poor scalability to high dimensional action spaces. Moreover, amortized generation usually unstable and prone mode collapse, issue similar GANs. Therefore, SQL is outperformed by SAC Haarnoja et al. (2188) on benchmark tasks like MuJoCo.
APPROACH
We introduce AC, a new actor-critic MaxEnt RL algorithm that uses SVGD as the underlying actor generate action samples from policies represented using EBMs. This choice is motivated by the expressivity of distributions that can be fitted via SVGD. Additionally, we show that we can derive a closed-form entropy estimate of the SVGD-induced distribution, thanks the invertibility of the update rule, which does not necessarily hold for other EBM samplers. Besides, we propose a parameterized version of SVGD to enable scalability smooth Q-function landscapes. S AC is hence capable of learning more optimal solution to the MaxEnt RL objective (Eq 2 as illustrated Figure 2
high-dimensional action spaces and non-
3.1 STEIN SOFT ACTOR CRITIC
Like SAC, AC performs soft policy iteration which alternates between policy evaluation and policy improvement. The difference that we model the actor as a parameterized sampler from an EBM. Hence, the policy distribution corresponds to expressive EBM as opposed to a Gaussian.
Critic. The critic's ' parameters & are obtained by minimizing the Bellman loss as traditionally:
o² arg min (( [[Qo ae) ")
(8)
with the target و = (a a.) ) + +7 1. 1)-p. [2((0+11 (1411 + (-(((111))) Here Ø is an exponentially moving average of the value network weights (nnih et al. 2015)
Actor as an EBM sampler. The actor is modeled as sampler from an EBM over the Q-values. To generate set of valid actions, the actor first samples distribution gd (e.g., Gaussian). These particles are then updated over several iterations E L], L). following the sampler dynamics characterized by transformation
of particles {a²) from an initial
i.e., {a/+1) - {a + eh( h (e.g., for SVGD, h = Af in Eg (b). If qd is tractable and h is invertible, it's possible to compute a closed-form expression of the distribution of the particles the 1h iteration via the change of variable
formula Devore et al. (002)): g'(a'ls) = (a
|det(1 + eV at haa",, (
VI E [1, In
this case, the policy is represented using the particle distribution at the final step L of the sampler dynamics, i.e., r[ass) ga(a"1s and the entropy can be estimated by averaging log (a over a set of particles (Section 3.2 We study the invertibility of popular EBM samplers in Section 3.3
4

---

Published as a conference paper at ICLR 2024
Parameterized initialization. To reduce the number of steps required to converge to the tar- get distribution (hence reducing computation cost), we further propose modeling the initial distribution as parameterized isotropic Gaus- sian, i.e., a ~ N [Nu((s), (s)).. The parame- terization trick is then used to express a" as function of f. Intuitively, the actor would learn 0 such that the initial distribution is close to the target distribution. Hence, fewer steps are re- quired to converge, as illustrated in Figure B Note that if the number of steps L = 0, S AC is reduced to SAC. Besides, deal with the non-smooth nature of deep Q-function landscapes which might lead to particle divergence in the sampling process, we bound the particle updates to be within a few standard deviations (t) from the mean of the learned initial distribution, i.e., too S ag S to VI E [1, L]. Eventually, the initial distribution de learns to contour the high-density region of the target distribution and the following updates refine it by converging to the spanned modes. Formally, the parameters I are computed by minimizing the expected KL-divergence between the policy % induced by the particles from the sampler and the EBM of the Q-values:
10
21 10
12
-8
-4
-12 -8
SFAC()))
SPAC
Figure 3: S²AC(,, 0) achieves faster conver- gence to the target distribution (in orange) than S AC)(() by parameterizing the initial distribution Nuuo 0e) of the SVGD sampler.
0* =arg max. E,, E,,,Daa -ay [Qo ab) ] + aES,-D [+((T(()(s))
s.t. to dg S toy, E [].
Here, D is the replay buffer. The derivation is in Appendix Note that the constraint does not truncate the particles as it is not an invertible transformation which then violates the assumptions of the change of variable formula. Instead, we sample more particles than we need and select the ones that stay within the range. We call ACAC(é, 0) and S AC(o) as two versions of AC with/without the parameterized initial distribution. The complete S AC algorithm is in Algorithm of Appendix
3.2 A CLOSED- FORM EXPRESSION OF THE POLICY'S S ENTROPY
A critical challenge in MaxEnt RL is how to efficiently compute the entropy term H( in iterative sampler from the EBM, under certain
Eq 2 We show that, if we model the policy conditions, we can derive a closed-form estimate the entropy at convergence.
Theorem 3.1. Let Fet IR" "R"++ R" be an invertible transformation of the form F(a)= ch(a). ) the distribution obtained from repeatedly applying F to set of samples {a"}
We denote by (aaaa) from an initial distribution (aa)) (a") over I. steps, i.., a² = F o F o.... F(a"). Under the condition < a Maa,)lox
E [1, I.). the distribution of the particles at the T step is:
Tr(Vaih(d') + +OLdL).. log logat (a+) log (a") 1-1
(10)
Here, d is the dimensionality of a, i.e., a E Rd and O(e"dL) is the order of approximation error. Proof Sketch: As F is invertible, we apply the change of variable formula (Appendix C+2 on the trans-
formation Fo .... and obtain: log (a²) = ( - log (T + Vat hr
Un-
der the assumption el/V., a. laa;)lloo < 1. we apply the corollary of Jacobi's formula (Appendix
C33
and get Eq. ITO.. The detailed proof is in Appendix [F Note that the condition cl </|V (a,ha))
can always be satisfied when we choose a sufficiently small step size e, or the gradient of h(aa) is small, i.e., h((a) is Lipschitz continuous with sufficiently small constant.
follows from the theorem above, that the entropy of policy modeled as an EBM sampler (Eq can be expressed analytically
-eeg--geaaa H( (-18) = Bag-ag log d (n (ob1s)) -Bag) -g
(a))
Tr Tr(V (Vo.hhas )) (11)
1=0
In the following, we drop the dependency of the action on for simplicity of the notation.
3.3 INVERTIBLE POLICIES
Next, we study the invertibility of three popular EBM samplers: SVGD, SGLD, and HMC as well as the efficiency of computing the trace, i.e.. (f s)) in Eg I0 for the ones that are invertible. Proposition 3.2 (SVGD invertibility). Given the SVGD learning rate e and RBF kernel k(.,.) with variance a, ife a, the update rule of SVGD dynamics defined in Eq (1 is invertible.
5

---

Published as a conference paper at ICLR 2024
2013
20
Oh, 2000
01
10
100
0096
ONP
Tagat Oharbulinn
(a) Recovering the GT entropy
(b) Effect of on H(p²)
(c) Effect of m and on Hl(9")
Figure 4: Entropy evaluation results.
Proof Sketch: We use the explicit function theorem show that the Jacobian V F,a 8) of the update rule Fa s) is diagonally dominated and hence invertible. This yields invertibility of F(a, s). See detailed proof in Appendix G3
(a*1s) (s) for the SVGD based sampler with an RBF Theorem 3.3. The closed-form estimate of (a4
kernel k(.-)) is
log q (a" ls))= (a (s) mo²
Σ
M(aj, Mlaj.a') ((a -d,) Vo Q(s ()s - -a''
1=0
Here, (.) denotes the transpose of a matrix/vector. Note that the entropy does not depend on any matrix computation, but only on vector dot products and first-order vector derivatives. The proof is in Appendix H.T Intuitively, the derived likelihood is proportional to (1) the concavity of the curvature of the Q-landscape, captured by a weighted average of the neighboring particles' Q-value gradients and (2) pairwise-distances between the neighboring particles (~ ||a! - ad cxp (lla) 12)), i.e., the larger the distance the higher is the entropy. We elaborate on the connection between this formula and non-parametric entropy estimators in Appendix B
Proposition 3.4 (SGLD, HMC). The SGLD and HMC updates are not invertible w.E.t. a.
Proof Sketch: SGLD is stochastic (noise term) and thus not injective. HMC is only invertible if conditioned on the velocity v. Detailed proofs are Appendices G.TG.22
From the above theoretic analysis, we can that SGLD update is not invertible and hence is not suitable as a sampler for S AC. While the HMC update is invertible, its derived closed-form entropy involves calculating Hessian and hence computationally more expensive. Due these considerations, we choose to use SVGD with an RBF kernel as the underlying sampler of S²AC.
RESULTS
We first evaluate the correctness of our proposed closed-form entropy formula. Then we present the results of different RL algorithms on multigoal and MuJoCo environments.
4.1 ENTROPY EVALUATION
This experiment tests the correctness of our entropy formula. We compare the estimated entropy for distributions (with known ground truth entropy or liglliiklhhoods) using different samplers and study the sensitivity of the formula to different samplers' parameters. (1) Recovering the ground truth entropy. In Figure 4al we plot samples (black dots) obtained by SVGD, SGL.D, DLDD and HMC at convergence to a Gaussian with ground truth entropy H(p) = 3.41, starting from the same initial distribution (leftmost sub-figure). We also report the entropy values computed via Eq.III Unlike SGLD, DLD, and HMC, SVGD recovers the ground truth entropy. This empirically supports Proposition 3.4 that SGLD, DLD, and HMC are not invertible. (2) Effect of the kernel variance. Figure 4b shows the effect of different SVGD kernel variances a. where we use the same initial Gaussian from Figure 4a We also visualize the particle distributions after SVGD steps for the different configurations in Figure of Appendix small (e.g., o = 0.1), the invertibility is violated, and thus the estimated entropy is wrong even at convergence. On the other extreme when the kernel variance is too large (e.g.. o 100), i.e., when the particles are too scattered initially, the particles do not converge to the target Gaussian due to noisy gradients in the first term of Eq. The best configurations hence lie somewhere in between (e.g., E (3. 5, 7)). (3) Effect of SVGD steps and particles. Figure 4c and Figure 10b (Appendix. T show the behavior of our entropy formula under different configurations of the number of SVGD steps and particles, on two settings: (i) GMM M with an increasing number of components M, and (ii) distributions with increasing ground truth entropy values, i.e., Gaussians with increasing variances a. Results show that our entropy consistently grows with an increasing M (Figure 4c) and increasing (Figure 10b.. even when a small number of SVGD steps and particles is used (e.g., I. = 10, m 10).
! We can see that when the kernel variance is too

---

Published as a conference paper at ICLR 2024
4.2 MULTI-GOAL EXPERIMENTS
To check AC learns a better solution to the max-entropy objective (Eq 2)) design new multi-goal environment shown in Figure The agent is 2D point mass at the origin trying to reach one of the goals (in red). Q-landscapes are depicted by level curves, Actions are bounded in [- along both axes. Critical states for the analysis are marked with blue crosses. It is built on the multi-goal environment in Haarnoja et al. (017)) with modifications such that all the goals have (i) the same maximum expected future reward (positive) but (ii) different maximum expected future entropy. This is achieved by asymmetrically placing the goals (two goals on the left side and one on the right, leading to a higher expected future entropy The problem setup and hyperparameters are detailed in Appendix (1) Multi-modality, Figure visualizes trajectories (blue lines) collected from 20 episodes of S S*AC(o, 0), S2,()),)) SAC, SQL
Figure 5: Multigoal env.
the left side) while assigning the same final rewards to all the goals.
and SAC-NF (SAC with a normalizing flow policy. Mazoure et al. (0200) agents (rows) at test time for increasing entropy weights (columns). S AC and SQL consistently cover all the modes for all a values, while this is only achieved by SAC and SAC-NF for large a values. Note that, in the case of SAC, this comes the expense of accuracy. Although normalizing flows are expressive enough in theory, they are known to quickly collapse to local optima in practice Kobyzev et al. (2020). The dispersion term AC encodes an inductive bias mitigate this issue. (2) Maximizing the expected future entropy. We also see that with increasing more S AC and SAC-NF trajectories converge to the left goals (Gz) This shows both models learn to maximize the expected future entropy. This is not the case for SQL whose trajectory distribution remains uniform across the goals. SAC results do not show a consistent trend. This validates the hypothesis that the entropy term in SAC only helps exploration but does not lead to maximizing future entropy. The quantified distribution over reached goals is in Figure 12 of Appendix D
(3) Robustnes/aadaptabliiy To assess place obstacle (red bar in Figure 7 on the path to G2-
the robustness of the learned policies, We show the test time trajectories of 20 episodes using S²AC, SAC, SQL and SAC-NF agents trained with different a's s. We observe that, for S AC and SAC-NF, with increasing o, more trajectories reach the goal after hitting the obstacles. This is not the case for SAC, where many trajectories hit the obstacle without reaching the goal. SQL does not manage to escape the barrier even with higher a. Additional results on the (4) effect of parameterization of q² and the (5) entropy's effect on the
learned Q-landscapes are respectively reported Figure and Figure 14 of Appendix 0
20
6 Saoon
SOL
SACAP
Figure 6: SCAC and SAC-NF learn to maxi- mize the expected future entropy (biased towards G2/G3) while SAC and SQL do not. S² con- sistently recovers all modes, while SAC-NF with smaller a's does not, indicating its instability.
Figure 7: A2 and SAC-NF are more robust to perturbations. Obstacle O is placed diagonally at [[1, 1]. Trajectories that did and did not reach the goal after hitting O are in green and red, respectively.
7

---

Published as a conference paper at ICLR 2024
Dig
(a) Hopper-v2
(b) Walkarddv22
(a) HallChootah-v2
(d) AnFiv2
(e) Humanoid-22
Algerise
50L
Slaaa
SPACIE
5P4000)
visue
001G
S4C
SAC
01 Median
(g) IQM
(h) Maan
Opiiaally Gap
( Pipss))
Figure 8: (a)-(e): Performance curves on the MuJoCo benchmark (training). AC outperforms SQL and SAC-NF on all environments and SAC on out of environments. (f)-(i): Comparison of Median, IQM, Mean, and Optimality Gap between S AC and baseline algorithms. (j): The probabilities of S AC outperforming baseline algorithms.
4.3 MUJoCO EXPERIMENTS
We evaluate AC on five environments from MuJoCo [Brockman et al. 2016): Hopper-v2, Walker2d-
v2, HalfCheetah--2, Ant-v2, and Humanoid-v2. As baselines, we use (1) DDPG (Gu al. 2017), (2)
PPO (Schulman et al. 2015). (3) SQL (Haarnoja et al. 2017). (4) SAC-NF (Mazoure et al. 2020).
and (5) SAC (Haarnoja et al. 2018a). Hyperparameters are in Appendix K
(1) Performance and sample efficciecy. We train five different instances of each algorithm with different random seeds, with each performing 100 evaluation rollouts every 1000 environment steps. Performance results are in Figure (aareBaa)-a a))) The solid curves correspond to the mean returns over the five trials and the shaded region represents the minimum and maximum. S AC( 0) is consistently better than SQL and SAC-NF across all the environments and has superior performance than SAC in four out of five environments. Results also show that the initial parameterization was key to ensuring the scalability (S² AC(g) has poor performance compared to SCAC(o, 0)). Figure X f)-(j) demonstrate the statistical significance of these gains by leveraging statistics from the rliable library (Agarwal
al.
202T) which we detail in Appendix K
(2) Run-time. We report the run-time of action selection of SAC, SQL, and AC algorithms in
Hopper Walker2d HalfCheetah Ant
Table SC f) run-time increases linearly with the action space. To improve the scalability. train an amortized version that deploy at test-time, following (Haarnoja et al., 2017 Specifically, train feed-forward deepnet (s 2) to mimic the SVGD dynamics during testing, where is random vector that allows mapping the same state to different particles. Note that we cannot use (s, during training as we need to estimate the entropy in Eq (α.. which depends on the unrolled SVGD dynamics (details in Appendix K). The amortized version S ACO 0, 1b) has a similar run-time to SAC and SQL with slight tradeoff in performance (Figure
Action dim State dim
11
17
17
111
SAC SQL
0.723
0.714
0.731
0.708
0.839
0.828
0.815
0.836
S AC(C, 0)
3.267
4.622
4.583
5.917
S AC(ád 0, +))
0.850
0.817
0.830
0.837
Table I: Action selection run-time on MuloCo.
5 RELATED WORK
MaxEnt RL Tooorov 2006 Ziebart 2010 Rawlik al. 2012) aims learn a policy that gets high rewards while acting as randomly as possible. To achieve this. it maximizes the sum of expected future reward and expected future entropy. It different from entropy regularization (Schulman et al.
2015 O' O'Donoghue et al. 2016: Schulman et al. 2017) which maximizes entropy at the current time step. It is also different from multi-modal RL approaches (Tang & Agrawal 2018) which recover different modes with equal frequencies without considering their future entropy. MaxEnt RL has been broadly incorporated in various RL domains, including inverse RL ZZebart et al.) 2008 Finn
et al. 2016). stochastic control (Rawlik et al. 2012) Toussaint 2009), guided policy search (Levine &
Koltun 2013). and off- off-policy learning (Haamojj et 20.12 b). MaxEnt RL shown to maximize
lower bound of the robust RI. objective ((yysenbach & Levine 2022 and is hence less sensitive
8

---

Published as a conference paper at ICLR 2024
to perturbations in state and reward functions. From the variational inference lens, MaxEnt RL aims to find the policy distribution that minimizes the KL-divergence to an EBM over Q-function. The desired family of variational distributions is (1) expressive enough to capture the intricacies of the Q-value landscape (e.g., multimodality) and (2) has a tractable entropy estimate. These two requirements are hard to satisfy. SAC (Haarnoja et al. 80) uses a Gaussian policy. Despite having a tractable entropy. it fails to capture arbitrary Q-value landscapes. SAC-GMM (Haarojj 2018) extends SAC by modeling the policy as a Gaussian Mixture Model, but it requires an impractical grid search over the number of components. Other extensions include IAPO (Marino et al., 2021)
which also models the policy as uni-modal Gaussian but learns collection of parameter estimates (mean, variance) through different initializations. While this yields multi-modality, it does not optimize a MaxEnt objective. SSPG (Cetin Celiktutan| 2022) and SAC-NF (Mazoure et al.) 2020 respectively improve the policy expressivity by modeling the policy as Markov chain with Gaussian transition probabilities and as a normalizing flow. Due to training instability, the reported multi-goal experiments in (Cetin & Celiktutan| 2022) show that, though both models capture multimodality, they fail to maximize the expected future entropy in positive reward setups. SQL (Haarnoja et al.
2017). 1. on the other hand, bypasses the explicit entropy computation altogether via soft version of value iteration. It then trains an amortized SVGD (Wang & Liu, 2016) sampler from the EBM over the learned Q-values. However, estimating soft value functions requires approximating integrals via importance sampling which is known to have high variance and poor scalability. We propose new family of variational distributions induced by a parameterized SVGD sampler from the EBM over Q-values. Our policy is expressive and captures multi-modal distributions while being characterized by a tractable entropy estimate.
EBMs (LeCun et al. 2006 Wu et al. 2018) are represented as Gibbs densities p(x) = exp E(z) /Z. where [(x) E K is an energy function describing inter-variable dependencies and 2 = exp E E(x) is the partition function. Despite their expressiveness, EBMs are not tractable the partition function requires integrating over an exponential number of configurations. Markov Chain Monte
Carlo (MCMC) methods (Van Ravenzwaaij et al. 2018) (e.g., HMC (Hottman & Gelman 2014).
SGLD (Weling & Teh 2011)) are frequently used to approximate the partition function via sampling.
There have been recent efforts to parameterize these samplers via deepnets (Levy al., 2017 Gong
et al. 2018 Feng et al. 2017) to improve scalability. Similarly to these methods, we propose a
parameterized variant of SVGD (Lu & Wang 2016) as an EBM sampler enable scalability to high- dimensional action spaces. Beyond sampling, we derive a closed-form expression of the sampling distribution as an estimate of the EBM. This yields tractable estimate of the entropy. This is opposed to previous methods for estimating EBM entropy which mostly rely on heuristic approximation, lower bounds Dai et al.) 2017 2019a), or neural estimators of mutual information (Kumar et al. 2019). The idea of approximating the entropy of EBMs via MCMC sampling by leveraging the change of variable formula was first proposed in Dai et al. 2019b). The authors apply the formula to HMC and LD. which, augment the EBM family with the noise or velocity variable for LD and HMC respectively. But the derived log-likelihood of the sampling distribution turns out to be wwwonter--inteeeeeeeee Dounter-intuitivvly- independent of the sampler's 's dynamics and equal to the initial distribution, which is then parameterized using a flow model (details in Appendix B.2). We show that SVGD is invertible, and hence we sample from the original EBM, so that our derived entropy is more intuitive as it depends on the SVGD dynamics. SVGD-augmente RL (Liu & Wang 2016) has been explored under other RL contexts. Liu et al.] (2177) use SVGD to learn distribution over policy parameters. While this leads to learning diverse policies, is fundamentally different from our approach as we are interested in learning a single multi-modal policy with a closed-form entropy formula. Castanet et al.) (023);; Chen et al. (2021) use SVGD to sample from multimodal distributions over goals/tasks. We go beyond sampling and use SVGD to derive a closed-form entropy formula of an expressive variational distribution.
show previously, violate the invertibility assumption. To go around this, they
6 CONCLUSION
We propose S AC, actor-critic algorithm that yields more optimal solution to the MaxEnt RL objective than previously proposed approaches. A²AC achieves this by leveraging a new family of variational distributions characterized by SVGD dynamics. The proposed distribution has high expressivity, i.e., it is flexible enough to capture multimodal policies in high dimensional spaces, and a tractable entropy estimate. Empirical results show that S AC learns expressive and robust policies while having superior performance than other MaxEnt RL. algorithms. For future work, we plan to study the application of the proposed variational distribution to other domains and develop benchmarks to evaluate the robustness of RL agents.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
Bo An is supported by the National Research Foundation Singapore and DSO National Laboratories under the AI Singapore Programme (AISGAward No: AISG2-GC-2223-009). Haipeng Chen is supported by William & Mary FRC Faculty Research Grants.
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenherg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. arXiv preprint (et::00069999 2018.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Mare G Bellemare. Deep reinforcement learning the edge of the statistical precipice. NeurIPS, 2021.
Ahmad and Pi-Erh Lin. A nonparametric estimation of the entropy for absolutely continuous distributions (corresp.). JEEE Trans. Inf. Theory, 1976.
Jan Beirlant and M.C.A van Zuijlen. The empirical distribution function and strong laws for functions of order statistics of uniform spacings. J. Multivar. Anal., 1985.
Jan Beirlant, Edward J Dudewicz, László Györfi, Edward C Van der Meulen, et al. Nonparametric entropy estimation: An overview. IJSRMSS, 1997.
Laura T Bernhofen, Edward Dudewiez, Janos Levendovszky, and Edward C van der Meulen. Ranking of the best random number generators via autropy-unifomme theory. AJMMS, 1996.
Peter Bickel and Leo Breiman. Sums of Functions of Nearest Neighbor Distances, Moment Bounds, Limit Theorems and Goodness of Fit Test. Ann. Probab., 1983.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai Gym. arXiv preprint arXiv: 0100000001540 2016.
Giuseppe Carleo and Matthias Troyer. Solving the quantum many-body problem with artificial neural networks. Science, 2017.
Nicolas Castanet, Olivier Sigaud, et al. Stein variational goal generation for adaptive exploration in multi-goal reinforcement learning. 2023.
Edoardo Cetin and Oya Celiktutan. Policy gradient with serial markov chain reasoning. NeurIPS, 2022.
Jiayu Chen, Yuanxin Zhang, Yuanfan Xu, Huimin Ma, Huazhong Yang, Jiaming Song, Yu Wang, and Yi Wu. Variational automatic curriculum learning for sparse-reward cooperative multi-agent problems. NeurlPS, 2021.
Thomas M Cover. Elements of information theory. John Wiley Sons, 1999.
Noel Cressie. Power results for tests based on high-order gaps. Biometrika, 1978.
Bo Dai, Hanjun Dai, Arthur Gretton, Le Song. Dale Schuurmans, and Niao He. Kernel exponential family estimation via doubly dual embedding. AISTAT, 2019a.
Bo Dai, Zhen Liu, Hanjun Dai, Niao He, Arthur Gretton, Le Song, and Dale Schuurmans. Exponential family estimation via adversarial dynamics embedding. NeurIPS, 2019b.
Zihang Dai, Amjad Almahairi, Philip Bachman, Eduard Hovy, and Aaron Courville. Calibrating mbaed generative adversarial networks. arXiv preprint arXiv: 1702:0170, 2017.
Jay L Devore, Kenneth N Berk, Matthew A Carlton, et al. Modern mathematical statistics with applications. Springer, 2012.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In ICML, 2016.
Edward Dudewicz and Edward C Van Der Meulen. Entropy-based tests of uniformity. JASA, 1981.
10

---

Published as a conference paper at ICLR 2024
Benjamin Eysenbach and Sergey Levine. Maximum entropy rl (provably) solves some robust rl problems. In ICLR, 2022.
Yihao Feng, Dilin Wang, and Qiang Liu. Learning to draw samples with amortized stein variational gradient descent. arXiv preprint arXiv:1707.06660 2017.
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In ICML, 2016.
Ruiqi Gao, Yang Song, Ben Poole, Ying Nian Wu, and Diederik P Kingma. Learning energy-based models by diffusion recovery likelihood. arXiv preprint arXiv:2012.0811 2020.
Xun Gao and Lu-Ming Duan. Efficient representation of quantum many-body states with deep neural networks. Nature communications, 2017.
Wenbo Gong, Yingzhen Li, and José Miguel Hemández-Looato. Meta-learning for stochastic gradient 2018.
mcmc. arXiv preprint arXiv:/806.044
Shixiang Gu, Timothy Lillicrap, Zoubin Ghahramani, Richard E Turner, and Sergey Levine. Q-prop: Sample-efficient policy gradient with an off-policy critic. ICI.R, 2017.
László Györfi and Edward C. van der Meulen. Density-free convergence properties of various estimators of entropy. CSDA, 1987.
Tuomas Haarnoja. Acquiring diverse robot skills via maximum entropy deep reinforcement learning (Ph.D. thesis). University of California, Berkeley, 2018.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In ICML, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In ICML, 2018a.
Tuomas Haarnoja. Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 18/2.05000 2018b.
Peter Hall. On powerful distributional tests based on sample spacings. JMVA, 1986.
Hideitsu Hino and Noboru Murata. A conditional entropy minimization criterion for dimensionality reduction and multiple kernel learning. Neural Comput., 2010.
Matthew Hoffman and Andrew Gelman. The No-U-Turn sampler: Adaptively setting path lengths in hamiltonian monte carlo. JMLR, 2014.
Aleksandr Vasil' evich Ivanov and MN Rozhkova. On properties of the statistical estimate of the entropy of random vector with probability density. Problemy Peredachi Informatsii, 1981.
Harry Joe. Estimation of entropy and other functionals of multivariate density. Ann. Inst. Stat. Math., 1989.
Hilbert J Kappen. Path integrals and symmetry breaking for optimal control theory. JSTAT, 2005.
Ivan Kobyzev, Simon JDD Prince, and Mareus Brubaker. Normalizing flows: An introduction and review of current methods. PAMI, 2020.
Lyudmyla F Kozachenko and Nikolai N Leonenko. Sample estimate of the entropy of random vector. Problemy Peredachi Informatsii, 1987.
Rithesh Kumar, Anirudh Goyal, Aaron Courville, and Yoshua Bengio. Engan: Latent space meme and maximum entropy generators for energy-based models. ICLR. 2019.
Alessandro Lazaric, Marcello Restelli, and Andrea Bonarini. Reinforcement learning in continuous action spaces through sequential monte carlo methods. NewrIPS, 2007.
11

---

Published as a conference paper at ICLR 2024
Erik G Ghamedd-Milee and John W Fisher III. Ica using spacings estimates of entropy. JMLR, 2003.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Fujie Huang. A tutorial on mnegy-bbssed learning. Predicting Structured Data, 2006.
Kyowoon Lee, Sol-A Kim, Jaesik Choi, and Seong-Whin Whan Lee. Deep reinforcement learning in
continuous action spaces: case study in the game of simulated curling. In ICML, 2018.
Sergey Levine and Vladlen Koltun. Guided policy search. ICML, 2013.
Daniel Levy, Matthew Hoffman, and Jascha Sohl-Dickstein. Generalizing hamiltonian monte carlo with neural networks. arXiv preprint 1tt.:::11111.9000000 2017.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, arXiv preprint 1xXiv:550900299) 2015.
Haozhe Liu, Bing Li, Haoqian Wu, Hanbang Liang, Yawen Huang, Yuexiang Li, Bernard Ghanem, and Yefeng Zheng. Combating mode collapse in gans via manifold entropy estimation. AAAI, 2022.
Qiang Liu. Stein variational gradient descent as gradient flow. NeurIPS, 2017.
Qiang Liu and Dilin Wang. Stein variational gradient descent: a general purpose bayesian inference algorithm. In NeurlPS, 2016.
Yang Liu, Prajit Ramachandran, Qiang Liu, and Jian Peng. Stein variational policy gradient. UAI, 2017.
Jan R Magnus and Heinz Neudecker. Matrix differential calculus with applications in statistics and econometrics. John Wiley & Sons, 2019.
Shie Mannor, Dori Peleg, and Reuven Rubinstein. The cross entropy method for classification. In ICMI., 2005.
Joseph Marino, Alexandre Piché, Alessandro Davide lalongo, and Yisong Yue. Iterative amortized policy optimization. NeurIPS, 2021.
Bogdan Mazoure, Thang Doan, Audrey Durand, Joelle Pineau, and R Devon Hjelm. Leveraging exploration in off-policy algorithms normalizing flows. In CoRL, 2020.
Roger Melko, Giuseppe Carleo, Juan Carrasquilla, and Ignacio Cirac. Restricted boltzmann machines in quantum physics. Nature Physics, 2019.
Safa Messaoud. Toward More Scalable Structured Models. Phl) thesis, University of Illinois rrbaaa-Champaige 2021.
Safa Messaoud, David Forsyth, and Alexander G Schwing. Structural consistency and controllability for diverse colorization. In ECCV, pp. 596-612, 2018.
Safa Messaoud, Maghav Kumar, and Alexander G Schwing. Can we learn heuristics for graphical model inference using reinforcement learning? In CVPR Workshops, pp. 766-767, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 2015.
Radford M Neal et al. Meme using hamiltonian dynamics. Handbook of Markov Chain Monte Carlo, 2011.
Brendan O' O'Donoghue, Remi Munos, Koray Kavukcuoglu, and Volodymyr Mnih. Combining policy gradient and gelearnin. arXiv preprint arXiv:1611.0666 2016.
Bo Pang. Tianyang Zhao, Xu Xie, and Ying Nian Wu. Trajectory prediction with latent belief energy-based model. In CVPR, pp. 1114-11824, 2021.
12

---

Published as a conference paper at ICLR 2024
Liam Paninski. Estimation of entropy and mutual information. Neural Comput., 2003.
Fernando Pérez-Crus Estimation of information theoretic measures for continuous random variables. In NeurlPS, 2008.
Konrad Rawlik, Mare Toussaint, and Sethu Vijayakumar. On stochastic optimal control and rein- forcement learning by approximate inference. Proceedings of Robotics: Science and Systems VIII, 2012.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In ICML. PMLR, 2015,
Édgar Roldán, Jérémie Barral, Pascal Martin, Juan MR Parrondo, and Frank Jülicher. Quantifying entropy production in active fluctuations of the hair-cell bundle from time irreversibility and uncertainty relations. New Phys., 2021.
Reuven Y. Rubinstein and Dirk P. Kroese. The Cross Entropy Method: Unifted Approach To Com- binatorial Optimization, Monte-Catee Carlo Simulation (Information Science and Statistics). Springer- Verlag, 2004.
Ruslan Salakhutdinov, Andriy Mnih, and Geoffrey Hinton. Restricted boltzmann machines for collaborative filtering. ICML, 2007.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In ICMI, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06777 2017.
Claude Elwood Shannon. mathematical theory of communication. ACM SIGMOBILE, 2001.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In ICML, 2014.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. NeurIPS, 1999.
Yunhao Tang and Shipra Agrawal. Boosting trust region policy optimization by normalizing flows policy. arXiv preprint arXiv: 1809.0026. 2018.
F.P. Tarasenko. On the evaluation of unknown probability density function, the direct estimation of the entropy from independent observations of a continuous random variable, and the distribution- free entropy test of goodness-of-fit Proceedings of the IEEE. 1968.
Evangelos Theodorou, Jonas Buchli, and Stefan Schaal. A generalized path integral control approach to reinforcement learning. JMLR, 2010.
Emanuel Todorov. Linearly-solvable markov decision problems. In NeurIPS, 2006.
Giacomo Torlai, Guglielmo Mazzola, Juan Carrasquilla, Matthias Troyer, Roger Melko, and Giuseppe
Carleo. Neural-network quantum state tomography. Nature Physics, 2018.
Mare Toussaint. Robot trajectory optimization using approximate inference. In ICML, 2009.
Don Van Ravenzwaaij, Pete Cassey, and Scott D Brown. A simple introduction to markov chain monte-carlo sampling. Psychon. Bull. Rev., 2018
Oldrich Vasicek. A Test for Normality Based on Sample Entropy. JSTOR, 2015.
Nino Vieillard, Olivier Pietquin, and Matthieu Geist. Munchausen reinforcement learning. In NewrlPS, 2020.
Dilin Wang and Qiang Liu. Learning to draw samples: With application to amortized mle for generative adversarial learning. arXiv preprint arXiv: 1.1100122, 2016.
13

---

Published as a conference paper at ICLR 2024
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In ICML, 2011.
Ying Nian Wu, Jianwen Xie, Yang Lu, and Song-Chun Zhu. Sparse and deep generalizations of the frame model. Annals of Mathematical Sciences and Applications, 2018.
Markus Wulfmeier, Peter Ondruska, and Ingmar Posner. Maximum entropy deep inverse reinforce- ment learning. arXiv preprint 1 2015.
Jianwen Xie, Yang Lu, Ruiqi Gao, Song-Chun Zhu, and Ying Nian Wu. Cooperative training of descriptor and generator networks. IEEE PAMI.
Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu. A theory of generative convnet. In ICML, 2016.
Jianwen Xie, Zilong Zheng, Ruiqi Gao, Wenguan Wang, Song-Chun Chun Zhu, and Ying Nian Wu. Generative voxelnet: learning energy-basee models for 3d shape synthesis and analysis. IEEE Transactions on Puttern Analysis and Machine Intelligence, 44(5), 2020.
Jianwen Xie, Zilong Zheng, Xiaolin Fang, Song-Chun Zhu, and Ying Nian Wu. Learning cycle- consistent cooperative networks via alternating meme teaching for unsupervised cross-domain translation. AAAI, volume 35, pp. 10430-10440, 2021a.
Jianwen Xie, Zilong Zheng, and Ping Li. Learning energy-based model with variational auto-encoder as amortized sampler. In AAAI, 2021b.
Jianwen Xie, Yaxuan Zhu, Jun Li, and Ping Li. A tale of two flows: Cooperative learning of langevin flow and normalizing flow toward energy-based based model. arXiv preprint arXiv:2205.06924 2022.
Yang Zhao, Jianwen Xie, and Ping Li. Learning energy- -based generative models via coarse-to-fin to-fine expanding and sampling. In ICLR, 2021.
Zilong Zheng, Jianwen Xie, and Ping Li. Patchwise generative convnet: Training energy-based models from single natural image for internal learning. In CVPR, pp. 2961-2970, June 2021.
Mo Zhou and Jianfeng Lu. Single timescale actor-critic method solve the linear quadratic regulator with convergence guarantees. JMLR, 2023.
Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegic Mellon University, 2010.
Brian Ziebart, Andrew L. Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In AAAI, 2008
14


Reviews:
[{"review_id": "Z717eQAvH7", "reviewer": "Reviewer_mSs2", "summary": "This paper deals with creating an SAC like algorithm, but while allowing\nmultimodality of the policy. This is achieved by using a Stein Variational\nGradient Descent sampler. A neural network parameterizes the mean and variance\nof the initial distribution of the sampler. Moreover, the samples produced\nby the SVGD method are restricted to lie within 3 standard deviations of the\ninitial distribution. The policies entropy can be computed thanks to the\nSVGD sampling process being invertible, similar to how a normalizing flow\nworks.\n\nThere are experiments to check the correctness of the formulation in simple 2D landscape fitting tasks. Moreover, they perform experiments\non 5 MuJoCo benchmark tasks. The performance seems similar to SAC in\n4/5 tasks, and better in one of the tasks.\n\n**Update**\n____________________________\n\nThanks for the extensive update.\n\nMany things improved, but I also still have concerns.\n\nMainly, the rebuttal claims: \"SSPG ... Specifically, the middle plot in Fig. 4(A) is an identical setup to the Multi-goal environment in Fig. 2 in our paper.\" But this does not seem to be true. The task in the SSPG is a bandit problem with only 1 environment step (and the action selects the location on the landscape); hence, there is no \"future entropy\" as there is just one step in the environment. Whereas in the current paper, there is an agent that moves around on the landscape, so the tasks are different. I see no reason why SSPG would also not be able to maximize the future entropy, as it uses a soft critic that includes the future entropy similarly to this paper.\n\nAnother concern is that your re-implementation of SAC-NF does not improve over SAC, whereas it did in the references, both in the SAC and REDQ cases, so perhaps your implementation is not well-performing.\n\nWhile the use of the rliable library was a good step to improve the analysis, there is a fairly large overlap in the error bars for SAC and your newly proposed method. As the number of random number seed experiments per environment was 5, I think this should be increased to reduce the error bars and receive a more reliable result.\n\nBased on the above, I decided to keep my score. What may have changed my assessment would have been performing more experiments so that the statistical significance of the results is clear (this would have increased my score to 5), not making claims that the setup is the same as the one in SSPG, better results for SAC-NF, etc. However, I will increase the contribution rating by 1 point, as I think there are also other contributions in the work.\n\n[3] SSPG: \"Policy Gradient With Serial Markov Chain Reasoning\"", "questions": "How does your method compare to other methods that create multi-modal\npolicies in RL? Please include a comparison.\n\nHow is it related to normalizing flows?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The correctness of the method is properly checked with experiments.", "weaknesses": "- There are many works looking at multimodality in MaxEnt RL that were\nnot discussed or cited. Moreover several of these obtain better\nresults than the current work (although they include additional features).\nFor example one earlier work is \"Boosting trust region policy optimization with\nnormalizing flows policy\" (https://arxiv.org/pdf/1809.10326.pdf). Their equation\n(5) seems similar to equations (10) and (11) in the current paper (it's more\nclear from the proof sketch). Moreover, there are the recent works:\n\"Reparameterized Policy Learning for Multimodal Trajectory Optimization\"\n(https://arxiv.org/pdf/2307.10710.pdf), although it is model-based, and\nalso the work: \"Policy Gradient With Serial Markov Chain Reasoning\"\n(https://openreview.net/forum?id=5VHK0q6Oo4M) that is model-free and obtains\ngood performance (it's also based on the MaxEnt principle); this work\nalso included a comparison with a normalizing flow in their appendix,\nand showed that their method achieves better performance; the experiments\nin this work were also more substantial than the current work.\nFinally, there are some other works that can be found in the related\nworks section of this paper: \"Leveraging exploration in off-policy algorithms via\nnormalizing flows\" (https://arxiv.org/pdf/1905.06893.pdf),\n\"Iterative Amortized Policy Optimization\" (https://arxiv.org/pdf/2010.10670.pdf).\nNone of these works were cited, so I think the literature review was insufficient\n(they cite many works, but miss the most relevant ones).\n\n- The experimental results are not very strong. The improvement over\nSAC is marginal. The other works I referenced above have more substantial experiments,\nand show greater improvements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Iow2XnRqgi", "reviewer": "Reviewer_1y8M", "summary": "This paper proposes a novel maximum entropy reinforcement learning algorithm that leverages Stein Variational Gradient Descent (SVGD) that allows the analytic computation of entropy.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The exposition of the paper is clear. The paper explicitly presents its problem, and the proposed solution explicitly addresses the problem.\n* The proposed method is theoretically sound and enjoys interesting connections to existing algorithms.\n* The paper's claims are well supported by the toy experiments. Experiment results on Mujoco environment seem promising.", "weaknesses": "* The main limitation seems to be the slow inference of SVGD. However, the paper argues that this limitation can be addressed by amortization. It would be better if the main manuscript provided more details on how amortization is performed.\n* It would be great if the paper mentions the scalability of SVGD with respect to dimensionality. How large the method can be scaled in terms of the dimensionality of the problem? Scaling SVGD to higher dimensional spaces seems to be challenging because we need exponentially more particles to represent a distribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YGCPYaHsqA", "reviewer": "Reviewer_ThdK", "summary": "The author present an algorithm to use steins gradient descent to learn stochastic policies using soft-actor critic. The specific advantage is to not rely on variational approaches  (such as Gaussian-like) for the stochasticity of the policy.", "questions": "- How does the armotized version of the method perform on all benchmarks?  Figure 9 (right) only shows one performance curve with no indication which benchmark it is\n- Because the armotized version is much faster and performs similarly I wonder why it is not the main method, but it details are hidden in the appendix. \n- Also I would be curious how this version performs on the toy problems  (Fig. 5-7).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well written\n- Methodology appears sound and novel\n- Related literature is checked and carefully introduced  (e.g. Relations to SQL and SAC) \n- Design choices (such as Parameterized initialization and amortized inference) are valid and thoroughly discussed\n- Theoretical insights, such as a closed form expression of the entropy and invertibility of SVGD  are given", "weaknesses": "Significance:  My biggest reservation for acceptance is significance.  How much does the increased expressiveness of the stochastic policy  (at the cost of computation) matter? The authors could only show significant differences to existing methods on hand-crafted toy problems. For the traditional benchmarks, i.e. Mujoco the method does not appear to perform better than standard approaches such as PPO. The problem here, is not the fact that it not performs better, but that it appears better experiments could be designed to show how large/if there is a significant advantage.\n\nThe general arguments for stochastic policies are, to my knowledge e.g.  \"multi-modality also has application in real robot tasks, as demonstrated in (Daniel et al., 2012) Quote taken from SAC paper. Additionally, as the authors write themselves: \"this enables better explo-\nration during training and eventually better robustness to environmental perturbations at test time, i.e., the agent learns multimodal action space distributions which enables picking the next best action in case a perturbation prevents the execution of the optimal one.\"\n\nIf that is the argumentation I would expect the authors to design experiments under these scenarios that are not toy.  This could have been achieved for instance by  augmenting the MuJoCo tasks with perturbation events that prevent execution of certain actions in certain situations. Then a more expressive stochastic policy could perform better.  For now it makes it hard to asses if the increase time- and algorithmic complexity is worth the cost of obtaining better performing policies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1Roarewq4h", "reviewer": "Reviewer_4Btw", "summary": "This paper argues that in MaxEnt RL, when policy is based on EBMs, estimating the entropy of policies could be an issue. The authors discussed related work, such as SQL and SAC, which have issues to not learn optimal regularized policies. They then proposed S2AC, which learns a more optimal solution to the MaxEnt RL objective. This is achieved by modelling the policy as a Stein Variational Gradient Descent (SVGD) sampler. They show that both SQL and SAC can be recovered with small modifications over S2AC. The authors then conducted experiments to verify the capability of S2AC in estimating entropy, in learning multimodal policies and maximizing entropy, as well as its performance in MuJoCo tasks.", "questions": "1. Does the result apply to other regularizers other than standard entropy?\n2. There are also variants of SAC for discrete settings. Do you think the S2AC would benefit in those scenarios?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Estimating entropy in learning MaxEnt RL is a quite interesting question and the authors showed that this actually matters.\n2. The idea of using the invertibility of samplers and methods are novel to my knowledge.\n3. The experimental results are promising, and they support the proposed methods.\n4. The backbone of S2AC, i.e., the new variational inference algorithm, could be of independent interest.\n5. The paper is well written. I can easily follow the logic.", "weaknesses": "I understand that the paper mostly focuses on discussions with SQL and SAC. However, from reading the paper it seems calculating/estimating entropy of high dimensional policies is an essential part. It could be worth discussing existing literature on entropy estimation and compare the proposed estimation method with them in terms of accurately estimating entropy and computational efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vGf1Wv8U0Q", "reviewer": "Reviewer_4hdR", "summary": "The authors propose a method that combines soft actor-critic and Stein variational gradient descent to solve the reinforcement learning problem with continuous state and action space. The critic is a common Q-learning method. For the actor, the authors learn an initial distribution for the action that is close to the desired one, in order to improve the sufficiency of the algorithm. The authors provided detailed numerical tests to validate their algorithm.", "questions": "1.\tPage 3. The notation \\rho_\\pi. Is it the stationary distribution of the state under \\pi, or the state distribution s_t, which depends on the initial distribution? If it is the former, please clarify. If it is the latter, then \\rho_\\pi should depend on t.\n2.\tPage 3 basic setting. It seems that the authors are considering an RL problem with infinite horizon and without discount. With such a setting, it is not a trivial issue to guarantee the total reward is finite. However, the authors make no comments on this issue.\n3.\tPage 3 eqn (3). The equivalence between MaxEnt RL and (3) is not clear to me. Can you also provide a short proof in the appendix? It is also surprising to me that (3) does not involve \\alpha. (Is there any typo here?) \n4.\tPage 3 eqn (4). -\\alpha should be +\\alpha?\n5.\tPage 4 bottom. In the change of variable formula, there should be an epsilon after I+?\n6.\tPage 5 equation (9). The motivation for the actor update is to minimize the KL divergence between the policy distribution and EBM of Q-values, which is clear to me. But it is not obvious that minimizing this KL divergence is equivalent to (9). Can you also add a short proof in the appendix. Also, I think \\mathcal{D} is used without definition. Q(a|s) looks like a typo, is it Q(a,s) or q(a|s)?\n7.\tPage 5 theorem 3.1. Eqn (10) should not be equal, but an approximation. Maybe it is better to add the order of the approximation error. I think it is O(\\epsilon^2 * L), provided sufficient regularity.\n8.\tPage 6. Prop 3.5 says HMC is not invertible, but the following paragraph says, “While the HMC update is invertible”. Is it a contradiction?\n9.\tRelated work. I think the paper “Single Timescale Actor-Critic Method to Solve the Linear Quadratic Regulator with Convergence Guarantees” (JMLR 2023) could also be added to the related work. The LQR setting also has continuous state and action space, and the actor is a soft policy.\n10.\tPage 17. The first \\epsilon should not appear in the second last line.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The combination of soft actor-critic and Stein variational gradient descent is a good idea. The presentation of the is clear. The numerical results are comprehensive.", "weaknesses": "Certain sections of the article, particularly the basic setting and theorems, are somewhat unclear and could benefit from additional clarification. A more rigorous treatment from the authors in these areas would greatly enhance the overall quality of the paper. Further details can be found in the \"Questions\" section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ABVPi1kFtD", "reviewer": "Reviewer_4Fze", "summary": "The paper proposes a new algorithm, named Stein Soft Actor Critic (S$^2$AC), for policy learning with entropy regularization. The goal is to encourage sufficient reward maximization as well as a reasonable coverage of actions, i.e., return a stochastic policy instead of greedy deterministic one. The algorithm is related to existing methods like SAC and SQL algorithms, however, with improved performance demonstrated in empirical results.", "questions": "Please see the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is mostly well written and easy to follow. The graphical demonstrations help a lot in understanding key concepts. To my knowledge, the proposed algorithm is new, and the empirical results corroborate the improved performance.\n\nI like the idea behind the S$^2$AC algorithm, which is not complicated but to the point.", "weaknesses": "1. Technical issues\n\nEquation (2) seems to be inconsistent with the setting in SQL, as there is no discount factor in Equation (2). Is this a typo (as well as in Equation (3))?\n\nTheorem 3.1 is not rigorous. My main concern is around the condition $\\epsilon \\lVert \\nabla h \\rVert_{\\infty} \\ll 1$. First, this is not a precise statement. Second, under this condition, Equation (10) cannot hold with equality (as already displayed in the appendix, there is an approximately equal argument). My suggestion is to either make a precise statement articulating the relationship between the condition and the final assertion to respect the rigor of a theorem, or make it casual by changing it to be a claim with approximate equality. Afterall, this theorem serves as a benign consequence of preserving the invertibility and as a motivation to properly choose $\\sigma$.\n\nIn Proposition 3.2, it is better to recall the definition of $\\sigma$, as there is an ambiguity of $\\sigma$ being the variance of Gaussian or kernel function.\n\n2. Claims in empirical results\n\nI am not confident in the claim that \"This empirically confirms that SGLD, DLD, and HMC update rules are not invertible\" (beginning of page 7) can be obtained from the entropy is not accurately estimated. Can authors provide more context around it? Moreover, why the target distribution is chosen so, as the mean and covariance are not natural (mean $[-0.69, 0.8]$, and variance ...).\n\nI would suggest to avoid the claim of \"maximizing the future reward and maximizing the future entropy\". The objective in Equation (2) only maximizes the sum of the future reward and future entropy, which is not to maximize both terms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o8QuK9iJ7d", "reviewer": "Reviewer_FpMW", "summary": "This paper proposed a new algorithm for MaxEnt RL that can find a policy with better entropy. The main benefit comes from an explicit evaluation of the entropy term thanks to the closed-form entropy of the based measure and invertiability of the transformation. Experimental results demonstrate the effectiveness of the proposed algorithm on finding maximum entropy policy.", "questions": "* The relationship between the proposed methods and the references I mentioned in the weakness part.\n* Is it possible to directly estimate the entropy of the energy-based policy?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The presentation is easy to follow.\n* The demonstration can clearly show the differences between SQL, SAC and the proposed method.\n* The computation for the invertibility of SVGD can be of independent interest.", "weaknesses": "* I believe that the authors missed an important related work [1]. There are lots of similarities between the proposed method and [1], e.g. using SVGD to approximate the energy-based policy parameterized by the $Q$-function. In fact, if [1] does not amortize the policy, I believe it is nearly identical to the proposed methods.\n\n* I feel there are several modifications that make the proposed method alleviating from the concept of SVGD, e.g. the truncation of the particle update. Furthermore, as there will be some discretization error from the gradient flow of KL on Wasserstein space (which is the motivation of SVGD), I'm thinking if the entropy of the discretized SVGD is a proper estimate of the KL on energy-based policy. I'm also wondering that given the $Q$-function, is it possible to directly estimate the entropy of the energy-based policy, which potentially gets rid of the other issues.\n\n* The experimental results in fact do not beat the baseline a lot. \n\n[1] Liu, Yang, et al. \"Stein variational policy gradient.\" 33rd Conference on Uncertainty in Artificial Intelligence, UAI 2017", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)