Particle Based Stochastic Policy Optimization
Authors: Qiwei Ye
Yuxuan Song
Chang Liu
Fangyun Wei
Tao Qin
Tie-Yan Liu
Venue: ICLR 2022 Submitted (2022)
Keywords: reinforcement learning
deep learning

Abstract: Stochastic polic have been widely applied for their good property in exploration and uncertainty quantification.  Modeling policy distribution by joint state-action distribution within the exponential family has enabled flexibility in exploration and learning multi-modal policies and also involved the probabilistic perspective of deep reinforcement learning (RL). The connection between probabilistic inference and RL makes it possible to leverage the advancements of probabilistic optimization tools.  However, recent efforts are limited to the minimization of reverse KLdivergence which is confidence-seeking and may fade the merit of a stochastic policy.  To leverage the full potential of stochastic policy and provide more flexible property, there is a strong motivation to consider different update rules during policy optimization.  In this paper, we propose a particle-based probabilistic pol-icy optimization framework, ParPI, which enables the usage of a broad family of divergence or distances,  such asf-divergences, and the Wasserstein distance which could serve better probabilistic behavior of the learned stochastic policy. Experiments in both online and offline settings demonstrate the effectiveness of the proposed algorithm as well as the characteristics of different discrepancy measures for policy optimization.

Under review as a conference paper at ICLR 2022
PARTICLE BASED STOCHASTIC POLICY OPTIMIZA- TION
Anonymous authors Paper under double-bling review
ABSTRACT
Stochastic policy has been widely applied for its good property in exploration and uncertainty quantification. Modeling policy distribution by joint state-action distribution within the exponential family has enabled flexibility in exploration and learning mulli--ooda policies and also involved the probabilistic perspective of deep reinforcement learning (RL). The connection between probabilistic inference and RL makes it possible to leverage the advancements of probabilistic optimiza- tion tools. However, recent efforts are limited to the minimization of reverse KL divergence which is confidence-seekingg policy. To leverage the full potential of stochastic policy and provide more flexible property, there is strong motivation to consider different update rules during policy optimization. In this paper, we propose a particle-basee icy optimization framework, ParPI, which enables the usage of broad family divergence or distances, such as f-divergences and the Wasserstein distance which could serve better probabilistic behavior of the learned stochastic policy. Experiments in both online and offline settings demonstrate the effectiveness of the proposed algorithm as well as the characteristics of different discrepancy measures for policy optimization.
and may fade the merit of stochastic
probabilistic pol-
INTRODUCTION
Deep reinforcement learning (DRL) leverages the power of neural-network function approximators and has shown great promise in diverse field, such as control (Lillicrap al., 2015; Andrychow- icz et al., 2020), robotics (Haarnoja et al., 2017; Gu et al., 2017; Plappert et al., 2018). Classic views on the notion of optimality in reinforcement learning state that an optimal policy can be deterministic (Sutton & Barto, 2011). However, recent studies by Daniel et al. (2012); Ziebart (2010) demonstrate that deterministic policy may suffer from several drawbacks, e.g., the multi-modal behaviors have significant applications in real robotic control tasks (Daniel et al., 2012) and inflex- ibility on exploration (Ostrovski et al., 2017). Particularly offline RL, the deterministic policy lacks the structure to manage risk towards uncertainty dynamics. The branch of probabilistic RL then introduced where the optimal solution is stochastic. Probabilistic RL not only enables the stochasticity of the learned policy but also provides different perspective to reveal the connection between reinforcement learning and probabilistic inference (Todorov, 2008), where the optimization procedure of the policy could be considered a statistical divergence minimization procedure towards distribution in the state-action or trajectory space, which is implied by the corresponding optimal RL policy.
Besides the different definitions of optimality and above mentioned priorities, another appealing prop- erty probabilistic reinforcement learning lies in that advanced techniques in approximate inference can then be applied in the optimization for RL. There are two widely applied frameworks, i.e., the pseudo- leddl--i framework and maximum entropy (MERL) framework. The pseudo-likelihodd de- fines the optimality by considering a specified positive and bounded function of reward as the density function. By regressing to the reweighted samples, the not an explicit probabilistic model but also suffers from high variance and poor risk handling (Levine, 2018). MERL is a prominent choice for policy improvement for stochastic policy which tends to maximize the expected reward and the expected conditional entropy. It avoids the drawbacks of pscudo-likelihook that it suffers from high variance and poor risk handling ability (Levine, 2018). In practice, the MERL. is generally implemented with the reverse KI minimization which promotes
based methods not only is

---

Under review as a conference paper at ICLR 2022
the mode-seeking behavior and could fade the merits of a stochastic policy. setting, the mode-secking policy will result in an overly-optmmistic estimation of Q-function (Kumar et al., 2020) which could be problematic when generalizing in the test scenario.
For example, in offline
In this work, we explore the possibility of using a broad class of distribution discrepancies for stochastic policy improvement, and find it is possible by leveraging particle-based methods. By providing better distribution matching methods, our proposed framework, ParPI, enjoys the power of probabilistic RL. Moreover, ParPI is naturally compatible with the offline RL setting. And we show that with specific discrepancy measure selected, ParPI could conjoin the benefit of different policy constraint methods. We firstly demonstrate that using a broader class of distribution discrepancies achieves significant improvements in stabilizing the policy optimization and achieving the state-of- are-art expected returns on high-dimensional and complex continuous control tasks (Todorov et al., 2012). Besides, we conduct extensive experiments in the offlin setup (Yu et al., 2020) to show that the ParPI could be more defensive for optimization thus results in more generalizable optimal policy.
2 PRELIMINARIES
2.1 PROBABILISTIC INFERENCE FRAMEWORK FOR RL
The reinforcement learning (RL) problem can be modeled the following infinite-horizon Markov Decision Process (MDP), which is implied by the tuple (S,A, r, P. 7). We consider the common cases where the state space S and action space A are assumed to be continuous. p(s (s, ae) refers to the transition probability, i.e. the probability density of next state $t+1 given the current state 8, and action dt- when an agent chooses action
state 8, the environment emits reward r(ss, a) SS x A ["min: ('minyYmax] during the transition. The policy distribution T(ass) is presented as distribution over the action space conditioned on any state. The distribution of state-action pairs when navigating the environment with policy π could be defined as Pa (s, a), which is also referred to as occupancy measure (Ho & Ermon, 2016), and the marginal distribution on the state space is pr(ss. With predefined MDP, the notation trajectory Tı refers to the following Markov sequence Ta = [(50: do), ... (ST, ar) generated by interacting with environment under policy π. The discount factor is denoted 7. Standard reinforcement learning seeks to find the optimal policy that could maximize the expected return, and the optimal solution could be deterministic (Sutton & Barto, 2). However, considering the exploration (Schulman et al., 2015), robustness (Ziebart, 2010) and multi- modal objectives (Daniel et al., 2012) in world application scenarios, a stochastic/mulli-moia policy is desired. To address this desideratum, optimal maximum entropy policy (Ziebart et al., 2008; Haarnoja et al., 2017) is defined as:
Σ 1E (ne (r(s,, a.) + [-H[
argmax
π
and the soft Bellman optimality equation (Ziebart, 2010) is defined as:
Q' (st, az) = + 7 (s++)))
which gives the soft Bellman operator T=QT Q²(()) and T=. r(h) + (M and also referred to as Maximum Entropy RM(MERL). Ziebart (2010); Haarnoja et al. (2017) and Haarnoja et al. (2018) show that the target of the policy update is given by the Q-function in an energy- based form, π (a s) x exp( ((ss, a)/a). Due to the intractability in estimating the normalizing constant (s)
exp( ((, aa/a da for continuous or large discrete action space, the energy- based form poses difficulties in utilizing the policy such as sampling or density estimation, so various methods use a standalone parametric policy model n (a (s) to distill from the energy function, and the MERL task is turned into iteratively updating the value model Qo and the policy model TT (a)ss..
3 PARTICLE-BASED POLICY IMPROVEMENT
3.1 DISTRIBUTION MATCHING FOR POLICY IMPROVEMENT
We aim to update a policy model to match a better target p(a]s), which is demonstrated by particles. We now formalize it as distribution matching task under general distribution metrre/dtisrepancy DI-II-- The first problem is that the policy model parameterizes a conditional distribution ( ss). so need to match the target p(als) by minimizing D(T (a]s), p(a s)), however, it is intractable for
2

---

Under review as a conference paper at ICLR 2022
any given s. To make the minimization practical, we note that in almost all modern policy update tasks, we also have an associated state trajectory, e.g., experiences in a replay buffer in off-policy RL, or simulated trajectories in inverse RL (Finn et al., 2016) and demonstrative trajectories in imitation learning (Ho & Ermon, 2016). This state trajectory demonstrates marginal distribution on the state, (s)), which is widely assumed to be stationary among different policies within the same MDP (Tsitsiklis & Van Roy, 1996; Haarnoja et al., 2017; 2018). Hence we instead minimize D(p(s) ( ( ())ss, p( p(a(s))) When the joint distributions match, the conditionals also match. This idea has been exploited in other fields, such as expectation propagation algorithms (Minka, 2001) for Bayesian inference. And we further provide discussions in the Appendix F. Along with the state trajectory, we also have samples/particles from the augmented target, p())p(als)) Particularly for ex- isting MERL methods, they minímize the reverse KI. divergence DRL. (T (m (-[s), p(-1s)) averaged over state distribution p(s) which can be implemented by conducting sampling from the replay buffer: Epts) [DLL (- [s). p(-[s)]. This objective can be reformulated as DKL (p(s)TT S) (a]s), p(s)p(a)s)) corresponding to our proposed method with D instantiated as the reverse KL.
As stressed, this framework allows matching the distribution by minimizing more powerful met- rics/discrepancies that do not have particular propensity, like the reverse KL. To balance utility and implementabiliiyy we consider the f f-divergences Dr and the Wasserstein distance (WD) Dw as two instances of D[-II-] of the proposed framework.
Minimizing f-divergences for policy updates. The f-divergence is defined as Dr(g.p) = (/(F(//)) where F is real-valued convex function with F(1) 0. Although the forward and reverse KL are also f-divergence instances, we do not consider them here since they would introduce the respective propensity of the learned model. Nowozin et al. (2016) show a useful lower bound of an f-divergence that benefits practical optimization:
Dr(a,)) sup (Dr(q P5 1) == E,( - E (FO))
f
where F*(t) == suppp (zt F(x)} is the Fenchel conjugate of F. If the parametric model is defined via a reparameterization h (h) : h = go(e) with e following a fixed and sasplle distribution q(c) like the standard Gaussian, the first term can also be estimated as (ga((() By also parameterizing f in some function class, the supremum can be estimated after optimizing over f, which also tightens the lower bound. Thus minimizing Dr well serves as minimizing Dy-
Minimizing WD for policy updates. The WD is defined as the minimal cost of transferring from one distribution to the other by a probabilistic transferring plan. It is shown to have an optimization utility even when the two distributions do not have overlapping support (Arjovsky et al., 2017). Its formulation under Kantorovichh-Ruinsee duality makes it convenient to optimize:
Dw(g sup 1
E., [J] [/]
For distributions q. P with bounded expectation on a Polish metric space (Villani, 2008, Particular Case 5.) (c.g., the common Euclidean space), where |1-I1z denotes the Lipschitz constant of a function. To enforce the Lipschitz constant constraint, various implementations are proposed, e.g. parameter clipping (Arjovsky et al., 2017), gradient penalty (Gulrajani et al., 2017) and spectral norm / hinge loss regularizations (Miyato et 2018).
Framework 1. The Particle-based ming ((p(s)T (())))) p(s)p(als)) can be used for the sub-task of updating policy in RL, as long the improved policy can be demonstrated by particles. The process is shown in Fig. 1.
Policy Improvement (ParPI) framework,
Remark 1. The ParPl framework can be applied to policy update in (Ziebart, 2010; Haarnoja et al,, 2017) and SAC (Haarnoja et al., 2018). In SQL, once the optimal Q-function Q" is achieved, the optimal policy is shown to be x ={(as) constant. In SAC, it is shown that n' (a)s) := expl (8, a)/ox) /((s) a better policy than m where Q is the Q-function under policy In both cases, the policy update target is given in energy-based form, whose particles can be drawn using MCMC algorithms.
exp(Q" (8, /a)/ (Z')()()))))))))())))))))() where Z²(s) is normalizing
3

---

Under review as a conference paper at ICLR 2022
3.2 PARPI FOR MERL
The particle-based policy improvement (ParPI) framework can be applied to any RL task where a state-action trajectory demonstrates a better pol- icy. We now show in detail how it helps to up- date policy in the MERL framework. As intro- duced, MERL (Ziebart et al., 2008; Ziebart, 2010; Haarnoja et al., 2017; Liu et al., 2017; Haarnoja et al., 2018; Zhang et al., 2018) defines the op- timal policy which also considers the entropy to cover multiple optimalities and encourage explo- ration. Based on the definition, a policy update tar- get is given in the form p(a)s) s) oc exp(Q(s, a)/a), where Q(s, a) is either the optimal Q-function or the Q-function of the current policy, which can be esti- mated from simulated trajectories. These methods update the policy model by minimizing the reverse KL DRL (p()) p(s)p(als)) The merit of it is that estimating the troublesome normalizing constant is not required in optimization, which is also widely exploited in variational inference. However, it has been pointed out (Huszár, 2015; Theis et al., 2016) that minimizing the reverse KL promotes the mode-seeking behavior in finite optimization steps, making (())=a(a))) concentrate to one mode of the target p( p((ss))) This is because a substantial penalty can be incurred when ( puts more mass to where p( ((s))(als) is dilute, but a regular loss in the other way. The behavior is also observed in practice, and there are attempts to ameliorate in other fields, like variational inference (Hernández-Lobes goal of MERL is to capture the diversity and multi-modalit of policy, such behavior would weaken the spirit of MERL.
^
^
Felo
[glojo
Meenax d
Kny
Figure 1: The Particle-based Policy Improve- ment (ParPI) framework. The current Q- function provides better policy in an energy- based form, whose particles can be drawn by MCMC algorithm. The policy model is updated to match the better policy by mini- mizing broader class of distribution metrics using the particles.
et al., 2016; Li & Gal, 2017) As the
The ParPI framework allows MERL to benefit from more powerful metrres/diisrrpanciie as long as the trajectory samples are available. To draw from the target policy, we employ the Langevin dynamics, which only requires an unnormalized density function of the target distribution.
Sampling with Langevin Dynamics. To minimize the general metries/ddisceppaniie above, sam- ples from the target distribution Pr (s)ssp(a)s) are required. Fortunately, this can be done by subse- quently sampling from (s( and p(a[s), and both can be implemented. Samples from (=((s can be drawn by simulating the (s, trajectory following the current policy x{aas) and environment transition. Sampling from p(als) can be done by running dynamics-based MCMC algorithms, which only require the unnormalized density, i.e. exp(Q(s, aa/ar. From various particle-bat
inference algorithms, employ the Langevin dynamics (Roberts et al., 1996; Roberts & Stramer, 2002; Welling & Tch, 2011) due to the low computational cost, whose transition is given by:
(++1) a V log ((a)(( s) + (O, 2=I) ④ + (E/a)Va s) + (O (4)
It has been shown to achieve 5 precision in total variance (Dalalyan, 2017), Wasserstein distance (Dur- mus & Moulines, 2016) and KL divergence (Cheng & Bartlett, 2017) in 0(1/5²) steps for strongly log-concave densities. Note that although we can use MCMC draw samples from the policy target, the sampling is slow in deploying the policy, where we need to run Markov chains for every state along the trajectory, so explicitly modeling policy n. (a(a)s) is still necessary.
Value Function Update. The proposed policy improvement rule described above only requires model for the state-action value function Qo(s, a), so it can be applied with any value function update method. The soft-Q learning method (Haarnoja et al., 2017) aims to update the value function towards optimal by a temporal differencc implementation based on the soft Bellman optimality equation 1:
(Qo (st,a,) a.) (St, az) log E Passipp(eeessss.. (exp (V (s<+1)))) ?] (5)
1o()) aL 2
4

---

Under review as a conference paper at ICLR 2022
The soft actor-critic (SAC) method (Haamojja et al., 2018) instead updates the state-value function model towards the state-action value of the current policy which gives a better policy target as following:
(Vi (s:) -Barnn [Qo (Se, ar) log The (ar [s:)))))
Jv(e) E pps)
1 (Qe (8c, a,) as) + T [v (8c+1)))²
(0) [ Se
2
(6)
And we provide convergence analysis of ParPI in Appendix E.
3.3 PARPI FOR OFFLINE RL
Another important advantage of ParPI lies in that it is naturally compatible with the offline RI. setting. In the offline RL setting, we only have access some static dataset, i.e. D = (s, a, r, 80); which is usually collected by the behavior policy лв. And during the learning procedure, the algorithm could not interact further with the environment. Thus the key challenge for offline RL algorithms is to generalize beyond the state and action support of the offline data. There are two mainstream offline RL algorithms: the value function regularization methods and the policy constrained methods. The value function regularization methods generally refers to the methods which conservatively regularized the estimation of value function either model-free or model based fashion, e.g. MOPO (Yu et al., 2020) and CQL (Kumar et al., 2020). Such methods do not have requirements on the policy improvement procedure, thus we could just directly change the vanilla policy optimization algorithm, e.g. entropy regularized update (Haarnoja et al., 2018), into ParPI as illustrated in Remark 1.
We mainly discuss the relationship between the policy constrained based offline RL algorithm and ParPI. The policy constrained offline RL algorithm tends to stabilize the training by adding constraints into the policy improvement procedure. Without loss of generality, the constrained policy objective could be then formalized as:
T = arg max Fas-Doorraaall ((s,, s.t. D (Mo(als), =T(aal)) E
(7)
Here D).,)) stands for some discrepancy measure between two distributions, and Tt is the behavior policy. Two main types of policy constraints are distributional constraints and support constraint. The distribution constraints (Nair et al., 2020; Wang et al., 2020; Peng et al., 2019; Wu et al., 2019) are generally more strict, which could provide a stable supervision signal yet limit the search space for the learned policy. And they are usually implemented by the minimization of density-based divergence, e.g. KL divergence and f-divergence. In comparison, the support constraints are relatively loose and only require the support of learned distribution to be equal to the behavior distribution. The support constraint enlarges the search space while it could bring unstable optimization when the support distance is large. Correspondingly, the support constraint compares samples regardless of density which is in line with the Integral Probability Metric (IPM), e.g. Maximum Mean Discrepancy and Wasserstein distance.
We then show that within a specific choose of divergence in ParPI, we could conjoin the benefit of both the distribution constraints and support constraint. With the D(,,) .) reverse KL divergence, the Lagrangian duality of Eq. 7 could be formulated as: Rs = arg max E,-D,asx, (anl))))) [Q(s, a)] a DEL (als)))) {a(ss) where a is the Lagrangian multiplier. The above formulation has the closed-form solution T.. (als) (a(ass) ((p, a) aa)a) and could then use ParPI to optimize To towards गे. Particularly, the V ((g(((()))) in Eq. 4 is (Taa(a)s) exp(Q(s, a)/oi in this case and the initial state of langevin dynamics is set as the behavior policy πØ. And the Wasserstein distance is used in particle optimization. Then we have the following fact:
Remark 2. With limited steps of langevin dynamics, ParPI is approximately solving the optimization problem:
Φk+1 = arg min (TT.TT((as)) To(als) exp(Q(s, ()/a) + oW2 (TT, + WW3 (T T62) (8)
Z
Here 7. stands for the normalizing constant and the W2( .) is the W₂ distance.
5

---

Under review as a conference paper at ICLR 2022
We leave the formal discussion of Remark. 2 in the Appendix D. The first two terms in Eq. 8 correspond to the distribution constraint and the support constraint, And the last term is a specific property introduced by ParPI which regularizes the update of policy to be close to the current policy. The intuition ensembles the trust region methods such as TRPO (Schulman et al., 2015) and PPO (Schulman et al., 2017) while the regularization of ParPI is conducted according to Wasserstein distance instead of KL divergence.
3.4 PRACTICAL OPTIMIZATION WITH PARPI
Estimating and Minimizing the Discrepancy Measure The occupancy measure from the current policy can be sampled by firstly sampling states from replay buffer and take action under the current policy (a(a]s) on each state, we carefully overload the notation and refer to the corresponding (s, a) distribution as Pa. As introduced in Eq.4, the occupancy measure implied by the Qe can approximately be retained by finite steps in Langevin Dynamics. In practice, we use the samples from Ps as the initial state, and the corresponding distribution is referred to PQ-
With the samples from Ps and pa available, we parameterize the discriminator((rrttc)) as fw to compute the objective in Eq.2 and Eq.3. More specifically, the Lipschitz constraint in minimizing Wasserstein Distance WD) cases imposed by spectral normalization following (Miyato et al., 2018). The weight matrix W in the discriminator network is regularized as WsN == . W where (W)
denotes the largest singular value of W. Empirically, the amortization optimization procedure of finding the appropriate & of policy model can lead to an intractable problem due to the complexity of the space S x A which leashes the very purpose of function approximation, To counter this obstacle, following the recent success in apprenticeship learning and imitation learning (Ho & Ermon, 2016), we add an entropy regularizer to the objective of the policy(actor) model. Without loss of generality, reparameterize the policy model (()()))))) as neural network transformation:
a2 = (s) + os((s)) 6 ~ p(e),
so (a(a))) = Pe( ( ((=
/ap(s). Before each policy update, we first update the discrimina- tive(critic) function fo for estimating Dp or Dw using particles of the target policy ((a), 8t) by maximizing
R (((e)a((a()) [log Sigm(S a))] Epterp(ale) [log Sigm(1 - [s() log (f (st, (s:) + (s((s)))) - log Sigm(1 (1 fw (st, (,),, for JSD,
Jy(w)
(9)
E (pte)as(ale) a)] - Eptortala (s,
for WD, an example of f-divergence and Sigm is the sigmoid func- tion. After updating w, we update the policy model by minimizing J= == - Ep (s) (To [s)]]
(s (s)) + ()))))))- - fs,
take Censen-Shannon Divergence(ISD)
(-])) +
((p(s)TT((as)) s), pl ())(s)))) as the final objective.
Squashing Correction and Sampling in Latent Space Following the default setting in (Haarnoja al., 2018), model the action distribution with unbounded Gaussian. As the experiment settings always limit the action space in finite interval, we apply tanh on the samples from Gaussian. However, if we conduct the Langevin Dynamics(Eq. 4) in the bounded support, e.g. (-1, we empirically find the method is susceptible to the selection of noise level e and some numerical issues could raise due that the produced samples could be out of the boundary. To alleviate the above issues, we do the MCMC steps in the raw action space before squashing. Here, we slightly overload the notation by denoting the random variable of the raw output as u which has infinite support and the variable of corrected action is tanh(u). With the stationary distribution in the action space as (ep(Q(ss a)/a). we could get the corresponding stationary distribution u space by applying the change of variable formula. The score V, log p(uss) used in Langevin Dynamics, i.e. Eq. 4, is:
V. logp(ule)= v.. tanh /a)) En V. log(1 (ttnh((..)
where u, denotes the i-th element of u.
3.5 DISCUSSION AND RELATED WORKS
The final objective is related to the regularized variants of apprenticeship learning algorithms in imitation learning (Ho et al., 2016; Syed et al., 2008). While in our setting. the corresponding target occupancy measure is defined implicitly also changes along with the optimization of Q-function

---

Under review as a conference paper at ICLR 2022
Algorithm ParPllParticle-Basee Policy Improvement
1: Input: Replay buffer D. Policy model Táa (a s). parameterized Q-function Qo, state function Ve and discrininattor(critie)) fr.
2: Set the step size e, the length of MCMC steps K and the total iterations T.
3: while not converged do
4: Sample batch of triple {(s, (),,ar
) from the replay buffer D.
5: Conducting update on the Qo:
6: 8,
- - Vo, (6,) for E {1,2)
Eq. 5 or Eq. Eq. 6 ParPI + SAC
7:
t + のh Av Jv (1b)
8:
Sample batch of state { from D.
9:
Draw action under current policy and get the state-action pairs {8t, Ong J
10:
Fixing the state, and conduct Langevin dynamics with Qo following equation to acquire the updated state- action pairs {st, dig at=-
w - w A J (w) 11:
Eq. 9.
12:
o - - AM J (6). end while
13:
which indicates more difficult optimization problem. Liu et al. (2017) consider a particle- based policy update, but rather than the target policy, their particles represent the posterior of the policy model parameter, and the update rule is also based on minimizing the reverse KI.. Zhang et al (2018) also propose a particle-based method for policy update by utilizing the minimal movement discretization of the Wasserstein gradient flow to minimize, but still the reverse KL. Many recent methods learn energy-based -based models especially with deep models (Du & Mordatch, 2019; Song & Ermon, 2019; Li et al., 2019a), while we instead learn parametric model to approximate distribution defined by an energy function for efficient prediction. We note that there are other methods to do so like the amortized MCMC methods (Li et al., 2017; 2019b; Feng et al., 2017), but the stationality-ories -rriettd objective may not be efficient for directly matching the target, and we cover scenarios where only the particles are available. Our method is also related to the policy constraint methods in the offline RL scenario. Wu ct al. (2019) and Kumar et al. (2019) proposed to constrained the learned policy with f-divergence IPM, while their methods differ from us in the fact that the target occupancy could be directly acquired from the static dataset and the divergence minimization term is only served as a regularizer.
506
AuPlcol
AuP S/C
Figure 2: Training curves continuous control benchmarks (Todorov al., 2012). Our ParPI framework improves the performance of base RL algorithms SQL and SAC, and achieves the best results consistently across all tasks. We average the return over the past 100 episodes, where the solid lines indicate the mean and shaded areas indicate the standard deviation.
4 EXPERIMENTS
We conduct extensive experiments on both online scenarios and offline scenarios. Besides, we delve deeply into how the component would affect the performance of ParPI task HalfCheetah-v3.
4.1
ONLINE SCENARIOS
We choose five well-known representative benchmark tasks with continuous control (Ant, Hopper, Humanoid, HalfCheetah and Walker2d) from Mujoco (Todorov et al., 2012). The method is compared
7

---

Under review as a conference paper at ICLR 2022
to SQL (Haarnoja et al., 2017) and SAC (Haarnoja et al., 2018), which represents the SoTA model- free oolicy methods for continuous control tasks. We choose Reverse KL (ivergence(RKL) and Jensen-Shannon Divergence(JSD) out of the f-divergence family. For the Wasserstein distance, adopt the hinge loss (Miyato et al., 2018). And the Lipschitz constraint is enforced through gradient penalty (Gulrajani et al., 2017) and spectral normalization (Miyato al., 2018). The proposed method is robust, We keep all of ParPI's 's hyper-parameter constant across all tasks in all domains for simplicity. For ppper-parameters and additional implementation details, please refer to Appendix G. All our experiments are conducted on machines equipped with Nvidia P100 GPUs using five different random seeds.
As shown in Figure 2.we see that ParPI yields significant improvements in both performance and sample efficiency across a wide range of environments. Comparing learning curves of different methods, ParPI improves both SQL. and SAC by a significant boost with almost no hyper-paramen tuning effort. One observation is that the ParPI with SQL shows more improvement then with SAC. The reason lies in the fact that the policy implied by the Q function in SAC is also shifted along with ParPI. And such situation
the training, which could be the unstable element for the optimization could be alleviated by optimizing the Q network for more steps than the policy network in each update. Here we only involve the same training configurations for fair comparison. Interestingly, we found that the Reverse KI. divergence with ParPI has still consistently outperformed the original SQL and SAC which optimized the reverse KI. divergence in the primal form. This could be due to that the intermediate target distribution constructed by MCMC smooths the density ratio estimation procedure (Rhodes et al., 2020) and favors the dual form optimization. The fact further justifies the effectiveness of proposed framework.
4.2 OFFLINE SCENARIOS
To further demonstrate the efficacy of ParPI, we then experiment with the D4RL offline reinforcement learning benchmark (Fu et al., 2020), including three environments(hopper, walker2d, and halfchee- tah), which consist of five different logged data-sets types (random, medium, medium-replay. expert, and crddium-expert)) in total of 15 sub-tasks. These datasets are generated by agent using SAC in rlkit (Pong, 2020), with each dataset containing million time-steps of environment interaction. As we discussed in Sec. 3.3, we study the performance of ParPI from the combination with both value function regularized methods and policy constraint methods. Particularly, we implement ParPI on top of MOPO (Yu et al., 2020) and BRAC (Wu et al., 2019)(see Appendix H), and the discrepancy measure in ParPI is set as wasserstein distance. We follow the same schema as the D4RL paper (Fu et al., 2020) to calculate the normalized return. And the result of 15 sub-tasks is presented in Tabel 1. Compared to the baseline, our approach achieves better performance in most environments, showing the power of ParPI.
4.3 COMPONENT ANALYSIS AND COMPUTATIONAL EFFICIENCY
We conduct ablation study of different discrepancy measures and MCMC parameters on the task HalsCheetah-v3 with the same settings as ParPI + SQL (see 1).
Discrepancy Measure We demonstrate the characteristics of different distribution metrics for policy optimization including: JSD, original WD and the hinge loss. Here we retain all the hyper-parameters of our method and the choice of the metric is the only difference. Interesting results can be observed from Figure 3: while the entropy of learned policy remains almost the same for different metrics, the standard deviation varies. The large standard deviation indicates that the learned policy tends to be more certain on frequently observed state regions and does not mislead in other regions. According to the empirical performance, we provide the following guidelines for the future usage of ParP1 in general the Wasserstein distance could be more desirable discrepancy measure and it also achieves the best performance in most tasks; Jensen- Shannon divergence, though it is reported to show some tendency of mode seeking (Theis et al., 2015), it also effectively ameliorates the problem as it consistently outperforms the exclusive KL. Hence, suggest that ParPI with W-distance could be the generally recommended setting. When the dimension of action is not large, JSD could also show good performance.
Langevin Dynamics We examine the choice of the step length K and step size of MCMC steps. We observe that the sample that have higher Q-value could be fetched along with the MCMC steps,
8

---

Under review as a conference paper at ICLR 2022
Table 1: Results for D4RL benchmarks. Each number is the normalized score computed as (score- random policy score) (expert policy score random policy score) of the policy at the last iteration of training, averaged over random seeds, standard deviation. Results of MOPO (Yu al., 2020), BEAR (Wu et al., 2019) and CQL (Kumar et al., 2020) are reported from their respective papers. Remaining results are taken from the D4RL white-paper (Fu al., 2020).
Offline Types
Tasks
BB
ParPIwD+MOPO
МОPO
CQL
BEAR
BRACV
BC
Walker2d
9.9=3,2
15.2+8.2
13.6+226
7.3
1.9
1.6
random
HalfCheetah
31.5++1.3
36.++11.5
35.4+225
35.4
25.1
31.2
2.1
Hopper
10.2:14.7
11.6::0.8
11.7::0.44
10.8
11.4
12.2
Walker2d
79.1#7.2
39.4#:3.2
17.8=19.3
79.2
59.1
81.1
6.6
medium
HalfCheetah
49.6++2.33
45.8++6.2
42.3++:1.6
44.4
41.7
46.3
36.1
Hopper
89.2.19.5
64.119.3
28.0=1224
52.1
Walker2d
41.8++7.9
38.6+12.6
39.0+:9.66
26.7
19.2
0.9
11.3
medium-replay
HallCheetah
39.91:1.7
54.8+3.11
53.14:2.0
38.6
Hopper
42.1+8.1
64.5+22711
67.5=24.7
48.6
33.7
0.6
11.8
Walker2d
105.613.4
91.21112
44.6=12.9
98.7
40.1
81.6
medium-cxpert
HalfChectah
93,9=12.2
104.1
63.3=38.0
62.4
53.4
41.9
35.8
Hopper
135.2-8.1
78.4 18.1
23.7+66.0
111
96.3
0.8
111.9
Walker2d
108.0=3.1
19.4
153.9
106.1
125.7
expert
HalfChectah
151.4=6.77
109.447.1
104.8
108.2
-1.1
107
Hopper
128.8+2.3
84.1+14.4
109.9
110.3
3.7
109
(a) Entropy of learnt policy
(b) The paper--arameter of Langevin Dynamic analysis.
Figure 3: Study of component of ParPI on HalfCheetah-v3. Every experiment ablation study was conducted using three different seeds.
and a longer MCMC run or larger step size is, the sample has higher Q value as illustrated by Figure 3.
Table 2: Wall-clock Time Comparison Ant
Training Time Efficiency To better understand the trade-off of ParPI on performance and train- ing efficiency, we conduct case study on walk clock time Ant shown in Table. 2. Here seconds to return indicates how much time is needed to gain the corresponding level of re- turns. ParPI would take slight more time as it takes additional time for MCMC steps. Moreover, the wall-clock time usually not the main obstacle comparing to the environment time, i.e., time spent during interaction with the environment, in online settings and sample efficiency for offline settings.
Seconds to Return
1000
2000
3000
ParPIw++SAC SAC
9463
11548
14544
5953
10006
12323
5 CONCLUSIONS AND DISCUSSION
We devise a novel algorithms framework for model-free reinforcement learning, ParPI, particle- based discrepancy/metric minimization framework for policy improvement, which can leverage the full potential of stochastic policy by enable the broad family of divergence and discrepancy, such as f-divergences and IPM-based metric. Our experiments on both online and offline settings show that the baseline algorithms benefit from the ParPI in that they achieve state-of-the-arr performance with less hyper- parameter tuning efforts. ParPI accumulates many desirable properties: robustness, stochasticity, and sample efficiency. Furthermore, ParPI exhibits stabilized training and has been shown to be time and sample efficient as compared to state-of-thee approaches. Moreover, the ParPI is model invariant, allowing it adapt different RI. task settings.
hyper-parameter

---

Under review as a conference paper at ICLR 2022
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In /2th {USENIX) Symposium on Operating Systems Design and Implementation ({OSDI) 16), pp. 265-283, 2016.
OpenAl: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in hand manipulation. The International Journal of Robotics Research, 39(1):3-20) 2020.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXiv:1701.07877 7777777..77. 2017.
Wasserstein GAN. arXiv preprint
Xiang Cheng and Peter Bartlett. Convergence of Langevin MCMC in KL-divergence. arXiv preprint 1rXi 09048, 2017.
Thomas M Cover and Joy A Thomas. Elements of information theory. John Wiley & Sons, 2012.
Arnak S Dalalyan. Theoretical guarantees for approximate sampling from smooth and log-concave densities. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 79(3): 651-676, 2017.
Christian Daniel, Gerhard Neumann, and Jan Peters. Hierarchical relative entropy policy search, In Artificial Intelligence and Statistics, pp. 273-281, 2012.
Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. In Advances in Neural Information Processing Systems 32, pp. 3603-3613, 2019.
Alain Durmus and Eric Moulines. High-dimensional Bayesian inference via the unadjusted Langevin algorithm. arXiv preprint 1rXiv:1605.01530 2016.
Yihao Feng, Dilin Wang, and Qiang Liu. Learning draw samples with amortized Stein variational gradient descent. arXiv preprint arXiv: 1707006666 2017.
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference on machine learning, pp. 49-58, 2016.
Justin Fu.
rail millbberkeleydAA4eeaaauttoooo rail [es1--errkleeyytttleeaaaaa
2020.
URL https:
com/
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep
data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 2020.
Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv: 1et..i002209777 2018
Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In Robotics and Automation (ICRA), 2017 IEEE International Conference on, pp. 1389-3396. IEEE, 2017.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of Wasserstein gans. In Advances in neural information processing systems, pp. 5767-5777, 2017.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In Proceedings of the 34th International Conference Machine Learning- Learning-Volume Volume 70, pp. 1352-1361. JMLR, org, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor- Off-policy maxi- mum entropy deep reinforcement learning with a stochastic actor. arXiy preprint arXiv:1801.0220 2018.
10

---

Under review as a conference paper at ICLR 2022
JM Hernández-Lobato, Y Li, M Rowland, D Hernández-Lobato, TD Bui, and RE Turner. Black-box a-divergence minimization. In Proceedings of the 33rd International Conference on Machine Learning, volume 48, pp. 151 1511-1520. International Machine Learning Society, 2016.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in neural information processing systems, pp. 4565-4573, 2016.
Jonathan Ho, Jayesh Gupta, and Stefano Ermon. Model-free imitation learning with policy optimiza- tion. In International Conference on Machine Learning, pp. 2760-2769, 2016.
Ferenc Huszár. How (not) train your generative model: Scheduled sampling, likelihood, adversary? arXiv preprint arXiv: 151.05501,, 2015.
Richard Jordan, David Kinderlehrer, and Felix Otto. The variational formulation of the fokker-plane Baaker-planck equation. SIAM journal on mathematical analysis, 29(1):1-17, 1998.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Neural Information Processing Systems (NeurIPS), 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.0477 2020.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv:1805.0000000 00909, 2018
Chongxuan Li, Chao Du, Kun Xu, Max Welling, Jun Zhu, and Bo Zhang. To relieve your headache of training an mrf, take advil. In International Conference on Learning Representations, 2019a.
Chunyuan Li, Ke Bai, Jianqiao Li, Guoyin Wang. Changyou Chen, and Lawrence Carin. Adversarial learning of a sampler based on an unnormalized distribution. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 3302-330 3311. PMLR, 2019b.
Yingzhen Li and Yarin Gal. Dropout inference bayesian neural networks with alpha-divergences. In Proceedings of the 34th International Conference on Machine JMLR. org, 2017.
70, pp. 2052-2061.
Yingzhen Li, Richard E Turnerr and Qiang Liu. Approximate inference with amortised MCMC. arXiv preprint arXiv: htth::/702084444 2017.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint (rXXv::509.02977 2015.
Chang Liu, Jingwei Zhuo, and Jun Zhu. Understanding meme dynamics as flows the wasserstein space. In International Conference on Machine Learning, pp. 4093-4103. PMLR, 2019.
Yang Liu, Prajit Ramachandran, Qiang Liu, and Jian Peng. Stein variational policy gradient. In 33rd Conference on Uncertainty in Artificial Intelligence, UAI 2017,
Thomas Peter Minka. A family of algorithms for approximate Bayesian inference. PhD thesis, Massachusetts Institute of Technology, Cambridge, 2001.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957. 2018.
Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael Jordan, al. Ray: A distributed framework for emerging {Al} applications. In 13th { {USENIX) Symposium on Operating Systems Design and Implementation (OSDI)) 18), pp. 561-577, 2018.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.093559 2020.
11

---

Under review as a conference paper at ICLR 2022
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In Advances in neural information processing systems, pp. 271-279, 2016.
Georg Ostrovski, Mare G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International conference on machine learning, pp. 2721-2730. PMLR, 2017.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weigghted regression Simple and scalable off-policy reinforcement learning. arXiv preprint 10X10:19100001777 2019, 2019.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goal reinforcement Iearn- ing: Challenging robotics environments and request for research. arXiv preprint arXiv: 2018.
Vitchyr Pong. vitchyr/rlkit, 2020. URL wwtpttttp//// / thub. wwt//oittttrr//veet..ccc
Benjamin Rhodes, Kai Xu, and Michael U Gutmann. Telescoping density-ratio estimation. arXiv
preprint arXiv:2006. 12X2: 2020.
Gareth O Roberts and Osnat Stramer. Langevin diffusions and Metropolis Hastting algorithms. Methodology and computing in applied probability, 4(4):337-337, 2002.
Gareth O Roberts, Richard Tweedie, et al. Exponential convergence of Langevin distributions and their discrete approximations. Bernoulli, 2(4):341-363, 1996.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint vrXiv:1707.06344 2017.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, pp. 11895-11907, 2019.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. 2011.
Umar Syed, Michael Bowling, and Robert E Schapire. Apprenticeship learning using linear program- ming. In Proceedings of the 25th international conference Machine learning, pp. 1032-1039, 2008.
L Theis, A van den Oord, and M Bethge. A note on the evaluation of generative models. In International Conference on Learning Representations (ICLR 2016), pp. 1-10, 2016.
Lucas Theis, Aäron van den Oord, and Matthias Bethge. note on the evaluation of generative models. arXiv preprint
2015.
Emanuel Todorov. General duality between optimal control and estimation. In 2008 47th IEEE Conference on Decision and Control, pp. 4286 4292. IEEE, 2008
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
JN Tsitsiklis and B Van Roy. An analysis of denforel- learning with function approximation- technical. Report LIDS-P-2322 Laboratory for Information and Decision Systems, Massachusetts Institute of Technology, Tech. Rep., 1996.
Cédric Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008.
Ziyu Wang, Alexander Novikov, Konrad Zomn, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Josh Merel. Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. arXiv preprint arXiv:2006. 1344, 2020.
12

---

Under review as a conference paper at ICLR 2022
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-1I), pp. 681-688, 2011.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint 1rXiv:/911.13350 2019.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-basee offline policy optimization. arXiv preprint arXiv:2005./3239,
2020.
Ruiyi Zhang, Changyou Chen, Chunyuan Li, and Lawrence Carin. Policy optimization wasserstein gradient flows. In International Conference on Machine Learning, pp. 5737-5746. PMLR, 2018.
Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. 2010.
Brian Ziebart, Andrew Maas, Andrew Bagnell, and Anind Dey. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433-1438. Chicago, II., USA, 2008
13

---

Under review as a conference paper at ICLR 2022
A PARTICLE IMPROVEMENT WITH LANGEVIN DYNAMICS
As in the practical implementation, only the finite-step Langevin dynamics could be conducted. Fortunately, the property of MCMC guarantees the improvement of the particle distribution. With the denotes the unique stationary distribution, e.g., the distribution implied by the Q Q-function in SQL and SAC. qe and qt-1 refer to the distribution which implicitly implied by an initial distribution and or / 1 steps Langevin dynamics, the following monotonic property is satisfied:
DEL (qe lp) - DKL (9-11p))pp)
(10)
And q, converges the stationary distribution p as - oc. The above proposition is the direct result of the following lemma, we also provide the proof here for the completeness.
Lemma 1. Cover & Thomas (2012) Let and be two distributions for Zn- Let qt and rt be the corresponded distributions of state z, at time induced by the transition kernel K. Then Dxl(u 2 Dxulac+1lleg
(+41) [++1] for all 2 0.
Proof.
Egg log (e(
DRI [xlleel
rx( (z()
qe (0 (z)
T (x.) (
log
re (Z)))((22111 [c)
log 90+1 (==1(
Eas (ze (eee)))etttsi (21le4)))))
re+ (2+1
Dkll9et//re441] + Egg DKL
B POLICY IMPROVEMENT THEOREM FOR PARPI
Given the objective function as the expected discounted sum of rewards, the policy improvement theorem corresponds to how policies can be improved monotonically. Similar theorems have been derived under the maximum entropy framework, i.e., SAC Haarnoja et al. (2018) and SQL Haarnoja et al. (2017). In this section, shows that optimization of the policy with ParPI does not hurt the monotonic property.
B. PARPISOL
In the SQL-version ParPI, considering the original policy improvement theorem of SQL: Theorem 1. Haarnoja et al. (2017) Given policy म, define a new policy ă as a(-ls) x exp (s,of((s.))) Assume that throughout our computation, is bounded and f exp(Q(ss da is bounded for any (s. (for both and i). Then we have fh (s,a)) a) 2 Qrofe (s. as a.
Vs
It should be notieed that there is constraint the policy optimization in Theorem. 1, hence could directly replace the original reverse KL minimization with ParPI without hurting the policy improvement theorem.
B.2 PARPISAC
In the situation of ParPIsac, need to generalize the previous objective: exp (Qp (St. ))
(Ø) ES,~D DKL
noe (-1s:) ||
Zo (Sx)
(z(0) E D (T (-|st) ||
exp (Qa (st, -)) Zo (St)
To make it clear, follow the proof provided in Haarnoja et al. (208)):
14

---

Under review as a conference paper at ICLR 2022
Lemma 2. Let Told II and let Nnew be the optimizer of the minimization problem defined in (+) = ((s)) Then Quaen (s, a) 2 Q (s, a) for all (s,a) A with
[A| <0
Proof. Let Vold E Il and let Q and Viola be the corresponding soft state-action value and soft state value, and let nnww be defined as
Tnew (-|sr) arg min DKL (I' (-|s:) || exp (QTT (st. - log FMu (s:)))
e'ell arg min Jan (T' (-1s:) n'ell
Note that Jus (Tnew (-1s:)) I Jam (Told (+1s;),, since we can always choose Tuew Vold E II. Hence
EagesToww [log Nnww (ar (s) - Quala (st, at) + log Z nald (s:) -
E Engwdraad [log Vold (a,(s:) QTola (st, a,) log A (s,)] Applying soft Bellman Equation, we could get the policy improvement for SAC.
(11)
Note in ParPlsAc. we could only need to guarantee the condition in Eq. 11 is also satisfied. This directly follows the monotonic property of Langevin dynamics as shown in Eq.
C DERIVATION OF SQUASH CORRECTION
To constrain the action space in a finite interval, the tanh is applied on the samples from the raw output u. And we also conduct the Langevin dynamics on u space. The change of variable formula indicates the following equation satisfied:
da
D
p(als log ((us(
det
logp(uls) Σ
log (1 - tanh? (u:)
du
The stationary distribution on the a space exp(Q(ss, a)), the corresponding density function on the u space satisfying that:
log p(uss) = exp(Q(s, (a))())a) Σ log(1 (tanh(u:))
We take derivative and get the score function as:
Vn logp(uls) V. (Q tanh tancha))///())))
Σ Vn log(1 (tanh(u:))
D DISCUSSION ON REMARK 2
The langevin dynamic special case of wasserstein gradient flow (Liu et al., 2019), At step 1, the particle simulation is to solve the following problem:
W3 (p. uk
(M)
1 arg min KL(Hllp(x)) +
2h
Here μ4 denotes the sampled distribution after k-th step and p(x) is the target distribution, i.e. (0((sa))) in our case. With limited steps langevin dynamics, according to the triangle inequality of wasserstein distance we have:
W
Ph-1:
(b)
W3
Pitni
(b)
(M)
C Σc
(12)
2h
2h
The above inequality is to say that if each step the wasserstein distance could be bounded (by CD). then the wasserstein distance between the final distribution and the original distribution is also (b) bounded, And we replace μ) with ny and 1 with T we get the W [TT Top is constrained.
Note that we optimize the wasserstein distance between Te Td. explicitly, and the wasserstein
distance W (To )o,) is also constrained. Applying the triangle inequality agian, get the
15

---

Under review as a conference paper at ICLR 2022
bounded condition W3 (To: (p) W (T&, TTx) W (Tg, Tox). Integrating the W (T TT) and 2 (TT, Tex) into Eq. 12 with Lagrangian multiplier a and 7, we then get the Remark 2.
E CONVERGENCE ANALYSIS OF PARPI
The convergence property of ParPI is highly correlated with the partiele updates in Eq. 4. Note that Jordan et al. (1998) indicates that with unlimited samples and infinitely small step size, the sample result could approach some stationary distribution ex(xx, Such property provide good theoretical intuition for convergence analysis on ParPI:
With unbiased gradient estimation the V W (Tg. Tramlled), where sampled indi- cates the empirical distribution acquired from the langevin dynamics. If the sample size M + oo and step size - 0, ah in PurPI would converge to the global minimum ma(als) (x)(ee))xp(())s.))aa))))
Proposition 1.
Note the Proposition is based on the fact the KT(-II-) convex and we leave the detailed derivation in the Appendix.
To start with, we introduce the following lemma:
Lemma 3. (Jordan et al., 1998) Assume that logp(x) S C1 is infinitely differentiable, and | lo)p(x))l S C2 (1 + C1 logp(x)) (Vx) for some constants (2,, Let T
hK, Mo (x)), and { (K)
1K=1 be the solution of the functional optimization problem:u (h)
w (2(m))) (1))
which are restricted to the space with finite second-order moments. Then i) the problem is convex;and ii) PkK converges to pr in the limit of h - i... lim - MK
arg min, {T((llp(xx) +
2h
(k) prr, where pr is the solution of Fokker-Planch (FP) equation: 0.14
V (-H4 VU + (H,o0T)) at T.
Note the stationary distribution of the FP Equation is proportional to ed(2)) Lemma 3 shows that [b) =ee In our case the stationary distribution refers to ma(als) (ee((e)))))(((
limg-soo, 1
2 Thus Lemma 3 suggests that with sample size M + oo and step size a + 0, ny would converge to the global minimum kefals) (xp(Q(s.a)/aa)
2
F DISCUSSION ON MATCHING THE JOINT STATE-ACTION DISTRIBUTION
Note that ultimate goal of learning stochastic policy could be formulated as matching the conditional distribution, i.e., for every value of the state: ming D (qe(- s) [p( s)), Vs. While in this situation, the optimization problems need to be solved independently in different states. For example, if we optimize JSD or WD), then will need many different discriminators critics for different states. The corresponding computational complexity is unacceptable. To make the training tractable, we "amortizedd the optimization problem on different states into a joint matching problem. Put in another way, using the joint distribution is obvious when considering the analogy to supervised learning: the task is used in the objective: Ep(x.)) [D (fo (z). The analogy is achieved by replacing the model fo (z) with ( s) and the target with p(- s).
match fo (z) to the corresponding target y for every x, but joint distribution
G EXPERIMENT DETAILS
All algorithms are implemented in Tensorflow (Abadi et al., 2016) and use a distributed implementa- tion powered by Ray (Moritz et al., 2018). Following the implementation of TD3 (Fujimoto et al., 2018), we use two Q-value functions parameterized by a two-layer feed-forward network to fend off overestimation. A diseriminator model sharing the same architecture is introduced for the divergence minimization purpose. Besides, we do not introduce any reward shaping for all tasks. When collecting rollouts for evaluations, we simply take the action selected by the policy at every state for every 1000 updates.
Policy, Q-value function & Discriminator architectures for both baselines and our algorithms:
16

---

Under review as a conference paper at ICLR 2022
12, a 12 E 1 NO
Affine Transformation
Dense layer 256, ReLU
Dense layer 256, ReLU
Dense layer 256, Tanh
Table 3: Policy
R* E R
Dense layer 256, ReLU
Dense layer 256, ReLU
Dense layer 256, ReLU
dense + 1
Table 4: (Double)Q Function
R a E R
Affine Transformation
Dense layer 256, ReLU
Dense layer 256, ReLU
Dense layer 256, Linear
Table 5: Discriminator
Table Hyper-parameter Settings of ParPI
Parameter
Policy Adam
Q-function
Discriminator
optimizer
Adam
Adam
Learning rate discount(y)
3 10
3 10
3 10 10-4
0.99
n/a
n/a
replay buffer size
106
n/a
n/a
entropy target
dim (A)
n/a
n/a
MCMC steps
n/a
n/a
5
MCMC step size gradient steps
n/a
n/a
3 10 -4
1
target update interval
1
H OFFLINE ALGORITHM DETAILS
For completeness, we list the detailed algorithms for ParPI +MOPO in Algorithm We use rollout length 5 for all tasks, and the same penalty coefficients reported by Yu et al. (2020).
17

---

Under review as a conference paper at ICLR 2022
Algorithm 2 ParPI on top of MOPO
1: Input: reward penalty coefficient À rollout horizon h, rollout batch size b.
2: Train batch data Panv ensemble of N probabilisttic dynamics (T' (s a), o' (s, a))]
3: Initialize policy and empty replay buffer Puodel +- Ø.
4: for epoch 1,2......... 2.... do
5: for 1, ..... (in parallel) do
6: Sample state 81 from Deny for the initialization of the rollout.
7: for 1,2, ......... do
8:
Sample an action aj (s)).
Randomly pick dynamics T from (T and sample 8,+1, az)),aj).
9:
10:
Compute f rj --y-ymman m | (s), az)lle.
11:
Add sample a5. J 85+1) to D medel-
12: end for
13: end for
14: Drawing samples from Dem U moddll. ParPI to update
15:
end for
For the ParPI +BRAC, following the original implementation (Wu et al., 2019; Fu, 2020), we using dual form for the behavior policy regulation(with KI.) and value penalty with fixed a. Then we use and Q-function update. We using (256,256) fully connected network as
ParPi for both the critic the minimax objective. We also adapt the gradient penalty in KL dual training. we using Adam for all optimizers.
18


Reviews:
[{"review_id": "yjK5vJ63KkY", "reviewer": "Reviewer_jv1X", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I found that this paper contains some very interesting ideas, using Langevin dynamics to efficiently incorporate difference divergence frameworks is quite novel. However I found the presentation of the paper to be quite messy and insufficient, additional detailed comments are as follows:\n\n- One major problem I found with this work is that the overall motivation seems unclear. While the authors demonstrated how their algorithm accommodates for f-divergence and Wasserstein distance. The authors failed to elaborate why using these divergence measures is desirable or where the performance boost they claim actually come from. While I appreciate the detail provided by the authors in the paper, the overall big picture not clear.\n- In Algorithm 1, why did you sample the tuple (s, a, r) instead of (s, a, r, s')? If my understanding is correct, you are using the SAC or SQL value function updates (Equation 5 and 6) so you would need the next state s' to calculate the value targets.\n- Can you clarify what \"# frames\" in the x-axis are referring to in Figure 2? If you are referring to the number of agent-environment interactions, then 1 millions samples is generally not considered sufficient for environments like Humanoid, at least 3 million or 10 million if you have the computational resources is generally recommended to get a good sense of the sample efficiency of the algorithm.\n- Can you clarify the part on Langevin dynamics for Section 4.3? What are you referring to here when you say samples with a higher Q value can be fetched with larger step sizes or longer MCMC runs? Is this a bias issue? Also the captions in Figure 3 are unreadable, I suggest adding the full size figures to the appendix if there is not enough room in the main text.\n- (Minor) Typo in paragraph before remark 2, specific 'choice' of divergence.\n- (Minor) Was there a typo in Table 4 since at the beginning of the appendix you mentioned that you used a two-layered neural network for the Q functions?", "summary_of_the_paper": "This work extends the popular RL as inference framework to accommodate for more general divergence measures such as the f-divergence and Wasserstein distance. The author proposed a particle-based optimization framework for learning stochastic policies where the policy is learned using samples generated via Langevin dynamics. The framework can be applied to both online and offline settings.", "summary_of_the_review": "I think this line of work has great potential in its current state, I do not believe this paper is ready for publication. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ej1UECVNFs8", "reviewer": "Reviewer_Rvy4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong points:\n\nThe paper proposes a very interesting idea to solve an important problem in stochastic policy optimization. In supervised learning, we all know optimizing the reverse KL is a bad idea as it's mode-seeking. However, in reinforcement learning, many algorithms considers reverse KL as it's easy to compute the policy gradient from on-policy sampled transitions. The particle-based sampling is a very interesting solution to mitigate this problem. The proposed method is empirically verified on both online and offline RL tasks. \n\nWeak points:\n\nMy concern is that this paper misses some important previous works that try to solve the same problem. See [1,2] for example. To compute the gradient of KL in policy optimization, one needs samples from the softmax policy defined by Q. The idea to implement this is to use self-normalized importance sampling. In fact, the methods proposed in [2] can exactly to be used for Eq 7. \nI think it is important to discuss and compare these previous methods as they consider the exactly same problem of the paper. \n\nOther problems:\n\nIn the online RL experiments, how is sampling from $\\rho_\\pi$ implemented? Does it need extra on-policy sampled trajectories, or just sample a transition from the replay buffer? \n\n\n[1] Nachum, O., Norouzi, M., & Schuurmans, D. (2016). Improving policy gradient by exploring under-appreciated rewards. arXiv preprint arXiv:1611.09321.\n\n[2] Mei, J., Xiao, C., Huang, R., Schuurmans, D., & Müller, M. (2019, August). On principled entropy exploration in policy optimization. In Proceedings of the 28th International Joint Conference on Artificial Intelligence (pp. 3130-3136).\n", "summary_of_the_paper": "This paper proposes a particle-based sampling method for stochastic policy optimization in reinforcement learning. The main observation is that many policy optimization algorithms use reverse KL as the objective to optimize policies, which is mode-seeking and could be problematic in practice. To mitigate this problem, the authors propose to use particle-based sampling such that one can directly use KL for policy optimization. ", "summary_of_the_review": "I recommend rejecting this paper as important baselines are missed. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "V2O6yzT0yTA", "reviewer": "Reviewer_MUM2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Review of: PARTICLE BASED STOCHASTIC POLICY OPTIMIZATION\n\nI found the paper very difficult to understand. This was true in the language (see minor comments), as well as the logic (examples follow), and some basic details (Figures are unreadable, captions incomplete, key details missing). \n\nThe logic is hard to follow. For example: \n\"Modeling policy distribution by joint state-action distribution within the exponential family has enabled flexibility in exploration and learning multi-modal policies and also involved the probabilistic perspective of deep reinforcement learning (RL). The connection between probabilistic inference and RL makes it possible to leverage the advancements of probabilistic optimization tools. However, recent efforts are limited to the minimization of reverse KL divergence which is confidence-seeking and may fade the merit of a stochastic policy.\"\n\nIt is unclear what the goal and contribution of the paper is. To connect probabilistic methods with RL? What does it meant to fade the merit of a stochastic policy? Also, what does it mean to provide more flexible property? \n\nIs framework 1 meant to be a mathematical statement? Is remark 1 supposed to follow from the framework? The structure of the math is difficulty to follow. \n\nThere is a lot of explaining in words that would be much clearer if stated and derived mathematically. \n\n\nMinor:\n- Stochastic policy has been <-- policies have been?\n- \"By providing better distribution matching methods\" Is this demonstrated? \n- Figure 1 is much too small. Also, I didn't understand it. \n- How does ParPI \"help\" update the policy?\n- Figure to is much too small. It is essentially unreadable. \n- Why choose reverse KL and JSD for the experiments? \n- What does PARPI + SQL mean? (Similar questions for all of the notation in the legend for figure 2)\n- \"We follow the same schema as the D4RL paper (Fu et al., 2020) to calculate the normalized return\" This is not acceptable. The paper should state clearly what was done. \n- \" Jensen-Shannon divergence, though it is reported to\nshow some tendency of mode seeking (Theis et al., 2015), it also effectively ameliorates the problem as it consistently outperforms the exclusive KL\" What does this mean? \n", "summary_of_the_paper": "The paper introduces a particle-based approach for model-free RL. Multiple experiments compare the performance of the approach to SQL and SAC in online and offline settings. Results suggest that the proposed particle based approach improves on the prior work and shows different divergence measures yield different performance. \n", "summary_of_the_review": "The paper proposes an interesting approach, but the writing and presentation were incomplete / unclear. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TvBD2OkinTd", "reviewer": "Reviewer_Uu6f", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I have fundamental concerns about the paper.\n\n- There are many typos and grammatical errors.\n- Very little notation is defined and there is inconsistent use of the notation that is defined, making it difficult to understand and evaluate technical correctness and significance.\n- The description of the algorithm is very convoluted and sparse. A large number of approximations are made, so it is not possible to infer it solely from the objective. Furthermore, the “policy updates” are never actually formally stated. A figure is included but it is not helpful in my opinion. I think this paper would greatly benefit from a general algorithm box early on that clearly explains the procedure (in contrast to Alg 1).\n- There are statements of theorems and propositions but no proofs.\n- The technical novelty is not clear. Although the particle-based idea could be new, many papers have considered general divergences and distances previously but there is little discussion of these. Minor point: the related work is abruptly in the middle of the paper. It would be helpful if the authors can clearly outline the differences between this work and prior work that has considered various forms of regularization and divergences.\n- There are a number of claims throughout the paper stated as facts that are either very confusing or arguably unsubstantiated. Example: “Particularly in offline RL, the deterministic policy lacks the structure to manage risk towards uncertainty dynamics.” What structure does it lack? What are “uncertainty dynamics.” The data being offline does not change the fact that the optimal policy for the MDP is still deterministic (for nearly all MDPs of interest).\n- The logic of the motivation is very hard to follow. It is continuously said that deterministic optimal policies are undesirable. Yet, if the policy is optimal, why does it matter if it is deterministic? Unless one is trying to also satisfy other desiderata, I’m not sure what the problem is. “Exploration” is cited as a reason, but this only matters during learning, not when the optimal policy is found. “Robustness” is cited but no robust problem is formulated. I would appreciate it if the authors can clarify this by formalizing mathematically the other desiderata that they would like the optimal policy to satisfy besides maximizing the reward. It appears that this is attempted in the objective in 2.1 with the entropy term, but it is not explained or proven that entropy alone satisfies these additional objectives beyond reward maximization.\n- Furthermore, it is not clear why we should expect that a different discrepancy measure in the implementation should lead to differences in the solution to the objective stated in Sec 2. I suspect (correct me if I am wrong) that the authors would like these properties *during training* and not necessarily as properties of the optimal policy.\n- One of the core problems that the paper tries to address is the fact that reverse KL is mode-seeking. However, this is never explained in the paper. Why is it mode-seeking? What effect does this have on the objective or the regularized objective? What properties of the other discrepancy measures suggest that they could solve this problem? I do not think it is sufficient to cite prior works that have also made this comment off-handedly. Instead, I think this is an important claim that should be understood via proof or experiments. \n\nSpecific comments and questions:\n- Framework 1: I am confused about what purpose this serves. Was there supposed to be a theorem statement here?\n- The “policy update” stated below Equation (1) is not explained/defined.\n- D[ . || . ] notation is used inconsistently.\n- Sec 2.1: V^* is never defined. Is this the actual optimal value function or the one for the regularized reward problem? These look like the normal Bellman equations, not the soft ones or regularized ones that are commonly used in the literature.\n- Sec 2 on minimizing f-divergences: \n    - What is the difference between q and q_\\phi? What is g_\\phi? Equation (2) is stated but seemingly never used. What is the purpose of this?\n    - “Thus minimizing D˜ F well serves as minimizing DF” I think formal justification of this is necessary. Minimizing a lower bound does not mean you also minimize the upper bound in general.\n    - It seems like many approximations are made in this paragraph but it is not clear what is lost by making these approximations or how close the approximations are to the truth. \n- Sec 3: \n    - V, V_{\\psi}, and V_{\\bar{\\psi}} are not defined.\n    - Equations (5) and (6) appear to be statements of residuals. What are the updates? \n    - The discussion of Remark 2 seems dubious. It is claimed that a particular minimizer is recovered, but the discussion attempts to lower bound certain values and claim the minimizer is the same. This requires a formal proof. \n    - “The first two terms in Eq. 8 correspond to the distribution constraint and the support constraint” It is not obvious to me that either constraint is satisfied since the constraints look “soft” now that they are additive terms in the objective. Can this be elaborated?\n    - Equation (9) is very hard to read. Additionally there is use of $\\approx$ without further discussion of what is being approximated. Can this be clarified?\n- The figures are very small and hard to read.\n", "summary_of_the_paper": "This paper proposes a new method for policy optimization in reinforcement learning. Building off of prior connections to probabilistic inference, the authors propose a framework for optimizing policies using general divergences/distances by way of particle-based methods for sampling. Some theory is presented. Experiments show that the method greatly out-performs state-of-the-art algorithms in both online and offline settings.\n", "summary_of_the_review": "I do not believe this paper is ready to be published. There are a number of fundamental problems with the presentation and technical results. It may be that there are interesting or novel ideas here, as the experiments would seem to suggest, but unfortunately I believe it would require significant revisions.\n\nI have the following high-level suggestions to improve the paper: The algorithm and basic method should be formally stated early on in a clear and concise way. When an approximation is made, it would be helpful to state it explicitly and justify it (if possible). It would be helpful to remove confusing claims and limit unnecessary discussion of prior work. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The manuscript extends the popular "RL as inference" framework with a generalized divergence minimization perspective. The authors observe that most policy optimization can be thought of as minimizing a reverse KL divergence, which has potentially undesirable mode-seeking properties. The authors propose a particle-based scheme wherein samples generated via Langevin dynamics are used for learning.

Several reviewers found the ideas presented interesting, and cited potential novelty and high potential for tackling an important problem. Unfortunately, all reviewers found major shortcomings, from presentation ("messy" presentation, lack of definition of notation and inconsistent use, issues around motivation and logical flow, vague and imprecise use of language, etc.). Several reviewers also had more fundamental criticisms, notably Uu6f who helpfully provided quite actionable feedback on the presentation. Unfortunately, discussion ended with the reviews: the authors offered no rebuttal or updates. The AC considers this a missed opportunity.

The AC concurs with, first and foremost, the concerns around presentation. The current state of the manuscript makes it difficult to parse apart the contribution being made, and in light of all 4 reviewers recommending rejection either strongly or weakly and with no rebuttals or responses put forth, I have no basis to recommend anything other than rejection.
Decision: Reject