Adaptive Discretization for Continuous Control using Particle Filtering Policy Network
Authors: Pei Xu
Ioannis Karamouzas
Venue:  (2021)
Keywords: Reinforcement Learning
Continuous Control
Action Space Discretization
Policy Gradient

Abstract: Controlling the movements of highly articulated agents and robots has been a long-standing challenge to model-free deep reinforcement learning. In this paper, we propose a simple, yet general, framework for improving the performance of policy gradient algorithms by discretizing the continuous action space. Instead of using a fixed set of predetermined atomic actions, we exploit particle filtering to adaptively discretize actions during training and track the posterior policy represented as a mixture distribution. The resulting policy can replace the original continuous policy of any given policy gradient algorithm without changing its underlying model architecture. We demonstrate the applicability of our approach to state-of-the-art on-policy and off-policy baselines in challenging control tasks. Baselines using our particle-based policies achieve better final performance and speed of convergence as compared to corresponding continuous implementations and implementations that rely on fixed discretization schemes. 

Under review as a conference paper at ICLR 2021
ADAPTIVE DISCRETIZATION FOR CONTINUOUS CON- TROL USING PARTICLE FILTERING POLICY NETWORK
Anonymous authors Paper under double-blind review
ABSTRACT
Controlling the movements of highly articulated agents and robots has been a Long-standing challenge to model-free deep reinforcement learning. In this paper. we propose a simple, yet general, framework for improving the performance of policy gradient algorithms by discretizing the continuous action space. Instead of using fixed set of predetermined atomic actions, exploit particle filtering to adaptively discretize actions during training and track the posterior policy rep- resented as mixture distribution. The resulting policy can replace the original continuous policy of any given policy gradient algorithm without changing its un- derlying model architecture. We demonstrate the applicability of our approach to state-of-the-arr on-policy and off-policy baselines in challenging control tasks. Baselines using our particle-based policies achieve better final performance and speed of convergence as compared to corresponding continuous implementations and implementations that rely on fixed discretization schemes.
INTRODUCTION
In the last few years, impressive results have been obtained by deep reinforcement learning (DRL)
both on physical and simulated articulated agents for wide range of motor tasks that involve learn-
ing controls in high-dimensional continuous action spaces (Lillicrap al.] 2015 Levine al.
2016
Heess et al. [0177] Haarnoja et al. [018c] Rajeswaran et al. 2018 Tan et al. [2018] Peng et al.
2018
2020 Many methods have been proposed that can improve the performance
DRL for continu-
ous control problems, e.g. distributed training (Mnih et al. 2016 Espeholt et al.) 2018). hierarchical
learning (Daniel et al. 2012
Haarnoja et al. 2018a), and maximum entropy regularization (Haarnoja
et al. [017] et al.
2017 Haarnoja et al. 2018b). Most of such works, though, focus on learning
mechanisms to boost performance beyond the basic distribution that defines the action policy, where
Gaussian- based policy or that with squashing function is the most common choice the basic policy to deal with continuous action spaces. However, the unimodal form of Gaussian distributions could experience difficulties when facing multi-modal reward landscape during optimization and prematurely commit to suboptimal actions (Daniel et al. 2012 Haarnoja et al.) 2017).
To address the unimodality issue of Gaussian policies, people have been exploring more expres- sive distributions than Gaussians, with a simple solution being to discretize the action space and use categorical distributions as multi-modal action policies (Andrychowicz et al. 2020 Jaskowski eet al. 2018 Tang & Agrawal 2019). However, categorical distributions cannot be directly extended many off-policy Trameworks as their sampling process is not reparameterizable. Importantly, the performance of the action space discretization depends a lot on the choice of discrete atomic actions, which are usually picked uniformly due to lack of prior knowledge. On the surface, increasing the resolution of the discretized action space can make fine control more possible. However, in prac- tice, this can detrimental to the optimization during training, since the policy gradient variance increases with increasing number of atomic actions (Tang& & Agrawal 2019).
Our work also focuses on action policies defined by an expressive, multimodal distribution. In- stead of selecting fixed samples from the continuous action space, though, we exploit particle- based approach to sample the action space dynamically during training and track the policy rep- resented as a mixture distribution with state-independent components. We refer to the resulting policy network as Particle Filtering Policy Network (PFPN). We evaluate PFPN on state-of-the-an art on-policy and off-policy baselines using high-dimensional tasks from the PyBullet Roboschool en-

---

Under review as a conference paper at ICLR 2021
vironments (Coumans & Bai 2016 2 and the more challenging DeepMimic framework PPeng et al. 2018). Our experiments show that baselines using PFPN exhibit better overall performance and/or speed of convergence and lead to more robust agent control. compared to uniform dis- cretization and to corresponding implementations with Gaussian policies.
Main Contributions, Overall, make the following contributions. We propose PFPN general framework for providing expressive action policies dealing with continuous action spaces. PFPN uses state-independent particles to represent atomic actions and optimizes their placement to meet the fine control demand of continuous control problems. We introduce a reparameterization trick that allows PFPN to be applicable to both on-policy and off off-policy policy gradient methods. PFPN outperforms unimodal Gaussian policies and the uniform discretization scheme, and is more sample- efficient and stable across different training trials. In addition, it leads to high quality motion and generates controls that are more robust to external perturbations. Our work does not change the underlying model architecture or learning mechanisms of policy gradient algorithms and thus can be applied to most commonly used policy gradient algorithms.
2 BACKGROUND
We consider standard reinforcement learning setup where given time horizon H and the trajectory T = ($1, al, a1.... SH. an)) obtained by a transient model MM(+++ [Se, ar) and a parameterized action policy To (a)))))) with S, E Rº" and a, E RT' denoting the state and action taken at time step 4, respectively, the goal of learning is to optimize that maximize the cumulative reward:
((0) Er-ma(r) [rx(r)] = f po(T)r(T)dr.
Here, (T) denotes the state-action visitation distribution for the trajectory T induced by the transient model M and the action policy Ty with parameter 0, and r(r) Σ, rs,, au) where r(s;, az) E IR is the reward received at time step t. We can maximize J(0) by adjusting the policy parameters f through the gradient ascent method, where the gradient of the expected reward can be determined according to the policy gradient theorem (Sutton et al., 2000). i.e.
[(J = (.- [Ac Ve log To Ta(a))) [s))
(2)
where A, E RR denotes an estimate to the reward term (T). In DRL., the estimator of A, often relies on separate network (critic) that is updated in tandem with the policy network (actor). This gives rise to a family of policy gradient algorithms known as actor-eritic.
On-Policy and Off-Policy Actor-Critics. In on-policy learning, the update policy is also the be- havior policy based on which a trajectory is obtained to estimate A, Common on-policy actor-critic algorithms include A3C (Mnih et al., 2016) and PPO (Schulman et al., 2017). and directly employ 2 for optimization. In off policy learning, the policy can be updated without the knowl- edge of whole trajectory. This results in more sample efficient approaches as samples are reusable. While algorithms such as Retrace (Munos et al.] 2016) and PCL (Nachum et al. 2017) rely on Equation 2 many off-policy algorithms exploit a critic network to estimate A, given a state-action pair (Q- or soft Q-value). Common off-policy actor-critic methods include DDPG (Lillicrap et al. 2013), SAC (Haarnoja et al. 2018b d) and their variants (Haarnoja et al. 2017 Fujimoto et al, 2018). These methods perform optimization maximize a state-action value (s), In order to update the policy network with parameter f, they require the action policy to be reparameterizable such that the sampled action a, can be rewritten function differentiable to the parameter O, and the optimization can be done through the gradient Va, Q(S ar) Vg
Equation
Policy Representation. Given multi-dimennsonal continuous action space, the most common choice in current DRL baselines is to model the policy To as a multivariate Gaussian distribu- tion with independent components for each action dimension (DDPG, SAC and their variants typ- ically use Gaussian with a monotonic squashing function to stabilize the training). For simplic- ity, let us consider a simple case with a single action dimension and define the action policy as Te( (N(po(ss) (s.), (s.)). Then, we can obtain log T (ao|st) x -(a, (s Given sampled action a, and the estimate of cumulative rewards A,, the optimization process based on the above expression can be imagined as that of shifting n4, (sx) towards the direction of at if A, is higher
2

---

Under review as a conference paper at ICLR 2021
than the expectation, or to the opposite direction if A, is smaller. Such an approach, though, can easily converge to a suboptimal solution, if, for example, the reward landscape has a basis between the current location of H4 (sx) and the optimal solution, scape is symmetric around 14, (st). These issues arise due to the fact that the Gaussian distribution is inherently unimodal, while the reward landscape could be multi-modal (Haarnoja et al.] (017)- Similar problems also could occur in Q-value based optimization, like DDPG and SAC. We refer to Appendix [ for further discussion about the limitations of unimodal Gaussian policies and the value of expressive multimodal policies.
hard to be optimized if the reward land-
3 PARTICLE FILTERING POLICY NETWORK
In this section, we describe our Particle Filtering Policy Network (PFPN) that addresses the uni- modality issues from which typical Gaussian- policy networks suffer. Our approach represents the action policy as a mixture distribution obtained by adaptively discretizing the action space using indepeeedden particles, each capturing Gaussian distribution. The policy network, instead of directly generating actions, it is tasked with choosing particles, while the final actions are obtained by sampling from the selected particles.
3.1 PARTICLE-BASED ACTION POLICY
Generally, we define P == UpsLA Ws, (s: = 1,... n2 = 1, .m} as a weighted set of particles for continuous control problems with a m-dimension action space and n particles dis- tributed on each action space, where H4, E IR representing an atomic action location on the k- th dimension of the action space, and (s,x(s: (0), satisfying E; Wi, (s:(0) = 1, denotes the as- sociated weight generated by the policy network with parameter 0 given the input state S,, Let Pa. (a..,//ppssss fi,k) denote the probability density function of the distribution defined by the loca- tion fa, and noise process Ei,e- Given P, we define the action policy as factorized across the action dimensions:
(a.ls:) T
= I1 1
wi..(si10p..aaaa.ll.. ELA),
(3)
where a, = {[a,11,'' Jt Ot.k E TR is the sampled action at the time step for the action dimension k and (-10 can be obtained by applying a softmax operation to the output neurons of the policy network for the k-th dimension. The state-independent parameter set, { gives us an adaptive discretization scheme that can be optimized during training. The choice of noise & relies on certain algorithms. It can be scalar, e.g., the Constein-Uhlenbee
noise in DDPG (Lillicrap) al. 2015)) or an independent sample drawn from the standard normal distribution N/ 1) in soft Q-learning (Haarnoja et al. 2017), or be decided by a learnable variable, for example, sample drawn from C(, with learnable standard deviation Eikk- In the later case, a particle become a
Gaussian component N VJGg, 2. Without loss of generality, we define the parameters of a particle as Φ [ei k for the following discussion.
While the softmax operation gives us categorical distribution defined by (s, the nature of the policy for each dimension is a mixture distribution with state-independent components defined by Φi.-- The number of output neurons in PFPN increases linearly with the increase in the number of action dimensions and thus makes it suitable for high-dimensional control problems. Drawing sam- ples from the mixture distribution can be done in two steps: first, based on the weights w. (st (0), we perform sampling on the categorical distribution to choose a particle ja for each dimension k; i.e. ik (s:) ~ P(- &. (s:)); then, we can draw actions from the components represented by the chosen particles with noise a, (s.(s.) (- (+1)),.s.)).
Certain algorithms, like A3C and IMPALA, introduce differential entropy loss to encourage explo- ration. However, may be infeasible analytically evaluate the differential entropy of mixture distribution without approximation (Huber et al.) 2008). As such, we use the entropy of the categor- ical distribution if a differential entropy term is needed during optimization.
We refer to Appendix C for the action policy representation in DDPG and SAC where an invertible squashing function is applied to Gaussian components.
3

---

Under review as a conference paper at ICLR 2021
3.2 TRAINING
The proposed particle-based policy distribution is general and can be applied directly to any algo- rithm using the policy gradient method with Equation 2 To initialize the training, due to lack of prior knowledge, the particles can be distributed uniformly along the action dimensions with a stan- dard deviation covering the gap between two successive particles. With no loss of generality, let us consider below only one action dimension and drop the subscript k. Then, at every training step, each particle will move along its action dimension and be updated by
J( = E Σ cew, (s.l0)
(4)
where or ~.T TT (-1s;) is the action chosen during sampling, and E, cient shared by all particles on the same action dimension. Our approach focuses only on the action policy representation in general policy gradient methods. The estimation of A, can be chosen as required by the underlying policy gradient method, e.g. the generalized advantage estimator (Schul- man et al.] (2155)) in PPO/DPPO and the V-trace based temporal difference error in IMPALA. Similarly, for the update of the policy neural network, we have
(ey) is coeffi-
JJ(0) = E Σ caps (oo
(5)
From the above equations, although sampling is performed on only one particle for each given dimension, all of that dimension's 's particles will be updated during each training iteration to move towards or away from the location of a, according to A₁. The amount of the update, however, is regulated by the state-dependent weight w; (s, (s, particles that have small probabilities to be chosen for given state 8, will be considered uninteresting and be updated with smaller step size or not be updated at all. On the other hand, the contribution of weights is limited by the distance between a particle and the sampled action: particles too far away from the sampled action would be considered as insignificant to merit any weight gain or loss. summary, particles can converge to different optimal locations near them during training and be distributed multimodally according to the reward landscape defined by A,, rather than collapsing to a unimodal, Gaussian-like distribution.
3.3 RESAMPLING
Similar to traditional particle filtering approaches, our approach would encounter the problem of degeneracy (Kong et al. 1994). During training, a particle placed near location at which sampling gives a low At value would achieve a weight decrease. Once the associated weight reaches near zero, the particle will not be updated anymore (cf. Equation 4 and become 'dead', We adapt the idea of importance resampling from the particle filtering literature (Doucet et al.) (001) to perform resampling for dead particles and reactivate them by duplicating alive target particles.
We consider a particle dead if its maximum weight over all possible states is too small, i.e. maxs, ws, ()) < where is small positive threshold number. In practice, we cannot check (x))) for all possible states, but can keep tracking it during sampling based on the observed states collected in the last batch of environment steps. A target particle т; is drawn for each dead particle independently from the categorical distribution based on the particle's 's average weight over the observed samples: T; P (-|Es, (s [0]] 1,2, 1.2......
Let D, be a set of dead particles sharing the same target particle т. Let also the logits for the weight of each particle le be generated by fully-connected layer with parameters we for the weight and by for the bias. The policy Th6 (ar]si is guaranteed to remain unchanged after resampling via duplicating dhe &- Φr, Vi E Dr, if the weight and bias used to generate the unnormalized logits of the target particle are shared with those of the dead one follows:
Theorem 1.
wi + ww; b,, by + b, - log (1D+|+ 11)
(6)
Proof. See Appendix B for the inference.
Theorem guarantees the correctness of our resampling process as it keep the action policy To (ar(st) identical before and after resampling. If, however, two particles are exactly the same
4

---

Under review as a conference paper at ICLR 2021
after resampling, they will always be updated together at the same pace during training and lose di- versity. To address this issue, we add some regularization noise to the mean value when performing resampling, iec. M4 t Mr Exs where E; is small random number to prevent μi from being too close to its target jAr-
3.4 REPARAMETERIZATTIN TRICK
The two-step sampling method described in Section 3.1
non-reparamettrizzalee because of the
standard way of sampling from the categorical distribution through which Gaussians are mixed. To address this issue and enable the proposed action policy applicable state-action value based off- policy algorithms, we consider the concrete distribution (Jang et al. 2016: Maddison et al., 2016) that generates a reparamcterized continuous approximation to a categorical distribution. We refer to Appendix D for formal definition of concrete distribution defined as CONCRETE below.
Let xsx (0) ~ CONCRETE( ww; (s:(ss))); ( = 1, J, 1), where (():(0) = {a; (s:]0)); = 1, ...
is a is reparametrizable sampling result of a relaxed version of the one-hot categorical distribution supported by the probability of [w, (s, (0); i 1,2,.... We apply the Gumbel-softnn trick (Jang
et al. 2017) to get sampled action value as
"arole arg max x(sx{0))
d'(s;) = STOP
where a; is the sample drawn from the distribution represented by the particle i with parameter di. STOP(-) is "gradient stop" operation, and 6(:,-) denotes the Kronecker delta function. Then, the reparameterized sampling result can be written as follows:
(s;) = Σ (a, - d'(s::) + (( arg max x) a' (s(s)
(8)
where m, : (s: 0) + STOP( arg max (() 2. (s) = 8(i, arg max x(s, ()) com- posing a one-hot vector that approximates the samples drawn from the corresponding categorical
distribution. Since (s,s)))) drawn from the concrete distribution is differentiable the parameter 0, the gradient of the reparameterized action sample can be obtained by
Vony (si) [(a=- (da=-a(ss)) (oz, V Vo, ay ō(i, arg max (0) a.
(9)
Through these equations, both the policy network parameter 0 and the particle parameters &; can be updated by backpropagation through the sampled action a' dS.)).
RELATED WORK
Our approach focuses the action policy representation exploiting more expressive distribution other than Gaussians for continuous control problem in DRL using policy gradient method. In on- policy gradient methods, action space discretization using a categorical distribution to replace the original, continuous one has been successfully applied in some control tasks (Andrychowicz et al., 2020 Tang & Agrawal 2019)). All of these works discretize the action space uniformly and convert the action space to a discrete one before training. While impressive results have been obtained that allow for better performance, such a uniform discretization scheme heavily relies on the number of discretized atomic actions to find a good solution. On the other hand, discretized action spaces cannot be directly applied on many off-policy policy gradient methods, since categorical distribu- tions are non-reparameterizable.. While DQN-like approaches are efficient for problems with dis- crete action spaces, such techniques without policy networks cannot scale well to high-dimensional continuous action spaces due to the curse of dimensionality (Lillicrap al. 2015). Recent work attempted to solve this issue by using a sequential model, at the expense, though, of increasing the complexity of the state space (Metz et al. 2017). Chou al.) (2017) proposed to use beta distribu- tions as a replacement to Gaussians. However, results from Tang & Agrawal (018a) show that beta distributions do not work well as Gaussians in high-dimensional tasks.
Our PFPN approach adopts a mixture distribution as the action policy, which becomes a mixture of Gaussians when learnable standard deviation variable is introduced generate normal noise
5

---

Under review as a conference paper at ICLR 2021
DepMiimieWaik
DospAiikk
Cunulation Rescaets
630
(3000
S000
430
2000
2000
230
Gosaida DISCIETE
11.5
1/1
0.1
1.2
# Sumples
Sacaples
of Sumples
<10
Samples
Figure 1:
Learning curves of PPO/DPPO using PFPN (blue) compared to Gaussian policies (red) and DISCRETE policies (green). Solid lines report the average and shaded regions are the minimum and maximum cumulative rewards achieved with different random seeds during training.
EutBBllleEur-xt)
Well
Paep
Cammotive Reward
500
500
3300
-000
30C
200
PETS
Diradaan
Samples
X
Sumples
-10
Samples
x10P
of Samples
Figure 2:
Comparison of SAC using PFPN (blue) and Gaussian policies (red). DISCRETE is not shown as the categorical distribution cannot be applied to SAC.
for sampling. It uses state-independent particles having state-dependent weights track the pol- icy distribution. While an early version of SAC also employed mixture of Gaussians, such mixture completely state dependent, and hence it cannot provide a discretization scheme that is consistent globally for all given states. In addition, the components in the fully state-dependent mix- ture of Gaussians could collapse, resulting in similar issues as unimodal Gaussian policies (Tang & Agrawal 2018a) Sap. Adopting state-independent components can reduce the network size and still provide expressive action policies when sufficient components are employed.
PFPN focuses on the policy distribution representation generated directly by the policy network without changing the underlying policy gradient algorithms or remodeling the problem, and thus can be applied directly most common used policy gradient methods in DRL. It is also comple- mentary to recent works that focus on improving the expressiveness of the action policy through normalizing flows (Haarooja et al.) 2018a| 'Tang & Agrawal 2018b] Mazoure et al. 2019 Delal- leau et al. 2019). where the mixture distribution provided by PFPN can be employed as a base distribution. Other techniques applicable policy gradient methods with a generally defined ba- sic action policy can be combined with PFPN as well, such as the use of ordinal architecture for action parameterization (Tang & Agrawal 2019) action space momentum as in the recent PPO- CMA (Hämäläinen et al. 2018). 1 and energy- based policy distribution optimization methods, like PGQL (O' D'Donoghue et al. 2016), Soft-Q learning (Haarnoja et al. 2017]017.. 7, SVPG (Liu et al. 2017
and policy optimization using Wasserstein Gradient Flows (Zhang al. 2018
EXPERIMENTS
The goal of our experiments is to evaluate whether existing policy gradient algorithms using PFPN can outperform the corresponding implementations with Gaussian policies, along with comparing our adaptive action discretization scheme generated by PFPN to the fixed, uniform one. For our comparisons, we use the set of particles as mixture of Gaussians with learnable standard deviation. We run benchmarks on a range of continuous torque-based control tasks from PyBullet Roboschool envirouments (Schulman al. 2015a). We also consider several challenging cosition-control tasks from the DeepMimic framework (Peng et al. (2018) where a 36-dimension humanoid agent learns a locomotion policy based motion capture data with 197-dimension state space.

---

Under review as a conference paper at ICLR 2021
5.1 COMPARISONS
Figure I and 2 evaluate our approach on two representative policy gradient methods: PPO/DPPO (Schulman al., 2017: Heess et al., 2017), which is a stable on-policy method that exhibits good performance, and SAC (Haarnoja et 20T8dd) an off-policy method that achieves state-of-the-art performance many tasks. The figures show the learning curve by cumulative rewards of evaluation rollouts during training. We also compare PFPN to the fixed discretization scheme (DISCRETE) obtained by uniformly discretizing each action dimension into a fixed number of bins and sampling actions from categorical distribution. In all comparisons, PFPN and DISCRETE exploit the same number of atomic actions, i.e., number of particles in PFPN and number of bins in DISCRETE. We train five trials of each baseline with different random seeds that the same across PFPN and the corresponding implementations of other methods. Evaluation was performed ten times every 1,000 training steps using deterministic action.
As it can be seen in Figure PFPN outperforms Gaussian policies and DISCRETE in all of these tested tasks. Compared Gaussian policies, our particle-based scheme achieves better final perfor- mance and typically exhibits faster convergence while being more stable across multiple trials. In the Roboschool tasks, DISCRETE performs better than the Gaussian policies. However, this doesn't t translate to the DeepMimic tasks where fine control demand is needed to reach DeepMimic's 's mul- timodal reward landscape, with DISCRETE showing high variance and an asymptotic performance that is on par with or worse than Gaussian-based policies. In contrast, PFPN is considerably faster and can reach stable performance that is higher than both Gaussian and DISCRETE policies.
In the SAC benchmarks shown in Figure 2 DISCRETE cannot be applied since the categorical distribution that it employs as the action policy is non-parameterizable. PFPN works by exploiting the reparameterization trick detailed in Section 3.4 Given the state-of-the-art performance of SAC, the PFPN version of SAC performs comparably to or better than the vanilla SAC baseline and has faster convergence in most of those tasks, which demonstrates the effectiveness of our proposed adaptive discretization scheme. In DeepMimic tasks, considering the computation cost of running stable PD controllers to determine the torque applied to each humanoid joint TTn al. 2011 PFPN can save hours of training time due to its sampling efficiency.
PFPN can be applied currently popular policy-gradient DRL algorithms. We refer Appendix H for additional benchmark results, as well as results obtained with A2C/A3C (Mnih et al. 2016).
IMPALA (Espeholt et al., 2018), and DDPG (Lillicrap al. 2015). In most of these benchmarks, PFPN outperforms the Gaussian baselines and DISCRETE scheme, and is more stable achieving similar performance across different training trials. Appendix H.5] we also compare the perfor- mance in terms of wall clock time, highlighting the sampling efficiency of PFPN over Gaussian and DISCRETE when facing complex control tasks. See Appendix G for all hyperparameters.
5.2
ADAPTIVE DISCRETIZATION
particlesi
AntHulleEEs--00
Humancid Bulketin-vdl
DeepMiiieWWlk
DeepMimicPasch
hins
PFPN
DISCRTTE
PTPN
DISCRETE
PFPN
DISCRETE
PFPN
DISCRETE
5
3154
:: 209
2938 + 1:47
2508
1: 293
2567
:: 416
438
:: 15
61 :: 96
37
:: 15
10
10
3163
323
2963
1: 281
2840
480
2351
343
489
308
56
426
48
281
155
35
2697
246
2367
+ 274
2276
+ 298
2255
376
684
+
164
+
31.
+
50
2571
163
2810
4: 239
2191
+: 322
1983
325
580
322 2: 190
521
19
198
159
100
2234
104
2181
4 175
1444
1 330
1427
2 358
579
2 18
277
: 197
533
1: 7
224
2: 164
150
2335-21
160
323
200
17
152
200
583
166
509
31
181
400
578 1:
14
111
- 159
478
- 68
126 - 137
Gaussian
2327 199
2462 1 105
540
1
19
359
181
Table 1: Comparison between PFPN and DISCRETE on four benchmarks using PPO/DPPO while varying the resolution of each action dimension. Training stops when fixed number of samples is met as shown in Figure D
Reported numbers denote final performance averaged over trials = std.
Compared to Gaussian-based policy networks, DISCRETE can work quite well in many on-policy tasks, as has been shown in recent prior work (Tang& & Agrawal 2019). However, in the comparative evaluations outlined above, showed that the adaptive discretization scheme that PFPN employs results in higher asymptotic performance and/or faster convergence as compared to the uniform discretization scheme. To gain better understanding of the advantages of adaptive discretization
7

---

Under review as a conference paper at ICLR 2021
X Axia
Y Axis
2. Axis
Anple
Taining Steps
1..0
40.
0.2
0.2
0.6
1.0
-1.0
10.5
40.2
0.2
0.66
1.0
-100
10.6
-1.2
0.2
0.6
1.0
-1.0
-0.65
-0.2
10.2
0.5
1.0
1.50
Figure 3: Evolution of how partieles are distributed along four action dimensions during training of the DeepMimicWalk task with DPPO. The depicted four dimensions represent the target right hip joint position expressed in an axis-angle representation. Each action dimension is normalized between -1 and 1. Particles are initially distributed uniformly along a dimension (dark colors) and their locations adaptively change as the policy network is trained (light colors). The training steps are measured by the number of samples exploited during training.
for learning motor tasks, Table 1 further compares the performance of PFPN and DISCRETE across different discretization resolutions (number of particles and number of bins, respectively). As it can be seen, PFPN performs better than DISCRETE for any given resolution.
Intuitively, increasing the resolution, and hence the number of atomic actions, helps both DIS- CRETE and PFPN. However, the more atomic actions employed the harder the optimization problem will be for both methods due to the increase of policy gradient variance (see Appendix D for theo- retical analysis). This can also be verified empirically by the performance decrease in Roboschool tasks when transitioning from 10 to 150 particles. In the more complex DeepMimic tasks, using too few atomic actions can easily skip over optimal actions. Although performance improves when the action resolution increases from 5 to 10, DISCRETE cannot provide any stable training results as denoted by the high variance among different trials. contrast, PFPN performs significantly better across all resolutions and can reach best performance using only 35 particles. And as the number of particles increases beyond 35, PFPN has only a slight decrease in performance. This is because the training of DeepMimic tasks final performance. We refer to Appendix for the sensitivity analysis of PFPN with respect to the number of particles, where employing more atomic actions beyond the optimal number results in slower convergence but similar final performance.
run long enough to ensure that it could reach a relatively stable
DeepMimic tasks rely stable PD controllers and exploit motion capture data to design the ward function, which is more subtle than the Roboschool reward functions that primarily measure performance based the torque cost and agent moving speed. Given a certain clip of motion, the valid movement of a joint may be restrained in some small ranges, while the action space covers the entire movement range of that joint. This makes position- based control problems more sensitive to the placement of atomic actions, compared to Roboschool torque-based control tasks in which the effective actions (torques) may be distributed in a relatively wide range over the action space. While uniform discretization could place many atomic actions blindly in bad regions, PFPN optimizes the placement of atomic actions, providing a more effective discretization scheme that reaches better performance with fewer atomic actions. As an example, Figure shows how particles evolve during training for one of the humanoid's 's joints the DeepMimic Walk task where PFPN reaches a cumu- lative imitation reward much higher than DISCRETE or Gaussian. We action spaces cover only some small parts of the entire action space. In Appendix H. we also show the particle evolution results for Roboschool's AntBulletEnv-vō task. Here, the active action spaces are distributed more uniformly across the entire action space, which explains why DISCRETE is able to exploit its atomic actions more efficiently than in DeepMimic tasks.
that the final active
5.3 CONTROL QUALITY
To highlight the control performance of the adaptive discretization scheme, Figure compares the motion generated by PFPN and DISCRETE in the DeepMimic Walk task. As can be seen in the figure, PFPN generates a stable motion sequence with a nature human-like gait, while DISCRETE is stuck at a suboptimal solution and generates walking motion sequence with an antalgic- gait where the agent walks forward mainly through the use of its left leg. From the motion trajectories,
8

---

Under review as a conference paper at ICLR 2021
Re
Togli Bot
Le
DISCRETS
Comparison of motion generated by PFPN and DISCRETE DeepMimic task during one step cycle. Both PFPN and DISCRETE are trained with DPPO using the best resolution parameters from Table D (35 particles and 200 bins, respectively). PCA embedding of the trajectories of the agent's each method. Lines with "x" are the ground truth trajectories extracted from the motion capture data that the agent learns imitate.
Figure 4:
feet are shown at the right and their time sequence expansions the bottom of
it can also be seen that PFPN results in more stable gait with clear eyclic pattern strictly following the motion capture data. We also assess the robustness of the learned poli- cies external perturbations. Table reports the minimal force needed to push the humanoid agent down. All experiments were performed af- ter training using deterministic actions. It evident that the PFPN agent can tolerate much higher forces than the Gaussian one. The fixed discretization in DISCRETE is less flexible, and result cannot learn generalized features that will enable robust agent control. We also refer to Appendix for additional results including motion trajectories obtained in the AntBulletEnv-vO task, as well as the controls (torque profiles) of the corresponding ant agents.
Task
Force Direction
PLPN
DISCRETE
Gaussian
DeepMimicWalk
Forward
340
512
Sideway
602
420
560
DeepMinniPunch
720
Sideway
896
576
748
Table 2: Minimal forwards and sideways push needed to
make a DPPO DeepMimic agent fall down. Push force is in Newtons (N) and applied on the chest of the agent for 0s
H
6 CONCLUSION
We present a general framework for learning controls in high-dimensional continuous action spaces through particle-based adaptive discretization. Our approach uses mixture distribution represented by set of weighted particles to track the action policy using atomic actions during training. By introducing the reparameterization trick, the resulting particle-based policy be adopted by both on-policy and off-policy policy gradient DRL algorithms. Our method does not change the underly- ing architecture or learning mechanism of the algorithms, and is applicable to common actor-critic policy gradient DRL algorithms. Overall, our particle filtering policy network combined with ex- isting baselines leads to better performance and sampling efficiency as compared to corresponding implementations with Gaussian policies. As way to discretize the action space, we show that our method is more friendly to policy gradient optimization by adaptive discretization, compared to uni- form discretization, as it can optimize the placement of atomic actions and has the potential to meet the fine control requirement by exploiting fewer atomic actions. In addition, leads to high quality motion and more robust agent control. While currently track each action dimension indepen- dently, accounting for the synergy that exists between different joints has the potential to further improve performance and motion robustness, which opens an exciting avenue for future work.

---

Under review as a conference paper at ICLR 2021
REFERENCES
OpenAl: Marcin Andrychowicz, Bowen Baker, Macick Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
Po-Wei Chou, Daniel Maturana, and Scbastian Scherer. Improving stochastic policy gradients in continuous control with deep reinforcement learning using the beta distribution. In International conference on machine learning, pp. 834 843, 2017.
Erwin Coumans and Yunfei Bai. Pybullet, python module for physics simulation for games, org 2016-2019.
robotics and machine learning.
Christian Daniel, Gerhard Neumann, and Jan Peters. Hierarchical relative entropy policy search. In Artificial Intelligence and Statistics, pp. 273-281, 2012.
Olivier Delalleau, Maxim Peter, Eloi Alonso, and Adrien Logut. Discrete and continuous action representation for practical rl in video games. arXiv preprint arXiv: 1077, 2019.
Arnaud Doucet, Nando De Freitas, and Neil Gordon. An introduction to sequential monte carlo methods. In Sequential Monte Carlo methods practice, pp. 3-14. Springer, 2001.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint arXiv: 1802.0158 2018.
Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv: 1802.884777 2018.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International Conference Machine Learning, pp. 1352-1361, 2017.
Tuomas Haarnoja, Kristian Hartikainen, Pieter Abbeel, and Sergey Levine. Latent space policies for hierarchical reinforcement learning. arXiv preprint arXiv: 1004: 02808, 2018a.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. arXiv preprint arXiv:1801.01290, 2018b.
Tuomas Haarnoja, Aurick Zhou, Sehoon Ha, Jie Tan, George Tucker, and Sergey Levine. Learning to walk via deep reinforcement learning. In Robotics: Science and Systems, 2018c.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv:1812.05000 2018d.
Perttu Hämäläinen, Amin Babadi, Xiaoxiao Ma, and Jaakko Lehtinen. Ppo-cma: Proximal policy optimization with covariance matrix adaptation. arXiv preprint arXiv: 2018.
Nicolas Heess, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, SM Eslami, Martin Riedmiller, et al. Emergence of locomotion behaviours in rich 2017.
environments. arXiv preprint
Marco Huber, Tim Bailey, Hugh Durrant-Wiie Whyte, and Uwe D Hanebeck. On entropy approxima- tion for gaussian mixture random vectors. In 2008 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, pp. 181-188. IEEE, 2008.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv: 1. 0. 144, 2016.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparametrization with gumble-softmax. In International Conference Learning Reprexentations (ICLR 2017), 2017.
10

---

Under review as a conference paper at ICLR 2021
Wojeiech Jaškowski, Odd Rune Lykkebo, Nihat Engin Toklu, Florian Trifterer, Zdeněk Buk, Jan Koutník, and Faustino Gomez. Reinforcement learning to run... fast. In The NIPS' 17 Competi- fion: Building Intelligent Systems, pp. 155-167. Springer, 2018.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6988 2014.
Augustine Kong, Jun S Liu, and Wing Hung Wong. Sequential imputations and bayesian missing data problems. Journal of the American statistical association, (9(425):278-281 1994.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. The Journal of Machine Learning Research, 17(1):1334-1373 2016.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: MXXiv:1509.02777 2015.
Yang Liu, Prajit Ramachandran, Qiang Liu, and Jian Peng. Stein variational policy gradient. arXiv preprint arXiv: /t1.//444...9999 2017.
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv: 1.0:7//1110001... 2016.
Bogdan Mazoure, Thang Doan, Audrey Durand, R Devon Hjelm, and Joelle Pineau. Leveraging
exploration in off-policy algorithms via normalizing flows. arXiv preprint MKKivI995.06881 2019.
Luke Metz, Julian Ibarz, Navdeep Jaitly, and James Davidson. Discrete sequential prediction of continuous actions for deep rl. arXiv preprint arXiv:/755.000 2017.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937, 2016.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare Bellemare. Safe and efficient off-policy reinforcement learning. In Advances in Neural Information Processing Systems, pp. 1054-1062, 2016.
Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. Advances Neural Information Processing Systems, pp. 2775-2785, 2017.
Brendan Donoghue, Remi Munos, Koray Kavukcuoglu, and Volodymyr Mnih. Combining policy gradient and q-learning, arXiv preprint UrXiv:1611.01620 2016.
George Papandreou and Alan L Yuille. Perturb-and-man random fields: Using discrete optimization to learn and sample from energy models. In 20/ / International Conference on Computer Vision pp. 193-200 IEFF, 2011.
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: Example- guided deep reinforcement learning of physics-based character skills. ACM Transactions Graphics (TOG), 3-14) 2018.
Xue Bin Peng, Erwin Coumans, Tingnan Zhang, Tsang- Wei Lee. Jie Tan, and Sergey Levine. Learn- ing Agile Robotic Locomotion Skills by Imitating Animals. In Proceedings of Robotics: Science and Systems, 2020.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. In Robotics: Science and Systems, 2018.
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynam- ics of learning in deep linear neural networks. arXiv preprint arXiv: /122.6/20. 2013.
11

---

Under review as a conference paper at ICLR 2021
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, 2015a.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint wrXiv:1506.02458 2015b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint urXiv:1707.063477 2017.
Richard Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in neural informa- tion processing systems, pp. 1057-1063, 2000.
Jie Tan, Karen Liu, and Greg Turk. Stable proportional -deevvaataleee controllers. IEEE Computer Graphics and Applications, 3144)344-44 2011.
Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, and Vincent Vanhoucke. Sim-to-real: Learning agile locomotion for quadruped robots. In Robotics: Science and Systems, 2018.
Yunhao Tang and Shipra Agrawal. Boosting trust region policy optimization by normalizing flows policy. arXiv preprint arXiv: 1809.10330 2018a.
Yunhao Tang and Shipra Agrawal. Implicit policy for reinforcement learning. arXiv preprint
1rXiv:1806.06798, 2018b.
Yunhao Tang and Shipra Agrawal. Discretizing continuous action space for on-policy optimization. arXiv preprint 1901. 10500, 2019.
Ruiyi Zhang, Changyou Chen, Chunyuan Li, and Lawrence Carin. Policy optimization as wasser- stein gradient flows. arXiv preprint arXiv: /808. 03030, 2018.
12

---

Under review as a conference paper at ICLR 2021
A ALGORITHM
Algorithm Policy Gradient Method using Particle Filtering Policy Network
Initialize the neural network parameter 0 and learning rate o;; initialize particle parameters o; to uniformly distribute particles on the action dimension; initialize the threshold e detect dead particles using small number; initialize the value of interval n to perform resampling. loop
for each environment step do Il Record the weight while sampling.
-T.p(-(s;))
W; + W. (wi(s:10)} end for for each training step do
II Update parameters using SGD method.
Φ; f 0; + aV ((zz)
0 t 0 0-0+aVJ(0)
end for
for every environment steps do I/ Detect dead particles and set up target ones. for each particle do if max EW; < then
Wk We E Wa], k = 1,2.
Ti
T T TUTT: Dr, t Dr, Uti
end if end for // Resampling.
for each target particle 7 E T do for each dead particle i E D, do
11 Duplicate particles.
di & Ôr with μ4s + fr + Es
Duplicate parameters of the last layer in the policy network.
W; wr: by <- by - log( (D, 1)
end for
b, <- b,
log([D,| +
D, +-
end for
T + 0: W; < 9
end for end loop
B POLICY NETWORK LOGITS CORRECTION DURING RESAMPLING
Theorem 1. Let D, be set of dead particles sharing the same target particle т. Let also the logits for the weight of each particle be generated by fully-connected layer with parameters wr for the weight and by for the bias. The policy To (s)s)) is guaranteed to remain unchanged after resampling via duplicating di &- Φr, Vi E Dy, if the weight and bias used to generate the unnormalized logits of the target particle are shared with those of the dead one follows:
w; + Wri bi, by +- by log (|Dr1 + 1).
(10)
Proof. The weight for the i-th particle achieved by softmax operation, which is applied to the unnormalized logits L,, i.e. the direct output of the policy network:
cLi(s))
wi (st) SOFTMAX((A(s:)) Σx ehulee)
(11)
Resampling via duplicating makes dead particles become identical to their target particle. Namely, particles in D, U (r) will share the same weights well the same value of logits, say 1/ after
13

---

Under review as a conference paper at ICLR 2021
resampling, To ensure the policy identical before and after sampling, the following equation must be satisfied
Eehalen
E
e (ot)
E
cha(+e)
(12)
D.U(T)
RED,
where Lş is the unnormalized logits for the k-th particle such that the weights for all particles who are not in D, U {r} unchanged, while particles in D, U {r} share same weights.
A target particle will not be tagged as dead at all, i.e. E DD for any dead particle set Dr. since a target particle drawn according to the particles' weights and since dead particles are defined as the ones having too small zero weight to be chosen. Hence, Equation 12 can be rewritten
Σ
c..)))) + tebr(()
((Dr) + L)e()(s))
(13)
iED
Given that cfr())) ss for any dead particle D, and that the number of particles limited, implies that
e (1D,) + ())
(14)
Taking the logarithm of both sides of the equation leads to that for all particles in D, U (7), their new logits after resampling should satisfy
I, ($t) R (xt) - (Pr] 1).
(15)
Assuming the input of the full-connected layer who generates L; is ((s),, i.e. L,(st) = w;x(s;) +bi we have
<(x(st) + by = w,x(s:) + by - log (p++|+1)
(16)
Then, Theorem
| can be reached.
If we perform random sampling not based on the weights during resampling (see Appendix I it is possible to pick a dead particle as the target particle. In that case
Tr, (8+) (, + (1 5tt, Dx))),
5(T,
(17)
where (st) is the new logits shared by particles in D, and 8(T, D&) is the Kronecker delta function
{
1 if E Dx otherwise
Ö(T, Dx)
(18)
that satisfies O(T D&) < 1. Then, for the particle its new logits can be defined
(s.) (1 Σ 5(T, L(), + Σ 5()..D)LL.
(19)
Consequently, the target particle 7 may or may not share the same logits with those in D,, depending on is tagged dead or not.
C POLICY REPRESENTATION WITH ACTION BOUNDS
In off-policy algorithms, like DDPG and SAC, an invertible squashing function, typically the hyperbolic tangent function, will be applied enforce action bounds samples drawn from Gaussian distributions, c.g. in SAC, the action is obtained by s.) = tanh u,, where u4.k ~ (s:) (s))), and ( and (s))) are parameters generated by the policy network with parameter 6.
Let a, (tanh ur where u drawn from the distribution represented by a particle with pa- rameter d is random variable sampled support the action the k-tth dimension. Then, the probability density function of PFPN represented by Equation 3 can be rewritten as
TH (ar(S))
= II Σ
w. (t. w)
tanh? u.,
(20)
and the log- probabiilit function becomes
log n. (ar (st)
Σ log Σ wi. (v,, 2 (log 2 -e,6 - softplus(-2uns
(21)
14

---

Under review as a conference paper at ICLR 2021
D CONCRETE DISTRIBUTION
Concrete distribution was introduced by Maddison et al. (016)) It is also called Gumbel-Softmax and proposed by Jang et al. 2016 concurrently. Here, directly give the definition of concrete random variables X ~ CONCRETE((, X) by its density function using the notion from Maddison al. 2 as below:
A-1 0g.Tk t ii=l
(x) = (n 1)
(22)
k=1
where X for E | E [0, E0 Th 1).0 a fai,... ann) E (0,+) is the location
parameter and X E +00) is the temperature parameter. A sample X = {X1,- Xn can be drawn by
exp((logax + Ga)/A)
X E exp((log aq + Gt)/A)*
(23)
where Gg ~ GUMBEL i.i.d., or more explicitly, G3= = log( log Uk) with Ug drawn from (N,)
From Equation X can be reparameterized using the parameter and X, and gives us relaxed version of continuous approximation to the one-hot categorical distribution supported by the logits a. As À is smaller, the approximation is more discrete and accurate. In all of our experiments, we pick À = 1.
For convenience, in Section 3.4
we use CONCRETE with parameter w E [wi,-.. 2y We E [0, 1], Sk=1 Wg }} as the probability weight to support a categorical distribution instead of the logits a. We can get in terms of w by
0g = log(ww/(11 wx)).
(24)
Since X CONCRETE(a, A) a relaxed one-hot result, we use arg max X to decide which particle to choose in the proposed reparameterizzaioo trick.
E VARIANCE OF POLICY GRADIENT IN PFPN CONFIGURATION
Since each action dimension is independent to others, without loss of generality, we here consider the action o with only one dimension along which n particles are distributed and the particle to represent a Gaussian distribution W ((P; o?). In order to make it easy for analysis, we set up the following assumptions: the reward estimation is constant, i.e. A, = A; logits to support the weights of particles are initialized equally, i.e. wa(s:[0) = x for all particles i and Vown (s,
Vow, (s. (0); particles are initialized to equally cover the whole action space, i.e. μ i-n of + where 1,...,"" n.
From Equation
5 the variance of the policy gradient under such assumptions is
= E, pl a,da;
L
Σ V (s f o,)da
En + of) (suc
(25)
( 11-
n2
1 2
0 )
Given [())a())) = when n = 1, from Equation 25 for any n > 0, the variance of policy gradient V (())()))a will increase with n. Though the assumptions usually are hard to meet per- fectly in practice, this still gives us an insight that employing a large number of particles may result in more challenge optimization.
15

---

Under review as a conference paper at ICLR 2021
This conclusion is consistent with that in the case of uniform discretization (Tang & Agrawal 2019)
where the variance of policy gradient is shown to satisfy
J(O) aa DISCRETE
(26)
n
That is to say, in either PFPN or uniform discretization scheme, we cannot simply improve the control performance of the police by employing more atomic actions, i.e. by increasing the number of particles or using more bins in the uniform discretization scheme, since the gradient variance increases as the discretization resolution increases. However, PFPN has slower increase rate, which implies that it might support more atomic actions before performance drops due to the difficulty in optimization. Additionally, compared the fixed, uniform discretization scheme, atomic actions represented by particles in PFPN are movable and their distribution can be optimized. This means that PFPN has the potential to provide better discretization scheme using fewer atomic actions and thus be more friendly to optimization using policy gradient.
F MULTI-MODAL POLICY
In this section, we show the multi-modal representation capacity of PFPN on special designed tasks.
Reward
PFPN
PFPN
Gonesiza
Commlativv Rewwar
Gauzelan
-0.1
0.3
Deriiity
-0.2
-0.3
-0.4
Action
# of Sample
(a) One-step Bandit
(b) Multi-goal Navigation
Figure 5: (a) One-step bandit task with asymmetric reward landscape. The reward landscape is defined as the gray line having two peaks asymmetrically at -0.25 and 0.75. The probability densities of stochastic action samples drawn from PFPN (blue) and Gaussian policy (red) are counted after training with fixed number of iterations.
(b) Illustration of 2D multi-goal navigation. Left: one-step trajectories generated by PFPN and Gaussian policy via stochastic sampling after training- Red dots are the four goal points placed symmetrically and the contour line depicts the reward landscape by action costs proportional to the distance to the closest goal point. The Gaussian policy is initialized around the origin. Right: The learning curve of PFPN and Gaussian policy measured by cumulative rewards over five training trials with PPO algorithm.
One-step Bandit. This is a simple task with one dimension action space A = 1-1, It has an asymmetric 2-peak reward landscape inversely proportional to the minimal distance points 00.25 and 0.75, as the gray line shown in Figure ((a) The goal of this task is to find out the optimal points close -0.25 and 0.75. In Figure [(a)] we show the stochastic action sample distributions of PFPN and the naive Gaussian policy after training with the same number of iterations. It is clear that PFPN captures the bi-modal distribution of the reward landscape, while the Gaussian policy gives an unimodal distribution capturing only one of reward peaks.
2D Multi-goal Navigation. In this task, the reward landscape is designed symmetrically with four goal points at [=5,,, +0.5] and the agent has the task to reach any the goal points. The cost (negative of the reward) function of the state S E I²² and action A E R² is defined as the distance from the target position to the closet goal point. The naive Gaussian policy is initialized with mean value around the origin such that it is placed at the center point of the reward basin initially. Figure shows the training performance curve and the one-step path trajectories given by PFPN and the Gaussian policy after training. While PFPN successfully learns diverse trajectories
16

---

Under review as a conference paper at ICLR 2021
reaching all four goals, the Gaussian policy fails to learn anything due to the symmetry of the reward landscape.
In the following, we limit our discussion to the expressivity of the basic action policy distributions generated directly by the policy network in actor-critic policy gradient methods, without the con- sideration of other methods to enhance the expressivity of action policies based on base ones, e.g. normalizing flows, or those to help better policy exploration by learning mechanisms, like entropy- based policies.
For simplicity, let us consider a 1-dimension Gaussian policy with log-probability of
log log = (a. - po(ss))² ( log or (S:) log 2n.
(27)
(x)
where a2 (Huo(s)), os)s) In order to update the policy network with parameter 8, we use Equation
with
lo (o(y)s)) = Vao(s,) log (o((arls))Vooo(()) + Vootee) log Voro(s)
(28)
where
V (o((s) log (a(()())) = (a. - vo(ss)) Voo(er) log To(as (s:) = (a: - 1o (8x)))) o (sx))
(29)
(s(ss)
(st))
The optimization of (s)ss) is along the direction of A,(a, (s(ss)) Therefore, the optimization
process is analogous to sliding po(s;), the location of policy distribution, towards or opposite to the sampled action a regularized by the advantage estimation A,. If Ag is higher than the average E, [At], the optimization will push (s(ss) towards to the location of ay; if Ag is smaller than the average, uo (s:) will be pushed away from the direction of a, -po(s)). Regarding the update of (soss), (s,) will increase and make the distribution cover a; if A, is higher than the average and the distance between a and (e(s;) is out of the range of (s)(s:: otherwise, (s;)) will decrease and make the distribution shrink. Such an optimization strategy will encounter problems when, for example, facing symmetric reward landscape defined by A,, just like the 2D multi-goal navigation problem shown above. The Gaussian policy distribution in such a case would be optimized to move towards multiple directions simultaneously. This makes the policy distribution staying around its current location and unable to be optimized. Similarly, when facing an asymmetric, multimodal reward landscape as shown in the one-step bandit test case, the Gaussian policy would only be able to capture just one mode at best due to its unimodality.
Though in nigh-dimensional control problems, the reward landscape has a more complex shape, the multimodality is quite common in various, practical tasks. As an empirical proof, Figure B shows the evolution of the particle distributions during training, which captures the multimodality along different dimensions in DeepMimic Walk task. In such task, Gaussian policy may face the problem of premature convergence and be stuck at suboptimal solutions because of its unimodality. There- fore, a multimodal policy distribution should be preferred for better capturing the reward landscape shape.
Fixed, uniform discretization (DISCRETE) schemes can also provide multimodal policy repre- sented by the categorical distribution. However, it is hard for DISCRETE to find an optimal solution due to the blindness of uniform discretization. Without loss of generality, we define the action space A as the range of [-1, 1] and assume that the action space is discretized with n bins. Under this setup, we will have atomic actions of {or,1, 4r,2, ... at...) with distance of between two con- secutive atomic actions. Let aj (S:) be the optimal solution given the state St. We can measure the error between the optimal solution az (s:) and the best solution that DISCRETE could provide by
min (s: dr,a 1 k
(30)
This only gives us an upper bound of the minimal error the half size of bin. We can decrease the upper bound by exploiting more atomic actions. However, this cannot necessarily help decrease the lower bound. This analysis gives
insight of the limitations that DISCRETE has.
Table reports the lower bound of the minimum error that DISCRETE would achieve with varying number of bins. We can see that the best discretization scheme for DISCRETE is to use 4 bins.
17

---

Under review as a conference paper at ICLR 2021
Number of Bins
3
5
6
7
9
10
Lower Bound Error
0.25
0.25
0.08
0.05
0.08
0.036
0.125
0.03
0.05
Table 3: Lower bound of the minimum error that DISCRETE attains with different action resolu- tions in the one-step bandit task.
However, in practice, we are usually unable to quantitatively analyze the lower bound error with respect the number of bins in complex DRL tasks. Increasing, though, the number of bins would reduce the possible maximum error. For example, it is better to use bins compared to the case with 7 5 bins, or better use 10 bins compared that using variance problem caused by introducing more atomic actions (See Appendix L would influence the
bins. However, the policy gradient
optimization negatively. To solve this dilemma, PFPN can optimize the placement of atomic actions during training and thus has the potential to meet the fine control requirement with fewer atomic actions.
G HYPERPARAMETERS
Parameter
Value
Shared
optimizer
Adam (Kingma & Ba 2014)
activation function
ReLU
resampling interval
25 environment episodes
dead particle detection threshold (e)
0.05/# of particles per action dimension
clip range (PPO/DPPO)
0.2
GAE discount factor
0.95
(PPO/DPPO, A2C/A3C, A)
truncation level (IMPALA, č, P)
1.0
reply buffer size (SAC, DDPG)
1 1000
Roboschool Environments
learning rate
3 10-1
weight initializer
Orthogonal (Saxe al. 2013 [256, 256]
number of neurons in hidden layers
number particle per action dimension discount factor (?))
35 (Humanoid), 10 (others)
0.99
coefficient of policy entropy loss term (A2C/A3C, IMPALA)
0.01
DeepMimic Environments
learning rate
1 10-4
weight initializer
Truncated Normal with std. dev. of 0.05
number of neurons in hidden layers
[1024, 512]
number particle per action dimension
35
discount factor (7)
0.95
coefficient of policy entropy loss term
0.00025
(A3C, IMPALA)
Table 4: Default Hyperparameters in Baseline PFPN Benchmarks
Table 4lists the default hyperparameters used in all of our experiments. Regarding PPO and A2C, in all Roboschool tasks except for the HumanoidBulleennwwy one, we use a single worker thread; for HumanoidBulletEnvwwe and DeepMimic tasks, we exploit the advantage of distributed training and use DPPO (synchronous PPO) and A3C (asynchronous A2C) with multiple worker threads, while IMPALA is natively multi-thread.
18

---

Under review as a conference paper at ICLR 2021
H ADDITIONAL RESULTS
Roaddon Eano-
AutBalle E
Humomiic EllleEEnse
Conaaiize Rewwokk
PPO
PPO
PPO
DPPO
3000
10
1000
1000
XUP
<32
-110P
110
Conultiio Rewards
A20
000
2000
1000
1000
100
<32
-100
Cuundetive Rowords
IMPLLA
3000
IMPALL
DEPALA
DMPALA
20000
1000
15
5000
1000
901
WiilMinniWWWlkk
DispMinni Punch
DeepMiniKKik
DapMiniidRo
Coumlttive Rewords
3070
DETO
DIVO
DPPD
320
331
Commbaiive Rewards
A30
ADC
400
430
400
3x0
230
2041
201
X.00
<.27
1.107
x100
Commbttre Rewards
600
600
C00
000
-00
400
-00
200
0.0
05
13
6.0
105
1.0
0.0
0.5
1.0
155
0.1
0.5
313
20
Samples
x00"
8 of Sampler
<32
Sumples
Samples
PEPN
Gauaton
DISCRETT
Figure 6: Training curves continuous control tasks from the Roboschool and DeepMimic envi- ronments using on-policy policy gradient algorithms and IMPALA with v-trace correction.
19

---

Under review as a conference paper at ICLR 2021
Front Left Foot
Front Right Foot
Left Back Foot
Right Back Foot
PEPN
Ganosian
DISCRETE
Figure 7: Motion trajectories of end effectors (four feet) of the ant agent in AntBulletEnv-vO task with PFPN-PPO, Gaussian-basee PPO and PPO with fixed, uniform discretization (DISCRETE), We apply PCA to visualize the 2D trajectories; the trajectories are measured by the relative positions of effectors with respect to the root link of the agent.
Front Left Foot
Front Right Foot
Back Left Foot.
Back Right Fool
Mtegos
Figure 8: Evolution of particles distributed on action dimensions for the four feet joints of the agent in AntBulletEnv-vO task during training using PPO.
H.1 COMPARISON TO GAUSSIAN AND UNIFORM DISCRETIZATION POLICIES
Rngure/ccoppane baselines that employ Gaussian policies to their PFPN counterparts on a variety of Roboschool and DeepMimic tasks and the fixed, uniform discretization scheme (DISCRETE) with the same number of atomic actions. The results are obtained as discussed in Section 5.1 where ten evaluation trials run every 1,000 training steps using deterministic actions. PFPN outperforms Gaussian policies and fixed, uniform discretization scheme in those baselines and is more stable across different training trials.
To further highlight the value that particle-basee based discretization adds to the action exploration prob- lem, we also compare the motion generated by PFPN-PPO policy in the AntBulletEnv-v0 task to the ones obtained by vanilla PPO with Gaussian action policy and that using fixed, uniform dis- cretization scheme. We project the one-episode motion trajectories of the four end effectors (feet) of the ant agent into two dimensions in Figure 7 As it can be seen, there is a significant differ- ence in the motion taken by the three agents, with PFPN leading to higher performance shown in Figure ? Similarly to our analysis in Figure 4 Gaussian-basse PPO and DISCRETE-PI have more noise in the generated manifold, while PFPN-PPO performs robustly with clear eyclic
20

---

Under review as a conference paper at ICLR 2021
motion pattern. We also show the particle evolution of the ant agent during training in Figure g which reflects the optimization of particle distribution.
In Figure 9 compare the torques generated by PFPN to those generated by Gaussian and DIS- CRETE. As it can been seen, PFPN generates significantly different torque profiles than the ones obtained by Gaussian both in terms of frequency and/or amplitude that result in higher asymptotic performance shown Figure PFPN has similar torque frequency and amplitude with DISCRETE, but also has significant difference details, which leading to better performance.
NN And
mn
W
W
JU
nut 7
1/ 1
& W
Mama Mn
nan m
in
n
LA
(a) PFPN
(b) Gaussian
(c) DISCRETE
Figure 9: Torque patterns generated by Gaussian- based PPO and PFPN-PPO on AntBulletEnv-VO. Each row denotes the corresponding policy per action dimension executed for 200 frames. Torque values are normalized lie between and
H.2 ADDITIONAL OFF-POLICY RESULTS
Besides the SAC benchmarks shown Section 5.1 we test PFPN with DDPG, another popularly used off-policy algorithm, which runs policy gradient method only through state-action value. Dif- ferentiate the above experiments where each particle is assigned with a learnable variable and rep- resents a Gaussian distribution, we follow the default configuration of DDPG and use scalar noise during action sampling. DDPG provides relatively worse performance compared to SAC and cannot work well in HumanoidBulletEnwwwe and DeepMimic tasks. As supplement, three environments, HalfCheetah-v2. Ant-v2 and Humanoid-v2. from OpenAl Gym benchmark suite are introduced in Figure PFPN shows advantage in most of those tasks.
BeucherBllleeeen
BalCCler Euriie
Comniatro Remords
3300
3000
3300
2000
Gaaadas
01
x02
X:30"
Ant-v2
D0PG
DEPG
2301
Condaabin
2000
2,001
2000
2300
8 vl Samples
of Samppas
Samples
Figure 10: Learning curves on continuous control tasks using DDPG in Roboschool and OpenAl Gym environments.
21

---

Under review as a conference paper at ICLR 2021
Environment
PH-N
DISCRETE
Gaussian
Goussian-Be Big
GMM
3163
2863 :
2327 E 199
2398 = 107
2813 380
AntBallelEns-vO
(93,856)
(93,776)
(75,272)
(94,352)
(134,896)
HomaoidBulleFFm--v0
2840 =: 480
2351 +: 343
2462 :: 195
2519 d: 219
2026 4: 346
(121,172)
(121,002)
(81,681)
(125,521)
(208,382)
DeepMimicWalk
5884 1:
245 4: 164
540 1: 19
517 1: 33
203 d: 1335
(1,375, 192)
(1,373,932)
(746,020)
(1,380,498)
(2.666,692)
537 1.
317 76
359 181
309 140
210 75
DeepMimicPunch
(1,375,192)
(1,373,932)
(746,020)
(1.380,498)
(2,666,692)
Table 5:
Performance comparison of PPO/DPPO using PFPN, uniform discretization (DISCRETE), Gaussian baselines (Gaussian), Gaussian baselines with a larger policy network (Gaussian- Baaussian-BBgg,.. and policies of fully state-dependent Gaussian mixture model (GMM). Reported numbers denote final performance averaged over 5 trials + std after training is done. In each test case, the number of the policy network parameters is listed in parentheses below the reported performance.
H.3 COMPARISON TO GAUSSIAN POLICIES WITH BIG NETWORK
The policy network in PFPN has similar size with the one in DISCRETE. Therefore, the performance advantage of PFPN over DISCRETE clearly comes from its more flexible discretization scheme. Compared to the Gaussian baselines, PFPN exploits a relatively larger policy network, though the number of hidden neurons is the same. To demonstrate that the performance gain of PFPN is not due its large network size, we increase the number of hidden neurons in Gaussian basclines such that they have similar number of policy network parameters with PFPN. The performance comparison between PFPN and the scaled up version of Gaussian baselines (Gaussian-Bigg is shown in T Training is done after fixed number of samples are collected shown in Figure | As can be seen from the table, increasing the number of hidden neurons does not lead to any significant improvements in Gaussian policies, with PFPN still reaching better final performance.
H.4 COMPARISON TO FULLY STATE-DEPENDENT GAUSSIAN MIXTURE POLICIES
AntDulkeEnwsst
Delp
DeepMrnoePuchh
Commbstiv Rowwres
PPO
DPPO
031
DPPO
D9P0
3000
2000
300
700
1000
1000
11.
1.0
Commlaatvv Rewords
400
000
400
-33
2000
2300
200
5000
1000
GMM
of Samples
of Saamlees
of Samples
of Sumples
Comparison of PFPN Gaussian baselines and fully state-dependent mixture of Gaus- sians (GMM). GMM in SAC uses the reparameterization trick described in Section 3.4 for state- action value based optimization.
Figure 11:
PFPN in our experiments employs a mixture of Gaussians with state-dependent weights but state- independent components. The state-independemt components can provide a global configuration of particle distributions that can be used as a discretization scheme. In theory, fully state-dependent Gaussian mixture model (GMM) can also work as the action policy in DRL. However, in our exper- iments depicted in Figure IT we found that GMM usually does not work quite well and even worse than Gaussians for complex continuous control problems. This is consistent with the results reported by Tang & Agrawal 80 As shown in Figure II GMM can only reach better performance than Gaussian in the AntBulletEnv-VO task with PPO algorithm and in the HumanoidBulletEnvww task using SAC, and is worse than PFPN in all tested benchmarks. GMM needs a much larger policy network as shown in Table 5 This may pose challenge to optimization. Another issue of GMM
22

---

Under review as a conference paper at ICLR 2021
observed during our experiments is that GMM components easy to collapse together and lose the advantage of multimodality. In the test cases of SAC, we use the reparamcterization trick introduced in Section 3.4 to perform state-action value based optimization.
H.5 TIME COMPLEXITY
FuranciilBillle
Deep Walk
Deep Mrenieeceenne
Commbsttv Rirwards
PPO
DPPN
JFPO
DPPO
2000
3303
2000
2003
1000
PFPN
000
30C
201
Oranlatre Resnads
4000
100
900
5AC
2000
5000
1000
200
200
8030
PEIN Ganedoos
0.0
25.
1.5
20
20
59
23 Ranning Time [houd]
Running
Time [tou]]
Rouning
Time ]oour]
Running Time [bous]
Figure 12: Cumulative rewards obtained during training as a function of the actual wall clock time. Training stops when a fixed number of samples is collected as reported in Figures D and 2
Sampling from categorical distributions directly using Gumbel tricks (Papandreou & Yuille 2011 is typically more expensive than that from Gaussians. In addition, both PFPN and DISCRETE need more time than the Gaussian baselines in order to perform action sampling and complete training iteration due to their larger network sizes. Compared to DISCRETE, PFPN has higher time com- plexity it needs an extra process of sampling from the Gaussian distributions represented by the particles. While the resampling process can cost time as well, in practice, it is only performed every tens of thousand interactions with the environment during training. Despite its higher time com- plexity, though, PFPN is more sample efficient than Gaussian baselines and DISCRETE, needing less actual clock time to reach better performance. To highlight this, Figure 12 reports the training performance as a function of running time (wall clock time). All training was done on machines having the same configuration with Nvidia Tesla V100 GPUs and stopped when a fixed number of samples was collected for training.
PFPN needs more time to leverage the same number of samples as Gaussian baselines. However, the performance obtained at fixed time can be significantly higher shown in Figure 12 Especially in DeepMimic tasks, the sampling efficiency of PFPN is quite apparent. In DeepMimic Walk using DPPO, PFPN needs about 6 hours to reach cumulative reward of 500, while Gaussian baselines needs around 8 hours to obtain a similar score. The gap between PFPN and Gaussian baselines is more evident in SAC cases. As our SAC implementation used single thread, it took much more time to finish the training compared to DPPO that worked distributedly. While PFPN and Gaussian baselines using SAC have almost the same final performance, PFPN converged about ten hours before the Gaussian baseline. Simulation in DeepMimic tasks is quite expensive in term of wall clock time, mainly due to the complexity of running stable PD controllers (Tan et al. 2011 and computing the reward by comparing the agent motion to the motion capture data. Once the simulated humanoid agent falls down (one of the termination conditions in the training of DeepMimic tasks), the simulation environment needs to reset and start a new environment episode. This process can take lots of time and poses a challenge to the algorithm's 's ability to do better action space exploration at the initial stage of training. It is also the reason why DISCRETE needs more wall time than PFPN, even though its time complexity is lower.
I SENSITIVITY ANALYSIS
Number of Particles. Since the particle configuration in PFPN is state- -nteeeindeeendnn,, needs a sufficient number of particles to meet the fine control demand. Intuitively, employing more particles
23

---

Under review as a conference paper at ICLR 2021
Cumulativv Rewards
500
DPPO
600
931
DPPO
300
200
Hoszaping
Rocaeeilig
0000005
0.0
10
1
0.0
#
Sample
of Sumple
of Sample
Sample
(a) Number of Particles
(b) Resampling Strategy (c) Resampling Frequency (d) Resampling Threshold
Figure 13:
Sensitivity of PFPN to the number of partieles and resampling strategies and hyper- parameters on DeepMimic Walk task using DPPO. (a) Comparison of PFPN using 35 particles per action dimension, which is the default parameters used in above benchmarks, to that using 5 (red), 10 (green). 50 (yellow) and 100 (azure) particles. (b)-(d) show the performance of PFPN with 35 particles on each action dimension but different resampling strategies or hyperparameters, where the blue line is the default hyperparameters used in our benchmark tests.
will increase the resolution of the action space, and thus increase the control capacity and make fine control more possible. However, in Appendix E we prove that due to the variance of policy gradient increasing as the number of partieles increases, the more particles employed, the harder the optimization would be. Therefore, it may negatively influence the performance to employ too many particles. This conclusion is consistent with the benchmark results of Roboschool tasks shown in Table M as the final performance decreases while the number of particles increases. The Deep- Mimic task with more than 35 particles, though reaching similar final performance, gets slower convergence as the number of particles increases. To clarify this, we show the learning curve of the DeepMimic Walk task with different number of particles in Figure [3 where PFPN-x represents the case that there are particles per action dimension. As it can be seen, there is an obvious con- vergence delay as the number of particles per action dimension increases from 35 to 100. On the other hand, as we can see, coarse discretization resolution employing too few particles, i.e., 5 or 10 particles per action dimension, will cause performance drops evidently, since the too few number of particles limits the control capacity.
Resampling Strategies. In Figure [3(b] We compare PFPN with default resampling strategy ex- plained in Section 3.3 to random resampling strategy (red) and that without resampling (green). The default resampling strategy is to draw targets for dead particles according to the weights of remaining alive ones. The random resampling strategy is to draw targets randomly from remaining alive particles. It can be seen that reactivating dead particles by resampling could help improve the training performance. Even the random resampling, though probably reactivating dead particles and place them on suboptimal locations, it still leads to better performance by keeping exploiting more effective particles, compared to the case without resampling. However, random resampling could lead to high variance and make the training process unstable by introducing too much uncertainty.
Figure further compares PFPN's s resampling strategy to no resampling for Roboschool and Deep- Mimic environments. As can be seen, in the more challenging DeepMimic tasks, our resampling strategy leads to an obvious improvement (note that 600 is the best reward that an algorithm can achieve). As compared to position-based DeepMimic control tasks, Roboschool agents directly learn torque-based controls where the effective atomic action space is uniformly distributed across the specified joint torque limits (see Section 5.2 for details). Taking also into account that PFPN requires only 10 atomic actions per dimension in the Roboschool tasks, particles are less likely to become degenerate since each of them will possibly become active at a certain observed state during training. While employing more particles would trigger the resampling process, this could lead to worse performance because of the increased difficulty in optimization (See Appendix
Resampling Hyperparameters. We also analyze the sensitivity of PFPN to the resampling hy- perparameters: the resampling threshold c, which determines whether a particle is dead or not, and the resampling interval, which denotes how often the resampling is performed and is measured by environment episodes. Results in Figure [3cc] and and [3d]] show that the resampling process itself is robust and not very sensitive to these two hyperparameters. However, extremely small value of
24

---

Under review as a conference paper at ICLR 2021
HumerrodBBallee---0000
Deep.Minic
DeepMiinicPuchh
Cummlative Rewards
600
DFPO
DPPO
DPPO
3000
400
2000
200
1000
PFPN
w/o resampling
0
8.0
0.0
of Samples x10
of Samples
110"
of Samples
x10"
Figure 14: Learning curves of PPO/DPPO using PFPN with the default resampling process (blue)
compared to that without resampling (green).
resampling threshold or large value of resampling interval still would hurt the performance by pre- venting resampling and push the learning curves towards to the case of PFPN without resampling. On the other hand, too large resampling threshold will lead alive particles to being labeled as dead, increasing the variance of the training performance. Resampling with too small interval will incur resampling before sufficient action exploration and thus cause the same problem with a too large value of resampling interval. In our tests, choose 20 environment episodes as the default resam- pling interval, and a dynamic value of resampling threshold depending on the number of particles that each action dimension has, which is around 0.0015 in DeepMimic tasks with 35 particles per action dimension.
25


Reviews:
[{"review_id": "vNmS-qRCTqM", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n----------\n\nThis paper presents an approach to multimodal policies based on Gaussian mixtures. The policy is parameterized as a set of Gaussian distributions (with state-invariant mean and variance) weighted by state-dependent mixture weights, which are the output of a (softmaxed) network. The weighting network and the means and covariances of the Gaussians are updated with standard RL losses. The authors propose a resampling scheme for mixture elements that consistently have low weight, in the style of resampling in particle filters. The authors evaluate the method with several RL algorithms including PPO and SAC, and on a variety of environments. \n\nComments\n----------\n\nOverall, this paper is pointing in an interesting direction. Expressive action distributions are often an afterthought in RL (although they have seen increasing attention with SAC), and this work is an interesting step in that direction. However, there are several limitations that, if addressed, would strengthen the paper substantially. \n\nOverall, the evaluation is quite thorough. However, the authors do not include a comparison to a standard mixture of gaussians policy, in which the Gaussians mixture elements are also state-dependent. The authors state that the mixture of Gaussians presented \"cannot support any discretization scheme\". What is meant by this? The PFPN approach does not actually \"discretize\" the action space, it simple uses a mixture of Gaussians to parametrize the policy to yield a multimodal action distribution. Thus, the fully state-dependent mixture of Gaussians seems like the most relevant comparison for the PFPN approach, and should be able to rely on the sample Gumbel-softmax trick for differentiable sampling. \n\nThe most interesting contribution of the paper is the resampling scheme. However, there is minimal evaluation of the benefit of this scheme. While section H of the appendix is a good start, I would expect a much more thorough investigation of the resampling, as it is probably the most substantial novel contribution of this work. Indeed, the current results for the resampling are somewhat worrying, as they show only a minor performance drop for removing the resampling entirely. An evaluation of the value of resampling (and the hyperparameters associated with resampling) on a variety of environments would strength the paper. \n\nFinally, the paper should better address the value of expressive multimodal policies beyond just performance increases in standard environments. This is addressed briefly in section E of the appendix, but a broader discussion of the benefits of the approach would help make clear the benefits associated with the PFPN approach. \n\nPost-Rebuttal\n----------\n\nI thank the authors for their response. The additional baseline comparisons do strengthen the paper, and I have increased my score from 4 to 5. I agree with the authors that the mixture of Gaussians policy is substantially weaker than their method, and is a useful baseline experiment to have. \n\nHowever, the added experiments with random sampling are somewhat worrying---the performance improvement of the proposed re-sampling scheme is quiet minor over random resampling. In the future, the authors may want to investigate the random resampling for the systems in figure 14.", "extra_scores": {}, "extra_text": {}}, {"review_id": "hLQsQ6GiF_q", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In the paper \"Adaptive Discretization for Continuous Control using Particle Filtering Policy Network\", the authors introduce a new way to discretise the action space of agent in RL settings by using a Particule Filtering approach. The main idea is that the learned policy will output the weight of each particle to define which one should be used, while the position of the particle changes during the learning process. Particles that are not moved (because they have a weight that is systematically too low) are removed and resampled from other particles. \n\nThe paper is well structured and clearly illustrated. \nHowever, I am unable to understand from the text why the proposed approach is not subject to the curse of dimensionality. In particular, if for every action dimension 10 to 35 weights should be defined by the policy, this creates a significant increase in the size of the search space (35^N). In particular, I would appreciate seeing a discussion about the size (in terms of the number of parameters) of the policies. \n\nIt is also quite surprising to see that the considered task in figure 2 can mostly be solved with very few particles (for both of the compared approaches). Therefore, I don't understand why in the last experiment of the paper (figure 3), the compared approaches use 35 vs. 200 particles. The complexity of the search space is certainly impacted by the difference and the observed results might just be the result of this. I am also wondering how the analysis can be extended to the other experiments. It seems that the performance difference on the other tasks is significantly more subtle and it will be important to discuss this. \n\nOverall, the paper proposes an interesting method but should be more didactic on why it is not impacted by the issues that are affecting the other approaches.  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "U4ukrLKsqxh", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Post rebuttal update: The authors have addressed my concerns and the revised submission is much clearer, so I'm increasing my score to 7.\n\n\nI must preface this with a caveat that I am not up to date with the latest results in continuous deep RL. That being said I think this is a nice and reasonably well-written paper, albeit with some weaknesses.\n\nMajor comments:\n\nThe biggest weakness of this paper appears to be that the results are only minor improvements (if they improve at all over the baseline). This is a major shortcoming because the proposed method is significantly more complicated than the baselines. If this additional complexity, and the additional hyper-parameters, does not bring major benefits then this begs the question as to whether or not it is worth it. On this point, what is the additional computational cost of the proposed method over the baselines?\nIf there is some other advantage to using this proposed method over the others then this should be explained more clearly up-front. I think a 'Contributions' section that listed the major takeaways of the paper would be useful.\n\nPractically none of the mathematical quantities are defined correctly. If you have a mathematical object it should be defined as a member of a particular space, e.g., x in R^n, or s \\in \\S, or \\pi \\in \\Delta_A. Otherwise it's very hard to reason about what these objects are. This is a major clarity problem.\n\nMinor comments:\n\nWhy is eq 6 the product over the action dimensions 'k'? Isn't each dimension independent in this setup? The current formulation makes the policy appear to be a scalar and I was under the impression that it was a vector, although as previously mentioned this isn't clear.\n\nThe notation in eq 9 (mu and sigma on top of each other) is very strange, probably better to use \\theta = (\\mu, \\sigma).\n\nI don't see what you used for the At estimate. Is that explained anywhere?\n\nI don't fully understand section 3.4, it's quite unclear as written. Why is the two-step sampling method 'non reparameterizable'? What does 'non reparameterizable' even mean? This needs a serious definition. Also the 'concrete' distribution is not defined here, it needs to be discussed more generally for completeness, possibly in the appendix is space is an issue.\n\nWhy is your method 'more friendly to policy gradient optimization' than simpler methods? Can this be quantified some way more concretely?\n\nA few missing related works occurred to me (from the policy gradient / value learning literature):\nhttps://arxiv.org/abs/1606.02647\nhttps://arxiv.org/abs/1611.01626\nhttp://papers.nips.cc/paper/6870-bridging-the-gap-between-value-and-policy-based-reinforcement-learning.pdf\n\nConclusion and next steps for the authors:\n\nOverall I liked the paper, but cannot recommend it for publication in it's current form.\n\nIf the paper is improved to be clearer I will increase my score, in particular it needs better explanation of the concepts in some sections and every mathematical object should be defined rigorously. I dropped a point for clarity alone.\n\nIf the significance of the work, and the computational results can be better explained or improved I could increase my score again.", "extra_scores": {}, "extra_text": {}}, {"review_id": "E43wl6LyCKk", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. Optimal policy should be unique, and please make corrections for motivation. Flexible policy representation is helpful when the agent is not confident about the environment (exploration). The energy-based policy is helpful for efficient exploration, as it can better model the uncertainty of an agent.\n\n2. Particle filtering is relatively computationally intensive. Could you please provide a comparison of the running time?\n\n3. The main contribution of this paper is a GMM policy using particle filtering. This direction is not that exciting. The initial version of SAC does use the GMM, but the final version uses isotropic Gaussian. Particle-based variational inference with RL has also been explored by Stein Variational Policy Gradient and Policy Optimization as WGFs. It is good to see improvement empirically, but I am not sure the improvement source. An ablation study on this should be useful.\n\n4. It is interesting to see the differentiable reparameterization trick. I cannot verify its correction. What does 'arg max x' mean?  If one prefers the flexible representation, the energy-based policy as in Soft Q-Learning should be a good choice. The dead particle issue will not happen as a repulsive force added in SVGD gradient step.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is rejected.

The authors contributions are:
* Propose PFPN as an expressive action policy for continuous action spaces.
* Introduce a reparameterization trick for training PFPN with off-policy methods.
* Experiments claiming PFPN outperforms unimodal Gaussian policies and a uniform discretization scheme and that it is more sample efficient and stable across different training trials.

I and the reviewers appreciate the additions by the authors. The GMM baseline is an important addition addressing concerns from several reviewers. However, I agree with R2's comment that "most interesting contribution of the paper is the resampling scheme. However, there is minimal evaluation of the benefit of this scheme [...] However, the added experiments with random sampling are somewhat worrying---the performance improvement of the proposed re-sampling scheme is quiet minor over random resampling. In the future, the authors may want to investigate the random resampling for the systems in figure 14." Without resampling, the proposed method is a location/scale state-independent GMM policy. It is interesting that this outperforms the fully state-dependent GMM, and the authors could investigate that further. To justify the additional complexity of the resampling step, the authors need to perform further investigation and move that to the main text.

In addition, the evaluated environments omit standard OpenAI gym environments (which the authors do have access to as evidenced by their experiments w/ DDPG on them in the Appendix). This makes evaluating baseline method performance challenging. Furthermore, the authors cite Tang & Agrawal (2018) which introduces a normalizing flow policy that outperforms GMM. It would be natural to compare to that baseline. Finally, Figurnov et al. (2018) among others shows how reparameterization gradients can be computed through GMMs. The authors should explain why this is not applicable.
Decision: Reject