Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: In the paper "Adaptive Discretization for Continuous Control using Particle Filtering Policy Network", the authors introduce a new way to discretise the action space of agent in RL settings by using a Particule Filtering approach. The main idea is that the learned policy will output the weight of each particle to define which one should be used, while the position of the particle changes during the learning process. Particles that are not moved (because they have a weight that is systematically too low) are removed and resampled from other particles. 

The paper is well structured and clearly illustrated. 
However, I am unable to understand from the text why the proposed approach is not subject to the curse of dimensionality. In particular, if for every action dimension 10 to 35 weights should be defined by the policy, this creates a significant increase in the size of the search space (35^N). In particular, I would appreciate seeing a discussion about the size (in terms of the number of parameters) of the policies. 

It is also quite surprising to see that the considered task in figure 2 can mostly be solved with very few particles (for both of the compared approaches). Therefore, I don't understand why in the last experiment of the paper (figure 3), the compared approaches use 35 vs. 200 particles. The complexity of the search space is certainly impacted by the difference and the observed results might just be the result of this. I am also wondering how the analysis can be extended to the other experiments. It seems that the performance difference on the other tasks is significantly more subtle and it will be important to discuss this. 

Overall, the paper proposes an interesting method but should be more didactic on why it is not impacted by the issues that are affecting the other approaches.  
review_title: The paper proposes an interesting method but should be more didactic on why it is not impacted by the issues that are affecting the other approaches.  

final_score: rating: 5: Marginally below acceptance threshold
review_content: 1. Optimal policy should be unique, and please make corrections for motivation. Flexible policy representation is helpful when the agent is not confident about the environment (exploration). The energy-based policy is helpful for efficient exploration, as it can better model the uncertainty of an agent.

2. Particle filtering is relatively computationally intensive. Could you please provide a comparison of the running time?

3. The main contribution of this paper is a GMM policy using particle filtering. This direction is not that exciting. The initial version of SAC does use the GMM, but the final version uses isotropic Gaussian. Particle-based variational inference with RL has also been explored by Stein Variational Policy Gradient and Policy Optimization as WGFs. It is good to see improvement empirically, but I am not sure the improvement source. An ablation study on this should be useful.

4. It is interesting to see the differentiable reparameterization trick. I cannot verify its correction. What does 'arg max x' mean?  If one prefers the flexible representation, the energy-based policy as in Soft Q-Learning should be a good choice. The dead particle issue will not happen as a repulsive force added in SVGD gradient step.
review_title: GMM Policy implemented by Particle Filtering

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary
----------

This paper presents an approach to multimodal policies based on Gaussian mixtures. The policy is parameterized as a set of Gaussian distributions (with state-invariant mean and variance) weighted by state-dependent mixture weights, which are the output of a (softmaxed) network. The weighting network and the means and covariances of the Gaussians are updated with standard RL losses. The authors propose a resampling scheme for mixture elements that consistently have low weight, in the style of resampling in particle filters. The authors evaluate the method with several RL algorithms including PPO and SAC, and on a variety of environments. 

Comments
----------

Overall, this paper is pointing in an interesting direction. Expressive action distributions are often an afterthought in RL (although they have seen increasing attention with SAC), and this work is an interesting step in that direction. However, there are several limitations that, if addressed, would strengthen the paper substantially. 

Overall, the evaluation is quite thorough. However, the authors do not include a comparison to a standard mixture of gaussians policy, in which the Gaussians mixture elements are also state-dependent. The authors state that the mixture of Gaussians presented "cannot support any discretization scheme". What is meant by this? The PFPN approach does not actually "discretize" the action space, it simple uses a mixture of Gaussians to parametrize the policy to yield a multimodal action distribution. Thus, the fully state-dependent mixture of Gaussians seems like the most relevant comparison for the PFPN approach, and should be able to rely on the sample Gumbel-softmax trick for differentiable sampling. 

The most interesting contribution of the paper is the resampling scheme. However, there is minimal evaluation of the benefit of this scheme. While section H of the appendix is a good start, I would expect a much more thorough investigation of the resampling, as it is probably the most substantial novel contribution of this work. Indeed, the current results for the resampling are somewhat worrying, as they show only a minor performance drop for removing the resampling entirely. An evaluation of the value of resampling (and the hyperparameters associated with resampling) on a variety of environments would strength the paper. 

Finally, the paper should better address the value of expressive multimodal policies beyond just performance increases in standard environments. This is addressed briefly in section E of the appendix, but a broader discussion of the benefits of the approach would help make clear the benefits associated with the PFPN approach. 

Post-Rebuttal
----------

I thank the authors for their response. The additional baseline comparisons do strengthen the paper, and I have increased my score from 4 to 5. I agree with the authors that the mixture of Gaussians policy is substantially weaker than their method, and is a useful baseline experiment to have. 

However, the added experiments with random sampling are somewhat worrying---the performance improvement of the proposed re-sampling scheme is quiet minor over random resampling. In the future, the authors may want to investigate the random resampling for the systems in figure 14.
review_title: Review 2

final_score: rating: 7: Good paper, accept
review_content: Post rebuttal update: The authors have addressed my concerns and the revised submission is much clearer, so I'm increasing my score to 7.


I must preface this with a caveat that I am not up to date with the latest results in continuous deep RL. That being said I think this is a nice and reasonably well-written paper, albeit with some weaknesses.

Major comments:

The biggest weakness of this paper appears to be that the results are only minor improvements (if they improve at all over the baseline). This is a major shortcoming because the proposed method is significantly more complicated than the baselines. If this additional complexity, and the additional hyper-parameters, does not bring major benefits then this begs the question as to whether or not it is worth it. On this point, what is the additional computational cost of the proposed method over the baselines?
If there is some other advantage to using this proposed method over the others then this should be explained more clearly up-front. I think a 'Contributions' section that listed the major takeaways of the paper would be useful.

Practically none of the mathematical quantities are defined correctly. If you have a mathematical object it should be defined as a member of a particular space, e.g., x in R^n, or s \in \S, or \pi \in \Delta_A. Otherwise it's very hard to reason about what these objects are. This is a major clarity problem.

Minor comments:

Why is eq 6 the product over the action dimensions 'k'? Isn't each dimension independent in this setup? The current formulation makes the policy appear to be a scalar and I was under the impression that it was a vector, although as previously mentioned this isn't clear.

The notation in eq 9 (mu and sigma on top of each other) is very strange, probably better to use \theta = (\mu, \sigma).

I don't see what you used for the At estimate. Is that explained anywhere?

I don't fully understand section 3.4, it's quite unclear as written. Why is the two-step sampling method 'non reparameterizable'? What does 'non reparameterizable' even mean? This needs a serious definition. Also the 'concrete' distribution is not defined here, it needs to be discussed more generally for completeness, possibly in the appendix is space is an issue.

Why is your method 'more friendly to policy gradient optimization' than simpler methods? Can this be quantified some way more concretely?

A few missing related works occurred to me (from the policy gradient / value learning literature):
https://arxiv.org/abs/1606.02647
https://arxiv.org/abs/1611.01626
http://papers.nips.cc/paper/6870-bridging-the-gap-between-value-and-policy-based-reinforcement-learning.pdf

Conclusion and next steps for the authors:

Overall I liked the paper, but cannot recommend it for publication in it's current form.

If the paper is improved to be clearer I will increase my score, in particular it needs better explanation of the concepts in some sections and every mathematical object should be defined rigorously. I dropped a point for clarity alone.

If the significance of the work, and the computational results can be better explained or improved I could increase my score again.
review_title: Adaptive Discretization for Continuous Control using Particle Filtering Policy Network 

Meta-review: This paper is rejected.

The authors contributions are:
* Propose PFPN as an expressive action policy for continuous action spaces.
* Introduce a reparameterization trick for training PFPN with off-policy methods.
* Experiments claiming PFPN outperforms unimodal Gaussian policies and a uniform discretization scheme and that it is more sample efficient and stable across different training trials.

I and the reviewers appreciate the additions by the authors. The GMM baseline is an important addition addressing concerns from several reviewers. However, I agree with R2's comment that "most interesting contribution of the paper is the resampling scheme. However, there is minimal evaluation of the benefit of this scheme [...] However, the added experiments with random sampling are somewhat worrying---the performance improvement of the proposed re-sampling scheme is quiet minor over random resampling. In the future, the authors may want to investigate the random resampling for the systems in figure 14." Without resampling, the proposed method is a location/scale state-independent GMM policy. It is interesting that this outperforms the fully state-dependent GMM, and the authors could investigate that further. To justify the additional complexity of the resampling step, the authors need to perform further investigation and move that to the main text.

In addition, the evaluated environments omit standard OpenAI gym environments (which the authors do have access to as evidenced by their experiments w/ DDPG on them in the Appendix). This makes evaluating baseline method performance challenging. Furthermore, the authors cite Tang & Agrawal (2018) which introduces a normalizing flow policy that outperforms GMM. It would be natural to compare to that baseline. Finally, Figurnov et al. (2018) among others shows how reparameterization gradients can be computed through GMMs. The authors should explain why this is not applicable.
Decision: Reject