Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a new deterministic policy gradient method (TDPO). The main idea and its derivations are based on the use of a deterministic Vine (DeVine) gradient estimator and the Wasserstein metric. The paper shows that a closed-form computation of Wasserstein distance can be derived without any approximation on the deterministic policies. A monotonic policy improvement guarantee is provided. Deterministic Vine (DeVine) is proposed as an optimization method for the proposed surrogate loss function, which is also inspired by the TRPO's surrogate loss. The proposed method is evaluated on two new robotic control domains that are proposed by the authors. The comparisons are against PPO, TRPO, DDPG, and TD3. Additional experiment results on Open AI Gym environments are provided in Appendix.

Overall, the studied idea is interesting. The finding of a closed-form Wasserstein distance on deterministic policies is useful. The new algorithm might be of interest to the RL community. Here are some of my main concerns regarding to this idea and the quality of the paper.

- The motivation of using the first experiment domain with non-local rewards is not clear on why this can other approaches have challenges? Why it could be a showcase of the use of a deterministic policy gradient method. In addition, as DDPG is an off-policy and deterministic policy gradient method, I was wondering why it does not perform well. More detailed experiment settings will be helpful.

- The second domain is basically a Mujoco task with a long-horizon. The results show it favors a deterministic method like TDPO. The same question on why DDPG and TD3 do performs well. They are off-policy methods that can be considered more sample-efficient than on-policy counter-parts like PPO/TRPO. On the other hand, much recent work show TRPO and PPO would perform comparably [1]. Therefore, a detailed experiment setting and a discussion on how each method's hypeparameters get tuned would be important.

- It is also questionable on why the TDPO does not perform well in comparisons to the baseline on Gym suite environments. Because those Gym domains are also based on Mujoco and programmed with deterministic dynamics.


[1] Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, Aleksander Madry:
Implementation Matters in Deep RL: A Case Study on PPO and TRPO. ICLR 2020
review_title: Interesting theoretical results but not convincing experiments.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper introduces a policy gradient method based on deterministic policies and deterministic gradient estimates. By assuming such a deterministic setting, the authors show that the proposed technique can estimate gradients on long-horizon tasks without the need to inject noise into the system for exploration. The authors propose using a Wasserstein-based model to perform regularization in this particular setting involving deterministic policies. They also establish the conditions under which it is possible to guarantee monotonic policy improvement.

Overall, this is a well-written paper with sound mathematical arguments. The authors present a convincing review of related work and argue that the proposed method is better-suited than existing techniques when dealing with problems where stochastic exploration is not efficient or feasible. The authors formally show that it is possible to lower bound the payoff improvement, which is a non-trivial result. All update equations are carefully derived and discussed in the appendix. 

The experiments seem to show that the method works well in a possibly restricted set of settings, involving either control tasks with resonant frequencies or problems with rewards associated with achieving oscillatory behaviors. The experiments on these particular handcrafted problems are well-designed and demonstrate that the proposed method (TDPO) achieves better performance than its competitors. However, the proposed method generally performs worse (on average) than others when evaluated on standard control benchmarks. Even though the experiments may suggest that the method is not widely applicable, I believe that the underlying theory is, on its own, an interesting contribution. 

I believe that this paper introduces a relevant contribution to the RL community that is concerned with scaling up learning to long-horizon problems. It presents a principled method and introduces non-trivial bounds. In my opinion, this conference's community would benefit from having this paper accepted to its proceedings.


I have a few questions for the authors:

1) in the introduction, the authors discuss rewards defined in the frequency domain. Could you please clarify what those are, in terms of the corresponding structure that they may imply in the reward function?

2) the authors also argue that the proposed method is robust in control environments with resonant frequencies. Could you formally define what this means, and what are the implications of this assumption when deploying this method in more general MDPs which may not be modeling robotics problems?

3) in the Background section, P is defined as DeltaS x DeltaA -> DeltaS. Shouldn't it be SxA -> DeltaS, instead?

4) in the Background section, there is an issue with the sentence that begins with "We may abuse notation and replace (...)". I believe that the terms mentioned here should have been delta_s, delta_a, s, a.

5) notation: P is sometimes used to refer to the transition dynamics, but sometimes it refers to the probability of an event. This isn't very clear. See, e.g., the reuse of this symbol when defining P(mu_s, pi).

6) what is P(mu_s, pi(. | s~mu_s))? Is this the joint distribution over mu_s and pi? If "P" denotes a probability, I'm confused: the terms used at this point do not refer to random variables or events, but to distributions. Could you please improve the notation used when defining such a generalized dynamics?

7) you assume the existence of Lip(Q^pi; a). Could you please discuss a few practical problems (with continuous actions) where the Q-function does not vary smoothly with actions? A discussion on the practical implications of this assumption would be interesting.

8) what is the intuition behind Eqs. 5 and 6? What do they mean, intuitively, and how do they relate to the goal of regularizing deterministic policies, and to the goal of ensuring monotonic policy improvement? These seem to be key assumptions to the method, but (in my opinion) they were not well motivated or discussed.

9) in this paper, your focus is on robotic environments with state-reset capabilities. This might be an unrealistic assumption, particularly in robotics problems where the goal is precisely to autonomously learn how to reach particular states or configuration. This assumption might also be reflecting deeper requirements; e.g., that the tasks being tackled are not goal-achieving tasks. Could you please discuss this assumption and its implications in more detail?

10) in practice, how do you pick the coefficients C_1 and C_2 (Eq.13)? How sensitive is your method to this choice?


review_title: This paper introduces a policy gradient method based on deterministic policies and deterministic gradient estimates. The authors show that the proposed technique can estimate gradients on long-horizon tasks without the need to inject noise into the system for exploration. Even though the experiments may suggest that the method is not widely applicable to general MDPs, I believe that the underlying theory is, on its own, an interesting contribution.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a deterministic policy gradient method using the Wasserstein distance to quantify the difference of deterministic policies. It also introduces a new deterministic estimator for the policy gradient with theoretical justification. The paper uses a similar path as TRPO to first introduce a surrogate objective then replace the complex term by a tractable term for practical performance. The method has been shown to work well under long-horizon and non-local reward environments whereas existing methods struggle to solve.

Strength: 
- Being able to deal with non-traditional RL settings can be beneficial to the RL community.
- The adaptation of Wasserstein distance into the surrogate objective of TRPO is new.
- Theoretical justification for the main results.
- Numerical experiments show advantage of the proposed method in non-local or long-horizon settings.

Weakness:
- The presentation of background section can be improved as it is too dense with different notations. It may be a good idea to summarize these notations into a table.
- Only consider 2 environments: 1 with non-local reward and 1 with long horizon and resonant frequencies.

I have the following questions and comments:
Q1. What is the motivation of changing frequency of pendulum to 1.7? instead of 0.5Hz? Have you tried different settings of the pendulum environment (use the original frequency of 0.5Hz)? If yes I am curious how other methods perform compare to TDPO.

C1. It would be nice to have a discussion to compare the computation cost of each iteration of TDPO and other methods like TRPO and TD3.

C2. As shown in the experiments, TDPO does not work well in the common gym environments, is there any changes to algorithm design possible to improve the current algorithm for these setting?

C3. From the way the method is presented, it seems that it is not simple to implement the method? It is better to have a discussion on the implementation aspect of the method.

Small comments: 
- Secion 1, first paragraph, line 6, there are two \delta_s
- The definition of KL divergence in terms of Hilbert space inner product missing a subscript?
- Introduce the definition of the term h.o.t. before use although it can be inferred from the context

Overall, I suggest a weak-accept decision based on the following reason:
- Novelty in algorithm development with theoretical justification for the monotonic improvement.
- Although experiments indeed show the clear advantage of the proposed method over existing ones, more environments or more setting in the same environment should be presented to better evaluate its performance.

review_title: Blind Review #4

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper builds on the template of TRPO and proposes an algorithm that searches for deterministic policy. A novel policy improvement lemma is derived for Wasserstein distance and thus a new algorithm is proposed based on iteratively optimizing policy performance lower bound.

I feel overall this paper may not be quite well motivated.  When describing the disadvantage of stochastic policies, it is better to make it clear. It is hard to get why stochastic policies are not able to handle non-local rewards, long time horizons, and naturally-resonant environments, in what aspects? What exactly means non-local rewards? Why TRPO can not handle long time horizons? There is no citation for "exploratory noise injection"? Where does the noise be injected? Many terms used here may lack of precise definitions. Overall I suggest the authors to make a clear comparison with TRPO that in what aspect you can improve TRPO. This is better to present the novelty of current works. Some improvements on MuJoCo over TRPO might not be sufficiently enough since RL algorithms involve too much tricks to make the comparison completely fair.

In (9), the theoretical lower bound to be optimized involves C_1 and C_2 that has a very complicated form. For the practical algorithm, I feel you just treat them as a tuning parameter when translating the regularized form to constrained form. Will it be problematic? In Section 5.1, the modification of reward function for the pendulum is a bit artificial. Anyone did this before or what's the reason to do that? Again, this may correspond the motivation of the proposed algorithm.

The presentation of Section 2.1 may not be in an optimal way. This should be a main result but several key derivations (Inequality (31) and Theorems A.5 and A.4) are deferred to the appendix. 
review_title: A variant of TRPO for deterministic policy

Meta-review: This paper proposes a deterministic policy gradient method that does not require to inject noise in the action selection.
Although the reviewers acknowledge that this paper has merits (novel and interesting idea, well written, technically sound), they have some doubts about the motivations for the proposed approach and about its empirical performance: a deeper analysis is requested.
The paper is borderline and needs to be revised before being ready for publication.
Decision: Reject