Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors present a (mostly) deterministic policy gradient method. They derive a theoretical algorithm showing it recovers the same policy gradient estimator.

This is then evaluated in practice showing that it performs similarly to other on-policy methods when tackling standard mujoco tasks. In addition, they show impressive performance gains when tackling long-horizon non-markovian rewards that rely on frequency domain matching.
strengths_and_weaknesses: Strengths:

- They present a (mostly) deterministic method that is capable of solving long-horizon problems in the frequency domain. A task that prior works are shown to be incapable of tackling.
- Results in the frequency domain are impressive.

Weaknesses:

- The premise of the paper is a truly deterministic policy gradient method, yet Algorithm 2 is not truly deterministic. Even though q is sampled from less often than off-policy exploration schemes, this still results in additional noise injection.
- Overall the work is presented as if the major benefit is true determinism whereas I feel the major benefit here is in the ability to tackle long horizon sparse reward tasks.

Minor points:

- I argue the authors to go over the paper with a spelling/grammar correction tool (such as grammarly). There are many mistakes such as "appeare" which would be trivially solved with minimal effort.
questions: - DDPG and TD3 are shown to fail. These tasks are by design partially observable or history dependent. DDPG (and variants) are by design off-policy history agnostic methods. Are PPO/TRPO/TDPO utilizing memory modules or are they only observing the current state?

- What happens when the problem is shorter term with more frequent rewards? For instance, the frequency can be estimated using STFT (for instance) and combining multiple per-step rewards. This might not be the exact same objective, but it's a denser task that is interesting to investigate.

- How do the various methods compare on sparse, non-markovian, rewards that are not necessarily frequency based?

Overall, while the experiments are impressive, I feel they do not cover the full analysis expected of this method. If the point is fully deterministic, the question is where does this help the agent perform better? How does this improve performance. Is it only in sparse reward tasks? Does it demand non-markovian nature or not? Why do previous on-policy methods fail in non-markovian returns (TRPO/PPO)?
limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: A solid paper, which I think represents a valuable contribution to the NeurIPS community because it tackles an important problem (improving actor-critic methods) in a novel way. The major weakness is in presentation, especially of issues around exploration. With deterministic policies, we need to be very careful to explore enough, but the paper is remarkably silent about what policy is used for exploration. 

The literature review section is missing OAC [1], whose deterministic version explores using deterministic policies, solving an optimisation problem that involves the Wasserstein divergence. 

Typos:
- line 152: well-suited form => in a form well-suited
- line 287: widely-test => widely tested

References:
[1] Ciosek, K., Vuong, Q., Loftin, R., & Hofmann, K. (2019). Better exploration with optimistic actor critic. Advances in Neural Information Processing Systems
strengths_and_weaknesses: Strengths:
1. The paper addresses an important issue of improving policy-gradient-based RL agents. This has huge practical significance since most RL agents used in practice are based on policy gradients.
2. The empirical evaluation is limited in scope, but very solid, with 100 independent runs.
3. As far as I can tell, the proposed update to the policy is novel.
4. There is an experiment on a real robot.

Weaknesses:
1. The presentation could be improved, particularly concerning:
    - how the algorithm explores
    - the pseudocode needs to be made more specific (make it clear that algorithm 2 is invoked in line 2 of algorithm 1).
2.  The simulated benchmarks seem a bit artificial.
questions: 1. Please provide more details about how the exploration policy is chosen. Moreover, can you clarify the pseudocode of Algorithm 1 to make it clear where the exploration policy comes form?
2. Can you explain what you mean by constraints in line 160? The optimisation objective in equation (7) does not seem to come with any constraints.
3. In line 116, you take the 2-norm of a Lipschitz constant. What does that mean given the Lipschitz constant is defined as a scalar?
4. In practice, can you explain how injecting noise into the transition dynamics you describe in the beginning of section 4.3 is different from injecting noise into the action distribution (which is what vanilla TD3 and DDPG do).
5. In the top part of Figure 1, can you explain why the different algorithms have vastly different performance at zero samples. 
limitations: The authors do not explicitly discuss the limitations of the work, except for the last sentence in the Discussion section. In my view, the major limitations are:
1. Empirically, while the evaluation is statistically excellent (the authors use 100 random seeds), the authors only test on relatively simple environments. Moreover, other than pendulum, the used environments are non-standard and seem to have been introduced with the proposed algorithm in mind. This presents a risk that the good performance of the proposed method is due to overfitting a cherry-picked benchmark. However, the authors do alleviate these concerns to a certain extent by testing on a real robot.
2. Theoretically, the authors do not show, even in the bandit case, why their method explores better than alternative approaches. You do not have to provide a regret bound (although that would be ideal), but providing some intuitions about how the proposed method is able to explore would be nice.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work proposes a gradient-based algorithm for updating a deterministic policy’s parameters in a reinforcement learning setting. While RL algorithms for training deterministic policies exist, they require the use of a stochastic exploration policy that can introduce variance into policy updates. This variance is especially problematic in environments with non-local rewards and long horizons. The TDPO algorithm uses a deterministic version of the Vine estimator to compute exact advantage estimates, and regularizes the policy updates using the Wasserstein metric, which is uniquely suited for deterministic policies. The algorithm is benchmarked against state-of-the-art baselines on robotic domains with non-local rewards and long horizons, and clearly shows a benefit in achieved performance.
strengths_and_weaknesses: Strengths:
This paper is clearly written, and derives its approach with appropriate rigor. Moreover, it identifies an important problem, proposes a sensible solution, and thoroughly demonstrates the benefits of the work through experiments. 

Weaknesses:
Though this is a very strong paper, I think there is inadequate discussion of its limitations. Namely, it seems to be the cases that any exploration can only result from perturbing the initial condition. It seems that some exploration is necessary in many domains, and it would be worth spelling out how TDPO would fare in such domains. 

There are also a few typos, such as “Since Optuna is widely-test[ed] and…“.
questions: My primary question is: in what environments do you expect the algorithm to perform poorly? My main concern is that it would have inadequate exploration. Though $\epsilon$-greedy is a notoriously inefficient strategy, would this algorithm fair better than, say, Thompson sampling or UCB-exploration, if those algorithms could be tractably applied to the same problem setting?
limitations: The discussion of limitations in this work is restricted its scope, i.e. that the algorithm is designed for continuous state and action spaces only. These is no discussion of negative societal impact, though I think such a discussion is unnecessary for this work. Regarding scope, and as stated in the Strengths & Weaknesses section, I would add a discussion on how the algorithm would fare in settings where exploration is required, and when to avoid using it in favor of another algorithm.
review_title: null

Meta-review: metareview: The paper addresses the problem of long-horizon policy learning with non-local rewards via PG that avoids noise injection. The reviewers are in consensus, and I concur, that this paper would be a welcome contribution to NeurIPS because it addresses an important problem, makes a solid contribution, and shows impressive results. The authors' rebuttal sufficiently addressed putting more emphasis on the long-horizon policy learning over the deterministic setting and discussion of the limitations. 

If possible, it would be a nice addition to add a video of the robot experiments in the the final revision.
confidence: Certain
Decision: Accept