Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new policy improvement scheme called conditional cross-entropy method (CCEM), which leverages the cross-entropy method in global optimization to keep track of the set of maximal actions. The proposed algorithm is a variant of the actor-critic methods and called GreedyAC, which (i) uses CCEM for updating the actor and (ii) applies entropy regularization on the loss function of the actor. The proposed method enjoys the advantages of (i) being non-sensitive to the entropy regularization parameter, (ii) concentrating on the greedy policy more slowly, (iii) having better empirical results in some environments than SAC, and (iv) enjoying policy improvement guarantee and convergence.
strength_and_weaknesses: **Strength**
- This paper first introduces a new perspective of applying CEM to RL and thereby proposes the GreedyAC algorithm with theoretical convergence and improvement guarantee.
- The proposed method is not sensitive to the entropy regularization parameter and appears to have some empirical advantage over the popular benchmark SAC.

**Weaknesses**
- While the theoretical results are interesting and could have their own merits, the algorithm analyzed in theory appear to deviate from the actual algorithm introduced in the paper (cf. Algorithms 2-3). 
- There are several concerns about the experimental results:
    - SAC: The reported empirical performance of SAC looks quite different from the existing benchmarks. For example, from Figure 2, it is surprising that SAC fails to solve the simple Acrobot and MountainCar tasks. By contrast, in the existing benchmarks (e.g., Table 1 in https://arxiv.org/abs/1907.02057), SAC appears quite strong in these classic tasks of OpenAI gym. It is not clear what makes the difference, and further explanation is needed.
    - Sensitivity: One of the claims is that GreedyAC empirically could be less sensitive to the entropy regularization parameter than SAC. However, it is not clear why this is true from the design or the convergence analysis. Further justification is needed for this claim, either from intuition, theoretical analysis or ablation studies.
- Some components of GreedyAC look somewhat ad hoc and would require more explanation. For example, the proposal distribution plays an important role in the policy improvement of GreedyAC, and it relies on an entropy regularization term.
- Several missing references on CEM for RL: There are several prior works on the combination of CEM and value-based methods, such as [Simmons-Edler et al. 2018] and [Shao et al. 2022]. These references are missing in this paper.

[Simmons-Edler et al. 2019]  Riley Simmons-Edler, Ben Eisner, Eric Mitchell, Sebastian Seung, Daniel Lee (2019). “Q-Learning for Continuous Actions with Cross-Entropy Guided Policies”. In The Reinforcement Learning for Real Life (RL4RealLife) Workshop in the 36th International Conference on Machine Learning (ICML), 2019.

[Shao et al., 2022] Lin Shao, Yifan You, Mengyuan Yan, Shenli Yuan, Qingyun Sun, Jeannette Bohg, "GRAC: Self-Guided and Self-Regularized Actor-Critic." Conference on Robot Learning. PMLR, 2022.


clarity,_quality,_novelty_and_reproducibility: **Clarity**: 
The writing is clear in most places. Here are some places that could be improved:
- Regarding the convergence analysis (cf. Theorem A.1), only informal results are provided in the main text. While this arrangement could provide intuition, on my first read, it is a bit confusing to me what kind of convergence result has been established in the paper. I would suggest that the authors at least provide a formal statement of Theorem A.1 as this result appears to be one of the major components of the paper.
- The caption of Figure 1 is somewhat confusing. It would be helpful to clarify the description of Figure 1.
- It is not clear how to parametrize the percentile-greedy policy by some parameter $w$. 

**Novelty**

Please see the above comments about the weaknesses and the missing references.

**Reproducibility**

As mentioned above, the performance of the baseline method are somewhat different from the existing public benchmarks.

summary_of_the_review: This paper presents a new perspective of applying CEM to policy improvement. While the theoretical result is interesting, there are some issues with whether the analysis actually captures the behavior of the proposed algorithm as well as with the empirical evaluation of the baselines and claim on sensitivity. Moreover, the novelty of this paper should be further clarified as there are some missing references that look relevant but not discussed in this paper.

========== Post-rebuttal ==========
I would like to thank the authors for providing a very detailed response. My main concerns, the difference between the algorithm and the analysis as well as the experimental results of the benchmark methods, have been addressed. As a result, I decided to raise my score from 5 to 6.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes an Actor-Critic algorithm in which the actor learns to pursue (imitate) the greedy policy of the critic, using, with continuous actions, a method inspired from the Cross-Entropy method (pursuing the top-K actions from a sample of relatively good actions). With continuous actions, the core contributions of the paper are the suggestion of using CEM to approximate $\argmax_a Q(s, a)$, and the introduction of an entropy-regularized proposal policy $\pi_{w'}$, to choose which actions to sample for computing the top-K of $Q(s, a)$.
strength_and_weaknesses: Strengths:

- The algorithms is simple, elegant and performs very well even on challenging tasks (Atari games, Mujoco tasks)
- The paper is clear and well-written

Weakness:

My use of "pursue" in "Summary of the paper" is on purpose. This paper actually presents an algorithm for continuous-action Conservative Policy Iteration [1], which can also be seen as continuous-action state-dependent Pursuit [2]. There is a large and exciting field of Reinforcement Learning that considers gradient-free optimization of the actors, using imitation learning to pursue the greedy policy of critics. Such work include:

- The Actor-Mimic [3], in which a DQN policy is distilled into an actor
- Dual Policy Iteration [4], the formalism behind AlphaZero, in which the actor pursues a proposal policy (in the case of MuZero, obtained by planning with MCTS)
- Approximate Conservative Policy Iteration [1], in which the proposal policy is the greedy policy of an on-policy critic. This paper, with discrete actions (Algorithm 2, discrete), is Approximate Conservative Policy Iteration
- Bootstrapped Dual Policy Iteration [5], close to Approximate Policy Iteration, but that shows that off-policy critics (learned with Q-Learning instead of SARSA) work, leading to extremely high sample-efficiency and hyper-parameter robustness (claims also made in this paper, and mostly due using Dual Policy Iteration instead of Policy Gradient)
- Batch/Offline Reinforcement Learning also has extensive literature on proposal policies and how to sample actions. For instance, Batch-Constrained Q-Learning [6] learns the proposal policy using a conditional Variational Autoencoder. This work is however only distantly related to this paper.

To my knowledge, the proposal policy presented in this paper, and its use to compute approximate greedy policies with continuous actions, is novel. The fact that it works is however not surprising, as the literature mentioned above consistently, over the past years, showed that doing something else that Policy Gradient for actor-critic training is very good indeed.

My points above lead to two remarks:

- Would it be possible for the authors to have a look at the world of Conservative Policy Iteration, Dual Policy Iteration and Policy Distillation, and briefly discuss that in the paper?
- Why is the critic learned with SARSA in this paper? [6] shows that using SOTA critic-learning algorithms (the proposed algorithm combines twin-delayed critics (TD3) with Boostrapped DQN) leads to significant gains in sample-efficiency. The only challenge is that all these learning algorithms assume the existence of $\argmax_a Q(s, a)$, so, discrete actions, but this paper happens to propose a way to compute that max and argmax using CEM and a proposition policy.

[1]: Kakade, S., Langford, J.: Approximately optimal approximate reinforcement learning. In: International Conference on Machine Learning (ICML). pp. 267–274 (2002)

[2]: Agache, M., & Oommen, B. J. (2002). Generalized pursuit learning schemes: New families of continuous and discretized learning automata. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 32(6), 738-749.

[3]: Parisotto, E., Ba, J. L., & Salakhutdinov, R. (2015). Actor-mimic: Deep multitask and transfer reinforcement learning. arXiv preprint arXiv:1511.06342.

[4]: Sun, W., Gordon, G. J., Boots, B., & Bagnell, J. (2018). Dual policy iteration. Advances in Neural Information Processing Systems, 31.

[5]: Steckelmacher, D., Plisnier, H., Roijers, D. M., & Nowé, A. (2019, September). Sample-efficient model-free reinforcement learning with off-policy critics. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (pp. 19-34). Springer, Cham.

[6]: Scott Fujimoto, David Meger, and Doina Precup. 2019. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning. 2052–2062
clarity,_quality,_novelty_and_reproducibility: - Significance: very high. The results presented in this paper are very strong, and the approach solves a problem (continuous-action Conservative Policy Iteration)
- Novelty: moderate. There is a whole field of RL in which the actor imitates some (greedy) function of the critic, and this work builds on/reinvents parts of that.
- Calrity: high. The paper is well-written and easy to follow.
summary_of_the_review: Very good idea, but learning the critic with SARSA? It seems that it helps with the proof, but maybe not the empirical performance. Other algorithms that train the actor towards target actions (as opposed to using Policy Gradient) should be discussed in the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes the Greedy Actor-Critic approach, which uses two policies, and a new CCEM approach: a conditional (on states) varianta of CEM. The critic is trained using Sarsa updates (with a replay buffer). A *proposal policy* (one of the two policies) is used to sample a distribution of actions, which are then evaluated and ranked by the critic. The top N of these actions are used for training the policies. The main policy (the second of the two policies) is used to select actions to be executed in the environment. Both policies are updated such that they increase the likelihood of the top N actions mentioned previously, but the proposal policy additionally has an entropy regularization term (which the main policy doesn't).
strength_and_weaknesses: **Strengths**:
- While the paper is rather technical and dense, in general it is possible to follow along with the main ideas.
- Theoretical and algorithmic contributions appear to be sound, with good emprical results.

**Weaknesses**:
- In several parts of the more technical writing, it seems that there are slight mistakes and confusing notation (notation suddenly changes or is not explained at all), making some parts difficult to follow (see detailed comments below).
- There are some statements where I'd appreciate a bit more intuition/explanations (see detailed comments below).

---

**After discussion**: the revision of the paper has fixed and clarified many of my concerns, and I have updated my score.
clarity,_quality,_novelty_and_reproducibility: **Clarity**: at a high level good, but have some concerns for some details.

**Quality**: sound as far as I can tell.

**Novelty**: good.

**Reproducibility**: only concern I have is due to clarity concerns for some technical details, as mentioned above.

---

**Detailed Comments**:
- p.2: "attempt to obtain (unbiased) estimates of this gradient" --> what is "this gradient"? I think you mean "the gradient of this objective"?
- Third paragraph of Section 2: suddenly a boldface $\boldsymbol{w}$ is introduced as subscript for a policy $\pi$. Earlier in the paper, $\theta$ was used to denote the parameters of a policy $\pi_{\theta}$. Why did this suddenly change to $\boldsymbol{w}$? Is it the same thing, or something else? Without explanation, I would assume it was just the same thing... but later in the paper I keep seeing unexplained mixes of $\boldsymbol{w}$, $\theta$, and even yet again a new $\boldsymbol{w}'$. I can guess that that last one is used for the proposal policy, due to how it is used in the update that also involves entropy regularization... but I really shouldn't have to guess this, this should be clearly explained. In the later parts I think $\theta$ is more commonly used only as subscript for $Q$, but (especially early in the paper) also for policies, so... I continue to be confused throughout all of the paper.
- p.3: "for $h < N$" --> I think this should be $h \leq N$?
- p.3: "narrows around higher valued $\theta$" --> technically this can be read as saying that it is simply trying to make the values of the parameters $\theta$ high. I think you rather mean that it narrows around parameters $\theta$ that lead to policies that achieve high returns.
- "CEM, however, finds the single-best set of optimal parameters" --> does it really find the optimal parameters? Or does it just try to approximate them?
- Figure 1 has equations saying that policies are equal to certain gradients. Are they really equal to the gradients? Or are their parameters just updated according to those gradients?
- p.4: "The actor $\boldsymbol{w}_t$" / "The proposal $\boldsymbol{w]_t'$" --> do these symbols actually denote the full actor/proposal [policy]? Or just their parameters?
- At the end of page 4, I do not understand why a proposal policy is no longer necessary for problems with discrete action spaces, and why it would no longer be useful to sample a distribution of actions. To me, exactly the same intuition would still apply. I think it would be useful to provide more explanation here.


**Other Minor Comments**:
- p.3: "are sample from" --> are sampled from
- Caption Fig 1: "more quickly that" --> more quickly than
- The legends for Figures 2 and 3 need to be slightly bigger to be readable.
- p.9: SOTA --> I know what the acronym means, but it is never fully written out and should not be used without fully writing out at least once.
summary_of_the_review: Potentially a good paper, but currently having slightly too many issues with imprecise statements and confusing / unexplained notation.

---

**After discussion**: the revision of the paper has fixed and clarified many of my concerns, and I have updated my score.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: One reviewer was quite excited by the paper to the point where they replicated the results.  Quoting from that review:

This paper presents an algorithm for continuous-action Conservative Policy Iteration which can also be seen as continuous-action state-dependent Pursuit.  To my knowledge, the proposal policy presented in this paper, and its use to compute approximate greedy policies with continuous actions, is novel.

A couple initial reviews were negative based on disappointments in the quality of the paper but scores were later raised in response to author feedback and improvements in the paper.  While one was excited, they also said:

The fact that [this] works is however not surprising, as existing literature consistently, over the past years, showed that doing something other that Policy Gradient for actor-critic training is very good indeed.

The novelty of the work and the reproducibility of the results were, in the end, judged to warrant publication.


justification_for_why_not_higher_score: The methods, while simple and novel and reproducible, are not surprising given existing the literature.
justification_for_why_not_lower_score: The methods do make a contribution that should appear in the literature.
Decision: Accept: poster