Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary
=========
Authors investigated the effect of approximation error for actor-critic. They derived an upper bound of approximation showing that minimizing the KL divergence between the two consecutive policies can drive this upper bound down. Based on their finding they  introduced the Error Controlled Actor-critic (ECAC) algorithm. They ran ablation study showing the positive impact of minimizing the KL divergence. Furthermore they compared ECAC against 4 state-of-the-art techniques showing their advantage across 4 out of 5 Mujoco domains.

[+] Aside from minor grammar issues (see details) the paper is easy to follow

[+] The trick to calculate alpha and beta automatically is very interesting

[+] Empirical results are encouraging 

[-] The experimental result section can enjoy more rigor. It is not clear how many seeds were used to generate these results. As Joelle discussed in one of NeurIPS conferences, it is easy use limited number of seeds and deduce very different outcomes (https://media.neurips.cc/Conferences/NIPS2018/Slides/jpineau-NeurIPS-dec18-fb.pdf). Also why showing min and max of returns instead of something more common such as std-err.

[-] I think Actor-critic has been shown to converge before. Why the proof using Banach’s Fixed Point Theorem important compared to other proofs?

Details
=========
P2, overestimation. => Need space after .
Section 2.2: hypothesis. And the => Remove "And"
Section 4.2: 
	- that can be views adapting => that can be viewed as adapting
	- I recommend providing the difference between Q-networks and its target at n+1 iteration in a source distribution before the corresponding value on the target distribution (e.g. replace Eqn 11 and 12 and the accompanying text)
	- While the difference between D^n and D^n+1 is small, the corresponding error introduced can be big
	- I believe the use of minimized KL divergence between policies has been practiced before in the RL literature.
Section 4.3: ).In order => ). In order
Page 7: 
	- Curves are smoothed uniformly for visual clarity" => What do you mean?
	- using different random seeds => How many random seeds? Why not mention here? I could not find it in the Appendix either
	- shaded region to the minimum and maximum returns over the five runs => Why not showing standard error instead to have a better statistical significance understanding visually?
	- Use vectorized image formats so your images are zoomable without the loss of quality
Eqn 23: Q_{ture} => Q_{true}
Page 8:
	- "with KL limitation have" => "with KL limitation has"
Section 5.2:
	- "ECAC outperforms all other algorithms on the tasks except" => What does beating mean? Have you ran statistical analysis or are you relying on the mean values?
review_title: A more analysis on a known trick in reinforcement learning

final_score: rating: 3: Clear rejection
review_content: Before beginning with the review, I would strongly request the authors to correct grammar errors, spelling errors, and notation inconsistencies which seriously hinder understanding in some cases.

This paper explores error accumulation in actor-critic methods. The authors claim to present an analysis of approximation error in actor-critic methods (though I couldn't find any concrete analysis of this fact) and then suggest that bounding the KL-divergence of the policy against the previous policy aids actor-critic algorithms by deriving an upper bound on the change in TD error due to faster changes in the policy.

My overall opinion is that this paper restates results that are already well known and the general theme of the method ECAC is already explored in RL literature (see KL-control literature). So, I don't really think there is any novel concept presented in the paper. I move to specific points next:

- Theorem 1: I don't see what's novel here. Such results have been shown in the absence of function approximation for off-policy actor-critic (Degris et al. 2012). What's new here?

- Analysis of approximation error: In the abstract, it is claimed that the paper performs an *analysis* of this error. I could not find this anywhere in this paper. If this refers to Section 3.2 (Hidden dangers in AC methods), I would not call that "analysis". It states some known facts without actually discussing anything formally there. So, either this claim should be removed or a valid analysis needs to be presented.

- The reason for why KL-constraints work: I can see what the derivation does and how the authors arrive at the conclusion of KL-constraint I do not buy the argument that KL-constraints help learning because they control TD error -- they do stabilize the critic, as would a smaller learning rate for the policy would and lead to lower accumulation of error (for e.g., see A theory of Regularized MDPs, Geist et al. or Leverage The Average: An Analysis of KL regularization in RL) and these analyses are far more sophisticated that the argument presented in this paper. 

- The empirical results in the paper are OK, but the baselines utilized are a bit suboptimal, for e.g. SAC and TD3 in HalfCheetah -- having used these myself, it seems like official codebases of these papers do get better performance than what is reported.  I think the numbers are this unreliable.

I think overall the algorithm with the KL-constraint is a good idea -- as has been explored several times in literature -- but I don't think this paper cites that work, nor compares to them along with not a rigorous enough evaluation and poor writing quality.
review_title: Lack of novelty, Inadequate comparison to prior work, Suboptimal baselines 

final_score: rating: 3: Clear rejection
review_content: 
Clarity:
---------
Major:
******
It is not clear what the actor critic algorithm is. What is presented in Fig 1 seems to be policy iteration, and Theorem 1 seems to be stating policy iteration (yet $\pi^{n+1}$ has not been defined). 

Minor:
*****
1) What is $V^{\pi^n}$? Is it the value function of the policy $\pi^n$.
2) What is the difference between $\pi^{*,n}$ and $\pi^{n+1}$.
3) What is $\pi$ is eq(3)?


Quality:
-----------
1) The argument following eq (15) is very informal: "the number of samples in $\mathcal{D}^{n+1}$ is only a little more than in $\mathcal{D}^n$".
2) For the contraction property of the Bellman operator, the author can consider citing standard text books instead of "https://towardsdatascience.com/ mathematical-analysis-of-reinforcement-learning-bellman-equation-ac9f0954e19f"


Significance:
------------------
1) The  authors claim "We present a convergence analysis for actor-critic methods by Banach’s Fixed Point Theorem" to be one of the contributions. The fixed point theorem here is to show that Bellman operator is a contraction operator (a well known fact).
2) The experiments are on only 5 domains. 


Overall Feedback: Adding the KL based penalty is perhaps a useful idea (this may also not be an entirely new one). It will be great if a) the algorithm can clearly stated,  b) the novelty of the method pointed out clearly by comparing it with related works c) the ideas can be supported by formal theoretical statements.
review_title: Seems to be a heuristic (perhaps not entirely new) presented in a not so clear manner

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper, the authors study the error introduced by the estimation of critic function in the Actor-Critic algorithm. Then the author proposed an algorithm that utilizes the idea of double Q learning and using a KL-divergence like regularization method to control this error. Experimentally the proposed algorithm achieves good results comparing to the vanilla Actor-Critic algorithm. This paper shows a successful routine from the theoretical analysis to a practical algorithm.

However, one of the drawbacks of this paper to me is, the authors proposed the double Q learning method. It is not clear how this double Q help to reduce the estimation error? I am also a little bit concerned that the gap between theoretical analysis and the practical algorithm is large. In practice, the authors always use a one-step update, but there is no guarantee that the one-step update for both actor and critic can decrease the error or improve the performance. 

Also, since the key idea of how to control the critic is to use a regularization based on the last policy to make sure the policy move not so 'fast'. It would be interesting to ask what will happen if we only use a smaller learning rate for the actor and a larger learning rate for the critic. Some theoretical work [1] [2] have already adopted this method to get the error controlled and make the actor converge to the stationary point. Therefore, it would highlight the result of this paper if the author can compare their algorithm and show the advantage over these algorithms that only set different learning rates. 

Given the contribution of the newly proposed algorithm, I will suggest accepting this paper.

[1] Wu, Yue, et al. "A Finite Time Analysis of Two Time-Scale Actor Critic Methods." arXiv preprint arXiv:2005.01350 (2020).

[2] Hong, Mingyi, et al. "A two-timescale framework for bilevel optimization: Complexity analysis and application to actor-critic." arXiv preprint arXiv:2007.05170 (2020).

--- Update after discussion

I agree with other reviewers' idea that the contribution of this paper is somehow limited. Since my previous suggestion (7, accept) is based on that the author can successfully address how to control the error of the critic theoretically. However, with only the experimental elaboration study, as I said in the response to the author, the contribution of this paper is limited. Thus, I would switch back to 5 (marginally reject) based on that after reading other reviewers' discussion.
review_title: Good paper starting from theoretical analysis to practical algorithms

Meta-review: The majority of the reviewers believe that this paper is not ready for publication. Among their concerns is that the paper has limited novelty, especially in relation to existing work that use the KL constraint. Some of the reviewers also believe that the arguments are sometimes hand-wavy and not rigorous. For example, in the discussion period after Nov. 24th, it is mentioned that the argument by the authors that "KL constraint==>decrease the approximation error==>increase performance" is not precise enough. I encourage the authors to take these comments into account and improve their paper.
Decision: Reject