Error Controlled Actor-Critic Method to Reinforcement Learning
Authors: Xingen Gao
Fei Chao
Changle Zhou
Zhen Ge
Chih-Min Lin
Longzhi Yang
Xiang Chang
Changjing Shang
Venue:  (2021)
Keywords: reinforcement learning
actor-critic
function approximation
approximation error
KL divergence

Abstract: In the reinforcement learning (RL) algorithms which incorporate function approximation methods, the approximation error of value function inevitably cause overestimation phenomenon and have a negative impact on the convergence of the algorithms. To mitigate the negative effects of approximation error, we propose a new actor-critic algorithm called Error Controlled Actor-critic which ensures confining the approximation error in value function. In this paper, we firstly present an analysis of how the approximation error can hinder the optimization process of actor-critic methods. Then, we *derive an upper boundary of the approximation error of Q function approximator, and found that the error can be lowered by placing restrictions on the KL-divergence between every two consecutive policies during the training phase of the policy.* The results of experiments on a range of continuous control tasks from OpenAI gym suite demonstrate that the proposed actor-critic algorithm apparently reduces the approximation error and significantly outperforms other model-free RL algorithms.

Under review as a conference paper at ICLR 2021
ERROR CONTROLLED ACTOR-CRITIC METHOD TO REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
In the reinforcement learning (RL) algorithms which incorporate function ap- proximation methods, the approximation error of value function inevitably causes overestimation phenomenon and impacts algorithm performances. To mitigate the negative effects caused by approximation error, we propose a new actor-critic algorithm called Error Controlled Actor-critic which ensures confining the ap- proximation error in value function. In this paper, we derive an upper boundary of approximation error for Q function approximator in actor-critic methods, and find that the error can be lowered by keep new policy close to the previous one during the training phase of the policy. The results of experiments on range of continuous control tasks from OpenAl gym suite demonstrate that the proposed actor-critic algorithm apparently reduces the approximation error and significantly outperforms other model-free RL algorithms.
INTRODUCTION
Reinforcement learning (RL) algorithms are combined with function approximation methods to adapt the application scenarios whose state spaces combinatorial, large,
continuous. Many function approximation methods RL methods, including the Fourier basis (Konidaris et al.) 2011), kernel regression (Xul 2006, Barreto et al. Bhat Bhat al. al.| 2012)) and neural neworks (Bar- to et al. 1982: Tesauro 1992; Boyan et al. 1992 Gullapalli 1992) have been used to learn value functions. recent years, many deep reinforcement learning (DRL) methods were implemented by incorporating deep learning into RL methods. Deep Q-learning Network (DQN) (Mnih et al.) 2013) reported by Mnih in 2013 a typical work that uses a deep convolutional neural network (CNN) to represent suitable action value function estimating future rewards (returns); it successfully learned end-to-end control policies for seven Atari 2600 games directly from large state spaces. Thereafter, deep RL methods, such Deep Deterministic Policy Gradient (DDPG) (Lillicrap al., 2016), Proximal Policy Optimization (PPO) (Schulman et al., 2017). Twin Delayed Deep Deterministic policy gradient (TD3) (Fujimoto et al.) 2018), and Soft Actor-Critt (SAC) (Haarnoja et al. 2018), started to become mainstream in the field of RI.
Althouth function approximation methods have assisted reinforcement learning (RL) algorithms to perform well in complex problems by providing great representation power; however, they also cause an issue called overestimation phenomenon that jeopardize the optimization process of RL algorithms. Thrun & Schwartz (1993) presented a theoretical analysis of this systematic overes- timation phenomenon in Q-learning methods that function approximation methods. Similar problem persists in the actor-critic methods employed function approximation methods. Thomas (2014) reported that several natural actor-critic algorithms use biased estimates of policy gradient to update parameters when using function approximation to approximate the action value function. Fujimoto et al.) (2018) proved that the value estimation in the deterministic policy gradient method also lead to overestimation problem. In brief, the approximation errors of value functions caused the inaccuracy of estimated values, and such inaccuracy induced the overestimation on value function; so that poor performances might be assigned to high reward values. As result, policies with poor performance might be obtained.
Previous works attempted to find direct strategies to effectively reduce the overestimation. Hasselt (2010) proposed Double Q-learning, in which the samples are divided into two sets to train two independent Q-function estimators. To diminish the overestimation, one Q-function estimator is

---

Under review as a conference paper at ICLR 2021
used to select actions, and the other one is applied to estimate its value. Fujimoto et al. (2018) proposed mechanisms, including clipped double Q-learning and delayed policy updates, to minimize the overestimation.
In contrast to these methods, we focus on actor-critic setting and manage to reduce the approxima- tion error of value function, which is the source of the overestimation, in an indirect but effective way. We use the concepts of domain adaptation (Ben-David et al.) (2100) to derive an upper boundary of the approximation error in Q function approximator: Then, we find that the least upper bound of this error can be obtained by minimizing the Kullback-Leibler divergence (KL divergence) between new policy and its previous one. This means minimizing the KL divergence when traning policy can stabilize the critic and then confine the approximation error in Q function. Interestingly, we arrive at similar conclusion as two literatures Geist et al. (2019): Vieillard et al. (2020) by somewhat dif- ferent route. In their works, the authors directly studied the effect of KI. and entropy regularization in RI. and proved that a KL regularization indeed leads to averaging errors made at each iteration of value function update. While our idea is very different from theirs: It is impracticable to minimize the approximation error directly, so instead we try to minimize an upper bound of approximation
error. This is similar to Expectation-Me Maaitiiiinaaiiiiiittiiin Algorithm (Bishop 2006) which maximize lower bound of log-likelihood instead of log- eiglikelihood directly. We derive an upper boundary of approximation error for function approximatorin actor-critic methods, and arrive at more general conclusion: approximation error can be reduced by keep new policy close to the previous one. Note that KL penalty a effective way, but not the only way.
Furthermore, the mentioned indirect operation (i.e. the KL penalty) can work together with the men- tioned direct strategies for reducing overestimation, for example, clipped double Q-learning. Then, a new actor-critic method called Error Controlled Actor-critic (ECAC) is established by adopting an effective operation that minimizes the KL divergence to keep the upper bound as low as possible. In other words, this method ensures the similarity between every two consecutive polices in training process and reduces the optimization difficulty of value function, so that the error in Q function approximators can be decreased. Ablation studies were performed to examine the effectiveness of our proposed strategy for decreasing the approximation error, and comparative evaluations were conducted to verify that our method can outperform other mainstream RL algorithms.
The main contributions of this paper are summarized as follow: (1) We presented an upper boundary of the approximation error in Q function approximator; (2) We proposed a practical actor-critic method Bethod-ECAAC which decreases the approximation error by restricting the KL divergence between every two consecutive policies and adopt mechanism to automatically adjust the coefficient of KI. term.
2 PRELIMINARIES
2.1 REINFORCEMENT LEARNING
Reinforcement learning (RL) algorithms are modeled as a mathematical framework called Markov Decision Process (MDP). In each time-step of MDP, an agent generates an action based on current state of its environment, then receives a reward and new state from the environment. Environmental state and agent's 's action at time t are denoted as E S and a, E A, respectively: S and A denote the state and action spaces respectively, which may be either discrete or continuous. Environment is described by a reward function, r r(s:, az), and a transition probability distribution, Pr($t+1 = s'|s; = s, a a). Transition probability distribution specifies the probability that the environment will transition next state. Initial state distribution is denoted as (ro
Let denotes policy, 7²x) denotes its expected discounted rewards:
00
<( = En [Ri + +2 + 2-228 R3 + ... Exp
* Re+1],
(I)
where denotes discount rate and 0 < < 1. The goal of RL is to find a policy, x², that maximizes a performance function over policy, (T), which measures the performance of policy:
πt= = arg max (T))
(2)
2

---

Under review as a conference paper at ICLR 2021
A natural form of (xx is n(r). Different interpretations of this optimization goal lead to different routes to the their solutions.
Almost all reinforcement learning algorithms involve estimating value functions, including state- value and action-value functions. State-value function, (*)s), gives the expected sum discounted reward when starting in and following given policy, V (s) specified by:
(*(()) = F Retkt1 s|
(3)
k=
Similarly, action-value function, (s, a), is given by:
("(s,a)= R44441/s: s, a].
(4)
2.2 ACTOR-CRITIC ARCHITECTURE
To avoid confusion, by default, we discuss only RL methods with function approximation in this section.
RL methods can be roughly divided into three categories: 1) value-based, 2) policy-based, and 3) actor-critic methods. Value-based method only learn value functions (state-value or action-value value functions), and have the advantage of fast convergence. Policy-bused methods primarily learn pa- rameterized policies.
parameterized policy (with parameter vector, 0) is either distribution over actions given state, (a|s),) or deterministic function, (s). Their basic update is 0n+1 = On + a V ((,) where is learning rate. Policy based methods show better convergence guar- antees but have high variance in gradient estimates. Actor-critic methods learn both value functions
and policies and use value functions to improve policies. In this way, they trade off small bias in gradient estimates to low variance in gradient estimates.
Actor-critic architecture (Peters & Schaal 2008 Degris et al. 2013 Sutton & Barto 2018) consists of two components: actor and critic modules. Critic module learns learns vaatev function, V (s), or action-value function, (s, a) or both of them, usually by temporal-difference (TD) methods. Actor module learns stochastic policy, To (a(s)), or a deterministic policy, a (s).,. and utilizes value function to improve the policy. For example, in actor module of DDPG (Lillicrap al. 2016), the policy is updated by using the following performance function
(() E., ({()s,)((s)))
(5)
where Th (s:) is a deterministic policy.
2.3 DOMAIN ADAPTATION
Domain adaptation is a task which aims at adapting well performing model from a source domain a different target domain. It is used describe the task of critic module in section 3.2 The learning task of critic module is viewed as adapting learned Q function approximator to next one, and the target error equates the approximation eтrоr at current iteration of critic update. Here, we present some concepts in domain adaptation, including domain, source error, and target error.
Domain is defined as specific pair consisting of a distribution, P, on an input space, X, and labeling function, ʃ X - IR. In domain adaption, source and target domains are denoted as (Ps, fs) and (Pr, [T), respectively. function, X IR, is defined as hypothesis.
Source error is the difference between a hypothesis, h(x), and labeling function of source domain, fs(x), on a source distribution which denoted as follow:
(h, fs) I~Ps
E [lh(x) fs(#)]]
Target error is the difference between a hypothesis, h(x)), and a labeling function of target domain, fr (z), target distribution which is denoted as follow:
er fr. = E [lb(x) [x()]]]
For convenience, we use the shorthand es(h) (h, fs) and er(h) = er()
3

---

Under review as a conference paper at ICLR 2021
3 ERROR CONTROLLED ACTOR-CRITIC
To reduce the impact of the approximation error, we propose new actor-critic algorithm called error controlled actor-critic (ECAC). We present the details of a version of ECAC for continous control tasks, and, more importantly, explain the rationale for confining the KL divergence. In section 3.2 we will show that, each iteration of critic update, the target error equals to the approximation error of Q function approximator. Then, we derive an upper boundary of the error, and find that the error can be reduced by limiting the KL divergence between every two consecutive policies. Although this operation is conducted when training the policy, it can indirectly reduce the optimization difficulty of Q function. Moreover, this indirect operation can work together with the strategies for diminishing overestimation phenomenon. We incorporate clipped double-Q learning into the critic module.
3.1 CRITIC MODULE-LEARNIIH ACTION VALUR FUNCTIONS
The learning task of critic module is to approximate Q functions. In the critic module of ECAC, two Q functions are approximated by two neural networks with weight (1)) and ((²2, respectively. As noted previously, we adopt clipped double-Q strategy (Fujimoto et al. 2018) to directly reduce overestimation. Furthermore, we adopt experience replay mechanism (Lin) 1992) agent's 's experi- ences at each timestep. (st, de. Pt+1,, $t+1), are stored in replay buffer, D; and training samples are uniformly drawn from this buffer. The two Q networks, Qac1) and Qa( are trained by using temporal-differenn learning. Notice that the bootstrapping operation in this setting-uss uses function approximation- means to minimize the following two TDD errors of Q funtion:
= Q Ges (8t, 0t) (Re+1 + min Qato (8t+1; a++1))-
(8)
1=1,2
Notice that clipped double- Q learning uses the smaller of the two values to form the TD error. With the minimum operator, it decreases the likelihood of overestimation by increasing the likelihood of underestimation. The two networks are respectively trained by minimizing the following two loss functions:
L( =
E (a.a.n"))>P s ()),a.)) a' a'~x(:x) x²)
10a (8, a) - (r + min Qsto (8', aa))))
(9)
1112
((
Q0i (r + min s Pl),a,), a' a'~x(- 8')
E ".a...")'PPs
Jases (s', a()))²
(10)
1=12
where D denotes replay buffer, PD denotes the distribution that describes the likelihoods of samples drawn from D uniformly, Pr(-|s, a) denotes the transition probability distribution, and denotes the target policy.
3.2 AN UPPER BOUND OF THE APPROXIMATION ERROR OF Q FUNCTION
For convenience, we analyze the setting with only one Q function. The concept of domain adaptation is used to describe the task of critic module. The learning task of critic module can be viewed as adapting the learned Q-network to a new Q function for newly learned policy. Thus, naturally, target error, i.e. Eq. 7 equates to the approximation error at current iteration of critic update. In this section, we derive an upper bound of the approximation error of Q function and find that the upper bound of approximation error will be smaller if the more similar the two consecutive policies.
Fig T illustrates the training process of an actor-critic method which is a alternating process of value function. At (n + 1) 1)-th iteration, the critic module tries to fit the value function, QTe,, by means of Te.. But because of appriximation error, the actually obtained Q-network (approximator), Q
is not equal to QTa,,, This can be expressed by the following equation:
Q (s, a) = TT (s, a) + 9.a c,11>
(11)
where g.a denotes the appriximation error in Q function given state, s, and action, a. This can be viewed as adapting the learned Q-network, Q to the value function, T Hence, the source distribution is Pr., and the larget distribution is PD As mentioned at the argument following
4

---

Under review as a conference paper at ICLR 2021
Eq. ( and (TO), PD denotes the distribution that describes the likelihoods of samples drawn from replay buffer D uniformly. D and D are replay buffers at n-th and (n + 1) iteration, respec- tively. Qsn is the labeling function. Clearly, the target error here equals the approximation error at current iteration of critic update. This can be expressed by the following equation:
er er(Q0)=
E 8,0~ Dail
[Qo(s, n0n (s,
E
(12)
s,a~
Fit %
Critic
Q %
Critic
TLe
Critic
Q
Critic
Te
Th + EL
Qe.: Q
Qai
n&
Q
nTz
Qs
TT.
Quaa
Actor Improve 7%
Actor Improve Re TE
Actor Improve
To
TB.
Figure 1: Alternating process of Actor-critic alternates between value function and policy updates.
In addition, because TD method is use to estimate Q the real labeling function in target domain actually the following one:
Yme, (s,, a) = Re+1 + 7Q. (s',
Pr(-(s, a' ng, (-18);
(13)
where Pr(-(s, a) denotes the transition probability distribution, and To, denotes the policy updated actor module by using Q Hence, the target error here means the difference between the Q-network, (s a), and its target (labeling function) at (n + 1)-th iteration, Vna.. a target distribution. The actual target error given by
er((s)=
E [1Qg(s,a) Unen a)] s.ampon41
110((, - [Be41 700 (8', a a'llll
s.a pn411
(14)
Furthermore, two types of error are used derive upper bound of appximation error, including source error (Q())) and error es(@u, (ro,,) Source enror error here is the difference between the Q-network, Qo a), and target (labeling function) at the n-th iteration, Vma.-:' on a source distribution. Hence, the approximation error is given by
cs((2)=
E (1Qo(s a) -moo (s.a)]= a.a~Paa
(8)(s, [R411 QQo. a'),
saa Pan ~ Pr(-laaa)
(15)
Error (Q(Q Yno.)) ) is the difference between the Q-network, Q(s, a), and its target (n + 1)-th iteration, Yno. on a source distribution, which is given by
es (Qar Moe.) s.a-Pon
E [10((, a) -Ure, (s) =
E
(1{(8,)) - [Re4 + 20,,(' (8.a"' a')]]]
8, Paa. (r(.
(16)
Target error can be derived into the following inequality:
Ses(Qo) + les(( Vrro.)) - (Qx) + perlle((000 - es(2s) Vrt. )l er er(Q2) =er((()) + es(Qs)
es + es Vne. es(Qsn Vre.
(17)
Ses(Qa) + E [|UTe, (8, ,oan_, (8, a) + (Qr) - es(Q0s VTu. )11
8.a-Ppn

---

Under review as a conference paper at ICLR 2021
where the third term in the third line, (er(Qu) - ess( Mro, )l. is transformed further as:
(Qs) - E5( More )1
(1Qp(s.)) a) Uren (s. a)l
11Q((s.))
Vru, (s, a)ll
s,a~ P Dnll
a, 8,anPoon
(18)
Sy
E 8pa~Pon
E (a
Qes (s',
E doupPet-aaaa) al- (11o))
100, (8', a')]
Recall that actually the replay buffer; and Pp denotes the distribution that describes the likeli- hoods of samples drawn from replay buffer D uniformly. Because, in experience replay mechanism (Lin 1992), the number of samples in D is only a little more than D the difference between PDD and P can be ignored. Finally, the upper bound of error in Q-network is determined by
eT(Qo) (es( +
E e
IlVno, (8, a) Msoo, (8. a]
(19)
E Ppp
E pa.allaa.)) a'mngn (ilo))
19 (6',
E dl d'mme,_1 (16)
10o,
noticeable that the third term in the upper bound will be smaller if the more similar the two consecutive policies, =0.-, and Tp,, are. Hence, we can conclude that confining the KL divergence between every two consecutive policies can help limit the approximation error during the optimiza- tion process of actor-critic. This conclusion is used to design the learning method of the policy.
3.3 ACTOR MODULE-LEARNING A POLICY
The learning task of the actor module is to learn parameterized stochastic policy, To (a(a(s).In order to lower the upper bound of approximation error Q function (or to reduce the optimization dif- ficulty of the Q functions), the goal of the policy is converted from maximizing expect discounted return two parts: maximizing the estimated Q values the minimum of the two Q approximators and, concurrently, minimizing the KI. divergence between successive policies. The optimization objective is specified by
max
E [min (s, (+())) - D (Te(1s)) (- (-18))], P~Po 1-12
(20)
where D is the replay buffer; Pp denotes the distribution that describes the likelihoods of sam- ples drawn from D uniformly; 0 is the parameters of the policy network; Oold is the parameters of the policy updated in the last iteration; (s(s) is the samples drawn from the target stochastic palicyro(alss. Note that, in order to back-propagan the error through this sampling operation, we use a Diagonal Gaussian policy and the reparameterization trick, i.e. samples are obtained according to
ag 8)
(s) + (s(s) @ $ <~N(0, I),
(21)
where (s) and (s)s) are the output of the policy network, and denotes the mean and covariance matrix's 's diagonal elements, respectively.
The KL divergence between two distributions, for example p(z) and q(x), can be thought of the difference between the cross entropy and entropy, which is specified by
Dxl(pll4) = H(p. q) H()),
(22)
where (p, q) denotes the cross entropy between p(z) and q(x), and H()) denotes the entropy of p(z). In practice, we find it is more effective to minimize the cross entropy between two succes- sive policies and to maximize the entropy of current policy, separately, than to minimize the KL divergence directly. Hence, we expand the original objective Eq. 20 into the following one:
max E' [in Qo. (s, (3(s))s) ald (Te Tosi (1s)) + (=)((s)(s
(23)
0 8~Pp 1=12

---

Under review as a conference paper at ICLR 2021
where and denote the coefficients of the cross entropy and the entropy, respectively.
Moreover, we adopt mechanism to automatically adjust a and 3. To do this, a is adjusted by keep- ing the current cross entropy close target value. This mechanism is specified by the following optimization objective:
min
E. [logar a ((ōKL + Sentropy) - (TT((-))) Te ( (+s)))) PNPD
(24)
where dK denotes the target KL value and dentropy denotes the target entropy. Note that log(-) is use to ensure that greater than O. 3 is adjusted the same way:
min E [lg B (H(T((-)s) Sentropy]].
(25)
3 s~
The overall training process is summarized in Appendix and code can be found on our GitHub https://github: wtt.auub..ooo/iggerraa////aa/a
EXPERIMENTS
The experiments aim to evaluate the effectiveness of the proposed strategy to lower the approxima- tion error and to verify that the proposed method can outperform other mainstream RL algorithms. The experiments for ablation study and comparative evaluation ing continuous control tasks from the OpenAI Gym(Brockman et al. 2016) environments, which
conducted
few challeng-
includes Mujoco[Todorov Todorov et al. (2022) and Pybullet (Coumans & Bai 2016 2019) versions. Imple- mentation details and hyperparameters of ECAC are presented in Appendix
4.1 ABLATION STUDY
Ablation studies are performed to verify the contribution the operation of KL limitation. We compared the performance of ECAC with the method removing KL limitation from ECAC.
Average return
Average retur
4000
3000
3000
8000
dlgo with limitaion
with KL Iniisiion
without KL Initaion
without KL limitaion
16
Million steos
Million steos
(a) Hopper-v3
(b) Walker2d-v3
Figure 2: Performance comparison of the method with KL limitation and the without KL lim- itation on the Hopper-v3 and Walker2d-v3 benchmark. The method with KL limitation performs better. Curves smoothed uniformly for visual clarity.
Figure compares five different instances for both methods with and without KL limition using dif- ferent random seeds; and each instance performs five evaluation episodes every 1, 000 enviromment steps. The solid curves corresponds to the mean and the shaded region to the minimum and maxi- mum returns over the five runs. The experimental result shows that our method with KL limitation performs better than one without KL limitation. Figure B demonstrates that by using ECAC the KL divergence remains comfortably low during all the training process.
Furthermore, to verify that confining the KI. divergence can decrease the approximation emror in Q function, measured the normalized approximation error in 100 random states every 10, 000 environment steps. The normalized approximation enror is specified by
1apprror Qtrwel
eQ
(26)
1Qturel
where Qupprax is the approximate Q value given by the current Q network, and Qtare is the true discounted return. The true value is estimated using the average discounted return over 100 episodes

---

Under review as a conference paper at ICLR 2021
KL Divergence
Divergence
wi KL Imitatio
with KL Imitaton
without RL limiation
18"
13"
127
P
Million steas
Million steps
(a) H3pper-v3
(b) Walker2d-v3
Figure 3: The KL divergence comparison of the method with KL limitation and the one without KL. limitation the H3pper-v3 and Walker2d-v3 benchmark.
with limiaioo
Emitaion
Error rate
50
22
00
04
68
6a
10
Million staos
Million stecs
(a) Hopper-v3
(b) Walker2d-v3
Figure 4: The approximation error comparison of the method with KL limitation and the one without KL limitation on the Hopper-v3 and Walker2d-v3 benchmark.
following the current policy, starting from states sampled from the replay buffer. Figure shows that the method with KL limitation has lower eтror in the Q function.
The results of all the ablation studies indicates that the approximation error of Q functioncan be decreased and the performance of the RL algorithm can be improved by placing restrictions on KL divergence between every two consecutive policies.
4.2 COMPARATIVE EVALUATION
Comparative evaluation are conducted to verify that our method can outperform other traditional RL methods including A2C, PPO, TD3, and SAC. Five individual runs of each algorithm with different random seeds done; and each run performs five evaluation episodes every 1, 000 environment steps. Our results are reported five random seeds (one random seed for one individual run) of the Gym simulator, the network initialization, and sampling actions from policy during the training.
The results of max average return over five runs on all the 10 tasks are presented in Table I ECAC outperforms all other algorithms the tasks except Hopper-v3 and HumanoidBullelBnvww only next to TD3 on Hopper-v3 and HumanoidBulletEnn--000 Figure 5 and Figure demonstrates learning curves of comparative evaluation on the 10 continuous control tasks (Mujoco and PyBullet version, respectively).
CONCLUSION
This paper presented a model-free actor-critic method based on finding that the approximation enror in value function of RL methods can decreased by placing restrictions on KL--divergence between every two consecutive policies. Our method increases the similarity between every two consecutive polices in the training process and therefore reduces the optimization difficulty value function. In the ablation studies, compare the approximation error in function, KL divergence, and performance of the methods with and without KL. limitation. The results of ablation study show that the proposed method can decrease the approximation error and improved the performance. Moreover, the results of comparative evaluation demonstrate that ECAC outperforms other model- free deep RL algorithm including A2C, PPO, TD3, and SAC.
8

---

Under review as a conference paper at ICLR 2021
5000
Averog
1500
3003
100
Million slips
Million slops
Millin shaps
(a) Hopper-v3
(b) Walker2d-v3
(c) HalfCheetah-v3
6000
3000
Averaags
2000
8:30
-100
00
02
08
(d) Ant-v3
(e) Humanoid-v3
Figure 5: The results of comparative evaluation on Mujoco version the OpenAl gym continuous control tasks. Curves smoothed uniformly for visual clarity.
2000
2000
Avdrago relun
Meergg
1000
-300
Million staps
Milloo ahaps
Million slops
(a) HopperBullctEnn--00
(b) Walker2DBulletEnvww
(c) HalfChoctaBBulleeEnwww
600
(d) AntBullctEne-v0
(c) HumanoidBalletEnw-00
Figure 6: The results of comparative evaluation on Pybullet version of the OpenAl gym continuous control tasks. Curves are smoothed uniformly for visual clarity.

---

Under review as a conference paper at ICLR 2021
Table 1:
Max average return five all the 10 tasks. Maximum value for each task is
bolded. = corresponds to standard deviation over runs.
Environment (Total steps)
ECAC
A2C
PPO
TD3
SAC
Hopper (10")
3395.4 69.5
2875.8 1 118.0
3317.6 1 150.3
3493.6 134.2
3175.2 1: 110.0
Walker2d (10%)
6146.2 = 243.8
2479.4
561.7
4811.1 + 295.7
3960.3 382.5
4435.1 945.4
HalfCheetah (10%)
11744.7 + 746
2653.5 + 975.9
1423
9492.6 852
0211.0+1061.3
Ant (10%)
5420.88 1069.6 6 1651.6 + 140.5
2502.1 + 442.8
3462.1 + 488.5
3832.2 = 927.7
Humanoid (5- 10")
8953.4
844.2
6969.2 403.2
937.1
HopperBulletLAnv (10%)
2642.5 39.3
1688.7 1. 68.9
2255.5 1: 342.9
2488.4 172.6
2397.7 1 56.9
Wallee2D (10%)
2323 161.8
1005,1 5.7 5.7
1035 + 201.2
2116.6 151.9
1651.0 = 408.9
HallCheetahBullellBav (10%)
2794.9 281.8
2027.7 87.8
1730 +
2012.8 + 182.2
2202. 213
AntBulletEnv--d (10%)
2997.6 6 220.9
2292.1 + 175.1
969. + 31.5
2953.9 - 81.3
2630.2 188
HumanoidBulletEnn (3 10%
1226.7 1 45.3
110.5 1 5.4
208.6 1 10.9
1471.1 113.9
1052.5 85.6
REFERENCES
Andre Barreto, Doina Precup, and Joelle Pineau. Reinforcement learning using kernel-based stochastic factorization. In Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 24, pp-
720-728. Curran Associates, Inc., 2011. URL http:://paper: httpppperr..isss.ccpppepeeee
4217 reinforcement wwwporrceceeeteeannnnnee using kernel based baaeeeeaaaaaaaaaaaaaaaaaaceeeeae factos fwwtoriattoaaaooo zation.
pdf
Andrew G Barto, Charles W Anderson, and Richard S Sutton. Synthesis of nonlinear control sur- faces by a layered associative search network. Biological Cybernetics, 4353):175-185, 1982.
Shai Dev-Da John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory of learning from different domains. Mach. Learn 79(1-2)151-- 175. 2010. doi: 10. (10007//10994000--552222 s 610994-009-5152-2
URL attps:
Nikhil Bhat, Vivek Farias, and Ciamac C Moallemi. Non-parametric approximate dynam- ic programming via the kernel method. In F. Pereira, C. J. C. Burges, L. Bottou, and
K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 25, pp. Curran Associates. Inc.. 2012. URL htttp://papers nips cctpaper/ 4547 non- carb-ppaamttiiccco approximated -ww.aid--nangaaanaiocao----
via the kernel meenneommhhhd
pdf
Christopher M Bishop. Pattern Recognition and Machine Learning (Information Science and Statis- tics). Springer- Veringe--Verlag New York, Inc., 2006.
Justin A. Boyan, Justin A. Boyan, Justin A. Boyan, and Justin A. Boyan. Modular neural network- for learning context-dependdnt game strategies. Technical report, Masters thesis, Computer Speech and Language Processing, 1992.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Erwin Coumans and Yunfei Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. http:: http::/pybullet..o org 2016-2019.
Thomas Degris, Martha White, and Richard S. Sutton. Off-policy actor-critic, 2013.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. Jennifer Dy and Andreas Krause (eds.). Proceedings of the 35th In- ternational Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1587-1596. Stockholmsmssan, Stockholm Sweden, 10-15 Jul 2018. PMLR. URL http://proceed// ngs ml wwwss/vv0000
18........ html
10

---

Under review as a conference paper at ICLR 2021
Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized markov decision processes, 2019.
Vijaykumar Gullapalli. Reinforcement Learning and Its Application to Control. PhD thesis, USA, 1992.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stuckholmsmüssan, Stockholm, Sweden, July 10-15 15, 2018, volume 80 Proceedings of Machine Learning Research, pp. 1856-1865, PMLR, 2018. URL http::tttcceeeiiiog/ mlr. www.spresssee.... /v80 https//////aaaroooeaa jalib html
Hado V. Hasselt. Double q-learning. In J. D. Lafferty, C. K. I. Williams, J. Shawe-Taylor R. S. Zemel, and A. Culotta (eds.), Advances in Neural Information Processing Systems 23,
pp. 2613-2621. Curran Associates, Inc., 2010. URL http: pwppraappaaapaaapppep nips tttccc/ppppp// cc/paper/
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv: Learning, 2014.
George Dimitri Konidaris, Sarah Osentoski, and Philip S. Thomas. Value function approximation in reinforcement learning using the fourier basis. Wolfram Burgard and Dan Roth (eds.), Proceed- ings of the Twenty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2011, San Francisco, California, USA, August 7-11, 2011. AAAI Press, 2011. URL http:/ httt:://ww... /wwww aaai. waai.org/ .rrg/
ocs/ ocs/index: index. php/ AAAI CALAI/AAA Aww.AAAAAaaaa........
wttp/viiwi//////e////eeeeeeeeeee 3569
Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URLhttps
7/arxiv. wwwp://orgaaaaaaaaaaee
02971
Long-Ji Lin. Reinforcement Learning for Robots Using Neural Networks. PhD thesis, USA, 1992.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, ab- s/1312.5602, 2013. URL htttt::://xxii.oo//bbb...
5602
Jan Peters and Stefan Schaal. Natural actor-critic. Neurocomputing, 71(7):1180 - 1190, 2008. ISSN 0925-2312. doi: http://dd..rrr........o//o.o.mmm sciencedirect com/ sc scienceeeeeeeeeeeeccccccce Modeling, Theory, and Application of Computational Intelligenc.
URL http://wwee //www. Progress in
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford. and Oleg Klimov. Proximal policy optimization algorithms. CoRR, Abs//707.006347 2017. URL sttp:///rxxv..rrg/DD.
Richard Sutton and Andrew G. Barto. Reinforcement Learning: An InntrrddctionnSeeeeee (Second Edition). Bradford Books, 2018 ISBN 0262039249. URL htllattt:///we. htt.t:/sse/eebbsttatfddddee/e//a
stwnfordd pdf
psych209/Readinge wttp:/ttttnottttoooaoooooooaaaaa Besaa2200/eaadnnnng/aattttaceaee
Gerald Tesauro. Practical issues in temporal difference learning. Mach. Learn., 8:257-277, 1992.
doi: 10. N0.10077/F000992697 URL Inttps:///o// htt.:/i..oorrgg//o 0tt///101100/00000 1tt.000000/0096699
Philip Thomas. Bias in natural actor-critic algorithms. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014, volume 32 of JMLR Workshop and Conference Proceedings, pp. 441-448. JMLR.org, 2014, URL httt:
//proceedings. mlr. wwwps/eeeee. httaasooossssl/ html
Sebastian Thrun and Anton Schwartz. Issues in using function approximation for reinforcement learning. In In Proceedings of the Fourth Connectionist Models Summer School. Erlbaum, 1993.

---

Under review as a conference paper at ICLR 2021
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, 2012.
Nino Vieillard, Tadashi Kozuno, Bruno Scherrer, Olivier Pietquin, Rmi Munos, and Matthieu Geist. Leverage the average: an analysis of regularization in rl, 2020.
Xin Xu. sparse kernel-based least-squares temporal difference algorithm for reinforcemen- t learning. In Licheng Jiao, Lipo Wang. Xinbo Gao, Jing Liu, and Feng Wu (eds.), Advances in Natural Computation, Second International Conference, ICNC 2006, Xi'an, China, Septem- ber 24-28, 2006. Proceedings, Part I, volume 4221 of Lecture Notes in Computer Science, pp. 47-56. Springer, 2006. doi: 10. 10.1100/11180 188007/1881070..8 11881070 8
URL https: /ttp////oiii//oooooooo/ //t/i..oo///.11000000000
12

---

Under review as a conference paper at ICLR 2021
APPENDICES
A
PSEUDO CODE OF ECAC
Algorithm Error controlled Actor-critic.
Require:
initial policy patameters, 0; Q function parameters, Φ1 and Φ2; discount rate, the coefficient of KL. term, 3; empty replay buffer D; the number of episodes, M; the maximum number of steps in each episode, T; minibatch size, N.
Ensure: optimal policy parameters 0*
1: for episode 1, do
2: Reset enviromment.
3: for = 1, do
4: Obeserve state 8 and select action a ~ r-1ss).
5:
Execute in the enviroment.
6:
Observe next state s' and reward r.
7:
Store transition (8, a, r, 8') in replay buffer D.
8:
Randomly sample a minibatch of N transitions, B = ((s,a,r,s') from D.
9:
Compute targets for the Q functions:
u(r, s') + + min Qo, (s', ā'), a' nT(-1s))
1-1
Update Q functions by one step of gradient descent using
V
Σ
(Qo, a)
v(r,,")) -(tr,s)'"(,s'")) 1,2.
(a.0.r.s")BB
11:
Backup old policy, Oold t 0. Update a and 3 by one step gradient ascent using
12:
V
log ((0KL + Ōentropy) (Tn
B
Σ log 3 (Te(ss)))) dentropu)]
where गKt. and Sentropy denote target KI. divergence and target entropy, respectively. Update policy by step of gradient ascent using
13:
1 V N
Σ (Q0. (8, (s) + (H((((()))) - (Do(()T(()))) Te
SEB
where (s) is a sample from T-(ss), which is differentiable with respect to via the reparameterization trick.
14:
end for
15: end for
13

---

Under review as a conference paper at ICLR 2021
Table 2: ECAC Hyperparameters
Parameter
Value
learning rate discount(?)
10
0.99
replay buffer size batch size
5-10
128
target KI.
5 5-10-3
target entropy
-dim((A) (e.g. -3 for Walker2d-v3)
target smoothing coefficient (7)
5-
number of hidden layers (all networks)
number of hidden units per layer
256
nonlinearity
ReLU
Table 3: Reward Scale Parameter
Environment
Reward Scale
Hopper-v3
Walker2d-v3
HalfCheetah-v3
Ant-v3
Humanoid-v3
20
HopperBulletEnn-v00
Walker2DBulletEnn--00
HallCheetahBuulleeEn--y
AntBullceEnv--O
HumanoidBulletEnwwww
20
B IMPLEMENTATION DETAILS AND HYPERPARAMETERS OF ECAC
For the implementation of ECAC, two layer feedforward neural network of 256 hidden units, with rectified linear units (ReLU) between each layer are used build the two functions an the policy. The parameters of the neural networks and the two coefficients (i.e a and (3) are optimized by using Adam(Kingma & Ba 2014). The hyperparameters of ECAC are listed in Table [2] Moreover, we adopt the target network technique in ECAC, which is common in previous works (Lillicrap et al.) 2016 Fujimoto al. 2018). We also adopt reward scale trick which is presented Haarnoja et (2018; and the reward scale parameter is listed in Table 3
14


Reviews:
[{"review_id": "wnZ1m814UZO", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n=========\nAuthors investigated the effect of approximation error for actor-critic. They derived an upper bound of approximation showing that minimizing the KL divergence between the two consecutive policies can drive this upper bound down. Based on their finding they  introduced the Error Controlled Actor-critic (ECAC) algorithm. They ran ablation study showing the positive impact of minimizing the KL divergence. Furthermore they compared ECAC against 4 state-of-the-art techniques showing their advantage across 4 out of 5 Mujoco domains.\n\n[+] Aside from minor grammar issues (see details) the paper is easy to follow\n\n[+] The trick to calculate alpha and beta automatically is very interesting\n\n[+] Empirical results are encouraging \n\n[-] The experimental result section can enjoy more rigor. It is not clear how many seeds were used to generate these results. As Joelle discussed in one of NeurIPS conferences, it is easy use limited number of seeds and deduce very different outcomes (https://media.neurips.cc/Conferences/NIPS2018/Slides/jpineau-NeurIPS-dec18-fb.pdf). Also why showing min and max of returns instead of something more common such as std-err.\n\n[-] I think Actor-critic has been shown to converge before. Why the proof using Banach’s Fixed Point Theorem important compared to other proofs?\n\nDetails\n=========\nP2, overestimation. => Need space after .\nSection 2.2: hypothesis. And the => Remove \"And\"\nSection 4.2: \n\t- that can be views adapting => that can be viewed as adapting\n\t- I recommend providing the difference between Q-networks and its target at n+1 iteration in a source distribution before the corresponding value on the target distribution (e.g. replace Eqn 11 and 12 and the accompanying text)\n\t- While the difference between D^n and D^n+1 is small, the corresponding error introduced can be big\n\t- I believe the use of minimized KL divergence between policies has been practiced before in the RL literature.\nSection 4.3: ).In order => ). In order\nPage 7: \n\t- Curves are smoothed uniformly for visual clarity\" => What do you mean?\n\t- using different random seeds => How many random seeds? Why not mention here? I could not find it in the Appendix either\n\t- shaded region to the minimum and maximum returns over the five runs => Why not showing standard error instead to have a better statistical significance understanding visually?\n\t- Use vectorized image formats so your images are zoomable without the loss of quality\nEqn 23: Q_{ture} => Q_{true}\nPage 8:\n\t- \"with KL limitation have\" => \"with KL limitation has\"\nSection 5.2:\n\t- \"ECAC outperforms all other algorithms on the tasks except\" => What does beating mean? Have you ran statistical analysis or are you relying on the mean values?", "extra_scores": {}, "extra_text": {}}, {"review_id": "Tq8A76BtXk", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors study the error introduced by the estimation of critic function in the Actor-Critic algorithm. Then the author proposed an algorithm that utilizes the idea of double Q learning and using a KL-divergence like regularization method to control this error. Experimentally the proposed algorithm achieves good results comparing to the vanilla Actor-Critic algorithm. This paper shows a successful routine from the theoretical analysis to a practical algorithm.\n\nHowever, one of the drawbacks of this paper to me is, the authors proposed the double Q learning method. It is not clear how this double Q help to reduce the estimation error? I am also a little bit concerned that the gap between theoretical analysis and the practical algorithm is large. In practice, the authors always use a one-step update, but there is no guarantee that the one-step update for both actor and critic can decrease the error or improve the performance. \n\nAlso, since the key idea of how to control the critic is to use a regularization based on the last policy to make sure the policy move not so 'fast'. It would be interesting to ask what will happen if we only use a smaller learning rate for the actor and a larger learning rate for the critic. Some theoretical work [1] [2] have already adopted this method to get the error controlled and make the actor converge to the stationary point. Therefore, it would highlight the result of this paper if the author can compare their algorithm and show the advantage over these algorithms that only set different learning rates. \n\nGiven the contribution of the newly proposed algorithm, I will suggest accepting this paper.\n\n[1] Wu, Yue, et al. \"A Finite Time Analysis of Two Time-Scale Actor Critic Methods.\" arXiv preprint arXiv:2005.01350 (2020).\n\n[2] Hong, Mingyi, et al. \"A two-timescale framework for bilevel optimization: Complexity analysis and application to actor-critic.\" arXiv preprint arXiv:2007.05170 (2020).\n\n--- Update after discussion\n\nI agree with other reviewers' idea that the contribution of this paper is somehow limited. Since my previous suggestion (7, accept) is based on that the author can successfully address how to control the error of the critic theoretically. However, with only the experimental elaboration study, as I said in the response to the author, the contribution of this paper is limited. Thus, I would switch back to 5 (marginally reject) based on that after reading other reviewers' discussion.", "extra_scores": {}, "extra_text": {}}, {"review_id": "HU150XC7ywq", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Before beginning with the review, I would strongly request the authors to correct grammar errors, spelling errors, and notation inconsistencies which seriously hinder understanding in some cases.\n\nThis paper explores error accumulation in actor-critic methods. The authors claim to present an analysis of approximation error in actor-critic methods (though I couldn't find any concrete analysis of this fact) and then suggest that bounding the KL-divergence of the policy against the previous policy aids actor-critic algorithms by deriving an upper bound on the change in TD error due to faster changes in the policy.\n\nMy overall opinion is that this paper restates results that are already well known and the general theme of the method ECAC is already explored in RL literature (see KL-control literature). So, I don't really think there is any novel concept presented in the paper. I move to specific points next:\n\n- Theorem 1: I don't see what's novel here. Such results have been shown in the absence of function approximation for off-policy actor-critic (Degris et al. 2012). What's new here?\n\n- Analysis of approximation error: In the abstract, it is claimed that the paper performs an *analysis* of this error. I could not find this anywhere in this paper. If this refers to Section 3.2 (Hidden dangers in AC methods), I would not call that \"analysis\". It states some known facts without actually discussing anything formally there. So, either this claim should be removed or a valid analysis needs to be presented.\n\n- The reason for why KL-constraints work: I can see what the derivation does and how the authors arrive at the conclusion of KL-constraint I do not buy the argument that KL-constraints help learning because they control TD error -- they do stabilize the critic, as would a smaller learning rate for the policy would and lead to lower accumulation of error (for e.g., see A theory of Regularized MDPs, Geist et al. or Leverage The Average: An Analysis of KL regularization in RL) and these analyses are far more sophisticated that the argument presented in this paper. \n\n- The empirical results in the paper are OK, but the baselines utilized are a bit suboptimal, for e.g. SAC and TD3 in HalfCheetah -- having used these myself, it seems like official codebases of these papers do get better performance than what is reported.  I think the numbers are this unreliable.\n\nI think overall the algorithm with the KL-constraint is a good idea -- as has been explored several times in literature -- but I don't think this paper cites that work, nor compares to them along with not a rigorous enough evaluation and poor writing quality.", "extra_scores": {}, "extra_text": {}}, {"review_id": "CopCioX0eI", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nClarity:\n---------\nMajor:\n******\nIt is not clear what the actor critic algorithm is. What is presented in Fig 1 seems to be policy iteration, and Theorem 1 seems to be stating policy iteration (yet $\\pi^{n+1}$ has not been defined). \n\nMinor:\n*****\n1) What is $V^{\\pi^n}$? Is it the value function of the policy $\\pi^n$.\n2) What is the difference between $\\pi^{*,n}$ and $\\pi^{n+1}$.\n3) What is $\\pi$ is eq(3)?\n\n\nQuality:\n-----------\n1) The argument following eq (15) is very informal: \"the number of samples in $\\mathcal{D}^{n+1}$ is only a little more than in $\\mathcal{D}^n$\".\n2) For the contraction property of the Bellman operator, the author can consider citing standard text books instead of \"https://towardsdatascience.com/ mathematical-analysis-of-reinforcement-learning-bellman-equation-ac9f0954e19f\"\n\n\nSignificance:\n------------------\n1) The  authors claim \"We present a convergence analysis for actor-critic methods by Banach’s Fixed Point Theorem\" to be one of the contributions. The fixed point theorem here is to show that Bellman operator is a contraction operator (a well known fact).\n2) The experiments are on only 5 domains. \n\n\nOverall Feedback: Adding the KL based penalty is perhaps a useful idea (this may also not be an entirely new one). It will be great if a) the algorithm can clearly stated,  b) the novelty of the method pointed out clearly by comparing it with related works c) the ideas can be supported by formal theoretical statements.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The majority of the reviewers believe that this paper is not ready for publication. Among their concerns is that the paper has limited novelty, especially in relation to existing work that use the KL constraint. Some of the reviewers also believe that the arguments are sometimes hand-wavy and not rigorous. For example, in the discussion period after Nov. 24th, it is mentioned that the argument by the authors that "KL constraint==>decrease the approximation error==>increase performance" is not precise enough. I encourage the authors to take these comments into account and improve their paper.
Decision: Reject