Model-based Value Exploration in Actor-critic Deep Reinforcement Learning
Authors: Huihui Zhang
Venue: Submitted to ICLR 2023 (2023)

Abstract: Off-policy method has demonstrated great potential on model-free deep reinforcement learning due to the sample-efficient advantage. However, it suffers extra instability due to some mismatched distributions from observations. Model-free on-policy counterparts usually have poor sample efficiency. Model-based algorithms, in contrast, are highly dependent on the goodness of expert demonstrations or learned dynamics.
In this work, we propose a method which involves training the dynamics to accelerate and gradually stabilize learning without adding sample-complexity. The dynamics model prediction can provide effective target value exploration, which is essentially different from the methods on-policy exploration, by adding valid diversity of transitions.
Despite the existence of model bias, the model-based prediction can avoid the overestimation and distribution mismatch errors in off-policy learning, as the learned dynamics model is asymptotically accurate.
Besides, to generalize the solution to large-scale reinforcement learning problems, we use global gaussian and deterministic function approximation to model the transition probability and reward function, respectively. To minimize the negative impact of potential model bias brought by the estimated dynamics, we adopt one-step global prediction for the model-based part of target value. By analyses and proofs, we show how the model-based prediction provides value exploration and asymptotical performance to the overall network. It can also be concluded that the convergence of proposed algorithm only depends on the accuracy of learnt dynamics model.

Under review as a conference paper at ICLR 2023
MODEL-BASED VALUE EXPLORATION IN ACTOR- CRITIC DEEP REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Off-policy method has demonstrated great potential on model-free deep reinforce- ment learning due to the sample-efficient advantage. However, it suffers extra in- stability due to some mismatched distributions from observations. Model-free on- policy counterparts usually have poor sample efficiency. Model-based algorithms,, in contrast, are highly dependent on the goodness of expert demonstrations or learned dynamics. In this work, we propose method which involves training the dynamics to accelerate and gradually stabilize learning without adding sample- complexity. The dynamics model prediction can provide effective target value ploration, which is essentially different from the methods on-policy exploration, by adding valid diversity of transitions. Despite the existence of model bias, the model-based prediction can avoid the overestimation and distribution mismatch errors in off-policy learning, the learned dynamics model is asymptotically ac- curate. Besides, to generalize the solution to large-scale reinforcement learning problems, we use global gaussian and deterministic function approximation model the transition probability and reward function, respectively. To minimize the negative impact of potential model bias brought by the estimated dynamic- s, we adopt one-step global prediction for the model-based part of target value. By analyses and proofs, we show how the model-based prediction provides val- exploration and asymptotical performance to the overall network. It can also be concluded that the convergence proposed algorithm only depends on the accuracy of learnt dynamics model.
INTRODUCTION
Model-free reinforcement learning (RL) algorithms have been applied to wide range of tasks, ranging from simple games (Mnih al. 2013) | Oh al., (016) to robotic locomotion skills (Schul- man et al. 2015). To tackle the large-scale continuous control problems, the function approximators implement some neural networks to represent the high-dimensional state and action spaces in deep reinforcement learning (DRL). However, model-free DRL is notoriously expensive in terms of its sample efficiency, which is deadly difficult to be employed in reality where samples are valuable to achieve. Among the recent model-free DRL algorithms, on-policy methods (Schulman al. 2015] 2017 Fujimoto et al.) 2018) typically require multiple samples to be collected for each rollout at every gradient step, which is quite extravagant in consuming samples because multiplied data re- quirement does not necessarily bring corresponding performance gain. In comparison, off-policy methods aim to reuse the past experience by storing the collected observations in a memory buffer, typically, combining Q-learning with neural networks (Mnih et al.) 2015). Unfortunately, the com- bination of off policyy learning and nigh-diimnsioonll, nonlinear function approximation are exposed to issues in terms of instability and divergence (Maei et al. 2009).
The causes for the emergent problems are very complicated, for example, some works (Fujimoto et al. 2018; 2019 Duan et al. (221)) blame them on the overestimation bias, which says the con- tinually maximized value during the actor-critic optimization will accumulate the overestimation emrors and break the training stability. Some others try to find extrapolation error induced by the mismatch between the distribution of sampled data from experience and true state-action visitation of the current policy (Fujimoto et al. 2019). There have been several ways tackle the distribution mismatch. The authors in (Wu et al. 2019) address the distribution errors by extra value penalty policy regularization, (Wang & Ross 2019) changes the rule of experience replay reduce the

---

Under review as a conference paper at ICLR 2023
distribution mismatch by sampling more aggressively from recent experience while ordering the updates to ensure that updates from old data do not overwrite updates from new data, and (Martin et al. (02T) relabels successful episodes as expert demonstrations for the agent to match. Despite their efforts, the overestimation bias and mismatched distribution from past experience can only be mitigated, and sometimes may induce new problems.
The paper has the following contributions. First, instead of using immediate rewards or assuming known reward function, we adopt neural networks to approximate the reward function as part of dy- namics. Meanwhile, we train the parameters of modeled transition probability and reward function based on the replay buffer from off-policy observations. Second, the prediction from the learned dynamics will be used to foresee the target value according to certain percentage, Since the jynamies-prediction is essentially different from the observations from environment, it can pro- vide extra exploration which is not conditioned on the state-action visitation history. Besides, well trained dynamics model is free of overestimation and distribution mismatch errors, and can provide more accurate target value and stabilize the asymptotic performance. Third, the related algorith- m is proposed and the final results prove good efficiency and stability of the proposed algorithm. Fourth, the accuracy of learned model is tested by setting maximum online time step, which is the beginning of off-line planning that is isolated from the environment.
2 RELATED WORK
Due to the various problems arising from the sample complexity of model-free algorithms, task- Deisenroth et al., 2013) as well as the model-based Levine et al. [2016] 2018 Kaiser et al.) 2019) using planning, which optimize the policy under a learned or given dynamics model, are more preferable in real physical systems, such as robots and autonomous vehicles. However, task-specific repre- sentations have limited range of learnable tasks and greater requirement for domain knowledge. Model-based DRL algorithms are considered being more efficient (Deisenroth et al.) (2013), because it constructs a dynamic probabilistic model via lots of data and avoids interaction with the environ-
specific representations (Peters et al. 2010
algorithms (Deisenroth & Rasmussen 2011
ment by training the strategy based on the learned dynamics model (Hua et al. 202I), but it limits the policy to only be as good as the learned model (Gu et al. 2016).
For the model-free part, the agent needs to interact with the environment to collect enough knowl- edge for training, which poses the importance of the tradeoff between exploration and exploitation (Mnih et al. 2016). Soft actor-critic (SAC) (Haarnoja et al. 2018a] b) achieves good performance on set of continuous control tasks by adopting stochastic function approximation and maximum en- tropy for policy exploration. Among these techniques, stochastic policies have the advantage of allowing on-policy exploration and off-policy experience replay over deterministic counterpart- s (Heess et al. 2015), and the maximum entropy exploration improves robustness and stability (Ziebart et al. 2008 Ziebart 2010).. Overall, the existing exploration strategies are limited to the policy, which raises the concern about whether and how the value exploration can play a positive role in model-free learning.
While some works combine both model-free and model-based DRL in the literature (Sutton 1990; Lampe& & Riedmiller] 20144), the following works particularly relevant to our work in this pa- per. Specifically, (Gu et al. 2016 Nagabandi et al. 2018) add synthetic imagination rollouts to an additional replay buffer for model-guided exploration in some off-policy methods at the price of much higher storage and computation costs. Besides, model ensembles are adopted in (Chua et al. 2018 Kurutach et al., 2018 Janner et al. 2019) reduce misguided policy or inaccurate planning caused by model bias. Moreover, value expansion of fixed multi-step prediction by dynamics model adopted in (Feinberg et al. 2018 [Buckman et al. 2018) to make proper value expansion and con- trol imagination depth. However, making multi-step prediction from global dynamics model may suffer cumulative model estimation errors and is usually replaced by iteratively refitted time-varying linear models (Levine& & Abbeel 2014). VIME (Houthooft et al. (2016) introduces maximization of information gain about the dynamics' certainty, which is overwhelmed by theoretical analyses and lack bit intuitive judgement. In this paper, we adopt one-step prediction, which avoids the costs of storage and computation from multi-step synthetic rollouts, from global dynamics model used for value exploration to achieve diversity, accuracy and generality.
2

---

Under review as a conference paper at ICLR 2023
3 PRELIMINARIES
We consider the extended Markov Decision Process (MDP) continuous state and action spaces, denoted by the tuple (S, A, P.r) where 8 is the state space, A is the action space, S is the space of next state, P(s'|s, (s, a. is the transition denoting the conditional probability of the next state s' E S given the current state connected with (s, a. s') from the environment. The purpose of reinforcement learning (RL) is to optimize the policy by maximizing the reward return, which is denoted by the expected discounted cumulative reward of a rollout. DRI. employs the function approximation to parameterize the reward return so that the optimization can work under the setting of continuous control. In DRI., an action is sampled from the policy и and "judged" according to a value estimate determined by the observa- tion 8, then the next state s' and the future reward r can be determined by the transition probability combined to define the dynamics
S and action E A, and r E 8 x A is the reward function which is
P(-|s, a) and the reward function r(s, a), respectively, which in this work. In recent researches of DRL, the action-value (Q-value) function with respect to the state-action pair is usually chosen as the surrogate of the reward return, in the form of
an( Σ EepPPt ayna [7'r(8,, as) 8, 0g = =a
(1)
where 7 E (0, 1) is the discount horizon factor for future rewards, x is the policy for action section every time step, and pt is the distribution of 8t, which is joint distribution of transitions. If conditioned on initial state-action pair (so, on). it is given by
[-1
Pt (s an) = P(s1 ao)
II -=
E. as P(si+1|8s,00 a.).
(2)
Then we have
P'(ss)
| (( (8t-1)]
(3)
Lemma 1. Assume the expected KL-divergence between two transition distributions is bounded by
max Bu Pa Drv (p(st+1|st az) P az)) S 6,
(4)
t
then we have Egr (st) P's)) 2t5.
Proof See Appendix A (submitted in the supplementary material).
The Q-value function in Eq. T is mapping from the input observation-action pair (s.a) to the
Q-value, and it has the property of satisfying Bellman equation, so the temporal difference (TD) (Tesauro 1995) is generally used to minimize Bellman errors by the transition tuple (s,a, s') at every critic evaluation step, which is given by ((s.a.r,e")) [(r + (s'(s', <Ts')) - Q(s, Lilli- crap et al. 2015), where Q' stands for target Q network. In algorithms using the experience replay, a, s') will be stored in replay buffer at every environment step, a is sampled from the ex- perience pool, and the next action has be judged by the current policy, represented as (s'). In off-policy methods, the distribution of sampled action a is different from the current policy, which will cause distribution mismatch. In the context, we use the term of 'iteration' to represent the index of updates. In the actor-critic paradigm, each iteration contains the evaluation step and the policy improvement step, which are used to update Q-value function and optimize the policy, respectively. After minimizing the Bellman errors, the policy improvement is performed by maximizing the ex- pected return J(0) = E, IQ (s, (s)). In some algorithms, the policy regularization may be attached to the expected return for the stability of training (Kumar et al. 2019 Jaques et al. 2219).. which is aimed to restrain the policy gradient
(o) to keep away from potential gradient vanishing or exploding problems well as reducing the estimation variance.
MODEL-BASED ACTOR-CRITIC VALUE EXPLORATION WITH ASYMPTOTIC PLANNING
The actor-critic target value exploration with asymptotic planning is a method that blends the one- step global model-based prediction into target critic value, whose importance lies in how and why it
3

---

Under review as a conference paper at ICLR 2023
work well. The diversity of state distributions between model-based prediction and observations produces extra value exploration, and asymptotically accurate learned model has the potential to overcome emrrrs from the overestimation and mismatched policy distributions, when experience play is applied in off-policy algorithms. The asymptotic planning is realized by gradually increasing the weight of model-basen prediction in the target critic value.
4.1 MODEL-BASED TARGET VALUE EXPLORATION
By instinct, the model-based target value is able to explore the future states from different view- points with some certainty. Without the model-based prediction, the target value usually chooses the input next states from random samples in replay buffer. The distributions of sampled next states which meet 5 ((++(++++) 1) ( are not really stationary in off-policy methods, since the policy will go through multiple updates in a rollout. These unstationary distributions can distort the real stationary transition probability from the view of the current policy, which means false dynamics could be experienced or "felt" with the real observations from the replay buffer, which are achieved by interacting with the environment. The problem of distribution mismatch will be incurred when off-policy method is used, since actions (a,,... ag) are random samples from the replay buffer following different distributions. The analyses tell that observations from the real environment are not necessarily more accurate than the prophecy of dynamics model.
Based on the choice of target value, the evaluation step for the critic updates can be separated into the off- off-policy training and the on-policy planning, We introduce the backup operator for model-based Q-value prediction as
PTQ(s,, az) (r((Ss 01, s642) + yBaa4 (0(5++1:0++1)
(5)
where 141 p(-|se, oz), ag ~ ((((s)), a++1 T(([s++)) and ry and p are reward functions and transition probability learnt from the dynamics model, respectively. We use s441 to distinguish the subtle difference between the model-based prediction and St+1 ~ (s++() which follows a distribution determined by the complete rollout. And 4+41 ~ (-15<+11 because the limitation on one-step global prediction, otherwise, choosing ar+1 ~ x+-15"+1) will induce cumulative model bias. If replacing Ta and t,+1 with the immediate reward and the next state s' from past experience. (19) reduces to the target Q-value of deep deterministic policy gradient (DDPG) (Liliicrap et al.) 2013) Plus regularization term concerning the entropy policy exploration, it then becomes the target Q-value of SAC.
The change of target value seems subtle, however, the model- prediction produces valid state and reward diversity for value exploration. Unlike the action which is usually bounded, the state s- pace is continuously unbounded in many tasks like Gym environments, then the exploration strategy commonly used in policy exploration, for example the gaussian exploration noise, will be invalid in value exploration. Throughout the referred literature, few works have attempted to apply sim- ple and/or feasible value exploration. More importantly, as the trained reward model and transition model grow more accurate, the on-policy prediction (19) will greatly reduce or avoid originally ex- isting overestimation errors and distribution mismatch in target Q-value without value exploration. Although model bias is induced by model- planning during learning, it can be controlled by asymptotically increased impact of model-based prediction.
Lemma 2. Consider the sequence Q++1 PM Q, constructed by 19). then given the condition that the Q-values are bounded, i.e., (Q.(s, a)] < oo, V(s,a) E x A, the sequence Q₁ will converge to unique optimal value as l. + 00.
The proof of Lemma 5 can be found in Appendix B In this work, 19) will be combined with the target Q-value of SAC according to an asymptotically increasing percentage. To expand the value exploration, the current Q-value function to be predicted by the target Q-value is also divided into two parts sharing the same critic network parameter. However, they take actions following different distributions inputs. By this means, the diversity can be enlarged and some convergence conditions can be satisfied, which will be shown later in Theorem 4
In the policy improvement step, we directly adopt the entropy policy exploration of SAC with the actor network, which does not involve computation on the dynamics. When the model-free DRL is applied to large-scale continuous control problems, the dynamics is unknown and the state-action
4

---

Under review as a conference paper at ICLR 2023
spaces continuous, the policy improvement over S X A at every iteration, which is called the absolute policy improvement, cannot be guaranteed by estimation with distribution mismatch and estimation biases. Some researches continuous control also show empirical results which degrade after reaching a good point. As told in related work part, there have been several methods trying to apply the model-based synthetic rollouts the policy improvement step, however, they do not work well with the proposed value exploration strategy according to our practice since the one-step global prediction will be violated.
4.2 DYNAMICS LEARNING
Compared with descriptive models that are feasible in small state spaces (Deisenroth & Rasmussen, Khansari-Zadeh & Billard, 201T), neural network approximation can scale better to high- dimensional state spaces. We parameterize the dynamics px (x a) and ra,s,, a, 8') with deep gener- ative models (Moerland et al. 2020). where the parameters and μ reparameterize the transition and the reward functions, respectively. Considering the fact that the transition function is difficult to train, we choose to learn a relative transition function to forecast the difference between the current state and next state, which is given by s, ++ px px(s, a). and the state difference follows the relative transition probability density function (pdf) px(-ss a). The use of in does not induce the distribu- tion mismatch since the relative transition function is unaffected by the policy. The representation of unknown reward function differs in the inputs among various tasks, for example, the informa- tion determining the rewards is not included in observations for the default setting of MuJoCo suite Todorov al.) 2012 Brockman al. 2016). This complicates the training of the reward function, but we will show that taking inputs as (s, s') is applicable in our selected benchmarks.
2011
Lemma 3. Assume the absolute value of expected reward function and the expected KL-divergence between the dynamics model and the real transition probability are respectively bounded by
[Eawr (s) S Tm. a~
max Eu pr Drv (ol
S 6,
(6)
then we have
E asrar (Q(s) BerP [Q(s,, az)] S 088).
The proof of Lemma 6 can be found in Appendix C which tells the distance between predicted
Q-value and true Q-value is bounded linearly by 8. The dynamics Model is trained accompanying the iteration, using random samples from experience buffer. Given the four-tuple sample (s, a, r, s'), the surrogate objective of relative transition function is given by
D(A) = (.,.,
(s (
-))
(8)
where R represents the replay buffer that random samples come from.
Then (8) can be optimized with stochastic gradient
V D(A) = (((,,,,')) [(s + px(s, 8')&xPx(s,0))
(9)
Similarly, the surrogate objective for updating the reward function be formulated
D(4)= = E(,,a, 2/ (ruls, (, - -n70
(10)
and it gradient is computed as
D(K) ((s.,rss)) [(r,(s, a,)) r)
(s.
(11)
well trained reward function is necessary for (9)). Without the real-time model reward function, sampled rewards from the replay buffer will induce distribution errors in model-based prediction.
5

---

Under review as a conference paper at ICLR 2023
4.3 MODEL-BASED ACTOR-CRITIC VALUE EXPLORATION ALGORITHM
As mentioned above, we adopt the minimized pairwise critics to serve as the target Q-value for the purosee of mitigating the effect of overestimation (Watkins 1989). Besides, current and target networks are separated to execute soft updates (Lillicrap et al. 2015 Haarnoja et al. (2018) for all surrogate objectives, for the good of stability. In this work, the asymptotical model-based prediction based on (T9) is merged into the target value of SAC, so the loss function for the update of critic parameters in the evaluation step can be estimated by
L(w') (ssas)s)) [(KQ' a') + (1 k)) (s') (s, -1 (1- (() (12)
where i E {1,2). a (s) and a' = - (s') are the on-policy actions chosen from the current
policy and the target policy parameterized by and 0', respectively, (s, a, r, is a tuple of history data sampled from the experience pool, and k is the asymptotic variable increasing from 0 to 1 as the time step proceeds. And
,')) - - Log(To (a'1s))) Q'(s',a')= = + min (Qwn (wa
(13)
a'() = min where w², ",w,,'' w and w² parameterize two critic networks and their target estimates, respectively. Besides, sy, = pssss a) is the on-policy next state predicted by the transition model parameterized by X, and opr (-Is') is the target policy distribution conditioned on the next state s'. The action input of reward function ry follows the current policy instead of being sampled from the replay buffer. By minimizing (24. the critic parameters can be updated for each evaluation step.
min (Qw" (d a'), Qw (sp: d')) + rr(s, s/),
(14)
Theorem 1. Assume E ((s..,)))) [(Q' (s', d') - Qui a)) S for > Tr then ET > 0 so that I(w") S 26 for t T T.
The proof of Theorem 3 can be found in Appendix [ Theorem 3 means that the critic loss function 24 of this work will converge under the assumption that the critic loss of SAC converges. Besides the conclusion of convergence, we are more curious about how it converges or by what factors both the accuracy of transition model and reward function
is affected. From Theorem 4 model will affect the target value prediction, and the transition model influences more. Moreover, a well-trained model can guarantee its convergence.
Theorem 2. Assume the absolute value of expected reward function, the expected KL-divergence between the dynamics model and the real transition probability, and the MSE of expected difference between the modeled reward function and the immediate reward are respectively bounded by
max Peawr(ss, a)) Ti
a~S
max Eu Pr Drv (p(s az)) 0t)) S 6,
max E(e.)) { ( ams awvz [ra(s, ae (44) - +]} 2
S E,
(15)
then we have the MSE of target prediction error bounded by 2 + 0
Proof See Appendix This theorem interprets that the target prediction error does not suffer from overestimation bias and mismatched distribution, and can be negligible once the learnt model is accurate, then the prediction error distance is bounded by / O(62).
Generally, the policy improvement step aims to maximize the current Q-value or it variant, which does not need the dynamics model to predict the reward the next state. Then the surrogate objec- tive function used to update the actor parameters can be directly given by
J(Ø)= = E, (Q. (8. a) a log(To(a|s))),
(16)
where Qu a) = min (Qwa (s, a). Qwa a)), $ comes from the tuple of history data, a = no (s) is the reparameterized action based is the current policy distribution conditioned on the current state s. By maximizing (16). the actor parameter can be updated for each policy improvement step.
and the policy network parameterized by 0, and a (-ls)

---

Under review as a conference paper at ICLR 2023
(a)
(b)
(c)
(d)
(c)
Figure (a) Ant-v3; (b) Halfcheetah-v35 v3; (c) Hopper-v3; (d) Walker2d-v3; (e) Humanoid-v3
For updates of target critic parameters, we adopt "soft" target updates (Lillicrap et al.) 2015) using a weighted factor < r < to control the speed of policy updates for the sake of small value enror at each iteration. Except the critic parameters, ters in this work. In 99) and (TT), the gradients in expectation forms are approximated by averaging over the sampled results of rollouts from past experience. We organize the above procedures as the model-based actor-critic value exploration with asymptotic planning (MAVE) algorithm, whose pseudocode is described by Algorithm The algorithm alternates between running the environ- ment steps to collect experience, and updating the network parameters using the stochastic gradients computed by the sampled batches from the experience pool. It is composed off-line planning, separated by the maximum online time step Ti. At the stage of online training, Step 10 Algorithm| [ requires tiny computation using the dynamics model without the burden of great computing and storage processing from virtual or synthetic rollouts, as explained by (4)).
adopt immediate updates for other parame-
online training and
Algorithm MAVE Algorithm
1: Initialize parameters w t- who w² +- wp. A t- og. X + Ao. <- Ho
2: Initialize target parameters w/l <- w w² + wg H' + 6
3: Initialize the learning rates le, las La for the critic, the actor and the dynamics model, the time step < O, the asymptotic variable & <- 0, the soft update hyperparameter т, the maximum online time step Ti, the maximum overall time step T, the batch size B and the replay buffer
+++0
4: while t < Ti do
5: Select action a₃ ~ No, (ag|s;)
6: Observe the reward and next state from the interaction feedback
7: Store transition R + R U {(s,, a2 T2, $c4
8: Sample batch of transitions B = (8, r,)),, from R
9:
for each time step do
10:
wH+1 f wt - 1.
we: (ww)) for i E (1,2) following Vii Lww))
0.t1 < + 1, ((0,) following g. ((O))
12:
det1 + d 2 1 (()) following Eq. 9
13:
Pt+1 + uh 1 La ( following Eq. (II)
w" + Tw]+1 + (1 T)w", for E (1,2]
15:
end for
8141 f- 84:
17:
t+t+1
18: end while
19: & 1+1
20: while < do
21: Repeat Steps
10
12
13
14
and 17
22: end while
EXPERIMENTS
5.1 BENCHMARKS
The performance of our proposed method is compared with several prior model-free and model- based reinforcement learning algorithms in the sample complexity and stability set of gym
7

---

Under review as a conference paper at ICLR 2023
12000
8000
8000
3000
FRAC
AROn
MEFO
MFO
MAVEL
MAVE
2000
9000
0
0.0
0.2
0.4
06
0.0
1.0
12
1.4
0.0
0.5
1.0
1.5
2.0
2.5
300
Timm stops
166
Time steps
1u6
(a)
(b)
4000
8500
5000
Aggo
Ago
2000
BAC
BAC
BRANT
IRAC
MEF
2000
MEO
1000
500
0.4
0.4
Time
126
Time
106
(c)
(d)
3000
Axtrgg roward
4000
Algo
TO3
BAC
2000
MAWE_P
1000
(c)
Figure 2: Average reward of off-line training and auline-planning versus time step in (a) Ant-v3; (b) Halfcheetah-vš. (c) Hopper-v3; (d) Walker2d-v3; (e) Humanoid-v3
continuous control tasks, several of which are selected in this paper as benchmarks illustrated in Fig.1
BASELINES
The baselines adopted for reference includes TD3, SAC, BRAC and Model-based policy optimiza- tion (MBPO) (Janner et al., 2019). Before the appearance of SAC, DDPG is regarded as one of the most efficient off-policy DRL methods (Duan al. 2016), followed by TD3 extension. SAC
8

---

Under review as a conference paper at ICLR 2023
has achieved state-of-the-an sample efficiency in multiple challenging continuous control domains [Christodoulou 2019), and BRAC can be regarded as a variant of SAC by adopting an extra policy regularization based on the KL divergence between updated and older policy. In this work, adopt (8) to train the transition probability instead of maximum likelihood since the logarithm of transition probability parameterized by the global gaussian network tends to be unbounded.
We apply the shared hyperparameters to our proposed algorithm with other baselines for every benchmark to keep faimess.. In the process of collecting the off-policy rollouts, the gaussian ex- ploration noise is added to every time step with fixed variance 0.2 when choosing the action, and then the noisy action clipped within the set boundary avoid uutoo--iittriuutin (OOD) actions (Kumar et al. [219 (020). The discount horizon factor is selected as 0.99, and all algorithms adopt stochastic policies and maximum prior action entropy except for TD3. The stochastic policies follow gaussian distributions with mean and variance parameterized by fully connected networks with two hidden layers, each of which has 256 units. TD3 uses deterministic policy, also parameterized by fully connected networks with two hidden layers. We organize the network architectures and hyper- respectively. The Adam optimizer (Kingma Ba 2014) is used
parameters in Appendix F and G to update the network parameters.
6.1 RESULTS
We run 10 seeds numbered from O to 9 for each algorithm to keep a fair comparison. After every 500 iterations (time steps), we launch evaluation procedure, which averages 10 rollouts for test. The average reward of a test will be recorded at every evaluation procedure, and all tests throughout the time step scale give the result of each algorithm.
The average rewards of algorithms tested in chosen benchmarks are shown Fig. 2 with standard devi- ation as the confidence interval (CI). From Figs. ((a) 2(b) and we can observe higher converged value and smaller standard deviation of MAVE at late time steps over other baselines. At early stage
before million time steps, MAVE vibrates due to the training dynamics, as we note from Figs. 2(a) 2(0) and 2(d) In Hopper environment, since the converged value is far lower than other benchmark- s, the tolerance for the fluctuation around convergence is much lower, which causes the instability problems of tested baselines. However, MAVE shows strong robustness and has converged value up to 3700 compared with other baselines, which outweighs the limits of state-of-the-ar results in Hopper task, shown in 2(c) In Fig. [ MAVE has relatively stable performance over 5000. For Humanoid with nigh-dimensional action space, Fig. [[e] shows that MAVE is much better than other baselines and can converge around the score of 6200. Over all figures in Fig. 2 SAC and BRAC both have their up and downs, and TD3 gives the worst performance, considering its lack of adopting the policy exploration.
We also show the goodness of trained dynamics model by plotting the results of off-line learning without interacting with the environment after 7i in Fig. labeled by 'MAVE-P", which is analyzed in Appendix
By the way, due to the differences in code details and the Mujoco version (which we use version 3),
the converged maximum may be a bit different from those of (Haarnoja et al., (2018)) and (Feinberg al. 2018). For example, the best result of Halfcheetah 8000 in (Feinberg et al. 2018). however, it is up to 15000, which is 12000 in our case. In contrast, the best result of Ant is 6000 in Haarnoja jet al., 2018b), which is lower than 7000 in our work and similar to Janner al. (0199) Except for Ant, the best results in Janner al. (2019) is closer ours'.
Due to the page limit, the ablation studies can be found Appendix
7 CONCLUSION
In this paper, we proposed a method that combines notations of dynamics training, model prediction, off-line training and on-line planning, which jointly deduce simple solution to the value explo- ration. Our work is sensitive to the dynamics precision, especially to the transition model, however, it is free from costs extra storage and computation and can greatly reduce estimation errors and distribution mismatches. The minor costs of our work are the necessary networks for the dynamics and hyperparameter (given in Appendix G control the speed of asymptotical dynamics training.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.011 01540, 2016.
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample- efficient reinforcement learning with stochastic ensemble value expansion. Advances in neural information processing systems, 31, 2018.
Petros Christodoulou. Soft actor-critic for discrete action settings. arXiv preprint arXiv: 1910.07207 2019.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing in a handful of trials using probabilistic dynamics models. Advances in neural information processing systems, 31, 2018.
Marc Deisenroth and Carl E Rasmussen. Pilco: A model-based and data-efficient approachh to policy search. In Proceedings of the 28th International Conference on machine learning (ICML-II), pp. 465-472. Citeseer, 2011.
Marc Peter Deisenroth, Gerhard Neumann, Jan Peters, et al. A survey policy search for robotics. Foundations and trends in Robotics, 2(1-2):388-403, 2013.
Jingliang Duan, Yang Guan, Shengbo Eben Li, Yangang Ren, Qi Sun, and Bo Cheng. Distributional soft actor-critic: Off-poliey reinforcement learning for addressing value estimation errors. IEEE Transactions on Neural Networks and Learning Systems, 2021.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep rein- forcement learning for continuous control. In International conference on machine learning, pp- 1329-1338. PMLR, 2016.
Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I Jordan, Joseph E Gonzalez, and Sergey Levine Model-based value estimation for efficient model-free reinforcement learning. arXiv preprint 00000 2018.
Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. 80:1587-1596, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. pp. 2052-2062 2019.
Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep q-learning with model-based acceleration. In International conference on machine learning, pp. 2829-2838. PMLR, 2016.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. pp. 1861-1870, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv: 1811v:181..05905 2018b.
Nicolas Heess, Greg Wayne, David Silver, Timothy Lillicrap, Yuval Tassa, and Tom Erez. Learning continuous control policies by stochastic value gradients. arXiv preprint
2015.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Vari- ational information maximizing exploration. Advances in neural information processing systems, 29, 2016.
Jiang Hua, Liangcai Zeng, Gongfa Li, and Zhaojie Ju. Learning for a robot: Deep reinforcement learning, imitation learning, transfer learning. Sensors, (14):1177, 2021.
Michael Janner, Justin Fu. Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. Advances in Neural Information Processing Systems, 32, 2019.
10

---

Under review as a conference paper at ICLR 2023
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint 1rX1::197700045666 2019.
Lukasz Kaiser, Mohammad Babacizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, al. Model-based reinforcement learning for atari. arXiv preprint arXiv: ha0::///3300444 2019.
S Mohammad Khansari-Zadeh and Aude Billard. Learning stable nonlinear dynamical systems with gaussian mixture models. IEEE Transactions on Robotics, 27(5):443-957, 2011.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off off-policy q- learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.0477 2020.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. arXiv preprint 18X2.:/802. 2018.
Thomas Lampe and Martin Riedmiller. Approximate model-assisted neural fitted q-iteration. In 2014 International Joint Conference on Neural Networks (IJCNN), pp. 2698-2704. IEEE, 2014.
Sergey Levine and Pieter Abbeel. Learning neural network policies with guided policy search under unknown dynamics. Advances in neural information processing systems, 27, 2014.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. The Journal of Machine Learning Research, 17(1):1334-1373, 2016.
Sergey Levine, Peter Pastor, Alex Krizhevsky, Julian Ibarz, and Deirdre Quillen. Learning hand- eye coordination for robotic grasping with deep learning and large-scale data collection. The International journal of robotics research, 37(4-5):421 -366, 2018.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint wrXiv:/509.02977 2015.
Hamid Maei, Csaba Szepesvari, Shalabh Bhatnagar, Doina Precup, David Silver, and Richard Sutton. Convergent lemporal-difference learning with arbitrary smooth function approximation. Advances in neural information processing systems, 22, 2009.
Jesus Bujalance Martin, Raphaël Chekroun, and Fabien Moutarde. Learning from demonstrations with sacr2: Soft actor-critic with reward relabeling. arXiv preprint arXiv:21 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arX- 13:55225 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533 533, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia. Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937, 2016.
Thomas M Moerland, Joost Brockens, and Catholijn M Jonker. Model-based reinforcement learning: A survey. arXiv preprint arXiv:2006. 16 2020.

---

Under review as a conference paper at ICLR 2023
Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In 2018 JEEE Interna- tional Conference on Robotics and Automation (ICRA), pp. 7559-7566. IEEE, 2018.
Junhyuk Oh, Valliappa Chockalingam, Honglak Lee, et al. Control of memory, active perception, and action in minecraft. In International Conference on Machine Learning, pp. 2790-2799, PML- R, 2016.
Jan Peters, Katharina Mulling. and Yasemin Altun. Relative entropy policy search. Twenty-Fourth
AAAI Conference Artificial Intelligence, 2010.
John Schulman, Sergey Levine, Pieter Abbeel, Michael I Jordan, and Philipp Moritz, Trust region policy optimization. In Ieml, volume 37, pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (OXxv:/777.06447, 2017.
Richard S Sutton. Integrated architectures for learning, planning, and reacting based on approxi- mating dynamic programming. In Machine learning proceedings 1990, pp. 216-224. Elsevier, 1990.
Gerald Tesauro. Temporal difference learning and td -l-gammon... Communications of the ACM, 38(3):
38-68, 1995.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Che Wang and Keith Ross. Boosting soft actor-critic: Emphasizing recent experience without for- getting the past. arXiv preprint arXiv: 19XX, 04009, 2019.
Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. 1989.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 1911. 13361,1,11,111 2019.
Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegic Mellon University, 2010.
Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433-1433 Chicago, IL, USA, 2008
A PROOF OF LEMMA
Lemma 4. Assume the expected KL-divergence between two transition distributions is bounded by max Eu, Drv (p(s1+1|5t, a.)) ((5411)) az)) S 5,
(17)
then we have Esps (s:) - Pt((s)) ) s 215.
Proof
EENS /p' ptl
=En,SSM 1~8,0 1-t pp 11
P(s 1111 1(
SBamS - ~~ pp
at 8
11- pl- + pt- 0--)) -
BBn-InS,a-1e -p- p1-
+ E
8. INI
- ((
= EMI-nn pp 1-1 pl- 2
Drv P
SEM,nnS 1p-11 pl- 1| +.28
< <lp(so) P(so)] 2t8
=2tō.
(18)
where pz (st) and p = (st),, and the last equality holds because the initial distribution is not affected by transitions. This proof is partly referred to Janner et al. (2019).
0
12

---

Under review as a conference paper at ICLR 2023
B PROOF OF LEMMA 5
pd (s az) (5r. 0z, (642) + > (0(6+1100+1))
(19)
Lemma's 5. Consider the sequence Q(+1 p* constructed by (19). then given the condition that the Q-values are bounded, ie.. [Qr(s.a) < oc, V(s,a) & S x A, the sequence Q₁ will converge to a unique optimal value as -> 00.
Proof
at] S 1Q( 19(5) a(+1) - 0' (' 4-1)]
Y max [Q a++1) Q' (0 a+11)
1++1
=1 - 2x
(20)
where |/- means the norm. Since the Q-value is assumed be bounded, the second inequality holds. We reach a conclusion that V((z,, at) E S x A. (20) holds, which can be rewritten as ||7 Q S ? x
lolloo. which means pa ((. a) max-norm contraction mapping, According to the contraction property, the sequence Qk+1 = PTQu will converge to unique fixed point.
C PROOF OF LEMMA 6
Lemma 6. Assume the absolute value of expected reward function and the expected KL-divergence between the dynamics model and the real transition probability are respectively bounded by
max [as a.) S Tm, am8
max Barx DTV oalp (e))) S 6,
(21)
then we have
Beppp" aant 1Q(s), ac)
Pe (Q(ss, at)] S 50)).
(22)
Proof
E Sup oere [Q(s a:)] Ss [Q(s
Σ Bans,an ('r(s) P*()))
Ears [aIN a:) (z)) (*))))
<rm Σ Eers [7 (p'(p'(s)) - (s{)]]]
<2r
liyd
1=0
2rmo7
(23)
(1-
(1
where the second last inequality is due to Lemma 4
D PROOF OF THEOREM
Theorem 3. Assume E ((s.)s)) [(Q"('', Qw: 5 for t > Tú. then ET that
M(w') 26 for > T.
13

---

Under review as a conference paper at ICLR 2023
Proof According to Lemma then for >> max 1,, Ti,T2 Llw") (,, 2e,
3T2
0, E (s.a.)))))) [(Q' (w, - s for T2,
(s, + (1 -uu
(24)
for > T.
E PROOF OF THEOREM 4
Theorem 4. Assume the absolute value of expected reward function, the expected KL-divergence between the dynamics model and the real transition probability, and the MSE of expected difference between the modeled reward function and the immediate reward are respectively bounded by
max TPawrr(s, a)) S mm
a
max En, p Drv (p(5 at)l| 0z)) S 6,
max E(a.)) { (Ee aim [ra(s, ae, 24+1) - }]] ;
S s.
(25)
then we have the MSE of target prediction error bounded by 2 + 0
Proof
Es Egg ppptt " aenmaruinn [ra(s, az s4t) + 7Q(s 4411 +0( Q(: [(s,
E mptt1 ptt) Ber [r - 7Q (0 -QQ00-0Q01000000000 |++1)] }
(s,))
22 27²
Eo / p (0(5 - -(1041,
<26 27²
[ef mpttl P a4 0441) Q(8++1)
<2 + 8r 622
(26)
7)
where (s.r) is the random sample from the replay buffer, and the last inequality can be referred to
Lemma 6 Then the prediction error is bounded by / /25 8r2
(1-7)*
F NETWORK ARCHITECTURE
We construct the critic network using fully-connected MLP with two hidden layers. The input is composed of the state and action, outputting a value representing the Q-value. The ReLU functions adopted activate the two hidden layers. The setting of policy network follows normal random distribution, whose expectation and variance are dully-connected networks fed only by the state. Both of them have hidden layers activated by the ReLU function. After the hidden layers, Tanh function and a Softplus function follows to form the expectation and variance, respectively. With the expectation and variance, normal distribution can be achieved represent the random policy. The network of transition probability is constructed similarly to that of the policy without of the Tanh clipping, except for the input, which is composed of the state and action instead. And the network of reward function is similar to that of the critic, with the input composed of the state, action and the next state. The architecture of networks are plotted Fig. 3 For simplicity, we omit illustration of reward network in this figure. The above mentioned network architecture is adopted for the random policy. For the algorithm using the deterministic policy, critic is constructed in the same way, however, the actor network is deterministic as the fully connected dense layer.
G HYPERPARAMETERS
Table I lists the common hyperparameters shared by all experiments and their respective settings. In this table, T., means the learning rate of the actor, I,, means the learning rate of critics, and La means
14

---

Under review as a conference paper at ICLR 2023
Fully connected Laver Bdua
Fally cornected laser Bela
Teper
State laper
(
Action layee
Iepet Iayer
Bain
Twrhh
plP
State Torzr
Fully connected Layse
Pally connected layer
Bela
Selai
Suftlls
c
Trpes Taper
Jayer Bell
State Tayer
s(-s,a)
Aetion larer
(s)
Figure 3: Architecture networks.
the learning rate of dynamics including the transition probability and the reward. J, and Ta represent soft update hyperparameter of the actor and the critic, respectively, and т. 1 means we adopt immediate update for the actor. The symbol var represents the variance of gaussian exploration noise, and is the fixed temperature hyperparameter for the term of maximum posteriori action entropy. which is applied of KL divergence for policy regularization applied in BRAC, 8 is the temperature hyperparameter to tune the impact of posteriori transition entropy in MAVE, and is the asymptotic rise rate for k, =1 11 S
algorithms except DDPG and TD3. ag represents the Wight factor
15

---

Under review as a conference paper at ICLR 2023
Table List of hyperparameters
Hyperparameter
Value
Description
Algorithm applied
La
0.0003
Learning rate of actor Learning rate of critic Learning rate of dynamics Soft update parameter of actor Soft update parameter critic
All
Le
0.0003
All
La
0.0003
MAVE
Ta
1
All
Te
0.005
All
y
0.99
Discount horizon factor
All
var
0,2
The variance of exploration noise
All
a
0.1
Fixed temperature
Except DDPG and TD3
B
0.1
Fixed temperature
MAVE
ad
0.1
Wight factor of KL regularization
BRAC
T
0.9999999995
Asymptotic rise rate
MAVE
Batch
256
Size of each mini-batch
All
Units
256
Hidden layer units
All
Memory
1000000
Size of replay buffer Evaluation period
All
Intereal
500
All
Test
10
Rollouts per evaluation
All
Moreover, Batch represents the size of mini-batches sampled for training, and Memory is means the size of replay buffer. The rest in Table [ are the hyperparameters for the evaluation procedure, specifically, Interval means how many time steps between two successive evaluation procedures, and Test means the number of rollouts run during each evaluation procedure.
H OFF-LINE PERFORMANCE
In Halfcheetah environment, the maximum online time step Ti is set as million, which means from million to 3 million steps, the agent stops interacting with the environment and performs the train- ing and planning totally based the reserved fixed experience buffer, more specifically, based on the initial states s in the experience four-tuple slots (s, a, s¹). From Fig. 2(b) MAVE-P represents the off-line performance after Ti, which shows that the model preserve the performance before the online training is stopped, when MAVE still has great potential to continue improving perfor- mance. Similar phenomena can be observed from Figs. [(a) 2(d) [ with the same T1 for Ant, Walker2d and Humanoid environments, respectively. The maximum online time step in Hopper is set as 0.5 million, which is smaller than other tasks because the upper converged value in Hopper is much lower. Besides, we note that Hopper does not reach a good point after stoping the on-line training. however, MAVE-P still manages to converge to 3500 solely counting on off-line training.
ABLATION STUDY
To investigate the contribution of individual parts in the proposed value exploration, we replace the modeled reward function with the immediate reward, replace the model predicted next state s with buffer-sampled s', and replace the on-policy current action a with buffer-sampled a in Q-value function, labeled as 'MAVE-R'. 'MAVE-P" and 'MAVE-Q', respectively. The figures of the three conditions are compared with MAVE in Fig. 4 From these figures, we see these fragments are all necessary. Specifically, lacking the modeled reward function as part of the dynamics and the pre- dicted next state foreseen by the modeled transition probability will induce distribution mismatches for MAVE-R and MAVE-P, respectively, and MAVE-Q will cause estimation error, which can be analyzed by the proof of Theorem
16

---

Under review as a conference paper at ICLR 2023
12000
1000
0000
Algo
Aloo
MAVE
MAVE
MAVER
4000
2000
stops
(a)
(b)
4000
2500
Algo
MAVE
2000
Algo
MAVE
MAWE-Q
2000
MAVE-Q
1000
800
0.4
Time stece
Time steps
(d)
8000
9000
Eoer 320 10w3rd
3000
MAUT-R
MAWE-P
2000
MAVE-
1000
00
02
0.4
8.
0.5
1.0
12
1.4
Timme aleccs
(e)
Figure 4: Ablation Study components in (a) Ant-v3; (b) Halfcheetah-v3; (c) Hopper-v3; (d) Walker2d-v3; (c) Humanoid-v3
17


Reviews:
[{"review_id": "mYOGe_2vBEP", "reviewer": "Reviewer_Zou8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Summary: \nThe paper proposes to mix off- & on-policy critic updates to improve sample efficiency and increase exploration. More specifically, the critic loss function is L = (k * TD_Q_off_policy + (1-k) * TD_Q_on_policy)**2. To enable the on-policy updates, a reward and dynamics model is learned from the collected data. In the experiments, the paper compares learning curves averaged over 10 seeds on the standard openAi gym continuous control tasks. The proposed algorithm performs better than the baselines in terms of sample efficiency as the reward goes up higher faster. \n\n", "summary_of_the_review": "* Conclusion:\nThe idea of the paper is not bad but also not innovative. I am happy that somebody tried mixing the off- & on-policy critic targets and reported their results on it. My personal takeaway from the paper is, that the off-policy nature of the critic targets does not seem to be a major limiting factor in the general approaches. Using the mixed critic target, one only gains 10-20% of performance. Unfortunately, the paper does not provide much insight into how this improved performance is obtained or what the limiting factors are. Is it the model error or not? Why is the a_t+1 chosen w.r.t. to the sampled and not the predicted state? Therefore, the paper is most likely not going to influence the trajectory of the community which is fine and could still be presented at ICLR. From a technical perspective, the paper is a borderline accept paper. However, in combination with the bad presentation and the need for a major revision the paper falls below the acceptance threshold and is a borderline reject for me. ", "strength_and_weaknesses": "* Technical Aspects:\nThe idea to mix off- & on-policy critic updates using a learned model is sound but also not super innovative. It is good that somebody tried it and reported the differences compared to the off-policy critic updates. Most design decisions are technically correct. Only the decision to evaluate the Q-function at t+1 with the predicted state of the dynamics model, i.e. s^p_{t+1}, with the action of the observed state, i.e., a_{t+1} \\sim p(s_{t+1}) seems weird. I do not fully understand why one would evaluate a Q-function with an action that would not have been selected in this state. The argument:\n\n> And a_t+1 ∼ π(·|s_t+1) because of the limitation on one-step global prediction, otherwise, choosing a_t+1 ∼ π(·|sp_t+1) will induce cumulative model bias.\n\ndoes not convince me. First of all, I do not see the cumulative model bias, because you are still using only a one-step prediction. Second, what is worse the model bias or the bad combination of state and action? An ablation study, looking deeper into this design decision would be great. \n\nThe experimental section focuses purely on benchmarking the algorithm compared to existing algorithms in terms of the learning curve. These curves show that the Mave learning curve is higher than the others. I would wish for more experiments that look deeper into what is going on in the algorithm. There would be many different questions, how big is the difference between the on- & off-policy targets? Is the off-policy target an underestimation of the critic or an overestimation? In my opinion, the learning curve is above your learning curve experiments do not contain much information anymore. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "fnDwb-TLCnB", "reviewer": "Reviewer_WVWG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies off-policy actor-critic reinforcement learning. It proposes the MAVE algorithm where an additional dynamics model is learned. In the TD-target of the value function is defined as a mixture of the usual TD-target computed from the next state observed during collection of the experience and the TD-target computed from the next state estimated by the learned dynamics model from the previous state. In the experiments slight improvement over baselines are demonstrated.", "summary_of_the_review": "Overall, the paper goes into an interesting direction but needs a lot of improvements to meet the acceptance threshold.", "strength_and_weaknesses": "Better value estimated in actor-critic setting is an important area of research. Using model-based approaches to get better TD-targets is an interesting direction to improve such algorithms. However, the paper has several weaknesses. The writing should be improved. There are many grammatical incorrect sentences making it a little difficult to follow the paper. Although the proposed method itself is simple it requires much more effort than necessary to understand what the authors propose. For example, first stating the equation for the loss and then in another equation the resulting gradient (eq. 9 and eq.11) seems unnecessary to me and only takes space. Figure 1 showing the mujoco environments is also not needed and the plots in Figure 2 could be arranged much more space efficient. This would allow the ablation studies to be in the main paper instead of the appendix. \n\nConceptually, I do not understand were the advantage of MAVE should come from. As the dynamics model is trained on the same distribution of data that is also used to train the value function, the estimated next state from the dynamics model should just be a noisy version of the true next state which is used for the classical TD-target. An appropriate baseline would then be to use MAVE but to replace the next state estimated by the dynamics model with the true next state plus some additional noise. This would be similar to how TD3 adds noise to the action at the next state but instead adding noise to the next state.\n\nThe experiments show slight improvements over the baselines, however the reported results for some baselines are worse than in the literature - especially for HalfCheetah. This is also true for the v3 versions in gym (cf. https://spinningup.openai.com/en/latest/spinningup/bench.html). The authors likely mean the gym v3 versions when they speak about mujoco version 3, as there is no mujoco version 3.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cS_iEwtheuB", "reviewer": "Reviewer_3k6t", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a model-based value exploration technique to improve actor-critic RL algorithms. This approach involves training models for transition probability and rewards. By one-step prediction using the models, this approach provides effective target value (Q function) exploration. By theoretical analysis and empirical results, the authors show that this technique is effective. ", "summary_of_the_review": "Due to lack in clarity and significance in theoretical analysis, I recommend rejection. But the direction seems interesting. ", "strength_and_weaknesses": "### Strength\nThe use of learned transition probability and reward in constructing a Q function target is novel and interesting. This is relevant to the RL community. Furthermore, the experimental results seem promising. \n\n### Weakness\n\nThe theoretical analysis does not seem to provide sufficient evidence for answering the major claim of this paper – this method can provide effective target value exploration and improve sample efficiency. The analysis (Theorem 2) bounds the error in Q function estimation by the errors in the learn models, but they do not necessarily show that the proposed method is more sample efficient. It’s unclear to me whether the proposed method is better than random exploration. Off-policy methods such as DDPG or SAC work just fine if the data collected have good coverage. \n\nThe clarity of this paper can be improved. The lack of clarity leads to large friction in understanding the correctness and contributions. \n\n- I don’t quite follow some claims throughout the text. For example\n    - “By instinct, the model-based target value is able to explore the future states from different view-points with some certainty”\n    - “Fourth, the accuracy of the learned model is tested by setting a maximum online time step, which is the beginning of off-line planning that is isolated from the environment.”\n- The problem setting has not been very clear. Is this an off-policy setting? What data is given? Is interacting with the environment allowed? \n- The math derivation is not rigorous and the notation is sometimes confusing. Some examples:\n    - In Eq (5), randomness in s_{t+1} and s_{t+1}^p are not considered. \n    - Above Section 4, J(\\theta), but the right hand size does not have \\theta. \n    - P^t under Eq (1) should be a function of \\pi. \n    - In Eq (3), what’s the distribution of s_{t-1}?\n- The number of Lemma and equations are mislabelled throughout the text.\n\nOther issues: \n- The layout of the experimental section can be improved. It seems to be too much space taken by 5 figures in Page 8. \n- In Lemma 1 and Lemma 3, should the expected KL-divergence be total variation distance?\n- In Lemma 3, Eq (7) neglects the RL discount factor in the rates, which is important to understand how the rate is dependent on horizon. \n- Using “dynamics” include both transition probability and reward function is a bit confusing. \n- \"planning\" usually involves relatively long horizon optimization, I did not connect one-step prediction to planning. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3b8AKnG514O", "reviewer": "Reviewer_YZy8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to use dynamics model prediction as the value target value to mitigate several drawbacks of off-policy methods. The authors provide experiments and analysis in the model-based regime.", "summary_of_the_review": "The idea itself is interesting, but the authors fail to establish either theoretical justification or experimental studies to support their claims. The existing results are based on strong assumptions, based on which the conclusion is questionable. ", "strength_and_weaknesses": "Pros: \n\n1. The motivation discussed in the paper is clear. The experiments show a clear performance boost in several tasks.\n\n2.  The problem of interest, i.e. using MBRL to solve the distribution mismatch and over-estimation is a fruitful area.\n\nCons: \n\n1. The claim that the proposed method can mitigate the issues in off-policy methods is weak. The authors only provide several standard theorems with very strong assumptions. Specifically, similar results are established in various MBRL works, e.g., MBPO [1]. The convergence results are not surprising based on the strong assumption: the model is accurate under every training policy. In other words, without the proper way of collecting diverse transition data, which requires exploration, claiming that model-based value target \"provide extra exploration\" is problematic. The conclusion that \"convergence only depends on the accuracy of the learned model\" is also crippled as it requires a globally accurate model. \n\n2. As a result of Con 1, the claimed properties of the proposed method in the intro are not well supported by the theorem.\n\n3. The contribution is limited, with connections to prior works not discussed. Can the authors discuss the connections with the model-based value expansion for model-free RL, e.g. by setting $H=1$ in MVE [2]? Since the reasons that the proposed method improves exploration and mitigates over-estimation are not clear, I am wondering if the method itself makes novel contributions.\n\n4. The experiment shows performance improvement. But the underlying reasons why it benefits training are still not clear. The authors discuss several drawbacks of off-policy methods, so I would like to see experimental designs to demonstrate this (may in tabular MDPs) and how a learned model mitigates these issues.\n\n[1] When to Trust Your Model: Model-Based Policy Optimization.\n\n[2] Model-Based Value Expansion for Efficient Model-Free Reinforcement Learning", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject