FORK: A FORward-looKing Actor for Model-Free Reinforcement Learning
Authors: Honghao Wei
Lei Ying
Venue:  (2021)
Keywords: Reinforcement Learning
Actor Critic
Policy Gradient
Model Free

Abstract: In this paper, we propose a new type of Actor, named forward-looking Actor or FORK for short, for Actor-Critic algorithms. FORK can be easily integrated into a model-free Actor-Critic algorithm. Our experiments on six Box2D and MuJoCo environments with continuous state and action spaces demonstrate significant performance improvement FORK can bring to the state-of-the-art algorithms. A variation of FORK can further solve BipedalWalkerHardcore in as few as four hours using a single GPU.

Under review conference paper at ICLR 2021
FORK: A FORWARD-LOOKING ACTOR FOR MODEL-FREE REINFORCEMENT LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
In this paper, we propose new type of Actor, named forward-looking Actor or FORK for short, for Actor-Critic algorithms. FORK can be easily integrated into model-free Actor- Critic algorithm. Our experiments on six Box2D and MuJoCo environments with continu- ous state and action spaces demonstrate significant performance improvement FORK can bring to the state-of-the-arr algorithms. A variation of FORK can further solve Bipedal- WalkerHardcore in few four hours using single GPU.
1 INTRODUCTION
Deep reinforcement learning has had tremendous successes, and sometimes even superhuman performance, in wide range of applications including board games (Silver al., 2016), video games (Vinyals al., 2019), and robotics (Haarnoja et al. 201 a). A key to these recent successes is the use of deep neural net- works as high-capacity function approximators that can harvest a large amount of data samples approxi- mate nigh-dimensional state or action value functions, which tackles one of the most challenging issues in reinforcement learning problems with very large state and action spaces.
Many modern reinforcement learning algorithms are model-free, so they are applicable in different envi- ronments and can readily react to new and unseen states. This paper considers model-free reinforcement learning for problems with continuous state and action spaces, in particular, the Actor-Critic method, where Critic evaluates the state or action values of the Actor's policy and Actor improves the policy based on the value estimation from Critic. To draw analogy between Actor-Critic algorithms and human deci- sion making, consider the scenario where a high school student is deciding on which college to attend after graduation. The student, like Actor, is likely to make her/his decision based the perceived values of the colleges, where the value of a college is based on many factors including (i) the quality of education it offers, its culture, and diversity, which can be viewed instantaneous rewards of attending the college; and (ii) the career opportunities after finishing the college, which can be thought as the future cumulative reward. We now take this analogy one step further, in human decision making, we often not only consider the "value" of current state and action, but also further forecast the outcome of the current decision and the value of the next state. In the example above, student often explicitly takes into consideration the first job she/he may have after finishing college, and the "value" of the first job. Since forward-looke cision making, we are interested in understanding whether such forward-looking decision making can help Actor; in particular, whether it is useful for Actor to forecast the next state and use the value of future states to improve the policy. To our great surprise, a relative straightforward implementation of forward-looking Actor, an add-on to existing Actor algorithms, improves Actor's 's performance by large margin.
is common in human de-
Our new Actor, named FOrward-looKing Actor or FORK for short, mimics human decision making where we think multi-step ahead. In particular, FORK includes a neural network that forecasts the next state given the current state and current action, called system network; and a neural network that forecasts the reward given a (state, action) pair, called reward network. With the system network and reward network,

---

Under review conference paper at ICLR 2021
FORK can forecast the next state and consider the value of the next state when improving the policy. For example, consider the Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al. 2015). which updates the parameters of Actor as follows:
Ô + VV Q Ag))
where 8t is the state at time t, e are Actor's parameters, 3 is the learning rate, Qu a) is the Critic network, and As(s) is the Actor network. With DDPG-FORK, the parameters can be updated as follows:
+ Φ (V Qe (s.s. V. Ap(s)) + VV 1. (5)++)))
2VQQu(5442 A0(5++2))
(1)
where R2 is the reward network, and St+1 and 5,+2 the future states forecast by the system network Fe.
We will see that FORK can be easily incorporated into most deep Actor-Critic algorithms, by adding two additional neural networks (the system network and the reward network), and by adding extra terms to the loss function when training Actor, e.g. adding term
R (8t, Ast):) + 1 (5(+)1)) Q8442
for each sampled state Sr implement
We remark that Equation (T) is just one example of FORK, FORK can have different implementations (a detailed discussion can be found in Section B We further remark that learning the system model is not new idea and has long history in reinforcement learning, called model-based reinforcement learning (some state-of-the-ard model-based reinforcement learning algorithms and the benchmark can be found in (Wang jet al. (.19)). Model-base reinforcement learning uses the model in a sophisticated way, often based on deterministic stochastic optimal control theory to optimize the policy based on the model. FORK only uses the system network as blackbox to forecast future states, and does not use it as a mathematical model for optimizing control actions. With this key distinction, any model-free Actor-Critic algorithm with FORK remains to be model-free.
our experiments, we added FORK two state-of-the-ar model-free algorithms, according recent benchmark studies (Duan et al. 2016a Wang et al.] (019):: TD3 (Fujimoto et al. 2018) (for deterministic policies) and SAC (Haarnoja et al. (2018b) (for stochastic policies). The evaluations on six challenging envi- ronments with continuous state space and action space show significant improvement when adding FORK. particular, TD3-FORK performs the best among the we tested. For Ant-v3, it improves the average cumulative reward by more than 50% than TD3, and achieves TD3's best performance using only 35% of training samples. BipedalWalker-v2 is considered "solved" when the agent obtains an average cumulative reward of at least 30Q" TD3-FORK only needs 0.23 million actor training steps solve the problem, half of that under TD3. Furthermore, a variation of TD3-FORK solves BipedalWalkerHaddoor a well known difficult environment, with as few four hours using single GPU.
1.1 RELATED WORK
The idea of using learned models in reinforcement learning is not new, and actually has long history in reinforcement learning. At a high level, FORK shares similar spirit as model-based reinforcement learning and rollout. However, in terms of implementation, FORK is very different and much simpler. Rollout in general requires the Monte-Carlo method (Silver et al. 2017) to simulate a finite number of future states from the current state and then combines that with value function approximations to decide the action to take at the current time. FORK does not require any high-fidelity simulation. The key distinction between FORK and model-based reinforcement learning is that madel-based reinforeement learning uses the learned
https: / http:///ftttttcceooe////////l/tt ___a_________ ker .py
2

---

Under review conference paper at ICLR 2021
model in a sophisticated manner. For example, in SVG (Heess al., 2015). the learned system model is integrated as part of the calculation of the value gradient, in (Gu et al. 2016). refitted local linear model and rollout are used derive linear- Gaussian controller, and (Bansal et al. 2017) uses learned dynamical model to compute the trajectory distribution of given policy and consequently estimates the corresponding cost using a Bayesian optimization-bn
policy search. More model-based reinforcement learning algorithms and related benchmarking can be found in (Wang et al. 2019). FORK, on the other hand, only uses the system network to predict future states, and does not use the system model beyond that. Other related work that accelerates reinforcement learning algorithms includes: acceleration through exploration strategies (Gupta et al. 2018), optimizers (Duan et al. 2016b). and intrinsic reward (Zheng et al., 2018), just name few. These approaches complementary to ours. FORK can be added further accelerate learning.
2 BACKGROUND
Reinforcement Learning algorithms aim at learning policies that maximize the cumulative reward by inter- acting with the environment. We consider a standard reinforcement learning setting defined by a Markov decision process (MDP) (S,A, Po. p, 7), where S is set of states, A is the action space, Po()) is the probability distribution of the initial state, : S x S x A -> [0, 20) is the transition density function, which represents the distribution of the next state 9t+-1 given current state 5; and action d,, r SxA + [Tmin T Tmax] is the bounded reward function on each transition, and 7 E (0, 1] is the discount factor. We consider a discrete-time system. At each time step t, given the current 8 E S, the agent selects an action a, E A based on a (deterministic or stochastic) policy (a (t), which moves the environment to the next state st+1. and yields reward r, r(st, a,) to the agent. We consider stationary policies this paper under which the action is taken based on 8t, and is independent of other historical information.
Starting from time 0, the return of given policy ı the discounted cumulative reward
T
(1)
Σ 7/r(s at), given 80 i.
1=0
(i) is also called the state-value function. Our goal is to learn a policy "" that maximizes this cumulative reward
n E arg max (i) Vi.
We assume our policy is parameterized by parameter é, denoted by Te, e.g. by the Actor network in Actor- Critic Algorithms. In this case, our goal is to identify the optimal parameter di that maximizes
t= E arg max Jr, (i).
Instead of value function, it often convenient to work with action-value function, Q-function, which is defined as follows:
(s, a) = E' (s, + (s)))
where s' is the next state given current state and action a. The optimal policy is policy that satisfies the following Bellman equation (Bellman 1957):
Qn-(s, = E r(s,a) + max Qu- (s' (5,
+ a'EA
When neural networks are used to approximate action-value functions, we denote the action-vall value function by (s, a), where & is the parameters of the neural network.
3

---

Under review conference paper at ICLR 2021
3 FORK FORWARD-LOOKING ACTOR
This paper focuses on Actor-Critic algorithms, where Critic estimates the state or action value functions of the current policy, and Actor improves the policy based on the value functions. We propose new type of Actor, FORK. More precisely, new training algorithm that improves the policy by considering not only the action-value of the current state (or states of the current mini-batch), but also future states and actions forecast using learned system model and learned reward model. This forward- Actor is illustrated FORK, we introduce two additional neural networks:
Figure
The system network Fg. The network is used to predict the next state of the environment, i.e,, given current state 8 and action a,, predicts the next state 8141 = Fo a,). With experi- ences (st, at, St+1), training the system network is a supervised learning problem. The neural net- work can be trained using mini-batch from replay- buffer and smooth-LI loss L(0) = |[+1 Fo(se, smooth
Forward LrevarrdLooking.oo Actor Ag(s) Fe(s,a) R₂ a)
Loss functions:
Actor:
((4)) El Agls))))Ap(s))) ( R.(ss))())) -y,((rss/A(Bu))) (r
system network: L(0)-T/Rs.r: -Tals ae) lamootheral
reward network: 1(())-((Irr -R, (.)
The reward network RG. This network predicts the reward given current state 8, and action at, i.e. iz (8, ae). The network can be trained from experience (8t, Ot; rt), with MSE loss L(n) = ||,, (s;,,a,))
Figure FORK includes three neural networks, the policy network Ag, the system model Fe, and the re- ward model Ry
FORK. With the system network and the reward network, the agent can forecast the next state, the next next stat and so on. Actor can then use the forecast improve the policy, For example, we consider the following loss function
( = E (u
- (s, As(s)) (a
- 22 (5442, A
(2)
the loss function above, 8 are from data samples (e.g. replay buffer), 8141 and $242 are calculated from the system network shown below:
1 Fø(se, As(s and 5c42 F6 Ag(5++1))
(3)
Note that when training Actor As with loss function L(Ø), other parameters L(Ø) are regarded as constants except & (see the PyTorch code in the supplemental materials).
The funiiinfunction Q, without function approximation, under current policy Ag satisfies
(st, As(s)) E [r(s, As(s:) (r(s++1, Ag(st+1)) + 2 (8t+2, A As(sc+2))],
where r, $1+1 and St+2 the actual rewards and states under the current policy, not estimated values.
Therefore, the loss function L(d) can be viewed as the average two estimators.
Given action values from Critic and with mini-batch of size N, FORK updates its parameters follows:
where 8e is the learning rate and
N
voL())
N Σ
Vo (ad1( R)
+y R
4

---

Under review conference paper at ICLR 2021
where 8 and 3 are the next state and the next next state estimated from the system network.
We note that it is important to use the system network to generate future states as in Equation because they mimic the states under the current policy. the replay buffer, then the sequence is from the old policy, which does not help the learning. Figure 2 compares TD3-FORK, TD3, and TD3-MT, which samples a sequence of three consecutive states, on BipedalWalker-v3 environment. We can clearly see that simply using consecutive states from experiences does not help improve learning. In fact, it significantly hurts the learning.
would sample a sequence of consecutive states from
Modified Reward Model: We found from our experiments that the reward network can more accurately predict reward ra when including the next state 81+1 as input into the reward network (an example can found in Appendix A..I). Therefore, use modified reward network R,y (8t, Oe, S1+1) in FORK.
300
Average Cunulative Reward
200
Adaptive Weight: Loss function L(Ø) in our algorithm uses the system network and the reward network to boost learning. In our experiments, we found that the forecasting can signifi- cantly improve the performance, except at the end of learning, Since the system and reward networks are not perfect, the er-
100
-100
rors in prediction can introduce errors/noises. To overcome
this issue, we found it is helpful to use an adaptive weight so that FORK accelerates learning at the beginning but weight decreases gradually as it gets close to the learning goal. comparison between fixed weights and adaptive weights can
0.2M
0.4M
0.5M
0.5M
1M
Timestaps
Figure 2: TD3-FORK versus TD3-MT
be found in Appendix A.2 We use simple adaptive weight w= (1)) Wa, where F is the moving average
of cumulative reward (per episode), and ro is predefined goal, wo is the initial weight, and (a)} if 0 < a < 1, = 0 a < 0 and = if a > 1. The loss function with adaptive weight becomes
[(Ø) P. [- Qv As(s))) (s, As(ss)) wy (5e+1, Ag(B+11) 2 (8et2, (+(+2)))
Furthermore, we set threshold and let w = 0 if the loss of the system network is larger than the threshold. This is avoid using FORK when the system and reward networks are very noisy. We note that in our experiments, the thresholds were chosen such that w 0 for around 20, 000 steps at the beginning of each instance, which includes the first 10,000 random exploration steps.
Different Implementations of FORK: It is easy to see FORK can be implemented in different forms. For example, instead of two-step ahead, we can use one-step ahead as follows:
((Φ) E ( As(s:) (R,,( Ao(st)) w (Set1, Ab(
(5)
or only use future action values:
L(Φ) E 1- (st, As(st)) (Qw (ge+1, Ag(+++1)) + w Q2 (5+42, Ao(Fr42)))
(6)
We compared these versions with FORK. The performance comparison can be found in Appendix
B.3
4 EXPERIMENTS
In this section, evaluate FORK add-on existing algorithms. We name algorithm with FORK algorithm-FORK e.g. TD3-FORK or SAC-FORK. As an example, a detailed description of TD3-FORK can We focused two algorithms: TD3 (Fujimoto al.) 2018) and SAC (Haarnoja
be found in Appendix A.3
5

---

Under review conference paper at ICLR 2021
et 2018b) because they were found have the best performance among model-free reinforcement learn- ing algorithms in recent benchmarking studies (Duan et al. 2016a Wang et al. 2019). We compared the
performance of FD3-FORK and SAC-FORK with TD3, SAC DDPG (Lillierap 2013).
4.1 Box2D AND MuJoCo ENVIRONMENTS
We selected six environments: BipedalWalker-v3 from Box2D (Catto) 201
Ant-v3, Hooper-v3.
HalfCheetah-v3, Humanoid-v3 and Walker2d-v3 from MuJoCo (Toorovv 2012 2 shown in Figure All these environments have continuous state spaces and action spaces.
(a) v3 (b) Ant-v3
Bipedal
(c) Hopper-v3
(d) HaliCheetah-v3
(e) Humanoid-v3
(f)
Walker2d-v3
Figure The six environment used in our experiments
4.2 IMPLEMENTATION DETAILS
operation, e.g. training Actor with mini-batch; episode: single-run of the environment from the beginning to the end, consisting of many steps; and instance: the entire training consisting of multiple episodes.
Terminology. step (or time):
Hyperparameters. Because FORK is an add-on, for TD3, we used the authors' implementation (https: 7/github. ftm ID33 for SAC, we used PyTorch version (https: (https://qithi) //ttt/b/itttbbbt com/ Eitchyr/rlkisg recommended by the authors without any change except adding FORK. The hyper- parameters of both TD3 and SAC are summarized in Table B Appendix A.4 and the hyperparameters related to FORK are summarized in Table in the same appendix.
We can see TD3-FORK does not require much hyperparameter tuning. The system network and reward network used in the environments are the same except for the Humanoid-v3 for which we use larger system and reward networks because the dimension the system is higher than other systems. The base weight wo is the same for all environments, the base rewards are the typical cumulative rewards under TD3 after successful training, and the system thresholds are the typical estimation errors after about 20,000 steps.
SAC-FORK requires slightly more hyperparameter tuning. The base weights were chosen be smaller values, the base rewards are the typical cumulative rewards under SAC, and the system thresholds are the same as those under TD3-FORK.
Initial Exploration. For each task and each algorithm, we use a random policy for exploration for the first 10,000 steps. Each step is one interaction with the environment.
Duration of Experiments. For each environment and each algorithm, we ran five different instances with different random seeds. Since we focus Actor performance, Actor trained for 0.5 million times for each instance. Since TD3 uses delayed Actor with frequency (i.e. Actor and Critic are trained with 1:2 ratio), Critic trained one million times under TD3 and TD3-FORK. For SAC, SAC-FORK and DDPG,
6

---

Under review conference paper at ICLR 2021
Critic was trained 0.5 million times. The performance with the same amount of total training, including Critic training and Actor training, can be found in Appendix B.2 where for each algorithm, Critic and Actor, together, were trained 1.5 millions times.
4.3 RESULTS
Figure 4 shows the average cumulative rewards, where we evaluated the policies every 5,000 steps without exploration noise during training process. Each evaluation was averaged 10 episodes. We train five dif- ferent instances for each algorithm with same random seeds. The solid curves shows the average cumulative rewards (per episode), and the shaded region represents the standard deviations.
The best average cumulative rewards (its definition can be found in Appendix B.I are summarized in Table D We can see that TD3-FORK outperforms all other algorithms. For Ant-v3, TD3-FORK improves the best average cumulative reward by more than 50% (5699.37 (TD3-FORK) versus 3652. (TD3)).
DDPG
SAC
SAC-FORK
TD3-FORK
TD3
0002
Average Cumuialive Reward
000
5002
Average Cumulalive Reward
2002
200
Average Curddllvv
2000
2000
1000
1032
-100
-200
0:1M
0.3M
0.MM
0.5M
-1000
0:1M
0.2M
BM
0.4M
0.5M
0.1M
0.2M
0.5M
0.4
0.5M
Numbor Training Stops (Actor)
Number Training Stops (Actor)
Number Training Steps (Actor)
(b) Bipedal Walker-33
(c) Ant-v3
(d) Hopper-v3
12000
0000
10000
Avorage Canuistive Foward
5000
Carmulltive Rewward
2002
4000
8000
6000
2000
2000
Aveeage
2000
1030
1000
0.1M
0.281
0.3M
0-4M
0.154
0.20
0.391
0.4M
0.5M
0.1M
0.211
0:38
DAM
Number of Training Staps (Actor)
Number of Thaining Steps (Actorr
Number of Training Steps (Actor)
(e) HallCheetah-v3
(i) Humanoid-v3
(g) Walker2d-v3
Figure 4: Learning curves of six environments. Curves were smoothed uniformly for visual clarity.
We also studied the improvement in terms of sample complexity. In Table Actor training required under TD3-FORK (SAC-FORK) to achieve the best average cumulative reward under TD3 (SAC). For example, for BipedalWalker-v3.. TD3 achieved the best average cumulative reward with 0.4925 million steps of Actor training; and TD3-FORK achieved the same value with only 0.225 million steps of Actor training, reducing the required samples by more than half.
summarized the number
7

---

Under review conference paper at ICLR 2021
Table 1: The best average cumulative rewards the algorithms. highlighted in bold text.
The best value for each environment
Environment
TD3-FORK
TD3
DDPG
SAC
SAC-FORK
BipedalWalker-v3
317.40
307.69
125.90
223.20
303.18
Ant-v3
5699.37
3652.11
995.59
1892.50
3091.00
Hopper-v3
3522.57
3517.57
1713.52
3381.01
3359.18
HalfCheetah-v3
10818.89
9893.04
5886.75
9314.64
8781.06
Humanoid-v3
5439.31 31
5386.21
456.35
5293.89
5233.89
Walker2d-v3
4616.46
4278.90
2404.60
4037.87
4243.69
Table Sample complexity (million). The number of training steps needed for TD3-FORK (SAC-FORK) to achieve the same best average cumulative reward under TD3 (SAC). The numbers under TD3 (SAC) are the time steps at which the TD3 (SAC) achieved the best average cumulative rewards.
Environment
TD3-FORK
TD3
SAC-FORK
SAC
BipedalWaller-v3
0.225
0.492
0.370
0.495
Ant-v3
0.272
0.475
0.285
0.495
H3pper-v3
0.405
0.457
0.500+
0.335
HalfChhetah--3
0.295
0.495
0.500+
0.480
Humanoid-v3
0,430
0.497
0.500+
0.495
Walker2d-v3
0.360
0.500
0.396
0.475
In summary, FORK improves the performance of both TD3 and SAC after being included as add-on. The improvement is more significant when adding to TD3 than adding to SAC. FORK improves TD3 all six environments, and improves SAC in three of the six environments. Furthermore, TD3-FORK performs the best in all six environment. More statistics about this set of experiments can be found in Appendix B.T
also presented experimental results where Actor and Critic together have the same amount of training across all algorithms (i.e. under TD3 and TD3-FORK, Actor was trained 0.5 million times and Critic was trained million times; and under other algorithms, both Actor and Critic were trained 0.75 million times). In this case, TD3-FORK performs the best among four of the six environments, and SAC-FORK performs the best in the rest two environments.
In Appendix B.2
4.4 BIPEDAL WAR.WAALLEEEEAACRRRRRYY
HARDCORE-V3
A variation of TD3-FORK can also solve well-known difficult environment, BipedalWallee-Harddoon v3, in as few as four hours using a single GPU. From the best of our knowledge, the known algorithm needs to train for days put (hctps: //qithub.... com/ dgriff777/a33 S00E:77/a3ccooniinooooooo continuuus). You can view the performance on Walkerl during and after training at ht/htpss:///mm htwp:aaeeeennnnnnnneee YnYQpXxxx-Q The im- plementation details can be found in Appendix
72 cpu AWS EC2 with 64 worker processes taking the raw frames in-
5 CONCLUSIONS
This paper proposes FORK, forward-looking Actor, as an add-on to Actor-Critic algorithms. The evalua- tion of six environments demonstrated the significant performance improvements by adding FORK two state-of-thh--at model-free reinforcement learning algorithms. A variation of TD3-FORK further solved Bipedal WalkerHaddooo in few as four hours with single GPU.
8

---

Under review conference paper at ICLR 2021
REFERENCES
Somil Bansal, Roberto Calandra, Kurtland Chua, Sergey Levine, and Claire Tomlin. MBMF: Model-based priors for model-free reinforcement learning. arXiv preprint arxiv: 2017.
Richard Bellman. Dynamic Programming. Dover Publications, 1957. ISBN 9780486428093..
Erin Catto. Box2D, 2D physics engine for games., 2011. URL
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforce- ment learning for continuous control. In International Conference on Machine Learning, pp. 1329-1338, 2016a.
Yan Duan, John Schulman, Xi Chen, Peter Bartlett, Ilya Sutskever, and Pieter Abbeel. RI 2: Fast rein- forcement learning via slow reinforcement learning. arXiv preprint arXiv:/611.02777 2016b.
Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation enror in actor-critic methods. arXiv preprint 1802 09077, 2018.
Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep q-learning with model- based acceleration. In International Conference Machine Learning, pp. 2829 2238, 2016.
Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta-reinforcement learning of structured exploration strategies. In Advances in Neural Information Processing Systems, pp. 5302-5311 2018.
Tuomas Haarnoja, Vitchyr Pong, Aurick Zhou, Murtaza Dalal, Pieter Abbeel, and Sergey Levine Compos- able deep reinforcement learning for robotic manipulation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 6251.. IEEE, 2018a.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pp. 1861-18700 2018b.
Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap, Tom Erez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. In Advances Neural Information Processing Systems (NeurlPS), pp. 2944 2952, 2015.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters, arXiv preprint arXiv:1709.06500 2017.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint S.Ktv:1500.0291..0 2015.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam. Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484448) 2016.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-359, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 20/2 JEEE/RS/ International Conference on Intelligent Robots and Systems, pp. 5026-50000000 5033. IEEE, 2012.

---

Under review conference paper at ICLR 2021
Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using mulli--gem reinforcement learning. Nature, 575(7782) 354, 2019.
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Picter Abbeel, and Jimmy Ba. Benchmarking model-based reinforcement learning. CoRR, 0tsp/19070200577 2019.
pettp://arxinn org ab9// .20577
7eyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. In Advances in Neural Information Processing Systems, pp. 4644-4654, 2018.
10

---

Under review conference paper at ICLR 2021
This appendix provides additional details about FORK and additional experiments. The appendix is orga- nized as follows:
In Section we provide additional details about FORK used in our experiments, including a description of TD3-FORK and the hyperparamters.
In Section we present additional experimental results, including additional statistics of the ex- periments conducted in Section 4 performance comparison under the same number of Actor+Critic training, and performance of different implementations of FORK.
we present additional changes we made when using variation of TD3-FORK, in particular, TD3-FORK-DQ, solve the BipedalWallerFFrrderre....
In Section C
A ADDITIONAL DETAILS OF FORK
A.1 REVISED REWARD NETWORK
We found from our experiments that the reward network can more accurately predict reward 7t when in- cluding the next state 5 input into the reward network. FigureSJshows the mean-square-error (MSE) the reward network with (st, ae) as the input versus with (s,, ag 5(+1) as the input for Bipeda/Walker-33 during the first 10,000 steps. We can clearly that MSE is lower in the revised reward network.
30
Training Loss (5, s') Training Loss a)
25
Mean Square Error
20
15
10
5
Uv
0
10K
Timesteps
Figure Training losses under the two different reward networks
A.2 ADAPTIVE WEIGHTS VERSUS FIXED WEIGHTS
We compared TD3-FROK with the fixed weights, named as TD3-FORK-F, where the weight is chosen to be 0.4. TD3-FORK performs the best in four out of the six environments. TD3-FORK-F has worse performance than TD3 on Walker2d-v3. We therefore proposed and used the adaptive weight because of this observation.
11

---

Under review conference paper at ICLR 2021
TD3-FORK-F
TD3-FORK
TD3
400
4000
Avenge Cumulaaive Rowwed
Avenage Cumulalive Reward
3000
300
3000
Average Cumuishve
200
2200
2000
100
3000
1500
2000
1000
1200
0.2M
0.4
0:EM
Timeslaps
Timesteps
Timesteps
(h) Bipeda/Walleerteeeee
(c) Ant-v3
(d) Hopper-v3
5000
10020
Avarag Curubilive Rewand
6200
Average Curulelive Reward
4000
4C00
Cumutative
8200
4000
2000
2000
2000
1000
1000
6.2M
0.4M
0.EM
0.8M
1M
0.2M
0.4M
0.6M
0.8M
0.20
0.4M
0.6M
0.BM
1M
Timestaps
Timostopp
Timastops
(c) HalfChectah-v3
(f) Humanoid-vā
(g) Walker2d-v3
Figure 6: Learning curves of TD3, TD3-FORK-F and FD3-FORK. Curves are smoothed uniformly for visual clarity.
A.3 TD3-FORK
The detailed description of TD3-FORK can be found in Algorithm [ and the codes are also submitted as a supplemental material.
A.4 HYPERPARAMETERS
Table 3 lists the hyper-paramen parameter used in DDPG, SAC, SAC-FORK and TD3-FORK. We kept the same hyperparamter values used in SAC and TD3 codes provided or recommended by the authors. We did not tune these parameters because the goal is to show that FORK is a simple yet powerful add-on to existing Actor-Critic algorithms.
Table summarizes the environment specific parameters. In particular, the base weight and base cumulative reward used implementing the adaptive weight, and threshold for adding FORK. The base cumulative rewards for TD3-FORK are the typical cumulative rewards under TD3 after training Actor for 0.5 million steps. The base cumulative rewards for SAC-FORK are similarly chosen but with a more careful tuning. The thresholds are the typical loss values after training the system networks for about 20,000 times including the first 10,000 exploration steps. the implementation, FORK added to Actor training only after the system network can predict the next state reasonably well. We observed TD3-FORK with intuitive choices of
12

---

Under review conference paper at ICLR 2021
Algorithm I 1'D3-FORK
Initialize critic network Quis Qii. system network Fo, R, and actor network As with random parame- ters 1/1, 1/2, 0, 17, Φ
1: Initialize target network Ø' Φ, 3y 1. <- 1/2
Initialize replay buffer B, soft update parameter T
Initialize base reward ro, wi; threshold I and moving average reward +- Initialize noise clip bound c, state bound (Omin- Omax)
2: for episode e I....,A. M do
3: Initialize observation state
4: Initialize episode reward I' 6: Select action a, according to the current policy and exploration noise or ~ As(s) + Et. where (0)
5: for f ,, do
7:
Execute action a and observe reward Tt, new state $1+1 Store transition tuple (st, dt, Ft, S1+1) into replay buffer B Sample random minibatch of N transitions (8i, az Ti 8641) from B d; Th (8++1) + 6, (O, -c,))
8:
9:
10:
11:
Set = = m1n Qu (5(+1)
12:
++
13:
Update critic network by minimizing the loss: L( = N4 Σj=1 E. - -Q, a:))
14:
Update state system network by minimizing loss: L(0) = | Fo(s;, a;) smmooth
15:
Update reward system network by minimizing the loss: L(n) = N E, (ri Rn aj 81+1)) if mod then
17:
Update & by the sampled policy gradient:
18:
if (0) > I then
19:
V VoL(o) 1/4 E. Vo (8t, a)la= V (sA((s))
20:
else
21:
1 clip Fo (s, (s:), Ornini Omux)) 5/u42
(5(5541)) }), Ominn Omax
22:
V Vol(d) " Σ. (V. vn (8t, ()) V (0A + w. V. Ra a, 6(4+1 a= VoAp(si)
23:
+w> R., ( +2 la=Ao(((_ VoAp(5 + wy 2 VoQ (5 a) (od4()((22)
24:
end if
25:
Update target networks:
26:
Φ' T (1 7)
27:
2/ + TW + (1-T)
28:
end if
29:
end for
30:
Update + ((e 1)r + r)/e
31:
Update adaptive weight w &- min(1 max(0, (,, 1)wo
32:
end for
hyperparameters worked well across different environments and required little tuning, while SAC-FORK required some careful tuning choosing the base weights and the base cumulative rewards.
13

---

Under review conference paper at ICLR 2021
Table 3: Hyperparameters
Parameter
Value
Shared
optimizer
Adam
learning rate
3 10
discount (7)
0,99
replay buffer size
106
number of hidden layers (all networks)
2
Batch Size
100
Target Update Rate
5 10-3
Target update delay (TD3, TD3-FORK)
2
nonlinearity
ReLU
number of hidden units per layer (Critic) number of hidden units per layer (Actor)
256
256
TD3-FORK
number of hidden units the system network
[400, 300]
(Humanoid 1024, 1024)
number of hidden units of the reward network
256, 256
(Humanoid 1024, 1024))
SAC-FORK
number of hidden units the system network
512,,122
(Humanoid 1024, 1024))
number of hidden units of the reward network
512, 512)
(Humanoid 1024, 1024)
Table 4: Environment Specific Parameters
Environment
Base Weight wo
Base Cumulative Reward ro
System threshold /
TD3-FORK
Bipeda/Walker-v3
0.6
320
0.01
Ant-v3
0.6
6200
0.15
Hopper-v3
0.6
3800
0.0020
HalfCheetah-v3
0.6
12000
0.20
Humanoid-v3
0.6
5200
0.20
Walker2d-v3
0.6
4500
0.15
SAC-FORK
Bipedal
0.40
320
0.01
Ant-v3
0.40
5200
0.020
Hopper-v3
0.10
2500
0.0020
HalfCheetah-v3
0.10
6000
0.10
Humanoid-v3
0.10
4500
0.10
Walker2d-v3
0.10
3000
0.15
B ADDITIONAL EXPERIMENTAL RESULTS
B.1 BEST AVERAGE CUMULATIVE REWARD, STANDARD-DEVIATION. AND BEST INSTANCE CUMULATIVE REWARD
Table 5 summarizes the best average cumulative rewards, the associated stance cumulative rewards. They are defined follows. Recall that each algorithm is trained for five instances, where each instance includes 0.5 million steps of Actor training. During the training process, we evaluated the algorithm every 5,000 steps without the exploration noise. For each evaluation, calculated
and best in-
14

---

Under review conference paper at ICLR 2021
the average cumulative rewards (without discount) over 10 episodes, where each episode is 0 1, 600 un- der SipedaWWalker-yy is 0 ~ 1, 000 under Ant-v3, Walker2d-v3, Hopper-v3, Humanoid-v3, and is exactly 1,000 under HalfCheetah-v3.
Now let x denote the average cumulative reward the rth evaluation during the Ith instance. Then
5
Best Average Cumulative Reward (Best Average): max
x(9
1=1
Mtandard-Deviatioe
( -X,)
5
Best Instance Cumulative Reward (Best Instance): max max X!O)
Table 5: Best Average Cumulative Rewards, Standard Deviations, and Best Instance Cumulative Rewards of TD3-FORK, TD3, DDPG, SAC, SAC-FORK over Six Environments. The Best Value for Each Environment is in Bold Text.
Environment
TD3-FORK
TD3
DDPG
SAC
SAC-FORK
BipedalWalker-vy
Best Average Cumulative Reward
317.40
307.69
125.90
223.20
303.18
Standard Deviation
+44688
+11.18
+130.06
+120.36
+13.58
Best Instance Cumulative Reward
322.97
317.47
254.17
314.91
322.58
Ant-v3
Best Average Cumulative Reward
5699.37
3652.11
995.59
1892.50
3091.00
Standard Deviation
4:234.62
+510.50
+2267
+523.40
11000
Best Instance Cumulative Reward
6015.47
4546.47
999.79
2595.72
4134.06
Hopper-v3
Best Average Cumulative Reward
3522.57
3517.57
1713.52
3381.01
3359.18
Standard Deviation
+120.22
+72.37
+957.20
+164.42
+119.3
Best Instance Cumulative Reward
2659.27
3591.42
3573.53
3595.08
3511.14
HalfCheetah-v3
Best Average Cumulative Reward
10818.89
9893.04
5886.75
9314.64
8781.06
Standard Deviation
+174.77
1679.21
2499.82
+-58.47
+816.97
Best Instance Cumulative Reward
11044.74 74
10361.92
9595.08
9802.70
9848.15
Humanoid-v3
Best Average Cumulative Reward
5439.31
5386.21
456.35
5293.89
5233.89
Standard Deviation
+152.26
+115.92
+75.22
226.75
#65.38
Best Instance Cumulative Reward
5685.77
5513.46
529.84
5333.69
5284.72
Walker2d-v3
Best Average Cumulative Reward
4616.46
4278.90
2404.60
4037.87
4243.69
Standard Deviation
+449.26
+195.35
-1399.97
+740.40
1467.76
Best Instance Cumulative Reward
5192.52
4541.51
4618.44
5116.54
5042.80
15

---

Under review conference paper at ICLR 2021
B.2 COMPARISON WITH THE SAME AMOUNT OF TOTAL TRAINING
In Section the algorithms were compared assuming the amount of Actor training since focus is on the performance of Actor. Since TD3 uses delayed Actor training, Critic of TD3 and TD3-FORK trained twice much Critic of SAC and SAC-FORK when Actor is trained the same number of steps, which gives advantage to TD3 and TD3-FORK.
To further compare the performance of TD3-FORK and SAC-FORK, present the results where for each algorithm, Actor and Critic, together, were trained 1.5 million steps. In particular, Actor was trained 0.5 million steps and Critic is trained I million steps under TD3 and TD3-FORK; and Actor and Critic were trained 0.75 million steps each under SAC and SAC-FORK, The results can be found in Figure 7
DDPG
SAC
SAC-FORK
TD3-FORK
TD3
0000
A000
Avrerage Cumuus Ieridstive Reward
Averagao Gumulefiv Reward
300
5000
Avarage Cumulelive
200
4000
2000
100
2000
1000
-200
-1000
0.75M
0.EM Numbar Training Stops (Actor Cittcc)
Numbor Training Stops (Actor + Criii))
Number of Training Staps (Actor Critie Critc)
(h) Sareda//Wllee-v33
(c) Ant-v3
(d) Hopper-v3
Average Cumuistive Roward
10030
Average Curujelive Reward
Roward
5000
4000
3002
8000
8000
2002
Average
0000
1000
1000
Number
Steps (Actor
Number Thaining Steps (Actoo-
Number Training Staps Cittie)
(e) HallCheetah--3
(I) Humanoid-v3
(g)
Walker2d-v3
Figure 7: Learning curves of the six environments. Under each algorithm, Actor and Critic, together, were trained for 1.5 million steps. Curves are smoothed uniformly for visual clarity.
Table summarizes the best average cumulative rewards, standard-deviations, and the best instance cumu- lative rewards. We can see that in terms the best average cumulative rewards, TD3-FORK performed the best in four out of the six environments, including BiledalWalker, Ant, Hopper and HalfChectah; and SAC-FORK performed the best in the remaining two Humanoid and Walker2d.
B.3 DIFFERENT IMPLEMENTATIONS OF FORK
As we mentioned in Section 3 FORK can have different implementations. We considered two examples in Section and compared their performance add-on to TD3. We call FORK with loss function Equation (5)
16

---

Under review conference paper at ICLR 2021
Table 6: Best Average Cumulative Reward, Standard- Deviiatinne,i and Best Instance Cumulative Reward. The Best Value for Each Environment is in Bold Text.
Environment
TD3-FORK
TD3
DDPG
SAC
SAC-FORK
Bipedal W
Best Average Cumulative Reward
317.40
307.69
125.90
304.56
311.49
Standard Deviation
+4.68
#11.1
1130.06
11.18
19.26
Best Instance Cumulative Reward
322.97
317.47
254.17
319.16
325.56
Ant-v3
Best Average Cumulative Reward
5699.31
3652.11
995.59
3108.41
3897.19
Standard Deviation
+234.62
#50.550
12.67
1663.05
1905.42
Best Instance Cumulative Reward
6015.47
4546.47
999.79
4048.27
5080.86
Hopper-v3
Best Average Cumulative Reward
3522.57
3517.57
1713.52
3381.01
3407.15
Standard Deviation
#120.22
#72.37
957. 20
#164.42
113.2286
Best Instance Cumulative Reward
3659.27
3591.42
3573.53
3595.08
3592.56
HalfCheetah-v3
Best Average Cumulative Reward
10818.89
9893.04
5886.75
10440.85
9704.72
Standard Deviation
+174.77
+679.21
+2499.8 822
+415.20
11213.09
Best Instance Cumulative Reward
11044.74
10361.92
9595.08
10737.45
10505.13
Humanoid-v3
Best Average Cumulative Reward
5439.31
5386.21
456.35
5415.61
5443.00 04
Standard Deviation
#152.26
#115.92
#75.22
55338
199.44
Best Instance Cumulative Reward
5685.77
5513.46
529.84
5490.09
5575.66
Walker2d-v3
Best Average Cumulative Reward
4616.46
4278.90
2404.60
4468.58
4693.66
Standard Deviation
+499.26
115.335
+1359.97
+622.17
+:468.90
Best Instance Cumulative Reward
5192.52
4541.51
4618.44
5466.40
5514.77
FORK-S, standing for single-step FORK; call FORK with loss function Equation and w 0.5 FORK- DQ. standing for Double-Q FORK; and FORK with loss function Equation and w = FORK-Q. standing for Q FORK. From T performs the best four out of the six environments and TD3-FORK-S performs the best in the remaining two. This is the reason selected the current form of FORK.
that in terms of best average cumulative reward, TD3-FORK
C BIPEDAL
TD3-FORK-DO can solve the difficult Bipedal SipedalWWlker--aarcoone-- environment with as few as four hours. The hardcore version is much more difficult than BipedalWalker. For example, a known algorithm needs train for days 72 cpu AWS EC2 with 64 worker processes taking the raw frames as input (https: //github. com/ dgrifff 7 ionuuu TD3-FORK-DO, a variation TD3-FORK, can solve the problem in few four hours by using default GPU setting provided by Google Colat? and with sensory data (not images). The performance on WaleeaaaWWleeerrdoorree during and after training can be viewed at hhttp:://youteee wntpe//OOOXXtxhheep The codes have been submitted supplementary materials.
To solve BipedalWalkerrHrdddonn, made several additional changes.
17

---

Under review conference paper at ICLR 2021
TD3-FORK-S
TD3-FORK-Q
TD3-FORK-DQ
TD3-FORK
TD3
400
Avengn Cumuiaaive Rowwed
Avenage Cumulalive Rewward
3500
300
3000
Cumuishie
200
2500
2000
100
3000
1500
2000
1000
1200
0.2M
0.454
0:EM
Timesteps
Timesteps
(h) Bipeda//aalaeeeea Walker//
(c) Ant-v3
(d) Hopper-v3
12000
5000
Cumurilia Roward
10020
Average Cumuuativv Rowward
Averge Cumulabve Rewarrd
4000
8020
4200
8020
3:00
2300
2000
2020
1000
1000
0.7M
0.4M
0.MM Timestaps
0.8M
1M
0.7M
D4M
0:MM
0.8M
1M
0.2M
D:MM
D.BM
0.8M
TM
T Timesteps
Timesteps
(c) HalfChectah-v3
(f) Humanoid-v3
Walker2d-v3
Figure Learning curves FD3- TD3, TD3-FORK-S, TD3-FORK-Q and TD3-FORK-DQ. Curves are smoothed uniformly for visual clarity.
(i) We changed the -100 reward to -5.
(ii) We increased other rewards by a factor of
We implemented a replay buffer where failed episodes, in which the bipedalwalker fell down at the end, and successful episodes are added to the replay-buffe with 5:1 ratio.
The changes to the rewards (i) and (ii) were suggested in the blog Using reward scaling to improve performance has been also reported in (Henderson al.) 2017). We made change (iii) because found failed episodes are more useful for learning than successful ones. The reason we believe is that when the bipidedalwalker already knows how handle terrain, there is need further train using the same type of terrain. When the training is near the end, most of the episodes are successful so adding these successful episodes overwhelm the more useful episodes (failed ones), which slows down the learning.
18

---

Under review conference paper at ICLR 2021
Table Best Average Cumulative Rewards, Standard Deviations, and Best Instance Cumulative Rewards of TD3-FORK, TD3, TD3-FORK-S, TD3-FORK-Q, and TD3-FORK-DQ over Six Environments. The Best Values Bold Text.
Environment
TD3-FORK
TD3
TD3-FORK-S
TD3-FORK-Q
TD3-FORK-DO
BipedalWalkeraa Best Average
317.40
307.69
314.63
302.96
306.44
Standard Deviation
#4.68
#11.18
#3.61
19.17
15.98
Best Instance
322.97
317.47
320.11
313.93
315.55
Ant-v3
Best Average
5699.37
3652.11
5226.37
3905.39
3287. .929
Standard Deviation
+:234.22
1510.50
+748.03
11019.78
2200.10
Best Instance
6015.47
4546.47
5748.94
5563.98
3499.86
Hopper-v3
Best Average
3522.57
3517.57
3605.09
3426.54
3482.14
Standard Deviation
+120.22
+72.37
189.04
1119.51
113.336
Best Instance
3659.27
3591.42
3740.71
3588.89
3677.66
HallCheetah-v3
Best Average
10818.89
9893.04
11077.10
10405.40
9942.23
Standard Deviation
+174.77
+679.21
+337.05
11154.00
=675.52
Best Instance
11044.74
10361.92
11450.83
11524.62
10668.00
Humanoid-v3
Best Average
5439.31
5386.21
5345.92
5255.54
5270.73
Standard Deviation
+152.26
+115.92
+:204.92
+210.65
#996.9
Best Instance
5685.77
5513.46
5706.35
5669.17
5499.36
Walker2d-v3
Best Average
4616.46
4278.90
4089.02
4175.40
4177.34 34
Standard Deviation
1:499.26
+195.35
2:20.61
+-601.8
-372.90
Best Instance
5192.52
4541.51
4450.75
4934.03
4920.74
19


Reviews:
[{"review_id": "lpb-7rl2ND2", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe authors proposed a simple modification to popular off policy algorithms, such as SAC and TD3. By employing a model network and reward network, the authors can expand the bellman update operation using predicted next few steps, similar to the GAE (general advantage estimation). The authors demonstrated that algorithms such as TD3, DDPG, and SAC can all benefit from their approach. \n\n\nPros:\nThe paper is clearly written and easy to understand.\n\nThe concept is simple and the implementation is straight forward. The results indicated that the training sample efficiency and final policy performance of the tested algorithms have improved for 6 benchmark tasks, compared with their \"vanilla\" version baseline. \n\n\nCons:\nWhile it is generally understandable that GAE-like approaches can help balance between bias and variance in the Q-value estimation, I am not convinced if the proposed networks are needed. The authors used two additional networks: the model network (they call \"system network\") and the reward network. The model network computes the standard transition s_t, a_t -> s_t+1, and the reward network estimates the true reward r = R(s_t, a_t, s_t+1). However, as reward function is generally provided, I don't see an additional reward network is necessary here. Second, in off-policy learning, the state transition and next states are already known, instead of using a model network to predict the next states, one can simply sample a small trajectory containing multiple consecutive state-action transitions, and use them to do the Q-value estimation. \n\nRecommendations:\nTo address the concern above, I propose the authors to add two more ablation studies: (1) Remove the reward network and only use the reward function. (2) Remove both the reward network and the model/system network, and use the recorded future states to estimate the Q-values. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "PqCWz-yrPLv", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThis paper focuses on the field of off-policy reinforcement learning. Specifically, the authors propose a model-based reinforcement learning method on top of actor-critic methods. The proposed method trains a dynamics model and a reward function on the off-policy data with supervised learning, and then uses the trained model to generate synthetic future states and rewards during the actor update. During policy update for a given state, the method computes the sum of the Q value estimate of the state and the Q value expansion for a few steps using the learned dynamics model and reward function.\n\nThe authors implement the proposed method on top of SAC, and TD3, and evaluate their performances on several MuJoCo and Box2D environments. The experiment results show that the proposed method outperforms the model-free baseline in terms of sample efficiency.\n\n### Comments\nThe paper is well written and the idea proposed in this paper is really easy to understand. The authors also include a wide suite of experiments to demonstrate the sample efficiency of the proposed method. Despite these advantages, I cannot recommend acceptance of this paper due to the lack of novelty and absence of fair baseline comparison, which I will elaborate on next.\n\nFirst of all, despite the title of the paper, the proposed method is really a model-based reinforcement learning method since the system network and reward network are just dynamics model and reward model. The proposed objective for the policy (eqn 2), is merely a sum of current Q value estimate and Q value expansion for a few steps using the learned model, which has been proposed before in various papers such as [1] and [2]. The only difference is that when computing the gradient with respect to the policy, the authors leave out the gradient that passes through the learned model, which results in biased estimate of the policy gradient. Therefore, I’m not convinced about the novelty of the proposed method.\n\nMoreover, while proposing a model-based method. The authors do not include baseline comparisons with other model-based RL methods. It is widely known that on low-dimensional control tasks, model-based method outperforms model-free methods ([3]), and therefore merely comparing to model-free baselines is unfair. It would be important to include comparisons to model based methods ([3]).\n\nDue to the lack of novelty and fair comparison to existing model-based methods, I cannot recommend acceptance for this paper.\n\n\n\nReferences\n\n[1] Heess, Nicolas, et al. \"Learning continuous control policies by stochastic value gradients.\" Advances in Neural Information Processing Systems. 2015.\n\n[2] Clavera, Ignasi, Yao Fu, and Pieter Abbeel. \"Model-Augmented Actor-Critic: Backpropagating through Paths.\" International Conference on Learning Representations. 2019.\n\n[3] Langlois, Eric, et al. \"Benchmarking model-based reinforcement learning.\" arXiv preprint arXiv:1907.02057 (2019).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "LCIhBQg_UNS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Correctness Issue**: The loss in equation 2 is the sum of the regular actor loss using the critic + an n-step return version of policy value, which would be reasonable for a policy update. However, the expression provided for the gradient (and the calculations in the code) are incorrect in that they are not the gradient of equation 2 with respect to the policy. In particular, it is not reflective of policy performance, as it fails to account for the fact that the future states in the imagined rollout are also functions of the policy. \nThe resulting policy update in FORK actually consists of:\n1. Regular actor update using the critic at the current state\n2. Updating to greedily maximize the reward at each intermediate timestep (which does not reflect the policy performance)\n3. Another regular actor update from the last state in the rollout.\n\nThe straightforward way of correcting this to make it optimize the loss would be to simply differentiate through the model, which may not work well as differentiating through learned models often leads to instability (though it may be fine for such short rollouts). Alternatively, a REINFORCE estimator can instead be used for the gradient of the n-step term for stochastic policies. \n\nWe note that the loss in eq. 6 (if we keep the dependence of the future state on the policy) is also not reflective of policy returns (since it overweights future returns over the present), but can provide reasonable interpretation of the FORK update if we drop the dependence of the future state on the policy as the authors do. \nThe gradient of the FORK-Q variant the authors take from eq.6 is the actually the update we would obtain by using the regular actor update (the $\\nabla_{\\theta} \\pi_{\\theta}(s) Q(s, \\pi(s))$ term for deterministic policies) but under a different state distribution. Instead of using the state distribution in the replay buffer, the update takes into account the distribution obtained by running the policy for a few steps starting from the buffer distribution. As running the policy from the buffer distribution would result in a distribution closer to the on-policy state distribution, one explanation for why the FORK-Q update would improve is that the gradient is closer to the on-policy policy gradients by changing the state distribution. \n\nThe same reasoning can be applied to primary FORK update presented, as it includes an actor update on the value in future states (3rd term in my previous list). The issue is with the greedy reward maximization in the inner steps (2nd term), but maybe it simply doesn't hurt on the environments tested or it perhaps takes advantage of a bias-variance tradeoff in greedily maximizing immediate reward for a few intermediate timesteps. I would like the authors to explicitly address these issues in the paper and present a clear explanation of why their FORK should be a better policy update.\n\n**Relation with Model Based RL Methods**: Overall, I also disagree with the authors' claims that FORK is very different and much simpler than other model based algorithms. With regards to their comment on how their method is somehow simpler? than rollout based methods, their policy update is using a short Monte-Carlo rollout for estimating the policy gradients; the difference being that the rollouts are being used to update a policy rather than to explicitly plan at test time. \n\nThe authors' claim that FORK does not require high-fidelity simulation seems unsupported to me, and the claim that model-based RL algorithms use the model in a sophisticated way is vague. In particular, Dyna-style algorithms (like STEVE https://arxiv.org/abs/1807.01675, MBPO https://arxiv.org/abs/1906.08253) which use the model to generate experience to help learn the critic, seem to be using the model in the same way as FORK (in the sense that they only use the model to generate samples with short rollouts). There is also no discussion of methods like ME-TRPO https://arxiv.org/abs/1802.10592, SLBO https://arxiv.org/abs/1807.03858, or the algorithms in https://arxiv.org/abs/2004.07804, all of which only the model to generate trajectories to use with a policy gradient algorithm. Overall, I would appreciate much more discussion about how FORK relates to and compares empirically against past RL algorithms that only use the model to generate samples, as well as clarifying the statements about FORK uses the model in a less sophisticated way.\n\nOn a separate note, using the model to generate n-step-return estimates of the policy value has been previously done in https://arxiv.org/abs/1807.01675 for example. The key difference is that prior work used it to generate target values for learning the Q-function, while here it is used purely for policy updates. Given how similarly the models are used however, I would recommend discussing this line of work explicitly in the related work, even though they are complementary. \n\n**Experimental Evaluation**: Despite the aforementioned correctness issue, the method seems to provide improvements when applied to TD3, and seemingly smaller improvements on top of SAC. However, I find it extremely strange that the authors chose in Figure 4 to plot returns against the number of training steps instead of the number of samples. As acknowledged by the authors, this makes the SAC vs TD3 comparisons incomparable, with TD3 and TD3-FORk enjoying the advantage of having seen twice as many samples. Moreover, comparing only on the number of actor/critic updates isn't even a fair comparison between FORK and baselines, as FORK additionally has to train a dynamics model and reward predictor. I would highly recommend simply showing learning curves with respect to the actual number of environment samples, rather than arbitrarily using the number of actor updates.\n\nI would also like to see comparisons against model based RL baselines, particularly MBPO, which uses a Dyna-style update with SAC to compare which method of utilizing the model is better. The authors could also see if the FORK actor update further improves upon MBPO or other model based RL methods.\n\n**Summary**:  As it stands, I believe the paper should be rejected due to the correctness issues (and resulting lack of justification for why FORK should give better actor updates), and insufficient discussion of how it relates to prior in model based RL. To consider accepting the paper, I would at least need to see these issues addressed by the authors.\n\nRegarding novelty and significance, using a model to predict n-step value estimates (as the paper claims to be doing) or using the model to explicitly adjust the state distribution of the policy update (as I suspect this might be doing instead) for the policy update in an off-policy actor critic algorithm has not been done before as far as I know. However, this change in how the model is used seems fairly minor, and to be convinced it were useful, I would like to see evidence of how it compares against the other model based RL algorithms. In particular, the benefit I imagine it might have over other model based RL methods is in being more robust to poorly fit models, but I would need to see empirical evidence supporting this.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "E1WWYvF7wvh", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to combine ideas from model-based RL into model-free off-policy policy gradient algorithms (like SAC and TD3). Specifically, the paper proposes to learn auxiliary models of environment rewards and dynamics and use a two-step rollout from these models during the computation of the policy gradient. The paper presents results of this mechanism applied to standard SAC and TD3 implementations on a variety of continuous control environments with favorable results.\n\nStrengths:\n\n-- The story is generally easy to follow.\n\n-- I appreciate that the authors didn't just evaluate on the extremely common (and somewhat saturated) MuJoCo benchmarks, and presented additional results on Bipedal Walker.\n\n-- As far as I can tell, the policy update proposed here is novel.\n\nWeaknesses:\n\n-- While the policy update appears novel, it is very similar to techniques in the model-based RL literature. Given this, it would greatly improve the paper if it had appropriate comparisons to similar model-based techniques, especially those which claim to combine model-free updates with model-based techniques; for example https://arxiv.org/abs/1906.08253\n\n-- Moreover, experimentally it would be nice to see comparisons to state-of-the-art model based RL methods. For example, https://arxiv.org/abs/1802.10592\n\n-- In terms of the current experiment results, I found the conclusions favorable to the proposed technique, but not a very compelling demonstration. For example, in Table 1, almost all the environments produce only a *slight* benefit for the proposed method. It appears the only significant benefit is on Ant. Similarly in Table 2, we see the results of SAC-FORK are sometimes much worse than SAC on its own.\n\n-- In terms of motivation for the method, I was not entirely convinced of why the proposed update is needed. The paper appeals to the idea of needing to reason about values in the future. But shouldn't the Q-value already encapsulate this? Moreover, the proposed update ends up only optimizing *actions* in the future, rather than somehow reasoning about the values at steps t+1, t+2 to decide the best action at step t.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agreed that the novelty of the method was not at the level expected for publication, and also raised a number of technical concerns regarding the approach. There was no response from the authors on these issues, hence the reviewer consensus is that the paper is not ready for publication at this time.
Decision: Reject