Solving Complex Manipulation Tasks with Model-Assisted Model-Free Reinforcement Learning
Authors: Jianshu Hu
Paul Weng
Venue: CoRL 2022 Poster (2022)
Keywords: Reinforcement learning
Data augmentation
Imaginary exploration
Optimistic initialization

Abstract: In this paper, we propose a novel deep reinforcement learning approach for improving the sample efficiency of a model-free actor-critic method by using a learned model to encourage exploration. The basic idea consists in generating artificial transitions with noisy actions, which can be used to update the critic. To counteract the model bias, we introduce a high initialization for the critic and two filters for the artificial transitions. Finally, we evaluate our approach with the TD3 algorithm on different robotic tasks and demonstrate that it achieves a better performance with higher sample efficiency than several other model-based and model-free methods.

Solving Complex Manipulation Tasks with Model-Assisted Model-Free Reinforcement Learning
Jianshu Hu
Paul Weng
UM-SITU Joint Institute Shanghai Jiao Tong University sjs1998@sjtu. edu. cn
UM-SITU Joint Institute Shanghai Jiao Tong University paul. wengas www.u/sss..uuu.....m cn
Abstract: In this paper, we propose novel deep reinforcement learning approach for improving the sample efficiency of model-free actor-critic method by using learned model to encourage exploration. The basic idea consists in generating imaginary transitions with noisy actions, which can be used to update the critic. To counteract the model bias, we introduce a high initialization for the critic and two filters for the imaginary transitions. Finally, the TD3 algorithm on different robotie tasks and demonstrate that achieves a better performance with higher sample efficiency than several other model-based and model-free methods.
evaluate our approach with
Reinforcement learning, Data augmentation, Imaginary exploration, Optimistic initialization
Keywords:
Introduction
Deep reinforcement learning (DRL) has shown its potential in solving difficult robotic tasks espe- cially when facing complex dynamics and contact-rich environment. For instance, dexterous ma- nipulation tasks like rotating a cube or ball to desired position and orientation
be solved with state-of-the-ar model-free methods [1], model-based methods [2, 3]. However, the inherent issues related the stability and sample efficiency of DRL algorithms still make them challenging to ap- ply on real robots. To tackle those issues, various approaches have been investigated. In this paper, specifically investigate how learned model can help accelerate model-free method. Previous research in this direction (see Section 2) used a learned model for data augmentation [4], to improve critic estimation [5], for gradient computation [6], or for guiding exploration in the true environment [7].
In contrast to previous model-based methods, propose to "explore" inside the learned model, which is arguably safer than in the real environment. the learned model, noisy actions are exe- cuted in states visited in the true environment to obtain imaginary transitions, which can be used to compute imaginary target Q-values for updating the Q-function of the current policy. With per- feet transition model, this approach would help accelerate learning the Q-function. However, since both the learned model and the estimated Q-function may be incorrect, directly using those imagi- nary transitions may lead potential issues. To counteract them, we introduce several techniques: high Q-value initialization and filtering of imaginary transitions to favor optimistic targets that are uncertain.
Intuitively, with a higher initialization, we approximate the "optimism in face of uncertainty" prin- ciple when evaluating random actions. In the spirit of this principle, only keep those that lead to higher evaluations than real transitions. Moreover, to avoid unnecessary updates, those imaginary transitions are selected with higher probability if there is larger uncertainty in the corresponding Q-values. Note that since the noisy actions are performed in the learned model, our model is not true exploration strategy. However, if such imaginary targets indeed used to update the Q- function, these can lead to higher evaluations of the corresponding actions, which would later favor
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

selecting them in the true environment. In this paper, we implement those techniques in the TD3 algorithm (see Section 3) [8] However, they may be beneficial in other DRL algorithms as well.
We propose a novel approach to exploit a learned model in DRL (see Section 4). We validate the proposed method and show that it outperforms relevant state-of-the-art algorithms in MuJoCo [9] robot environments and especially in some complex manipulation tasks (see Section 5) 1. Moreover, we analyze and discuss the different proposed techniques.
Contributions:
2 Related Work
Researchers have explored various methods for combining a learned model with model-free rein- forcement learning algorithms. They can mainly be divided into four categories: (1) model-based data augmentation, (2) model-based value estimation, (3) analytic gradient calculation, and (4) model-guided exploration.
Dyna [4] is typical architecture for learning dynamics model with the true experience and using the dynamics model to generate imaginary data for training a value function and a policy. Gu et al. [10] introduce a local linear model for representing the dynamics and show the improvement of using the imaginary rollouts with this model. ME-TRPO [1]] is method leveraging an ensemble dynamics model in model-free reinforcement learning algorithm TRPO [12]. Another work, MA- BDDPG [13], aims to alleviate the effects of model bias by considering uncertainty when using imaginary transitions stored in a replay buffer. The uncertainty of imaginary transitions is measured as the variance of an ensemble of critics. In contrast to this method, we use the disagreement between the two target crities in TD3 [8] as an uncertainty measure and generate imaginary transitions online instead of maintaining another imaginary replay buffer.
Focusing improving the estimation of the Q function, Feinberg et al. [5] propose method called MVE which uses the observed states and a learned dynamics model to simulate for a fixed horizon with current policy. With this imaginary segment, better target Q-value (value expansion, which is a variant of n-step return) is applied in the training of the critic. Instead of using the dynamics model to forward for several steps, our method simulates only one step (to reduce the impact of the model error) but not directly with the current policy. The performance of MVE is sensitive to difficult-tosect hyperparameterr the simulation horizon, which is limited by the quality of the learned model. To overcome this problem. Buckman et al. [14] use weighted sum of value expansions from different horizons and different models. They learn ensemble models to approximate the transition function, reward function, and Q-function. Weights are then assigned to the value estimations of different prediction horizons according to the variance from those models. To ensure a monotonic improvement under the model bias, Janner et al. [15] provided theoretical analysis on how to decide the simulation horizon.
Instead of using the learned model as an environment simulator, researchers have also exploited its differentiabilitt. Deisenroth and Rasmussen [16] propose method called PILCO to learn probabilistic dynamics model and use it in policy search by calculating analytical gradient of the object with respect the policy parameters. Clavera al. [6] further extend the idea and train bootstrap ensemble probabilistic dynamics model.
As for guiding the exploration, a natural idea is to try to explore more in regions where the learned model is uncertain. For instance, Pathak et al. [17] formulate the disagreement across an ensemble model an intrinsic reward. Another example is the work of Shyam al. [7] in which they measure the novelty of state-action pairs with a learned model and use this novelty as the objective of an exploration Markov Decision Process (MDP) to find an exploratory policy. In contrast these methods, we influence the exploration in the true environment by trying noisy actions in the learned model and setting a high initialization for the critics.
Background
A Markov Decision Process (MDP) is composed of set of state S, set of action A, transition function : S A P(S) (with P(S) denoting the set of probability distributions over S), a
'Code is available this link

---

reward function r SS - R, and a distribution over initial state μ E P(S). Given a deterministic policy x S - A. the value function V (*(s) E E X'r, So s) is defined as the expected discounted cumulative reward an agent will receive starting from a feasible state and following the policy. The discounted factor is defined as 7 E [0, 1]. To solve this MDP is to find an optimal policy x² that maximizes the expected value function n -T"(s) arg max, R, [V* (s) 8 n4). To find this optimal policy, we often need an action-value function (= a) = Ex, [m a) + +>V"(s') which corresponds to the expected discounted sum of rewards obtained by executing action in state and acting according to policy π thereafter. Since we focus on robotic tasks, we only consider deterministic MDPs in this work, although our approach could certainly be applied in stochastic settings as well.
Deep Deterministic Policy Gradient (DDPG) DDPG [18] is DRL algorithm with an actor- critic structure for solving an MDP with continuous state and action spaces. The policy π (actor) and its Q-function Q (s, a) (critic) are approximated by neural networks parameterized by 0 and respectively. In DDPG, the actor interacts with the environment generating transitions that are stored in replay buffer. At each training step, mini-batch {(s, as. F', 81+1) i 1,..., .N) is sampled from the replay buffer and used for updating the actor's parameters according to the deterministic policy gradient:
Vol(T)= 1
N E VaQ(sn, Φ) la=n(s;10)) VOT(s; 10),
(1)
while the critic's parameters @ are updated minimize the following loss function:
((Q)
Σ (Q(s,,a 1 Φ) N
1
y1) -
(2)
where y is the target Q-value for the sampled transition (st, ai ri, 8141):
y: = + 70 (0( x (8t+1 (8)) 10))
+
(3)
To improve the learning stability, the target Q-value is calculated with target Q function Q(+ d') and a target actor T--- 0¹). The target networks (ø', 0') are initialized to the same parameters as the original networks (ø, A), but then updated slowly towards the original ones with ø' + (1 T) + TO and 8' < T)0' + T8, where T E (0, 1) is a hyperparameter.
Twin Delayed DDPG (TD3) TD3 [8] improves DDPG with three tricks: clipped double-Q learn- ing, target policy smoothing, and delayed policy update. The first trick aims to prevent overes- timation by learning two critics Q(., Φ1). Q( Φ2) (with their corresponding target critics Q(--- o1). Q(-,- (2)). When calculating the target Q-value, the target critic with smaller value used for training. The second trick aims to smooth the objective funetion by injecting a truncated Gaussian noise é to the output of the target policy. Note that the perturbed actions are clipped to ensure they remain feasible. For legibility, we do not write this step in equations. With these two tricks, the target Q- value becomes:
y/4 + y min (Q(#:+1, 7( 0¹) + d1)), (S041,
00) + (b))
(4)
Lastly, the delayed policy update enforces that the actor be updated at a lower frequency than the critic. With those tricks, it has been empirically observed that the performance and stability of learning the Q-function are substantially improved.
4 Methodology
Our proposed algorithm (see Algorithm 1) extends TD3 to use a learned model to enhance the update of its critics. Note that while the dynamics model is not known, we assume that the reward function known. This assumption is natural in robotics since the reward function is defined by the system designer guide the learning of the robot.
The whole training process can be divided into two phases. Phase (time step h)), transitions collected by interacting with the environment using random policy. No learning occurs up to
3

---

Algorithm MAMF
1: initialize critics Qss, Φ1), Q(s, a Φ2)
2: initialize actor (s 0). model I and empty replay buffer R
3: set the parameters of targets of Φ1, 02 t 22 and 0' +- 0.
4: initialize 1) and p
5: start with initial state so μ
6: for t 0... do
7: generate transition with random policy and keep track of the max reward y*
8: save transition (St, dz Tt, $t+1) in replay buffer R
9: if episode ends then reset the environment 81+1 μ end if
10: t> ho then update ff with {ss, Og, Ft, 81+1]
,, sampled from R end if
11: end for
12: use max reward r" to set the bias in the critics according to Equation (6).
13: start with initial state 8p J
14: for t. h + 1...., H do
interact with the environment with current policy and small noise save transition (st, az, ft,, Sr+1) to replay buffer R
16:
if episode ends then reset the environment 8 f end if
18:
sample {(5), Gg. ry. 85+1) ) j -,.... .N} from R and train f sample {(sk, ag, Tk: 8k+1)
......N N] from R and create imaginary transitions
20:
<- X P
filter the imaginary transitions with the two filters update critic according to Equation (10)
22:
23:
if t mod policy_des _eloy = then
24:
update actor
25:
update target networks end if
26:
27:
end for
time step ho after which the dynamics model starts be trained. Data from Phase is also used to set a high initialization for the critics. During Phase 2 (time step h + 1 to H), transitions are now generated by the actor and both actor and critic start to be trained. The dynamics model further updated online and used for creating imaginary transitions with noisy actions.
The goal in our approach is train the crities with those imaginary transitions, which either improve the Q-estimation or guide the exploration such that potentially promising actions could be tried in the true environment. To that aim, we apply filters the imaginary transitions, since using any imaginary transitions is generally detrimental due to model bias and Q-value estimation error. The first filter keeps the imaginary transitions with higher target Q-values. The second filter selects imaginary transitions whose Q-values are uncertain with higher probability. With all these components, we propose the model-assisted model-free (MAMF) algorithm (see Algorithm 1). To ensure the reproducibility of our results, the source code of MAMF will be released after publication.
Next explain how learn the dynamics model, how implement the high initialization, how we perform exploration in the learned model, and how we filter the imaginary transitions.
Learning the Dynamics Model. Our algorithm starts by training dynamics model M : S - with samples generated by a random policy (e.g.. which uniformly randomly selects actions). Before starting updating M, ho transitions are first collected. Then, M is repeatedly updated via stochastic mini-batch gradient descent until / transitions have been generated by the random policy, which ends Phase 1. After that, the model is still updated with the same procedure, but the transitions are now generated by the current policy. Online DRL training is performed until H transitions have been generated. The model is (pre)trained by minimizing a mean -quared error between predicted next states and true next states computed over a mini-batch ((3), a4 Ti, S141)
N.::
(f(st) (()
LM
(5)
N

---

High Initialization of Critics. The crities are initialized to optimistic values by changing the bias term in the last layer of the critic networks. This bias set as follows:
b= == (r" x () X c
(6)
where r" is the reward observed during Phase 1, is the max episode length, and [0, 1] is hyperparameter controlling how optimistic the high initialization is.
Exploration with Learned Model. In the ceitrrcritic architecture of TD3, the critic is trained us- ing transitions {(s,, a,, F,, $1+1) I i 1, ,, ,. sampled from the replay buffer. For the same set of states {&; =, .,, ,N), noise from a truncated normal distribution
clip(N((0, 1), -7 n) with large clipping bound 77 is applied on the policy action (s; 0) to generate imaginary tran- sitions. The bound is initialized as the max action amax the agent can take in each dimension. After each iteration, the bound for truncating the Gaussian noise exponentially decays with fixed decaying rate Formally, the imaginary transitions (st, a,, $,+1) generated by the learned model and known reward function r are written:
à. clip(p((s (s; 0) + v, mmax mmax),
8141 f(1 (s)),d))
A (=),,,;
Filtering. To only use the imaginary transitions whose actions are potentially better than the origi- nal ones, these transitions are filtered by their target Q-values. Only the imaginary transitions whose target Q-values are higher than the original ones are considered:
(ft,, 8,+1 0', &1 Φ22 yi(rs, 81+1 10'001.. 02)
where i and y; are calculated by Equation (4)
The remaining imaginary transitions are filtered with respect to the uncertainty in the Q-value esti- mates. This uncertainty be measured as the disagreement of the two target critics in TD3, which can be expressed as the absolute value of the difference of the target Q-values given by the two target critic networks:
A = (0($:1I.T$::: + + (8)) o1) - Q($++1 (8.41 + e (*) Φ2)1.
This difference is then normalized by the max difference observed far in the sampled transitions to obtain a ratio P max E [0. 1]. With probability P. an imaginary transition is selected for training the critics, otherwise it is dropped. Therefore, imaginary transitions with larger uncertainty have a higher chance to be kept. The number of remaining imaginary transitions after the two filters is denoted by
Finally, the loss for the crities composed of two parts, the original loss and an additional loss obtained from the filtered imaginary transitions:
Lg =
M (Q(s., --)) Σ (Q(s,, à, 9.)²
(10)
where the target value is calculated with Equation (4) and the target values ÿ; for imaginary transitions follow almost the same calculation but with the action, next state, and reward substituted by their imaginary counterparts.
5 Experimental Results
Our experiments are designed to answer the following questions:
Do our propositions improve the performance with respect relevant baselines?
Does each component of our method (i.c., decaying noise, high initialization, filtering) contribute to its performance?
In which aspects does our method help?
Before answering those questions, we present next the experimental environments and discuss the experimental set-up used.
5

---

(a) EggCatchUnderam
(b) EggCatchOverarm
(c) PenSpin
Figure 1: Visualization of three environments in Dexterous Gym
5.1 Environments and Experimental Set-Up
We evaluated our approach on several robot environments in MuloCo [9] and three complex manip- ulation tasks proposed in Dexterous Gym [19]. The visualizations of these manipulation tasks shown in Figure 1. The task is throwing and catching an object at desired position and orientation with two dexterous manipulation hands in EggCatch and EggCatchOverarm. The task of PenSpin is rotating a pen without dropping it. The details of these environments are described in Appendix A. We chose these three representative environments from Dexterous Gym [19]. The other environments like BlockCatchUnderumg are simply variants with different objects.
Across all the environments, all the hyperparameters such learning rates, batch size and network architectures for the actor and critic are kept the same except for the hyperparameters related to setting the bias. The hyperparameters and brief discussion about setting them shown in Ap- pendix B. All the experiments are performed over five different random seeds and the performance measured by the sum of rewards averaged over 10 episodes.
5.2 Results
5.3 Do our propositions improve the performance w.r. relevant baselines?
To answer this question, we evaluate our method on the environments introduced above. We com- pare with the following baselines: (1) TD3, (2) MVE-TD3: MVE with TD3, and (3) MA-TD3: MA-BDDPG with TD3. In Appendix C, we present some additional experiments where we com- pare and integrate approach with the state-of-thes art model-free RL method called SAC [20], which encourages exploration by incorporating an entropy term.
Both MVE [5] and MA-BDDPG [13] are methods that use a dynamics model to generate imaginary transitions for training the critics. Since their source codes are not publicly available, we imple- mented those methods in TD3 for consistency. For MVE, we set the prediction horizon to 3 which is applicable in all the environments. For MA-BDDPG, we actually implemented a version similar to it, which corresponds to TD3 with only imaginary transitions filtered by uncertainty. It is equiva- lent to our method without high initialization, decaying noise and the filter favoring higher target Q values. Similar to the original method proposed by Charlesworth and Montana [19], demonstrations are also used in all the experiments in Dexterous Gym.
The training results are shown in Figure 2. We can see the outperformance of our method across different environments. Especially complex environments like EngCaach our method achieves a much higher final return and has low variability across different runs. We also notice that combining MVE with TD3 can not guarantee an improvement in some environments. This might be due to the target Q smoothing in TD3 (clipped noise added to the action when calculating the target Q-values). Similar results of MVE can also be observed in the work by Buckman et al. [14]. Their results also show that MVE does not guarantee improvement when applied on DDPG [18].
5.4 Does each component of our method contribute to its performanee?
To prove the significance of each component in our method, we performed an ablation study. By comparing the results shown in Figure 3(a)-(b), we can find that high initialization is necessary to counteract the effect of the model bias and encourage the exploration at the beginning. Using decaying noise and choosing imaginary transitions with higher target Q-values achieve the ideas of both seeking optimistic state-action during training and of exploring in the model. The filter with uncertainty guides the learning of the Q-function to emphasize on uncertain part and thus helps the estimation of Q-vlluss. All the components in synergy improve the training of the critic.

---

-20
Hovir Sourd)
-25
2000
752
-35
703 MISMOonn
-55
-65
$1e0leeu0000
(a) HalfCheetah
(b) Hopper
(c) Pusher
5333
2000
wetrour)
000
200
300
Sradecos500
(d) Reacher
(c) Swimmer
(f) Walker
1300
Jounn
INU-TE3
HAUTTD
155-720
1000
502-
1990
1090
1750
3200
1750
(g) Undderrn
(h) FggCaathOveeaamm
PenSpin
Figure 2: Results of training in MuJoCo environments and Dexterous Gym
5.5 In which aspects does our method help?
We conjecture that our method helps in three ways: better exploration, better estimation of Q-values, and exploitation of independence among action components. For the first point, we believe that the higher final performance of our method compared to the baselines provides some evidence of the improved exploration.
For the second point, we compare our method with a variant of TD3 where critics are updated more frequently. This variant could achieve better Q-estimations and thus better performances in some environments. Due to the page limit, the results and details for the experiments are shown in Appendix C. We can see that simply updating the critic more often is still not sufficient to achieve the performance gains of our method.
For the last point, we believe that our method also exploits potential action independence (i.c., in some states, parts of the actions do not have impact on the next environment state). This kind of independence is common in robotic tasks e.g., in a walking robot, the actions of any limb that does not touch the ground may have less effects on the next states, or more concretely. in EggCatchUn- derarm, once hand has thrown a ball, the actions of this hand is not important anymore. To test this conjecture, we augment HalfCheetah by doubling its action space A to an enlarged action space A x A. The transitions in this augmented environment depend only on the first or second part of the enlarged actions. This dependence switches from one part to the other every H time steps. Following this pattern, part of the actions has no direct effeec.. The results of training in this arti-
7

---

HOSTE3
30
50
HALTO3
T25-9.00
135.60 6-306
VARE4-200
566 150 1900 1750 1500 1756
2000
250
390 1800
1356
1752
2000
520 600
1PAS9aps.0000
(a) EggCarchUnderarm
(b) EggCatchOverarm
(c) HallCheetah
Figure 3: Figures (a) and (b) show the ablation study. Figure (c) show the results of training in the environments defined in Section 5.5
ficial environment with different frequencies of switching are shown in Figure 3(c). Although the augmented environment has a larger action space, method can still improve the performance.
6 Limitations
The limitations of our work mainly include these three aspects:
(I) Using too large too small bias in the critic will hurt the performance of our method. Although our heuristic for setting the bias quite robust (except HalfCheetah), it is not clear how to analytically determine it. A more theoretical analysis may be required for applying it in more general situation.
(2) While much work has investigated exploration strategies in model-free algorithms, in our method, a simple truncated Gaussian noise applied generate imaginary transitions. more sophisticated way of choosing exploratory actions might further improve our approach.
Although our proposed method significantly improves the sample efficiency and the performance, especially on the harder tasks of Dexterous Gym, we believe that more efforts may still be needed to fully the algorithm in real robot. Potential avenues make the approach even more practical could be to use it in combination with sim2real methods [21, 22], exploit any extra a priori known information such as symmetries [23], or exploit the learned model in some other ways [24] for instance.
7 Conclusion
We propose method called MAME, which leverages dynamics model to help train the critic in model-free algorithm. One key novelty is to generate imaginary data with exploratory actions. Our experiments demonstrate the sample efficiency and performance improvements in some complex manipulation tasks. We believe that our method is step forward towards making DRL practical on real robots and shows a promising way of solving those complex tasks. Although proposition is implemented with TD3, it is actually independent from this DRL algorithm. Thus, one interesting future work would be to adapt and combine our proposition with other model-free algorithms.
this method even

---

Acknowledgments
This work is supported in part by the program of National Natural Science Foundation of China (No. 62176154) and the program of the Shanghai NSF (No. 19ZR1426700).
References
[1] OpenAI, I. Akkaya, M. Andrychowicz. M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, J. Schneider, N. Tezak, J. Tworek, P. Welinder, L. Weng, Q. Yuan, W. Zaremba, and L. Zhang. Solving rubik's cube with a robot hand. CoRR, abs/1910.07113, 2019. URL
071133
[2] K. Lowrey, A. Rajeswaran, S. M. Kakade, E. Todorov, and I. Mordatch. Plan online, learn offline: Efficient learning and exploration via model-based control. CoRR, abs/181 2811,01844
2018. URL http: /tt////aavvvv..e/e////eeeeeeaev/ org/ ha////bs/11111 01848.
[3] A. Nagabandi, K. Konolige, S. Levine, and V. Kumar. Deep dynamics models for learning dex- terous manipulation. CoRR, abs/1909.11652, 2019. URL http:// //arxiv.//xii..//////e/e org/ abs/ /tss//99.. 1909. 11652.
[4] R. S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting. SIGART Bull., 264):160-163, jul 1991. ISSN 0163-5719. 0o1:10.114///2334411277. URL https: 101111//222/4 122377.
[5] V. Feinberg, A. Wan, I. Stoica, M. I. Jordan, J. E. Gonzalez, and S. Levine. Model-based value estimation for efficient model-free reinforcement learning. CoRR, abs/1803.00101, 2018. URL http://arxiv. org/ abs 1803.0030000 00101.
[6] I. Clavera, V. Fu, and P. Abbeel. Model-augmented actor-critic: Backpropagating through paths. CoRR, bbs//2005000068, 2020. URL https://arxiv: wwtgg//////aaaaeeeeeeeeee 08068.
[7] P. Shyam, W. Jaskowski, and F. Gomez. Model-based active exploration. 121/18 2018. URL. http:///axxv..orrabbbaess/ 12161.. 12162.
CoRR,
[8] Fujimoto, H. van Hoof, and D. Meger. Addressing function approximation error in actor- critic methods. CoRR, Abs/1802.09477, 2018. http://xxx/ wwtps///.g/abbbb///// URL. 09477.
[9] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026- 5033, 2012. N00.1110//ROS.2022...6
[10] S. Gu, T. Lillicrap, Sutskever, and S. Levine. Continuous deep q-learning with model- based acceleration. CoRR, abs/1603.000748, 2016. URL http://arxiv, org. ////////////// 1603. 00748.
[11] T. Kurutach, I. Clavera, Y. Duan, A. Tamar, and P. Abbeel. Model-ensemble trust-region optimization. CoRR, abs/1802. 10592, 2018. URL http:://aaxii..tg/////////////// haaggg/aa//////
wrwp:///..rr/aaa//...... 10592.
[12] Schulman, S. Levine, P. Moritz, M. I. Jordan, and P. Abbeel. Trust region policy optimiza- tion. CoRR, abs/1502.05477, 2015, URL http: http:///www...rrrra//////2/00000
[13] G. Kalweit and J. Boedecker. Uncertainty-driven imagination for continuous deep reinforce- ment learning- In S. Levine, V. Vanhoucke, and Goldberg, editors, Proceedings of the 1st Annual Conference on Robot Learning, volume 78 of Proceedings of Machine Learning Research, pages 195-206. PMLR, 13-15 Nov 2017. URL //proee////oooeiingg...... mlr.
press/v78/kalvvit177.. html.
[14] Buckman, D. Hafner, G. Tucker, E. Brevdo, and H. Lee. Sample-efficient reinforcement learning with stochastic ensemble value expansion. CoRR, abs/1807.01675, 2018. URL http: //arziv. org/ abs 1800/ 01675.
[15] M. Janner, J. Fu, M. Zhang, and Levine. When to trust your model: Model-based policy optimization. CoRR, abs/1906.08253, 2019. URL http: / arxiv. .rg. wwtgg/bs/1990..000 08253.

---

[16] M. P. Deisenroth and C. E. Rasmussen. Pilco: A model-baset and data data-efficient approach to policy search. In Proceedings of the 28th International Conference on International Confer- ence on Machine Learning, ICMI'11, page 465-472, Madison, WI, USA, 2011. Omnipress. ISBN 9781450306195.
[17] D. Pathak, D. Gandhi, and A. Gupta. Self-supervised exploration via disagreement. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Con- ference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5062-5071. PMLR, 09-15 Jun 2019. URL https://proceeedngg.. mlr ptt.ee/nng........s////////// press/v97/ pathak19a 19aaak19a... html.
[18] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. In ICLR (Poster), 2016. URL http:
// //arxiv. org/ abs/1509.022
[19] H. Charlesworth and G. Montana. Solving challenging dexterous manipulation tasks with trajectory optimisation and reinforcement learning. CoRR, abs/2009.05104, 2020. URL
https://arxiv.. orgo ab09 05104.
[20] T. Haarnjja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. CoRR, abs//801.01290, 2018. URL http: //arxiv. org/ abs/ 1101/ 01290.
[21] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. CoRR, abs/ abs/1703.06907, 2017. URL http://arxiv. org/ abaa /10.. 06907.
[22] OpenAI, M. Andrychowicz, Baker, M. Chociej, R. Józefowicz, B. McGrew, Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder,
L. Weng, and W. Zaremba. Learning dexterous in-hand manipulation. CoRR, abs/1 880.00077.0
2018. URL http: / /arxivv.... org/ abs/ 110//88888.. 0011/1088800007 00177.
[23] Y. Lin, J. Huang, M. Zimmer, Y. Guan, J. Rojas, and P. Weng. Invariant transform experience replay: Data augmentation for deep reinforcement learning. IEEE Robotics and Automation Letters, 5(4):6615-6622, 2020. doi:
[24] P. F. Christiano, Z. Shah, I. Mordatch, J. Schneider, T. Blackwell, J. Tobin, P. Abbeel, and W. Zaremba. Transfer from simulation to real world through learning deep inverse dynamics model. CoRR, abs/1610.003118 2016. URL http:/ www.vv.org/abs/eeeee
10


Reviews:
[{"review_id": "yFeVrU_Vhio8", "reviewer": "Reviewer_jm78", "summary": "The paper presents an algorithm that combines model based and model free RL for control. Specifically the authors propose a learning framework where a learned model is used, after being pre-trained, to roll-out imaginary transitions that are used to update a critic in order to eventually facilitate exploration in the real environment to solve a task. The authors present experimental evaluation of the proposed algorithms on a variety of simulated environments and compares to a number of baselines. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: The paper is written clearly and is easy to follow, the algorithm is also presented well and it would be in principle possible to reimplement it. The direction of combining model based and model free is relevant and interesting and looking at the model as a possible avenue for doing exploration is also an interesting direction. The results overall seem promising and show that the method works. \n\nWeaknesses: I am not super convinced by the results, compared to TD3 there might not be a clear benefit as the model was also already pre-trained. The final performance is mostly comparable to TD3. I would not be so worried about this itself, however since you don't show any transfer/generalisation experiments learning a model without reusing it, where TD3 actually also performs well, seems a bit of an overhead to me.  \n\nI think the paper would be very strong if you could show that you can reuse the model across tasks, or are you doing this already but did not make clear in the paper? \n\nI think that for that it would also be important to clarify the pre-training of the model more, and if you pre-trained for each experiment a new model. \n\n- robot experiments are missing \n\nQuestions:\n\n- could you give a better intuition or understanding of why the initialisation of the bias help? Also how did you come up with this equation, what why were you expecting this would help? \n\n- How much data did you use to pre-train and how close is this data to the experimental domain?\n\n- Why did you not compare to a purely model based RL algorithm like this one https://arxiv.org/pdf/1912.01603.pdf ?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "There are a couple of questions that need clarification but I'm willing to adjust my assessment during the authors respond period.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "see above ", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "VHO1ioRJF9", "reviewer": "Reviewer_4nCH", "summary": "This paper presents a deep reinforcement learning approach to improve the sample efficiency of an actor-critic method. It adds random noise to action values to create imaginary transitions. To address the model bias from using the transitions, it introduces high Q-value initialization of critics and two filtering methods that favor transitions with high target Q-values and high uncertainty in the Q-value estimates. They compare their method with different variants of TD3.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n* This paper presents several interesting tricks to counteract the model bias of using imaginary transitions. \n* The paper is well structured and written.\n* Figures are clear and useful for understanding the results.\n\n**Weaknesses:**\n* Since the proposed method pre-trains a dynamics model, it is unfair not to show the pre-training steps in the experimental results (Figures 2 and 3) when compared to the baselines. The proposed method's sample efficiency could be overestimated as the number of pre-training steps is not taken into account.\n* The experimental results in Figure 2 will look more convincing if non-TD3 algorithms are used for comparison, such as SAC. Figure 2 looks more like an ablation than a baselines comparison since TD3 and MA-TD3 are implementations without some of the components in your method. \n* Lack of real robot experiments.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper presents an interesting method to improve the sample efficiency of an actor-critic method. However, two issues need to be addressed. First, the proposed method pre-trains a dynamics model in the environment, but the number of pre-training steps is not shown in the experimental results, resulting in an unfair comparison. Second, most baselines (TD3 and MA-TD3) in Figure 2 are implementations without some of the proposed method's components, making it more like an ablation study than a baselines comparison. The authors should consider adding state-of-the-art non-TD3 methods in Figure 2. Additionally, having real robot experiments will certainly improve the paper.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Results:\n* [5.2.1] It will be clearer if there is an additional sentence that describes the details of MVE, similar to what you did for MA-BDDPG.\n\nConclusion:\n* The sentence “one key novelty is to generate imaginary data with exploratory actions” is not true. Some prior works also add Gaussian noise to the action values, such as [1]. \n\nTypos:\n* Line 46: “see sec:result”.\n* Line 179: there is an extra “to”.\n\n**References:**\n\n[1] Hado van Hasselt and Marco A. Wiering. Reinforcement Learning in Continuous Action Spaces, 2007.\n"}}, {"review_id": "HEocZYka5kL", "reviewer": "Reviewer_BZ4p", "summary": "This paper proposes an approach (called MAMF) to augment a model-free reinforcement learning algorithm (namely TD3) with a learned dynamics model. The learned model is used to sample imaginary transitions from actually observed states, which are then used to train the critic. The imaginary transitions are filtered based on two criteria: 1) higher Q values, and 2) higher uncertainty. Experimental results in simulation show that this method outperforms prior methods in simulated manipulation tasks, achieving higher performance and better sample efficiency.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper is clear and easy to read, and has the relevant references (to the best of my knowledge). It also clarifies the differences with other closely related prior approaches well. Experiments are performed on many diverse environments, showing the effectiveness of the approach across domains and problem types. Ablation studies are also presented for various aspects of the algorithm, which is useful to see. I believe a key part of what makes this method works well is that the learned model is only used for extrapolating a single time-step into the future, limiting the amount of bias introduced by an inaccurate model. Another good feature is that the proposed strategy is not necessarily limited to augmenting TD3: it could (in theory) be applied to any model-free RL algorithm, although this is not tested in the paper. This definitely extends the applicability of the algorithm.\n\nWeaknesses:\n\nI believe the main weakness of this paper is probably the lack of real-world robot experiments. However, I believe this is mitigated by: 1) extensive experiments in a variety of simulation domains, and 2) the generality of the approach to any model-free RL algorithm, not just TD3.\n\nAnother potential issue which could make real-world application more challenging is the difficulty of training dynamics models for high-dimensional state/observation spaces such as camera images.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This is a good paper, which experimentally evaluates a model-based augmentation to model-free RL algorithms. While there are some weaknesses and limitations when considering real-world application, it nonetheless seems like a useful strategy to consider, and I would consider this a good contribution to the RL community.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "I couldn't find/think of anything that needed revision.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "DclSkeUULn", "reviewer": "Reviewer_4eEP", "summary": "This paper studies improving the sample efficiency of model-free DRLs using a learning transition model. The proposed method uses a learning model to generate imaginary transitions, which are then used to update the critic. The authors applied the proposed method to various simulated robot tasks and demonstrated that it outperformed other methods with high sample efficiency.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* When the proposed method is applied to TD3, sample efficiency and performance are improved simultaneously\n* Comparison with other methods on various robot simulation tasks\n* An ablation study of the proposed method is conducted\n\nWeaknesses:\n* The proposed method is a general method to improve DRL sample efficiency. It is not specialized for the complex manipulation tasks in the title. The association between the proposed method and complex manipulation is not strong.\n* No comparison in real robot task\n* As the authors describe in their limitations, the sample efficiency is not high enough to learn complex manipulations on a real robot. The combination of sim2real and the proposed method is given as an example as a solution; however, the proposed method would not be necessary if a simulator could be used.\n* No comparison of applying the proposed method to Soft Actor Critic (SAC), which is more sample efficient than TD3.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The approach is general enough to apply to a variety of tasks. Simulation results show high sample efficiency and performance for various robot tasks.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "It would be necessary to explain how the proposed method can be used to train real robot tasks by comparing it with previous sample-efficient methods.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)