Parallel deep reinforcement learning withmodel-free and model-based methods
Eiji Uchibe. Dept. of Brain Robot Interface, Computational Neuroscience Labs., ATR International

Introduction⚫ Background⚫ Contribution of this study

Reinforcement Learning (RL) can be categorized into model-based methods that
exploit an (estimated) environmental model, and model-free methods that directly
learn a policy through the interaction with the environment. To improve learning
efficiency, we have proposed CRAIL (Uchibe, 2018), which dynamically selects alearning module from multiple heterogeneous model-free RL modules according tolearning performance while multiple modules are trained simultaneously. However,
CRAIL does not consider model-based methods.

This study extends CRAIL to deal with model-based and model-free methods andinvestigates whether dynamic switching between them contributes to theimprovement of learning efficiency. The proposed method was evaluated byMuJoCo benchmark tasks. Experimental results show that a model-based methodwith a simple model was selected at the early stage of learning, and a modelbased method with a complicated model was used at the later stage. Furthermore,model-free methods were selected when the network did not have sufficient
capacity to represent the environmental dynamics.

Inverse RL using Density Ratio Estimation⚫ Model-free Cooperative Reinforcement And Imitation Learning (CRAIL)[Uchibe, 2018]• Multiple learning modules withdifferent algorithms and networks

mixed policy
value andpolicyDPGselfimitation
SAC

REINFORCE⚫ Prepared Model-free RL modules
Soft Actor-Critic (SAC)• Learning from the replay buffer– Mixed policy collect experiences

experience– All the modules share the experiences

replay buffer– Trained with RL loss and self-imitation loss

• Mixing weights are determined by thevalue function• The appropriate module is selected
automatically– Early stage: RBF x SAC– Late stage: NN x DPG

Gaussian Process based SAC (GP-SAC)

Replay buffer• Learning from the estimated model

action value 𝑄• Uniform sampling from the replaybuffer(Haarnoja et al., 2018)⚫ Prepared Model-based RL modules
Probabilistic Inference for Learning
Control (PILCO)

Replay buffer

Each module has a policy and a state value function• Each module has a policy and a state value function– 𝜋𝑖 𝑢 𝑥 : prob. of the 𝑖-th module to select 𝑢 at 𝑥, and 𝑉𝑖 (𝑥): expected value of returns at 𝑥
– 𝛾: discount factor，and 𝑟(𝑥, 𝑢): immediate reward• Behavior policy is given by the weighted sum of policies
𝜋𝑏 𝑢 𝑥= σ𝑀𝑖=1 𝛼𝛼 𝑖 𝑥 ∝ exp 𝛽𝑉𝑖 𝑥𝑑𝐽𝑑𝜃

Stochastic Value Gradient (SVG)• NN model for computing two gradients
Replay buffer

𝑑𝑉 𝑑𝑉,𝑑𝑥 𝑑𝜃• Multiple importance sampling foroff-policy learning• Multiple importance sampling foroff-policy learning• Evaluate the long-term predictions
by cascading 1-step prediction• Expand the Bellman recursion for1-step(Deisenroth and Rasmussen, 2011;
Deisenroth et al., 2015)

𝑖 𝑥 𝜋𝑖 𝑢 𝑥– 𝛼(𝑖 ∣ 𝑥): mixing weight is defined by• Dynamics and reward are
estimated by Gaussian Process• DYNA-style (Sutton, 1990)• Compute policy gradients analytically⚫ Entropy-regularized reinforcement learning

𝑄

Replay buffer(Heess et al., 2015)

the agent tends to select the module that has alarge state value with high probability.

Experimental Results⚫ Simulation: MuJoCo control tasks⚫ Comparison of learning curves

Benchmark tasks provided by OpenAI. For comparison, each module is trained
independently• The performance of MB-CRAIL increased
after about 5 × 104 steps• PILCO learned faster, but its performancesaturated• SAC learned more slowly but achieved
the best performance• The model-based methods (PILCO andSVG) have a large weight in the middle oflearning⚫ Training curves on
all the control tasks• SAC is often selected at the end oflearning

Conclusion⚫ Summary
We developed Model-based CRAIL, which integrates model-free and model-based RL modulesand selects one module according to the learning progress. Our method outperformed themodules that were trained independently. We will consider the computing time in each step.
For example, a simple module can make a decision faster than a complicated module. It is often
appropriate in robot control.

⚫ Acknowledgments
This work was supported by Japan Society for the Promotion of Science KAKENHI Grant Numbers JP19H05001.
This work was also supported by Innovative Science and Technology Initiative for Security Grant Number
JPJ004596, ATLA, Japan and JST-Mirai Program Grant Number JPMJMI18B8, Japan.

⚫ References
1. Deisenroth, M., et al. (2015). Gaussian processes for data-efficient learning in robotics and control. IEEE
Transactions on Pattern Analysis and Machine Intelligence, pp. 408-423.
2. Haarnoja, T., et al. (2018). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with astochastic actor. In Proc. of the 35th International Conference on Machine Learning, pp. 1861-1870.
3. Heess, N. et al. (2015). Learning continuous control policies by stochastic value gradients. In Advances in
Neural Information Processing Systems 28. 2015.
4. Uchibe, E. (2018). Cooperative and Competitive Reinforcement and Imitation Learning for a Mixture ofHeterogeneous Learning Modules. Frontiers in Neurorobotics.
5. Sutton, R.S. (1990). Integrated Architecture for Learning, Planning, and Reacting Based on Approximating
Dynamic Programming. In Proc. of the 7th International Conference on Machine Learning.