（論文）DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills（2018）

# Source
- [DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills](https://arxiv.org/abs/1804.02717)
- [Towards a Virtual Stuntman](http://bair.berkeley.edu/blog/2018/04/10/virtual-stuntman/)

# Introduction
The authors present a reinforcement learning method to enables simulated characters to learn highly dynamic and acrobatic skills from reference motion clips (either motion captured or self-generated).  Compared to previous research, the movement of the actor is much more fluid and natural.

Furthermore, they can also provide goal-oriented objectives, such as walking in a target direction or hitting a target while performing the trained action.  On top of this, they have explored methods to develop multi-skilled agents capable of performing a rich repertoire of diverse skills.

[Introduction Video](https://www.youtube.com/watch?v=vppFvq2quQ0)


# Implementation
The task is structured as a standard reinforcement learning problem that attempts to learn the optimal policy parameter θ<sup>*</sup> by using policy gradient method.  The policy gradient is estimated as:
<img width="473" alt="Policy gradient" src="https://qiita-image-store.s3.amazonaws.com/0/264921/c506a71c-13e3-e107-8771-b2530e9b81eb.png">
Where d<sub>θ</sub>(s<sub>t</sub>) is the state distribution under the policy π<sub>θ</sub>.  A<sub>t</sub> is the advantage of taking an action a<sub>t</sub> at a given state s<sub>t</sub>
<img width="152" alt="Advantage" src="https://qiita-image-store.s3.amazonaws.com/0/264921/e624dd20-0cb5-5cf7-9dab-c66401455439.png">
<img width="130" alt="Rt" src="https://qiita-image-store.s3.amazonaws.com/0/264921/75ff8e05-d786-1a98-dab6-c7c4d047fee1.png"> is the total return received starting from state s<sub>t</sub> at time t, while V(s<sub>t</sub>) is the value function starting in s<sub>t</sub>.

The policy itself is trained using PPO ([Proximal Policy Optimization](https://arxiv.org/pdf/1707.06347.pdf)),  an extension of vanilla policy gradient methods (e.g. REINFORCE).  It introduces the **clipped surrogate objective**, which restricts the amount of policy update per step (except when the action is highly probable while the reward is negative).


###States
- Relative positions of each link with respect to the root, their rotations expressed in quaternions, and their linear and angular velocities.
- A phase variable φ∈[0,1].  φ=0 denotes the start of a motion, and φ=1 denotes the end. 
- Goal *g* 
for policies trained to achieve additional task objectives, such as walking in a particular direction or hitting a target.

###Actions
Target orientations for PD controllers at each joint.

###Network
The policy network is illustrated as:
<img width="604" alt="Network" src="https://qiita-image-store.s3.amazonaws.com/0/264921/faa39e27-ea56-865e-7dab-4d046ff6ffc2.png">
The heightmap H is sampled on a uniform grid around the character, and it is defined as a grayscale image, with black indicating minimum height and white representing maximum height.

The value function is modeled by a similar network, except the output layer consists of a single linear unit.

###Reward
Reward r<sub>t</sub> at each step t is calculated by:
<img width="166" alt="Reward" src="https://qiita-image-store.s3.amazonaws.com/0/264921/31cfcc11-cc73-c2d0-2b22-813b16e8cad0.png">
Where r<sup>I</sup><sub>t</sub> is the imitation objective (how well it follows the reference motion) and r<sup>G</sup><sub>t</sub> is the task objective (how well it completes the given task).  ω<sup>I</sup> and ω<sup>G</sup> are scalar weight values.

The imitation objective r<sup>I</sup><sub>t</sub> is further decomposed into:
<img width="367" alt="Imitation Objective" src="https://qiita-image-store.s3.amazonaws.com/0/264921/a099d154-1a19-9109-245d-0726c4b15923.png">
The pose reward r<sup>p</sup><sub>t</sub> indicates how well the joint orientations are matched with the reference motion.
<img width="293" alt="Pose reward" src="https://qiita-image-store.s3.amazonaws.com/0/264921/7cd4c43e-1305-59ec-6a78-fecf530318cc.png">
The velocity reward is computed from the difference between character and reference motion joint velocities.
<img width="294" alt="Velocity reward" src="https://qiita-image-store.s3.amazonaws.com/0/264921/fd0f5a46-5f2c-62bc-8e98-4b12ecc11589.png">
The end-effector reward r<sup>e</sup><sub>t</sub> tells how good the positions of the character's hands and feet are with respect to the reference motion.
<img width="295" alt="Endeffector reward" src="https://qiita-image-store.s3.amazonaws.com/0/264921/a050b683-ce62-468a-e2d4-66a7bc404f63.png">
Finally, r<sup>c</sup><sub>t</sub> outputs the deviation in the the character's center-of-mass from that of the reference motion.

#Training
There are two networks, one for the policy while another for the value function.  The policy network is updated using PPO, while the value function is updated using TD(λ).  There are two key points introduced in the paper which greatly impact the effectiveness of the training procedure; **Reference State Initialization (RSI)** and **Early Termination (ET)**.

###Reference State Initialization (RSI)
The initial state when learning from an episode is generally set to the state at the first step.  However, for the case of mimicking a motion, the policy must learn the early phases of the motion before progressing towards later phases.  This can be problematic for actions such as backflips, where learning the landing is a prerequisite for the character to receive a high return from the initial jump.  If the policy cannot land successfully, jumping will result in worse returns.
(This is slightly suspicious as the reward should reflect which action is yielding a better performance as the reward is calculated at each time step?).
<img width="838" alt="RSI" src="https://qiita-image-store.s3.amazonaws.com/0/264921/c880fa9f-0b7a-8686-0334-1060da46b3b1.png">

Another disadvantage of a fixed initial state is the resulting exploration challenge.  As the policy receives reward retrospectively, it is unable to learn that a state is favorable until it has actually visited the state and received a high reward.  For motions such as a backflip, which is very sensitive to the initial conditions at takeoff and landing, it is unlikely to execute a successful trajectory through random exploration if they always start from the ground.

The reference state initialization is therefore implemented by sampling a state in the episode length and use it as the initial state of the agent.

###Early Termination (ET)
An early termination is activated when certain links, such as the torso or head, makes contact with the ground.  Once this happens, the episode will be terminated immediately to prevent further training.  For example, if the character has fallen to the ground while trying to learn to walk, it is almost impossible for them to get back up, and they should not learn from movements they induce while laying on the ground.

![Image](http://bair.berkeley.edu/static/blog/stuntman/backflip_ablation.gif)
*Comparison of policies trained without RSI or ET. RSI and ET can be crucial for learning more dynamics motions. Left: RSI+ET. Middle: No RSI. Right: No ET.*

#Result
<img width="417" alt="Result" src="https://qiita-image-store.s3.amazonaws.com/0/264921/f4bf4e66-2b7b-cf56-b77a-559f680d1e1e.png">
<img width="477" alt="Result description" src="https://qiita-image-store.s3.amazonaws.com/0/264921/16f64ca0-5f1b-eb28-ae28-ee373be8f8bf.png">

<img width="500" alt="Result2" src="https://qiita-image-store.s3.amazonaws.com/0/264921/4c4925b5-a6e2-dc15-48bb-acb8916e41a4.png">

#Additional Functions
###Multi-Skill Integration
There are three different methods to enable the performance of multiple skills sequentially.

1. Multi-Clip Reward
The policy is introduced to different reference motion clips.  While the action to perform will be constant, the imitation objective will become diverse (as they are logged separately).  The actual reward used to update the network is calculated as:
<img width="150" alt="Multi-clip reward" src="https://qiita-image-store.s3.amazonaws.com/0/264921/9ee5b22b-65f2-d572-e3b8-0df387164e35.png">
where r<sup>j</sup><sub>t</sub> is the imitation objective with respect to the *j* th clip, and *k* is the number of motion clips.

2. Skill Selector Policy
The policy is supplied with an one-hot vector (in additional to the original inputs), where each entry corresponds to the motion that should be executed.  Note that there is no additional task objective r<sup>G</sup><sub>t</sub>.

3. Composite Policy
Divise separate policies for different skills, then integrate them together into a composite policy.  The agent determines which is the best skill to perform by examining the value function of each policies under the given state.  Given a set of policies and their value functions <img width="133" alt="policy value functions" src="https://qiita-image-store.s3.amazonaws.com/0/264921/a3390b2e-af16-1e8c-da07-25127df0f827.png">, a composite policy Π(a|s) is constructed as <img width="400" alt="composite policy" src="https://qiita-image-store.s3.amazonaws.com/0/264921/539189d6-1b4e-a01a-9b08-1ac7a0483bc0.png">
where Τ is the temperature parameter.

###Varying Characters
Policy trained on one particular type of humanoid can also be used to perform motions on another humanoid models with vastly different mass distribution and actuators.  It is also possible to train policy for quadrupedal models such as dinosaur and cheetah.
![Image](http://bair.berkeley.edu/static/blog/stuntman/t-rex.gif)

###Task Goals
The policies can be trained to perform tasks while preserving the style prescribed by the reference motions.  The tasks demonstrated include following a target direction, striking a target, throwing a ball at a target and and terrain traversal.  These exercises are implemented by providing additional input goal g<sub>t</sub> and modifying the task objective r<sup>G</sup><sub>t</sub> accordingly.
![Image](http://bair.berkeley.edu/static/blog/stuntman/humanoid_strikc_spinkick.gif)
![Image](http://bair.berkeley.edu/static/blog/stuntman/humanoid_throw.gif)

#Conclusion
- RSI is probably applicable for a wide range of tasks.  It is especially useful for problems that only merit one final reward at the end of the episode.
- ET has been employed prevalently already, the paper merely solidifies the assumption of their effectiveness with evaluated results.
- It is uncertain how many motions can be learnt by a single policy.
- The imitation reward has parameters that are shared amongst different motions.  Perhaps there are particular values that will be more effective for certain actions?  There could even be correlations between the parameter values and the genre of the motion.