user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Policy Transfer with Strategy Optimization

Paper content:
Computer simulation provides an automatic and safe way for training robotic control
policies to achieve complex tasks such as locomotion.
However, a policy
trained in simulation usually does not transfer directly to the real hardware due
to the differences between the two environments.
Transfer learning using domain
randomization is a promising approach, but it usually assumes that the target environment
is close to the distribution of the training environments, thus relying
heavily on accurate system identification.
In this paper, we present a different
approach that leverages domain randomization for transferring control policies to
unknown environments.
The key idea that, instead of learning a single policy in
the simulation, we simultaneously learn a family of policies that exhibit different
behaviors.
When tested in the target environment, we directly search for the best
policy in the family based on the task performance, without the need to identify
the dynamic parameters.
We evaluate our method on five simulated robotic control
problems with different discrepancies in the training and testing environment
and demonstrate that our method can overcome larger modeling errors compared
to training a robust policy or an adaptive policy.
Recent developments in Deep Reinforcement Learning (DRL) have shown the potential to learn complex robotic controllers in an automatic way with minimal human intervention.
However, due to the high sample complexity of DRL algorithms, directly training control policies on the hardware still remains largely impractical for agile tasks such as locomotion.A promising direction to address this issue is to use the idea of transfer learning which learns a model in a source environment and transfers it to a target environment of interest.
In the context of learning robotic control policies, we can consider the real world the target environment and the computer simulation the source environment.
Learning in simulated environment provides a safe and efficient way to explore large variety of different situations that a real robot might encounter.
However, due to the model discrepancy between physics simulation and the real-world environment, also known as the Reality Gap BID2 BID18 , the trained policy usually fails in the target environment.
Efforts have been made to analyze the cause of the Reality Gap BID20 and to develop more accurate computer simulation to improve the ability of a policy when transferred it to real hardware.
Orthogonal to improving the fidelity of the physics simulation, researchers have also attempted to cross the reality gap by training more capable policies that succeed in a large variety of simulated environments.
Our method falls into the second category.To develop a policy capable of performing in various environments with different governing dynamics, one can consider to train a robust policy or to train an adaptive policy.
In both cases, the policy is trained in environments with randomized dynamics.
A robust policy is trained under a range of dynamics without identifying the specific dynamic parameters.
Such a policy can only perform well if the simulation is a good approximation of the real world dynamics.
In addition, for more agile motor skills, robust policies may appear over-conservative due to the uncertainty in the training environments.
On the other hand, when an adaptive policy is used, it learns to first identify, implicitly or explicitly, the dynamics of its environment, and then selects the best action according to the identified dynamics.
Being able to act differently according to the dynamics allows the adaptive policy to achieve higher performance on a larger range of dynamic systems.
However, when the target dynamics is notably different from the training dynamics, it may still produce sub-optimal results for two reasons.
First, when a sequence of novel observations is presented, the learned identification model in an adaptive policy may produce inaccurate estimations.
Second, even when the identification model is perfect, the corresponding action may not be optimal for the new situation.In this work, we introduce a new method that enjoys the versatility of an adaptive policy, while avoiding the challenges of system identification.
Instead of relating the observations in the target environment to the similar experiences in the training environment, our method searches for the best policy directly based on the task performance in the target environment.Our algorithm can be divided to two stages.
The first stage trains a family of policies, each optimized for a particular vector of dynamic parameters.
The family of policies can be parameterized by the dynamic parameters in a continuous representation.
Each member of the family, referred to as a strategy, is a policy associated with particular dynamic parameters.
Using a locomotion controller as an example, a strategy associated with low friction coefficient may exhibit cautious walking motion, while a strategy associated with high friction coefficient may result in more aggressive running motion.
In the second stage we perform a search over the strategies in the target environment to find the one that achieves the highest task performance.We evaluate our method on three examples that demonstrate transfer of a policy learned in one simulator DART, to another simulator MuJoCo.
Due to the differences in the constraint solvers, these simulators can produce notably different simulation results.
A more detailed description of the differences between DART and MuJoCo is provided in Appendix A. We also add latency to the MuJoCo environment to mimic a real world scenario, which further increases the difficulty of the transfer.
In addition, we use a quadruped robot simulated in Bullet to demonstrate that our method can overcome actuator modeling errors.
Latency and actuator modeling have been found to be important for Sim-to-Real transfer of locomotion policies BID20 .
Finally, we transfer a policy learned for a robot composed of rigid bodies to a robot whose end-effector is deformable, demonstrating the possiblity of using our method to transfer to problems that are challenging to model faithfully.
We have demonstrated that our method, SO-CMA, can successfully transfer policies trained in one environment to a notably different one with a relatively low amount of samples.
One advantage of SO-CMA, compared to the baselines, is that it works consistently well across different examples, while none of the baseline methods achieve successful transfer for all the examples.We hypothesize that the large variance in the performance of the baseline methods is due to their sensitivity to the type of task being tested.
For example, if there exists a robust controller that works for a large range of different dynamic parameters µ in the task, such as a bipedal running motion in the Walker2d example, training a Robust policy may achieve good performance in transfer.
However, when the optimal controller is more sensitive to µ, Robust policies may learn to use overly-conservative strategies, leading to sub-optimal performance (e.g. in HalfCheetah) or fail to perform the task (e.g. in Hopper).
On the other hand, if the target environment is not significantly different from the training environments, UPOSI may achieve good performance, as in HalfCheetah.
However, as the reality gap becomes larger, the system identification model in UPOSI may fail to produce good estimates and result in non-optimal actions.
Furthermore, Hist did not achieve successful transfer in any of the examples, possibly due to two reasons:
1) it shares similar limitation to UPOSI when the reality gap is large and
2) it is in general more difficult to train Hist due to the larger input space, so that with a limited sample budget it is challenging to fine-tune Hist effectively.We also note that although in some examples certain baseline method may achieve successful transfer, the fine-tuning process of these methods relies on having a dense reward signal.
In practice, one may only have access to a sparse reward signal in the target environment, e.g. distance traveled before falling to the ground.
Our method, using an evolutionary algorithm (CMA), naturally handles sparse rewards and thus the performance gap between our method (SO-CMA) and the baseline methods will likely be large if a sparse reward is used.
We have proposed a policy transfer algorithm where we first learn a family of policies simultaneously in a source environment that exhibits different behaviors and then search directly for a policy in the family that performs the best in the target environment.
We show that our proposed method can overcome large modeling errors, including those commonly seen on real robotic platforms with relatively low amount of samples in the target environment.
These results suggest that our method has the potential to transfer policies trained in simulation to real hardware.There are a few interesting directions that merit further investigations.
First, it would be interesting to explore other approaches for learning a family of policies that exhibit different behaviors.
One such example is the method proposed by BID11 , where an agent learns diverse skills without a reward function in an unsupervised manner.
Another example is the HCP-I policy proposed by BID4 , which learns a latent representation of the environment variations implicitly.
Equipping our policy with memories is another interesting direction to investigate.
The addition of memory will extend our method to target environments that vary over time.
We have investigated in a few options for strategy optimization and found that CMA-ES works well for our examples.
However, it would be desired if we can find a way to further reduce the sample required in the target environment.
One possible direction is to warm-start the optimization using models learned in simulation, such as the calibration model in or the online system identification model in BID38 .A
DIFFERENCES BETWEEN DART AND MUJOCO DART BID19 and MuJoCo BID35 are both physically-based simulators that computes how the state of virtual character or robot evolves over time and interacts with other objects in a physical way. Both
of them have been demonstrated for transferring controllers learned for a simulated robot to a real hardware , and there has been work trying to transfer policies between DART and MuJoCo BID36 . The
two simulators are similar in many aspects, for example both of them uses generalized coordinates for representing the state of a robot. Despite
the many similarities between DART and MuJoCo, there are a few important differences between them that makes transferring a policy trained in one simulator to the other challenging. For the
examples of DART-to-MuJoCo transfer presented in this paper, there are three major differences as described below:1. Contact
Handling Contact modeling is important for robotic control applications, especially for locomotion tasks, where robots heavily rely on manipulating contacts between end-effector and the ground to move forward. In DART
, contacts are handled by solving a linear complementarity problem (LCP) (Tan et al.) , which ensures that in the next timestep, the objects will not penetrate with each other, while satisfying the laws of physics. In MuJoCo
, the contact dynamics is modeled using a complementarity-free formulation, which means the objects might penetrate with each other. The resulting
impulse will increase with the penetration depth and separate the penetrating objects eventually.

assistant: text: We propose a policy transfer algorithm that can overcome large and challenging discrepancies in the system dynamics such as latency, actuator modeling error, etc.