# Analysis of Cost Functions for Reinforcement Learning of Reaching Tasks in Humanoid Robots
## 3 Motion Adaptation by Reinforcement Learning  
The process of stable motion imitation effectively addresses both the kinematics and dynamics of the robot. However, this results in a modified reproduction of the initially demonstrated trajectory. The extracted joint angle motion thus results in a suboptimal control policy for tasks requiring high precision, such as reaching. Thus, the transferred motion needs to be adapted to fulfill the task requirements. This can be achieved by RL algorithms, where the imitated human motion serves as an initial control policy. The aim of the RL is to enhance this initial policy, taking into consideration the specific task requirements and physical constraints of the robot.  
As mentioned before, when applying RL to complex robotic systems like humanoid robots, the problem known as the \"curse of dimensionality\" poses a significant challenge. Despite RL algorithms' ability to create new control policies, the curse of dimensionality is still an ongoing research concern and often prevents fast and reliable learning. As was shown in [16], the PI\({}^{2}\) algorithm is able to handle the complexity of a humanoid robot while also providing fast convergence. This approach stands out when measuring the algorithm performance and speed of convergence when compared to gradient-based RL algorithms such as REINFORCE [23] and Natural-Actor-Critic [22].  
The Policy Improvement with Path integrals is a probabilistic reinforcement learning algorithm derived from the principles of stochastic optimal control and path integrals. The algorithm uses probability-weighted averaging to perform a parameter update instead of using an estimate of the gradient, as is the case with many RL algorithms. PI\({}^{2}\) requires minimal tuning of the parameters with the exploration noise being the only tunable parameter. However, the modification used in this work involves a method that is able to automatically determine the exploration magnitude, as proposed by [PERSON] et al. [19]. The modified method, called PI\({}^{2}\)-CMA incorporates a covariance matrix adaptation based  
Figure 1: Block diagram of the two-task inverse kinematics process for obtaining stable motion reproduction.  
Figure 2: Sequence of snapshots of the demonstrated and reproduced reaching motion.  
on [25; 26], which enables automatic adaptation of the exploration matrix [36]. Consequently, the robot should be able to determine the trade-off between exploration and exploitation autonomously.  
The PI\({}^{2}\)-CMA learns a parameterized policy. Using a parameterized policy, the curse of dimensionality is avoided and the algorithm optimizes a relatively small number of parameters representing the parameterized task trajectories. In this study, we utilize Dynamical Movement Primitives (DMPs) [37; 38], a widely used method for representing robot trajectories. The DMP encoding is utilized to parameterize the reproduced human demonstrations resulting from the stable motion imitation system, generating an initial trajectory that is subsequently improved by the RL process. The trajectory parameterization is a crucial step when learning a task trajectory. The DMP parameterization offers several advantages: it represents the motion with a small number of parameters, reduces the impact of noise present in demonstrations, and generates smooth motion trajectories. The DMP formulation is based on a stable dynamical system and, therefore, ensures stable dynamics of the encoded trajectories. Another advantage is that the learned trajectory can be easily adapted to different goals without modifying its underlying shape. Additionally, this representation proves resilient to external disturbances, further enhancing the robustness of the learned motion trajectories.