# Analysis of Cost Functions for Reinforcement Learning of Reaching Tasks in Humanoid Robots
###### Abstract  
In this paper, we present a study on transferring human motions to a humanoid robot for stable and precise task execution. We employ a whole-body motion imitation system that considers the stability of the robot to generate a stable reproduction of the demonstrated motion. However, the initially acquired motions are usually suboptimal. To successfully perform the desired tasks, the transferred motions require refinement through reinforcement learning to accommodate the differences between the human demonstrator and the humanoid robot as well as task constraints. Our experimental evaluation investigates the impact of different cost function terms on the overall task performance. The findings indicate that the selection of an optimal combination of weights included in the cost function is of great importance for learning precise reaching motions that preserve both the robot's postural balance and the human-like shape of the demonstrated motions. We verified our methodology in a simulated environment and through tests on a real humanoid robot, TALOS. humanoid robots; imitation learning; reinforcement learning +
Footnote †: journal: Applied  
0000-0001-999-4000-0000-0000-0000-0000-0000-0000-0000-0000-0000-0  
by also considering task constraints besides the robot's physical constraints [12]. Machine learning algorithms can assist in generalizing and extrapolating these motions, especially when accurate models are unavailable [13; 14]. Many studies exploit the synergy of LfD and Reinforcement Learning (RL) to adapt the demonstrated motions to specific task requirements. This way the strengths of both methodologies are utilized to facilitate successful task execution and generate human-like movements. By narrowing the exploration to the neighborhood of the initial policy derived from LfD, exploration is confined to the search space within the humanoid robot's motor space, averting potentially harmful or unpredictable behaviors, such as falls. This strategy accelerates the optimal control policy identification process via RL, benefiting from the prior knowledge obtained by LfD while retaining the essence of the initial motion patterns. Thus, these learning strategies effectively complement each other, enhancing the overall learning efficacy. The application of RL in humanoid robots remains a challenge due to the high complexity and dimensionality of state and action spaces. Traditional algorithms that aim to learn state-action-reward functions across the entire state space are unfeasible in such environments. Consequently, many RL strategies focus on learning a parameterized policy through complete trajectory rollouts. A notable solution that follows this approach is the PI\({}^{2}\) algorithm (Policy Improvement with Path Integrals). It is a stochastic policy search algorithm that effectively scales to high-dimensional states [15]. [PERSON] et al. [16] successfully applied the PI\({}^{2}\) algorithm to teach a 34-DOF humanoid robot full-body motor skills, thus demonstrating the algorithm's efficiency and robustness in high-dimensional spaces [17; 18]. PI\({}^{2}\) was further extended by [PERSON] et al. [19] to automatically determine the magnitude of the exploration noise by Covariance Matrix Adaptation (CMA). The modified PI\({}^{2}\)-CMA algorithm has been proven to be scalable to high dimensions while also being easy to implement because it has no open tuning parameters. A variation of PI\({}^{2}\)-CMA has been proposed by [PERSON] et al. [20] where additional heuristics are used to guide the CMA process, which leads to faster convergence. Another stochastic RL method is the PoWER algorithm developed by [PERSON] and [PERSON] [21]. This algorithm shares the update strategy with PI\({}^{2}\) and has also been reported to be applicable to the learning of high dimensional humanoid robot motions.