# Analysis of Cost Functions for Reinforcement Learning of Reaching Tasks in Humanoid Robots
## 6 Conclusions  
In this study, we combined imitation learning and reinforcement learning to effectively teach a humanoid robot how to perform a reaching task while maintaining its stability. Our approach involves transferring an observed human motion to a humanoid robot, accounting for the stability constraints of humanoid robots. The transferred motion is then refined using reinforcement learning with a suitably selected cost function to consider several specific objectives with respect to the desired task. In our study, the selection of suitable cost function components is grounded in statistical analysis of the influence of different cost function terms on the performance of the reaching behavior. The gained insights helped us determine the most effective weight configuration for the selected cost function, ensuring the best performance across multiple task objectives. Our analysis finally led to the conclusion about the best weight configuration and highlighted the importance of various constraints with respect to the overall task performance. The proposed algorithm was successfully applied to compute reaching motions for a high degree of freedom humanoid robot. It can learn stable reaching trajectories in less than 30 updates. With the proposed approach, the robot can learn a reaching motion to a \begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multicolumn{2}{c}{**Position (m)**} & \multicolumn{2}{c}{**Orientation (rad)**} & \multicolumn{2}{c}{**Distance from CSP (m)**} \\
**Simulation** & **Robot** & **Simulation** & **Robot** & **Simulation** & **Robot** \\ \hline
0.0012 & 0.0056 & 0.0061 & 0.0114 & 0.0337 & 0.0441 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Accuracy of reaching motion and stability performance in simulation and on the real robot. The cost function weights when computing the reaching motion were set to the following values: \(w_{stab}=20\), \(w_{pos}=20\), \(w_{ori}=4\), \(w_{traaj}=0.0\), \(w_{acc}=0.0\). Figure 14: Comparison of the execution of the initial motion and the optimal motion with the real robot. Figure 13: Comparison of the execution of the initial motion and the optimal motion in simulation. desired point in 3D space, based on a single observation of human motion. The optimal reaching motions computed by RL in simulation were transferred and assessed on the real robot, demonstrating a negligible gap between simulation and real world. Nevertheless, to tackle more intricate tasks, it is likely that further adjustments to the transferred motion will be necessary to accommodate the variations in the dynamics between the simulated and the real robot. The proposed framework offers various possibilities for extension. It can be adapted to include more complex whole-body tasks, integrating additional constraints into the learning process. Addressing these more complex tasks will necessitate the adoption of advanced approaches for determining optimal weights. In this context, an automated curriculum learning approach could be utilized to learn an optimal weight configuration by progressively increasing task complexity. On a smaller scale, the method presented in this work is comparable to curriculum learning. In our case, the analysis of task performance was used, whereas curriculum learning would require specifically designed heuristics to identify the optimal combination of weights. Conceptualization, methodology, investigation, validation, visualization and writing--original draft preparation, K.S. ; supervision and writing--review and editing, A.U. All authors have read and agreed to the published version of the manuscript. This research was supported by program group Automation, Robotics and Biocybernetics (P2-0076) and Young Researcher Grant (PR-12398), both funded by the Slovenian Research and Innovation Agency. Not applicable. Publicly available data is analyzed in this study.