# Analysis of Cost Functions for Reinforcement Learning of Reaching Tasks in Humanoid Robots
## 4 Analysis of Influence of Different Cost Function Terms on the Learnt Motions
### Experimental Setting
#### 4.1.1 Evaluation of the Influence of the Stability Cost  
We started our analysis by setting only the weights of the cost terms addressing the stability and reaching accuracy in cost function (14) to non-zero values. The weights of the trajectory similarity term as well as the smoothing term were thus set to zero. The weights associated with position and orientation cost terms were set to \(w_{pos}=20\) and \(w_{ori}=4\), respectively. As discussed in Section 3.1, it is essential to maintain the proper relative scaling between these weights. In this context, the choice of \(w_{pos}=20\) and \(w_{ori}=4\) is made based on a 5:1 ratio between the position and orientation costs observed during the initial demonstration. This ensures that both position and orientation have a similar influence on the overall cost function, contributing to balanced convergence between these two terms. To evaluate the effect of different stability weights, we performed the RL process with 4 different \(w_{stab}\), i.e., \(w_{stab}=0,\ 10,\ 15,\ 20\), aiming to provide a comparable scaling as the final position and orientation weights. Although learning always starts with an initially stable motion (acquired by the motion mapping approach described in Section 2.2), the robot could become unstable due to the exploration process. The aim of RL is to maintain and possibly improve the stability of the robot as well as other task requirements.  
The results coming from RL sessions with four different stability weights are displayed in Figure 5. The learning curves in Figure 5a show convergence after approximately 30 updates in all 4 cases. Good reaching accuracy was achieved by all four weights, as plotted in Figure 5c,d. The main difference is visible in the stability term. Figure 5b depicts the average distance of the measured center of pressure to the center of the support polygon throughout the learning process. As expected, the higher stability weight contributes to the improved stability of the robot.  
Good task performance is evident also from the trajectory of the ZMP and the 3D spatial path of the end-effector, respectively, as shown in Figure 6a,b. The robot maintains the ZMP within the support polygon for each configuration of weights. As expected, the learning session with the highest stability weight shows the most stable motion of the ZMP. Figure 6b provides insight into the evolution of the end-effector trajectory. Despite the lack of the trajectory similarity term in the cost function, the end-effector learns to accurately reach the desired pose while preserving the initially demonstrated trajectory shape.  
Table 1 provides the statistical analysis of reaching accuracy and stability of the robot when using different stability cost weights to optimize the reaching motion. The table presents the average and minimal errors for position and orientation observed in the last 50 updates as well as the average and the minimal distance between the measured center of pressure and the center of the support polygon in the last 50 updates. All weight configurations demonstrate a reaching accuracy with low positional and orientational errors. As the highest stability weight \(w_{stab}=20\) only negligibly influences the accuracy of reaching, we used this value in the rest of our analysis.  
Figure 5: Learning performance when using different weights for the stability cost term. (**a**) Cost. (**b**) Distance from the center of the support polygon. (**c**) Distance from the desired goal position. (**d**) Distance from the desired goal orientation.  
Figure 6: Stability of the robot and spatial course of the reaching motions when using different weights for the stability cost term. (**a**) ZMP trajectories of the initial reaching motion and of the motions toward the new reaching pose computed by RL, viewed in the support polygon of the two feet (**left**), and as separate \(x\) and \(y\) components of the ZMP over time (**right**). (**b**) Spatial path of the initial reaching motion and the motions toward the new reaching pose computed by RL.