# Analysis of Cost Functions for Reinforcement Learning of Reaching Tasks in Humanoid Robots
## 4 Analysis of Influence of Different Cost Function Terms on the Learnt Motions
### Experimental Setting
#### 4.1.3 Evaluation of the Influence of the Smoothing Cost  
Finally, we also investigated the impact of the smoothing term in cost function (14). Its purpose is to ensure the generation of smooth joint trajectories. Based on the results of Sections 4.1.1 and 4.1.2, we set \(w_{pos}=20\), \(w_{ori}=4\), \(w_{stab}=20\), and \(w_{traj}=0.0\). In Figure 9, we compare the performance of learning when varying acceleration cost weight, specifically \(w_{acc}=0.0,\ 1.0,\ 1.5,\ 10\), which will provide comparable but also higher scaling than the other cost terms included in the learning process. As evident from Figure 9e, the configuration with the highest smoothness weight results in the lowest cost for acceleration.  
Figure 8: Stability of the robot and spatial course of the reaching motions when using different weights for the trajectory similarity cost term. (**a**) ZMP trajectories of the initial reaching motion and of the motions towards the new reaching pose computed by RL, viewed in the support polygon of the two feet (**left**), and as separate \(x\) and \(y\) components of the ZMP over time (**right**). (**b**) Spatial path of the initial reaching motion and he motions toward the new reaching pose computed by RL.  
Figure 7: Learning performance when using different weights for the trajectory similarity cost term. (**a**) Cost. (**b**) Distance from the center of the support polygon. (**c**) Distance from the desired goal position. (**d**) Distance from the desired goal orientation. (**e**) Deviation of the initial and the reproduced trajectory.  
The resulting ZMP trajectories and end-effector paths are depicted in Figure 10a,b, respectively. Note that the configuration with the highest acceleration weight compromises the reaching accuracy and also alters the shape of the reaching trajectory.  
Table 3 provides the statistical analysis of the reaching accuracy and smoothing cost when evaluating different smoothing cost weights when learning the reaching motion. Average and minimal positional and orientational errors are presented as well as the average and minimal smoothing cost between the initial reaching motion and the computed reaching motions in the last 50 updates. The statistical analysis in Table 3 makes the trend of increased deviation from the desired final reaching pose as the smoothness weight increases clear. However, the configuration with a smoothness weight of \(w_{acc}=1.0\) maintains good reaching accuracy while achieving a smoother reaching trajectory, as evident from  
Figure 10: Stability of the robot and spatial course of the reaching motions when using different weights for the smoothing cost term. (**a**) ZMP trajectories of the initial reaching motion and of the motions toward the new reaching pose computed by RL, viewed in the support polygon of the two feet (**left**), and as separate \(x\) and \(y\) components of the ZMP over time (**right**). (**b**) Spatial path of the initial reaching motion and the motions toward the new reaching pose computed by RL.  
Figure 9: Learning performance when using different weights for the smoothing cost term. (**a**) Cost. (**b**) Distance from the center of the support polygon. (**c**) Distance from the desired goal position. (**d**) Distance from the desired goal orientation. (**e**) Acceleration squared.  
Figure 10b. Consequently, we conclude that the weight configurations with \(w_{acc}=0.0\) and \(w_{acc}=1.0\) both offer good solutions.