# Analysis of Cost Functions for Reinforcement Learning of Reaching Tasks in Humanoid Robots
## 4 Analysis of Influence of Different Cost Function Terms on the Learnt Motions
### Experimental Setting
#### 4.1.2 Evaluation of the Influence of the Trajectory Similarity Cost  
Next, we analyzed the impact of the trajectory similarity term. As justified in Section 4.1.1, the weight for the stability was chosen to be \(w_{stab}=20\). The weights associated with the accuracy of reaching position and orientation cost were kept at \(w_{pos}=20\) and \(w_{ori}=4\), respectively, while the weight of the smoothing term was set to 0. We analyzed the learning processes by varying the trajectory similarity cost weight, where we selected \(w_{traj}=0.0,\ 0.5,\ 1.5,\ 5\), aiming to provide similar or higher scaling than the other cost terms. Learning curves for the different weight configurations are shown in Figure 7a. As expected, the highest trajectory similarity weight yields the smallest deviation from the initial trajectory. However, this comes at the expense of the compromised reaching accuracy as the weight increases. We observe a consistent pattern of the reduced reaching accuracy as the trajectory similarity weight increases.  
The performance of different trajectory similarity weights in terms of stability are visualized through the ZMP trajectories in Figure 8a. It turns out that the trajectory similarity term did not significantly influence the stability of the robot. Finally, in Figure 8b we observe that the spatial course of all of the computed reaching motions is similar to the spatial course of the initial reaching trajectory. Due to the generalization properties inherent in the DMP representation, even the trajectory computed with \(w_{traj}=0\) approximates the overall spatial course of the resulting reaching motion well.  
Table 2 provides numerical values for the accuracy of reaching and for trajectory similarity. The table presents the average and minimal errors for position and orientation observed in the last 50 updates. Furthermore, the average and minimal DTW distance between the initial reaching motion and the computed reaching motions in the last 50 updates are also provided. The results presented in Table 2 confirm the negative impact of the trajectory similarity term on the reaching accuracy. While the optimal trajectory from the learning session with the highest weight (\(w_{traj}=5\)) is the closest to the initially demonstrated trajectory, the reaching accuracy is compromised and the robot fails to reach the desired pose. Given that the results in Figure 8a and Table 2 demonstrate that the trajectory similarity weight only negligibly improves the trajectory shape and that the initial trajectory shape is well preserved even when \(w_{traj}=0\), we conclude that the best choice for the weight of the trajectory cost term is \(w_{traj}=0\).  
\begin{table}
\begin{tabular}{c c c c c c c} \hline  & \multicolumn{2}{c}{**Position [m]**} & \multicolumn{2}{c}{**Orientation [rad]**} & \multicolumn{2}{c}{**DTW Distance**} \\  & **Avg** & **Min** & **Avg** & **Min** & **Avg** & **Min** \\ \hline \(w_{traj}=0.0\) & 0.0020 & 0.0003 & 0.0047 & 0.0008 & 1.9202 & 1.8585 \\ \hline \(w_{traj}=0.5\) & 0.0053 & 0.0035 & 0.0055 & 0.0003 & 1.2956 & 1.2847 \\ \hline \(w_{traj}=1.5\) & 0.0120 & 0.0102 & 0.0086 & 0.0005 & 0.9465 & 0.9375 \\ \hline \(w_{traj}=5\) & 0.0234 & 0.0227 & 0.4946 & 0.4868 & 0.2672 & 0.2653 \\ \hline \end{tabular}
\end{table}
Table 2: Statistical analysis of the reaching accuracy and trajectory similarity when using different trajectory similarity cost weights to optimize the reaching motion. The weights of the other cost terms were set as follows: \(w_{pos}=20\), \(w_{ori}=4\), \(w_{stab}=20\), \(w_{acc}=0.0\).