# Analysis of Cost Functions for Reinforcement Learning of Reaching Tasks in Humanoid Robots
## 2 Stable Motion Imitation Framework
### Stability Control and Motion Mapping  
Since we are interested in safe and stable motion imitation, solely mapping the kinematic motion of the human demonstrator to the humanoid robot is insufficient. The directly transferred joint angle trajectories may lead to falls as the robot's stability is not considered during the motion transfer. To address this issue, we incorporated a stability control mechanism based on the linear inverted pendulum dynamics. This approach accounts for the robot's mass distribution to ensure the stability of the robot while executing the transferred motions. The issue of preserving the stability of a humanoid robot is one of the fundamental challenges of humanoid robotics. The most popular stability concept, the zero moment point (ZMP), was introduced by [PERSON] [27] and has been foundational in developing various stability control methods ever since. The ZMP represents an important physical characteristic of a humanoid robot, denoting the point on the ground where the sum of allthe moments of the active forces equals zero. The stability criterion derived from the ZMP states that the robot is balanced if its ZMP falls within the convex hull defined by the soles of its feet. The humanoid robot's motion space is highly complex and multi-dimensional, making its dynamics nonlinear. To address this complexity, researchers often approximate the robot's dynamics using point mass models. The linear inverted pendulum model (LIPM) dynamics are widely used, assuming that the entire robot mass is concentrated at its center of mass (CoM) [28; 29; 30]. Although simplified, this model has proven valuable for stabilization and gait control in humanoid robots [31; 32; 33; 34; 35]. By employing the LIPM, the idea is to exploit the dynamic relationship between the robot's CoM and the ZMP to generate whole-body motion while ensuring that ZMP stays within the support polygon [33]. Considering the dynamics of the LIPM where the whole mass is concentrated at \(\mathbf{x}_{CoM}\), the following relation between the CoM and ZMP can be obtained  
\[\bar{x}_{CoM}=\omega(x_{CoM}-x_{ZMP}), \tag{1}\]  
\[\bar{y}_{CoM}=\omega(y_{CoM}-y_{ZMP}). \tag{2}\]  
Assuming the constant height of the point mass \(z_{c}\), we can compute \(\omega=\sqrt{g/z_{c}}\), the natural frequency of the linear inverted pendulum, where \(g\) is the gravity constant. Here, the \(z\) axis is assumed to be parallel to the direction of gravity. Alternatively, instead of assuming constant height for the center of mass, the \(z\) component of the CoM dynamics can be controlled by defining the natural frequency as \(\omega=\sqrt{\frac{\bar{x}_{CoM}+g}{z_{CoM}-z_{ZMP}}}\). In essence, this approach closely resembles the dynamics of the linear inverted pendulum, allowing for effective ZMP control by controlling the center of mass motion. Based on the relationship between the center of mass and the ZMP given by Equations (1) and (2), [PERSON] et al. [33] showed that using any control method for the inverted pendulum control, a reference ZMP can be maintained within the support polygon by controlling the CoM. By employing a simple P-controller, the reference ZMP velocity can be calculated so that the measured ZMP is being pushed toward the center of the support polygon, \(\mathbf{x}_{CSP}=[x_{CSP},y_{CSP}]\), as  
\[\dot{x}_{ZMP}^{ref} = \gamma(x_{CSP}-x_{ZMP}), \tag{3}\] \[\dot{y}_{ZMP}^{ref} = \gamma(y_{CSP}-y_{ZMP}), \tag{4}\]  
where \(\gamma\) is a positive gain. Equations (3) and (4) are integrated to obtain the new reference ZMP. Then, using the equations of motion (1) and (2), referential dynamics for the horizontal part of the center of mass can be calculated using the referential ZMP as  
\[\dot{x}_{CoM}^{ref} = \omega(x_{CoM}-x_{ZMP}^{ref}), \tag{5}\] \[\dot{y}_{CoM}^{ref} = \omega(y_{CoM}-y_{ZMP}^{ref}). \tag{6}\]  
The desired dynamics of the center of mass can be attained by integrating the Equations (5) and (6).