## 3 PDM and Parameter Identification
### RL-Based Parameter Identification Method  
To improve the model accuracy while simplifying the identification process, a parameter identification method integrating CAD technology, least squares estimation, and reinforcement learning is presented. There are two stages, namely, preliminary identification and reinforcement learning-based parameter tuning. 1. Preliminary identification stage  
The parameters within a flexible dynamic model can be categorized into three distinct groups based on their inherent characteristics and origins: those associated with inertia, those related to stiffness, and those pertaining to friction. The inertia-related parameters encompass the mass and the distribution of weight across each individual link and joint. Link inertias can be extracted from the CAD model, and joint parameters can be queried from product manuals. Similarly, stiffness parameters can be obtained from the manuals of  
Figure 3: Elastic joint model. reducers. However, these obtained parameters are only statistically accurate but with error distributions, which needs further fine-tuning for specific individuals. Friction characteristics, particularly the reducer, are influenced by a variety of variables and exhibit substantial individual variation. Therefore, joint-by-joint studies are required to determine the friction parameters. When the motor rotates at a constant velocity, the inertia force can be neglected, although the links and some components of the reducer might still have a small amount of acceleration due to the nonlinear friction and stiffness. In this situation, the following equation for the i-th joint can be derived from Equations (2)-(6):  
\[\tau_{smi}\text{sgn}(\dot{q}_{mi})+\sigma_{vmi}\dot{q}_{mi}+[\tau_{ci}+(\tau_ {si}-\tau_{ci})e^{-(\frac{\dot{q}_{di}}{4\alpha i})^{1.5}}]\text{sgn}(\dot{q}_ {ei})/R+\sigma_{ei}\dot{q}_{ei}/R=\tau_{mi} \tag{7}\]  
where the motor torque \(\tau_{mi}\) can be collected from the servo driver. Based on a series of constant velocity experiments, \(\dot{q}_{mi}\), \(\dot{q}_{ei}\), and the corresponding \(\tau_{mi}\) can be obtained. The LS criterion is then utilized to estimate the friction parameters. (b) Reinforcement learning-based parameter tuning stage  
After preliminary identification, independent parameters are obtained instead of the minimum parameter set. Nevertheless, a significant discrepancy exists between the initial estimated parameters and their actual counterparts. [PERSON] et al. proposed a method for robot joint parameter recognition based on a cuckoo search backpropagation neural network to improve the accuracy of robot dynamic recognition in unknown parameter scenarios [39]. [PERSON] et al. studied the automatic adjustment of reward function in gait control of a quadruped robot based on reinforcement learning and a near-end strategy optimization algorithm [40]. [PERSON] and [PERSON] developed a reinforcement learning approach to drive cable-driven parallel robots with uncertain parameters and redundant constraints [41]. In this stage, parameter tuning based on continuous action reinforcement learning automata (CARLA) [42] is conducted to improve the model accuracy. CARLA is a parameter optimization algorithm based on the RL framework and has been utilized in parameter selection of the controller [43]. The principle of CARLA is probability-based trial and error. Each parameter is assigned a probability density (PD) function, and each update of the parameters is referred to as an action. The actions of all parameters are firstly determined by random selection and PD. Then, the performance evaluation module is employed to evaluate the parameters and update each PD function. With several iterations, the cost function is minimized, and the parameter values with maximum PD can be selected.