## 3 PDM and Parameter Identification
### RL-Based Parameter Identification Method
Parameter tuning based on CARLA is illustrated in Figure 4, and the detailed process is as follows:  
Step 1: Initialize the learning space of each parameter  
Assuming the parameter set in Equations (2)-(6) is \(X=\{x_{i}|i=1,2,\ldots,m\}\) and represents the i-th independent parameter, a value range \([x_{\text{imin}},x_{\text{imax}}]\) called the learning space needs to be determined, which is used to limit the tuning process and is of great significance in improving the convergence rate and stability. The preliminary parameters acquired previously are taken as the intermediate values to construct the learning space. Since CAD models usually have high accuracy, the actual inertia-related parameters are close to the preliminary values. Hence, the corresponding learning spaces can be configured to smaller ranges. The stiffness- and friction-related parameters always have bigger gaps compared to the actual values. And the boundary of learning spaces should be set far from the initial values. Then, Step 2 is performed. Step 2: Define cost function based on dynamic model  
Minimizing the cost function \(J(X)\) is the target of parameter tuning. Hence, the cost function should be relevant to all unknown parameters. The observable values in experiments include motor torque \(\tau_{mi}\) and motion parameters \(q_{mi}\) and \(q_{ei}\). Therefore, the gap between \(\tau_{mi}\) and the simulated toque \(\hat{\tau}_{mi}\) from the dynamic model can be used as the cost function based on the LS criterion, as follows:  
\[J(X)=\sqrt{\sum_{i=1}^{n}{\left({\tau_{mi}-\hat{\tau}_{mi}}\right)^{2}}} \tag{8}\]  
However, as can be seen from Equations (2)-(4), Equation (8) does not contain stiffness parameters. Thus, the weighted cost function is defined as follows:  
\[J^{*}(X)=\sqrt{\left({1-e^{-\lambda}}\right){\sum_{i=1}^{n}{\left({\tau_{mi}- \hat{\tau}_{mi}}\right)^{2}}+e^{-\lambda}}{\sum_{i=1}^{n}{\left({\hat{\tau}_{ ei}-\hat{\tau}_{ei}^{\prime}}\right)^{2}}}} \tag{9}\]  
where \(\hat{\tau}_{ei}\) and \(\hat{\tau}_{ei}^{\prime}\) are the simulated joint torque obtained from Equations (2) and (3), respectively, and \(\lambda\) is the weighting factor and can be selected from 0.8 to 1.4. Then, Step 3 is performed. Step 3: Excite robot to collect training dataset  
A dataset \(\Phi=\{q_{e},q_{m},\tau_{m}\}\) needs to be constructed for the training of [PERSON]. A series of excitation trajectories are constructed based on the Fourier series, where the amplitude and frequency can be selected randomly without optimization. The excitation trajectory is given as follows:  
\[q_{mi,d}^{*}(t)=\sum_{j=1}^{N}{\frac{1}{{j2\pi f_{i,d}}}\left[{a_{i,d_{j}}\sin \left({j2\pi f_{i,d}t}\right)+b_{i,d_{j}}\cos\left({j2\pi f_{i,d}t}\right)} \right]} \tag{10}\]  
where \(q_{mi,d}^{*}(t)\) is the d-th target trajectory of the i-th motor, \(f_{i,d}\) is the basic frequency, and \(a_{i,d_{j}}\) and \(b_{i,d_{j}}\) are the amplitudes. Through a series of excitation experiments, \(\Phi\) can be collected. Concurrently, the actual velocity and acceleration of the motor and joint can be derived through the differentiation and subsequent filtering of \(\Phi\). Then, Step 4 is performed. Figure 4: Flowchart of parameter tuning based on CARLA. Step 4: Tune parameters with CARLA  
Initially, the distribution of parameter \(x_{i}\) in learning space \([x_{i\text{min}},x_{i\text{max}}]\) is assumed to be uniform. The corresponding PD of \(x_{i}\) can be calculated by  
\[f_{i,k=1}(x_{i})=\left\{ \begin{array}{l}\frac{1}{x_{i\text{max}}-x_{i\text{ min}}},x_{i}\in[x_{i\text{min}},x_{i\text{max}}]\\ 0,\text{ otherwise}\end{array} \right. \tag{11}\]  
where k is the iteration number and k = 1 refers to the initial state. Hence, the probability of \(x_{i}=x_{ik}\) where \(x_{ik}\in[x_{i\text{min}},x_{i\text{max}}]\) can be given by  
\[p_{i}(k)=\int_{x_{i\text{min}}}^{x_{ik}}f_{i,k}(x_{i})dx_{i} \tag{12}\]  
In each iteration, \(p_{i}(k)\) is selected randomly, and the corresponding \(x_{ik}\) can be calculated by Equation (12), which is the so-called action.