# Compound Heuristic Information Guided Policy Improvement for Robot Motor Skill Acquisition
## 4 PI\({}^{2}\)-CMA with Kernel Canonical Correlation Analysis
### Robot Intelligent Trajectory Inference with KCCA  
KCCA [14] is a nonlinear correlation analysis method. In this paper, we employ KCCA on the elite samples from the robot's first joint to other joints, and make heuristic inference. After the \(p\)th iteration of PI\({}^{2}\)-CMA, the program records the total reward \(J^{p}(\tau)\) (one episodic samples), based on the current \(\mathbf{\tilde{\omega}}^{p}_{1,\cdots,D}\) (perturbation vectors of D joints). Then, \(J^{p}(\tau)\) is compared with the total reward \(J^{p-1}(\tau)\) based on \(\mathbf{\tilde{\omega}}^{p-1}_{1,\cdots,D}\) at the (\(p\)-1)th iteration to obtain a decline rate \(T_{r}\). If \(T_{r}\) is greater than its upper threshold \(T_{max}\), \(T_{r}\) is updated to \(T_{max}\), and the program executes KCCA learning. At this time, the weight perturbation sample \(\mathbf{\tilde{\omega}}^{s}_{1}=\{\mathbf{\tilde{\omega}}_{1,1},\mathbf{\tilde{\omega}}_{1,2 },\cdots,\mathbf{\tilde{\omega}}_{1,N}\}\) on the first joint and the weight perturbation sample \(\mathbf{\tilde{\omega}}^{s}_{2}=\{\mathbf{\tilde{\omega}}_{2,1},\mathbf{\tilde{\omega}}_{2,2 },\cdots,\mathbf{\tilde{\omega}}_{2,N}\}\) on the second joint in the \(p\)th iteration are respectively mapped in high dimensions to obtain \([\mathbf{\phi}(\mathbf{\tilde{\omega}}_{d,1})\quad\mathbf{\phi}(\mathbf{\tilde{\omega}}_{d,2}) \quad\cdots\quad\mathbf{\phi}(\mathbf{\tilde{\omega}}_{d,N})]\), denoted as \(\Phi(\mathbf{\tilde{\omega}}^{s}_{d})\in\mathbb{R}^{V\times N}(d\in[1,2])\). Next, find two sets of vectors \(\mathbf{w_{1}}\), \(\mathbf{w_{2}}\in\mathbb{R}^{V\times 1}\), so that the correlation coefficient between the data \(u\) and \(v\) after the projection of \(\mathbf{\phi}(\mathbf{\tilde{\omega}}_{1})\) and \(\mathbf{\phi}(\mathbf{\tilde{\omega}}_{2})\) is maximized. We will have:  
\[ \begin{split} u&=\mathbf{w}^{T}_{1}\mathbf{\phi}(\mathbf{\tilde{\omega}}_{1}),\\ v&=\mathbf{w}^{T}_{2}\mathbf{\phi}(\mathbf{\tilde{\omega}}_{2}).\\ \end{split}  \tag{11}\]  
The covariance of \(u\) and \(v\) is given by:  
\[var(u,v)=\frac{1}{N-1}\mathbf{w}^{T}_{1}\mathbf{\phi}(\mathbf{\tilde{\omega}}^{s}_{1})\mathbf{ \phi}(\mathbf{\tilde{\omega}}^{s}_{2})^{T}\mathbf{w}_{2}. \tag{12}\]Obviously, the vectors \(\mathbf{w}_{1}\) and \(\mathbf{w}_{2}\) are located in the space spanned by data \(\[PERSON](\tilde{\mathbf{\omega}}_{1}^{s})\) and \(\Phi(\tilde{\mathbf{\omega}}_{2}^{s})\):  
\[\mathbf{w}_{1} =\[PERSON](\tilde{\mathbf{\omega}}_{1}^{s})\mathbf{\alpha}_{1}, \tag{13}\] \[\mathbf{w}_{2} =\Phi(\tilde{\mathbf{\omega}}_{2}^{s})\mathbf{\alpha}_{2},\]  
where \(\mathbf{\alpha}_{1}\), \(\mathbf{\alpha}_{2}\in\mathbb{R}^{N\times 1}\). From Equations (12) and (13), correlation coefficient \(\rho\) can be obtained:  
\[\rho =\frac{\mathbf{\alpha}_{1}^{T}\[PERSON]^{T}(\tilde{\mathbf{\omega}}_{1}^{s}) \[PERSON](\tilde{\mathbf{\omega}}_{1}^{s})\[PERSON]^{T}(\tilde{\mathbf{\omega}}_{2}^{s})\[PERSON]( \tilde{\mathbf{\omega}}_{2}^{s})\mathbf{\alpha}_{2}}{\sqrt{\mathbf{\alpha}_{1}^{T}(\[PERSON]^ {T}(\tilde{\mathbf{\omega}}_{1}^{s})\[PERSON](\tilde{\mathbf{\omega}}_{1}^{s}))^{2}\mathbf{ \alpha}_{1}}\sqrt{\mathbf{\alpha}_{2}^{T}(\[PERSON]^{T}(\tilde{\mathbf{\omega}}_{2}^{s}) \[PERSON](\tilde{\mathbf{\omega}}_{2}^{s}))^{2}\mathbf{\alpha}_{2}}} \tag{14}\] \[=\frac{\mathbf{\alpha}_{1}^{T}K_{\omega_{1}}K_{\omega_{2}}\mathbf{\alpha }_{2}}{\sqrt{\mathbf{\alpha}_{1}^{T}K_{\omega_{1}}K_{\omega_{1}}\mathbf{\alpha}_{1}} \sqrt{\mathbf{\alpha}_{2}^{T}K_{\omega_{2}}K_{\omega_{2}}\mathbf{\alpha}_{2}}}.\]  
Kernel method is introduced in Equation (14), where \(K(\cdot)\) is a kernel matrix. Without loss of generality, we fixed the denominator \(\mathbf{\alpha}_{1}^{T}K_{\omega_{1}}K_{\omega_{1}}\mathbf{\alpha}_{1}=1\), \(\mathbf{\alpha}_{2}^{T}K_{\omega_{2}}K_{\omega_{2}}\mathbf{\alpha}_{2}=1\) of Equation (14) to find a suitable \(\mathbf{\alpha}_{1}\) and \(\mathbf{\alpha}_{2}\) to maximize \(\mathbf{\alpha}_{1}^{T}K_{\omega_{1}}K_{\omega_{2}}\mathbf{\alpha}_{2}\).