# Compound Heuristic Information Guided Policy Improvement for Robot Motor Skill Acquisition
## 4 PI\({}^{2}\)-CMA with Kernel Canonical Correlation Analysis
### Nonlinear Correlation Heuristic Information  
According to Equations (3) and (4), perturbation is generated with equal probability for each joint's weight vector. In other words, there is no correlation between the joints. Therefore, \([\tilde{\mathbf{\omega}}_{1}^{T},\tilde{\mathbf{\omega}}_{2}^{T},\cdots,\tilde{\mathbf{ \omega}}_{D}^{T}]^{T}\in\mathbb{R}^{DM\times 1}\) is denoted as \(\tilde{\mathbf{\omega}}\), and covariance matrix of \(\tilde{\mathbf{\omega}}\) is given by:  
\[\Sigma_{\tilde{\mathbf{\omega}}}=diag[\sigma_{1}^{2},\sigma_{2}^{2},\cdots,\sigma_ {D}^{2}]\in\mathbb{R}^{DM\times DM}, \tag{8}\]  
where \(diag[\cdot]\) represents diagonal matrix and \(\sigma_{1}^{2}=\cdots=\sigma_{D}^{2}=\sigma^{2}\). When the multi-joint robot completes the task, we believe that there are some hidden patterns between the joints. The implicit patterns between the perturbation vectors of the multi-joint robot are expressed as nonlinear correlations. With the help of the kernel method, the \(d\)th joint's perturbation vector \(\tilde{\mathbf{\omega}}_{d}\in\mathbb{R}^{M\times 1}\) is mapped to high-dimensional feature space \(\phi(\tilde{\mathbf{\omega}}_{d})\in\mathbb{R}^{V\times 1}\). \(\tilde{\mathbf{\omega}}\) is also mapped to high-dimensional feature space \(\phi(\tilde{\mathbf{\omega}})\in\mathbb{R}^{DV\times 1}\), and then covariance matrix of \(\tilde{\mathbf{\omega}}\) can be described as:\[\Sigma_{\mathbf{\phi}(\mathbf{\tilde{\omega}})}=\left[ \begin{array}{ccccc}\Gamma(\mathbf{ \tilde{\omega}}_{1},\mathbf{\tilde{\omega}}_{1})&\Gamma(\mathbf{\tilde{\omega}}_{1},\bm {\tilde{\omega}}_{2})&\cdots&\Gamma(\mathbf{\tilde{\omega}}_{1},\mathbf{\tilde{\omega}}_ {D})\\ \Gamma(\mathbf{\tilde{\omega}}_{2},\mathbf{\tilde{\omega}}_{1})&\Gamma(\mathbf{\tilde{ \omega}}_{2},\mathbf{\tilde{\omega}}_{2})&\cdots&\Gamma(\mathbf{\tilde{\omega}}_{2}, \mathbf{\tilde{\omega}}_{D})\\ \vdots&\vdots&\ddots&\vdots\\ \Gamma(\mathbf{\tilde{\omega}}_{D},\mathbf{\tilde{\omega}}_{1})&\Gamma(\mathbf{\tilde{ \omega}}_{D},\mathbf{\tilde{\omega}}_{2})&\cdots&\Gamma(\mathbf{\tilde{\omega}}_{D}, \mathbf{\tilde{\omega}}_{D})\\ \end{array} \right], \tag{9}\]  
where \(\Gamma(\mathbf{\tilde{\omega}}_{d_{i}},\mathbf{\tilde{\omega}}_{d_{j}})=cov(\mathbf{\phi} (\mathbf{\tilde{\omega}}_{d_{i}}),\mathbf{\phi}(\mathbf{\tilde{\omega}}_{d_{j}}))\) is the covariance of \(\mathbf{\tilde{\omega}}_{d_{i}}\) and \(\mathbf{\tilde{\omega}}_{d_{j}}\) projected on high-dimensional space, expressing the implicit nonlinear pattern between the perturbation vectors of the \(d_{i}\)th joint and the \(d_{j}\)th joint. The intuitive policy is to obtain \(\Sigma_{\mathbf{\phi}(\mathbf{\tilde{\omega}})}\) based on empirical samples, and then use \(\Gamma(\mathbf{\tilde{\omega}}_{d_{i}},\mathbf{\tilde{\omega}}_{d_{j}})\) as the heuristic information. That is, given the perturbation vector \(\mathbf{\tilde{\omega}}_{d_{i}}\), the exploration of \(\mathbf{\tilde{\omega}}_{d_{j}}\) is guided by the covariance \(\Gamma(\mathbf{\tilde{\omega}}_{d_{j}},\mathbf{\tilde{\omega}}_{d_{j}})\). Note that \(\Gamma(\mathbf{\tilde{\omega}}_{d_{i}},\mathbf{\tilde{\omega}}_{d_{j}})\) is in the unified space coordinate system (i.e., \(\phi(\mathbf{\tilde{\omega}})\)), but analyzing the correlation between the perturbation vector \(\mathbf{\tilde{\omega}}_{d_{i}}\) of the \(d_{i}\)th joint and the perturbation vector \(\mathbf{\tilde{\omega}}_{d_{j}}\) of the \(d_{j}\)th joint in a uniform coordinate system is not the best choice. Because after the perturbation vectors of different joints are mapped to a high-dimensional space, the correlation coefficients between the joints can be maximized only after a proper projection transformation, and such projection matrices are usually not necessarily identical.