# Compound Heuristic Information Guided Policy Improvement for Robot Motor Skill Acquisition
## 3 Path Integral Policy Improvement with Covariance Matrix Adaption
\begin{table}
\begin{tabular}{l l} \hline \hline
**Symbol** & **Definition** \\ \hline \(\tilde{y}_{t},y_{t},y_{t}\) & The desired acceleration, velocity, position of the joint \\ \hline \(h(x_{t})\) & Fitting function. \(x_{t}\) is a phase variable \\ \hline \(p_{,k,i}\) & The probability-weighted value of the \(k\)th trajectory of the \(d\)th DOF at time \(i\) \\ \hline \(\delta\mathbf{\omega}_{d,i}\) & The \(d\)th joint’s correction weight vector at time \(i\) \\ \hline \(\delta\mathbf{\omega}_{d}\) & Average value of \(\delta\mathbf{\omega}_{d,i}\) over time \\ \hline \(\mathbf{\omega}_{d}\) & The \(d\)th joint’s weight vector \\ \hline \(\tilde{\mathbf{\omega}}_{d}\) & The \(d\)th joint’s weight vector with perturbation \\ \hline \(\tilde{\mathbf{\omega}}\) & \([\tilde{\mathbf{\omega}}_{1}^{T},\tilde{\mathbf{\omega}}_{2}^{T},\cdots,\tilde{\mathbf{ \omega}}_{D}^{T}]^{T}\) \\ \hline \(\phi(\cdot)\) & Mapping function. Mapping a variable to V-dimensional space \\ \hline \(\tilde{\mathbf{\omega}}_{d}^{s}\) & The weight perturbation sample of the \(d\)th joint, i.e., \(\tilde{\mathbf{\omega}}_{d}^{s}=\{\tilde{\mathbf{\omega}}_{d,1},\tilde{\mathbf{\omega}}_{ d,2},\cdots,\tilde{\mathbf{\omega}}_{d,N}\}\) \\ \hline \(J_{d}\) & The \(d\)th joint’s cost for given task \\ \hline \(J^{p}\) & The total cost of D joints at the \(p\)th iteration \\ \hline \(K(\cdot)\) & Kernel matrix \\ \hline \(T_{r}\) & Decline rate of cost between \(J^{p-1}\) and \(J^{p}\) \\ \hline \(\{J_{d},\delta\mathbf{\omega}_{d,i}\}_{K}\) & \(J_{d}\) and \(\delta\mathbf{\omega}_{d,i}\) which are calculated from K roll-outs \\ \hline \(\{J_{d},\delta\mathbf{\omega}_{d,i}\}_{Ke}\) & \(J_{d}\) and \(\delta\mathbf{\omega}_{d,i}\) which are calculated from \(Ke\) elite roll-outs, and \(Ke\) elite roll-outs are obtained by sorting \(K\) roll-outs according to the cost \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Summary of the notation frequently used in the article. \[P_{,k,i}=P\left(\tau_{,i,k}\right)=\frac{e^{-\frac{1}{4}\left(l\left(\tau_{,i,k} \right)\right)}}{\sum_{k=1}^{K_{\epsilon}}e^{-\frac{1}{4}\left(l\left(\tau_{,i }\right)\right)}}, \tag{5}\]  
where \(\lambda\) is an appropriate constant. Later, \(\delta\mathbf{\omega}_{d,i}\) is averaged over time to get \(\delta\mathbf{\omega}_{d}\) further to obtain the new weight (i.e., \(\mathbf{\omega}_{d}^{\text{new}}=\mathbf{\omega}_{d}^{\text{old}}+\delta\mathbf{\omega}_{d}\)). By searching the parameter space iteratively, PI\({}^{2}\) will eventually find a solution for the new task. Classical PI\({}^{2}\) only updates the mean \(\mathbf{\omega}\), and the covariance \(\sigma^{2}\) is a constant (\(\sigma^{2}=\lambda_{init}\mathbf{I}_{M}\)). \(M\) is the number of base functions. \(\lambda_{init}\) determines the magnitude of initial exploration noise. PI\({}^{2}\)-CMA aims to determine the magnitude of the exploration noise automatically and to infer the implicit linear correlation between the basis functions within the joint of the robot. In other words, covariance matrix adaption is expressed as:  
\[\Sigma_{d,i}^{new}=\sum_{k=1}^{K_{\epsilon}}P_{,k,i}(\delta\mathbf{\omega}_{d,i})( \delta\mathbf{\omega}_{d,i})^{\text{T}}. \tag{6}\]  
In addition, the covariance update equation is  
\[\Sigma_{d}^{new}=\frac{\sum_{i=1}^{N}(N-i)\Sigma_{d,i}^{new}}{\sum_{i=l}^{N}( N-l)}. \tag{7}\]  
As we can see that the vanilla PI\({}^{2}\)-CMA only infers linear correlation of weight within a DMP independently.