# Latent Structure Matching for Knowledge Transfer in Reinforcement Learning
## 3 Method
### Latent Structure Matching (LSM)  
The algorithmic basis of LSM is manifold alignment [32], through which LSM derives latent structures of tasks and finds matched source task for a new target task. LSM is actually one of the variants of LRE. Like normal LRE, it firstly calculates a reconstruction coefficient matrix for source and target tasks, and then minimizes the reconstruction error in low-dimensional space. However, compared with normal LRE, several modifications are made in the reconstruction error function and reconstruction coefficient matrix for efficient estimation of similarity between source and target tasks. Given action value functions learned for \(S\) source tasks, we arrange all these functions into a vector \(Q=[Q_{l}^{(s)}]_{1\times L}\), where \(Q_{l}^{(s)}\) is the \(l^{th}\) action value function for the \(s^{th}\) source task. Similarly, all of the action value functions learned for \(T\) target tasks are also arranged into the vector \(Q=[Q_{\gamma}^{(t)}]_{1\times\Gamma}\), where \(Q_{\gamma}^{(t)}\) is the \(\gamma^{th}\) action value function for the \(t^{th}\) target task. The Equation (2) can be used to derive source and target tasks' latent structures, namely the low-dimensional embeddings of action value functions \(Q\) and \(Q\). To match the embeddings in low dimension, we add an alignment item and obtain our error function:  
\[S(\Phi)=(1-\mu)\|\Phi-B\Phi\|_{F}^{2}+\mu\sum_{i,j=1}^{m}\|\Phi_{i}-\Phi_{j}\|^ {2}C_{i,j}, \tag{3}\]  
where \(\mu\in[0,1]\) is a weight parameter. The first item of (3) represents the reconstruction error, and it is consistent with the right side of (2); the second item is added to represent the alignment error. \(\Phi\in\mathbb{R}^{m\times d}\), \(B\in\mathbb{R}^{m\times m}\), and \(C\in\mathbb{R}^{m\times m}\) are all block matrices:  
\[\Phi=\left[ \begin{array}{c}E\\ \mathcal{E}\end{array} \right],B=\left[ \begin{array}{cc}\hat{Q}&0\\ 0&\hat{Q}\end{array} \right],C=\left[ \begin{array}{cc}0&C^{(Q,\mathcal{Q})} \\ C^{(Q,\mathcal{Q})}&0\end{array} \right],\]  
where \(m=L+\Gamma\) and \(d\) is the dimension of low-dimensional space. \(E\) and \(\mathcal{E}\) are the embeddings respectively for \(Q\), \(Q\). \(C^{(Q,\mathcal{Q})}\) is the proximity matrix, which can be obtained by using cosine similarity metric. \(\hat{Q}\) and \(\hat{Q}\) are the reconstruction coefficient matrices of \(Q\) and \(\mathcal{Q}\), respectively, which can be solved by minimizing (1). However, the minimization of (1) in LRE requires frequent iterative calculation. For this, we use the closed-form solution of subspace estimation [33] to calculate the  
Figure 1: Overview of the two-staged method composed of latent structure matching and value function transfer. In the first stage, latent structure matching (LSM) is used to measure the similarity between tasks, so as to find the highly-matched source task for the target task. In the second stage, value function transfer is used to improve the agent’s exploration strategy, thereby improving the agent’s performance. reconstruction coefficient matrix, thereby reducing the calculation. The specific procedure is described in Appendix A. The error function in (3) can be reduced to a sum of matrix traces:  
\[ \begin{split} S(\Phi)&=(1-\mu)tr((\Phi-B\Phi)^{T}( \Phi-B\Phi))+\mu\sum_{z=1}^{d}\sum_{i,j=1}^{m}\|\[PERSON]_{i,z}-\[PERSON]_{j,z}\|_{2}^{2} C_{i,j}\\ &=(1-\mu)tr(((\mathbb{I}-B)\Phi)^{T}(\mathbb{I}-B)\Phi)+2\mu\sum _{z=1}^{d}\[PERSON]_{,z}^{T}L\Phi_{,z}\\ &=(1-\mu)tr(\Phi^{T}(\mathbb{I}-B)^{T}(\mathbb{I}-B)\[PERSON])+2\mu tr (\Phi^{T}L\Phi).\end{split}  \tag{4}\]  
Similar to locally linear embedding [34] and LRE, we introduce the constraint \(\Phi^{T}\Phi=\mathbb{I}\) to ensure that the minimization problem of error function \(S\) is a well-posed problem.