# Latent Structure Matching for Knowledge Transfer in Reinforcement Learning
## Appendix A  
In this supplementary material, we describe the steps of the reconstruction coefficient matrix mentioned in Section 3. First, Singular value decomposition (SVD) [41] is used to decompose \(Q\) into the form \(Q=UKV^{T}\), where \(U\) is the left singular matrix having only a single element, \(K=[k_{i}]_{1\times L}\) is the singular value matrix, and \(V=[v_{ij}]_{L\times L}\) is the right singular matrix. According to the value of elements in \(K\), we divide \(K\) into two submatrices, namely \(K_{1  
the \(\epsilon\)-greedy exploration strategy, and \(\pi^{{}^{\prime}}\) to denote the modified exploration strategy with knowledge transfer, namely LSM-based exploration strategy. \(Q_{j}^{\pi}\) is used to denote the action value function with the policy \(\pi\) on task \(j\). Suppose we have learnt action value functions \(Q_{i}^{\pi}\) and \(Q_{i}^{\pi^{{}^{\prime}}}\). To examine the performance improvement of LSM-based exploration strategy, we compare it with traditional \(\epsilon\)-greedy exploration strategy by calculating their difference. Based on the Bellman equation [42], we have  
\[Q_{i}^{\pi}(s,a)-Q_{i}^{\pi^{{}^{\prime}}}(s,a)\] \[=R_{i}(s,a)+\gamma\sum_{s^{\prime}}p(s^{\prime}|s,a)\sum_{a^{ \prime}}\pi(a^{\prime}|s^{\prime})Q_{i}^{\pi}(s^{\prime},a^{\prime})-R_{i}(s,a )-\gamma\sum_{s^{\prime}}p(s^{\prime}|s,a)\sum_{a^{\prime}}\pi^{{}^{\prime}}(a ^{\prime}|s^{\prime})Q_{i}^{\pi^{{}^{\prime}}}(s^{\prime},a^{\prime})\] \[=\gamma\sum_{s^{\prime}}p(s^{\prime}|s,a)\sum_{a^{\prime}}\pi(a^{ \prime}|s^{\prime})Q_{i}^{\pi}(s^{\prime},a^{\prime})-\gamma\sum_{s^{\prime}} p(s^{\prime}|s,a)\sum_{a^{\prime}}\pi^{{}^{\prime}}(a^{\prime}|s^{\prime})Q_{i}^{ \pi^{{}^{\prime}}}(s^{\prime},a^{\prime})\] \[=\gamma\sum_{s^{\prime}}p(s^{\prime}|s,a)[\sum_{a^{\prime}}\pi(a^ {\prime}|s^{\prime})Q_{i}^{\pi}(s^{\prime},a^{\prime})-\sum_{a^{\prime}}\pi^{{ }^{\prime}}(a^{\prime}|s^{\prime})Q_{i}^{\pi^{{}^{\prime}}}(s^{\prime},a^{ \prime})]\]  
Then, according to the exploration rules of the \(\epsilon\)-greedy exploration strategy and the LSM-based exploration strategy as mentioned above, the above equation is rewritten as  
\[Q_{i}^{\pi}(s,a)-Q_{i}^{\pi^{{}^{\prime}}}(s,a)\] \[=\gamma\sum_{s^{\prime}}p(s^{\prime}|s,a)[(1-\epsilon)maxQ_{i}^{ \pi}(s^{\prime},a^{\prime})+\epsilon/w\sum_{a^{\prime}}Q_{i}^{\pi}(s^{\prime },a^{\prime})-(1-\epsilon)maxQ_{i}^{\pi^{{}^{\prime}}}(s^{\prime},a^{\prime}) -\epsilon maxQ_{j}^{\pi}(s^{\prime},a^{\prime})]\] \[=\gamma\sum_{s^{\prime}}p(s^{\prime}|s,a)[(1-\epsilon)maxQ_{i}^ {\pi}(s^{\prime},a^{\prime})-(1-\epsilon)maxQ_{i}^{\pi^{{}^{\prime}}}(s^{ \prime},a^{\prime})]+\gamma\sum_{s^{\prime}}p(s^{\prime}|s,a)[\epsilon/w\sum_{a ^{\prime}}Q_{i}^{\pi}(s^{\prime},a^{\prime})-\epsilon maxQ_{j}^{\pi}(s^{ \prime},a^{\prime})]\] \[\leq\gamma\sum_{s^{\prime}}p(s^{\prime}|s,a)[(1-\epsilon)maxQ_{i} ^{\pi}(s^{\prime},a^{\prime})-(1-\epsilon)maxQ_{i}^{\pi^{{}^{\prime}}}(s^{ \prime},a^{\prime})]+\gamma\sum_{s^{\prime}}p(s^{\prime}|s,a)\sum_{a^{\prime}} Q_{i}^{\pi}(s^{\prime},a^{\prime})-\epsilon/w\sum_{a^{\prime}}Q_{j}^{\pi}(s^{ \prime},a^{\prime})]\] \[\leq\gamma\sum_{s^{\prime}}p(s^{\prime}|s,a)(1-\epsilon)max[Q_{i} ^{\pi}(s^{\prime},a^{\prime})-Q_{i}^{\pi^{{}^{\prime}}}(s^{\prime},a^{\prime} )]+\gamma\sum_{s^{\prime}}p(s^{\prime}|s,a)\sum_{a^{\prime}}\epsilon/w\mid Q_{ i}^{\pi}(s^{\prime},a^{\prime})-Q_{j}^{\pi}(s^{\prime},a^{\prime})\mid\] \[=\gamma(1-\epsilon)\Delta+\gamma\sum_{s^{\prime}}p(s^{\prime}|s,a )\sum_{a^{\prime}}\epsilon/w\mid Q_{i}^{\pi}(s^{\prime},a^{\prime})-Q_{j}^{\pi }(s^{\prime},a^{\prime})\mid,\]  
where \(\Delta=max[Q_{i}^{\pi}(s^{\prime},a^{\prime})-Q_{i}^{\pi^{{}^{\prime}}}(s^{ \prime},a^{\prime})]\), and \(w\) represents the total number of actions. Since all values of \([Q_{i}^{\pi}(s^{\prime},a^{\prime})-Q_{i}^{\pi^{{}^{\prime}}}(s^{\prime},a^{ \prime})]\) satisfy the above formula, the maximum value of \([Q_{i}^{\pi}(s^{\prime},a^{\prime})-Q_{i}^{\pi^{{}^{\prime}}}(s^{\prime},a^{ \prime})]\), that is, \(\Delta\), also satisfies the above formula.