# Latent Structure Matching for Knowledge Transfer in Reinforcement Learning
## 1 Introduction  
Reinforcement learning (RL) is where an agent guides its actions based on the rewards obtained from the trial-and-error interaction with the environment [1; 2]. The ultimate goal of RL is to achieve a mapping from states to actions with maximum return [3]. In recent years, reinforcement learning has been successfully applied to some aspects of life such as medical care [4], finance [5], and transportation [6]. However, in RL, a small change on the problem configuration usually leads to a complete new training, which inevitably brings a slow convergence and the growing requirement of a large amount of empirical samples. The introduction of transfer learning [7] in RL provides a promising solution to this problem. Transfer learning guides the transfer of useful knowledge from well-learned source tasks to target tasks [8]. In RL, the knowledge obtained from previous situations can be reused as heuristics to achieve effective knowledge transfer, thus speeding up the learning procedure in new situations and reducing sample request [3]; moreover, knowledge transfer is able to mitigate much the issue caused by a change on the problem configuration as mentioned above. However, if an inappropriate source task is selected in knowledge transfer, it will lead to negative transfer [9], which will slow down or even disrupt the learning of a new target task. Therefore, it is very important for knowledge transfer to select appropriate source tasks that have a high degree of matching with target tasks. The similarity estimation of tasks is the main way to select matched source tasks in the existing works on knowledge transfer for RL. The works proposed at the beginning are semi-automatic, because they require human intervention for similarity estimation. A typical work [10] defined a one-to-one mapping between states and actions to estimate similarity. In the subsequent work [11], a many-to-one mapping of relevant states and actions was defined. These semi-automatic works require manual parameterization of defined mappings; nevertheless, the optimal values of the parameters are not easy to reach by experience or intuition [12]. In order to overcome this problem, some mechanisms [13; 14] to select optimal mappings were developed. However, human intervention is still needed in these works [12]. Later, in order to solve the problems in the earlier works mentioned above, some automatic similarity estimation works [15; 16; 17; 18; 19; 20; 21] are presented. These works made use data-driven similarity metrics, including the Markov decision process (MDP) similarity metric [20], the Hausdorff metric [21], and the Kantorovich metric [21]. These works are highly interpretative, but computationally expensive and have trouble when handling situations involving a large number of tasks [19]. Recently, clustering algorithms [22] were used in some works to tackle large number of tasks. In these works, the clustering of policies, value functions, rewards, and dynamics of tasks, were modeled as random process to estimate the similarity [23; 24]. However, these clustering-based works suffer from the bias from the modeling for clustering, and are sensitive to noise and variation in policies, dynamics, and value functions of tasks, resulting in the inaccurate estimation of task similarity. The classification and characteristics of existing similarity estimation methods mentioned above are summarized in Table 1.  
To address the limitation of existing works, we propose a novel task matching algorithm, namely latent structure matching (LSM), which uses low-dimensional embedding to extract latent essential factors and relationships of tasks. Furthermore, through low-dimensional alignment, LSM effectively matches tasks in large number, and estimates their similarity, thus finding highly-matched source tasks. Based on the matched source tasks, useful exploratory advice is formed to improve exploration strategy, and to accelerate the learning of target tasks.