# Latent Structure Matching for Knowledge Transfer in Reinforcement Learning
## 2 Related Work  
### Knowledge Transfer in RL  
The sample complexity and learning inefficiency of the RL algorithms are two of the most critical aspects that obstruct its feasibility in practical applications [25]. Transfer learning is introduced to address these problems. To perform effective knowledge transfer in RL, it is of great importance to select appropriate source tasks that are highly-matched with the target task. In most of the existing works, similarity estimation is the main way of selecting a matched source task. Early works were semi-automatic, as they conducted the similarity estimation of tasks according to human experience or intuition. [PERSON] and [PERSON] [26] defined one-to-one inter-task mapping (TVITM) as an action in one task and its image in another task being similar with each other. The similarity was loosely defined so that it depended on how the action affected the agent's state and what reward was received. TVITM was subsequently used by [PERSON] and [PERSON] [10] for knowledge transfer in RL to speed up learning. A many-to-one mapping for knowledge transfer in RL was presented in [11], which used a linear combination of relevant states and actions to initialize learning. However, it needed humans to provide the parameters of the linear combination, and the optimal values of the parameters were not easy to reach [12]. To avoid this, some researchers developed mechanisms for optimal mapping selection. [PERSON] and [PERSON] [13] proposed two algorithms to select the optimal mapping for both model-based and model-free RLs respectively. In another of their works [14], they proposed a method which autonomously selected mappings from a set of all possible inter-task mappings. In order to learn similarity of tasks without human intervention, some automatic works have been proposed. [PERSON] [15] measured the distance between Markov decision processes (MDPs) by constructing bisimulation metrics on rewards and probability functions. [PERSON] [16] gave a method called \"modeling approximate state transitions by exploiting regression (MASTER)\", which automatically learned a mapping from one task to another based on evidences gathered from task environments. A method using neural work to automatically map actions in one domain to actions in the other domain was proposed in [17]. [PERSON] and [PERSON] [18] defined similarity in terms of tasks, and proposed several possible task similarity measures based on the transfer time, policy overlap, Q-values, and reward structure respectively to measure the similarity between tasks. [PERSON] [19] proposed to use self-organization neural networks to make the effective use of domain knowledge to reduce model complexity in RL. Through minimizing the reconstruction error of a restricted Boltzmann machine simulating the behavioral dynamics of two compared Markov decision processes, [PERSON] and [PERSON] [20] gave a data-driven automated [PERSON] decision process similarity metric. [PERSON] and [PERSON] [21] introduced two metrics for measuring the distance between finite Markov decision processes: The Hausdorff metric measuring the distance between two subsets of a metric space, and the Kantorovich metric measuring the distance between probabilistic distributions. These works are from a purely computational perspective, so they are computationally intensive, and cannot handle scenarios with a large number of source tasks [19]. To solve the issue raised when handling large number of tasks, recent works used the clustering to learn the similarity automatically. [PERSON] and [PERSON] [23] used the intrinsic clustering property of the Dirichlet process to impose sharing of episodes among similar tasks, which effectively reduced the number of episodes required for learning a good policy in each task, when data sharing was appropriate. [PERSON] and [PERSON] [27] used a Gaussian process hierarchical BNP model to cluster different classes of source MDPs in sequential batch mode.