# Latent Structure Matching for Knowledge Transfer in Reinforcement Learning
## 4 Experiments
### Experiments on Mountain Car Problem  
These experiments are conducted on the mountain car [39; 40] environment under the classical control problem of the OpenAI gym environment library. The mountain car problem belongs to the continuous control problem. As shown in Figure 6, in the experimental environment, a car is located on a two-dimensional track between two mountains; it is required to slide down from the left side of the mountain and rush to the target position on the right side of the mountain.  
The detailed settings about the mountain car environment are listed below:  
* _Action_: The action space is one-dimensional and discrete. The car can perform three actions, including leftward, rightward, and neutral, namely: \[Action=(leftward,neutral,rightward).\]
* _Reward_: Each time the car arrives at a state, it obtains a reward of \(-1.0\).  
Figure 5: The learning curves of LSM-based and \(\epsilon\)-greedy exploration strategies with different values of \(\epsilon\) ( 0.9, 0.7, and 0.5 from left to right). The curves are smoothed with a window of 100 episodes with an error bar at every 100 episodes.  
Figure 6: The mountain car environment. The purpose of the agent is to start from the starting position marked “Start”, and use the energy obtained by sliding down from the left side of the mountain to rush to the position marked “Goal” on the right side of the mountain.  
* _State_: The state space is two-dimensional and continuous. A state consists of speed and position, defined as: \[\begin{cases}Velocity=(-0.007,0.007)\\ Position=(-1.2,0.6)\end{cases}\]
* _Initial state_: The position marked \"Start\" in Figure 6 denotes the initial state of the car. The initial state is set as: \[\begin{cases}Position=-0.5\\ Velocity=0.0\end{cases}\]
* _Goal state_: The position marked \"Goal\" in Figure 6 is the target state of the car. The target state is set as: _Position_\(\geq\) 0.5.
* _State update rule_: A state is updated as the rule: \[\begin{cases}Velocity=Velocity+0.001\times(Action-1)-0.0025\times cos(3\times Position )\\ Position=Position+Velocity\end{cases}\]  
As state values in these experiments are continuous, we use the RBF network to approximate action value functions instead of the tabular representation. The parameters of estimated action value functions of source and target tasks are the input in LSM, used to find the matched source tasks, from which knowledge is transferred to improve the \(\epsilon\)-greedy exploration strategy.  
We compare the traditional \(\epsilon\)-greedy exploration strategy with our strategy, namely the LSM-based exploration strategy, and the experimental results are given in Figure 7. The learning curves are smoothed with a window of five episodes with an error bar at every five episodes. According to the evaluation criteria for knowledge transfer methods in reinforcement learning proposed by [PERSON] [3], our strategy performs better than the initial performance, called Jumpstart [3], and the final performance, called Asymptotic Performance [3], of the \(\epsilon\)-greedy exploration strategy. In the mountain car environment, because the agent using \(\epsilon\)-greedy exploration strategy explores randomly, it needs to try many times to know which speed it should take to get to the left side of the mountain so that it can use the energy of gliding to reach the right side of the mountain. Further, because of the experience gained in previous tasks, the agent using LSM-based exploration strategy can find a good speed at the beginning, and then uses the feedback from the environment to quickly find the optimal speed. Therefore, our method can achieve better performance at the beginning and the end.  
Figure 7: Learning curves for the mountain car problem. The curves are smoothed with a window of five episodes with an error bar at every five episodes.