# Latent Structure Matching for Knowledge Transfer in Reinforcement Learning
## 4 Experiments
### Experiments on Maze Navigation Problem  
These experiments are conducted on a 10\(\times\)10 maze navigation environment in two-dimensional space, as shown in Figure 2. An agent is required to walk from the start point to the target point, while avoiding all obstacles in the path and maximizing the total rewards obtained. The detailed settings about the maze navigation environments are listed below. * _Action_: The action space is one-dimensional and discrete. The agent can perform four actions: Upward, downward, leftward, and rightward. * _State_: The state space is also one-dimensional and discrete. Since each grid in the maze represents one state, there are a total of 100 states. * _Reward_: When the agent reaches the target state, it will receive a reward of +10; when the agent encounters an obstacle, it will get a penalty of \(-10\); and when the agent is in any other state, it will get a living penalty of \(-0.1\). We design two maze environments as shown in Figure 2. In two maze environments, the action value functions obtained from source tasks (the grids marked in blue in Figure 1(a)) and the action value functions from target tasks (the grids marked in yellow in Figure 1(a),(b)) are used as the input of LSM. The experimental results are shown in Figure 3, the learning curves are smoothed with a window of 100 episodes with an error bar at every 100 episodes. The left curve shows the effect of knowledge transfer under the same environment settings, and the right curve shows the effect of knowledge transfer under different environment settings. It can be seen from Figure 3 that LSM-based modified \(\epsilon\)-greedy exploration strategy performs significantly better than normal \(\epsilon\)-greedy exploration strategy. For two exploration strategies, the value of \(\epsilon\) is set as 0.9. Figure 2: The maze navigation environments of two different settings. The red, yellow and gray grids in the figure denote respectively the starting positions, the target positions, and the obstacles. The goal of the agent is to walk from the start point (the red grids) to the target point (the yellow grids). To have an insight into the effect of \(\epsilon\), we change the value of \(\epsilon\) in the range \([0.5,0.9]\) to observe the performance variation of two exploration strategies, and the results are given in Figures 4 and 5. The upper part of Figure 4 shows the average return of 2000 episodes obtained by changing the value of \(\epsilon\) under the same environment settings, and the lower part of Figure 4 shows the average return of 2000 episodes obtained by changing the value of \(\epsilon\) under different environment settings. As can be seen, compared with \(\epsilon\)-greedy exploration strategy, the LSM-based exploration strategy can always achieve a higher average return regardless of the value of \(\epsilon\). Figure 5 shows the variation in average return of each episode obtained by changing the value of \(\epsilon\) under the same environment settings and different environment settings, respectively. When the value of \(\epsilon\) is set to 0.7 or 0.5, the Q-Learning algorithm using \(\epsilon\)-greedy exploration strategy almost stops learning; however, the Q-Learning algorithm using LSM-based exploration strategy continues to perform well whether under the same environment settings or different environment settings. The reason for this may be that the increase of randomness in action selection greatly will reduce the possibility of taking sequent actions to find valid paths to target state; the good performance of the  
Figure 4: The comparison of the average return of 2000 episodes using the LSM-based and the \(\epsilon\)-greedy exploration strategies, and the values of \(\epsilon\) are 0.9, 0.7, and 0.5 respectively from left to right. The upper part shows the results under the same environment settings and the lower part shows the results under different environment settings.