# Memory-Enhanced Knowledge Reasoning with Reinforcement Learning
## 5 Experimental Results and Analysis
### Path Diversity Exploration  
To compute the total number of unique paths explored by the agent during training and visualize their changes, we include edge labels and intermediate entities when calculating unique paths. This is shown in Figure 13.  
\begin{table}
\begin{tabular}{c c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{4}{c}{**NELL-995**} & \multicolumn{4}{c}{**FB15K-237**} & \multicolumn{4}{c}{**KINSHIP**} \\ \cline{2-13}  & **@1** & **@3** & **@10** & **MRR** & **@1** & **@3** & **@10** & **MRR** & **@1** & **@3** & **@10** & **MRR** \\ \hline ConvRL-KGM & 69.9 & 81.6 & 86.9 & 75.1 & 36.5 & 45.8 & 59.5 & 42.7 & 79.2 & 92.6 & 98.2 & 86.9 \\ ConvRL-KGM (-ConvR) & 65.6 & 79.2 & 84.4 & 72.7 & 34.6 & 44.3 & 58.1 & 42.1 & 78.9 & 92.7 & 98.0 & 84.5 \\ ConvRL-KGM (-SLSTM) & 63.7 & 75.6 & 83.2 & 72.1 & 33.1 & 43.8 & 57.2 & 41.2 & 77.3 & 90.2 & 96.1 & 84.9 \\ ConvRL-KGM (self-attention) & 68.9 & 80.2 & 84.5 & 74.1 & 34.8 & 44.3 & 58.3 & 41.6 & 78.9 & 91.9 & 97.2 & 86.2 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Results of ablation experiments.  
Figure 13: Number of unique paths in KINSHIP, NELL-995, and FB15K-237.  
In Figure 13, it is observed that a large number of paths need to be explored by the agent before reaching a good performance level across all datasets. As training progresses, the speed of path discovery slows down. On the smaller \(\mathcal{G}\) (KINSHIP), the rate of encountering new paths decreases significantly after a certain number of epochs, and the development set accuracy tends to stabilize accordingly. On larger \(\mathcal{G}\)s (FB15k-237 and NELL-995), there is no clear slowdown before severe overfitting occurs and development set performance begins to decrease. One possible reason is that larger \(\mathcal{G}\)s are more sparsely connected, so the efficiency of obtaining generalizable knowledge from \(\mathcal{G}\) by sampling and exploring a limited proportion of the path space is lower.  
Although removing some modules significantly reduces the effectiveness of path exploration, removing the ConvR reward shaping slightly improves the paths accessed during training across all datasets. This suggests that the correlation between explored paths and development set performance is not strictly positive. Typically, the best-performing model is not the one that explores the most paths.  
Influence of maximum path length on prediction datasets  
In the ConvRL-KGM framework, the maximum length of the pre-specified path is provided for the agent to search. The influence of different maximum path lengths was studied through experiments. The agent can traverse the graph in a shorter number of steps for datasets with shorter path lengths, while larger datasets require longer steps. Experiments were conducted on the maximum lengths of [2; 3; 4; 5] on NELL-995. The results are shown in Figure 14.  
Figure 14: _Cont._  
For larger datasets, the growth rate of performance slows down when the maximum length of the path exceeds three. Although the agent does not need excessively long paths, it can still use different strategies to reach the correct answer, showing strong flexibility.