# Causal Reinforcement Learning for Knowledge Graph Reasoning
## 1 Introduction  
In recent years, knowledge graphs have been widely used in several fields. However, for various professional fields, the constructed knowledge graphs are usually sparse and the implicit relationship between entities has not been fully examined, resulting in a poor practical application effect. Therefore, it is necessary to complete the knowledge graphs; the typical method is knowledge graph reasoning. Knowledge graph reasoning tasks include Knowledge Graph Completion (KGC), quality verification, path prediction, relationship reasoning, and conflict detection [1]. This paper mainly focuses on KGC and transforms the knowledge graph reasoning problem into a problem predicting the missing part of the triple. For example, as depicted in Figure 1, we obtain two paths based on the existing knowledge graph: \"[PERSON]\(\frac{\text{Born in}}{\text{London}}\)\" and \"London\(\frac{\text{Located in}}{\text{Britain}}\)\". Our task is to complete the triple: ([PERSON], nationality,?). Since we do not have the path \"[PERSON]\(\frac{\text{Nationality}}{\text{British}}\)\", we use the dotted line to represent the edge of the relationship. From knowledge graph reasoning, the above two paths are equivalent to \"[PERSON]\(\frac{\text{Nationality}}{\text{British}}\)\". So, we complete the triple: ([PERSON], nationality, British). In the above reasoning process, we start from the Nolan node. For the relationship of \"nationality\", the association degree of the edges linked to the Nolan node differs,highlighting that \"Born in\" is more relevant to finding a relationship of nationality than \"Direct\" and \"Partner\". This is because humans have prior knowledge that \"place of birth\" is more likely to determine \"nationality\" than \"director\" and \"partner\". Reinforcement learning is applied to knowledge graph reasoning tasks owing to its adaptability to complex environments [2]. Reinforcement learning regards the above reasoning process as the following problem: finding a path connecting the known head entity and tail entity, modeling this problem as a sequential decision problem, and solving it using the method based on policy gradient [3]. Since the existing reinforcement learning models do not conduct in-depth research on the knowledge graph, this paper employs causal inference to analyze the knowledge graph and obtain prior knowledge, thereby improving inference performance. For instance, in Figure 1, we want to complete the triple (person, language,? ), and there are two paths in the knowledge graph: \"personStarinfilm\"Britain\" and \"personStarinfilmLanguage\". We obtain prior knowledge through causal inference, i.e., the relationship \"language\" in the second path is more related to the relationship in the triple to be completed than the relationship \"country\" in the first path. So, we obtain the importance of the relationship in the path and thus guide the path selection. The major contributions of this work are as follows:  
1. We propose a prior knowledge generation method based on causal inference. Specifically, it is implemented through the relationship importance identification module. The generated prior knowledge is mainly used to measure the contribution of the relationship in the triple to be completed. 2. We introduce a new method combining causal inference and reinforcement learning that applies to knowledge graph reasoning. Specifically, the prior knowledge is integrated into the control strategy of the reinforcement model, allowing the agent to select the relation more accurately in each step. Our model also applies to large-scale knowledge graphs. 3. The experiments demonstrate that our method's performance outperforms the current baseline methods in most cases. Additionally, we conduct ablation experiments to highlight that our method is more effective than solely employing reinforcement learning. The remainder of this paper is organized as follows. Section 2 briefly introduces the related work.