# Combining Semantic and Structural Features for Reasoning on Patent Knowledge Graphs
## 2 Methodology
### Multi-Head Attention Mechanism Fusion Block  
In recent years, attention mechanisms based on adaptive features have proven effective in enhancing model performance and have been widely applied in knowledge graph reasoning. Due to the complexity of patent knowledge information analysis, and in order to further obtain comprehensive information from the knowledge graph and better integrate semantic and structural information to enhance the model's precise semantic understanding and robust reasoning capabilities, the multi-head attention mechanism is applied at the score fusion stage. The multi-head attention mechanism merges the scores derived from semantic and structural information, ensuring that the resultant scores reflect both the semantic and structural information of the knowledge graph. This mechanism allows the model to dynamically weight each element in a sequence, assign varying degrees of importance, and aggregate this information into a single output vector. This aggregation considers the content of each element in the sequence and their interactions. In natural language processing tasks, it aids models in understanding context, capturing long-distance dependencies, and within knowledge graphs, it facilitates the understanding of complex relationships between different entities. The computational method of the multi-head attention mechanism for merging semantic and structural information from graphs is illustrated in Equation (12):  
\[\left\{ \begin{array}{l}MultiHead(Q,K,V)=Concat(head_{1},head_{2})W^{o}\\ head_{i}=a\left(QW_{i}^{Q},KW_{i}^{K},VW_{i}^{k}\right)\end{array} \right. \tag{12}\]  
where \(MultiHead(Q,K,V)\) is the multi-head attention function, and \(Q,K,V\) represent the inputs for queries, keys, and values, respectively. The outputs of two attention heads are \(head_{1}\) and \(head_{2}\), with \(Concat\) denoting the concatenation of these two heads' outputs. \(W^{o}\) is the linear transformation matrix that integrates the outputs, with each head capturing different aspects of the information from the input sequence. Each head \(head_{i}\) computes  
Figure 5: Process of extracting graph structure information around target nodes in knowledge graphs. its own output of the attention function \(a\), where \(QW_{i}^{Q},KW_{i}^{K},VW_{i}^{k}\) are the inputs \(Q,K,V\) multiplied by their respective weight matrices. \(head_{1}\) and \(head_{2}\) specifically represent the attention for semantic information and structural information parts, respectively. The calculation method is shown in Equation (13), where softmax is used as the activation function, and \(\sqrt{d_{k}}\) is a scaling factor used to control the size of the dot product to prevent issues of gradient vanishing due to excessively large dot products. \[a(Q,K,V)=softmax\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)V \tag{13}\]  
When computing the attention for semantic information, the input consists of entity vectors from the knowledge graph, denoted as \(X=(x_{1},x_{2},\cdots x_{n})\). The calculation of attention weights for the semantic information component \(head_{1}\) is shown in Equation (14). \[a_{1}=softmax\left(\frac{XX^{T}}{\sqrt{d_{k}}}\right)X \tag{14}\]  
In the process of calculating the attention vector for structural features, the relationship between the target node's structural information and the final result is represented by the degree of the entity node, with the input for the weight coefficient calculation denoted as \(D=(d_{1},d_{2},\cdots d_{n})\), and the attention weights for the structural information component \(head_{2}\) shown in Equation (15). \[a_{2}=softmax\left(\frac{DD^{T}}{\sqrt{d_{k}}}\right)D \tag{15}\]  
Based on the output values from Equations (14) and (15), semantic and structural scoring information are merged. The fusion method is outlined in Equation (16). \[f=\sum\limits_{i=1}^{n}\alpha_{i}f_{i} \tag{16}\]  
The weight calculation parameters in Equation (16) are detailed in Equation (17), with the calculated \(\alpha_{i}\) being normalized weights, where the sum of all weights equals 1.