# A Novel Embedding Model for Knowledge Graph Entity Alignment Based on Graph Neural Networks
## 3 Proposed Framework
### Heterogeneous Graph Transformer with Relation Awareness (HGTRA)  
The process by which the graph transformer assimilates all the neighboring features of node \"h\" can be elegantly formulated as follows:  
\[e_{h}^{(l)}\leftarrow\underset{\forall t\in N(h)}{Aggregate}\left(Attention(h,t) \cdot Message(h,t)\right) \tag{7}\]  
In this equation, \(Attention\) is used to calculate the importance of each neighboring node; \(Message\) extracts features from each neighboring node; and \(Aggregate\) aggregates neighbor information using attention weights. However, as illustrated in Equation (7), the graph transformer fails to consider edge features. To address this, we designed a novel heterogeneous graph transformer with relation awareness (HGTRA) in this paper, inspired by previous work [26]. The proposed HGTRA enables our model to differentiate between the heterogeneous features of relations and properties, thus facilitating a better capture of neighborhood similarities among aligned entities. Let \(E^{(l)}\) represents the output of the \((l)\)-th layer of HGTRA, which also serves as the input to the \((l+1)\)-th layer. At first, the value of \(E^{(0)}\) is equal to \(E^{(n)}\).When HGTRA takes a relation triplet as input, the output is relation-based embedding. When it takes a property triplet as input, the output is property-based embedding. HGTRA mainly consists of the following four layers, as shown in Figure 2:  
**(a) Relation Embedding. ** Considering the possible similarity between the aligned relation and the head entity and tail entity, this paper generates relation features by combining the relevant entity features. In particular, the relation embedding \(r\) is approximately calculated by taking an average of the embeddings of its related head entity \(H_{r}\) and tail entity \(T_{r}\), as demonstrated in the subsequent formula:  
\[R^{l}(r)=\sigma\left[\frac{\sum_{e\in H_{r}}b_{h}e_{i}^{(l-1)}}{|H_{r}|}||\frac {\sum_{e\in T_{r}}b_{t}e_{j}^{(l-1)}}{|T_{r}|}\right] \tag{8}\]  
In this equation, \(|\cdot|\) denotes the size of a set; \(b_{h}\) and \(b_{t}\) are attention vectors; \([\cdot]\) denotes the operation of concatenation; and \(\sigma\) denotes the activation function Rectified Linear Unit (ReLU);  
**(b) Heterogeneous Attention. ** In this work, entity \(h\) is mapped to a key vector \(K^{i}(h)\), and its neighboring entity \(t\) is mapped to a query vector \(Q^{i}(t)\). In contrast to other methods, this work uses the dot product of their concatenation and \(R^{l}(r)\) as the value of attention, rather than directly using the dot product of the key and query vectors. \(R^{l}(r)\) is derived from the feature aggregation of the related head and tail entities (refer to Equation (8)), hence it does not stray too far from the embeddings of its linked entities. In addition, \(R^{l}(r)\) signifies the heterogeneous feature of the edge, thereby causing distinct effects on the contribution of neighboring pairs linked to different edges towards the entity \(h\). More specifically, this work calculates the multi-head attention of each neighbor relation \((h,r,t)\), evaluated in the following manner in this study:  
\[HAttention(h,r,t)=\frac{||}{i\in[1,h_{n}]\forall(r,t)\in RN(h)}Softmax(HATT_ {head^{i}}(h,r,t)) \tag{9}\]  
Among them, the detailed expression of \(HATT_{head^{i}}(h,r,t)\) is as follows:  
\[HATT_{head^{i}}(h,r,t)=\frac{a^{T}\left(\big{[}K^{i}(h)\big{|}\big{|}Q^{i}(t) \big{]}R^{l}(r)\right)}{\sqrt{d/h_{n}}} \tag{10}\]  
where \(K^{i}(h)=K\_Linear^{i}\Big{(}e_{h}^{(l-1)}\Big{)}\); \(Q^{i}(t)=Q\_Linear^{i}\Big{(}e_{t}^{(l-1)}\Big{)}\); the symbol \(RN(h)\) denotes the set of entities neighboring \(h\); the parameter \(a\) is the attention parameter of dimensionality \(d/h_{n}\times 1\), where \(h_{n}\) denotes the number of attention heads. It should be noted that the  
Figure 2: Overview of the framework of HGTRA. Softmax operation ensures that the sum of attention weights assigned to all neighboring entities is equal to one;  
**(c) Heterogeneous Message.