# Heterogeneous Graph Purification Network: Purifying Noisy Heterogeneity without Metapaths
## 4 Proposed Method
### Type-Specific Update  
After the inner-subgraph and intra-subgraph aggregation modules, we need to use the comprehensive message \(Z_{i}^{l}\) to update the original node representation. First, we use the following formula to transform the message back to the representation space of target node type:  
\[M_{i}^{l}=GeLU(BN_{\varphi(i)}(W_{u,\varphi(i)}^{L}Z_{i}^{l})) \tag{12}\]  
where \(W_{u,\varphi(i)}\) is the transform matrix corresponding to node type \(\phi(i)\) and \(GeLU\) means gelu activation function [28]. It is worth noting that we utilize a distinct batch normalization for each type of node, rather than a default shared one or layer normalization, and we name it type-specific batch normalization. The reason for this design is that the distribution of representations for different types of nodes is usually very different. Using the same batch normalization layer directly would lead to grossly inaccurate estimates of the node feature statistics, which may significantly reduce the effectiveness of our model. At the same time, the type-specific batch normalization also helps to maintain node representations of the same type in the same distribution space, serving to purify the node representation. Residual connection [29] has been widely used among models in various domains and proved to be very effective in alleviating problems such as gradient disappearance and oversmoothing in GNNs [30]. In contrast to the normal one with fixed residual weight, we design a novel and neighborhood smoothness-related dynamic residual connection scheme for an adapted version in the field of heterogeneous graph networks:  
\[\lambda_{\varphi(i)}^{l}=\textit{sigmoid}(\gamma_{\varphi(i)}\times\frac{|X_ {i}^{l-1}-M_{i}^{l}|_{F}}{\sqrt{d}}) \tag{13}\]  
\[X_{i}^{l}=(1-\lambda_{\varphi(i)}^{l})X_{i}^{l-1}+\lambda_{\varphi(i)}^{l}M_{ i}^{l} \tag{14}\]  
where \(|\ |_{F}\) means Frobenius norm and \(\gamma_{\varphi(i)}\) is a learnable coefficient specific to the type of node \(\phi(i)\). \(X_{i}^{l-1}\) is the node representation from the previous layer and \(d\) is the dimension of node embedding. We define neighborhood smoothness of nodes by calculating the Euclidean difference between the passed message and the original node representation. The distance value is divided by \(\sqrt{d}\) to avoid gradient vanishing in \(\textit{sigmoid}\) function. The basic design idea of our proposed dynamic residual connection is that the update weight of each node should be positively correlated to its neighborhood smoothness. The interpretation is that propagated messages can be considered as the information complement from neighboring nodes to central target nodes. And we believe that nodes with lower neighborhood smoothness represent higher original feature noise according to the widely accepted graph smoothness assumption, especially for most types of nodes without initial features. So, they should rely more on the information provided by the neighboring nodes for feature purification in the method of representation update. Meanwhile, if the neighborhood smoothness is already very high, it means that the original information of the node is sufficient and the additional information brought by the graph structure has limited value. Thus, a smaller update weight should be used to prevent the problem of over-smoothing in GNN training. Overall, our design can provide a two-fold positive effect. In addition,the evaluation metrics of neighborhood smoothing should not be consistent for different types of nodes, so the node category-related parameter \(\gamma_{\phi(i)}\) is introduced for modeling type difference. After the dynamic residual update method, we let \(X_{i}^{l}\) continuously come through a two layer MLP with normal residual connection and type-specific batch normalization, which is similar to the Transformer [25] architecture:  
\[X_{i}^{l+1}=X_{i}^{l}+BN_{\phi(i)}(MLP_{\phi(i)}(X_{i}^{l})) \tag{15}\]  
This design is motivated by the challenge of effectively scaling the number of GNN layers in large-scale graphs.