# MBHAN: Motif-Based Heterogeneous Graph Attention Network
## 5 Experiments
### Comparative Statistical Tests for the Different Algorithms  
The previous two subsections involved the comparison of multiple algorithms on multiple datasets. However, the same algorithm may not have the same ranking on different datasets. We therefore undertook some statistical tests to assess the overall performance of each particular algorithm across the multiple datasets.  
A Friedman test [51] can be employed to determine whether the performance of a particular algorithm (MBHAN, in this case) is significantly different from other algorithms. It is a classic non-parametric statistical test that is based on the hypothesis that there is no significant difference in the overall distribution of multiple pairs of algorithms' mean ranks. If we let \(Rank_{j}^{i}\) be the rank of the \(j\)-th method on the \(i\)-th data set, its mean performance ranking on a dataset can be calculated using Equation (15), where \(N\) is the number of methods to be compared and \(S\) is the number of datasets. The Friedman statistic can then be computed using Equation (16). As \(\tau_{x^{2}}\) was found to be too conservative, an improved version is given by Equation (17).  
\[Rank_{j}=\frac{1}{S}\sum_{i=1}^{S}Rank_{j}^{i},\ j\in[1,\ N], \tag{15}\]  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Datasets** & \multicolumn{2}{c}{**DBLP\_Four\_Area**} & \multicolumn{1}{c}{**ACM**} \\ \hline
**Metrics** & \multicolumn{2}{c}{**NMI**} & \multicolumn{1}{c}{**NMI**} \\ \hline
**Target Nodes** & **Author** & **Paper** & **Paper** \\ \hline DeepWalk & 73.49 & 57.91 & 43.89 \\ metapath2 vec & 66.17 & 53.48 & 22.47 \\ GCN & 76.31 & 60.47 & 54.26 \\ GAT & 76.80 & 60.70 & 58.47 \\ HAN & 79.87 & 62.79 & 61.56 \\ GTN & 80.05 & 63.09 & 61.77 \\ \hline MBHANnon\_type & 81.17 & 63.00 & 60.74 \\ MBHAN & **81.75** & **63.14** & **61.79** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Node clustering performance results (%).  
\[\tau_{\chi^{2}}=\frac{12S}{N(N+1)}\left(\sum_{i=1}^{N}Rank_{i}^{2}-\frac{N(N+1)^{2} }{4}\right) \tag{16}\]  
\[F=\frac{(S-1)\tau_{\chi^{2}}}{S(N-1)-\tau_{\chi^{2}}}, \tag{17}\]  
To perform an overall comparison, the experimental results for the different datasets need to be combined. \(S\) was set at 3 and \(N\) was set at 8. The Mac-F1, Mic-F1, and NMI Friedman test values were 41.200, 24.999, and 73.600, respectively. These are all much higher than the critical value (2.76) for a significance level of \(\alpha=0.05\). Therefore, the original hypothesis can be safely rejected. In other words, there are significant performance differences between the different methods.  
Because the null-hypothesis is rejected, a post-hoc test can be proceeded. The Nemenyi test [51], which is calculated using Equation (18), defines the critical difference \((CD)\) value for a case where two methods are significantly different with a certain confidence \((1-\gamma)\). Here, \(q_{\gamma}\) is the critical value based on the Studentized range statistic divided by \(\sqrt{2}\).  
\[CD=q_{\gamma}\sqrt{\frac{N(N+1)}{6S}}, \tag{18}\]  
The critical difference for the 95% confidence interval was calculated (\(CD=6.06\)). The results of the test are shown in Figure 4, where the mean rank of each method is marked by a dot, and the length of the horizontal bar across each dot shows the critical difference. For any two methods, a mothed outperforms another one if its mean rank is smaller. More strictly, if the mean ranks of any two methods differ by more than \(CD\), the superiority is significant. According to the mean ranks and the results of the Nemenyi test, we can conclude that our MBHAN method achieves the best overall performance, and significantly outperforms some of these methods.