Title: Revisiting the Message Passing in Heterophilous Graph Neural Networks
Venue: NeurIPS 2024
Accepted: no
Keywords: GNNs

Message Passing

Heterophily
Abstract: Graph Neural Networks (GNNs) have demonstrated strong performance in graph mining tasks due to their message-passing mechanism, which is aligned with the homophily assumption that adjacent nodes exhibit similar behaviors. However, in many real-world graphs, connected nodes may display contrasting behaviors, termed as heterophilous patterns, which has attracted increased interest in heterophilous GNNs (HTGNNs).
Although the message-passing mechanism seems unsuitable for heterophilous graphs due to the propagation of class-irrelevant information, it is still widely used in many existing HTGNNs and consistently achieves notable success. 
This raises the question: why does message passing remain effective on heterophilous graphs?
To answer this question, in this paper, we revisit the message-passing mechanisms in heterophilous graph neural networks and reformulate them into a unified heterophilious message-passing (HTMP) mechanism.
Based on HTMP and empirical analysis, we reveal that the success of message passing in existing HTGNNs is attributed to implicitly enhancing the compatibility matrix among classes.
Moreover, we argue that the full potential of the compatibility matrix is not completely achieved due to the existence of incomplete and noisy semantic neighborhoods in real-world heterophilous graphs.
To bridge this gap, we introduce a new approach named CMGNN, which operates within the HTMP mechanism to explicitly leverage and improve the compatibility matrix.
A thorough evaluation involving 10 benchmark datasets and comparative analysis against 13 well-established baselines highlights the superior performance of the HTMP mechanism and CMGNN method.

Abstract

Graph Neural Networks (GNNs) have demonstrated strong performance in graph mining tasks due to their message-passing mechanism, which is aligned with the homophily assumption that adjacent nodes exhibit similar behaviors. However, in many real-world graphs, connected nodes may display contrasting behaviors, termed as heterophilous patterns, which has attracted increased interest in heterophilous GNNs (HTGNNs). Although the message-passing mechanism seems unsuitable for heterophilous graphs due to the propagation of class-irrelevant information, it is still widely used in many existing HTGNNs and consistently achieves notable success. This raises the question: why does message passing remain effective on heterophilous graphs? To answer this question, in this paper, we revisit the message-passing mechanisms in heterophilous graph neural networks and reformulate them into a unified heterophilious message-passing (HTMP) mechanism. Based on HTMP and empirical analysis, we reveal that the success of message passing in existing HTGNNs is attributed to implicitly enhancing the compatibility matrix among classes. Moreover, we argue that the full potential of the compatibility matrix is not completely achieved due to the existence of incomplete and noisy semantic neighborhoods in real-world heterophilous graphs. To bridge this gap, we introduce a new approach named CMGNN, which operates within the HTMP mechanism to explicitly leverage and improve the compatibility matrix. A thorough evaluation involving 10 benchmark datasets and comparative analysis against 13 well-established baselines highlights the superior performance of the HTMP mechanism and CMGNN method.

1 Introduction

Graph Neural Networks (GNNs) have shown remarkable performance in graph mining tasks, such 24

as social network analysis [1, 2] and recommender systems [3, 4]. The design principle of GNNs is 25

26

typically based on the homophily assumption [5], which assumes that nodes are inclined to exhibit

behaviors similar to their neighboring nodes [6]. However, this assumption does not always hold 27

in real-world graphs, where the connected nodes demonstrate a contrasting tendency known as the 28

heterophily [7]. In response to the challenges of heterophily in graphs, heterophilous GNNs (HTGNNs) 29

have attracted considerable research interest [6, 8-10], with numerous innovative approaches being 30

introduced recently [11-24]. However, the majority of these methods continue to employ a message31

passing mechanism, which was not originally designed for heterophilous graphs, as they tend to 32

incorporate excessive information from disparate classes. This naturally raises a question: Why does 33

message passing remain effective on heterophilous graphs? 34

Recently, a few efforts [6] have begun to investigate this question and reveal that vanilla message 35

passing can work on heterophilous graphs under certain conditions. However, the absence of a unified 36

and comprehensive understanding of message passing within existing HTGNNs has hindered the 37 creation of innovative approaches. In this paper, we first revisit the message-passing mechanisms 38 in existing HTGNNs and reformulate them into a unified heterophilous message-passing (HTMP) 39 mechanism, which extends the definition of neighborhood in various ways and simultaneously utilizes 40 the messages of multiple neighborhoods. Specifically, HTMP consists of three major steps namely 41 aggregating messages with explicit guidance, combining messages from multiple neighborhoods, and 42 fusing intermediate representations. 43

44

45

46

47

48

49

Equipped with HTMP, we further conduct empirical analysis on real-world graphs. The results reveal that the success of message passing in existing HTGNNs is attributed to implicitly enhancing the compatibility matrix , which exhibits the probabilities of observing edges among nodes from different classes. In particular, by increasing the distinctiveness between the rows of the compatibility matrix via different strategies, the node representations of different classes become more discriminative in heterophilous graphs.

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

Drawing from previous observations, we contend that nodes within real-world graphs might exhibit a semantic neighborhood that only reveals a fraction of the compatibility matrix, accompanied by noise. This could limit the effectiveness of enhancing the compatibility matrix and result in suboptimal representations. To fill this gap, we further propose a novel Compatibility Matrix-aware Graph Neural Network (CMGNN) under HTMP mechanism, which utilizes the compatibility matrix to construct desired neighborhood messages as supplementary for nodes and explicitly enhances the compatibility matrix by a targeted constraint. We build a benchmark to fairly evaluate CMGNN and existing methods, which encompasses 13 diverse baseline methods and 10 datasets that exhibit varying levels of heterophily. Extensive experimental results demonstrate the superiority of CMGNN and HTMP mechanism. The contributions of this paper are summarized as:

We revisit the message-passing mechanisms in existing HTGNNs and reformulate them into a unified heterophilous message-passing mechanism (HTMP), which not only provides a macroscopic view of message passing in HTGNNs but also enables people to develop new methods flexibly.

We reveal that the effectiveness of message passing on heterophilous graphs is attributed to implicitly enhancing the compatibility matrix among classes, which gives us a new perspective to understand the message passing in HTGNNs.

Based on HTMP mechanism and empirical analysis, we propose CMGNN to unlock the potential of the compatibility matrix in HTGNNs. We further build a unified benchmark that overcomes the issues of current datasets for fair evaluation 1 . Experiments show the superiority of CMGNN.

2 Preliminaries

Given a graph G = ( V , E , X , A , Y ) , V is the node set and E is the edge set. Nodes are characterized by the feature matrix X ∈ R N × d f , where N = |V| denotes the number of nodes, d f is the features dimension. Y ∈ R N × 1 is the node labels with the one-hot version C ∈ R N × K , where K is the number of node classes. The neighborhood of node v i is denoted as N i . A ∈ R N × N is the adjacency matrix , and D = diag ( d 1 , ..., d n ) represents the diagonal degree matrix, where d i = ∑ j A ij . ˜ A = A + I represents the adjacency matrix with self-loops. Let Z ∈ R N × d r be the node representations with dimension d r learned by the models. We use 1 to represent a matrix with all elements equal to 1, and 0 for a matrix with all elements equal to 0.

Homophily and Heterophily . High homophily is observed in graphs where a substantial portion of connected nodes shares identical labels, while high heterophily corresponds to the opposite situation. For measuring the homophily level, two widely used metrics are edge homophily h e [12] and node homophily h n [15], defined as h e = |{ e u,v | e u,v ∈E , Y u = Y v }| |E| and h n = 1 |V| ∑ v ∈V |{ u | u ∈N v , Y u = Y v }| d v .

Both metrics have a range of

[0

indicate stronger heterophily.

Vanilla Message Passing (VMP) . The vanilla message-passing mechanism plays a pivotal role in 84 transforming and updating node representations based on the neighborhood [25]. Typically, the 85

,

1]

, where higher values indicate stronger homophily and lower values

* The correspondence between the full form and the abbreviation: Raw Neighborhood (Raw), Neighborhood Redefine (ReDef), Neighborhood Discrimination (Dis), Degree-based Averaging (DegAvg), Adaptive Weights (AdaWeight), Relation Estimation (RelaEst), Addition (Add), Weighted Addition (WeightAdd), Adaptive Weighted Addition (AdaAdd), Concatenation (Cat), Adaptive Dimension Concatenation (AdaCat).

mechanism operates iteratively and comprises two stages: 86

where the AGGREGATE function first aggregates the input messages Z l -1 from neighborhood A 87 into the aggregated one ˜ Z l , and subsequently, the COMBINE function combines the messages of 88 node ego and neighborhood aggregation, resulting in updated representations Z l . 89

90

3 Revisiting Message Passing in Heterophilous GNNs.

To gain a thorough and unified insight into the effectiveness of message passing in HTGNNs, we 91 revisit message passing in various notable HTGNNs [11-24] and propose a unified heterophilous 92 message passing (HTMP) mechanism, structured as follows: 93

Generally, HTMP extends the definition of neighborhood in various ways and simultaneously utilize 94 the messages of multiple neighborhoods, which is the key for better adapting to heterophily. We 95 use R to denote the number of neighborhoods used by the model. In each message passing layer l , 96 HTMP separately aggregates messages within R neighborhoods and combines them. The method97 ological analysis of some representative HTGNNs and more details can be seen in Appendix A. 98 Compared to the VMP mechanism, HTMP mechanism has advances in the following functions: 99

(i) To characterize different neigborhoods, the AGGREGATE function in HTMP includes the neigh100 borhood indicator A r to indicate the neighbors within a specific neighborhood r . The adjacency 101 matrix A in VMP is a special neighborhood indicator that marks the neighbors in the raw neigh102 borhood. To further characterize the aggregation of different neighborhoods, HTMP introduces the 103 aggregation guidence B r for each neighborhood r . In VMP, the aggregation guidance is an implicit 104 parameter of the AGGREGATE function since it only works for the raw neighborhood. A commonly 105 used form of the AGGREGATE function is AGGREGATE ( A r , B r , Z l -1 ) = ( A r ⊙ B r ) Z l -1 W l r , 106 where ⊙ is the Hadamard product and W l r is a weight matrix for message transformation. We take 107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

this as the general form of the AGGREGATE function and only analyze the neighborhood indicators and the aggregation guidance in the following.

The neighborhood indicator A r ∈ { 0 , 1 } N × N indicates neighbors associated with central nodes within neighborhood r . To describe the multiple neighborhoods in HTGNNs, neighborhood indicators can be formed as a list A = [ A 1 , ..., A r , ..., A R ] . For the sake of simplicity, we consider the identity matrix I ∈ R N × N as a special neighborhood indicator for acquiring the nodes' ego messages. The aggregation guidance B r ∈ R N × N can be viewed as pairwise aggregation weights in most cases, which has the multiple form B = [ B 1 , ..., B r , ..., B R ] . Table 1 illustrates the connection between message passing in various HTGNNs and HTMP mechanism.

(ii) Considering the existence of multiple neighborhoods, the COMBINE function in HTMP need to integrate multiple messages instead of only the ego node and the raw neighborhood. Thus, the input of the COMBINE function is a set of messages ˜ Z l r aggregated from the corresponding neighborhoods. In HTGNNs, addition and concatenation are two common approaches, each of which has variants. An effective COMBINE function is capable of simultaneously processing messages from various neighborhoods while preserving their distinct features, thereby reducing the effects of heterophily.

(iii) In VMP, the final output representations are usually the one of the final layer: Z = Z L . Some HTGNNs utilize the combination of intermediate representations to leverage messages from different localities, adapting to the heterophilous structural properties in different graphs. Thus, we introduce an additional FUSE function in HTMP which integrates multiple representations Z l of different layers l into the final Z . Similarly, the FUSE function is based on addition and concatenation.

4 Why Does Message Passing Still Remain Effective in Heterophilous Graphs?

Based on HTMP mechanism, we further dive into the motivation behind the message passing of existing HTGNNs. Our discussion begins by examining the difference between homophilous and heterophilous graphs. Initially, we consider the homophily ratios h e and h n , as outlined in Section 2. However, a single number is not able to indicate enough conditions of a graph. Ma et al. [6] propose the existence of a special case of heterophily, named "good" heterophily, where the VMP mechanism can achieve strong performance and the homophily ratio shows no difference. Thus, to better study the heterophily property, here we introduce the Compatibility Matrix [7] to describe graphs:

Definition 1 Compatibility Matrix (CM) : The potential connection preference among classes within a graph. It's formatted as a matrix M ∈ R K × K , where the i -th row M i denotes the connection probabilities between class i and all classes. It can be estimated empirically by the statistics among nodes as follows:

where Norm ( · ) denotes the L1 normalization and T is the matrix transpose operation. C nb ∈ R N × K is the semantic neighborhoods of nodes, which indicates the proportion of neighbors from each class in nodes' neighborhoods.

We visualize the CM of a homophilous graph Photo [29] and a heterophilous graph AmazonRatings [30] in Figure 1(a) and 1(b). The CM in Photo displays an identity-like matrix, where the diagonal elements can be viewed as the homophily level of each class. With this type of CM, the VMP mechanism learns representations comprised mostly of messages from same the class, while messages of other classes are diluted. Then how does HTMP mechanism work on heterophilous graphs without an identity-like CM? The "good" heterophily inspires us, which we believe corresponds to a CM with enough discriminability among classes. We conduct experiments on synthetic graphs to confirm this idea, with details available in Appendix C. Also, we find "good" heterophily in real-world graphs though it's not as significant as imagined. Thus, we have the following observation:

Observation 1 (Connection between CM and VMP). When enough (depends on data) discriminability exists among classes in CM, vanilla message passing can work well in heterophilous graphs.

With this observation, we have a conjecture: Is HTMP mechanism trying to enhance the discriminability of CM? Some special designs in HTMP intuitively meet this. For example, feature-similarity-based neighborhood indicators and neighborhood discrimination are designed to construct neighborhoods

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

[0.5, 0, 0.5]

(a)

Observed

CM of

Photo

(b)

Observed

CM of

Amazon-Ratings

(c)

New-constructed

CMofAmazon-Ratings

[0, 0.5, 0.5]

[0, 0.7, 0.3]

[0, 0.2, 0.8]

[0.8, 0, 0.2]

[0.3, 0, 0.7]

[0, 0, 1] ?

(d)

Overlap of semantic

neighborhood distribution

with high homophily, that is, an identity-like CM with high discriminability. We plot the CM of feature-similarity-based neighborhood on Amazon-Ratings in Figure 1(c) to confirm it. Moreover, we investgate two representative methods ACM-GCN [18] and GPRGNN [20], showing that they also meet this conjecture with the posterior proof in Appendix D. ACM-GCN combines the messages of node ego, low-frequency and high-frequency with adaptive weights, which actually motifs the edge weights and node weights to build a new CM. GPRGNN has a FUSE function with adaptive weights while other settings are the same as GCN. It actually integrates the CMs of multiple-order neighborhoods with adaptive weights to form a more discriminative CM. These lead to the answer to the aforementioned question:

Observation 2 (Connection between CM and HTMP). The unified goal of various message passing in existing HTGNNs is to utilize and enhance the discriminability of CM on heterophilous graphs. In other words, the success of message passing in existing HTGNNs benefits from utilizing and enhancing the discriminability of CM.

Furthermore, we notice that the power of CM is not fully released due to the incomplete and noisy semantic neighborhoods in real-world heterophilous graphs. We use the perspective of distribution to describe the issue more intuitively: The semantic neighborhoods of nodes from the same class collectively form a distribution, whose mean value indicates the connection preference of that class, i.e. M i for class i . Influenced by factors such as degree and randomness, the semantic neighborhood of nodes in real-world graphs may display only a fraction of CM accompanied by noise. It can lead to the overlap between different distributions as shown in Figure 1(d), where the existence of overlapping parts means nodes from different classes may have the same semantic neighborhood. This brings a great challenge since the overlapping semantic neighborhood may become redundant information during message passing.

5 Method

To fill this gap, we further propose a method named Compatibility Matrix-Aware GNN (CMGNN), which leverages the CM to construct desired neighborhood messages as supplementary, providing valuable neighborhood information for nodes to mitigate the impact of incomplete and noisy semantic neighborhoods. The desired neighborhood message denotes the averaging message within a neighborhood when a node's semantic neighborhoods meet the CM of the corresponding class, which converts the discriminability from CM into messages. CMGNN follows the HTMP mechanism and constructs a supplementary neighborhood indicator along with the corresponding aggregation guidance to introduce supplementary messages. Further, CMGNN introduces a simple constraint to explicitly enhance the discriminability of CM.

Message Passing in CMGNN. CMGNN aggregates messages from three neighborhoods for each node, including the ego neighborhood, raw neighborhood, and supplementary neighborhood. Following the HTMP mechanism, the message passing of CMGNN cen be described as follows:

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

where AdaWeight is the adaptive weighted addition implemented by an MLP with Softmax, ∥ denotes the concatenation. The neighborhood indicators and aggregation guidance of the three neighborhoods are formatted as follows:

where A sup and B sup are described below.

The supplementary neighborhood indicator A sup assigns K additional virtual neighbors for each node: A sup = 1 ∈ R N × K . Specifically, these additional neighbors are K virtual nodes, constructed as the prototypes of classes based on the labels of the training set. The attributes X ptt ∈ R K × d f , neighborhoods A ptt ∈ R K × N and labels Y ptt ∈ R K × K of prototypes are defined as follows:

where C train and X train are the one-hot labels and attributes of nodes in the training set. Utilizing class prototypes as supplementary neighborhoods can provide each node with representative messages of classes, which builds the basis for desired neighborhood messages.

The supplementary aggregation guidance B sup = ˆ C ˆ M indicates the desired semantic neighborhood of nodes, i.e. the desired proportion of neighbors from each class in nodes' neighborhoods according to the probability that nodes belong to each class. ˆ M is the estimated compatibility matrix described in below. Using soft logits instead of one-hot pseudo labels preserves the real characteristics of nodes and reduces the impact of wrong predictions. During the message aggregation in the supplementary neighborhoods, the input representations Z l -1 are replaced by the representations of virtual prototype nodes Z l -1 ptt , which are obtained by the same message-passing mechanism as real nodes.

Similar to existing methods [18, 19], we also regard topology structure as a kind of additional available node features. Thus, the input representation of the first layer can be obtained in two ways:

Note that in practice, we use ReLU as the activation function between layers. From the perspective of HTMP mechanism, our special design is to introduce an additional neighborhood indicator A sup by neighborhood redefining and aggregation guidance B sup , which can be seen as a form of relation estimation along with good interpretability. Meanwhile, these designs greatly reduce the time and space cost via the N × K form.

Compatibility Matrix Estimation. The CM can be directly calculated via Eq 3 with full-available labels. However, the label information is not entirely available in semi-supervised settings. Thus, we try to estimate the CM with the help of semi-supervised and pseudo labels. Since the pseudo labels predicted by the model might be wrong, which can lead to low-quality estimation, we introduce the confidence g ∈ R N × 1 based on the information entropy to reduce the impact of wrong predictions, where a high entropy means low confidence:

where ˆ C ∈ R N × K is the soft pseudo labels composed of labels from the training set and model predictions. Then the nodes' semantic neighborhoods C nb = Norm ( A ( g · ˆ C )) ∈ R N × K are calculated considering the confidence.

Further, the degrees of nodes also influence the estimation. As we mentioned in Section 4, the semantic neighborhood of low-degree nodes may display incomplete CM, leading to a significant gap between semantic neighborhoods and corresponding CM. Thus, they deserve low weights during the estimation. We manually set up two fixed thresholds and a weighting function range in [0 , 1] :

When a node's degree d i is smaller than the number of classes K , its semantic neighborhood is unlikely to display complete CM, corresponding to a low weight. And when the node degree is greater than 3 K , we believe it can display near-complete CM, corresponding to the maximum weight. ˆ R K × K

Finally, we can estimate the compatibility matrix M ∈ as follows:

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

Objective Function. As mentioned in Sec 4, the CMs in real-world graphs don't always have 236 significant discriminability, which may lead to low effectiveness of supplementary messages. Thus, we 237 introduce an additional discrimination loss L dis to reduce the similarity of the desired neighborhood 238 message among different classes, which enhances the discriminability among classes in CM. The 239 overall loss consists of a CrossEntropy loss L ce and the discrimination loss L dis : 240

̸

where Z ptt ∈ R K × d r is the representation of virtual prototypes nodes. More details about the implementation of CMGNN is available in Appendix E.

6 Benchmarks and Experiments

In this section, we conduct comprehensive experiments to demonstrate the effectiveness of the proposed CMGNN with a newly organized benchmark for fair comparisons.

6.1 New Benchmark

As reported in [30], some widely adopted datasets in existing works have critical drawbacks, which lead to unreliable results. Therefore, with a comprehensive review of existing benchmark evaluation, we construct a new benchmark to fairly perform experimental validation. Specifically, we integrate 13 representative homophilous and heterophilous GNNs, construct a unified codebase, and evaluate their node classification performances on 10 unified organized datasets with various heterophily levels.

Drawbacks of Existing Datasets. Existing works mostly follow the settings and datasets used in [15], including 6 heterophilous datasets (Cornell, Texas, Wisconsin, Actor, Chameleon, and Squirrel) and 3 homophilous datasets (Cora, Citeseer, and Pubmed). Platonov et al. [30] pointed out that there are serious data leakages in Chameleon and Squirrel, while Cornell, Texas, and Wisconsin are too small with very imbalanced classes. Further, we revisit other datasets and discover new drawbacks: (i) In the ten splits of Citeseer, there are two inconsistent ones, which have smaller training, validation, and test sets that could cause issues with statistical results; (ii) The data split ratios for Cora are not consistent with the expected ones. These drawbacks may lead to certain issues with the conclusions of previous works. The detailed descriptions of dataset drawbacks are listed in Appendix F.1.

Newly Organized Datasets. The datasets used in the benchmark include Roman-Empire, AmazonRatings, Chameleon-F, Squirrel-F, Actor, Flickr, BlogCatalog, Wikics, Pubmed, and Photo. Their statistics are summarized in Table 2, with details in Appendix F.2. For consistency with existing methods, we randomly construct 10 splits with predefined proportions (48%/32%/20% for train/valid/test) for each dataset and report the mean performance and standard deviation of 10 splits.

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

Baseline Methods. As baseline methods, we choose 13 representative homophilous and heterophilous GNNs, including (i) shallow base model: MLP; (ii) homophilous GNNs: GCN [1], GAT [28], GCNII [27]; (iii) heterophilous GNNs: H2GCN [12], MixHop [16], GBK-GNN [24], GGCN[23], GloGNN [19], HOGGCN [17], GPR-GNN [20], ACM-GCN [18] and OrderedGNN [21]. For each method, we integrate its official/reproduced code into a unified codebase and search for parameters in the space suggested by the original papers. More experimental settings can be found in Appendix F.4 and G.1.

6.2 Main Results

Following the constructed benchmark, we evaluate methods and report the performance in Table 2.

Performance of Baseline Methods. With the new benchmarks, some interesting observations and conclusions can be found when analyzing the performance of baseline methods. First, comparing the performance of MLP and GCN, we can find "good" heterophily in Amazon-Ratings, Chameleon-F, and Squirrel-F. Meanwhile, when the homophily level is not high enough, "bad" homophily may also exist as shown in BlogCatalog and Wikics. These results once again support the observations about CMs. Therefore, homophilous GNNs can also work well in heterophilous graphs as GCNII has an average rank of 4.1, which is better than most HTGNNs. This is attributed to the initial residual connection in GCNII actually playing the role of ego/neighbor separation, which is suitable in heterophilous graphs. As for heterophilous GNNs , they are usually designed for both homophilous and heterophilous graphs. Surprisingly, MixHop, as an early method, demonstrated quite good performance. In fact, from the perspective of HTMP, it can be considered a degenerate version of OrderedGNN with no learnable dimensions. As previous SOTA methods, OrderedGNN and ACM-GCN prove their strong capabilities again.

Performance of CMGNN. CMGNN achieves the best performance in 6 datasets and an average rank of 2.1, which outperforms baseline methods. This demonstrates the superiority of utilizing and enhancing the CM to handle incomplete and noisy semantic neighborhoods, especially in heterophilous graphs. Regarding the suboptimal performance in Actor, we believe that this is due to the CM in this dataset are not discriminative enough to provide valuable information via the supplementary messages and hard to enhance. In homophilous graphs, due to the identity-like CMs, the overlap between distributions is relatively less, leading to a minor contribution from supplement messages. Yet CMGNN still achieves top-level performances.

6.3 Ablation Study

We conduct an ablation study on two key designs of CMGNN , including the supplementary messages of the desired neighborhood (SM) and the discrimination loss (DL). The results are shown in Table 3. First of all , both SM and DL have indispensable contributions except for Flickr, BlogCatalog, and Pubmed, in which the discrimination loss has no effect. This may be due to the discriminability of desired neighborhood messages reaching the bottlenecks and can not be further improved by DL Meanwhile , the extent of their contributions varies across datasets. SM plays a more important role in most datasets except Roman-Empire, Wikics, and Photo, in which the number of nodes that need supplementary messages is relatively small and DL has great effects. Further , we notice that with SM and DL, CMGNN can reach a smaller standard deviation most of the time. This illustrates that CMGNN achieves more stable results by handling nodes with incomplete and noisy semantic neighborhoods. As for the opposite result on Chameleon-F, this may attributed to the small size of this dataset (890 nodes), which can lead to naturally unstable results.

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

(a) Amazon-Ratings Obs

(b) Amazon-Ratings Est

(c) BlogCatalog Obs

(d) BlogCatalog Est

6.4 Visualization of Compatibility Matrix Estimation

We visualize the observed and estimated CMs by CMGNN in Figure 2 with heat maps. Obviously, CMGNN estimates CMs that are very close to those existing in graphs. This shows that even with incomplete node labels, CMGNN can estimate high-quality CMs which provides valuable neighborhood information to nodes. Meanwhile, it can adapt to graphs with various levels of heterophily. More results can be seen in Appendix G.2.1.

6.5 Performance on Nodes with Various Levels of Degrees

To verify the effect of CMGNN on nodes with incomplete and noisy semantic neighborhoods, we divide the test set nodes into 5 parts according to their degrees and report the classification accuracy respectively. We compare CMGNN with 3 top-performance methods and show the results in Table 4. In general, nodes with low degrees tend to have incomplete and noisy semantic neighborhoods. Thus, our outstanding performances on the top 20% nodes with the least degree demonstrate the effectiveness of CMGNN for providing desired neighborhood messages. Further, we can find that OrderedGNN and GCNII are good at dealing with nodes with high degrees, while ACM-GCN is relatively good at nodes with low degrees. And CMGNN , to a certain extent, can be adapted to both situations at the same time.

7 Conclusion and Limitations

In this paper, we revisit the message passing mechanism in existing heterophilous GNNs and reformulate them into a unified heterophilous message passing (HTMP) mechanism. Based on the HTMP mechanism and empirical analysis, we reveal that the reason for message passing remaining effective is attributed to implicitly enhancing the compatibility matrix among classes. Further, we propose a novel method CMGNN to unlock the potential of the compatibility matrix by handling the incomplete and noisy semantic neighborhoods. The experimental results show the effectiveness of CMGNN and the feasibility of designing a new method following HTMP mechanism. We hope the HTMP mechanism and benchmark can further provide convenience to the community.

This work mainly focuses on the message passing mechanism in existing HTGNNs under the 335 semi-supervised setting. Thus, the other designs in HTGNNs such as objective functions are not 336 analyzed in this paper. The proposed HTMP mechanism is suitable for only a large part of existing 337 HTGNNs which still follow the message passing mechanism. 338

339

340

341

References

Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017.

Yanfu Zhang, Hongchang Gao, Jian Pei, and Heng Huang. Robust self-supervised struc342 tural graph neural network for social network prediction. In Proceedings of the ACM Web 343 Conference 2022, pages 1352-1361, 2022. 344

Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. Neural graph collabo345 rative filtering. In Proceedings of the 42nd international ACM SIGIR conference on Research 346 and development in Information Retrieval, pages 165-174, 2019. 347

348

349

350

351

Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. Lightgcn: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 639-648, 2020.

352

353

354

355

356

357

358

Miller McPherson, Lynn Smith-Lovin, and James M Cook. Birds of a feather: Homophily in social networks. Annual review of sociology, 27(1):415-444, 2001.

Yao Ma, Xiaorui Liu, Neil Shah, and Jiliang Tang. Is homophily a necessity for graph neural networks? In International Conference on Learning Representations, 2022.

Jiong Zhu, Ryan A Rossi, Anup Rao, Tung Mai, Nedim Lipka, Nesreen K Ahmed, and Danai Koutra. Graph neural networks with heterophily. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11168-11176, 2021.

Xin Zheng, Yixin Liu, Shirui Pan, Miao Zhang, Di Jin, and Philip S Yu. Graph neural networks 359 for graphs with heterophily: A survey. arXiv preprint arXiv:2202.07082, 2022. 360

Jiong Zhu, Yujun Yan, Mark Heimann, Lingxiao Zhao, Leman Akoglu, and Danai Koutra. 361 Heterophily and graph neural networks: Past, present and future. IEEE Data Engineering 362 Bulletin, 2023. 363

364

365

366

Chenghua Gong, Yao Cheng, Xiang Li, Caihua Shan, Siqiang Luo, and Chuan Shi. Towards learning from graphs with heterophily: Progress and future. arXiv preprint arXiv:2401.09769, 2024.

367

368

369

370

371

372

373

374

375

376

377

378

Deyu Bo, Xiao Wang, Chuan Shi, and Huawei Shen. Beyond low-frequency information in graph convolutional networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 3950-3957, 2021.

Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. Advances in neural information processing systems, 33:7793-7804, 2020.

Di Jin, Zhizhi Yu, Cuiying Huo, Rui Wang, Xiao Wang, Dongxiao He, and Jiawei Han. Universal graph convolutional networks. Advances in Neural Information Processing Systems, 34:10654-10664, 2021.

Wei Jin, Tyler Derr, Yiqi Wang, Yao Ma, Zitao Liu, and Jiliang Tang. Node similarity preserving graph convolutional networks. In Proceedings of the 14th ACM international conference on web search and data mining, pages 148-156, 2021.

Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. Geom-gcn: Geo379 metric graph convolutional networks. In International Conference on Learning Representations, 380 2020. 381

Sami Abu-El-Haija, Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard, Kristina Lerman, Hrayr 382 Harutyunyan, Greg Ver Steeg, and Aram Galstyan. Mixhop: Higher-order graph convolutional 383 architectures via sparsified neighborhood mixing. In international conference on machine 384 learning, pages 21-29. PMLR, 2019. 385

Tao Wang, Di Jin, Rui Wang, Dongxiao He, and Yuxiao Huang. Powerful graph convolutional 386 networks with adaptive propagation mechanism for homophily and heterophily. In Proceedings 387 of the AAAI conference on artificial intelligence, volume 36, pages 4210-4218, 2022. 388

Sitao Luan, Chenqing Hua, Qincheng Lu, Jiaqi Zhu, Mingde Zhao, Shuyuan Zhang, Xiao-Wen 389 Chang, and Doina Precup. Revisiting heterophily for graph neural networks. Advances in 390 neural information processing systems, 35:1362-1375, 2022. 391

392

393

394

Xiang Li, Renyu Zhu, Yao Cheng, Caihua Shan, Siqiang Luo, Dongsheng Li, and Weining Qian. Finding global homophily in graph neural networks when meeting heterophily. In International Conference on Machine Learning, pages 13242-13256. PMLR, 2022.

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovic. Adaptive universal generalized pagerank graph neural network. In International Conference on Learning Representations, 2021.

Yunchong Song, Chenghu Zhou, Xinbing Wang, and Zhouhan Lin. Ordered GNN: Ordering message passing to deal with heterophily and over-smoothing. In The Eleventh International Conference on Learning Representations, 2023.

Susheel Suresh, Vinith Budde, Jennifer Neville, Pan Li, and Jianzhu Ma. Breaking the limit of graph neural networks by improving the assortativity of graphs with local mixing patterns. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 1541-1551, 2021.

Yujun Yan, Milad Hashemi, Kevin Swersky, Yaoqing Yang, and Danai Koutra. Two sides of the same coin: Heterophily and oversmoothing in graph convolutional neural networks. In 2022 IEEE International Conference on Data Mining (ICDM), pages 1287-1292. IEEE, 2022.

Lun Du, Xiaozhou Shi, Qiang Fu, Xiaojun Ma, Hengyu Liu, Shi Han, and Dongmei Zhang. Gbk-gnn: Gated bi-kernel graph neural networks for modeling both homophily and heterophily. In Proceedings of the ACM Web Conference 2022, pages 1550-1558, 2022.

Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pages 1263-1272. PMLR, 2017.

Johannes Gasteiger, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. In International Conference on Learning Representations, 2019.

Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In International conference on machine learning, pages 1725-1735. PMLR, 2020.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In The International Conference on Learning Representations, 2018.

Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. arXiv preprint arXiv:1811.05868, 2018.

Oleg Platonov, Denis Kuznedelev, Michael Diskin, Artem Babenko, and Liudmila Prokhorenkova. A critical look at the evaluation of GNNs under heterophily: Are we really making progress? In The Eleventh International Conference on Learning Representations, 2023.

Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In International conference on machine learning, pages 5453-5462. PMLR, 2018.

Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks 432 on graphs with fast localized spectral filtering. Advances in neural information processing 433 systems, 29, 2016. 434

Bingbing Xu, Huawei Shen, Qi Cao, Yunqi Qiu, and Xueqi Cheng. Graph wavelet neural 435 network. In International Conference on Learning Representations, 2018. 436

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large 437 graphs. Advances in neural information processing systems, 30, 2017. 438

Yao Ma, Xiaorui Liu, Tong Zhao, Yozen Liu, Jiliang Tang, and Neil Shah. A unified view on 439 graph neural networks as graph signal denoising. In Proceedings of the 30th ACM International 440 Conference on Information & Knowledge Management, pages 1202-1211, 2021. 441

442

443

444

Meiqi Zhu, Xiao Wang, Chuan Shi, Houye Ji, and Peng Cui. Interpreting and unifying graph neural networks with an optimization framework. In Proceedings of the Web Conference 2021, pages 1215-1226, 2021.

445

446

447

448

449

450

451

Yixin Liu, Zhao Li, Shirui Pan, Chen Gong, Chuan Zhou, and George Karypis. Anomaly detection on attributed networks via contrastive self-supervised learning. IEEE Transactions on Neural Networks and Learning Systems, 2021.

Péter Mernyei and C˘ at˘ alina Cangea. Wiki-cs: A wikipedia-based benchmark for graph neural networks. arXiv preprint arXiv:2007.02901, 2020.

Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina EliassiRad. Collective classification in network data. AI magazine, 29(3):93-93, 2008.

Édouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomáš Mikolov. Learning 452 word vectors for 157 languages. In Proceedings of the Eleventh International Conference on 453 Language Resources and Evaluation (LREC 2018), 2018. 454

Leskovec Jure. Snap datasets: Stanford large network dataset collection. Retrieved December 455 2021 from http://snap. stanford. edu/data, 2014. 456

Benedek Rozemberczki, Carl Allen, and Rik Sarkar. Multi-scale attributed node embedding. 457 Journal of Complex Networks, 9(2):cnab014, 2021. 458

459

460

461

Jie Tang, Jimeng Sun, Chi Wang, and Zi Yang. Social influence analysis in large-scale networks. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 807-816, 2009.

Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for 462 word representation. In Proceedings of the 2014 conference on empirical methods in natural 463 language processing (EMNLP), pages 1532-1543, 2014. 464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

A More Details of HTMP Mechanism

In this part, we list more details about the HTMP mechanism, including additional analysis about HTMP, method-wise analysis and overall analysis.

A.1 Additional Analysis of HTMP Mechanism

A.1.1 Neighborhood Indicators

The neighborhood indicator explicitly marks the neighbors of all nodes within a specific neighborhood. In existing heterophilous GNNs, neighborhood indicators typically take one of the following forms: (i) Raw Neighborhood (Raw); (ii) Neighborhood Redefining (ReDef); and (3) Neighborhood Discrimination (Dis).

Raw Neighborhood. Raw neighborhood, including A and ˜ A , provides the basic neighborhood information. The only difference between them lies in whether there is differential treatment of the node's ego messages. For example, APPNP [26] applies additional weighting to the nodes' ego messages compared with GCN [1]. For the sake of simplicity, we consider the identity matrix I ∈ R N × N as a special neighborhood indicator for acquiring the nodes' ego messages. In heterophilous GNNs, ego/neighbor separation is a common strategy that can mitigate the confusion of ego messages with neighbor messages.

Neighborhood Redefining. Neighborhood redefining is the most commonly used technique in heterophilous GNNs, aiming to capture additional information from new neighborhoods. As a representative example, high-order neighborhood A h can provide long-distance connection information but also result in additional computational costs. Feature-similarity-based neighborhood A f is often defined by the k-NN relationships within the feature space. Fundamentally, it only utilizes node features and thus needs to be used in conjunction with other neighborhood indicators. Otherwise, the model will be limited by the amount of information in node features. GloGNN [19] introduces fully-connected neighborhood 1 ∈ R N × N , which can capture global neighbor information from all nodes. However, it can also cause significant time and space consumption. Additionally, there are some custom-defined neighborhood A c . For example, Geom-GCN [15] redefines neighborhoods based on the geometric relationships between node pairs. These neighborhood indicators may have limited generality, and the effectiveness is reliant on the specific method.

Neighborhood Discrimination. Neighborhood discrimination aims to mark whether neighbors share the same label with central nodes. The neighborhoods are partitioned into positive A p and negative ones A n , which include homophilous and heterophilous neighbors respectively. GGCN [23] divides the raw neighborhood based on the similarity of node representations with a threshold of 0. Explicitly distinguishing neighbors allows for targeted processing, making the model more interpretable. However, its performance is influenced by the accuracy of the discrimination, which may lead to the accumulation of errors.

A.1.2 Aggregation Guidance

After identifying the neighborhood, the aggregation guidance controls what type of messages to gather from the corresponding neighbors. The existing aggregation guidance mainly includes three kinds of approaches: (1) Degree Averaging (DegAvg), (2) Adaptive Weights (AdaWeight), and (3) Relationship Estimation (RelaEst).

Degree Averaging. Degree averaging, formatted as B d = D -1 2 1D -1 2 or B d = D -1 1 , is the most common aggregation guidance, which plays the role of a low-pass filter to capture the smooth signals and is fixed during model training. Further, combining negative degree averaging with an identity aggregation guidance I ∈ R N × N can capture the difference between central nodes and neighbors, as used in ACM-GCN [18]. Degree averaging is simple and efficient but depends on the discriminability of corresponding neighborhoods.

Adaptive Weights. Another common strategy is allowing the model to learn the appropriate aggregation guidances B aw . GAT [28] proposes an attention mechanism to learn aggregate weights, which guides many subsequent heterophilous methods. To better handle heterophilous graphs, FAGCN [11] introduces negative-available attention weights B naw to capture the difference between central nodes

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

and heterophilous neighbors. Adaptive weights can personalize message aggregation for different neighbors, yet it's difficult for models to attain the desired effect.

Relationship Estimation. Recently, some methods have tried to estimate the pair-wise relationships B re between nodes and use them to guide message aggregation. HOG-GCN [17] estimates the pair-wise homophily levels between nodes as aggregation guidances based on both attribute and topology space. GloGNN [19] treats all nodes as neighbors and estimates a coefficient matrix as aggregation guidance based on the idea of linear subspace expression. GGCN [23] estimates appropriate weights for message aggregation with the degrees of nodes and the similarities between node representations. Relationship estimation usually has theoretical guidance, which brings strong interpretability. However, it may also result in significant temporal and spatial complexity when estimating pair-wise relations.

A.1.3 COMBINE Function

After message aggregation, the COMBINE functions integrate messages from multiple neighborhoods into layer representations. COMBINE functions in heterophilous GNNs are commonly based on two operations: addition and concatenation, each of which has variants. To merge several messages together, addition (Add) is a naive idea. Further, to control the weight of messages from different neighborhoods, weighted addition (WeightedAdd) is applied. However, it is a global setting and cannot adapt to the differences between nodes. Thus, adaptive weighted addition (AdaAdd) is proposed, which can learn personalized message combination weights for each node, but it will result in additional time consumption. Although the addition is simple and efficient, some methods [12, 16] believe that it may blur messages from different neighborhoods, which can be harmful in heterophilous GNNs, so they employ a concatenation operation (Cat) to separate the messages. Nevertheless, such an approach not only increases the space cost but may also retain additional redundant messages. To address these issues, OrderedGNN [21] proposes an adaptive concatenation mechanism (AdaCat) that can combine multiple messages with learnable dimensions. This is an innovative and worthy further exploration practice, but the difficulty of model learning should also be considered.

A.1.4 FUSE Function

Further, the FUSE functions integrate messages from multiple layers into the final representation. For the FUSE function, utilizing the representation of the last layer as the final representation is widely accepted: Z = Z L . JKNet [31] proposes that the combination of representations from intermediate layers can capture both local and global information. H2GCN [12] applies it in heterophilous graphs, preserving messages from different localities with concatenation. Similarly, GPRGNN [20] combines the representations of multiple layers into the final representation through adaptive weighted addition.

A.1.5 AGGREGATE function

The most commonly used AGGREGATE function is AGGREGATE ( A r , B r , Z l -1 r ) = ( A r ⊙ B r ) Z l -1 r W l r . We take this as the fixed form of the AGGREGATE function following. Actually, the input representations Z -1 r and weight matrixes W l r also can be specially designed. Taking the initial node representations Z 0 as input is a relatively common approach as in APPNP [26], GCNII [27], FAGCN [11] and GloGNN [19]. Further, GCNII [27] adds an identity matrix I w to the weight matrixes to keep more original messages. However, the methods that specially design these components are few and with a similar form. Thus, we don't discuss them too much, but leave it for future extensions.

A.2 Revisiting Representative GNNs with HTMP Mechanism

In this part, we utilize HTMP mechanism to revisit the representative GNNs. We start from homophilous GNNs as simple examples and further extend to heterophilous GNNs.

A.2.1 GCN

Graph Convolutional Networks (GCN) [1] utilizes a low-pass filter to gather messages from neighbors as follows:

It can be revisited by HTMP with the following components: 563

Specifically, GCN has a raw neighborhood indicator ˜ A and a degree averaging aggregation guidance B d . Since there is only one neighborhood, the COMBINE function is meaningless in GCN. GCN utilizes a naive way to fuse messages about the original neighborhood and central nodes. However, it may confuse the representations in heterophilous graphs.

A.2.2 APPNP

PPNP [26] is also a general method whose message passing is based on Personalized PageRank (PPR). To avoid massive consumption, APPNP is introduced as the approximate version of PPNP with an iterative message-passing mechanism:

It can be revisited by with the following components:

Specifically, APPNP aggregates messages from node ego and neighborhoods separately and combines them with a weighted addition. Compared with GCN, APPNP assigns adjustable weights to nodes, for controlling the proportion of ego and neighbor messages during message-passing, which becomes a worthy design in heterophilous graphs.

A.2.3 GAT

Going a step further, Graph Attention Networks (GAT) [28] allows learnable weights for each neighbor:

where α ij is the weight for aggregating neighbor node j to center node i , whose construction process 580 is as follows: 581

e

ij

=

LeakyReLU

Z

l

-

i

Let P GAT be the matrix of aggregation weights in GAT: 582

HTMP can revisit GAT with the following components: 583

which is the matrix version of Eq 16. Specifically, GAT aggregate messages from raw neighborhood 584 ˜ A with adaptive weights B aw . Aggregation guidance with adaptive weights is a nice idea, but simple 585 constraints are not enough for the model to learn ideal results. 586

([

1

|

Z

l

-

j

1

]

a

)

.

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

A.2.4 GCNII 587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

GCNII [27] is a novel homophilous GNN with two key designs: initial residual connection and identity mapping, which can be formatted as follows:

where α and β are two predefined parameters and I w ∈ R d r × d r is an identity matrix.

From the perspective of HTMP, it can be viewed as follows:

where the COMBINE function is weighted addition. Specifically, the first design of GCNII is a form of ego/neighbor separation, and the second design is a novel transformation weights matrix. This can also be specially designed, but only GCNII does this, so we won't analyze it too much and leave it as a future extension.

A.2.5 Geom-GCN

Geom-GCN [15] is one of the most influential heterophilous GNNs, which employs the geometric relationships of nodes within two kinds of neighborhoods to aggregate the messages through bi-level aggregation:

where ∥ denotes the concatenate operator, { g, s } is the set of neighborhoods including the original graph and the latent space. R is the set of geometric relationships. A i,r is the corresponding adjacency matrix in neighborhood i and relationship r .

It can be revisited by HTMP with the following components:

where the COMBINE function is concatenation and the weight matrix W l in Eq 22 can be viewed as the combination of multiple W l i,r . Specifically, Geom-GCN redefines multiple neighborhoods based on the customized geometric relations in both raw and latent space. The messages are aggregated from each neighborhood and combined by a concatenation. This approach may be applicable to some datasets, yet it has weak universality.

A.2.6 H2GCN

H2GCN [12] is also an influential method with three key designs: ego- and neighbor-message separation, higher-order neighborhoods, and the combination of intermediate representations. Its single-layer representations are constructed as follows:

where ˆ A h 2 denotes the 2 -order adjacency matrix with normalization.

It can be revisited by HTMP with the following components:

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

630

631

632

633

634

635

636

which can form a weight matrix: 637

where the COMBINE function is concatenation. Meanwhile, H2GCN also uses the concatenation as the FUSE function. Specifically, H2GCN aggregates messages from the raw and 2-order neighborhoods in a layer of message passing and keeps them apart in the representations. The design of ego/neighbor separation is first introduced by H2GCN and gradually becomes a necessity for subsequent methods.

A.2.7 SimP-GCN

SimP-GCN [14] constructs an additional graph based on the feature similarity. It has two key concepts: (1) the information from the original graph and feature kNN graph should be balanced, and (2) each node can adjust the contribution of its node features. Specifically, the message passing in SimP-GCN is as follows:

where s l ∈ R n is a learnable score vector that balances the effect of the original and feature graphs, D l K = diag ( K l 1 , K l 2 , ..., K l n ) is a learnable diagonal matrix.

It can be revisited by HTMP with the following components:

where the COMBINE function is adaptive weighted addition. Specifically, SimP-GCN aggregates messages from ego, raw and feature-similarity-based neighborhoods, and combines them with node-specific learnable weights. The feature-similarity-based neighborhoods can provide more homophilous messages to enhance the discriminability of the compatibility matrix. However, it's still limited by the amount of information on node features.

A.2.8 FAGCN

FAGCN [11] proposes considering both low-frequency and high-frequency information simultaneously, and transferring them into the negative-allowable weights during message passing:

where α G ij can be negative as follows:

It can be revisited by HTMP with the following components: 638

where the COMBINE function is weighted addition, same as the matrix form of Eq 28. Specifically, 639 FAGCN aggregates messages from node ego and raw neighborhood with negative-allowable weights. 640 It has a similar form to GAT but allows for ego/neighbor separation and negative weights, which 641 means the model can capture the difference between center nodes and neighbors. 642

A.2.9 GGCN 643

GGCN [23] explicitly distinguishes between homophilous and heterophilous neighbors based on 644 node similarities, and assigns corresponding positive and negative weights: 645

where ˆ Z l = Z l -1 W l + b l , ˜ A l T = ˜ A ⊙T l is an adjacency matrix weighted by the structure property, β l 0 , β l 1 and β l 2 are learnable scalars. The neighbors are distinguished by the cosine similarity of node representations with a threshold of 0:

̸

It can be revisited by HTMP with the following components: 649

where A p and A n are discriminated by the representation similarities: 650

The COMBINE function is an adaptive weighted addition. Specifically, GGCN divides the raw neighborhood into positive and negative ones based on the similarities among node presentations. On this basis, it aggregates messages from node ego, positive and negative neighborhoods, and combines them with node-specific learnable weights. This approach allows for targeted processing for homophilous and heterophilous neighbors, yet can suffer from the accuracy of discrimination, which may lead to the accumulation of errors.

A.2.10 ACM-GCN

ACM-GCN [18] introduces 3 channels (identity, low pass and high pass) to capture different information and mixes them with node-wise adaptive weights:

where diag ( α l I ) , diag ( α l L ) , diag ( α l H ) ∈ R N × 1 are learnable weight vectors.

It can be revisited by HTMP with the following components:

651

652

653

654

655

656

657

658

659

660

661

where the COMBINE function is adaptive weighted addition. Specifically, ACM-GCN aggregates 662 node ego, low-frequency, and high-frequency messages from ego and raw neighborhoods, and 663 combines them with node-wise adaptive weights. With simple but effective designs, ACM-GCN 664 achieves outstanding performance, which shows that complicated designs are not necessary. 665

646

647

648

666

667

668

669

670

671

672

A.2.11 OrderedGNN

OrderedGNN [21] is a SOTA method that introduces a node-wise adaptive dimension concatenation function to combine messages from neighbors of different hops:

where P d ∈ R N × d r is designed to be matrix with each line P l d,i being a dimension indicate vector, which starts with continuous 1s while the others be 0s. In practice, to keep the differentiability, it's "soften" as follows:

where f l ξ is a learnable layer that fuses two messages.

It can be revisited by HTMP with the following components: 673

where the COMBINE function is concatenation with node-wise adaptive dimensions. Specifically, in 674 each layer, OrderedGNN aggregates messages from node ego and raw neighborhood and concatenates 675 them with learnable dimensions. Combined with the multi-layer architecture, this approach can 676 aggregate messages from neighbors of different hops and combine them not only with adaptive 677 contributions but also as separately as possible. 678

679

680

681

682

683

684

685

686

687

688

689

690

691

692

693

694

695

696

697

698

699

700

701

702

703

704

705

706

A.3 Analysis and Advice for Designing Models

The HTMP mechanism splits the message-passing mechanism of HTGNNs into multiple modules, establishing connections among methods. For instance, most message passing in HTGNNs have personalized processing for nodes. Some methods [24, 11, 13, 22] utilize the learnable aggregation guidance and some others [14, 18, 21, 23] count on learnable COMBINE functions. Though neighborhood redefining is commonly used in HTGNNs, there are also many methods [24, 11, 18, 20, 21] using only raw neighborhoods to handle heterophily and achieve good performance. Degree averaging, which plays the role of a low-pass filter to capture the smooth signals, can still work well in many HTGNNs [12, 14-16, 20]. High-order neighbor information may be helpful in heterophilous graphs. Existing HTGNNs utilize it in two ways: directly defining high-order [12, 13, 16, 17] or even full-connected [19] neighborhood indicators and by the multi-layer architecture of message passing [20, 21].

With the aid of HTMP, we can revisit existing methods from a unified and comprehensible perspective. An obvious observation is that the coordination among designs is important while good combinations with easy designs can also achieve wonderful results. For instance, in ACM-GCN [18], the separation and adaptive addition of ego, low-frequency, and high-frequency messages can accommodate the personalized conditions of each node. OrderedGNN's design [21], which includes an adaptive connection mechanism, ego/neighbor separation, and multi-layer architecture, allows discrete and adaptive combinations of messages from multi-hop neighborhoods. This advises us to take into account all components simultaneously when designing models. As an illustration, please be cautious about using multiple learnable components. Also, here are some additional model design tips and considerations. Please separate the messages from node ego and neighbors . When combining them afterward, whether by weighted addition or concatenation, this approach is at least harmless if not beneficial, especially when dealing with heterophilous graphs. Last but not least, try to design a model capable of personalized handling different nodes . Available components include but are not limited to, custom-defined neighborhood indicators, aggregation guidance with adaptive weights or estimated relationships, and learnable COMBINE functions. This is to accommodate the diversity and sparsity of neighborhoods that nodes in real-world graphs may have.

707

708

709

710

711

712

713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

728

729

730

731

732

733

734

735

736

737

738

739

740

741

742

743

744

745

746

747

748

749

750

751

752

753

754

755

756

757

B Related Works

Homophilous Graph Neural Networks . Graph Neural Networks (GNNs) have showcased impressive capabilities in handling graph-structured data. Traditional GNNs are predominantly founded on the assumption of homophily, broadly categorized into two classes: spectral-based GNNs and spatial-based GNNs. Firstly , spectral-based GNNs acquire node representations through graph convolution operations employing diverse graph filters [1, 32, 33]. Secondly , spatial-based methods gather information from neighbors and update the representation of central nodes through the message-passing mechanism [26, 28, 34]. Moreover, for a more comprehensive understanding of existing homophilous GNNs , several unified frameworks [35, 36] have been proposed. Ma et al. [35] propose that the aggregation process in some representative homophilous GNNs can be regarded as solving a graph denoising problem with a smoothness assumption. Zhu et al. [36] establishes a connection between various message-passing mechanisms and a unified optimization problem. However, these methods have limitations, as the aggregated representations may lose discriminability when heterophilous neighbors dominate [11, 12].

Heterophilous Graph Neural Networks . Recently, some heterophilous GNNs have emerged to tackle the heterophily problem [11-23]. Firstly , a commonly adopted strategy involves expanding the neighborhood with higher homophily or richer messages , such as high order neighborhooods [12, 13], feature-similarity-based neighborhoods [13, 14], and custom-defined neighborhoods [15, 22]. Secondly , some approaches [11, 17-19, 23] aim to leverage information from heterophilous neighbors , considering that not all heterophily is detrimental et al.[6]. Thirdly , some methods [12, 16, 20, 21] adapt to heterophily by extending the combine function in message passing, creating variations for addition and concatenation. On this basis, several works have reviewed existing heterophilous methods . Zheng et al. [8] and Zhu et al. [9] identifies effective designs in heterophilous GNNs and analyzes the relationship between heterophily and graph-related issues. Gong et al. [10] provide a higher-level perspective on learning heterophilous graphs, summarizing and classifying existing methods based on learning strategies, architectures, and applications. However, these reviews merely classify and list methods hierarchically, lacking unified understandings and not exploring the reason behind the effectiveness of message passing in heterophilous graphs.

C The Detail of Experiments on Synthetic Datasets

To explore the performance impact of homophily level, node degrees and compatibility matrix (CMs) on simple GNNs, we conduct some experiments on synthetic datasets.

C.1 Synthetic Datasets

We construct synthetic graphs considering the factors of homophily, CMs and degrees. For homophily, we set 3 levels including Lowh (0.2), Midh (0.5), and Highh (0.8). For CMs, we set two levels of discriminability, including Easy and Hard. For degrees, we set two levels including Lowdeg (4) and Highdeg (18). Note that with a certain homophily level, we can only control the non-diagonal elements of CMs. Thus, there are a total of 12 synthetic graphs following the above settings. These synthetic graphs are based on the Cora dataset, which provides node features and labels, which means, only the edges are constructed. We visualize the CMs of these graphs in Figure 3. Since there is no significant difference in CMs between low-degree and high-degree, we only plot the high-degree ones. Further, the edges are randomly constructed under the guidance of these CMs and degrees to form the synthetic graphs.

C.2 Experiments on Synthetic Datasets

We use GCN to analyze the performance impact of the above factors. The semi-supervised node classification performance of GCN is shown in Table 5 while the baseline performance of MLP (72.54 ± 2.18) is the same among these datasets since their difference is only on edges. From these results, we have some observations: (1) high homophily is not necessary, GCN can also work well on low homophily but discriminative CM; (2) low degrees have a negative impact on performance, especially when the CMs are relatively weak discriminative, this also indicates that nodes with lower degrees are more likely to have confused neighborhoods; and (3) when dealing with nodes with confused neighborhoods, GCN may contaminate central nodes with their neighborhoods' messages, which

0

1

2

3

4

5

6

0

1

2

3

4

5

6

0.80

0.10

0.00

0.00

0.00

0.00

0.10

0.10

0.81

0.10

0.00

0.00

0.00

0.00

0.00

0.10

0.80

0.10

0.00

0.00

0.00

0.00

0.00

0.10

0.81

0.09

0.00

0.00

0.00

0.00

0.00

0.09

0.81

0.10

0.00

0.00

0.00

0.00

0.00

0.10

0.80

0.10

0.09

0.00

0.00

0.00

0.00

0.10

0.81

highh_easy_highdeg

0.0

0.2

0.4

0.6

0.8

1.0

(a) Highh, Easy

0

1

2

3

4

5

6

0

1

2

3

4

5

6

0.51

0.25

0.00

0.00

0.00

0.00

0.24

0.25

0.50

0.25

0.00

0.00

0.00

0.00

0.00

0.24

0.51

0.25

0.00

0.00

0.00

0.00

0.00

0.25

0.49

0.26

0.00

0.00

0.00

0.00

0.00

0.26

0.50

0.24

0.00

0.00

0.00

0.00

0.00

0.25

0.51

0.24

0.24

0.00

0.00

0.00

0.00

0.24

0.52

midh_easy_highdeg

0.0

0.2

0.4

0.6

0.8

1.0

(b) Midh, Easy

0

1

2

3

4

5

6

0

1

2

3

4

5

6

0.20

0.40

0.00

0.00

0.00

0.00

0.40

0.40

0.21

0.39

0.00

0.00

0.00

0.00

0.00

0.40

0.20

0.40

0.00

0.00

0.00

0.00

0.00

0.39

0.20

0.40

0.00

0.00

0.00

0.00

0.00

0.40

0.20

0.40

0.00

0.00

0.00

0.00

0.00

0.40

0.20

0.41

0.39

0.00

0.00

0.00

0.00

0.41

0.20

lowh_easy_highdeg

0.0

0.2

0.4

0.6

0.8

1.0

(c) Lowh, Easy

0

1

2

3

4

5

6

0

1

2

3

4

5

6

0.81

0.04

0.03

0.02

0.03

0.03

0.04

0.04

0.80

0.04

0.03

0.03

0.03

0.03

0.03

0.04

0.81

0.04

0.03

0.03

0.03

0.03

0.04

0.04

0.81

0.04

0.03

0.03

0.03

0.03

0.03

0.04

0.81

0.04

0.03

0.03

0.03

0.03

0.02

0.04

0.81

0.04

0.04

0.03

0.03

0.03

0.03

0.04

0.81

highh_hard_highdeg

0.0

0.2

0.4

0.6

0.8

1.0

(d) Highh, Hard

0

1

2

3

4

5

6

0

1

2

3

4

5

6

0.50

0.11

0.07

0.07

0.07

0.07

0.11

0.11

0.48

0.12

0.07

0.07

0.07

0.07

0.07

0.11

0.50

0.10

0.07

0.07

0.07

0.07

0.07

0.11

0.51

0.10

0.07

0.07

0.07

0.07

0.07

0.10

0.51

0.10

0.07

0.07

0.07

0.07

0.07

0.11

0.51

0.10

0.11

0.07

0.07

0.07

0.07

0.10

0.51

midh_hard_highdeg

0.0

0.2

0.4

0.6

0.8

1.0

(e) Midh, Hard

0

1

2

3

4

5

6

0

1

2

3

4

5

6

0.18

0.22

0.10

0.10

0.09

0.09

0.22

0.21

0.19

0.21

0.10

0.10

0.09

0.10

0.09

0.22

0.18

0.22

0.09

0.10

0.10

0.09

0.10

0.21

0.19

0.22

0.10

0.09

0.09

0.10

0.09

0.22

0.19

0.21

0.09

0.09

0.09

0.09

0.10

0.22

0.18

0.23

0.22

0.10

0.09

0.10

0.09

0.21

0.19

lowh_hard_highdeg

0.0

0.2

0.4

0.6

0.8

1.0

(f) Lowh, Hard

leads to performance worse than MLP. This once again remind us the importance of ego/neighbor 758 separation. 759

760

D Empirical Evidence for the Conjecture about CM

In this part, we show the empirical evidence for the conjecture about CM as mentioned in Sec 4. 761 Specifically, we plot the observed and desired CM of ACM-GCN and GPRGNN in Figure 4. The 762 results show that ACM-GCN and GPRGNN have enhanced the discriminability of CM, which can be 763 empirical evidence for the conjecture. 764

The desired CMs are obtained as follows: For ACM-GCN, we leverage the learned weights in the 765 COMBINE function to rebuild a weighted adjacency matrix A acm based on the low-pass filter ˆ A 766 and high-pass filter I -ˆ A , then regard A acm as the neighborhood and calculate the desired CM. 767

0

1

2

3

4

0

1

2

3

4

0.45

0.34

0.15

0.04

0.02

0.24

0.45

0.22

0.06

0.02

0.17

0.38

0.30

0.11

0.04

0.13

0.32

0.31

0.17

0.08

0.14

0.27

0.27

0.18

0.12

Real M

0.0

0.2

0.4

0.6

0.8

1.0

(a) Observed

0

1

2

3

4

0

1

2

3

4

0.68

0.20

0.09

0.02

0.01

0.14

0.68

0.13

0.04

0.01

0.10

0.22

0.59

0.07

0.02

0.08

0.19

0.18

0.51

0.04

0.08

0.15

0.16

0.10

0.50

M

0.0

0.2

0.4

0.6

0.8

1.0

(b) ACM-GCN

0

1

2

3

4

0

1

2

3

4

0.51

0.28

0.14

0.04

0.02

0.20

0.51

0.19

0.07

0.03

0.15

0.30

0.40

0.10

0.04

0.13

0.26

0.25

0.29

0.07

0.15

0.22

0.23

0.13

0.26

M

0.0

0.2

0.4

0.6

0.8

1.0

(c) GPRGNN

For GPRGNN, we utilize the leaned weights in the FUSE function to rebuild a weighted adjacency 768 matrix A gpr based on the multi-hop adjacency matrixes [ I , A , A 2 , ..., A k ] then regard A gpr as the 769 neighborhood and calculate the desired CM. 770

771

772

773

E Additional Detailed Implementation of CMGNN

Overall Message Passing Mechanism. The overall message passing mechanism in CMGNN is formatted as follows:

where diag ( α l 0 ) , diag ( α l 1 ) , diag ( α l 2 ) R N × 1 are the learned combination weights introduced below. 774

COMBNIE Function with Adaptive Weights. Firstly, we list the aggregated messages ˜ Z l r from 3 775 neighborhoods: 776

The combination weights are learned by an MLP with Softmax:

where W l att ∈ R (3 d r +1) × 3 and W l mix ∈ R 3 × 3 are two learnable weight matrixes, d is the node degrees which may be helpful to weights learning.

The Message Passing of Supplementary Prototypes. In practice, the virtual prototype nodes are viewed as additional nodes, which have the same message passing mechanism as real nodes:

where A sup,ptt = 1 ∈ R K × K and B sup,ptt = ˆ C ptt ˆ M are similar with those of real nodes.

Update Strategy for the Estimation of the Compatibility Matrix. For the sake of efficiency, we do not estimate the compatibility matrix in each epoch. Instead, we save it as fixed parameters and only update it when the evaluation performance is improved during the training.

Predition of CMGNN. CMGNN leverages the prediction of the model during message passing. For initialization, nodes have the same probabilities belonging to each class. During the message passing, the prediction soft label ˆ C is replaced by the output of CMGNN, formatted as follow:

where CLA is a classifier implemented by an MLP and Z is the final node representations.

F More Detail about the Benchmark

777

778

779

780

781

782

783

784

785

786

787

788

789

790

791

792

793

In this section, we describe the details of the new benchmarks, including (i) the reason why we need a new benchmark: drawbacks of existing datasets; (ii) detailed descriptions of new datasets; (iii) baseline methods and the codebase; and (iv) details of obtaining benchmark performance.

F.1 Drawbacks in Existing Datasets 794

As mentioned in [30], the widely used datasets Cornell, Texas, and Wisconsin 2 have a too small scale 795 for evaluation. Further, the original datasets Chameleon and Squirrel have an issue of data leakage, 796

where some nodes may occur simultaneously in both training and testing sets. Then, the splitting 797 ratio of training, validation, and testing sets are different across various datasets, which is ignored in 798 previous works. 799

800

801

802

Therefore, to build a comprehensive and fair benchmark for model effectiveness evaluation, we will newly organize 10 datasets with unified splitting across various homophily values in the next Subsection F.2.

803

804

805

806

807

808

809

810

811

812

813

814

815

816

817

818

819

820

821

822

823

824

825

826

827

828

829

830

831

832

833

834

835

836

F.2 New Datasets

In our benchmark, we adopt ten different types of publicly available datasets with a unified splitting setting (48%/32%/20% for training/validation/testing) for fair model comparison, including RomanEmpire [30], Amazon-Ratings [30], Chameleon-F [30], Squirrel-F [30], Actor [15], Flickr [37], BlogCatalog [37], Wikics [38], Pubmed [39], and Photo [29]. The datasets have a variety of homophily values from low to high. The statistics and splitting of these datasets are shown in Table 6. The detailed description of the datasets is as follows:

Roman-Empire 3 [30] is derived from the extensive article on the Roman Empire found on the English Wikipedia, chosen for its status as one of the most comprehensive entries on the platform. It contains 22,662 nodes and 65,854 edges between nodes. Each node represents an individual word from the text, with the total number of nodes mirroring the length of the article. An edge between two nodes is established under one of two conditions: the words are sequential in the text or they are linked in the sentence's dependency tree, indicating a grammatical relationship where one word is syntactically dependent on the other. Consequently, the graph is structured as a chain graph, enriched with additional edges that represent these syntactic dependencies. The graph encompasses a total of 18 distinct node classes, with each node being equipped with 300-dimensional attributes obtained by fastText word embeddings [40].

Amazon-Ratings 3 [30] is sourced from the Amazon product co-purchasing network metadata dataset [41]. It contains 24,492 nodes and 186,100 edges between nodes. The nodes within this graph represent products, encompassing a variety of categories such as books, music CDs, DVDs, and VHS video tapes. An edge between nodes signifies that the respective products are often purchased together. The objection is to forecast the average rating assigned to a product by reviewers, with the ratings being categorized into five distinct classes. For the purpose of node feature representation, we have utilized the 300-dimensional mean values derived from fastText word embeddings [40], extracted from the textual descriptions of the products.

Chameleon-F and Squirrel-F 3 [30] are specialized collections of Wikipedia page-to-page networks [42], of which the data leakage nodes are filtered out by [30]. Within these datasets, each node symbolizes a web page, and edges denote the mutual hyperlinks that connect them. The node features are derived from a selection of informative nouns extracted directly from Wikipedia articles. For the purpose of classification, nodes are categorized into five distinct groups based on the average monthly web traffic they receive. Specifically, Chameleon-F contains 890 nodes and 13,584 edges between nodes, with each node being equipped with 2,325-dimensional features. Squirrel-F contains 2,223 nodes and 65,718 edges between nodes, with each node being equipped with a 2,089-dimensional feature vector.

837

838

839

840

841

842

843

844

845

846

847

848

849

850

851

852

853

854

855

856

857

858

859

860

861

862

863

864

865

866

867

868

869

870

871

872

873

874

875

876

877

878

879

880

881

882

883

Actor 4 [15] is an actor-centric induced subgraph derived from the broader film-director-actor-writer network, as originally presented by [43]. In this refined network, each node corresponds to an individual actor, and the edges signify the co-occurrence of these actors on the same Wikipedia page. The node features are identified through the presence of certain keywords found within the actors' Wikipedia entries. For the purpose of classification, the actors are organized into five distinct categories based on the words of the actor's Wikipedia. Statistically, it contains 7,600 nodes and 30,019 edges between nodes, with each node being equipped with a 932-dimensional feature vector.

Flickr and Blogcatalog 5 [37] are two datasets of social networks, originating from the blog-sharing platform BlogCatalog and the photo-sharing platform Flickr, respectively. Within these datasets, nodes symbolize the individual users of the platforms, while links signify the followship relationships that exist between them. In the context of social networks, users frequently create personalized content, such as publishing blog posts or uploading and sharing photos with accompanying tag descriptions. These textual contents are consequently treated as attributes associated with each node. The classification objection is to predict the interest group of each user. Specifically, Flickr contains 7,575 nodes and 479,476 edges between nodes. The graph encompasses a total of 9 distinct node classes, with each node being equipped with a 12047-dimensional attribute vector. BlogCatalog contains 5,196 nodes and 343,486 edges between nodes. The graph encompasses a total of 6 distinct node classes, with each node being equipped with 8189-dimensional attributes.

Wikics 6 [38] is a dataset curated from Wikipedia, specifically designed for benchmarking the performance of GNNs. It is meticulously constructed around 10 distinct categories that represent various branches of computer science, showcasing a high degree of connectivity. The node features are extracted from the text of the associated Wikipedia articles, leveraging the power of pretrained GloVe word embeddings [44]. These features are computed as the average of the word embeddings, yielding a comprehensive 300-dimensional representation for each node. The dataset encompasses a substantial network of 11,701 nodes interconnected by 431,206 edges.

Pubmed 7 [39] is a classical citation network consisting of 19,717 scientific publications with 44,338 links between them. The text contents of each publication are treated as their node attributes, and thus each node is assigned a 500-dimensional attribute vector. The target is to predict which of the paper categories each node belongs to, with a total of 3 candidate classes.

Photo 8 [29] is one of the Amazon subset network from [29]. Nodes in the graph represent goods and edges represent that two goods are frequently bought together. Given product reviews as bag-of-words node features, each node is assigned a 745-dimensional feature vector. The task is to map goods to their respective product category. It contains 7,650 nodes and 238,162 edges between nodes. The graph encompasses a total of 8 distinct product categories.

F.3 Baseline Methods and the Codebase

For comprehensive comparisons, we choose 13 representative homophilous and heterophilous GNNs as baseline methods in the benchmark, including (i) Shallow base model: MLP; (ii) Homopihlous GNNs: GCN, GAT, GCNII; and (iii) Heterophilous GNNs: H2GCN, MixHop, GBK-GNN, GGCN, GloGNN, HOGGCN, GPR-GNN. Detailed descriptions of some of these methods can be seen in Appendix A.2.

To explore the performance of baseline methods on new datasets and facilitate future expansions, we collect the official/reproduced codes from GitHub and integrate them into a unified codebase. Specifically, all methods share the same data loaders and evaluation metrics. One can easily run different methods with only parameters changing within the codebase. The codebase is based on the PyTorch 9 framework, supporting DGL 10 and PyG 11 . Detailed usages of the codebase are available in the Readme file of the codebase.

884

885

886

887

888

889

890

891

892

893

894

895

896

897

898

899

900

901

902

903

904

905

906

907

908

909

910

911

912

913

914

915

916

F.4 Details of Obtaining Benchmark Performance

Following the settings in existing methods, we construct 10 random splits (48%/32%/20% for train/valid/test) for each dataset and report the average performance among 10 runs on them along with the standard deviation.

For all baseline methods except MLP, GCN, and GAT, we conduct parameter searches within the search space recommended by the original papers. The searches are based on the NNI framework with an anneal strategy. We use Adam as the optimizer for all methods. Each method has dozens of search trails according to their time costs and the best performances are reported. The currently known optimal parameters of each method are listed in the codebase. We run these experiments on NVIDIA GeForce RTX 3090 GPU with 24G memory. The out-of-memory error during model training is reported as OOM in Table 2.

G More Details about Experiments

In this section, we describe the additional details of the experiments, including experimental settings and results.

G.1 Additional Experimental Settings

Our method has the same experimental settings within the benchmark, including datasets, splits, evaluations, hardware, optimizer and so on as in Appendix F.4.

Parameters Search Space. We list the search space of parameters in Table 7, where patience is for early stopping, nhidden is the embedding dimension of hidden layers as well as the representation dimension d r , relu_varient decides ReLU applying before message aggregation or not as in ACMGCN, structure_info determines whether to use structure information as supplement node features or not.

Ablation Study. In the ablation study, there are three variants of our methods: without SM, without DL, without SM and DL. For "without SM", we delete the supplementary messages during message passing, using only messages from node ego and raw neighborhood for combination. For "without DL", we simply set λ = 0 to delete the discrimination loss. For "without SM and DL", we just combine the above two settings.

G.2 Additional Experimental Results

In this subsection, we show some additional experimental results and analysis.

G.2.1 Additional Results of CM Estimations

The additional visualizations of CM estimations are shown in Figure 5. As we can see, our method can estimate quite accurate CMs among various homophily and class numbers, which provides a good foundation for the construction of supplementary messages.

0

1

2

3

4

5

6

7

8

9

10

1

12

13

14

15

16

17

0

1

2

3

4

5

6

7

8

9

10

1

12

13

14

15

16

17

0.0

0.05

0.15

0.06

0.02

0.05

0.18

0.04

0.07

0.04

0.0

0.03

0.01

0.05

0.05

0.01

0.0

0.17

0.02

0.0

0.4

0.14

0.1

0.05

0.03

0.05

0.0

0.01

0.03

0.01

0.02

0.01

0.01

0.02

0.01

0.06

0.06

0.4

0.02

0.1

0.06

0.05

0.03

0.01

0.05

0.01

0.01

0.0

0.01

0.0

0.01

0.01

0.02

0.09

0.04

0.26

0.20

0.0

0.1

0.05

0.07

0.02

0.06

0.01

0.03

0.0

0.02

0.0

0.02

0.01

0.01

0.09

0.01

0.23

0.10

0.15

0.08

0.07

0.07

0.03

0.07

0.02

0.03

0.0

0.02

0.0

0.02

0.01

0.01

0.07

0.05

0.1

0.09

0.05

0.09

0.15

0.04

0.18

0.05

0.03

0.02

0.01

0.02

0.01

0.01

0.01

0.01

0.07

0.20

0.09

0.06

0.1

0.09

0.05

0.02

0.02

0.02

0.03

0.03

0.03

0.03

0.0

0.0

0.01

0.10

0.12

0.07

0.18

0.02

0.05

0.09

0.35

0.03

0.0

0.05

0.03

0.02

0.0

0.02

0.0

0.01

0.01

0.01

0.06

0.1

0.0

0.18

0.12

0.1

0.09

0.02

0.04

0.0

0.01

0.02

0.01

0.01

0.0

0.0

0.03

0.07

0.17

0.12

0.07

0.09

0.06

0.08

0.08

0.05

0.05

0.02

0.09

0.0

0.01

0.02

0.04

0.0

0.0

0.03

0.17

0.01

0.25

0.08

0.14

0.12

0.08

0.07

0.03

0.05

0.0

0.05

0.0

0.03

0.0

0.02

0.01

0.0

0.05

0.15

0.07

0.0

0.0

0.0

0.06

0.12

0.01

0.02

0.02

0.0

0.05

0.01

0.05

0.03

0.0

0.09

0.31

0.03

0.15

0.08

0.07

0.07

0.09

0.09

0.04

0.04

0.03

0.03

0.01

0.09

0.01

0.02

0.01

0.02

0.1

0.29

0.10

0.0

0.0

0.0

0.08

0.0

0.01

0.01

0.06

0.0

0.09

0.01

0.0

0.19

0.0

0.05

0.10

0.20

0.05

0.09

0.1

0.10

0.06

0.0

0.02

0.01

0.01

0.03

0.04

0.02

0.15

0.0

0.01

0.04

0.08

0.03

0.23

0.17

0.04

0.07

0.04

0.06

0.02

0.10

0.01

0.01

0.0

0.01

0.0

0.01

0.01

0.02

0.16

0.0

0.12

0.12

0.04

0.02

0.02

0.24

0.01

0.14

0.04

0.0

0.06

0.02

0.02

0.03

0.01

0.0

0.10

0.10

0.10

0.13

0.08

0.05

0.05

0.05

0.02

0.06

0.04

0.01

0.04

0.03

0.01

0.01

0.03

0.02

0.18

Real M

0.0

0.2

0.4

0.6

0.8

1.0

(a) Roman-Empire Real

0

1

2

3

4

5

6

7

8

9

10

1

12

13

14

15

16

17

0

1

2

3

4

5

6

7

8

9

10

1

12

13

14

15

16

17

0.03

0.06

0.15

0.06

0.02

0.06

0.19

0.05

0.06

0.05

0.0

0.03

0.01

0.06

0.03

0.01

0.0

0.15

0.02

0.01

0.4

0.14

0.1

0.04

0.03

0.04

0.0

0.01

0.03

0.01

0.01

0.01

0.0

0.01

0.01

0.06

0.07

0.4

0.01

0.1

0.06

0.05

0.04

0.01

0.04

0.02

0.01

0.0

0.01

0.0

0.01

0.01

0.02

0.10

0.04

0.26

0.20

0.0

0.12

0.04

0.08

0.02

0.06

0.02

0.03

0.0

0.02

0.0

0.01

0.01

0.01

0.09

0.02

0.24

0.1

0.15

0.09

0.07

0.08

0.03

0.06

0.02

0.03

0.0

0.02

0.0

0.01

0.01

0.01

0.06

0.05

0.10

0.09

0.05

0.09

0.17

0.04

0.19

0.04

0.03

0.02

0.01

0.02

0.01

0.01

0.01

0.01

0.06

0.20

0.08

0.07

0.1

0.09

0.06

0.03

0.02

0.02

0.02

0.03

0.03

0.02

0.02

0.0

0.01

0.10

0.10

0.08

0.18

0.03

0.05

0.09

0.38

0.03

0.0

0.04

0.02

0.02

0.0

0.01

0.0

0.01

0.01

0.01

0.05

0.10

0.01

0.18

0.13

0.12

0.08

0.03

0.04

0.01

0.01

0.03

0.01

0.01

0.0

0.0

0.03

0.06

0.15

0.13

0.08

0.1

0.06

0.08

0.08

0.05

0.04

0.02

0.10

0.0

0.01

0.01

0.04

0.0

0.0

0.03

0.16

0.01

0.24

0.06

0.15

0.12

0.08

0.09

0.02

0.05

0.01

0.05

0.0

0.04

0.0

0.01

0.01

0.0

0.05

0.19

0.07

0.01

0.0

0.0

0.06

0.16

0.01

0.02

0.02

0.0

0.06

0.01

0.07

0.02

0.0

0.07

0.23

0.04

0.15

0.08

0.09

0.09

0.10

0.08

0.04

0.04

0.02

0.05

0.01

0.07

0.02

0.01

0.01

0.01

0.10

0.32

0.10

0.01

0.0

0.0

0.06

0.10

0.01

0.0

0.06

0.0

0.08

0.02

0.0

0.09

0.0

0.04

0.10

0.23

0.05

0.10

0.10

0.10

0.05

0.01

0.02

0.01

0.01

0.03

0.03

0.02

0.14

0.0

0.01

0.03

0.06

0.04

0.21

0.17

0.04

0.07

0.04

0.08

0.02

0.09

0.01

0.01

0.0

0.01

0.0

0.0

0.02

0.02

0.16

0.01

0.12

0.14

0.04

0.02

0.02

0.26

0.01

0.12

0.04

0.0

0.04

0.01

0.02

0.01

0.01

0.01

0.12

0.10

0.1

0.15

0.08

0.05

0.05

0.06

0.02

0.06

0.04

0.01

0.02

0.02

0.01

0.01

0.03

0.02

0.16

Est M

0.0

0.2

0.4

0.6

0.8

1.0

(b) Roman-Empire Est

0

1

2

3

4

0

1

2

3

4

0.22

0.22

0.11

0.12

0.34

0.18

0.17

0.20

0.20

0.25

0.11

0.12

0.23

0.22

0.32

0.06

0.11

0.23

0.24

0.35

0.07

0.08

0.20

0.24

0.41

Real M

0.0

0.2

0.4

0.6

0.8

1.0

(c) Chameleon-F Real

0

1

2

3

4

0

1

2

3

4

0.28

0.16

0.10

0.10

0.36

0.29

0.14

0.19

0.14

0.24

0.14

0.10

0.27

0.18

0.31

0.07

0.06

0.19

0.26

0.42

0.11

0.06

0.20

0.25

0.38

Est M

0.0

0.2

0.4

0.6

0.8

1.0

(d) Chameleon-F Est

0

1

2

3

4

0

1

2

3

4

0.17

0.18

0.17

0.20

0.29

0.15

0.17

0.18

0.21

0.28

0.14

0.17

0.19

0.22

0.28

0.14

0.17

0.17

0.24

0.28

0.14

0.16

0.17

0.24

0.29

Real M

0.0

0.2

0.4

0.6

0.8

1.0

0

1

2

3

4

0

1

2

3

4

0.25

0.14

0.12

0.22

0.28

0.26

0.15

0.13

0.20

0.25

0.27

0.16

0.13

0.20

0.24

0.29

0.16

0.12

0.20

0.23

0.28

0.15

0.13

0.20

0.25

Est M

0.0

0.2

0.4

0.6

0.8

1.0

0

1

2

3

4

0

1

2

3

4

0.11

0.17

0.19

0.29

0.24

0.10

0.17

0.19

0.30

0.24

0.11

0.15

0.20

0.29

0.25

0.10

0.16

0.19

0.30

0.24

0.11

0.17

0.19

0.28

0.26

Real M

0.0

0.2

0.4

0.6

0.8

1.0

(g) Actor Real

0

1

2

3

4

0

1

2

3

4

0.09

0.16

0.20

0.24

0.31

0.08

0.16

0.22

0.24

0.29

0.08

0.15

0.23

0.24

0.31

0.08

0.15

0.22

0.25

0.29

0.09

0.16

0.22

0.24

0.30

Est M

0.0

0.2

0.4

0.6

0.8

1.0

(h) Actor Est

0

1

2

3

4

5

6

7

8

0

1

2

3

4

5

6

7

8

0.41

0.09

0.03

0.06

0.05

0.07

0.18

0.04

0.07

0.13

0.21

0.10

0.06

0.06

0.10

0.16

0.09

0.09

0.08

0.11

0.21

0.06

0.10

0.06

0.12

0.19

0.08

0.11

0.11

0.08

0.23

0.07

0.09

0.12

0.10

0.10

0.10

0.18

0.12

0.06

0.12

0.10

0.13

0.10

0.08

0.15

0.11

0.05

0.06

0.07

0.17

0.18

0.13

0.08

0.17

0.11

0.05

0.06

0.06

0.09

0.29

0.09

0.09

0.09

0.11

0.10

0.06

0.08

0.09

0.16

0.22

0.10

0.10

0.10

0.06

0.06

0.06

0.08

0.13

0.08

0.33

Real M

0.0

0.2

0.4

0.6

0.8

1.0

(i) Flickr Real

0

1

2

3

4

5

6

7

8

0

1

2

3

4

5

6

7

8

0.40

0.10

0.03

0.08

0.04

0.06

0.18

0.05

0.07

0.13

0.20

0.09

0.09

0.07

0.08

0.16

0.09

0.09

0.08

0.12

0.19

0.08

0.10

0.06

0.12

0.17

0.09

0.11

0.11

0.08

0.24

0.07

0.07

0.12

0.10

0.10

0.10

0.16

0.12

0.09

0.11

0.09

0.13

0.10

0.09

0.15

0.11

0.06

0.09

0.07

0.14

0.17

0.12

0.10

0.17

0.11

0.05

0.08

0.05

0.08

0.28

0.09

0.09

0.09

0.12

0.10

0.09

0.08

0.07

0.15

0.21

0.10

0.11

0.10

0.06

0.08

0.06

0.08

0.12

0.08

0.31

Est M

0.0

0.2

0.4

0.6

0.8

1.0

(j) Flickr Est

0

1

2

3

4

5

6

7

8

9

0

1

2

3

4

5

6

7

8

9

0.75

0.04

0.03

0.01

0.01

0.02

0.00

0.01

0.02

0.06

0.01

0.56

0.07

0.03

0.03

0.02

0.01

0.05

0.03

0.07

0.00

0.01

0.66

0.11

0.04

0.03

0.04

0.03

0.01

0.07

0.00

0.01

0.15

0.72

0.02

0.01

0.01

0.03

0.01

0.03

0.00

0.01

0.09

0.03

0.71

0.05

0.01

0.03

0.02

0.02

0.00

0.01

0.10

0.03

0.09

0.64

0.01

0.04

0.03

0.03

0.00

0.01

0.25

0.06

0.04

0.02

0.54

0.03

0.01

0.02

0.00

0.04

0.11

0.07

0.06

0.04

0.02

0.51

0.03

0.06

0.00

0.03

0.11

0.03

0.06

0.08

0.01

0.08

0.43

0.12

0.00

0.03

0.12

0.05

0.02

0.02

0.01

0.02

0.02

0.70

Real M

0.0

0.2

0.4

0.6

0.8

1.0

(k) Wikics Real

0

1

2

3

4

5

6

7

8

9

0

1

2

3

4

5

6

7

8

9

0.73

0.04

0.05

0.02

0.02

0.01

0.00

0.01

0.03

0.09

0.02

0.66

0.10

0.02

0.03

0.01

0.01

0.05

0.03

0.09

0.00

0.01

0.74

0.08

0.04

0.02

0.03

0.02

0.01

0.06

0.00

0.00

0.15

0.76

0.01

0.01

0.01

0.02

0.00

0.03

0.00

0.01

0.10

0.02

0.74

0.05

0.01

0.03

0.01

0.02

0.01

0.01

0.10

0.03

0.09

0.67

0.01

0.03

0.03

0.03

0.00

0.01

0.24

0.04

0.04

0.01

0.61

0.01

0.00

0.02

0.01

0.05

0.14

0.07

0.07

0.05

0.01

0.49

0.05

0.07

0.02

0.04

0.11

0.02

0.08

0.07

0.01

0.08

0.45

0.14

0.01

0.03

0.12

0.03

0.02

0.01

0.01

0.02

0.02

0.73

Est M

0.0

0.2

0.4

0.6

0.8

1.0

(l) Wikics Est

0

1

2

0

1

2

0.69

0.12

0.19

0.05

0.85

0.10

0.08

0.13

0.79

Real M

0.0

0.2

0.4

0.6

0.8

1.0

(m) Pubmed Real

0

1

2

0

1

2

0.72

0.12

0.17

0.04

0.88

0.07

0.07

0.09

0.84

Est M

0.0

0.2

0.4

0.6

0.8

1.0

(n) Pubmed Est

0

1

2

3

4

5

6

7

0

1

2

3

4

5

6

7

0.89

0.04

0.00

0.02

0.00

0.00

0.04

0.00

0.01

0.83

0.00

0.09

0.01

0.00

0.01

0.02

0.00

0.01

0.95

0.00

0.02

0.00

0.00

0.00

0.01

0.13

0.00

0.71

0.03

0.00

0.05

0.06

0.00

0.03

0.01

0.05

0.79

0.00

0.10

0.02

0.00

0.00

0.01

0.00

0.00

0.96

0.00

0.00

0.01

0.01

0.00

0.02

0.03

0.00

0.86

0.05

0.01

0.06

0.00

0.14

0.02

0.00

0.22

0.55

Real M

0.0

0.2

0.4

0.6

0.8

1.0

0

1

2

3

4

5

6

7

0

1

2

3

4

5

6

7

0.90

0.04

0.00

0.01

0.00

0.00

0.04

0.00

0.01

0.86

0.00

0.09

0.01

0.00

0.01

0.02

0.00

0.00

0.98

0.00

0.01

0.00

0.00

0.00

0.01

0.13

0.00

0.72

0.04

0.00

0.04

0.07

0.00

0.02

0.01

0.05

0.80

0.00

0.09

0.03

0.00

0.00

0.00

0.00

0.00

0.99

0.00

0.00

0.01

0.01

0.00

0.01

0.04

0.00

0.87

0.06

0.00

0.06

0.00

0.15

0.03

0.00

0.20

0.55

Est M

0.0

0.2

0.4

0.6

0.8

1.0

(p) Photo Est

G.2.2 Additional Performance on Nodes with Various Levels of Degrees. 917

We show the additional performance on nodes with various degrees in Table 8. The results show that 918 CMGNN can achieve relatively good performance on low-degree nodes, especially on heterophilous 919 graphs. For the opposite results on homophilous graphs, we guess it may be due to the low-degree 920 nodes in homophilous graphs having a more discriminative semantic neighborhood, such as a one-hot 921 form. On the contrary, there are relatively more high-degree nodes with confused neighborhoods due 922 to the randomness, which leads to the shown results on homophilous graphs. 923

924

925

926

927

928

929

930

931

932

933

934

935

G.2.3 Efficiency Study

Complexity Analysis. The number of learnable parameters in layer l of CMGNN is 3 d r ( d r +1)+9 , compared to d r d r in GCN and 3 d r ( d r +1) + 9 in ACM-GCN. The time complexity of layer l is composed of 3 parts (i) AGGREGATE function: O ( Nd r 2 ) , O ( Nd r 2 + Md r ) and O ( Nd r 2 + NKd r ) for identity neighborhood, raw neighborhood and the supplementary neighborhood respectively, where M = |E| denotes the number of edges; (ii) COMBINE function: O (3 N (3 d r +1)+12 N ) for adaptive weights calculating and O (3 N ) for combination; (iii) FUSE function: O (1) for concatenations. To this end, the time complexity of CMGNN is O ( Nd r (3 d r + K + 9) + Md r + 18 N + 1) , or O ( Nd r 2 + Md r ) for brevity.

Experimental Running Time. we report the actual average running time (ms per epoch) of baseline methods and CMGNN in Table 9 for comparison. The results demonstrate that CMGNN can balance both performance effectiveness and running efficiency.

936

937

938

939

940

941

942

943

944

945

946

947

948

949

950

951

952

953

954

955

956

957

958

959

960

961

962

963

964

965

966

967

968

969

970

971

972

973

974

975

976

977

978

979

980

981

982

983

984

985

986

987

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The contributions and scope of this paper are included in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of this work are listed in the Sec 7.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

988

989

990

991

992

993

994

995

996

997

998

999

1000

1001

1002

1003

1004

1005

1006

1007

1008

1009

1010

1011

1012

1013

1014

1015

1016

1017

1018

1019

1020

1021

1022

1023

1024

1025

1026

1027

1028

1029

1030

1031

1032

1033

1034

1035

1036

1037

1038

1039

1040

1041

Justification: We provide the formalization analysis in Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The details of the method and experimental settings are provided in Appendix. Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

1042

1043

1044

1045

1046

1047

1048

1049

1050

1051

1052

1053

1054

1055

1056

1057

1058

1059

1060

1061

1062

1063

1064

1065

1066

1067

1068

1069

1070

1071

1072

1073

1074

1075

1076

1077

1078

1079

1080

1081

1082

1083

1084

1085

1086

1087

1088

1089

1090

1091

1092

Answer: [Yes]

Justification: The data and code are available in the supplementary material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The detailed experimental settings are provided in Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report the average accuracy and the standard deviation as the performance in experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

1093

1094

1095

1096

1097

1098

1099

1100

1101

1102

1103

1104

1105

1106

1107

1108

1109

1110

1111

1112

1113

1114

1115

1116

1117

1118

1119

1120

1121

1122

1123

1124

1125

1126

1127

1128

1129

1130

1131

1132

1133

1134

1135

1136

1137

1138

1139

1140

1141

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We list the hardware and software resources along with the space and space complexity in Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in this paper conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The potential positive societal impacts are provided in the Introduction while the potential negative societal impacts are meaningless since this work is a foundational research.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

1142

1143

1144

1145

1146

1147

1148

1149

1150

1151

1152

1153

1154

1155

1156

1157

1158

1159

1160

1161

1162

1163

1164

1165

1166

1167

1168

1169

1170

1171

1172

1173

1174

1175

1176

1177

1178

1179

1180

1181

1182

1183

1184

1185

1186

1187

1188

1189

1190

1191

1192

1193

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risk. The datasets we used are all publicly available online.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The datasets and code of baseline methods are publicly available online. We cite the original paper and mark the URL in both papers and codebase.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

1194

1195

1196

1197

1198

1199

1200

1201

1202

1203

1204

1205

1206

1207

1208

1209

1210

1211

1212

1213

1214

1215

1216

1217

1218

1219

1220

1221

1222

1223

1224

1225

1226

1227

1228

1229

1230

1231

1232

1233

1234

1235

1236

1237

1238

1239

1240

1241

1242

1243

1244

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide a public codebase along with an illustrative README file.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

1245

1246

1247

1248

1249

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\n1. The paper unifies existing heterophilous graph neural networks (HTGNNs) into a Heterophilous Message-Passing (HTMP) mechanism.\n2. The authors reveal that the effectiveness of HTMP is due to increasing differences among node representations belonging to different classes.\n3. Guided by this revelation, the paper then introduces Compatibility Matrix-aware Graph Neural Network (CMGNN) to further enhance HTGNNs.\n4. The authors conduct fair evaluations and comparative analysis on multiple benchmark datasets, highlighting the superior performance of the HTMP mechanism and the proposed CMGNN method.\n\nSTRENGTHS:\n1. The claims are supported empirically by a detailed comparison across multiple benchmark datasets.\n2. The paper is well-written and clearly structured, with each section logically building on the previous ones.\n\nWEAKNESSES:\n1. Several research publications [1, 2, 3] have used compatible matrices to boost the effectiveness of GNNs on heterophilic graphs. In-depth qualitative and quantitative comparisons are missing from this submission. Including such analyses would significantly increase the importance of the contributions.\n2. Some claims made in the paper, such as Observation 1 and Observation 2 in Section 4, would benefit from additional analysis. For instance, including theoretical analysis with formal notations would provide more rigorous support for these claims.\n3. Existing survey articles have unified and categorised message passing on heterophilic graphs [4,5,6]. This submission should compare and position the proposed HTMP unification against these categorisations.\n4. The experiments lack a comparison of training times with baseline models. Including an analysis of the tradeoff between accuracy and training time would greatly enhance the results.\n\n\n\n\nReferences:\n1. Simplifying Node Classification on Heterophilous Graphs with Compatible Label Propagation, In TMLR'22,\n2. Explicit pairwise factorized graph neural network for semi-supervised node classification, In UAI'21,\n3. Graph Neural Networks with Heterophily, In AAAI'21,\n4. Graph Neural Networks for Graphs with Heterophily: A Survey,\n5. Learning from Graphs with Heterophily: Progress and Future,\n6. Heterophily and Graph Neural Networks: Past, Present and Future.\n\n\n\n**Edit post Rebuttal:**\nThe authors have promised to include detailed comparisons in a future revised version. Since these details cannot be verified within the review period, I will lower my confidence from 4 to 3. However, given that other major concerns have been addressed, I will raise my rating by one point from 4 to 5.\n\nQUESTIONS:\n1. Were there insights on how the compatible matrices utilised in CMGNN compared with the methods in the referenced publications [1, 2, 3]? What were the pros and cons of CMGNN to tackle heterophily with existing methods that use compatibility matrices?\n2. Were there theoretical foundations that support Observation 1 and Observation 2? What formal notations and theoretical analyses could be included to strengthen the support for these observations?\n3. How did the proposed HTMP unification differ from the categorisations presented in the existing survey articles [4, 5, 6]? What unique insights did the HTMP unification offer compared to existing unifications?\n4. What did the tradeoff between accuracy and training times of the proposed methods and the baselines look like?\n\nLIMITATIONS:\nThe authors have provided some discussion on the limitations of their work (for instance, see section 7 on Page 9).\n\nPotential negative societal impacts are not relevant to this study.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work revisits the message-passing mechanisms in existing HTGNNs and reformulates them into a unified heterophilous message-passing (HTMP) mechanism. Based on HTMP, the authors propose a new framework named CMGNN. Experiments on 10 datasets with 13 different baseline models demonstrate the effectiveness of the proposed framework.\n\nSTRENGTHS:\n1. This work proposes a unified heterophilous message-passing (HTMP) mechanism, which could be a guideline for further research on heterophilous GNN. \n2. Based on the HTMP mechanism, this work proposes a new framework named CMGNN, which is novel and has basic value.\n3. The effectiveness of the HTMP mechanism and CMGNN framework is well supported by experiment results.\n\nWEAKNESSES:\n1. Paper presentation could be further improved. For example, the conception of \"good\" heterophily and \"bad\" homophily deserves further explanation. There are some spelling mistakes, such as \"heterophilious\" in line 12.\n2. If space permits, I feel like moving experiments in Appendix C to the main body would be better for the introduction of *Observation 1*. \n3. It might be hard to follow as this paper has so many equations, especially those about CMGNN. So I suggest providing a flow chart or a pseudo algorithm for better understanding.\n4. Conclusions in this paper are mainly based on experiment results. It would be better if corresponding theoretical analysis or proofs are provided.\n\nQUESTIONS:\n1. What are the contributions to the newly built benchmark datasets? It seems that this work just collects 10 existing datasets and sets them with the same train-valid-test split ratio.\n2. The weighting function (Eq.9) seems to be quite empirical. Is it possible to further discuss it?\n\nLIMITATIONS:\nAs the authors mentioned, this work mainly focuses on semi-supervised settings, which could be further generalized.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to address the question of \"why does message passing remain effective on heterophilous graphs\" and proposes a unified framework called heterophilous message-passing (HTMP) mechanism. It extensively reviews the architecture of existing heterophilous GNNs under this framework. It then moves on to discuss the empirical observation that the success of message passing in existing heterophilous GNNs is attributed to their implicitly enhancement of the compatibility matrix among classes, and proposed a new GNN approach called CMGNN to further enhance the separability of the compatibility matrix for different classes in the message passing process. The paper includes an extensive empirical analysis involving 10 benchmark datasets and 13 well-established baseline GNNs, and show that the proposed CMGNN approach has the best overall performance against the baselines.\n\nSTRENGTHS:\n- The writing is clear and well-organized for most parts of the paper; \n- The paper gives an extensive survey of existing message-passing GNNs under the HTMP mechanism in Table 1 and Appendix A. \n- The experiments are well-thought and extensive: it addresses the drawbacks of the previous homophilous and heterophilous node classification benchmarks identified in previous works by using more recent benchmark datasets, and include 13 baselines for a comprehensive evaluation of the proposed method.\n- The proposed approach, CMGNN, has the best overall performance against 13 baselines on 10 benchmark datasets.\n\nWEAKNESSES:\n- This work builds upon the findings of several previous works regarding the effective designs for GNNs under heterophily and when is heterophily challenging (or in other words, \"bad\") for GNNs. While the authors cited these works in some parts of the paper ([6,9,12,18] in References), I feel that **some of the observations in the paper overlapped with the findings in previous works, and their connections and differences are not clearly stated in the paper**. \n  - For example, Observation 1 seems to overlap with the previous observations made in [6] (\"to ensure that the neighborhood patterns for nodes with different labels are distinguishable, the inter- class similarity should be low\") and [9] (\"two key factors, low-degree nodes and complex compatibility matrices, deteriorate the distinguishability of the neighborhood label distributions when coupled with heterophily, thus making heterophily a unique challenge for GNNs in most cases\"). \n  - Given this, I also think that the claim in the related work section (line 732-734) that \"these reviews ... not exploring the reason behind the effectiveness of message passing in heterophilous graphs\" is inaccurate, as this paper is in fact built upon these analyses regarding the effectiveness of message passing in heterophilous graphs. \n\n- Section 5 (method) is too condensed to present a clear picture of how the proposed Compatibility Matrix-Aware GNN (CMGNN) works for the readers. For example, it is unclear what \"topology structure\" that the authors are considering as \"additional available node features\", and the term in Eq. 7 is not well explained. The authors also didn't explain clearly in the main paper how is the \"soft pseudo labels\" being generated for the model. It will help with the understanding if the authors can include a figure showing the architecture of the proposed CMGNN model. I feel the \"method\" section is the most novel part in the paper and deserves more length in the paper. \n\n- It would be good to analyze the computational complexity and/or compare the empirical runtime of the model with the baselines. \n\n- As a minor point, the \"Norm\" term in Eq. 3 should be explained as \"L1 normalization for matrix row vectors\" to avoid the confusion that the normalization is done with the L1 norm for *matrix* (instead of for vectors).\n\nQUESTIONS:\n- As per Weakness point 1, can you describe how the observations in the paper are connected to, and different with, prior observations in [6,9,12,18]? \n- As per Weakness point 2, what is the \"topology structure\" that you considered as \"additional available node features\"? How does the use of these features affect the performance of the proposed model? \n- Can you provide some analysis regarding the computational complexity and/or the empirical runtime of the model?\n\nLIMITATIONS:\nThe authors acknowledged the limitation that the proposed HTMP framework is only applicable to GNNs following the message-passing mechanism. One additional limitation is that the paper is mostly empirical and does not give theoretical underpinnings.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}