Node-wise Filtering in Graph Neural Networks: A Mixture of Experts Approach

## Abstract

Abstract Graph Neural Networks (GNNs) have proven to be highly effective for node classification tasks across diverse graph structural patterns. Traditionally, GNNs employ a uniform global filter—typically a low-pass filter for homophilic graphs and a high-pass filter for heterophilic graphs. However, real-world graphs often exhibit a complex mix of homophilic and heterophilic patterns, rendering a single global filter approach suboptimal. In this work, we theoretically demonstrate that a global filter optimized for one pattern can adversely affect performance on nodes with differing patterns. To address this, we introduce a novel GNN framework Node-MoE that utilizes a mixture of experts to adaptively select the appropriate filters for different nodes. Extensive experiments demonstrate the effectiveness of Node-MoE on both homophilic and heterophilic graphs.

### 1 Introduction

Graph Neural Networks (GNNs) [, ] have emerged as powerful tools in representation learning for graph structure data, and have achieved remarkable success on various graph learning tasks [, ], especially the node classification task. GNNs usually can be designed and viewed from two domains, i.e., spatial domain and spectral domain. In the spatial domain, GNNs [, , ] typically follow the message passing mechanism [], which propagate messages between neighboring nodes. In the spectral domain, GNNs [, ] apply different filters on the graph signals in the spectral domain of the graph Laplacian matrix.

Most GNNs have shown great effectiveness in the node classification task of homophilic graphs [, , , ], where connected nodes tend to share the same labels. These GNNs usually leverage the low-pass filters, where the smoothed signals are preserved. However, the heterophilic graphs exhibit the heterophilic patterns, where the connected nodes tend to have different labels. As a result, several GNNs [, , ] designed for heterophilic graphs introduce the high-pass filter to better handle such diversity. To adapt to both homophilic and heterophilic graphs, GNNs with learnable graph convolution [, , , ] can automatically learn different types of filters for different types of graphs. Despite the great success, these GNNs usually apply a uniform global filter across all nodes.

Figure: Figure 1: A toy example to illustrate the effect of global and node-wise filters. The node color represents features, and the number indicates the labels.
Refer to caption: x1.png

However, real-world graphs often display a complex interplay of homophilic and heterophilic patterns [, ], challenging this one-size-fits-all filtering approach. Specifically, while some nodes tend to connect with others that share similar labels, reflecting homophilic patterns, others are more inclined to form connections with nodes that have differing labels, indicative of heterophilic patterns.
Applying a uniform type of filter, tailored for just one of these patterns, across all nodes may hurt the performance of other patterns. To illustrate this, we provide an example as shown in Figure 1(a), where different colors represent distinct node features, and numbers indicate node labels. The nodes are marked as either solid or dotted circles to denote homophilic and heterophilic patterns, respectively. Applying a global low-pass filter, such as the adjacency matrix ${\mathbf{A}}$, uniformly across all nodes results in a scenario where nodes on the left possess the same feature, while those on the right possess another. Therefore, all the left nodes or the right nodes will have the same prediction. However, nodes on the left or right don’t share the same label.
Consequently, this global filtering approach leads to misclassification of nodes.

This toy example clearly illustrates the limitations of a one-size-fits-all filtering strategy and motivates the need for a more tailored approach. To address this, we propose applying different filters to nodes based on their specific structural patterns. Figure 1(b) provides an example that we apply a low-pass filter, such as ${\mathbf{A}}$, to homophilic nodes, and a high-pass filter, such as $-{\mathbf{A}}$, to heterophilic nodes. From the results, nodes in the same class would have the same features. Therefore, this node-wise filtering approach allows for the perfect classification of all nodes in this example.

Present work. In this work, we observe that nodes in many real-world graphs not only exhibit diverse structural patterns but also that these patterns vary significantly among different communities within the same graph.
Utilizing the CSBM model to generate graphs with mixed structural patterns, we theoretically demonstrate that a global filter optimized for one pattern may incur significant losses for nodes with other patterns, while node-wise filtering can achieve linear separability for all nodes under mild conditions. Building on these insights, we propose a Node-wise filtering method - Node-MoE, which leverages a Mixture of Experts framework to adaptively select appropriate filters for different nodes. Extensive experiments validate the effectiveness of the proposed Node-MoE on both homophilic and heterophilic graphs, illustrating significant performance improvement.

### 2 Preliminary

In this section, we explore the structural patterns present in various graph datasets, which usually exhibit mixed homophilic and heterophilic patterns. Then, we theoretically demonstrate that a global filter often fails in graphs characterized by such mixed structural patterns. In contrast, node-wise filtering can achieve linear separability under mild conditions. Before we start, we first define the notations used in this paper and background knowledge.

Notations. We use bold upper-case letters such as $\mathbf{X}$ to denote matrices. $\mathbf{X}_{i}$ denotes its $i$-th row and ${\mathbf{X}}_{ij}$ indicates the $i$-th row and $j$-th column element. We use bold lower-case letters such as $\mathbf{x}$ to denote vectors. Let $\mathcal{G}=(\mathcal{V},\mathcal{E})$ be a graph, where $\mathcal{V}$ is the node set, $\mathcal{E}$ is the edge set, and $|\mathcal{V}|=n$.
$\mathcal{N}_{i}$ denotes the neighborhood node set for node $v_{i}$.
The graph can be represented by an adjacency matrix $\mathbf{A}\in\mathbb{R}^{n\times n}$, where $\mathbf{A}_{ij}>0$ indices that there exists an edge between nodes $v_{i}$ and $v_{j}$ in $\mathcal{G}$, or otherwise $\mathbf{A}_{ij}=0$. For a node $v_{i}$, we use $\mathcal{N}(v_{i})=\{v_{j}:{\mathbf{A}}_{ij}>0\}$ to denote its neighbors. Let ${\mathbf{D}}=diag(d_{1},d_{2},\dots,d_{n})$ be the degree matrix, where $d_{i}=\sum_{j}\mathbf{A}_{ij}$ is the degree of node $v_{i}$. Furthermore, suppose that each node is associated with a $d$-dimensional feature $\mathbf{x}$ and we use $\mathbf{X}=[\mathbf{x}_{1},\dots,\mathbf{x}_{n}]^{\top}\in\mathrm{R}^{n\times d}$ to denote the feature matrix. Besides, the label matrix is ${\mathbf{Y}}\in\mathrm{R}^{n\times c}$, where $c$ is the number of classes. We use $y_{u}$ to denote the label of node $u$.

Graph Laplacian. The graph Laplacian matrix is defined as $\mathbf{L}=\mathbf{D}-\mathbf{A}$. We define the normalized adjacency matrix as ${\tilde{{\mathbf{A}}}}={\mathbf{D}}^{-\frac{1}{2}}{\mathbf{A}}{\mathbf{D}}^{-
\frac{1}{2}}$ and the normalized Laplacian matrix as ${\tilde{{\mathbf{L}}}}={\mathbf{I}}-{\tilde{{\mathbf{A}}}}$. Its eigendecomposition can be represented by ${\tilde{{\mathbf{L}}}}={\mathbf{U}}{\mathbf{\Lambda}}{\mathbf{U}}^{\top}$, where the ${\mathbf{U}}\in\mathbb{R}^{n\times n}$ is the eigenvector matrix and ${\mathbf{\Lambda}}=diag(\lambda_{1},\lambda_{2},\dots,\lambda_{n})$ is the eigenvalue matrix. Specifically, $0\leq\lambda_{1}\leq\lambda_{2}\leq\dots\leq\lambda_{n}<2$. The filtered signals can be represented by $\hat{{\mathbf{X}}}={\mathbf{U}}f({\mathbf{\Lambda}}){\mathbf{U}}^{\top}{
\mathbf{X}}$, where $f$ is the filter function. As a result, the graph convolution ${\tilde{{\mathbf{A}}}}{\mathbf{X}}$ can be viewed as a low-pass filter, with the filter $f(\lambda_{i})=1-\lambda_{i}$. Similarly, the graph convolution $-{\tilde{{\mathbf{A}}}}{\mathbf{X}}$ is a high-pass filter with filter $f(\lambda_{i})=\lambda_{i}-1$.

Homophily metrics. Homophily metrics in graphs measure the tendency of edges to connect nodes with similar labels []. There are several commonly used homophily metrics, such as edge homophily [], node homophily [], and class homophily []. In this paper, we adopt the node homophily $H(\mathcal{G})=\frac{1}{|\mathcal{V}|}\sum_{v_{i}\in\mathcal{V}}h(v_{i})$, where $h(v_{i})=\frac{\left|\left\{u\in\mathcal{N}\left(v_{i}\right):y_{u}=y_{v}
\right\}\right|}{d_{i}}$ measures the label similarity between node $v_{i}$ with its neighbors. A node with higher $h(v)$ exhibits a homophilic pattern while a low $h(v)$ indicates a heterophilic pattern.

#### 2.1 Structural Patterns in Existing Graphs

In this subsection, we examine the structural patterns present in existing graph datasets. Specifically, we select two widely used homophilic datasets, i.e., Cora and CiteSeer [], and two heterophilic datasets, i.e., chameleon and squirrel [].

Figure: Figure 2: Node homophily density.
Refer to caption: x2.png

We first calculate the homophily distribution for all nodes in the graph. As shown in Figure 3, while the majority of nodes in homophilic graphs predominantly exhibit homophilic patterns, and those in heterophilic graphs display heterophilic patterns, exceptions are evident. Notably, some nodes in homophilic graphs show heterophilic tendencies, and conversely, some nodes in heterophilic graphs demonstrate homophilic patterns. Consequently, all these graphs exhibit a mixture of homophilic and heterophilic patterns, which aligns with the findings in the previous works [, ].

We further analyze the position of nodes with different structural patterns within the graphs. To do this, we divide each graph into several subgraphs using community detection algorithms []. We focus on the largest 10 communities and calculate the homophily level for each subgraph. The results, as shown in Figure 3, reveal significant variations in homophily across different communities. For instance, in the Cora dataset, homophily levels in some communities approach 1, indicating strong homophily, while in some communities it drops below 0.5. Similarly, in the chameleon dataset, the lowest homophily levels are near 0, with the highest reaching above 0.6. These findings highlight the considerable diversity in node interaction patterns, even within the same graph, underscoring the complexity of graph structures in real-world datasets. The variability in homophily levels clearly illustrates that nodes in various parts of the graph may require distinct processing approaches. Therefore, applying the same global filter to all nodes may lead to suboptimal performance.

#### 2.2 Analysis based on CSBM model

To further illustrate why applying a global filter may result in suboptimal performance, we utilize the Contextual Stochastic Block Model (CSBM) [], which has been widely applied to graph analysis [, ], such as analyzing the behavior of GNNs [, , ]. The CSBM is a generative model, which is often used to generate graph structures and node features. Typically, CSBMs are based on the assumption that graphs are generated following a uniform patter, such as nodes with the same label are connected with probability $p$ while nodes with different labels are connected with probability $q$ []. However, the real-world complexity of graphs features a mixture of homophilic and heterophilic patterns, as illustrated in section 2.1. We adapt the CSBM by mixing two CSBMs to generate one graph, mirroring the approach [].

###### Definition 1 .

$CSBM(n,\boldsymbol{\mu},\boldsymbol{\nu},(p_{0},q_{0}),(p_{1},q_{1}),P)$. The generated nodes consist of two classes, $C_{0}=\left\{i\in[n]:y_{i}=0\right\}$ and $C_{1}=\left\{j\in[n]:y_{j}=1\right\}$. For each node, consider ${\mathbf{X}}\in\mathbb{R}^{n\times d}$ to be the feature matrix such that each row ${\mathbf{X}}_{i}$ is an independent $d$-dimensional Gaussian random vectors with ${\mathbf{X}}_{i}\sim N\left(\boldsymbol{\mu},\frac{1}{d}{\mathbf{I}}\right)$ if $i\in C_{0}$ and ${\mathbf{X}}_{j}\sim N\left(\boldsymbol{\nu},\frac{1}{d}{\mathbf{I}}\right)$ if $j\in C_{1}$. Here $\boldsymbol{\mu},\boldsymbol{\nu}$ are the fixed class mean vector with $\|\boldsymbol{\mu}\|_{2},\|\boldsymbol{\nu}\|_{2}\leq 1$ and ${\mathbf{I}}$ is the identity matrix. Suppose there are two patterns of nodes in the adjacency matrix ${\mathbf{A}}=(a_{ij})$, i.e., the homophilic pattern: $H_{0}=\{i\in[n]:a_{ij}=\operatorname{Ber}(p_{0})$ if $y_{i}=y_{j}$ and
$a_{ij}=\operatorname{Ber}(q_{0})$ if $y_{i}\neq y_{j},p_{0}>q_{0}\}$ and the heterophilic pattern: $H_{1}=\{i\in[n]:a_{ij}=\operatorname{Ber}(p_{1})$ if $y_{i}=y_{j}$ and
$a_{ij}=\operatorname{Ber}(q_{1})$ if $y_{i}\neq y_{j},p_{1}<q_{1}\}$. $P$ denotes the probability that a node is in the homophilic pattern. We also assume the nodes follow the same degree distribution $p_{0}+q_{0}=p_{1}+q_{1}$.

For simplification, we consider a linear model with parameters ${\mathbf{w}}\in\mathbb{R}^{d}$ and $b\in\mathbb{R}$, following the approach []. The predicted label for nodes is given by $\hat{{\mathbf{y}}}=\sigma(\tilde{{\mathbf{X}}}{\mathbf{w}}+b\mathbf{1})$, where $\sigma(x)=(1+e^{-x})^{-1}$ is the sigmoid function, and $\tilde{{\mathbf{X}}}$ represents the features after filtering. The binary cross-entropy loss over nodes $\mathcal{V}$ is formulated as $L(\mathcal{V},{\mathbf{w}},b)=-\frac{1}{|\mathcal{V}|}\sum_{i\in\mathcal{V}}y_
{i}\log(\hat{y_{i}})+(1-y_{i})\log(1-\hat{y_{i}})$.

###### Theorem 1 .

Suppose $n$ is relatively large, the graph is not too sparse with $p_{i},q_{i}=\omega(\log^{2}(n)/n)$ and the feature center distance is not too small with $\|\boldsymbol{\mu}-\boldsymbol{\nu}\|=\omega(\frac{\log n}{\sqrt{\operatorname
{dn}(p_{0}+q_{0})}})$ and $\|{\mathbf{w}}\|\leq R$. For the graph $G(\mathcal{V},\mathcal{E},{\mathbf{X}})\sim CSBM(n,\boldsymbol{\mu},
\boldsymbol{\nu},(p_{0},q_{0}),(p_{1},q_{1}),P)$, we have the following:

1. If the low-pass global filter, i.e., $1-\lambda$, is applied to the whole graph $G$, we can find a optimal ${\mathbf{w}}^{*},b^{*}$ that achieve near linear separability for the homophilic node set $H_{0}$. However, the loss for the heterophilic node set $H_{1}$ can be relatively large with:

$$ $\displaystyle L(H_{1},{\mathbf{w}}^{*},b^{*})\geq\frac{R(q_{1}-p_{1})}{2(q_{1} +p_{1})}\|\boldsymbol{\mu}-\boldsymbol{\nu}\|\left(1+o_{d}(1)\right).$ $$

2. If different filters are applied to homophilic and heterophilic sets separately, we can find an optimal ${\mathbf{w}}^{*},b^{*}$ that all the nodes are linear separable with the probability:

$$ $\displaystyle\mathbb{P}\left(\left(\tilde{{\mathbf{X}}}_{i}\right)_{i\in \mathcal{V}}\text{ is linearly separable }\right)=1-o_{d}(1).$ $$

The proof of these results is detailed in Appendix A. Theorem 1 reveals critical insights into the filtering strategies for graphs with mixed homophilic and heterophilic patterns, as generated by the CSBM model. The first part of the theorem illustrates that applying a global low-pass filter can create an optimal classifier for homophilic nodes, achieving near-linear separability. However, this classifier may result in a substantial loss for heterophilic nodes, highlighting the limitations of a uniform filtering strategy. Conversely, the second part of the theorem demonstrates that by applying different filters to different patterns of nodes separately, it is possible to achieve linear separability across all nodes. These findings strongly motivate the exploration of a node-wise filtering method, which can automatically apply different filters to distinct nodes based on their specific patterns, to improve the overall performance.

###### Definition 1 .

###### Theorem 1 .

### 3 The Proposed Method

The investigations presented in Section 2 underscore the complex nature of real-world datasets, revealing a mixture of homophilic and heterophilic patterns within them. Additionally, these patterns are not uniformly distributed throughout the graph; rather, the level of homophily varies significantly across different communities. Our theoretical analysis further demonstrates that global filtering, as commonly employed in numerous GNNs, may not effectively capture such complex patterns, often leading to suboptimal performance. In contrast, node-wise filtering, which applies distinct filters to individual nodes based on their specific patterns, shows great promise in handling the intricacies of such complex graphs.

However, implementing the node-wise filtering approach presents two significant challenges. First, how can we incorporate various filters into a single unified framework? It requires a flexible architecture that can seamlessly accommodate multiple filtering mechanisms without compromising the efficiency and scalability of the model. Second, without ground truth on node patterns, how can we select the appropriate filters for different nodes? In the following subsections, we aim to address these challenges.

Figure: Figure 4: The overall framework of the proposed Node-MoE. For each node, the gating model will assign different weights for each expert based on the node’s feature and context. The experts can be any GNNs with different filters.
Refer to caption: x4.png

#### 3.1 Node-MoE : Node-wise Filtering via Mixture of Experts

Mixture of Experts (MoE) [, ], which follows the divide-and-conquer principle to divide the complex problem space into several subspaces so that each one can be easily addressed by specialized experts, have been successfully adopted across various domains [, , ]. For node classification tasks in graphs exhibiting a mixture of structural patterns, the diversity of node interactions necessitates applying distinct filters to different nodes as we discussed in Sections 2. This necessity aligns well with the MoE methodology, which processes different samples with specific experts. Building on this principle, we introduce a flexible and efficient Node-wise Filtering via Mixture of Experts (Node-MoE) framework, designed to dynamically apply appropriate filters to nodes based on their structural characteristics.

The overall Node-MoE framework is illustrated in Figure 4, which consists of two primary components: the gating model and the multiple expert models. With the graph data as input, the gating model $g(\cdot)$ computes the weight assigned to each expert for every node, reflecting the relevance of each expert’s contribution to that specific node. Each expert model, implemented as any GNN with different filters, generates node representations independently. The final node classification is determined by a weighted sum of these representations, where the weights are those assigned by the gating model. The prediction for node $i$ can be represented by:

$$ $\displaystyle\hat{y}_{i}=h\left(\sum_{o=1}^{m}g({\mathbf{A}},{\mathbf{X}})_{i, o}E_{o}(\mathbf{A},\mathbf{X})_{i}\right),$ (1) $$

where $m$ is the number of experts, $E_{o}$ denotes the $o$-th expert, $g(\mathbf{A},\mathbf{X})_{i,o}$ represents the weight assigned to the $o$-th expert for node $i$ by the gating model, and $h$ is a classifier, which could be a model like a neural network or a simple activation function like Softmax. In the following, we will delve into the specific designs of the gating model and the expert models within the proposed Node-MoE framework.

#### 3.2 Gating Model

The gating model is a pivotal component of the Node-MoE framework, aimed at selecting the most appropriate experts for each node. Its primary function is to dynamically assign higher weights to experts whose filtering characteristics best match the node’s patterns. For instance, an expert utilizing a high-pass filter may receive a higher weight for a node that exhibits heterophilic patterns. However, a significant challenge arises as there is no explicit ground truth indicating which pattern each node belongs to. In traditional MoE models, the gating model often utilizes a straightforward feed-forward network that processes the features of the sample as input [, , , ]. Nevertheless, the nodes with different patterns may share similar node features, making this method ineffective.

To address this challenge, we estimate node patterns by incorporating the contextual features surrounding each node. If a node’s features significantly differ from those of its neighboring nodes, it is likely that this node exhibits a heterophilic pattern. Specifically, the input to our gating model includes a composite vector $[{\mathbf{X}},|{\mathbf{A}}{\mathbf{X}}-{\mathbf{X}}|,|{\mathbf{A}}^{2}{
\mathbf{X}}-{\mathbf{X}}|]$. This vector combines the node’s original features with the absolute differences between its features and those of its neighbors over one and two hops, respectively, to indicate the node’s structural patterns. Moreover, as discussed in Section 2.1, different structural patterns are not uniformly distributed across the graph, and distinct communities may exhibit varying structural characteristics. To capitalize on this phenomenon, we employ GNNs with low-pass filters, such as GIN [], for the gating model. These networks are chosen due to their strong community detection capabilities [, ], ensuring that neighboring nodes are likely to receive similar expert selections. Experimental results in Section 4.3 clearly demonstrate the proposed gating can efficiently assign different nodes to their suitable filters.

#### 3.3 Expert Models

The mixed structural patterns observed in real-world graphs necessitate that the expert models in our Node-MoE framework possess diverse capabilities. To achieve this, we consider multiple existing GNNs equipped with different filters. Traditional GNNs often utilize fixed filters, which may not adequately capture the complexity of diverse structural patterns. To address this limitation, we opt for GNNs with learnable graph convolutions [, , , ], which are capable of adapting their filters to better fit the graph structural patterns. However, the same experts would make the gating model hard to learn the right features [] and may result in all experts’ filters being optimized in the same direction. To encourage diversity and ensure that each expert is adept at handling specific structural patterns, we adopt a differentiated initialization strategy for the filters in the experts. Instead of using a fixed filter initialization, we initialize different experts with distinct types of filters, such as low-pass, constant, and high-pass filters.

Filter Smoothing Loss. While integrating multiple experts with diverse filters significantly enhances the expressive capacity of our Node-MoE framework, this complexity can also make the model more challenging to fit. For example, training multiple filters simultaneously may lead to oscillations in the spectral domain for each filter as shown in Appendix B. This not only complicates fitting the model to the data but also impacts its explainability. The specific role and function of each oscillating filter become difficult to discern, making it harder to understand and interpret the model’s behavior. To mitigate these issues, we introduce a filter smoothing loss designed to ensure that the learned filters exhibit smooth behavior in the spectral domain. This loss is defined as follows:

$$ $\displaystyle L_{s}^{o}=\sum_{i=1}^{K}|f_{o}(x_{i})-f_{o}(x_{i-1})|^{2},$ (2) $$

where $f_{o}(\cdot)$ is the learnable filter function of the $o$-th expert, $x_{0}\leq x_{1}\leq\cdots\leq x_{K}$ are $K+1$ values spanning the spectral domain. The overall training loss is then given by $L=L_{task}+\gamma\sum_{o=1}^{m}L_{s}^{o}$, where the $L_{task}$ is the node classification loss and $\gamma$ is a hyperparameter that adjusts the influence of the filter smoothing loss.

#### 3.4 Top-K gating

The soft gating that integrates all experts in the Node-MoE framework significantly enhances its modeling capabilities, but it also increases computational complexity since each expert must process all samples. To improve computational efficiency while maintaining performance, we introduce a variant of Node-MoE by leveraging the Top-K gating mechanism []. In this variant, the Node-MoE with Top-K gating selectively activates only the top k experts with the highest relevance for each node. Specifically, the gating function for a node $v_{i}$ is defined as $g(v_{i})=\operatorname{Softmax}\left(\operatorname{TopK}\left(g\left({\mathbf{
A}},{\mathbf{X}}\right)_{i},k\right)\right)$. To prevent the gating model from consistently favoring a limited number of experts, we incorporate a load-balancing loss as suggested by []. In the experiments detailed in Section 4.4, we set $k=1$ to ensure that the computational complexity of our Node-MoE is comparable to that of an individual expert model.

### 4 Experiment

In this section, we conduct comprehensive experiments to validate the effectiveness of the proposed Node-MoE. Specifically, we aim to address the following research questions:

- •
RQ1: How does Node-MoE perform compared with the state-of-the-art baselines on both homophilic and heterophilic graphs?
- •
RQ2: Do the experts within Node-MoE learn diverse structural patterns and does the gating model accurately assign each node to its most suitable experts?
- •
RQ3: How do different factors affect the performance of Node-MoE?

#### 4.1 Experimental settings.

Datasets. To evaluate the efficacy of our proposed Node-MoE, we conduct experiments across seven widely used datasets. These include four homophilic datasets: Cora, CiteSeer, Pubmed [], and ogbn-arxiv []; along with three heterophilic datasets: Chameleon, Squirrel, and Actor []. For Cora, CiteSeer, and Pubmed, we generate ten random splits, distributing nodes into 60% training, 20% validation, and 20% testing partitions. For the heterophilic datasets, we utilize the ten fixed splits as specified in []. The ogbn-arxiv dataset is evaluated using its standard split []. We run the experiments 3 times for each split
and report the average performance and standard deviation. More details about these datasets are shown in Appendix C.1.

Baselines. We compare our method with a diverse set of baselines, which can be divided into five categories: (1) Non-GNN methods like MLP and Label Propagation (LP) []; (2) Homophilic GNNs utilizing fixed low-pass filters such as GCN [], GAT [], APPNP [], and GCNII []; (3) Heterophilic GNNs including WRGCN [], GloGNN [] and LinkX []; (4) GNNs with learnable filters like GPRGNN [] and ChebNetII []; (5) MoE-based GNNs such as GMoE [].

Node-MoE settings. The proposed Node-MoE framework is highly flexible, allowing for a wide range of choices in both gating and expert models. In this work, we employ the GIN [] as the gating model due to its exceptional expressive power and ability to leverage community properties as discussed in Section 2. As for the expert models, we utilize ChebNetII [], known for its efficiency in learning filters. Specifically, we experiment with configurations of 2, 3, and 5 ChebNetII experts, each initialized with different filters. More details and parameter settings are in Appendix C.2.

#### 4.2 Performance Comparison on Benchmark Datasets

In this section, we evaluate the efficacy of the proposed Node-MoE across both homophilic and heterophilic datasets. The results of node classification experiments are detailed in Table 1. From the results, we can have the following observations:

- •
The proposed Node-MoE demonstrates robust performance across both homophilic and heterophilic datasets, achieving the best average rank among all baselines. This indicates its effectiveness in handling diverse graph structures.
- •
The GNNs and methods like LP that use fixed low-pass filters generally do well on homophilic datasets but tend to underperform on heterophilic datasets. Conversely, specialized models like GloGNN and LinkX, designed for heterophilic graphs, do not perform as well on homophilic datasets such as the ogbn-arxiv dataset.
- •
The GNNs equipped with learnable filters generally perform well on both types of datasets, as they can adapt their filters to the dataset’s structural patterns. However, their performance is still not optimal. The proposed Node-MoE, which utilizes multiple ChebNetII as experts, significantly outperforms a single ChebNetII, especially on heterophilic datasets. This result validates the effectiveness of our node-wise filtering approach.
- •
We also compare the proposed Node-MoE with GMoE, which adapts the receptive field for each node but still applies traditional graph convolution with low-pass filters. We can find Node-MoE consistently outperforms GMoE across all datasets.

**Table 1: Node classification accuracy (%) on benchmark datasets. OOM means out-of-memory. The bold and underline markers denotes the best and second-best performance respectively.**
|  | Homophilic Datasets | Heterophilc Datasets | Avg.<br>Rank |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Cora | CiteSeer | PubMed | ogbn-arxiv | Chameleon | Squirrel | Actor | / |
| MLP | 76.49 ± 1.13 | 73.15 ± 1.36 | 86.14 ± 0.64 | 55.68 ± 0.11 | 48.11 ± 2.23 | 31.68 ± 1.90 | 36.17 ± 1.09 | 10.71 |
| LP | 86.05 ± 1.35 | 69.39 ± 2.01 | 83.38 ± 0.64 | 68.14 ± 0.00 | 44.10 ± 4.10 | 31.92 ± 0.82 | 24.62 ± 0.51 | 12 |
| GCN | 88.60 ± 1.19 | 76.88 ± 1.78 | 88.48 ± 0.46 | 71.91 ± 0.15 | 67.96 ± 1.82 | 54.47 ± 1.17 | 30.31 ± 0.98 | 6 |
| GAT | 88.68 ± 1.13 | 76.70 ± 1.81 | 86.52 ± 0.56 | 71.92 ± 0.17 | 65.29 ± 2.54 | 49.46 ± 1.69 | 28.95 ± 1.08 | 7.29 |
| APPNP | 88.49 ± 1.28 | 77.42 ± 1.47 | 87.56 ± 0.52 | 71.61 ± 0.30 | 54.32 ± 2.61 | 36.41 ± 1.94 | 36.05 ± 0.91 | 7.43 |
| GCNII | 88.12± 1.05 | 77.30 ±1.58 | 90.17 ± 0.57 | 72.74 ± 0.16 | 55.54 ± 2.02 | 56.63 ± 1.17 | 34.36 ± 0.77 | 5.43 |
| GloGNN | 88.78 ± 1.21 | 75.21 ± 1.17 | 87.93 ± 0.63 | 65.68 ± 0.04 | 69.78 ± 2.42 | 57.54 ± 1.39 | 37.35 ± 1.30 | 5.43 |
| LinkX | 82.89 ± 1.27 | 70.05 ± 1.88 | 84.81 ± 0.65 | 66.54 ± 0.52 | 68.42 ± 1.38 | 61.81 ± 1.80 | 36.10 ± 1.55 | 8.14 |
| WRGCN | 88.06 ± 1.50 | 76.28 ± 1.98 | 86.39 ± 0.55 | OOM | 65.24 ± 0.87 | 48.85 ± 0.78 | 34.89 ± 1.11 | 9.57 |
| GPR-GNN | 88.54 ± 0.67 | 76.44 ± 1.89 | 88.46 ± 0.31 | 71.78 ± 0.18 | 62.85 ± 2.90 | 54.35 ± 0.87 | 33.12 ± 0.57 | 7.29 |
| ChebNetII | 88.71 ± 0.93 | 76.93 ± 1.57 | 88.93 ± 0.29 | 72.32 ± 0.23 | 71.14 ± 2.13 | 57.12 ± 1.13 | 35.67 ± 1.19 | 3.86 |
| GMoE | 87.27 ± 1.74 | 76.56 ± 1.57 | 88.14 ± 0.56 | 71.74 ± 0.29 | 71.88 ± 1.60 | 51.97 ± 3.16 | 35.75 ± 1.31 | 6.57 |
| Node-MoE | 89.38 ± 1.26 | 77.78 ± 1.36 | 89.58 ± 0.60 | 73.19 ± 0.22 | 73.64 ± 1.80 | 62.31 ± 1.98 | 36.28 ± 1.01 | 1.29 |

#### 4.3 Analysis of Node-MoE

In this section, we delve into an in-depth analysis of the behaviors of Node-MoE to demonstrate its rationality and effectiveness. We aim to uncover several key aspects of how Node-MoE operates and performs: What specific types of filters does Node-MoE learn? Are nodes appropriately assigned to these diverse filters by the gating model?
We conduct experiments on both CiteSeer and Chameleon datasets using configurations with 2 experts. The results for the Chameleon dataset are presented below. For more results and analysis, please refer to Appendix D.

Figure: Figure 5: Learned 2 filters by Node-MoE on Chameleon.
Refer to caption: extracted/5644463/Figures/chameleon_MoE_filter_2_1.png

Figure 6 showcases the two filters learned by Node-MoE on the Chameleon dataset, where filter 0 functions as a low-pass filter and filter 1 as a high-pass filter. To analyze the behavior of the gating model in Node-MoE, we split nodes into different groups based on their homophily levels. Figure 6 displays the weights assigned by the gating model to these two experts. The results reveal that nodes with lower homophily levels predominantly receive higher weights for the high-pass filter (filter 1), and as the homophily level increases, the weight for this filter correspondingly decreases. This pattern confirms our design that nodes with varying structural patterns require different filters, demonstrating the effectiveness of the proposed gating model.

#### 4.4 Ablation studies

In this section, we conduct ablation studies to further investigate the effectiveness of two key components within the Node-MoE framework: the gating model and the filter smoothing loss. For the gating model, we explore two variants: a traditional MLP-based gating mechanism that utilizes the input features ${\mathbf{X}}$, and the Top-k gating approach as detailed in Section 3.4. Specifically, we choose $k=1$ to ensure the proposed Node-MoE has similar efficiency with the single expert.
Figure 8 presents the results on CiteSeer, ogbn-arxiv, Chameleon, and Squirrel datasets. We observe two findings: (1) Traditional gating does not perform as well as the proposed gating in Node-MoE and only achieves results comparable to an individual ChebNetII expert. (2) The Top-1 gating, which selects only one expert, can achieve similar results to those of the soft gating Node-MoE that utilizes all experts. This indicates that the proposed Node-MoE can effectively enhance performance while maintaining a complexity level comparable to that of an individual expert model.

Figure: Figure 7: The performance comparison of different gating variants.
Refer to caption: extracted/5644463/Figures/ablation_gating.png

We also investigate the impact of the weight parameter, $\gamma$, of the filter smoothing loss on the overall performance. Specifically, we conduct experiments on the Citeseer and Squirrel datasets and the $\gamma$ is chosen in $[0,0.1,0.5,1,5]$. As shown in Figure 8, incorporating the filter smoothing loss generally enhances performance, especially for the Citeseer dataset. The reason is that the filter smoothing loss can mitigate filter oscillation, which may lead to the model being hard to learn. For more detailed insights into the effects of the filter smoothing loss, please refer to Appendix B.

### 5 Related Works

Graph Neural Networks (GNNs) [, ] have achieved remarkable success in graph representation learning across a wide range of tasks []. The design of GNN architectures is majorly motivated in the spatial domain [, , , ] and spectral domain [, , , ]. The GNNs in the spatial domain usually follow the message-passing mechanism [], which can be regarded as low-pass graph filters [, ]. As a result, these GNNs are usually suitable for homophilic graphs. To address heterophilic graphs, specialized models like GloGNN [], LinkX [], MixHop [], and ACM-GNN [] have been developed. Additionally, models such as Bernnet [], GPRGNN [], and ChebNetII [] feature learnable filters that adapt to various graph types. Recent studies have highlighted that real-world graphs often exhibit a mixture of structural patterns [, ], with [] specifically exploring GNN generalization under mixed patterns. Traditional GNNs typically apply the same global filter across all nodes, which can be suboptimal for such mixed scenarios. In response, our proposed Node-MoE introduces a node-wise filtering approach, applying distinct filters to nodes based on their individual patterns, enhancing adaptability and performance.

Mixture of Experts (MoE) [, ] architecture has been widely used in NLP [, ] and Computer Vision [] to improve efficiency of large models. In graph domain, GraphMETRO [] leverage
MoE to address the graph distribution shift issue. GMoE [] utilizes MoE to adaptive select propagation hops for different nodes. Despite these advancements, these method still face challenges in handling complex graph patterns with the same efficacy as our proposed Node-MoE.

### 6 Conclusion

In this paper, we explored the complex structural patterns inherent in real-world graph datasets, which typically showcase a mixture of homophilic and heterophilic patterns. Notably, these patterns exhibit significant variability across different communities within the same dataset, highlighting the intricate and diverse nature of graph structures. Our theoretical analysis reveals that the conventional single global filter, commonly used in many GNNs, is often inadequate for capturing such complex structural patterns. To address this limitation, we proposed the Node-wise filtering method, Node-MoE, a flexible and effective solution that adaptively selects appropriate filters for different nodes. Extensive experiments demonstrate the proposed Node-MoE demonstrated robust performance on both homophilic and heterophilic datasets. Further, our behavioral analysis and ablation studies validate the design and effectiveness of the proposed Node-MoE.

## Appendix

### Appendix A Proof of Theorem 1

In this section, we present the proof of Theorem 1. This theorem analyzes the separability when different filters are applied to graphs generated by a mixed CSBM model in Defination 1-$CSBM(n,\boldsymbol{\mu},\boldsymbol{\nu},(p_{0},q_{0}),(p_{1},q_{1}),P)$ using a linear classifier.

Notably, the following proof is derived based on [], which analyzes the linear separability of a single graph convolution under a single CSBM model with only one pattern - $CSBM(n,\boldsymbol{\mu},\boldsymbol{\nu},(p,q))$. We extend the analysis to graphs with mixed CSBM models. Besides, we analyze the scenarios in which different filters are applied to the same graph.

We follow the assumption 1 and 2 in []: The graph size n should be relatively large with $\omega(d\log d)\leq n\leq O(\operatorname{poly}(d))$, and the graph is not too sparse with $p_{0},q_{0},p_{1},q_{1}=\omega\left(\log^{2}(n)/n\right)$.

#### A.1 Proof of part 1 of Theorem 1

###### Proof.

For the low-pass filter, consider the filtered feature $\tilde{{\mathbf{X}}}={\mathbf{D}}^{-1}{\mathbf{A}}{\mathbf{X}}$. Then, the filtered feature of node $i$ still follows the normal distribution, the mean can be represented by:

$m(i)=E(\tilde{{\mathbf{X}}}_{i})=\left\{\begin{aligned} \frac{p_{0}\boldsymbol
{\mu}+q_{0}\boldsymbol{\nu}}{p_{0}+q_{0}}(1+o(1))\quad\text{for}\,i\in C_{0}\,
\text{and}\,i\in H_{0}\\
\frac{q_{0}\boldsymbol{\mu}+p_{0}\boldsymbol{\nu}}{p_{0}+q_{0}}(1+o(1))\quad
\text{for}\,i\in C_{1}\,\text{and}\,i\in H_{0}\\
\frac{p_{1}\boldsymbol{\mu}+q_{1}\boldsymbol{\nu}}{p_{1}+q_{1}}(1+o(1))\quad
\text{for}\,i\in C_{0}\,\text{and}\,i\in H_{1}\\
\frac{q_{1}\boldsymbol{\mu}+p_{1}\boldsymbol{\nu}}{p_{1}+q_{1}}(1+o(1))\quad
\text{for}\,i\in C_{1}\,\text{and}\,i\in H_{1}\\
\end{aligned}\right.,$

where $C_{0}$ and $C_{1}$ represent the class 0 and class 1, respectively; $H_{0}$ and $H_{1}$ are the homophilic and heterophilic node sets, respectively. The covariance matrix can be represented by:
$Cov(\tilde{{\mathbf{X}}}_{i})=\frac{1}{d{\mathbf{D}}_{ii}}{\mathbf{I}}.$. Lemma 2 in [] demostrate that for any unit vector ${\mathbf{w}}$, we have:
$\left|\left(\tilde{{\mathbf{X}}}_{i}-m(i)\right)\cdot\mathbf{w}\right|=O\left(
\sqrt{\frac{\log n}{dn(p_{0}+q_{0})}}\right).$

If we only consider the nodes with homophilic patterns, i.e., $i\in H_{0}$, we can find the optimal linear classifier with ${\mathbf{w}}^{*}=R\frac{\boldsymbol{\nu}-\boldsymbol{\mu}}{\|\boldsymbol{\nu}-
\boldsymbol{\mu}\|}$ and ${\mathbf{b}}^{*}=-\frac{1}{2}\langle\boldsymbol{\nu}+\boldsymbol{\mu},{\mathbf
{w}}^{*}\rangle$. We also have the assumption that the distance between $\boldsymbol{\mu}$ and $\boldsymbol{\nu}$ are relatively large, with $\|\boldsymbol{\nu}-\boldsymbol{\mu}\|=\Omega\left(\frac{\log n}{dn(p_{0}+q_{0}
)}\right)$.

Then, for $i\in C_{0}\,\text{and}\,i\in H_{0}$, we have:

$$ $\displaystyle\langle\tilde{{\mathbf{X}}}_{i},{\mathbf{w}}^{*}\rangle+b^{*}$ $\displaystyle=\frac{\langle p_{0}\boldsymbol{\mu}+q_{0}\boldsymbol{\nu},{ \mathbf{w}}^{*}\rangle}{p_{0}+q_{0}}(1+o(1))+O\left(\|{\mathbf{w}}^{*}\|\sqrt{ \frac{\log n}{dn(p+q)}}\right)-\frac{1}{2}\langle\boldsymbol{\nu}+\boldsymbol{ \mu},{\mathbf{w}}^{*}\rangle$ $\displaystyle=\frac{\langle 2p_{0}\boldsymbol{\mu}+2q_{o}\boldsymbol{\nu}-(p_{ 0}+q_{0})(\boldsymbol{\mu}+\boldsymbol{\nu}),{\mathbf{w}}^{*}\rangle}{p_{0}+q_ {0}}(1+o(1))+o(\|{\mathbf{w}}^{*}\|)$ $\displaystyle=\frac{p_{0}-q_{0}}{2(p_{0}+q_{0})}\langle\boldsymbol{\mu}- \boldsymbol{\nu},{\mathbf{w}}*\rangle(1+o(1))+o(\|{\mathbf{w}}^{*}\|)$ $\displaystyle=-\frac{R(p_{0}-q_{0})}{2(p_{0}+q_{0})}\|\boldsymbol{\mu}- \boldsymbol{\nu}\|(1+o(1))<0$ $$

Similarly, for $i\in C_{1}\,\text{and}\,i\in H_{0}$, we have:

$$ $\displaystyle\langle\tilde{{\mathbf{X}}}_{i},{\mathbf{w}}^{*}\rangle+b^{*}$ $\displaystyle=-\frac{R(q_{0}-p_{0})}{2(p_{0}+q_{0})}\|\boldsymbol{\mu}- \boldsymbol{\nu}\|(1+o(1))>0$ $$

Therefore, the linear classifier with $w^{*}$ and $b^{*}$ can separate class $C_{0}$.

However, if we apply this linear classifier to the heterophilic node set $H_{1}$, we have:

$\langle\tilde{{\mathbf{X}}}_{i},{\mathbf{w}}^{*}\rangle+b^{*}=\left\{\begin{
aligned} -\frac{R(p_{1}-q_{1})}{2(p_{1}+q_{1})}\|\boldsymbol{\mu}-\boldsymbol{
\nu}\|(1+o(1))>0\quad\text{for}\,i\in C_{0}\,\text{and}\,i\in H_{1}\\
-\frac{R(q_{1}-p_{1})}{2(p_{1}+q_{1})}\|\boldsymbol{\mu}-\boldsymbol{\nu}\|(1+
o(1))<0\quad\text{for}\,i\in C_{1}\,\text{and}\,i\in H_{1}\\
\end{aligned}\right.$

Therefore, all nodes in $H_{1}$ are misclassified. The binary cross-entropy over node set $H_{1}$ can be represented by:

$$ $\displaystyle L(H_{1},{\mathbf{w}}^{*},b^{*})$ $\displaystyle=\frac{1}{|H_{1}|}\sum_{i\in H_{1}}-y_{i}\log\left(\sigma\left( \left\langle\tilde{{\mathbf{X}}}_{i},{\mathbf{w}}^{*}\right\rangle+\tilde{b} \right)\right)-\left(1-y_{i}\right)\log\left(1-\sigma\left(\left\langle\tilde{ {\mathbf{X}}}_{i},{\mathbf{w}}^{*}\right\rangle+b^{*}\right)\right)$ $\displaystyle=\frac{1}{|H_{1}|}\sum_{i\in H_{1}}\log\left(1+\exp\left(\left(1- 2y_{i}\right)\left(\left\langle\tilde{X}_{i},\tilde{\mathbf{w}}\right\rangle+b ^{*}\right)\right)\right)$ $\displaystyle=\log\left(1+\exp\left(-\frac{R(p_{1}-q_{1})}{2(p_{1}+q_{1})}\| \boldsymbol{\mu}-\boldsymbol{\nu}\|(1+o(1))\right)\right)$ $$

As for $x=-\frac{R(p_{1}-q_{1})}{2(p_{1}+q_{1})}\|\boldsymbol{\mu}-\boldsymbol{\nu}\|>0$, we have $e^{x}\geq x$. As a result, we have

$$ $\displaystyle L(H_{1},{\mathbf{w}}^{*},b^{*})\geq\frac{R(q_{1}-p_{1})}{2(p_{1} +q_{1})}\|\boldsymbol{\mu}-\boldsymbol{\nu}\|(1+o(1))$ $$

∎

###### Proof.

#### A.2 Proof of part 2 of Theorem 1

###### Proof.

Suppose we apply a high-pass filter to the heterophilic nodes $H_{1}$ and the filtered features are $\tilde{{\mathbf{X}}}=-{\mathbf{D}}^{-1}{\mathbf{A}}{\mathbf{X}}$. For nodes in $H_{1}$,

$m(i)=E(\tilde{{\mathbf{X}}}_{i})=\left\{\begin{aligned} -\frac{p_{1}
\boldsymbol{\mu}+q_{1}\boldsymbol{\nu}}{p_{1}+q_{1}}(1+o(1))\quad\text{for}\,i
\in C_{0}\,\text{and}\,i\in H_{1}\\
-\frac{q_{1}\boldsymbol{\mu}+p_{1}\boldsymbol{\nu}}{p_{1}+q_{1}}(1+o(1))\quad
\text{for}\,i\in C_{1}\,\text{and}\,i\in H_{1}\\
\end{aligned}\right.$

Therefore, if we apply the same linear classifier with ${\mathbf{w}}^{*}$ and $b^{*}$, then we have:

$\langle\tilde{{\mathbf{X}}}_{i},{\mathbf{w}}^{*}\rangle+b^{*}=\left\{\begin{
aligned} \frac{R(p_{1}-q_{1})}{2(p_{1}+q_{1})}\|\boldsymbol{\mu}-\boldsymbol{
\nu}\|(1+o(1))<0\quad\text{for}\,i\in C_{0}\,\text{and}\,i\in H_{1}\\
\frac{R(q_{1}-p_{1})}{2(p_{1}+q_{1})}\|\boldsymbol{\mu}-\boldsymbol{\nu}\|(1+o
(1))>0\quad\text{for}\,i\in C_{1}\,\text{and}\,i\in H_{1}\\
\end{aligned}\right.$

As a result, the same linear classifier can separate both the homophilic set $H_{0}$ and heterophilic set $H_{1}$.

∎

###### Proof.

### Appendix B The Impact of Filter Smoothing Loss

In this section, we explore the impact of the proposed filter smoothing loss on the behavior of the learned filters in our Node-MoE framework. Figures 10 and 10 display the effects of the Node-MoE framework without and with the application of filter smoothing loss, respectively. Without the filter smoothing loss, as shown in Figure 10, the learned filters exhibit significant oscillations, making it challenging to discern their specific functions. In contrast, with the filter smoothing loss applied, as illustrated in Figure 10, the behavior of the filters becomes more distinct: filter 0 clearly functions as a low-pass filter, and filter 1 as a high-pass filter.

Figure: Figure 9: Learned 2 filters by Node-MoE on Chameleon without filter smoothing loss.
Refer to caption: extracted/5644463/Figures/chameleon_MoE_filter_2_0.png

Additionally, we assessed the training dynamics of the proposed Node-MoE framework by comparing performance with and without the filter smoothing loss, while keeping other hyperparameters constant. For the Citeseer dataset, applying the filter smoothing loss resulted in a higher average training accuracy of 99.37 ± 0.17, compared to 93.51 ± 1.27 when the loss was not applied. A similar pattern was observed on the Squirrel dataset, where the training accuracy was 96.54 ± 1.42 with the filter smoothing loss, versus 95.54 ± 0.94 without it. These results suggest that oscillations in the filters without the smoothing loss can hinder the model’s ability to fit the data effectively, resulting in suboptimal performance as shown in Section 4.4.

### Appendix C Datasets and Experimental Settings

In this section, we detail the datasets used and the experimental settings for both the baseline models and the proposed Node-MoE framework.

#### C.1 Datasets

We conduct experiments across seven widely recognized datasets, which encompass both homophilic and heterophilic types. The homophilic datasets include Cora, CiteSeer, and Pubmed [], along with ogbn-arxiv []; the heterophilic datasets comprise Chameleon, Squirrel, and Actor []. For Cora, CiteSeer, and Pubmed, we generate ten random splits, allocating nodes into training, validation, and testing sets with proportions of 60%, 20%, and 20%, respectively. For the heterophilic datasets, we adhere to the ten fixed splits as defined in []. The ogbn-arxiv dataset is assessed using its standard split as established by []. Detailed statistics of these datasets are shown in Table 2.

Table: Table 2: Statistics of datasets. The split ratio is for train/validation/test.

#### C.2 Experimental Settings

For the baseline models, we adopt the same parameter setting in their original paper. For the proposed Node-MoE, we adopt the GIN as gating and GCNII as experts. Notably, the GCNII model has different learning rates and weight decay for the filters and other parameters. All the hyperparameters are tuned based on the validation accuracy from the following search space:

- •
Gating Learning Rate: {0.0001, 0.001, 0.01 }
- •
Gating Dropout: {0, 0.5, 0.8}
- •
Gating Weight Decay: {0, 5e-5, 5e-4}
- •
Expert Learning Rate for Filters: {0.001, 0.01, 0.1}
- •
Expert Weight Decay for Filters: {0, 5e-5, 5e-3, 5e-2 }
- •
Expert Learning Rate: {0.001, 0.01, 0.1, 0.5}
- •
Expert Dropout: {0, 0.5, 0.8}
- •
Filter Smoothing loss weight: {0, 0.01, 0.1, 1}
- •
Load balancing weight for top-k gating: {0, 0.001, 0.01, 0.1, 1}
- •
Number of experts: {2, 3, 5}

For the initialization of filters in ChebNetII, which uses a K-order approximation, we employ a set of initialization strategies for the polynomial coefficients. These strategies include: decreasing powers $[\alpha^{0},\alpha^{1},\cdots,\alpha^{K}]$, increasing powers $[\alpha^{K},\alpha^{K-1},\cdots,\alpha^{0}]$, and uniform values $[1,1,\cdots,1]$. For configurations with 2 or 3 experts, we set $\alpha=0.9$. When expanding to 5 experts, we use two values of $\alpha$, setting them at $0.9$ and $0.8$, respectively, to diversify the response characteristics of the filters.

We use a single GPU of NVIDIA RTX A5000 24Gb, to run the experiments.

### Appendix D Analysis of the proposed Node-MoE

In this section, we provide more analysis of the proposed Node-MoE by comprehensive experiments.

#### D.1 The behavior of Node-MoE with 2 experts

The learned filters and the corresponding gating weights for nodes with different homophily levels are illustrated below. For the Chameleon dataset, these are displayed in Figure 12 for the filters and Figure 12 for the gating weights. Similarly, for the Citeseer dataset, the filters are shown in Figure 14 and the gating weights in Figure 14.

Figure: Figure 11: Learned 2 filters by Node-MoE on Chameleon.
Refer to caption: extracted/5644463/Figures/chameleon_MoE_filter_2_1.png

Figure: Figure 13: Learned 2 filters by Node-MoE on Citeseer.
Refer to caption: extracted/5644463/Figures/CiteSeer_MoE_filter_2.png

For both datasets, the learned filters demonstrate distinct characteristics: filter 0s function as low-pass filters, effectively smoothing signals, while filter 1s respond more strongly to high-frequency signals, characteristic of high-pass filters. Specifically, for the heterophilic dataset, such as Chameleon, the gating model generally assigns higher weights to filter 1, indicating a preference for high-pass filtering to accommodate the less homophilic nature of the dataset. Conversely, for the homophilic dataset, such as Citeseer, higher weights are typically assigned to filter 0, emphasizing low-pass filtering.

Moreover, within the Chameleon dataset, the weight assigned to the high-pass filter (filter 1) decreases as the homophily level increases. In contrast, in the Citeseer dataset, the weight to the low-pass filter (filter 0) increases with rising homophily levels. This pattern supports our initial hypothesis: nodes with lower homophily are better served by high-pass filters to capture the dissimilarity among neighbors, while nodes with higher homophily benefit from low-pass filters to reinforce the similarity among neighboring nodes.

#### D.2 Accuracy analysis between the Node-MoE with the individual expert

In this section, we evaluate the performance differences between the proposed Node-MoE framework and a single expert model, specifically ChebNetII, across nodes with varying homophily levels in the Chameleon dataset.

Figure: Figure 15: The filter learned by ChebNetII on Chameleon.
Refer to caption: extracted/5644463/Figures/chameleon_MoE_filter_1.png

The filter characteristics learned by the individual ChebNetII are depicted in Figure 16, which shows that this filter responds strongly to both low and high-frequency components. The comparative performance analysis is presented in Figure 16. From these results, it is evident that ChebNetII only outperforms Node-MoE when the node homophily level is exceptionally high. These findings indicate that the reliance on a single global filter, such as with ChebNetII, can lead to suboptimal performance. While such a filter might fit exceptionally well for nodes of a particular type, it may not adequately address the needs of nodes in other groups, thereby limiting overall effectiveness across diverse node types.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: How did the authors find GIN and ChevNetII as an gating and expert models respectively? What alternatives are considered and why these combination performs the best?
soundness: 3
strengths: - The use of the Contextual Stochastic Block Model (CSBM) to validate this limitation strengthens the motivation for NODE-MOE’s node-specific filtering approach.
- The NODE-MOE allows each node to dynamically access different filters through a mixture of experts.
- Extensive experiments show NODE-MOE’s effectiveness across both homophilic and heterophilic datasets.
summary: The paper introduces NODE-MOE, a GNN framework that uses a mixture of experts (MoE) model to apply node-specific filters, addressing the limitations of traditional GNNs that use a uniform global filter.  NODE-MOE’s architecture includes a gating model, which assigns each node a set of specialized filters from multiple expert models, allowing for dynamic adaptation to the local structure of each node. Experimental results across diverse datasets demonstrate NODE-MOE’s superior accuracy over standard GNNs, especially in noisy environments and with limited labels.
weaknesses: - The abstract and introduction of this paper require revision to reflect the existing literature more accurately. Currently, statements such as “Traditionally, GNNs employ a uniform global filter” in the abstract and introduction *could mislead readers*, including reviewers, by implying that no prior research has explored GNNs with multiple filters. However, as discussed in the related work section, numerous studies share similar motivations and have developed approaches that utilize multiple filters in GNNs. To provide a clearer foundation, the authors should acknowledge these existing contributions in both the abstract and introduction, clarifying how their work builds upon or differentiates from these prior approaches. This adjustment will ensure a more accurate representation of the field and help readers better understand the novelty and contributions of this paper.
- The paper claims that the proposed model offers better computational complexity than previous work discussed in the related section. However, no rigorous analysis is provided to substantiate this claim. Given the highly parallelizable nature of some existing “post-fusion” architectures, these prior models may, in fact, achieve lower computational costs.
- Theorem 1 only provides proof for linear models with lower expressivity than typical GNNs. Moreover, the proof in part 2 relies on a strong assumption that every homophilic and heterophilic node can be perfectly classified and assigned the appropriate filter.
- The proposed NODE-MoE seems to use a larger number of learnable parameters compared to other baselines. How many learnable parameters are used for both the baselines and NODE-MoE, as shown in Table 1? Is the same performance improvement observed, even if the number of parameters for NODE-MoE is adjusted to be on a similar scale as the baselines?
- The proposed NODE-MoE seems to have a broader hyperparameter search space than other baselines. Since its training time per epoch is longer, this broader search space would likely result in a significantly longer training time, including the validation time required to find the best hyperparameter set. Is the same performance improvement observed, even if the total training time, including hyperparameter search, is adjusted to be comparable to the baselines?
- Figures 6 and 15 seem inconsistent with the underlying motivation. While Figure 6 shows that the weights for the high-pass filter decrease as the homophily level rises, the difference between the two weights is not significant across the entire range of homophily. Moreover, even when the homophily level reaches 1, the high-pass filter’s average weight still exceeds that of the low-pass filter. Additionally, in Figure 15, even the trend observed in Figure 6 no longer exists.
- Minor: There appears to be an error in y-tick (density) values in Figure 2, as they are not within the 0-1 range.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1) Given that the performance gains in Table 1 are not very clear (due to the overlap in std. dev. with non MoE baselines) what were the winning hyper parameters from the search space?  The reason I ask is because the paper claims that due to efficiency reasons one can reduce the number of experts (say 2) and utilize gating with Top K = 1. But since that come with the trade-off with accuracy, I am curious about the setting used for the main results presented.
soundness: 3
strengths: 1) Good motivation and interesting use of MoE for dealing with the problem of node having varying levels of homophilic / heterophilic patterns in real world graphs. 

2) While the use of MoE to graph data is not new, the paper presents some new insights when used for node level tasks e.g., the use of filter smoothing loss, experts with node wise filtering. Experimentally, performs better than other graph based MoE techniques. 

3) Good and reasonable set of experiments on the analysis of the proposed method, detailed experiments on the time complexity and hyper parameter choices.
summary: The paper introduces a Mixture of Experts inspired technique for GNNs aimed to improve classification performance on heterophilic and homophilic nodes in a graph. The paper first, presents some analysis to show how real-world graph contain varying levels of homophily and a single fixed "low" or "high" pass filter may not be the best solution to learn reasonable node features. Thus, motivated by this analysis, the authors propose a mixture of experts technique, wherein a gating model is capable of "soft" selection of representation of a bunch of expert GNN models with varying node wise filter levels. Experiments show that the proposed method generally outperforms previous graph MoE based methods and other standard baselines. Detailed ablations and analysis into the hyper parameters are presented. Some considerations into the efficiency of the proposed methods are also presented.
weaknesses: 1) Limited novelty of the proposed problem setup (Section 2.1 and 2.2). It is well known that real world graphs contain nodes that exhibit variations in homophilic behavior. [ref - 1,2]. Limited technical novelty since, much of the techniques used in the proposed method is based on earlier work in the MoE area like Top K sampling, use of gating mechanism [3].

2) The gains in performance due to the proposed method "Node-MoE" is not clear compared to other non MoE baselines given the significant overlap of mean standard deviation. Moreover, the heterophilic datasets chosen are known to be problematic [4]. It would a stronger argument, if additional new datasets from [4] are used for the experiments. 

3) A clear algorithm detailing the different techniques can be presented for better clarity to the reader. 

Refs. 
[1] Peel et al. Multiscale mixing patterns in networks. PNAS (2018).
[2] Suresh et al. Breaking the limit of graph neural networks by improving the assortativity of graphs with local mixing patterns. KDD (2021).
[3] Shazeer et al. Outrageously large neural networks: The sparsely gated mixture-of-experts layer. ICLR (2017)
[4] Platonov, et al.  A critical look at the evaluation of GNNs under heterophily: Are we really making progress?. ICLR (2023)

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: see the details in the weaknesses part
soundness: 3
strengths: The proposal of MoE-like node label filtering is particularly interesting, as it enhances the method's suitability for handling real-world graphs with diverse patterns.

The authors provide a comprehensive analysis throughout the paper, complemented by a thorough set of experimental studies that effectively support their proposed approach.
summary: This paper tackles the challenges faced by Graph Neural Networks (GNN) in real-world graphs that exhibit both hemophilic and heterophilic patterns, which complicates processing with a single global filtering approach. The authors propose a Mixture of Experts (MoE) based node classification method specifically designed to address these issues. By utilizing node-level MoE, the proposed method effectively navigates the complexities of these patterns, demonstrating promising results through extensive experimental evaluations.
weaknesses: * Figure 1 serves as a poor illustrative example, as it may lead to confusion. The introduction in Section I states that “nodes tend to connect with others that share similar labels, reflecting homophilic patterns,” yet Figure 1 shows both solid and dashed edge nodes connecting to nodes labeled  and 1 on the right side, which contradicts this description.
* The message and patterns presented in Figure 2 lack clarity regarding their relevance to the key contributions of the work. The statement that “the majority of nodes in homophilic graphs predominantly exhibit homophilic patterns” raises questions about the definition of homophilic graphs. According to the authors' definition in the Homophily metrics section, this concept only applies to graphs with different labels; otherwise, h(v) will always equal 1.
* The paper suffers from overloaded notations that can lead to confusion for readers. For instance, h(v_i) is used in Section 1 to denote the label similarity between a node v_i and its neighbors, while in Section 3.1, the same notation is repurposed to refer to a node classifier. This inconsistency may hinder comprehension.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: See the questions in "Weaknesses".

For the statement: "... different structural patterns are not uniformly distributed across the graph, and distinct communities
may exhibit varying structural characteristics. To capitalize on this phenomenon, we employ GNNs
with low-pass filters, such as GIN (Xu et al., 2018), for the gating model. These networks are chosen
due to their strong community detection capabilities (Shchur & Günnemann, 2019; Bruna & Li,
2017), ensuring that neighboring nodes are likely to receive similar expert selections", I also have two questions:

1. Could you elaborate on why GIN has strong community detection capabilities?

2. Regarding "ensuring that neighboring nodes are likely to receive similar expert selections", does this apply to both homophilic and heterophilic graphs? If so, how would the approach differ between them? In heterophilic graphs, neighboring nodes are more likely to differ in features or labels, so how would similar expert selections across neighbors work?
soundness: 3
strengths: **originality**

The originality of the paper is marginal, as it builds on well-known observations (see "weakness").

**quality**

The paper appears to be technically sound, presenting a nice theoretical analysis based on the CSBM model, which motivates the development of its proposed node-wise filtering method.

**clarity**

The paper is well-structured and generally easy to follow. However, Figure 1 could be improved, as it currently reads a bit confusing. For instance, there are no numbers or labels on the right side of Figure 1(a), making it unclear which nodes correspond to those on the left side after applying the global filter. Also, what are the key messages of the figure? The figures in Section 4 (Experiments) could be enhanced, e.g., the font sizes for the x-axis and y-axis labels in Figures 6 and 7 are too small, which makes them difficult to read.

**significance**

The significance of the proposed method is limited due to its marginal originality.
summary: In this paper, the authors address limitations in graph neural networks (GNNs) that arise when only a single global filter is used. They observe that graphs may contain a combination of homophilic and heterophilic patterns within the same graph, with varying levels of homophily across different subgraphs or communities. Based on the patterns generated by the  CSBM model, they theoretically analyze the (near) linear separability of nodes when (1) a low-pass global filter is applied to the entire graph and (2) different filters are applied separately to nodes with different patterns. These theoretical insights lead to the development of a novel GNN model that employs a mixture-of-experts approach to adaptively select filters for nodes based on their specific patterns.
weaknesses: (W1) The observation of varying structural patterns (homophilic and heterophilic patterns within the same graph) is not highly original. While the paper provides a well-illustrated demonstration in Section 2.1, analyzing these patterns from two perspectives: node homophily density and homophily across different communities, this does not significantly enhance the originality. These structural variations are commonly known and have been addressed in previous works using various approaches as discussed in Section 5 Related Works.

(W2) The proposed method assumes that a heterophilic pattern is present when a node’s features significantly differ from those of its neighboring nodes. While this assumption is reasonable, it restricts the method's applicability. Node features do not always correlate with node labels in terms of homophilic and heterophilic patterns, which may reduce the method's effectiveness when feature-label alignment is weak. Although the authors argue that the proposed method is distinct from existing works because those approaches pass all nodes through all filters, these prior methods do not make specific assumptions about heterophilic patterns, instead allowing the filters to adaptively learn from data. Thus, the distinction between the current work and these prior methods seems more like a trade-off rather than a clear improvement.

(W3) The proposed method appears to be a combination of a gating function and a mixture of experts. When multiple GNNs with different filters are used, the model grows in complexity, introducing additional learnable parameters and hyperparameters (such as $\gamma$ for adjusting the influence of the filter smoothing loss, and $K$ for the number of experts). This added complexity typically leads to performance improvements, so the modest gains shown in the experimental results are not surprising.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: See weaknesses.

**Additional questions.**

Q1. The average training time per epoch for ChebNetII and Node-MoE-3/5 are quite similar. What is the end-to-end training time comparison between these methods?

Q2. In Appendix C.2, the strategies for polynomial coefficients include both increasing/decreasing powers of $\alpha^k$ and uniform values. How would initializing each filter randomly impact performance?

Q3. What are some limitations of this work?
soundness: 2
strengths: * Theoretical and empirical analysis highlighting the limitations of applying a single global filter.

* A suite of experiments across homophilic and heterophilic datasets demonstrating the effectiveness of the proposed method.

* Multiple ablation studies validating specific design choices, such as different gating variants.
summary: The paper identifies that real-world graphs exhibit a complex mix of homophilic and heterophilic patterns, meaning each node may display varying levels of homophily. Consequently, applying a single global filter leads to suboptimal performance. Using real-graphs and synthetic graphs constructed using the CSBM model, the authors empirically and theoretically demonstrate that a global filter optimized for one pattern increases the loss for other patterns. To address these limitations, they propose Node-MoE: Node-Wise Filtering Via Mixture of Experts.

Node-MoE dynamically applies distinct filters to individual nodes based on their specific patterns. The gating/routing model utilizes a composite vector, applying a GIN network to assign weights to each expert model for each node. Expert models are instantiated using ChebNetII/GCNII with fixed filter initializations, such as low-pass, high-pass, or constant filters. An additional smoothing loss is incorporated to enhance training stability and improve explainability. The proposed method is validated across multiple homophilic and heterophilic datasets, benchmarked against a diverse set of baselines.
weaknesses: * A comparable framework, Link-MoE [1], was previously proposed for link prediction and bears a close resemblance to the overall framework of Node-MoE. However, this work is neither cited nor discussed in the related literature. Given the presence of this recent work, the novelty of the proposed method appears limited.

* While the paper argues that applying different filters to homophilic and heterophilic separately can lead to linear separability, a similar effect might also be achieved by increasing the representation capacity of global filters. For instance, by increasing the order in polynomial filters, or by learning piecewise polynomial filters[2]. Moreover, [3] explores node-wise filters via polynomial graph transformers. A comparison with these methods is necessary to evaluate the effectiveness of Node-MoE.

* Table 1 compares Node-MoE with ACM-GCN [4]. However, [4] also includes additional results for ACM-GCN+ and ACMII-GCN++, which achieve 76.08% (Node-MoE: 73.64%) on the Chameleon dataset and 69.98% (Node-MoE: 62.31%) on the Squirrel dataset, with similar improvements across other datasets. GCN+ and GCN++ incorporate MLP(A) as additional features. Moreover, [5] demonstrated that incorporating structural information (A) as additional features alongside simple graph filters can yield notable improvements—reaching 77.48% on the Chameleon dataset and 74.17% on the Squirrel dataset. In light of these results, the necessity of an MoE-based approach remains unclear.

---
[1] Mixture of Link Predictors, NeurIPS24 (arXiv version available since 13 Feb 2024)

[2] A Piece-Wise Polynomial Filtering Approach for Graph Neural Networks, ECML22

[3] PolyFormer: Scalable Node-wise Filters via Polynomial Graph Transformer, KDD24

[4] Revisiting Heterophily for Graph Neural Networks, NeurIPS22
Supplementary:https://proceedings.neurips.cc/paper_files/paper/2022/file/092359ce5cf60a80e882378944bf1be4-Supplemental-Conference.pdf

[5] Simple Truncated SVD based Model for Node Classification on Heterophilic Graphs, DLG-KDD21

Meta-review: This paper proposes NODE-MOE, a Graph Neural Network (GNN) framework that combines a mixture of experts (MoE) with node-specific filters to address the limitations of traditional GNNs relying on uniform global filters. The architecture features a gating model that dynamically assigns each node a set of specialized filters from multiple expert models, enabling adaptive filtering based on local node structure.

The authors argue that real-world graphs exhibit complex patterns that cannot be captured by a single global filter, citing the coexistence of homophilic and heterophilic structures. However, this perspective oversimplifies the complexity of real-world graphs, which often defy categorization as solely homophilic or heterophilic.

While the experimental results demonstrate the effectiveness of NODE-MOE, particularly in noisy environments and with limited labels, I concur with reviewers AtKX and VQRg that the paper lacks a clear theoretical foundation. The gating mechanism's selection process and potential failure conditions are not well-understood, and the provided theory, based on linear models and Gaussian vectors, is overly simplistic and fails to provide meaningful insights into the model's behavior in practice.

Furthermore, the paper's approach is essentially an integration of homophily and heterophily concepts into GNN mixture models (e.g., GraphMETRO and GMoE), which feels ad hoc, lacking a solid theoretical justification. The authors should provide a more rigorous analysis of the relationships between these concepts and the proposed model, as well as a clearer explanation of when and why NODE-MOE is expected to work and under which conditions.

The reviewers provided thorough and insightful evaluations of the paper, identifying both its strengths and weaknesses. After carefully reading the manuscript, I concur with their assessments and share their concerns regarding the lack of a solid theoretical foundation. The reviewers' comments and critiques have helped to clarify the paper's contributions and limitations, and I believe that the authors should address these issues in order to strengthen the work. Overall, I appreciate the diligence of the reviewers and I believe that their feedback will be invaluable in helping the authors to improve the paper.

Decision: Reject