Graph Attention Multi-layer Perceptron

Abstract: Recently, graph neural networks (GNNs) have achieved a stride of success in many graph-based applications. However, most GNNs suffer from a critical issue: representation learned is constructed based on a fixed k-hop neighborhood and insensitive to individual needs for each node, which greatly hampers the performance of GNNs. To satisfy the unique needs of each node, we propose a new architecture -- Graph Attention Multi-Layer Perceptron (GAMLP). This architecture combines multi-scale knowledge and learns to capture the underlying correlations between different scales of knowledge with two novel attention mechanisms: Recursive attention and Jumping Knowledge (JK) attention. Instead of using node feature only, the knowledge within node labels is also exploited to reinforce the performance of GAMLP. Extensive experiments on 12 real-world datasets demonstrate that GAMLP achieves state-of-the-art performance while enjoying high scalability and efficiency.

Reviews and author responses:

The paper proposes an efficient method for inductive/transductive node labeling based on fast feature propagation, label propagation, and attention and yields good experimental results. The paper introduces an efficient way to perform node labeling tasks on inductive and transductive setting. It achieves good results on the widely used benchmarks. Here are the questions / concerns:

1- In some parts it is difficult to follow the paper. And at some points the information is missing. e.g., definition of T in eq(2) and s in eq (7). eq (7) and (8) seems to be identical. The dimensions are not explicitly mentioned which could have made the paper more clear.

2- In terms of contribution, the paper seems to have incremental contributions with respect to previous work but not a significant one. Using a weighted average over the features from different layers seems very similar to attention graph pooling and also feature propagation is not different from previous work. Also using label propagation has been previously explored.

3- The other potential limitation of the proposed model is that it is tailored to supervised setting and won't be able to perform in contrastive setting with linear evaluation protocol. The paper introduces some incremental contributions with empirical support. But at the end of day, the contributions are slight variations of previous work. 

 I would like to thank the authors for their responses. After going through the responses on my comments and other reviewers, I increased my score. However, I still think that the idea is somewhat incremental.

 Hi Reviewer eUi3,

Could you take a look at author response and update your review? The authors have provided detailed comments as well as additional experimental results based on your suggestions and questions. 

The paper proposed the Graph Attention Multi-Layer Perceptron (GAMLP) model that combines features extracted from two streams: 1) node-adaptive features that are obtained via "Graph-wise / layer-wise propagation" (Section 2.2), 2) features obtained from label propagation. The author also proposed two ways for performing the node-adaptive attention: JK attention, and Recursive attention (Section 3.3). Extensive experiments are conducted on 9 transductive datasets and 3 inductive datasets. The results show that the proposed GAMLP(JK) and GAMLP(R) can outperform the baselines.
 - Overall Novelty

Combining layer-wise propagation and label propagation is a good idea. However, it is not clear what's new in GAMLP except ensembling two types of models. Fusing these two streams do not look very challenging and it is foreseeable that the model can achieve competitive performance as GBP (based on layer-wise propagation) or UniMP (based on label propagation). The proposed "JK attention" and "Recursive attention" are also not very novel and are largely borrowed from previous works like JK-Net.

- Technical Quality + Writing

The technical quality of the paper is good. The author summarized the current landscape of scalable GNNs and categorized them as 1) sampling-based, 2) graph-wise propagation and 3) layer-wise propagation. The author also provided the reference to prior works such as label propagation and JK-Net. Also, the author clearly stated the general algorithm in Figure 2 and the detailed design in Section 3. However, the author can still improve the paper by providing insights on some technical designs, e.g.

    1. Why choose the `cos(...)` formula in designing the "Last Residual Connection" module?
    2. What's the fundamental differences between JK attention and Recursive attention? Why would you propose two variants?

- Experiments

The author conducted extensive experiments in 9 transductive datasets and 3 inductive datasets. However, I find that the baseline selection is not consistent across the experiments. For example, GBP is not compared in Table 2. UniMP is not compared in Table 1.

In addition, the trend of validation and test accuracy reported in Table 5 is not consistent (models with higher validation accuracy tend to have lower test accuracy). The author needs to clarify the inconsistency. The general idea of GAMLP is neat and reasonable. However, I'm hesitant to accept the paper because I haven't seen a contribution that is clearly novel. In addition, there are some inconsistencies in the experiments.

——— Post Rebuttal ———
The author has addressed my concerns regarding the experiments and the motivation of proposing two attention mechanisms (recursive attention v.s JK attention). Given the nice results and scalability of GAMLP, I’ve increased my score to 6.  However, I’m still concerned about the novelty of the paper because it proposed several ideas but none of them is really novel. The node-wise feature propagation idea looks like a direct extension of the previous layer-wise / graph-wise propagation mechanisms. The major difference is that the node-wise propagation will use the propagated features from multiple depths. Thus, I haven’t increased my score to accept due to lack of novelty. 

 Dear reviewers, 

Authors have provided us with a detailed author response for each of your reviews. 
Could you take a look and comment on how it changed your initial review? 
Even when your review and score does not change with author review, it would be helpful to know why. 
Thanks!

best,
Area Chair

 We thank reviewer Cwzz for your insightful reviews and valuable time. We address all concerns as follows. 
### Q1: Typo and dimensions.
Thanks for pointing out these typos. We have replaced T in eq(2) with $\mathbf{\hat{A}}$, which is the symmetric normalization adjacency matrix. Besides, $s$ in eq(7) is a learnable parameter vector, and $\sum_{k=0}^{l-1} w_i(k) \mathbf{X}_i^{(k)}$ in eq(8) has been replaced with $\mathbf{E}_i$. 

We apologize for we bring you inconvenience in your reading. We have updated the eq(2), eq(7), and eq(8) in the revised manuscript as follows.
eq(2):
\begin{equation}
\small
    \mathbf{X}^{(k)} \gets \hat{\mathbf{A}}\mathbf{X}^{(k-1)},\ \forall  k=1,\ldots, K,
\end{equation}

E(7):
\begin{equation}
\small
\widetilde{\mathbf{X}}\_i\^{(l)} = \mathbf{X}\_i^{(l)} \parallel \sum_{k=0}\^{l-1} w\_i(k) \mathbf{X}\_i\^{(k)}, \quad    {\widetilde{w}\_{i}(l)} = \delta(\widetilde{\mathbf{X}}\_i\^{(l)} \cdot s), \quad  w\_{i}(l) = {e\^{\widetilde{w}\_{i}(l)}}/{\sum \limits_{k=0}\limits\^{K} e\^{\widetilde{w}\_{i}(k)}}.
\end{equation}

eq(8):
\begin{equation}
\small
\widetilde{\mathbf{X}}\_i\^{(l)} = \mathbf{X}\_i\^{(l)} \parallel \mathbf{E}\_i, \quad    {\widetilde{w}\_{i}(l)} = \delta(\widetilde{\mathbf{X}}\_i\^{(l)} \cdot s), \quad  w\_{i}(l) = {e\^{\widetilde{w}\_{i}(l)}}/{\sum \limits_{k=0}\limits\^{K} e\^{\widetilde{w}\_{i}(k)}}.
\end{equation}

We have also added the following dimension descriptions:

eq(3): $\mathbf{W}\_k  = Diag(\eta\_{k}) \in \mathbb{R}^{n \times n}$, $\eta\_{k} \in \mathbb{R}^{n}$

eq(4): $\mathbf{Y} \in \mathbb{R}^{n \times c}$ 

eq(7): $s \in \mathbb{R}^{d}$, $\widetilde{\mathbf{X}}\_i^{(l)} \in \mathbb{R}^{d}$

eq(8): $\mathbf{E}\_i \in \mathbb{R}^{Kf}$


### Q2: Contrastive setting.
GAMLP can be applied in the DGI framework to support contrastive learning. Specifically, DGI originally takes GCN as the base model. It permutates the node features in each training epoch and feeds both the original features and the permutated features into GCN to generate node embeddings. DGI then computes the graph embedding and the contrastive loss using the generated node embeddings.

Here we replace the GCN in DGI with GAMLP without the label part. We execute the propagation operations in GAMLP still as preprocessing, yet two feature lists are generated: one from the propagation operations on the original features, the other one from the propagation operations on the permutated features. These two feature lists are then combined using the attention mechanism in GAMLP to produce node embeddings. The subsequent procedures are kept the same as in the DGI framework.

At last, we use Logistic Regression to evaluate the qualities of the generated node embeddings on the node classification task. In the experiment, DGI with GAMLP achieves 83.2% on the Cora dataset, outperforming DGI with GCN by a margin of 1.1%. The experiment result strongly verifies the feasibility of applying GAMLP in the contrastive setting.

 ### Q3: Novelty and Contribution.

**1.Novelty of weighted average over different features.**

(1) To the best of our knowledge, GAMLP is the first to weighted average over the features from different propagation steps in GNN. The most widely-applied attention mechanism in graph pooling is self-attention, which is often used to learn the importance of a specific node to other nodes or a specific subgraph. This popular design is much more similar to the attention mechanism in GAT [1] than ours. We have also noticed a recent graph pooling research that proposes to combine multi-level representations using learnable scalars [2]. However, the learnable scalars in [2] are not node-adaptive, and the individual properties of nodes are neglected.


(2) Our mechanisms to get the attention weights are new. Specifically, **Recursive attention** can capture the information gain compared with the previous combined feature, and **JK attention** can learn the importance of propagated features by comparing with the concatenated features at a different scale.

**2.Novelty of feature propagation.**

Admittedly, the feature propagation process is the same as SGC. However, SGC could not leverage multi-hop information, and it only leverages the information of the last feature propagation, which lost much information and so as the performance. Other methods such as JK-Net indeed incorporate multi-layer information, but the time and space cost is expensive. The model could not even apply to million scale graph, let alone a billion scale. The GAT uses an attention mechanism in the GCN, but the attention is neither node-wise nor scalable to a large graph.

**3.Novelty of label propagation.**

(1) We are the first to propose the **last residual connection** to address the problem of label leak when introducing label propagation to model training. By the Last residual, we can leverage node-wise multi-hop label information without seriously **over-fitting**. Unlike UniMP, which adopts the additional mask part to avoid over-fitting in each training batch, our GAMLP is much faster since the last residual connection only needs to be done once in the pre-process.

(2) We are the first to incorporate **multiple hop label information** in a node-wise manner, while previous work (e.g., UniMP and SAGN) only uses a single hop label information.

**4.Summary of the contributions.**

To sum up, our contributions mainly come from the following aspects: 

(1)We propose to consider the **node-wise feature propagation** in scalable GNNs, while previous methods adopt layer-wise propagation (e.g., GBP[3]) or graph-wise propagation (e.g., SGC[4]).

(2) While combining GNN and label propagation is not new, we are the first to propose **node-wise label propagation**. 

(3) We propose the **last residual connection module** to tackle the label leakage problem in label utilization.

(4) We combine multi-scale feature and label knowledge and learn to capture the underlying correlations between different scales of knowledge with two **attention mechanisms**.

[1] Veličković, Petar, et al. [*Graph Attention Networks.*](https://openreview.net/pdf?id=rJXMpikCZ) ICLR. 2018.

[2] Itoh, Takeshi D., Takatomi Kubo, and Kazushi Ikeda. [*Multi-Level Attention Pooling for Graph Neural Networks: Unifying Graph Representations with Multiple Localities.*](https://arxiv.org/pdf/2103.01488) arXiv preprint arXiv:2103.01488 (2021).

[3] Chen, Ming, et al. [*Scalable Graph Neural Networks via Bidirectional Propagation.*](https://arxiv.org/abs/2010.15421) NeurIPS. 2020.

[4] Wu, Felix, et al. [*Simplifying graph convolutional networks.*](http://proceedings.mlr.press/v97/wu19e/wu19e.pdf) ICML. 2019.

We are very glad to respond if you have any new questions.

Respectfully,

Paper610 Authors

 ### Q3: Heterophily-based data and data with heterogeneous node features.
We are sorry that the original GAMLP cannot perform well on heterophily-based data and data with heterogeneous node features. After several simple modifications, decent performance on these two special settings is expected for GAMLP.

(1) For heterophily-based data, if two nodes are connected by an edge, the representations of these two nodes will be dissimilar with high probabilities. An easy way to adapt GAMLP to heterophily-based data is to *generate two sets of propagated features, one on the graph with self-loops, the other one on the graph without self-loops*. This simple approach is first proposed in [1], and it achieves great performance on heterophily-based data. In GAMLP, these two sets of propagated features are unioned and then fed into the attention mechanism. The basic idea is to isolate the combination of the neighboring features from the features of the center node. It is inappropriate to combine these two features the same as in the popular GCN framework since they are dissimilar with high probabilities, and the combination will corrupt the representation of the center node. 
After this modification, GAMLP is able to learn adaptive weights for both the combination of centering nodes and neighboring nodes and neighboring nodes alone, which makes it perform well on both homophily-based data and heterophily-based data.

(2) For heterogeneous graphs with heterogeneous node features, we only consider GAMLP without the lable part. Suppose there are $\small R$ types of nodes in total and each type of nodes have their unique kind of features. For node $\small i$, we denote its kind as $\small r_i$, and the dimension of its feature is denoted by $\small Dim_{r_i}$. We change the dimensions of the initial features of nodes to $\small \sum_{r=1}^R Dim_r$. In the feature vector of node $\small i$, the $\small \left[\sum_{r=1}^{p-1} Dim_r, \ \sum_{r=1}^{p} Dim_r \right)$ positions stand for the features for the kind $\small p$. And the original features of node $\small i$ are filled in the $\small \left[\sum_{r=1}^{r_i-1} Dim_r, \ \sum_{r=1}^{r_i} Dim_r \right)$ positions of the vector. The other positions of node $i$'s feature vector are initialized as zeros. 
In the propagation process, we update the corresponding positions of a node's feature vector using its corresponding kinds of neighbors. For example, if a node has two type $\small p$ neighbors and three type $\small q$ neighbors, then only the  $\small \left[\sum_{r=1}^{p-1} Dim_r, \ \sum_{r=1}^{p} Dim_r \right)$ and $\small \left[\sum_{r=1}^{q-1} Dim_r, \ \sum_{r=1}^{q} Dim_r \right)$ positions of its feature vector are updated using the corresponding positions of feature vectors of the two type $\small p$ neighbors and three type $\small q$ neighbors, respectively.
After this modification, GAMLP can perform on heterogeneous graphs with heterogeneous node features.


[1] Maurya, Sunil Kumar, Xin Liu, and Tsuyoshi Murata. [*Improving Graph Neural Networks with Simple Architecture Design.*](https://arxiv.org/pdf/2105.07634.pdf) arXiv preprint arXiv:2105.07634 (2021).

We are very glad to respond if you have any new questions.

Respectfully,

Paper610 Authors

 We thank reviewer XqaP for your insightful reviews and valuable time. We address all concerns as follows. 
### Q1: Novelty and Contribution.

To sum up, our contributions mainly come from the following aspects: 

(1) We propose to consider the **node-wise feature propagation** in scalable GNNs. 

Details: Previous scalable (decoupled) GNNs adopt layer-wise propagation (e.g.,[1]) or graph-wise propagation (e.g.,[2]), thus each node in these methods cannot have a personalized combination of different steps of the propagated features.

(2) We combine multi-scale feature and label knowledge and learn to capture the underlying correlations between different scales of knowledge with two **attention mechanisms**.

Details: Several attempts are trying to apply the attention mechanisms in GNNs [3][4][5][8]. However, most of them are either not scalable [3][4] or not in a node-wise manner [5]. The attention mechanism in [8] shares many similarities with the one in GAMLP, but it cannot exploit the relationship between features at different propagation steps like GAMLP does. 

(3) To the best of our knowledge, we are the first to propose **node-wise label propagation** in GNN. 

Details: Despite that incorporating label information is not new in GNN [6][7][8], GAMLP is the first to take advantage of label information at different propagation steps to the best of our knowledge.


(4) We propose the **last residual connection module** to tackle the label leakage problem in label utilization.

Details:  With the aid of the last residual connections, GAMLP can apply the attention mechanism to the labels from propagation step $\small 0$ to $\small K$. This achievement is not minor since directly applying attention mechanisms to labels from propagation step $\small 0$ to $\small K$ would lead to serious **over-fitting**.

[1] Chen, Ming, et al. [*Scalable Graph Neural Networks via Bidirectional Propagation.*](https://arxiv.org/abs/2010.15421) NeurIPS. 2020.

[2] Wu, Felix, et al. [*Simplifying graph convolutional networks.*](http://proceedings.mlr.press/v97/wu19e/wu19e.pdf) ICML. 2019.

[3] Veličković, Petar, et al. [*Graph Attention Networks.*](https://openreview.net/pdf?id=rJXMpikCZ) ICLR. 2018.

[4] Liu, Meng, Hongyang Gao, and Shuiwang Ji. [*Towards deeper graph neural networks.*](https://dl.acm.org/doi/pdf/10.1145/3394486.3403076) SIGKDD. 2020.

[5] Chien, Eli, et al. [*Adaptive Universal Generalized PageRank Graph Neural Network.*](https://openreview.net/pdf?id=n6jl7fLxrP) ICLR. 2020.

[6] Wang, Hongwei, and Jure Leskovec. [*Unifying graph convolutional neural networks and label propagation.*](https://arxiv.org/pdf/2002.06755) arXiv preprint arXiv:2002.06755 (2020).

[7] Shi, Yunsheng, et al. [*Masked label prediction: Unified message passing model for semi-supervised classification.*](https://arxiv.org/pdf/2009.03509) arXiv preprint arXiv:2009.03509 (2020).

[8] Sun, Chuxiong, and Guoshi Wu. [*Scalable and Adaptive Graph Neural Networks with Self-Label-Enhanced training.*](https://arxiv.org/pdf/2104.09376) arXiv preprint arXiv:2104.09376 (2021).

### Q2: Does GAMLP solves the aforementioned limitations in Figure 1?
Figure 1 shows that different nodes have different optimal propagation steps, which most previous works paid little attention to. With the aid of two new attention mechanisms, GAMLP assigns varied weight distributions over the propagation steps to different nodes. In other words, GAMLP assigns varied propagation steps to different nodes through learning from the training data. An interpretability analysis about the varied weight distributions assigned to different nodes is provided in Appendix B.3 in the original manuscript. The results show that the weights of large-degree nodes drop faster than the weights of small-degree nodes when the propagation step increases. This phenomenon is consistent with the intuition that nodes with large degrees tend to have small optimal propagation steps. This analysis illustrates that GAMLP has addressed the limitations shown in Figure 1.

 We thank reviewer eUi3 for your insightful reviews and valuable time. We address all concerns as follows.

### Q1: GAMLP vs. SAGN+SLE
Thanks for mentioning the similar work -- SLE. In fact, the paper you mentioned contains two parts: the model **SAGN** and the training method **SLE**. SLE adopts the pseudo label[1] -- a commonly used self-training trick in Kaggle competition to further improve the model performance. However, we think such a design will make the model more complex and harder to analyze the true performance gain. This is also the reason why we just compared GAMLP with SAGN (without SLE) in Section 3.6 of our original manuscript. 

As we introduced in Section 3.6, SAGN+SLE also adopted the label propagation and node-wise feature propagation. The main differences between our method and SAGN+SLE are as follows. 

(1) The two attention mechanisms in GAMLP can exploit the **relationship between features** at different propagation steps, while SAGN+SLE cannot; 

(2) the label usage in GAMLP is the first attempt to take advantage of **label information at different propagation steps** with the aid of last residual connections, while SAGN+SLE can only adopt label information at one fixed propagation step.

These two differences are also the two advantages of GAMLP over SAGN+SLE. We provide the performance comparison between GAMLP and SAGN+0-SLE in Q2, which shows that these two advantages of GAMLP turn into performance superiority over SAGN+0-SLE in real-world practice.

[1] https://www.kaggle.com/cdeotte/pseudo-labeling-qda-0-969

### Q2: The performance comparison.
SLE is originally proposed to be a multiple-stage self-training method, yet GAMLP has no self-training designs. To evaluate the effectiveness of the attention mechanism and the label usage of both methods, here we only provide the performance comparison between SAGN+0-SLE and GAMLP (JK) for fairness. Under this scenario, the self-training designs in SAGN+SLE are disabled. We include the results on two large-scale datasets as follows: 

| Method     | ogbn-products | ogbn-papers100M |
| ---------- | ------------- | --------------- |
| GAMLP (JK) | 83.54±0.25    | 68.07±0.10      |
| SAGN+0-SLE | 83.29±0.18    | 67.55±0.15      |

The table shows that GAMLP (JK) outperforms SAGN+0-SLE by a large margin, which empirically illustrates the novelty of GAMLP on both the attention mechanism and the label usage.



### Q3: Comparisons among the proposed attention mechanisms in GAMLP, "learnable scalar" and concatenating $\small [X^{0},\ \dots, X^{k}]$.
We followed your kind advice and provided the performance comparison between the two attention mechanisms in GAMLP and the two variants you are interested in. Since directly concatenating $\small [X^{0},\ \dots, X^{k}]$ is quite similar to the design of SIGN [1], here we directly fetch the performance data of SIGN from the OGB leaderboard instead.

The performance comparison on the ogbn-products dataset is provided in the table below, where GAMLP (S) stands for the "learnable scalar" variant. Here we exclude the label part in GAMLP since this exclusion provides a clearer view of how our proposed attention mechanisms affect the performance. GAMLP (S) in the table stands for the "learnable scalar" variant.

| Method                | ogbn-products |
| --------------------- | ------------- |
| GAMLP (JK) w/o label  | 81.43±0.18    |
| GAMLP \(R\) w/o label | 81.30±0.09    |
| GAMLP (S) w/o label   | 80.84±0.20    |
| SIGN                  | 80.52±0.16    |

The comparison clearly shows that the two attention mechanisms in GAMLP are superior to the other two variants in terms of performance.

Although the "learnable scalar" variant can learn the varied importance of features at different propagation steps, it assigns the identical weight distribution to all the nodes, which neglects the personalized properties of each node. In contrast, the two attention mechanisms in GAMLP generate node-adaptive weights for each node. 

The variant that directly concatenates $\small [X^{0},\ \dots, X^{k}]$ has its number of parameters grows linearly with $k$, making it quite hard to train when $k$ is large. In GAMLP, the number of parameters for implementing the two attention mechanisms is a small constant, independent of $k$. This property of the two attention mechanisms in GAMLP makes it possible for the model to effectively exploit deep structural information when $k$ is large.


[1] Frasca, Fabrizio, et al. [*Sign: Scalable inception graph neural networks.*](https://arxiv.org/pdf/2004.11198) arXiv preprint arXiv:2004.11198 (2020).

We are very glad to respond if you have any new questions.

Respectfully,

Paper610 Authors


 ### Q5. Experiments
These experiments are not done before in their original papers, and we have completed them in the revision. Table 2 and parts of Table 1 are as follows.

**Table 1: Performance comparison on seven transductive datasets.**

| Method   | Cora     | Citeseer | PubMed   | Amazon Computer | Amazon Photo | Coauthor CS | Coauthor Physics |
| -------- | -------- | -------- | -------- | --------------- | ------------ | ----------- | ---------------- |
| GCN      | 81.8±0.5 | 70.8±0.5 | 79.3±0.7 | 82.4 ±0.4       | 91.2±0.6     | 90.7±0.2    | 92.7±1.1         |
| GAT      | 83.0±0.7 | 72.5±0.7 | 79.0±0.3 | 80.1±0.6        | 90.8±1.0   | 87.4±0.2    | 90.2±1.4        |
| UniMP    | 83.1±0.4 | 72.5±0.9 | 80.1±0.5 | 83.9±0.8     | 92.0±1.1  | 92.4±0.3 | 93.5±0.8      |
| GAMLP (JK) | 84.3±0.8|  74.6±0.4       |     80.7±0.4     | 84.5±0.7       |    92.8±0.7        | 92.6±0.5          |  93.6±1.0                |
| GAMLP \(R\)         |83.9±0.6|73.9±0.6    |  80.8±0.5        |  84.2±0.5               |  92.6±0.8            |       92.8±0.7      |  93.2±1.0               |

**Table 2: Performance comparison on the ogbn-products dataset.**

| Method      | Val Accuracy | Test Accuracy |
| ----------- | ------------------- | -------------------- |
| GCN         | 92.00±0.03          | 75.64±0.21           |
| GraphSAGE   | 92.24±0.07          | 78.50±0.14           |
| GraphSAINT  | 92.52±0.13          | 80.27±0.26           |
| GBP         | 92.45±0.03          | 80.48±0.05           |
| SIGN        | 92.99±0.04          | 80.52±0.16           |
| DeeperGCN   | 92.38±0.09          | 80.98±0.20           |
| SAGN        | 93.09±0.04          | 81.20±0.07           |
| UniMP       | 93.08±0.17          | 82.56±0.31           |
| GAMLP (JK)  | 93.25±0.04          | 83.54±0.25           |
| GAMLP \(R\) | 93.11±0.05          | 83.59±0.09           |

The trend of validation and test accuracy reported in Table 5 is not consistent. But this phenomenon is quite common due to the split of datasets. For example, in Table 2, SIGN [1] has higher validation accuracy than DeeperGCN [2] but lower test accuracy. Also, the SAGN [3] has similar validation accuracy with UniMP [4] but much lower test Accuracy. 

[1] Frasca, Fabrizio, et al. [*Sign: Scalable inception graph neural networks.*](https://arxiv.org/pdf/2004.11198) arXiv preprint arXiv:2004.11198 (2020).

[2] Li, Guohao, et al. [*Deepergcn: All you need to train deeper gcns.*](https://arxiv.org/pdf/2006.07739) arXiv preprint arXiv:2006.07739 (2020). 

[3] Sun, Chuxiong, and Guoshi Wu. [*Scalable and Adaptive Graph Neural Networks with Self-Label-Enhanced training.*](https://arxiv.org/pdf/2104.09376) arXiv preprint arXiv:2104.09376 (2021).

[4] Shi, Yunsheng, et al. [*Masked label prediction: Unified message passing model for semi-supervised classification.*](https://arxiv.org/pdf/2009.03509) arXiv preprint arXiv:2009.03509 (2020).


We are very glad to respond if you have any new questions.

Respectfully,

Paper610 Authors

 ### Q2. The novelty of the attention mechanisms
**Recursive attention:** To the best of our knowledge, we are the first to propose the recursive attention mechanism, which can capture the information gain compared with the previous combined feature.

**JK attention:** The JK attention mechanism in GAMLP is motivated by JK-Net, but their usage of the concatenated feature is totally different. The concatenated feature in JK-Net is directly applied for the downstream node class prediction, while it is just used as a reference vector for the attention-based combination branch in GAMLP.



### Q3. Why choose the `cos(...)` formula?
Our first choice for the $\small \alpha_l$ in the last residual connection module is $\small \alpha_l=\frac{L-l}{L}$. However, we find that GAMLP still encounters the **over-fitting** issue on some datasets. Thus, we instead choose $\small \alpha_l=\cos(\frac{\pi l}{2L})$ to give more penalties to labels at large propagation steps.

We provide the performance comparison on the ogbn-products dataset in the table below. Three weighting schemes for the last residual connection module are tested: "Cosine function" stands for $\small \alpha_l=\cos(\frac{\pi l}{2L})$, the one in GAMLP; "Linear-decreasing weight" stands for $\small \alpha_l=\frac{L-l}{L}$; and "Fixed weight" stands for $\small \alpha_l=0.7$. The table shows that the weighting scheme GAMLP adopts, $\small \alpha_l=\cos(\frac{\pi l}{2L})$, outperforms the other two options.

| Choices                   | Test Accuracy |
| ------------------------ | ------------- |
| Fixed weight             | 82.56±0.43    |
| Linear-decreasing weight | 82.72±0.93    |
| Cosine function          | 83.59±0.05    |


### Q4. Differences between two attention mechanisms
The differences are as follows:

**D1:** JK attention has more **parameters** than recursive attention, and it has a better ability to fit complex data, but this ability also leads to the risk of **over-fitting**. Therefore, recursive attention can work better on data that is easy to be over-fitted.

**D2:** The **design principle** between these two mechanisms is totally different. JK-attention learns the importance of propagated features by comparing with the concatenated features at a different scale, while recursive attention only learns the feature information gain compared with the previous combined feature.


| Method    | ogbn-products | ogbn-mag    | ogbn-papers100M | Reddit     |
| --------- | ------------ | ---------- | --------------- | ---------- |
| GAMLP \(R\)  | 83.59±0.05   | 54.01±0.21 | 67.46±0.02      | 96.62±0.01 |
| GAMLP (JK) | 83.54±0.25   | 53.62±0.21 | 68.07±0.10      | 97.04±0.01 |

According to the No Free Lunch Theorem [1], there is no single best machine learning algorithm for predictive modeling problems such as classification and regression.
This is also the reason why GraphSAGE [2] has three candidate aggregator functions (i.e., Mean aggregator, LSTM aggregator, and Pooling aggregator).
Similarly, we propose two different attention mechanisms to adapt to the diverse needs of different datasets. The experimental results in the above table show that GAMLP (JK) works better than GAMLP \(R\) on the ogbn-papers100M and the Reddit datasets, and GAMLP \(R\) outperforms GAMLP (JK) on the ogbn-mag and the ogbn-products datasets. The evaluation results verify the necessity of proposing these two attention mechanisms at the same time.

[1] Wolpert, David H. [*The lack of a priori distinctions between learning algorithms.*](https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.390.9412&rep=rep1&type=pdf) Neural computation 8.7 (1996): 1341-1390.
[2] Hamilton, William L., Rex Ying, and Jure Leskovec. [*Inductive representation learning on large graphs.*](http://papers.nips.cc/paper/6703-inductive-representation-learningon-large-graphs.pdf) NeurIPS. 2017.

 We appreciate your assessment about "The technical quality of the paper is good." Thanks for your constructive feedback! We believe that addressing this feedback will make our paper stronger.

### Q1. Novelty and contribution of GAMLP
Our contributions mainly come from the following aspects: 
(1) We propose to consider the **node-wise feature propagation** in scalable GNNs. 
Details: Previous scalable (decoupled) GNNs adopt layer-wise propagation (e.g.,[1]) or graph-wise propagation (e.g.,[2]), thus each node in these methods cannot have a personalized combination of different steps of the propagated features.

(2) We combine multi-scale feature and label knowledge and learn to capture the underlying correlations between different scales of knowledge with two **attention mechanisms**.
Details: Several attempts are trying to apply the attention mechanisms in GNNs [3][4][5][8]. However, most of them are either not scalable [3][4] or not in a node-wise manner [5]. The attention mechanism in [8] shares many similarities with the one in GAMLP, but it cannot exploit the relationship between features at different propagation steps like GAMLP does. 

(3) To the best of our knowledge, we are the first to propose **node-wise label propagation** in GNN. 
Details: Despite that incorporating label information is not new in GNN [6][7][8], GAMLP is the first to take advantage of label information at different propagation steps to the best of our knowledge.


(4) We propose the **last residual connection module** to tackle the label leakage problem in label utilization.
Details:  With the aid of the last residual connections, GAMLP can apply the attention mechanism to the labels from propagation step $\small 0$ to $\small K$. This achievement is not minor since directly applying attention mechanisms to labels from propagation step $\small 0$ to $\small K$ would lead to serious **over-fitting**.

**Differences between GAMLP and GBP**: (1) Node-wise propagation in GAMLP vs. layer-wise propagation in GBP; (2) To combine multi-scale node features, GAMLP adopts the learnable node adaptive weights, while GBP uses the heuristic and fixed weights.

We have also done experiments over GBP and GAMLP without label part model, which is shown below. The experimental results show that GAMLP can significantly outperform GBP even without the label part, which verifies the advantage of its adaptive and node-wise propagation.

| Method               | PPI   | Reddit |
| -------------------- | ----- | ------ |
| GAMLP (JK) w/o label | 99.6% | 99.3%  |
| GBP                  | 96.9% | 94.1%  |


**Differences between GAMLP and UniMP**: 
(1) The label usage in UniMP is coupled with the training process, making it hard to scale to large graphs. While GAMLP decouples the label usage from the training process, the label propagation process can be executed as preprocessing. (2) The label propagation steps in UniMP are restricted to the same number of model layers. And if the number of model layers becomes large, UniMP will encounter the efficiency and scalability issues even on relatively small graphs. While the label propagation steps in GAMLP can be quite large since the label propagation is performed as preprocessing. (3) Both UniMP and GAMLP propose approaches to fight against the label leakage issue. However, the random masking in UniMP has to be executed in each training epoch, while the last residual connection (composed of simple matrix addition) in GAMLP just needs to be executed once during preprocessing. Thus, UniMP consumes more resources than GAMLP to fight the label leakage issue.

[1] Chen, Ming, et al. [*Scalable Graph Neural Networks via Bidirectional Propagation.*](https://arxiv.org/abs/2010.15421) NeurIPS. 2020.

[2] Wu, Felix, et al. [*Simplifying graph convolutional networks.*](http://proceedings.mlr.press/v97/wu19e/wu19e.pdf) ICML. 2019.

[3] Veličković, Petar, et al. [*Graph Attention Networks.*](https://openreview.net/pdf?id=rJXMpikCZ) ICLR. 2018.

[4] Liu, Meng, Hongyang Gao, and Shuiwang Ji. [*Towards deeper graph neural networks.*](https://dl.acm.org/doi/pdf/10.1145/3394486.3403076) SIGKDD. 2020.

[5] Chien, Eli, et al. [*Adaptive Universal Generalized PageRank Graph Neural Network.*](https://openreview.net/pdf?id=n6jl7fLxrP) ICLR. 2020.

[6] Wang, Hongwei, and Jure Leskovec. [*Unifying graph convolutional neural networks and label propagation.*](https://arxiv.org/pdf/2002.06755) arXiv preprint arXiv:2002.06755 (2020).

[7] Shi, Yunsheng, et al. [*Masked label prediction: Unified message passing model for semi-supervised classification.*](https://arxiv.org/pdf/2009.03509) arXiv preprint arXiv:2009.03509 (2020).

[8] Sun, Chuxiong, and Guoshi Wu. [*Scalable and Adaptive Graph Neural Networks with Self-Label-Enhanced training.*](https://arxiv.org/pdf/2104.09376) arXiv preprint arXiv:2104.09376 (2021).

This paper proposes the scalable GAMLP architecture that can learn node-adaptive features and can utilize label propagation to improve the model performance. Comprehensive experiments are provided to demonstrate the effectiveness of GAMLP, discuss the importance of each component in GAMLP, and compare its efficiency against some classic and SOTA baseline models. Strengths:
1. The writing of this paper is very clear, the organization is easy to follow and the main ideas are delivered clearly.
2. I appreciate the comprehensive experiments. 
3. The model performance is impressive. 

===

Weakness：
1. I feel there are some similarities between this paper and the work of SLE [1]. According to my understanding, both papers try to improve the SIGN model [2] by introducing a node-wise attention mechanism and utilizing label propagation to further enhance the performance. I am wondering, what is the key difference between GAMLP and SLE, and what are the advantages of GAMLP compared to SLE?
2. I am also interested in the performance comparison between GAMLP and SLE.
3. For the ablation study (section 4.3) Q2 (2), the authors compared the attention mechanism with different reference vectors. However, I am also interested to see how can the proposed attention mechanism affect performance?  For example, what would happen if each Wk in eq(3) is a learnable scaler? or what would happen if directly concatenate [X^(0)...X^(k)]?

===

[1] Sun, Chuxiong, and Guoshi Wu. "Scalable and Adaptive Graph Neural Networks with Self-Label-Enhanced training." arXiv preprint arXiv:2104.09376 (2021).
[2] Frasca, Fabrizio, et al. "Sign: Scalable inception graph neural networks." arXiv preprint arXiv:2004.11198 (2020). My major concern is the novelty. 

The paper proposes a simplified graph attention model that operates in two stages: first label propagation of the labels and features and then learning MLP on obtained predictions. The experiments show improvement on 12 graph classification datasets.  The paper proposes an end-to-end model that first propagates separately node features and labels, combines them with attention weights, aggregates embeddings and trains MLP model. For node-adaptive attention mechanism the authors propose two schemes: Recursive attention and JK attention. Experiments clearly show state-of-the-art performance. 

Despite its advantages, I see the following limitations of this work: 
1) Limited novelty. All the blocks used in the model were presented in the past and showed superior performance. Hence it's expected to see superior performance when combining those. 

2. Figure 1 used to discuss limitations of the existing approaches is not plotted for the proposed approach. Does GAMLP solves the aforementioned limitations? 

3.Cora/citeseer/pubmed datasets are quite small to motivate such an approach. While there are OGB datasets, all of these data are homophily-based. It would be good to compare the model on heterophily-based data [1] (where LP may fail) and on data with heterogeneous node features [2] (where graph-agnostic MLP and GBM can lead to better results). It's also interesting to compare with solely MLP baseline and GBDT baseline [2]

[1] New Benchmarks for Learning on Non-Homophilous Graphs https://arxiv.org/abs/2104.01404

[2] Boost then Convolve: Gradient Boosting Meets Graph Neural Networks https://arxiv.org/abs/2101.08543 

 The paper is well-written and attains good performance metrics; however, contributions are limited and are composed of previously studied ideas. To improve the paper I would seek a good application of such model besides the standard node classification (e.g. such applications that require simplicity of ML models). 

Meta-review: The paper proposes an improvement to graph based neural network, by improving their attention mechanism (introducing recursive attention and jumping knowledge attention) to flexibly attend to its neighborhood. The paper shows solid experimental results over competitive baselines, as acknowledged by reviewers. The reviewers agree that the paper is clearly written, but overall have issues with the novelty of the approach. The paper combines multiple components (last residual connection module, improved attention mechanism) to show gains, but none of the pieces are very new.

Decision: Reject