Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: 
This paper observed interesting phenomenons: feature aggregation helps smooth out abnormal features but would lead to over-smoothing for normal features, residual link helps adjust feature smoothness for normal features but may hurt the performance when we have abnormal features. Therefore, it points out that we need to tradeoff between normal and abnormal features while designing GNNs. Then, based on the observation, this work proposed the AMP method and the AirGNN model, which are shown by experiments to be effective under various abnormal feature scenarios.


main_review: 
This paper provides very interesting observations and finds the tradeoff problem between normal and abnormal feature scenarios, the observations and the understandings (in section 2.2) make sense to me, and the results show in figure 5, figure 12(a)-16(a), are impressive.

However, I have the following questions/concerns:

1. What is the training objective function? Is it just Eq.(7) or Eq.(7) +classification loss?

2. Cora, Citeseer, and Pubmed are very classic datasets but they are not enough. It would be better if the authors can provide experimental results on other datasets in the main paper, instead of providing them in the appendix.

3. The author claimed that, in real-world applications, node features in graphs could often be abnormal such as being naturally noisy, then could you please provide some experimental results on some abnormal real-world datasets? For now, it seems to me that the authors are manually making the features in the rea-world dataset abnormal by replacing the raw features with Gaussian-generated features, so it is more like a synthetic scenario instead of a real-world scenario.

4. For figure 6, since they are on normal features, then why would the x-axis be "Ratio of Noisy Nodes(%)"? Why would the "Ratio of Noisy Nodes" influence the performance of GNNs with normal features? According to my understanding, noisy feature = abnormal features, am I misunderstanding?

5. I am curious, now you're using one Gaussian distribution to generate noisy features, what if the noise is generated in a more complicated way? (for example, what if different node class has different Gaussian distribution generation?)

6. I think the writing is not clear enough for me. 



limitations_and_societal_impact: 
The limitation of this work is not discussed in the paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper targets to design GNNs with stronger resilience to abnormal node features. Empirical examination is firstly conducted by replacing the features of randomly selected nodes with random Gaussian noise. The comparison between models with and without the residual connections show the helpfulness of residual connections on learning with abnormal features. 
main_review: Although the designed model is simply reasonable, the missing of key related work in discussion and in evaluation comparison makes it hard to know how the proposed solution is effective.  The related work discussion is short and missing the closely related work that also addresses abnormal or noisy feature problems in GNN. The baselines include only the representative GNNs, not taking into account the related work of other robust GNN models. 
For example:
Robust Graph Convolutional Networks Against Adversarial Attacks  in KDD 2019
GNNGuard: Defending Graph Neural Networks against Adversarial Attacks in NeurIPS 2020
And many other GNN defense models listed in Table 3 in paper entitled “Adversarial Attack and Defense on Graph Data: A Survey”.

limitations_and_societal_impact: Not applicable
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper discusses how message passing and feature aggregation are affected by abnormal node features, and designs a message passing module called Adaptive Message Passing (AMP) with adaptive residual connection and feature aggregation. In this way, the module can adaptively deal with the node that is inconsistent with the local feature during the message passing. The paper proposes the AirGNN that is built up with AMP as the message-passing layer in GNN, the experiment result shows that the AirGNN gets a more robust result on both the abnormal node and normal node classification comparing with the baselines.
main_review: The paper gives a solution for dealing with the abnormal node feature in the graph. The authors first give an observation of how the residual connection and number of layers affect the models' performance when the layer grows. Then the author gives the explanation and their solutions to balance the feature aggregation and residual connections affect during the message passing. The experiment result shows their advantages on various datasets comparing with the baselines.

The paper writes in good order, the discussion at the beginning of the paper helps a lot for me to understand the motivation of the paper. The authors give a straightforward solution by designing an adaptive module to control the feature aggregation and residual connection to deal with the abnormal feature in the graph. I am not sure if it is a novel method but seems reasonable to me. 

For the experiment part, the authors compared their method with GCN, GAT, APPNP, and GCNII and evaluate the performance on both normal and abnormal features. From the plot, the AirGNN consistently outperforms other baselines when an abnormal feature exists and increases. Overall, the experiment shows the model's advantage properly.

My questions and concerns are the follows:

1. Authors use a single noise setting in the whole paper, I think for this setting, the noise part is an important thing to understand, however, authors do not even write down the detailed noise setting ( what is the mean and var of the noise distribution). I am expecting to see a careful ablation study towards the noise to see if the method is robust to detect variations of the noise feature.

2. I see from the appendix that for other datasets like Coauthor Physics, Amazon photo, obg-arxiv, the AirGNN is pretty close or even worse than other methods like APPNP on the normal features. Do you have any explanation about this? I not confident for the statement that the model can deal with the real-world Graph with abnormal features. Combining with the first suggestions, it is necessary to see if the proposed model performs well when the noise setting changes.

3. I am not an expert in this area, but it seems that from the related works and the baseline, the authors do not mention some competing ''Robust GNN'' baselines.  I believe there should have some adversarial GNN paper that studies the noise feature during the training process. 

limitations_and_societal_impact: The authors do not give a section that discusses the limitations and potential negative societal impact. However, since the author stresses that the abnormal node feature exists a lot in the real-world dataset, I may expect the author has a brief discussion on this aspect.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper examines the behavior of GNNs when node features are perturbed during test time.  AirGNN is proposed based on the observations from the standard GNNs. AirGNN adaptively computes the weight of the initial embedding and aggregated embeddings and is robust to random/adversarial noise keeping the standard classification performance.
main_review: # Strengthes

* The proposed method is well motivated (Sections 2.1, 2.2, and 3.1). These sections argue that deep GNNs without residual connections are too smooth, whereas GNNs with residual connections can rely on noisy features. The proposed method tries to select the better scenario adaptively. This is reasonable.
* Section 3.3 provides interpretations of the proposed method. These explanations are intuitive and helpful for understanding.

# Weaknesses

* Many robust GNNs have been proposed [A, B]. These methods are not discussed or compared with the proposed method. I would like to see qualitative discussions and quantitative comparisons with them.
* The proposed method is a variant of APPNP. The proposed method adopts the L21 norm instead of the Frobenius norm. It is important to make more exhaustive comparisons with APPNP. See also the questions below.

# Questions

* Figure 1 (a) says APPNP wo/Res achieves accuracy 0.6 on abnormal features, but Figure 5 (a) says APPNP's accuracy is around 0.2. Why does this happen? Is APPNP w/Res used in Figure 5 (a)? (as a result of hyperparameter tuning?) I would like to see the performance of APPNP wo/Res in all settings. As far as I understand, AirGNN should perform worse on abnormal features than APPNP w/Res while it should perform better on normal features. This should be clarified in the experiments.
* See also Limitations And Societal Impact below.

## Reference

* [A] Robust Graph Convolutional Networks Against Adversarial Attacks. KDD 2019.
* [B] Adversarial attack and defense on graph data: A survey. https://arxiv.org/abs/1812.10528
limitations_and_societal_impact: No limitations are discussed in the manuscript. It is important to discuss the cons against APPNP as the proposed method is a variant of it. What information is lost using the L21 norm instead of L2? As the proposed method is related to adversarial attacks, the societal impact can also be discussed. Is the proposed method reliable enough to deploy in the real world?
review_title: null

Meta-review: The paper discovers the phenomenon that residual connections can amplify GNNs' vulnerability against abnormal node features. It analyzes possible reasons and based on the understanding designs an effective message passing scheme to help solve the issue. 

The observed phenomenon is interesting and important for related applications, and the study and design are reasonable with impressive results. The good response from the authors has clarified several concerns of the reviewers (better presentation, more thorough experiments, comparison with related work etc). The clarification in the response should be incorporated into the final version if accepted.  
Decision: Accept (Poster)