Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper studies the generalization problem of GNNs. The authors analyze the problem that the generalization of GNNs will be hindered by entangled representations, as well as the correlation between causal and bias variables. An edge masker is proposed to extract causal and bias substructures, which are further utilized to construct counterfactual unbiased samples and help GNN to capture causal relations. Experiments aim to validate the method in terms of debiasing, transferability, and interpretability.
strengths_and_weaknesses: Strengths
1. The research problem of the generalization of GNNs is interesting to the community.
2. The preliminary study and analysis (Section 3) further clarify the motivations of this paper.
3. The experiments show the effectiveness of the method on three datasets, including quantitative and qualitative evaluations.

Weaknesses
1. Line 73 " To our knowledge, we first study the generalization problem of GNNs in a more challenging yet practical scenario, i.e., the graphs are with severe bias." is overclaimed to me. Some works are overlooked[1-4].
2. It seems that the paper differs from DIR[2] only in some minor details: it changes the cross-entropy to generalized cross-entropy, etc. Why does DIR perform so poorly in Table 1? Why is DIR missing in Table 2? 
3. The experiment evaluations miss some necessary parts. The used datasets are all super-pixel datasets converted from images, but graph benchmarks on generalization are totally ignored in the experiments (such as OGB, etc.). And the hyperparameter analysis is also missing, which is also important since some hyperparameters could influence the performance significantly.
4. The related works about disentangled graph learning are missing.
5.  The main idea seems similar to DIR[2]. The authors are expected to clarify and highlight the difference between DIR and this paper. 
[1]Zhu, Qi, et al. "Shift-robust gnns: Overcoming the limitations of localized graph training data." 
[2]Wu, Ying-Xin, et al. "Discovering invariant rationales for graph neural networks." 
[3]Fan, Shaohua, et al. "Debiased graph neural networks with agnostic label selection bias." 
[4]Zhang, Shengyu, et al. "Stable Prediction on Graphs with Agnostic Distribution Shift." 

questions: Please kindly see the weaknesses above.
limitations: N/A


After Rebuttal Update: My rating will not change.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: In this work, the authors propose a disentangled GNN framework to learn the causal substructure and bias substructure for generalization. The proposed edge mask generator is to explicitly split the input graph into causal and bias parts. Based on the disentangled representations, the counterfactual unbiased training samples are used to decorrelate causal and bias variables. The authors conduct some experiments on some datasets.
strengths_and_weaknesses: In this work, the authors propose a disentangled GNN framework to learn the causal substructure and bias substructure for generalization. The proposed edge mask generator is to explicitly split the input graph into causal and bias parts. Based on the disentangled representations, the counterfactual unbiased training samples are used to decorrelate causal and bias variables. The authors conduct some experiments on some datasets. 

This work has the following strengths:
1) This work takes the SCM to model the data generation process and focuses on splitting the graph into causal and bias subgraphs. This pipeline can benefit the model interpretability.
2) The model details and framework are present in Figures.
3) The improvements shown in the experiments on some datasets are good.

However, I have the following concerns:
1) The novelty of this paper is so limited, which is the extension of existing works (such as DIR [1]). The technical contributions are straightforward. The problem analysis in section 3.2 is somehow similar to the analysis in [1] (see its appendix C Theory).  Some key part of the designed model has been proposed in DIR [1].  Although they improve upon the CE loss of DIR, the contributions are limited.
2) The expressions are confusing to the readers, and should be carefully revised. The authors propose to learn disentangled causal substructure but it is not explained why the learned representations are disentangled, according to the definition in [2].
3) The proposed method heavily relies on accurately identifying the causal and bias subgraphs at the beginning of the training process, which is hard to be satisfied in practice. 
4) The experimental evaluations are not convincing enough to support their claims. For example, some important baselines (such as top-k Pool, SAG Pool, etc.). How to quantitatively verify the learned causal subgraphs have causal relations with the labels? The evaluation datasets are also too weak. 
5) The authors claim that the proposed method can not only improve generalization ability but enjoy inherent interpretability, robustness, and transferability. But the experiments are also not enough to support it.
6) Some important literatures such as disentangled graph learning are ignored in the related works and experiments. I sincerely suggest the authors add them into the comparisons.
[1] Discovering Invariant Rationales for Graph Neural Networks.
[2] Towards a Definition of Disentangled Representations.

questions: Besides the concerns above, I still have some questions?
1) How about the results when using GAT as the GNN model besides GCN and GIN?
2) Why randomly permuting the latent vectors from different graphs can guarantee to generate unbiased counterfactual samples?

limitations: N/A
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper first studies the debiasing problem on GNNs, especially in the severe bias scenarios. They first conduct an investigation study to show than when the bias degree becomes larger, the bias information will dominate the prediction. In these scenarios, existing GNNs will has very poor generalization ability. The paper analyzes the reasons that lead to the failure by a causal graph. This causal graph also inspires two ways to debias GNNs: disentanglement and decorrelation. Based on this high-level idea, the authors propose a GNN disentanglement framework to disentangle the casual and bias substructures. From the extensive experiments on the newly constructed benchmarks, the proposed method outperforms baselines with a large margin. And the proposed model also has very good interpretability, robustness and transferability.
strengths_and_weaknesses: Strengths:
- Originality: The paper first studies the severe bias problem on graphs, which aims to investigate whether the GNNs are vulnerable to the spurious correlation. The paper motivates the proposed method well, both in experimental investigation and causal view. The proposed method could disentangle the desired subgraph well and has clear interpretability owing to the learned edge mask. Moreover, the paper conducts three benchmarks for the direction of study, which have controllable bias degrees and are easier to visualize and explain. All these contributions are original and significant.
-Quality: The studied problem is important, which makes the community being aware of the fundamental learning mechanism of GNN. Can we make more reliable GNNs that utilize human-understandable knowledge to make predictions? The proposed method is solid, which has clear motivation and effective techniques to achieve the goal. The proposed benchmark also has high-quality, which could be standard benchmarks in this direction.
-Clarity: The presentation of this paper is very clear and easy to follow. The contribution of this paper could be easily identified. The paper also provides the code and data to reproduce the experiments.
- Significance: In my opinion, the paper makes significant contributions on problem, technique and benchmark perspectives.

Weaknesses:
1. The presentation can be further improved. 
2. It seems that some related works are missing in Section 2, e.g. [1], 
[1] Li H, Wang X, Zhang Z, et al. Ood-gnn: Out-of-distribution generalized graph neural network[J]. arXiv preprint arXiv:2112.03806, 2021.

questions: I am curious about why the GNNs incline to utilize spurious correlation to make prediction rather than causal substructure. Although you have said that bias substructure has simpler structure and is easier to learn, it will be also interesting to conduct more theoretical studies in the GNN learning mechanism view in the further work.
limitations: The authors point out some limitations (e.g., more fine-grained splitting) and some societal impacts (e.g., the importance of stable model on safety-critical fields). The paper seldom introduces the negative social impacts. And I also could not foresee any negative aspects.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper pays attention on generalization problem of GNNs on the datasets with severe bias. From their investigation study, an interesting phenomenon shows that the GNNs prefer to utilize spurious correlation to make prediction rather than intrinsic causal information. Based on this observation, the authors analyze the bias problem in causal view and propose the debiasing strategy with the d-connection theory. The proposed framework is reasonable to solve the studied problem, which disentangles the bias and causal substructures by two well-designed functional GNN modules. To benchmark this new problems, three explainable and bias-controllable datasets are constructed. The proposed method significantly outperforms baselines. Meanwhile, the extensive qualitative evaluation could also validate that the proposed method has transparent interpretability.
strengths_and_weaknesses: The paper’s strengths and weaknesses are as follows:
Strengths:
1. The paper proposes an interesting problem, i.e., generalizing GNNs on graphs with severe bias, and also provides an interesting investigation study to show that the spurious correlation will dominate the prediction.
2. The authors utilize SCM to formulate the studied problem. And based on the proposed causal graph, it clearly motivates the methods that make the GNNs get rid of spurious correlation. They first design a parameterized edge mask generator to split the original graph into causal and bias subgraphs. And two GNN modules supervised by bias-aware and causal-aware loss respectively are developed to encode corresponding subgraphs into embeddings. With the learned embeddings, the model further decorrelates the causal and bias embedding by randomly swapping them. Overall, the proposed method is motivated well and technique sound.
3. The experiments are extensive. The paper conducts comprehensive quantitative and qualitative evaluation to validate the effectiveness and various properties of the proposed method. From the results, we can see that the proposed framework could disentangle original bias graph well, resulting in superior performance. Moreover, due to the learned mask, the proposed has appealing interpretability and transferability.
4. The new proposed datasets have various good properties, such as controllable bias degree, easy to explain and visualize. And the proposed benchmarks are desired for the community. The paper also provides the details to reproduce the experiments.
5. Overall, the paper is well-written and easy to follow.

Weaknesses:
Generally, I cannot find major weakness for this paper. This study looks novel in GNN community. I think some parts of model design is inspired by other work from image community, e.g., [20], which may influence its novelty. However, the overall framework is very effective and explainable for the graph data compared with general debiasing method.

questions: 1. Recently proposed method, DIR, seems to achieve unsatisfied results on the proposed benchmarks. Could you explain why?
2. The model obtains subgraph by learning the edge mask and do not delete the nodes based on the learned edge weights. I think the model could further remove the bias by deleting the bias nodes.

limitations: The proposed method may be limited to the problem they study, as they design the model based on the observation from experiments. The model could be further improved by tackling more types of bias, e.g., structure bias.
review_title: null

Meta-review: metareview: The paper proposes a GNN framework where the causal substructure and the bias substructure are disentangled by a mask generator and separate GNNs.  SOTA results on artificially generated severely biased data are reported.

Reviewers raised concerns mainly on the novelty, e.g., from [2, 20], insufficient experiments, missing baselines, and experiments only on artificially generated data based on images (not really graph data).  The authors addressed those concerns mostly well.

Although two reviewers kept their rejecting scores, they didn't further raise criticisms after rebuttal, and I find no good reason for rejection from their reviews.  

My concern that remains after the rebuttal is about the data.  The authors argue that existing benchmark graph datasets don't have severe bias, which is why they generated artificially biased data by using image data, which should not necessarily be treated as graphs.  This makes me wonder if the proposed method is really useful in practice.  Namely, are there some application scenarios where severe bias is expected on graph data?   If so, readers can expect that the authors would prepare real-world graph data that show severe bias (without manipulation), on which the proposed method outperforms the SOTA methods.  Can readers expect this in the author's near future follow-up work?  I strongly recommend the authors to discuss this point in the final version.

confidence: Less certain
Decision: Accept