Debiasing Graph Neural Networks via Learning Disentangled Causal Substructure
Authors: Shaohua Fan
Xiao Wang
Yanhu Mo
Chuan Shi
Jian Tang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Graph Neural Networks
Debiasing
Causal substructure

Abstract: Most Graph Neural Networks (GNNs) predict the labels of unseen graphs by learning the correlation between the input graphs and labels. However, by presenting a graph classification investigation on the training graphs with severe bias, surprisingly, we discover that GNNs always tend to explore the spurious correlations to make decision, even if the causal correlation always exists. This implies that existing GNNs trained on such biased datasets will suffer from poor generalization capability.  By analyzing this problem in a causal view, we find that disentangling and decorrelating the causal and bias latent variables from the biased graphs are both crucial for debiasing. Inspired by this, we propose a general disentangled GNN framework to learn the causal substructure and bias substructure, respectively. Particularly,  we design a parameterized edge mask generator to explicitly split the input graph into causal and bias subgraphs. Then two GNN modules supervised by causal/bias-aware loss functions respectively are trained to encode causal and bias subgraphs into their corresponding representations. With the disentangled representations, we synthesize the counterfactual unbiased training samples to further decorrelate causal and bias variables. Moreover, to better benchmark the severe bias problem, we construct three new graph datasets, which have controllable bias degrees and are easier to visualize and explain. Experimental results well demonstrate that our approach achieves superior generalization performance over existing baselines. Furthermore, owing to the learned edge mask, the proposed model has appealing interpretability and transferability.

Debiasing Graph Neural Networks via Learning Disentangled Causal Substructure
Shaohua Fan 1.24 1 Xiao Wang's Yanhu Mo', Chuan Shi"; Jian Tang
2
Beijing University of Posts and Telccommunications, China Mila Québec AI Institute, Canada
HEC Montréal, Canada
CIFAR Al Research Chair
(fanshaohua, xiaowang, moyanhu, shi Hbbptt.edpppppoe edu. cn, jian. tang@hec.ca
Abstract
Most Graph Neural Networks (GNNs) predict the labels of unseen graphs by learn- ing the correlation between the input graphs and labels. However, by presenting graph classification investigation prisingly, we discover that GNNs always tend to explore the spurious correlations make decision, even if the causal correlation always exists. This implies that existing GNNs trained on such biased datasets will suffer from poor generalization capability. By analyzing this problem and decorrelating the causal and bias latent variables from the biased graphs are both crucial for debiasing. Inspired by this, we propose a general disentangled GNN framework to learn the causal substructure and bias substructure, respectively. Particularly, we design a parameterized edge mask generator to explicitly split the input graph into causal and bias subgraphs. Then two GNN modules supervised by causal/bias-aware loss functions respectively are trained to encode causal and bias subgraphs into their corresponding representations. With the disentangled representations, we synthesize the counterfactual unbiased training samples to further decorrelate causal and bias variables. Moreover, to better benchmark the severe bias problem, we construct three new graph datasets, which have control- lable bias degrees and easier visualize and explain. Experimental results well demonstrate that our approach achieves superior generalization performance over existing baselines. Furthermore, owing to the learned edge mask, the proposed model has appealing interpretability and transferability!
the training graphs with severe bias, sur-
causal view, we find that disentangling
1 Introduction
Graph Neural Networks (GNNs) have exhibited powerful performance on graph data with various applications [T 35 g. One major category applications are the graph classification task. 20 44), superpixel graph classification [14], and well known that graph classification is usually
such molecular graph property prediction [15]
social network category classification [46] 44
determined by relevant substructure, but not the whole graph structure 4 26 45). For example, for MNIST superpixel graph classification task, the digit subgraphs are causal (i.c., deterministic) for
labels [3)] The mutagenic property of molecular graph depends the functional groups (i.e.,
"This work done when the first author visiting student at Mila.
Corresponding authors.
Code and data are available at: http://ggilbbbooogggggabbbbbbbi
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

nitrogen dioxide (NO2)), rather than the irrelevant patterns (i.e., carbon rings) [27] Therefore, it is a fundamental requirement for GNNs to identify causal substructures, so as to make correct prediction.
Ideally, when the graphs are unbiased, i.e., only the causal substructures are related with the graph labels, the GNNs are able to utilize such substructure to predict the labels. However, due to the uncontrollable data collection process, the graphs are inevitably biased, i.e., existing meaningless substructures spuriously correlates with labels. Taking colored MNIST superpixel graph dataset in Sec. 3.I as an example (illustrated in Fig. [ each category of digit subgraphs mainly correspond to one kind of color background subgraphs, e.g.. digit subgraph is related with red background subgraph. Therefore, the color background subgraph will be treated as bias information, which highly correlates with labels but does not determines them in the training set. Under this situation, will GNNs still stably utilize the causal substructure to make decision?
To investigate the impact of bias on GNNs, we conduct an experimental investigation to demonstrate the impact of bias (especially in the severe bias scenarios) on the generalization capability of GNNs (.ec.B.I). We find that GNNs actually utilize both bias and causal substructures make prediction. However, with severer bias correlation, even bias substructure still could not exactly determine labels like causal substructure, GNNs majorly utilize bias substructure as shorteuts make prediction, causing a large generalization performance degradation. Why this happens? We analyze the data- generating process and model prediction mechanism behind the graph classification using a causal graph (Sec. 3.2). The casual graph illustrates that the observed graphs are generated by the causal and bias latent variables and existing GNNs could not distinguish the causal substructure from entangled graphs. How can we disentangle the causal and bias substructures from observed graphs, so that GNNs can only utilize the causal substructures to make stable prediction when severe bias appears? ?
To address the question, two challenges need to be faced. 1) How to identify the causal substructure and bias substructure in the severe biased graphs? In the severe bias scenarios, bias substructure will be "easier to learn" for GNNs and finally dominate the prediction. Using the normal cross-entropy loss, like DIR 339 could not fully capture such aggressive property of bias. 2) How to extract the causal substructure from an entangled graph? The statistically causal substructure is usually determined by the global property of the entire graph population, rather than single graph. When extracting causal substructure from a graph, we need to establish the relations among all the graphs.
In this paper, we propose novel debiasing framework for GNNs via learning Disentangled Causal substructure, called DisC. Given an input biased graph, we propose to explicitly filter edges into causal and bias subgraphs by parameterized edge mask generator, whose parameters are shared across entire graph population. As a result, the edge masker is naturally capable to specify the importance for each edge and extract causal and bias subgraphs from a global view of the entire observations. Then, a "casual" -aware (weighted cross-entropy) loss and a cross-entropy) loss are respectively utilized to supervise two functional GNN modules. Based on the supervision, the edge mask generator could generate corresponding subgraphs and the GNNs could encode corresponding subgraphs into their disentangled embeddings. With the disentangled embeddings, we randomly permute the latent vectors extracted from different graphs to generate more unbiased counterfactual samples in embedding space. The new generated samples still contain both causal and bias information, while their correlation has been decorrelated. In this time, there is only correlation between causal variables with labels, so that the model could concentrate on the true correlation between the causal subgraphs and labels. Our major contributions are as follows:
(generalized
To our knowledge, we first study the generalization problem of GNNs in a more challenging yet practical scenario. i.c., the graphs are with severe bias. We systematically analyze the bias impact on GNNs from both experimental study and causal analysis. We find that the bias substructure, compared with causal substructure, is much easier to dominate the training of GNNs.
To debias GNNs, we develop a novel GNN framework for disentangling causal substructure, which is flexible to build upon various GNNs for improving generalization ability while enjoying inherent interpretability, robustness and transferability.
We construct three new datasets with various properties and controllable bias degrees, which can better benchmark the new problem. Our model outperforms the corresponding base models with a large margin (from 4.47% to 169.17% average improvements). Various investigation studies demonstrate that our model could discover and leverage causal substructure for prediction.

---

2 Related Works
Generalization of GNNs in wild environments. Most existing GNN methods are proposed under the IID hypothesis. i.e. training and testing set are independently sampled from the identical distribution (34 17 35
However, reality, thus ideal assumption is hard to be satisfied. Recently, several methods have been proposed to improve the generalization ability of GNNs in wild OOD environments. Several works [29 7 38) study the OOD problem of node classification. For OOD graph classification task, StableGNN 66] propose to learn the stable causal relationship in graphs. OOD-GNN [22] propose to constrain each dimension of learned embedding to be independent. DIR [39] discovers the invariant rationales for generalizing GNNs. Although they have achieved better OOD performance, they are not designed for the datasets with severe bias, which more challenging for guaranteeing the generalization ability of GNNs.
Disentangled graph neural networks. Recently, there are a couple of methods that study the disentangled GNNs. DisenGCN (2 utilizes neighbourhood routing mechanism to divide the neighbours of the node into several mutually exclusive parts. IPGDN [25] promotes DisenGCN by constraining the different parts of the embedding feature to be independent. DisenGCN and IPGDN are node-level disentanglement thus FactorGCN [42] considers the whole graph information and disentangles the target graph into several factorized graphs. Despite results of the previous works, they do not consider disentangling the causal and bias information for graphs.
General debiasing methods. Recently, debiasing problem has drawn much attention in machine learning community [16] 23
33
2
11. One category of these methods pre-defining a certain
bias type explicitly to mitigate [6
23
33 37 For example, Wang et al. [37] and Bahng et
al. [ design a texture- and color-guided model to adversarially train a debiased neural network against the biased one. Instead of defining certain types of bias, recent approaches [30] S 21 rely on the straightforward assumption that models are prone to exploit the bias as shortcuts to make prediction [10]. In the line with the recent studies, our study belongs to the second category. However, most of existing methods are designed for image datasets and could not effectively extract causal substructure from graph data. Distinctly, first study the severe bias problem on graph data, and our method could effectively extract causal substructure from graph data.
Origiel
2
3
4
images
60
Trxeforne
40
621
20
Jarind
Rasel beceg
sebgraii
sabgragi
3
0.5
0.25
D3es Degree
0.95
(a) Examples of graphs in CMNIST-75sp.
(b) Performance of GNNs.
Figure 1: Example graphs of CMNIST-75sp and the performance of GNNs on this dataset.
3
Preliminary Study and Analysis
In this section, we first illustrate the existing GNNs tend to exploit the bias substructure as shorteuts for prediction through motivating experiment. Then analyze the prediction process of GNNs in causal view. Based on this causal view, it motivates our solution to relieve the impact of bias.
3.1 Motivating Example
To measure the generalization ability of GNNs with the effect of bias, we construct a graph classifica- tion dataset with controllable bias degrees, called CMNIST-75sp. We first construct a biased MNIST image dataset like []. where each category of digit highly correlates with a pre-defined color their background. For example, in the training set, 90% of 0 digits are with red background (i.e., biased samples), and remaining 10% images are with random background color (i.e., unbiased samples), whose the bias degree is 0.9 in this situation. We consider four bias degrees {0.8, 0.85, 0.9, 0.95).
3

---

For the testing set, we construct both biased testing set and unbiased testing set. The biased testing set has the same bias degree with training set, aiming to measure the extent of models relying on bias. The unbiased testing set, where the digit labels uncorrelate with the background colors, aims to test whether the model could utilize the inherent digit signals for prediction. Note that training set and testing set have the same pre-defined color set. Then, we convert the biased MNIST images into superpixel graphs with at most 75 nodes each graph using [18), where the edges are constructed by the KNN method based on the 2D coordinates of superpixels and node features are the concatenation of coordinates and average color of superpixels. Each graph is labeled by its digit class, so that its digital subgraph is deterministic for label and background subgraph is spuriously correlated with labels but not deterministic. The examples of graphs are illustrated in Fig. (aa
We perform three popular GNN methods: GCN 17, GIN 41 and GCNII (3] on CMNIST-75sp
and the results are shown in Fig- [(b] The same color of dashed line and solid line represent the results of the corresponding methods on the biased testing set and the unbiased testing set respectively. Overall, the GNNs achieve much better performance on biased testing set than unbiased testing set. The phenomenon indicates that although GNNs could still learn some causal signals for prediction, the unexpected bias information is also being utilized for prediction. More specifically, with bias degree becoming larger, the performance of GNNs on biased testing set is increased and the value of accuracy is nearly in line with the bias degree, while the performance on unbiased testing drops dramatically. Hence, although causal substructure could determine labels perfectly. in severe bias scenarios, the GNNs lean to utilize the easier to learn bias information to make prediction rather than the inherent causal signals, and bias substructure will finally dominate the prediction.
B
E
B
Y
G
(a) SCM of the union of the (b) SCM of our debiasing GNN data generation and the existing method.
GNNs' prediction process.
Figure 2: SCMs. Grey and white variables represent unobserved and observed variables, respectively.
3.2 Problem Analysis
Debiasing GNNs for unbiased prediction requires understanding the natural mechanisms of graph classification task. We present causal view of the union of the data-generating process and the model prediction process behind the task. Here we formalize the causal view as a Structure Causal Model (SCM) causal graph 31) by inspecting
the causalities among five variables: unobserved causal variable C, unobserved bias variable B, observed graph G, graph embedding E, and ground truth label / prediction Fig. 2(a) illustrates the SCM, where each link denotes a causal relationship.
C G B. The observed graph data is generated by two unobserved latent variables: the causal variable C' and the bias variable B, such as digit subgraphs and background subgraphs in the CMNIST-75sp dataset. And all bellow relations are illustrated by CMNiST-75sp.
Y This link means that the causal variable is the only endogenous parent to determine the generation of ground-truth label Y. For example, C' is the oracle digit subgraph, which exactly explains why the label is labeled as Y.
C' +--+ B. This link indicates the spurious correlation between C' and B. Such probabilistic dependencies is usually caused by the direct cause unobserved confounder [32]. Here do not distinguish these scenarios and only observe the spurious correlation between B and C, such the spurious correlation between the color background subgraphs and digit subgraphs.
'We use variable Y for both the ground truth labels and prediction, as they optimized to be the same.

---

- Existing GNNs usually learn the graph embedding E' based on the observed graph Gand make the prediction based the learned embedding E.
According to the SCM, GNNs will utilize both information to make prediction. As bias substructure (e.g., background subgraph) usually has simpler structure than meaningful causal substructure (e.g., digit subgraph), if GNN utilizes such simple substructure, it could achieve low loss very fast. Hence, GNN inclines to utilizes bias information when most graphs are biased. Based on the SCM in Fig. 2(a) according d-connection theory [1 (see App. A): two variables are dependent if they connected by at least one unblocked path, we could find two paths that would induce the spurious correlation between the bias variable B and label Y: (1) B G E Y and (2) B - C Y. To make the prediction Y being uncorrelated with the bias B, we need to intercept the two unblocked paths. For this purpose, we propose debias GNNs in causal view, in Fig. 2(b)
C-G - B and C - To intercept the path (I), we should disentangle the latent variables C and B from the observed graph G and make prediction only based on the causal variable C.
C + B. To intercept the path (2), as we cannot change the link between C' and Y. one possible solution is to make C' and B uncorrelated.
4 Methodology
Motivated by the above causal analysis, in this section, we present our proposed debiasing GNN framework DisC, remove the spurious correlation. The overall framework is shown in Fig. First, an edge mask generator is learnt to mask the edges of original input graphs into causal subgraphs and bias subgraphs. Second, two separate GNN modules with their corresponding masked subgraphs are trained to encode corresponding causal substructure and bias substructure into disentangled representations, respectively. Last, after the disentangled representations are well-trained, we permute the bias representations among the training graphs to generate counterfactual unbiased samples, so that the correlation between causal representations and bias representations is removed.
Caucal GXKgle
Z Zuebeore
C2
W
Suge Malk
Mis buth Grais
C
Locg
CNN
Tedonad Floe
Figure 3: The overall framework of DisC.
4.1 Causal and Bias Substructure Generator
Given mini- batch of biased graphs g = (G,,..., G. our idea is that: we take a collection of graph instances and design a generative probabilistic model to learn mask the edges into causal subgraph or bias subgraph. Particularly, given a graph G (A, X), where A is the adjacency matrix and X is the node feature matrix, utilize multi-layer perceptron (MLP) upon the concatenation of node features x, of node i and X, of node j to measure the importance of edge (i,j) for causal subgraph:
aij MLP([x,,;;]]
(1)
Then a sigmoid function o(.) is employed to project a; into the range of (0,1), which indicates the probability of edge (i,j) being the edge in the causal subgraph as follows:
Cjj (ass).
(2)
5

---

Naturally, we could get the probability of edge (i.j) being the edge in the bias subgraph by: b; 1- Cig. Now we could construct the causal edge mask M. = [Gy]] and bias edge mask M, [bby]] Finally, we decompose the original graph G into causal subgraph G. = (M, o A, X} and bias subgraph G2 (M, @ A,X), X}. Intuitively, the edge mask could highlight different part of structure information of original graphs, thus GNNs built on the different subgraphs could encode different parts of graph information. Moreover, the mask generator has two advantages. (1) Global view: In individual graph level, the mask generator (i.e., MLP), whose parameters are shared by all the edges in graph, take a global view of all the edges a graph, which enables us to identify community in graph. It is well known that the effect of edge cannot be judged independently, because edges usually collaborate with each other, forming a community, to make prediction. Thus, it is critical to evaluate edge in global view. In whole graph population level, the mask generator takes a global view of all the graphs in the training set, which enables us to identify causal/bias subgraph. Particularly,
the causal/bias is the statistical information in the population level, it is necessary to view all the graphs to identify the causal/bias substructure. Considering both such coalition effects and population-level statistical information, the generator is able measure the importance of edges more accurately. (2) Generalization: The mask generator can generalize the mechanism of mask generation to new graphs without retraining, so capable and efficient to prune unseen graphs
4.2 Learning Disentangled Graph Representations
Given Ge and Gs. how to ensure they are causal subgraph and bias subgraph, respectively? Inspired by [2I) our approach simultaneously trains a pair of GNNs (9b.9e) with linear classifiers (Ca: C.) as follows: (1) Motivated by the observation in Sec. [3.1 that bias substructure is easier to learn, we utilize a bias-aware loss to train a bias GNN 9b and a bias classifier C, and (2) in contrast, we train a causal GNN Ie and causal classifier Ce on the training graphs that the bias GNN struggles to learn. Next, we would present each component in detail.
As shown in Fig. 3 GNN 9r. and 9h embed the corresponding subgraphs into causal embedding 2, = (G and bias embedding 2 = gb GGp 7/)), respectively, where 7 is the parameters of GNNs. Subsequently, concatenated vector 2 [2ci 26] is fed into linear classifiers C. and C. to predict the target label y. To train 9. and C2 as bias extractor, we utilize the generalized cross entropy (GCE) 47 loss to amplify the bias of the bias GNN and classifier:
1 - C²¹² (2)
CCE(C,(=;(:).))
(3)
g
where Cz(z;a)) and Ci 04p) are softmax output of the bias classifier and its probability belonging to the target category y. respectively, and a is the parameters of classifier. Here E (0. 1] is a hyperparameter that controls the degree of amplifying bias. Given 02, = [Tb, ay the gradient of the GCE loss up-weights the gradient of the standard cross entropy (CE) loss for the samples with a high confidence C₁ of predicting the correct target category as follows:
(GCE(C,(Z;;)) a0,
( @CE(C)(2;
(4)
00, Therefore, compared with CE loss, GCE loss will amplify the gradients of 0, on samples by the confidence score (² Based on our observation that the bias information is usually easier to be learned, so the biased graphs will have higher ( than unbiased graphs. Therefore, the model 9b and C trained by GCE loss will focus on bias information and finally get the bias subgraph. Note that, to ensure that Cs predicts target labels mainly based on this 2y, the loss from C2 is not backpropagated to ge, i.e., only update 0, in Eq. 4. and vice versa.
Meanwhile, we also train a causal GNN simultancously with the weighted CE loss. The graphs with high CE loss from Co can be regarded as the unbiased samples compared with the samples with low CE loss. In this regard, we could obtain the unbias score of each graph as CF(CA(=).v)
W(2) CE(C.(2).v) + CC(()(2)))))
(5)
Large value of W implies the graph is an unbiased sample, hence we could use these weights to reweight the loss of these graphs to train g. and C., enforcing them to learn the unbiased information. Thus, the objective function for learning disentangled representation is:
LD = (<=)CE(C.(=) v) + GCE(C,(=).
(6)

---

4.3 Counterfactual Unbiased Sample Generation
Until now, we have achieved the first goal analyzed in Sec. 3.2 that is the disentanglement of causal and bias substructures, Next, we will show how to achieve the second goal that makes the causal variable 2g and bias variable 2, uncorrelated. Although we have disentangled causal and bias information, they are disentangled from the biased observed graphs. Hence, there will exist statistical correlation between causal and bias variables inheriting from the biased observed graphs. To further decorrelate be and =g. according to the causal relation of data-generating process: C - G < B, we propose to generate the counterfactual unbiased samples in embedding space by swapping 26- More specifically, we randomly permute bias vectors in each mini-batch and obtain Funbiased [[zi &.), where & represents the randomly permuted bias vectors of žg. As &c and &. in Funbiased are randomly combined from different graphs, they will have much less correlation than = [ =6. where both are from the same graph. To make 9b and C still focus on the bias information, we also swap label as I along with 2, so that the spurious correlation between &, and still exists. With the generated unbiased samples, we utilize the following loss function to train two GNN modules:
Lg W(2)CE(CC Zumbiasea) y) + GCE Ci Punbiosea): .)),
Together with the disentanglement loss, total loss function is defined as:
L = LD + XgLG,
(8)
where ÅG is a hyperparameter for weighting the importance of generation component. Moreover, training with more diverse samples would also benefit with better generalization unseen testing scenarios. Our approach is summarized in App. B. Note that, as we need well-disentangled represen- tations to generate the high-quality unbiased samples, in the early stage of training, we only train the model with Lp- After certain epochs, we train the model with T..
5 Experiment
Datasets. We construct three datasets with various properties and bias ratios benchmark this new problem, where the datasets have clear causal subgraphs making the results explainable. Follow- ing CMNIST-75sp introduced in Sec. [.1] we use the similar way to construct CFashion-75sp and CKuzushiji-75sp datasets based on the Fashion-MNIST [40] and Kuzushiji-MNIS,
(4] datasets. As the causal subgraphs of these two datasets are more complicated (fashion product and hiragana char- acters), they are more challenging. Due to the page limits, here we set bias degrees as {0.8, 0.9, 0.95). We report the main results on unbiased test sets. Details are in App. C.I.
Baselines and experimental setup. As DisC is a general framework which could be built on various base GNN models, we select three popular GNNs: GCN (17), GIN 41), and GCNII K The corresponding models are termed as DisCGGN. DISCGIN and DisCGCNII- respectively. Hence, base models are the most straight basclines. Another kind of baselines are the causal-inspired GNN method DIR (39) and StableGNN (6. We also compare against general debiasing method LDD (11 by replacing its encoder with GNNs. Graph Pooling method DiftPool 44] and graph disentangling method FactorGCN [42 are also compared. To keep fair comparison, our model uses the same GNN architecture and hyperparameters with the corresponding base model. All the experiments are run times with different random seeds and we report the accuracy and the standard error. More details are in App. C.2.
5.1 Quantitative Evaluation
Main results. The overall results are summarized in Table I and we have following observations:
(1)
DisC has much better generalization ability than base models. DisC outperforms the corre- sponding base model consistently with a large margin. With heavier biases, our model achieves larger improvements over base models. Specifically, for CMNIST-75sp, CFashion- 75ssh and CKuzushiji-75sp with smaller bias degree (i.e., 0.8), our model achieves 40.02%. 4.47% and 29.82% average improvements over corresponding base models, respectively. Surprisingly, with severer biases (0.9 and 0.95), DisC achieves 169.175 14.67% 77% and 49.35% average improvements over base models on three datasets, respectively. It indicates that the proposed method is a general framework helping existing GNNs against the negative impact of bias.
7

---

(2) DisC significantly outperforms existing debiasing methods. We notice that DIR could not achieve satisfying results. The reason is that DIR utilizes CE loss extract bias information, which could not fully capture the property of bias in severe bias scenarios. And DIR sets one fixed threshold to spilt subgraphs, which is suboptimal. StableGNN outperforms their hase model DiffPool and achieve competitive results, indicating the effectiveness of their proposed causal variable distinguishing regularizer. However, their framework adjusts data distribution based on the original dataset, it is hard to generate unbiased distribution when the unbiased samples are scarce. DisC could generate more unbiased samples based on the disentangled representations. Moreover, LDD is a general debiasing method which is not designed for graph data. DisC outperforms corresponding LDD variants with average 23.15%, indicating that the seamless joint of clobbal-oopulation awwre edge masker with debiasing disentangle framework is very effective.
Table 1: Graph classification accuracy evaluated on unbiased testing sets, which have same color (bias) set with training set. The best performance within each base model variant is in bold.
Dalavel Bias
CMNIST-75pp
CFashion-75pp 25kp
CKuzushij--SS 7Sp
0.8
0.9
0.95
0.8
0.9
0.95
0.8
0.9
0.95
FactorCCNN
12.
02.35
42.502491
61.20_1.111
$1.500 20
45.7 40
42.87g119
52.86
27.87
DifiPool [44]
70.794g.08
68.45.p77
47.12.1.04
62.52-0.35
57.50.3.9
50.880 20.86
45.46-0.a
36.15.0.19
27.45-
DD 159
10:0310.r
13.122
12.80
10.35_0.2
10.72.0.27
10.59
StabloGNN
77.65.2.64
08.57
51.35
64 0..08-0.2
58 58.201.0.0
11.46.n.3
49. 19.41-900
39.30.0..2
28.254
LDD 21
64.95g1.99
56.65
63.55_1.17
64.30
62.28 62.28
42.98.0.3
38.76.0 m
33.08 03.08,
LDDorN 21
61.8821.44
50.592110
31 31.28
6L.6520.s
57. 57.
53. 38..5
37 14.8520.4
28.97
22. 23.13
78.03ga.00
00.50g0
51.06ga
30 30.63_1.7
54.09g254
57.90 57.90gn
48. 48.70g1 as
41.50g1
33.93
GCN
17
50.43+4.15
28.977444
13.50
63 03.00
57.22240.00
38.46-111
28,35-0.79
20.71 20.70
DisCoroy
82 62.60.0.s
78 24.14
63.477.5.5
66.85..1.1
65 45.0.70
63 98
48 13:2.5
36 6S
GIN
31.75ge
30.75g5.5
16.0111114
64,25_n.10
58.05g0.00
19. 19.74
41.82
50.Mg 57
11.18
DeCCory
82.10g1.00 so
74.90-1.11
58.58.424
67.10....
90,1.1
85.80=a0 au
56.18,100
41.75
30.25-e1.5 us
GONIT [
00.70.1.71
G7 G7.68....6
41.00/ars
66 68.68.00.5
53 18
48 48.53-0.25
36.25.0.20
25,60 25.00+a0m0
DisCgener
79.500,3.1
76.00_1m
60.5445.31
66.47 1.37
65 .0.r0
61. o7
54.90 1.30
44.73 44.73
36 96
Ablation studies. To validate the importance of each module in our method, in Fig. 4 we conduct ablation studies on our variants (w.o. G means without the sample generation module) and the related variants of LDD. The major difference between DisC/w.o. G with LDD /w.o. G is the edge mask module. most cases, DisC/w.o. o. G significantly outperforms LDD /w.o. G, indicating the necessity of learning edge mask for graph data. And DisC which has counterfactual sample generation module could further boost the performances based on the disentangled embeddings of DisC/w.o. G. However, LDD seldomly outperforms LDD /w.o. G or even achieves worse performances. That is, generating high-quality counterfactual samples needs well-disentangled causal and bias embeddings. If embeddings are not well-disentangled, counterfactual samples may act as noisy samples, which would prevent models from achieving further improvement. The edge masker could help the model generate well-disentangled embeddings, which is crucial for overall performance.
90
80
60
LEDIWO D DVC
LOO
Do DEC
LOON
DsLMaa
Accuracy
60
LOD
Accuracy
70
LDO
Accuracy
50
LDO
DEC
70
60
40
60
50
50
30
40
GCN
GCNII
40
GCN
GCNII
20
GCN
GCNII
Base model
Base model
Base madel
(a) CMNIST-75sp
(b) CFashion- 755p
(c) CKuzushiji-75pp
Figure 4: Ablation studies of the DisC vs. LDD average over three bias degrees of each dataset.
Robustness unseen bias. Table 2 reports the results of DisC compared with its corresponding base models on testing set with unseen bias, i.e., the pre-defined color (bias) sets of training set and testing set are disjoint. The performances of base models further drop compared with the results on seen bias scenario in Table I However, our model still achieves very stable performances, fully demonstrating the generalization ability of our model agnostic bias scenario.
Hyperparameter experiments Fig-5 is the hyperparameter experiments of the degree of amplifying bias in GCE loss and the importance of generation component Xç. For q. we fix AG = 10 and vary q from (0.1, 0.3, 0.5, 0.7, 0.9). For Ag, we fix q 0.7 and vary AG from {1,5, 10, 15). From the results, we can that our model achieves stable performance across different values of q and
8

---

Table 2: The results on unseen unbiased testing sets, i.c., the color has not been seen in training set.
Dataset
CMNIST-75sp
CFashion-7555 75sp
CKuzushiji-55p
Bias
0.8
0.9
0.95
0.8
0,95
0.9
0.95
DIR
10.38.0.25
10.14.0.40
9.77
16.7741.7
16.51.3.20
12.69-1.61
10.48-0.2
10.33
10.59-a.96
GCN
36.88+5.15
23.07=4.07
11.88-0.38
69.33
53.6540.47
45.600-..06
36.35-n.48
27.88.0.44
19.95-0.07
DisCoos
82.73:1a1
77.70 s7
65 o0
67 41.4
68.2840.18
63.77+1
57.80
51. 41.60=0.41
41 04
GIN
48.9342.90
34.95-0.56
14.53-0.97
+8.58
53.80.0.22
48 48.43.0.m
39.25-a.5r
30.75+1.4
22.35-n.a6
DisCouN
77.80 73.00
58 80 66
67 67.15
59. 59.98
51.70
55.47
43. 23.2061136
31 33
GCNII
53.50+6.25
45.62:2.26
32.0.5.c0
58,8541.89
46.97 53.98
53.
39.93_a.as
30.33
23.09-1.a
DisCOoNII
79.65
76.65 76.63
663g..3
38 60 60.00gg..
o6
60.50
63.051222
61.78g1 60
56 23g2 45
49.
49.10g2.0s 02
41.052o.11
Àg- When q = 0.1, it means the GCE loss will nearly reduce to normal CE loss. We can see the performance of DisCoeN is worse than other scenarios, demonstrating the effectiveness of utilizing GCE loss.
90
90
Ao
Accuracy
Accuracy
70-
0.1
0.3
0.5
0.7
0.9
10
15
AG
(a) q
(b) À
Figure 5: The hyperparameter experiments of q and ÀG
5.2 Qualitative Evaluation
Visualization of edge mask. To better illustrate the significant causal and bias subgraphs extracted by DisCGeN. we visualize the original images, original graph. and corresponding causal subgraph and bias subgraph of CMNIST-75sp with 0.9 bias degree in Fig. where the width of edge represents the value of learned weight Gj or bag- Fig. ((a) shows the visualization results of testing graphs with the bias (color) that has been seen in the training set. As we can see, our model could discover the causal subgraphs where the most salient edges are in the digital subgraphs. With these causal subgraphs that highlight the structure information of digital, the GNNs will more easily extract this causal information. Fig. bb(b) shows the visualization results of testing graphs with unseen bias. According to the visualization, our model could still discover the causal subgraph outline, indicating our model could recognize causal subgraphs, whether the bias is seen or unseen. The visualization results of CFashion-75sp and CKuzushiji-75y are shown in App. D.
Original image Original graph Causal subgraph Biased subgraph Original image Otiginal graph Causal subgraph Blased sabgraph
1
2
S
(n) Testing graphs with hias in training
(b) Testing graphs with bias training
Figure 6: Visualization of subgraphs extracted by DisC. The width of edge is edge weight Cıj bij-
Projection of disentangled representation. Fig. 7 shows the projection of latent vectors 2, and 2b extracted from the causal GNN ge and bias GNN go of DisCOON. respectively, using t-SNE [19 on

---

(a) E. labeled by digit.
(b) Ze labeled by color.
(c) 2,, labeled by digit.
(d) 2, labeled by color.
Figure 7: Visualization of 2, and 2 with colors labeled by the digit and bias (color) labels. We observe
that 2, and 2n are well clustered according to the groundtruth labels and bias labels, respectively.
CMNIST-75sp. Fig. (a-b) are the projections of ze labeled by the target labels (digit) and bias labels (color), respectively, Fig. (c-d) are the projections of 24 labeled by the target labels and bias labels, respectively. We observe that & are clustered according to the target labels while 24 are clustered with the bias labels. And &e are mixed with bias labels and 2 are mixed with target labels. The results indicate that DisC successfully learns the disentangled causal and bias representations.
Transferability of the learned mask. As our model could extract GNN-inde ONN-independent subgraphs, the learning edge weights can be used to purify original biased graphs. These sparse subgraphs represent sig- nificant semantic information and can be universally transferred to any GNNs. To validate this point, we learn the edge mask by DisCGGN and prune the edges with least (0%, 20%, 40%, 60%} weights while keeping the remaining edge weights. Then train vanilla GIN and GCNII on these weighted prumed datasets. Fig. is the comparison of the results, where the dashed lines represent the results of base model on original biased graphs and the solid lines represent the perfor- mance of GNNs weighted pruned datasets. The results show that the GNNs trained on the pruned datasets achieve better performances, indicating our learned edge mask has considerable transferability.
100
Figure 8: Performance of GIN and GCNII on the weighted pruned graphs found by DisCGeN-
Conclusion
In this paper, are first to study the generalization problem of GNNs on severe bias datasets, which is crucial to study the transparently knowledge learning mechanism of GNNs. We analyze the problem in causal view that the generalization of GNNs will be hindered by entangled representations as well as the correlation between causal and bias variables. To remove the impact from these two aspects, we propose general disentangling framework, DisC, which extracts causal substructure and bias substructure by two different functional GNNs, respectively. After the representations are well-disentangled, we proliferate the counterfactual unbiased samples by randomly swapping the disentangled vectors. With the new constructed benchmarks, we clearly validate the effectiveness, robustness, interpretability, and transferability of our method.
Acknowledgments and Disclosure of Funding
This work is supported part by the National Natural Science Foundation of China (No. U20B2045, 62192784, 62172052, 62002029, 62172052, U1936014). This work is also partially supported by the Natural Sciences and Engineering Research Council (NSERC) Discovery Grant, the Canada CIFAR Al Chair Program, collaboration grants between Microsoft Research and Mila, Samsung Electronies Co., Ltd., Amazon Faculty Research Award, Tencent AI Lab Rhino-Bin Bird Gift Fund and NRC Collaborative R&D Project (A14D-CORE--6). This project was also partially funded by IVADO Fundamental Research Project grant PRF-2019- 3583139722 The work Shaohua Fan is supported by the China Scholarship Council (No.202006470078) The computation resource of this project is supported by Compute Canada
hhaps//www..comuuccandddu.
10

---

References
[1] Hyojin Bahng, Sanghyuk Chun, Sangdoo Yun, Jaegul Choo, and Seong Joon Oh, Learning de-biased representations with biased representations. In ICML, 2020.
[2] Remi Cadene, Corentin Dancette, Matthieu Cord, Devi Parikh, et al. Rubi: Reducing unimodal biases for visual question answering. NeurIPS, 2019.
[3] Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In ICML, 2020.
[4] Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint a12.0:181..01118. 2018.
[5] Luke Darlow, Stanistaw Jastrzębski, and Amos Storkey. Latent adversarial debiasing: Mitigating collider bias in deep neural networks. arXiv preprint arXiv:2011.11486 2020.
[6] Shaohua Fan, Xiao Wang, Chuan Shi, Peng Cui, and Bai Wang. Generalizing graph neural networks on out-of-distribution graphs. In arXiv preprint arXiv:2111. 2021.
[7] Shaohua Fan, Xiao Wang, Chuan Shi, Kun Kuang, Nian Liu, and Bai Wang. Debiased graph neural networks with agnostic label selection bias. 7NNLS, 2022.
[8] Shaohua Fan, Xiao Wang, Chuan Shi, Emiao Lu, Ken Lin, and Bai Wang. One2multi graph autoencoder for multi-view graph clustering. WWW, pages 3070-3077 3076,07762200 2020.
[9] Shaohua Fan, Junxiong Zhu, Xiaotian Han, Chuan Shi, Linmei Hu, Biyu Ma, and Yongliang Li. Metapath-guided heterogeneous graph neural network for intent recommendation. In SIGKDD, pages 2478-2486, 2019.
[10] Robert Geirhos, Henn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wicland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intellignce, 2(1 (1166555567, 2020.
[11] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint 1rXiv:1811.122331 2018.
[12] Madelyn Glymour, Judea Pearl, and Nicholas Jewell. Causal inference in statistics: A primer. John Wiley & Sons, 2016.
[13] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NeurlPS, 2017.
[14] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.122 122661, 2019.
[15) Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In NeurIPS, 2020.
[16] Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Junmo Kim. Learning not to learn: Training deep neural networks with biased data. In CVPR, 2019.
[17] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2016.
[18] Boris Knyazev, Graham W Taylor, and Mohamed Amer. Understanding attention and general- ization in graph neural networks. In NeurIPS, 2019.
[19] G.E. Kvan der Maaten, L.J.P.; Hinton. Understanding attention and generalization in graph neural networks. Journal of Machine Learning Research, 2008
[20] John Boaz Lee, Ryan Rossi, and Xiangnan Kong. Graph classification using structural attention. In SIGKDD, 2018.

---

[21] Jungsoo Lee, Eungyeup Kim, Juyoung Lee, Jihyeon Lee, and Jaegul Choo. Learning debiased representation via disentangled feature augmentation. In NeurIPS, 2021.
[22] Haoyang Li, Xin Wang. Ziwei Zhang, and Wenwu Zhu. Ood-gnn: Out-of-distribution general- ized graph neural network. arXiv preprint arXiv:2112.03806, 2021.
[23] Yi Li and Nuno Vasconcelos. Repair: Removing representation bias by dataset resampling, In CVPR, 2019.
[24] Renjie Liao, Raquel Urtasun, and Richard Zemel. A pac-bayesian approach to generalization bounds for graph neural networks. In ICLR, 2020.
[25] Yanbei Liu, Xiao Wang. Shu Wu, and Zhitao Xiao. Independence promoted graph disentangled networks. AAAI, 2020.
[26] Ana Lucic, Maartje ter Hoeve, Gabriele Tolomei, Maarten de Rijke, and Fabrizio Silvestri. Ci-gnnexplainer: Counterfactual explanations for graph neural networks. In AISTATS, 2022.
[27] Dongsheng Luo, Wei Cheng, Dongkuan Xu, Wenchao Yu, Bo Zong, Haifeng Chen, and Xiang Zhang. Parameterized explainer for graph neural network. In NeurIPS, 2020.
[28] Jianxin Ma, Peng Cui, Kun Kuang, Xin Wang, and Wenwu Zhu. Disentangled graph convolu- tional networks. In ICML, pages 4212 -2212.2..222 PMLR, 2019.
[29] Jiaqi Ma, Junwei Deng, and Qiaozhu Mei. Subgroup generalization and fairness of graph neural networks. In NeurlPS, 2021.
[30] Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: De-biasing classifier from biased classifier. In NeurlPS, 2020.
[31] Judea Pearl. Causality. Cambridge university press, 2009.
[32] Hans Reichenbach. The direction of time, volume 65. Univ of California Press, 1991.
[33] Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks. In ICLR, 2019.
[34] Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neural Networks, 20(1):61-80, 2008.
[35] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In ICLR, 2017.
[36] Minh Vu and My T Thai. Pgm-explainer: Probabilistic graphical model explanations for graph neural networks. In NeurIPS, 2020.
[37] Haohan Wang, Zexue He, Zachary Lipton, and Eric P Xing. Learning robust representations by projecting superficial statistics out. arXiv preprint 1rXiv:1903.0620 2019.
[38] Qitian Wu, Hengrui Zhang, Junchi Yan, and David Wipf. Handling distribution shifts on graphs: An invariance perspective. In ICLR, 2022.
[39] Ying-Xin Wu, Xiang Wang, An Zhang, Xiangnan He, and Tat-Seng Chua. Discovering invariant rationales for graph neural networks. In ICLR, 2022.
[40] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: (00i:::700077447 2017.
[41] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In ICLR, 2019.
[42] Yiding Yang, Zunlei Feng, Mingli Song, and Xinchao Wang. Factorizable graph convolutional networks. NewrIPS, volume 33, pages 20286-20296, 2020.
[43] Rex Ying, Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovee, Gnnexplainer: Generating explanations for graph neural networks. In NeurlPS, 2019.
12

---

[44] Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hierarchical graph representation learning with differentiable pooling. In NeurIPS, volume 31, 2018.
[45] Hao Yuan, Haiyang Yu, Jie Wang, Kang Li, and Shuiwang Ji. On explainability of graph neural networks via subgraph explorations. ICML, 2021.
[46] Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen. An and-to-end deep learning architecture for graph classification. In AAAI, 2018.
[47] Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. NeurlPS, 2018.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See App.
(c) Did you discuss any potential negative societal impacts of your work? [No] We could not foresee any potential negative societal impacts of work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [Yes] See Sec. 3.2,
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] We provide the URL of code and data for reproducing the main results.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Sec. Scc.5land App. C.2.
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] See Sec.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Sec. 5)
4. If you are using existing assets (c.g., code, data, models) or curating/relcasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We construct new
assets based on existing assets. We have cited them in Sec.
(b) Did you mention the license of the assets? [Yes] See App. C.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] We release new assets through a URL in Sec.
(d) Did you discuss whether and how consent was obtained from people whose data you'le 're using/curating? [No] The source data for generating
data is publicly available.
(c) Did you discuss whether the data you are using/curating contains personally identifi- able information or offensive content? [No] We do not have personally identifiable information or offensive content.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "eGid1H2Z18v", "reviewer": "Reviewer_UrWX", "summary": "This paper first studies the debiasing problem on GNNs, especially in the severe bias scenarios. They first conduct an investigation study to show than when the bias degree becomes larger, the bias information will dominate the prediction. In these scenarios, existing GNNs will has very poor generalization ability. The paper analyzes the reasons that lead to the failure by a causal graph. This causal graph also inspires two ways to debias GNNs: disentanglement and decorrelation. Based on this high-level idea, the authors propose a GNN disentanglement framework to disentangle the casual and bias substructures. From the extensive experiments on the newly constructed benchmarks, the proposed method outperforms baselines with a large margin. And the proposed model also has very good interpretability, robustness and transferability.", "questions": "I am curious about why the GNNs incline to utilize spurious correlation to make prediction rather than causal substructure. Although you have said that bias substructure has simpler structure and is easier to learn, it will be also interesting to conduct more theoretical studies in the GNN learning mechanism view in the further work.", "limitations": "The authors point out some limitations (e.g., more fine-grained splitting) and some societal impacts (e.g., the importance of stable model on safety-critical fields). The paper seldom introduces the negative social impacts. And I also could not foresee any negative aspects.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Originality: The paper first studies the severe bias problem on graphs, which aims to investigate whether the GNNs are vulnerable to the spurious correlation. The paper motivates the proposed method well, both in experimental investigation and causal view. The proposed method could disentangle the desired subgraph well and has clear interpretability owing to the learned edge mask. Moreover, the paper conducts three benchmarks for the direction of study, which have controllable bias degrees and are easier to visualize and explain. All these contributions are original and significant.\n-Quality: The studied problem is important, which makes the community being aware of the fundamental learning mechanism of GNN. Can we make more reliable GNNs that utilize human-understandable knowledge to make predictions? The proposed method is solid, which has clear motivation and effective techniques to achieve the goal. The proposed benchmark also has high-quality, which could be standard benchmarks in this direction.\n-Clarity: The presentation of this paper is very clear and easy to follow. The contribution of this paper could be easily identified. The paper also provides the code and data to reproduce the experiments.\n- Significance: In my opinion, the paper makes significant contributions on problem, technique and benchmark perspectives.\n\nWeaknesses:\n1. The presentation can be further improved. \n2. It seems that some related works are missing in Section 2, e.g. [1], \n[1] Li H, Wang X, Zhang Z, et al. Ood-gnn: Out-of-distribution generalized graph neural network[J]. arXiv preprint arXiv:2112.03806, 2021.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "QMqQaiZNgFs", "reviewer": "Reviewer_JhdG", "summary": "In this work, the authors propose a disentangled GNN framework to learn the causal substructure and bias substructure for generalization. The proposed edge mask generator is to explicitly split the input graph into causal and bias parts. Based on the disentangled representations, the counterfactual unbiased training samples are used to decorrelate causal and bias variables. The authors conduct some experiments on some datasets.", "questions": "Besides the concerns above, I still have some questions?\n1) How about the results when using GAT as the GNN model besides GCN and GIN?\n2) Why randomly permuting the latent vectors from different graphs can guarantee to generate unbiased counterfactual samples?\n", "limitations": "N/A", "rating": 4, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "In this work, the authors propose a disentangled GNN framework to learn the causal substructure and bias substructure for generalization. The proposed edge mask generator is to explicitly split the input graph into causal and bias parts. Based on the disentangled representations, the counterfactual unbiased training samples are used to decorrelate causal and bias variables. The authors conduct some experiments on some datasets. \n\nThis work has the following strengths:\n1) This work takes the SCM to model the data generation process and focuses on splitting the graph into causal and bias subgraphs. This pipeline can benefit the model interpretability.\n2) The model details and framework are present in Figures.\n3) The improvements shown in the experiments on some datasets are good.\n\nHowever, I have the following concerns:\n1) The novelty of this paper is so limited, which is the extension of existing works (such as DIR [1]). The technical contributions are straightforward. The problem analysis in section 3.2 is somehow similar to the analysis in [1] (see its appendix C Theory).  Some key part of the designed model has been proposed in DIR [1].  Although they improve upon the CE loss of DIR, the contributions are limited.\n2) The expressions are confusing to the readers, and should be carefully revised. The authors propose to learn disentangled causal substructure but it is not explained why the learned representations are disentangled, according to the definition in [2].\n3) The proposed method heavily relies on accurately identifying the causal and bias subgraphs at the beginning of the training process, which is hard to be satisfied in practice. \n4) The experimental evaluations are not convincing enough to support their claims. For example, some important baselines (such as top-k Pool, SAG Pool, etc.). How to quantitatively verify the learned causal subgraphs have causal relations with the labels? The evaluation datasets are also too weak. \n5) The authors claim that the proposed method can not only improve generalization ability but enjoy inherent interpretability, robustness, and transferability. But the experiments are also not enough to support it.\n6) Some important literatures such as disentangled graph learning are ignored in the related works and experiments. I sincerely suggest the authors add them into the comparisons.\n[1] Discovering Invariant Rationales for Graph Neural Networks.\n[2] Towards a Definition of Disentangled Representations.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Q5UXFjom1h", "reviewer": "Reviewer_V4kV", "summary": "This paper pays attention on generalization problem of GNNs on the datasets with severe bias. From their investigation study, an interesting phenomenon shows that the GNNs prefer to utilize spurious correlation to make prediction rather than intrinsic causal information. Based on this observation, the authors analyze the bias problem in causal view and propose the debiasing strategy with the d-connection theory. The proposed framework is reasonable to solve the studied problem, which disentangles the bias and causal substructures by two well-designed functional GNN modules. To benchmark this new problems, three explainable and bias-controllable datasets are constructed. The proposed method significantly outperforms baselines. Meanwhile, the extensive qualitative evaluation could also validate that the proposed method has transparent interpretability.", "questions": "1. Recently proposed method, DIR, seems to achieve unsatisfied results on the proposed benchmarks. Could you explain why?\n2. The model obtains subgraph by learning the edge mask and do not delete the nodes based on the learned edge weights. I think the model could further remove the bias by deleting the bias nodes.\n", "limitations": "The proposed method may be limited to the problem they study, as they design the model based on the observation from experiments. The model could be further improved by tackling more types of bias, e.g., structure bias. ", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper’s strengths and weaknesses are as follows:\nStrengths:\n1. The paper proposes an interesting problem, i.e., generalizing GNNs on graphs with severe bias, and also provides an interesting investigation study to show that the spurious correlation will dominate the prediction.\n2. The authors utilize SCM to formulate the studied problem. And based on the proposed causal graph, it clearly motivates the methods that make the GNNs get rid of spurious correlation. They first design a parameterized edge mask generator to split the original graph into causal and bias subgraphs. And two GNN modules supervised by bias-aware and causal-aware loss respectively are developed to encode corresponding subgraphs into embeddings. With the learned embeddings, the model further decorrelates the causal and bias embedding by randomly swapping them. Overall, the proposed method is motivated well and technique sound.\n3. The experiments are extensive. The paper conducts comprehensive quantitative and qualitative evaluation to validate the effectiveness and various properties of the proposed method. From the results, we can see that the proposed framework could disentangle original bias graph well, resulting in superior performance. Moreover, due to the learned mask, the proposed has appealing interpretability and transferability.\n4. The new proposed datasets have various good properties, such as controllable bias degree, easy to explain and visualize. And the proposed benchmarks are desired for the community. The paper also provides the details to reproduce the experiments.\n5. Overall, the paper is well-written and easy to follow.\n\nWeaknesses:\nGenerally, I cannot find major weakness for this paper. This study looks novel in GNN community. I think some parts of model design is inspired by other work from image community, e.g., [20], which may influence its novelty. However, the overall framework is very effective and explainable for the graph data compared with general debiasing method.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JnB1yJiGhyU", "reviewer": "Reviewer_TFtC", "summary": "This paper studies the generalization problem of GNNs. The authors analyze the problem that the generalization of GNNs will be hindered by entangled representations, as well as the correlation between causal and bias variables. An edge masker is proposed to extract causal and bias substructures, which are further utilized to construct counterfactual unbiased samples and help GNN to capture causal relations. Experiments aim to validate the method in terms of debiasing, transferability, and interpretability.", "questions": "Please kindly see the weaknesses above.", "limitations": "N/A\n\n\nAfter Rebuttal Update: My rating will not change.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The research problem of the generalization of GNNs is interesting to the community.\n2. The preliminary study and analysis (Section 3) further clarify the motivations of this paper.\n3. The experiments show the effectiveness of the method on three datasets, including quantitative and qualitative evaluations.\n\nWeaknesses\n1. Line 73 \" To our knowledge, we first study the generalization problem of GNNs in a more challenging yet practical scenario, i.e., the graphs are with severe bias.\" is overclaimed to me. Some works are overlooked[1-4].\n2. It seems that the paper differs from DIR[2] only in some minor details: it changes the cross-entropy to generalized cross-entropy, etc. Why does DIR perform so poorly in Table 1? Why is DIR missing in Table 2? \n3. The experiment evaluations miss some necessary parts. The used datasets are all super-pixel datasets converted from images, but graph benchmarks on generalization are totally ignored in the experiments (such as OGB, etc.). And the hyperparameter analysis is also missing, which is also important since some hyperparameters could influence the performance significantly.\n4. The related works about disentangled graph learning are missing.\n5.  The main idea seems similar to DIR[2]. The authors are expected to clarify and highlight the difference between DIR and this paper. \n[1]Zhu, Qi, et al. \"Shift-robust gnns: Overcoming the limitations of localized graph training data.\" \n[2]Wu, Ying-Xin, et al. \"Discovering invariant rationales for graph neural networks.\" \n[3]Fan, Shaohua, et al. \"Debiased graph neural networks with agnostic label selection bias.\" \n[4]Zhang, Shengyu, et al. \"Stable Prediction on Graphs with Agnostic Distribution Shift.\" \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept