DISCOVERING INVARIANT RATIONALES FOR GRAPH NEURAL NETWORKS
Year: 2022

Abstract:
Intrinsic interpretability of graph neural networks (GNNs) is to find a small subset of the input graph’s features — rationale — which guides the model prediction. Unfortunately, the leading rationalization models often rely on data biases, especially shortcut features, to compose rationales and make predictions without probing the critical and causal patterns. Moreover, such data biases easily change outside the training distribution. As a result, these models suffer from a huge drop in interpretability and predictive performance on out-of-distribution data. In this work, we propose a new strategy of discovering invariant rationale (DIR) to construct intrinsically interpretable GNNs. It conducts interventions on the training distribution to create multiple interventional distributions. Then it approaches the causal rationales that are invariant across different distributions while filtering out the spurious patterns that are unstable. Experiments on both synthetic and realworld datasets validate the superiority of our DIR in terms of interpretability and generalization ability on graph classification over the leading baselines. Code and datasets are available at https://github.com/Wuyxin/DIR-GNN.

heading: 1 INTRODUCTION
text: The eye-catching success in graph neural networks (GNNs) (Hamilton et al., 2017; Kipf & Welling, 2017; Dwivedi et al., 2020) provokes the rationalization task, answering “What knowledge drives the model to make certain predictions?”. The goal of selective rationalization (aka. feature attribution) (Chang et al., 2020; Ying et al., 2019; Luo et al., 2020; Wang et al., 2021c) is to find a small subset of the input’s graph features — rationale — which best guides or explains the model prediction. Discovering the rationale in a model helps audit its inner workings and justify its predictions. Moreover, it has tremendous impacts on real-world applications, such as finding functional groups to shed light on protein structure prediction (Senior et al., 2020).
Two research lines of rationalization have recently emerged in GNNs. Post-hoc explainability (Ying et al., 2019; Luo et al., 2020; Yuan et al., 2021; Wang et al., 2021c) attributes a model’s prediction to the input graph with a separate explanation method, while intrinsic interpretability (Veličković et al., 2018; Gao & Ji, 2019) incorporates a rationalization module into the model to make transparent predictions. Here we focus on intrinsically interpretable GNNs. Among them, graph attention (Veličković et al., 2018) and pooling (Lee et al., 2019; Knyazev et al., 2019; Gao & Ji, 2019; Ranjan et al., 2020) operators prevail, which work as a computational block of a GNN to generate soft or hard masks on the input graph. They cast the learning paradigm of GNN as minimizing the empirical risk with the masked subgraphs, which are regarded as rationales to guide the model predictions.
Despite the appealing nature, recent studies (Chang et al., 2020; Knyazev et al., 2019) show that the current rationalization methods are prone to
∗Corresponding author.
exploit data biases as shortcuts to make predictions and compose rationales. Typically, shortcuts result from confounding factors, sampling biases, and artifacts in the training data. Considering Figure 1, when the most bases of House-motif graphs are Tree, a GNN does not need to learn the correct function to reach high accuracy for the motif type. Instead, it is much easier to learn from the statistical shortcuts linking the bases Tree with the most occurring motifs House. Unfortunately, when facing with out-of-distribution (OOD) data, such methods generalize poorly since the shortcuts are changed. Hence, such shortcut-involved rationales hardly reveal the truly critical subgraphs for the predicted labels, being at odds with the true reasoning process that underlies the task of interest (Teney et al., 2020) and human cognition (Alvarez-Melis & Jaakkola, 2017).
Here we ascribe the failure on OOD data to the inability to identify causal patterns, which are stable to distribution shift. Motivated by recent studies on invariant learning (IL) (Arjovsky et al., 2019; Krueger et al., 2021; Chang et al., 2020; Bühlmann, 2018), we premise different distributions elicit different environments of data generating process. We argue that the causal patterns to the labels remain stable across environments, while the relations between the shortcut patterns and the labels vary. Such environment-invariant patterns are more plausible and qualified as rationales.
Aiming to identify rationales that capture the environment-invariant causal patterns, we formalize a learning strategy, Discovering Invariant Rationales (DIR), for intrinsically interpretable GNNs. One major problem is how to get multiple environments from a standard training set. Differing from the heterogeneous setting (Bühlmann, 2018) of existing IL methods, where environments are observable and attainable, DIR does not assume prophets about environments. It instead generates distribution perturbations by causal intervention — interventional distributions (Tian et al., 2006; Pearl et al., 2016) — to instantiate environments and further distinguish the causal and non-causal parts.
Guided by this idea, our DIR strategy consists of four modules: a rationale generator, a distribution intervener, a feature encoder, two classifiers. Specifically, the rationale generator learns to split the input graph into causal and non-causal subgraphs, which are respectively encoded by the encoder into representations. Then, the distribution intervener conducts the causal interventions on the non-causal representations to create perturbed distributions, with which we can infer the invariant causal parts. Then, the two classifiers are respectively built upon the causal and non-causal parts to generate the joint prediction, whose invariant risk is minimized across different distributions. On one synthetic and three real datasets, extensive experiments demonstrate the generalization ability of DIR to surpass current state-of-the-art IL methods (Arjovsky et al., 2019; Krueger et al., 2021; Sagawa et al., 2019), and the interpretability of DIR to outperform the attention- and pooling-based rationalization methods (Veličković et al., 2018; Gao & Ji, 2019). Our main contributions are:
• We propose a novel invariant learning algorithm, DIR, for inherent interpretable models, improving the generalization ability and is suitable for any deep models. • We offer causality theoretic analysis to guarantee the preeminence of DIR. • We provide the implementation of DIR for graph classification tasks, which consistently achieves
excellent performance on three datasets with various generalization types.

heading: 2 INVARIANT RATIONALE DISCOVERY
text: With a causal look at the data-generating process, we formalize the principle of discovering invariant rationales, which guides our discovery strategy. Throughout the paper, upper-cased letters like G denote random variables, while lower-case letters like g denote deterministic value of variables.

heading: 2.1 CAUSAL VIEW OF DATA-GENERATING PROCESS
text: Generating rationales for transparent predictions requires understanding the actual mechanisms of the task of interest. Without loss of generality, we focus on the graph classification task and present a causal view of the data-generating process behind this task. Here we formalize the causal view as a Structure Causal Model (SCM) (Pearl et al., 2016; Pearl, 2000) by inspecting on the causalities among four variables: input graph G, ground-truth label Y , causal part C, non-causal part S. Figure 2a illustrates the SCM, where each link denotes a causal relationship between two variables.
• C → G ← S. The input graph G consists of two disjoint parts: the causal part C and the non-causal part S, such as the House motif and the Tree base in Figure 1.
Published as a conference paper at ICLR 2022
𝑺
𝒀
𝑮 𝑪
𝑮
𝑺
𝑪 𝒀 House Cycle Crane
House Cycle Crane
…
(b) Illustration of Constructing Interventional Distributions
House Cycle Crane
𝒅𝒐(𝑺 = ∅)
∅
𝒅𝒐(𝑺 = )
𝒅𝒐(𝑺 = )
…
…
…
𝑺
𝒀
𝑮 𝑪
Original Distribution
Multiple 𝒔-Interventional Distributions
(a) SCM
𝑺
𝒀
𝑮 𝑪
𝑮
𝑺
𝑪 𝒀 House
Cycle
Crane
House
Cycle
Crane
…
(b) Illustration of Constructing Interventional Distributions
House
Cycle
Crane
𝒅𝒐(𝑺 = ∅)
∅
𝒅𝒐(𝑺 = )
𝒅𝒐(𝑺 = )
…
…
…
𝑺
𝒀
𝑮 𝑪
Original Distribution
Multiple 𝒔-Interventional Distributions
(b) Interventional Distributions.
Figure 2: (a) Causal view of data-generating process; (b) Illustration of interventional distributions.
• C → Y . By “causal part”, we mean C is the only endogenous parent to determine the groundtruth label Y . Taking the motif-base example in Figure 1 again, C is the oracle rationale, which perfectly explains why the graph is labeled as Y .
• C L9999K S. This dashed arrow indicates additional probabilistic dependencies (Pearl, 2000; Pearl et al., 2016) between C and S. We consider three typical relationships here: (1) C is independent of S, i.e., C |= S; (2) C is the direct cause of S, i.e., C → S; and (3) There exists a common cause E, i.e., C ← E → S. See Appendix B for the corresponding examples.
C L9999K S can create spurious correlations between the non-causal part S and the ground-truth label Y . Assuming C → S, C is a confounder between S and Y , which opens a backdoor path S ← C → Y , thus making S and Y spuriously correlated (Pearl et al., 2016). We systematize such spurious correlations as Y 6⊥⊥ S. Wherein, we make feature induction assumption on S to avoid the confusion of the induced subset of S between C. See Appendix C for the formal assumption. Furthermore, data collected from different environments exhibit various spurious correlations (Teney et al., 2020; Arjovsky et al., 2019), e.g., one mostly picks House motifs with Tree bases as the training data, while another selects House motifs with Wheel bases as the testing data. Hence, such spurious correlations are unstable and variant across different distributions.

heading: 2.2 TASK FORMALIZATION OF INVARIANT RATIONALIZATION
text: Oracle Rationale. With the causal theory (Pearl et al., 2016; Pearl, 2000), for each variable X in a SCM, there exists a directed link from each of its parent variables PA(X) to X , if and only if the causal mechanism X = fX(PA(X), X) persists, where X |= PA(X) is the exogenous noise of X . For simplicity, we omit the exogenous noise and simplify it as X = fX(PA(X)). Hence, there exist a function fY : C → Y in our SCM, where the “oracle rationale” C satisfies:
Y = fY (C), Y |= S | C, (1) where Y |= S | C indicates that C shields Y from the influence of S, making the causal relationship C → Y invariant across different S. Rationalization. In general, only the pairs of input G and label Y are observed during training, while neither oracle rationale C nor oracle structural equation model fY is available. The absence of oracles calls for the study on intrinsic interpretability. We systematize an intrinsically-interpretable GNN as a combination of two modules, i.e., h = hŶ ◦ hC̃ , where hC̃ : G→ C̃ discovers rationale C̃ from the observed G, and hŶ : C̃ → Ŷ outputs the prediction Ŷ to approach Y . Distinct from C and Y which are the variables in the causal mechanisms, C̃ and Ŷ represent the variables in the modeling process to approximateC and Y . To optimize these modules, most of current intrinsicallyinterpretable GNNs (Veličković et al., 2018; Lee et al., 2019; Knyazev et al., 2019; Gao & Ji, 2019; Ranjan et al., 2020) adopt the learning strategy of minimizing the empirical risk:
min hC̃ ,hŶ R(hŶ ◦ hC̃(G), Y ), (2)
where R(·, ·) is the risk function, which can be the cross-entropy loss. Nevertheless, this learning strategy relies heavily on the statistical associations between the input features and labels, and can potentially exhibit non-causal rationales.
Invariant Rationalization. We ascribe the limitation to ignoring Y |= S | C in Equation 1, which is crucial to refine the causal relationship C → Y that is invariant across different S. By introducing this independence, we formalize the task of invariant rationalization as:
min hC̃ ,hŶ R(hŶ ◦ hC̃(G), Y ), s.t. Y |= S̃ | C̃, (3)
where S̃ = G \ C̃ is the complement of C̃. This formulation encourages the rationale C̃ seeking the patterns that are stable across different distributions, while discarding the unstable patterns.

heading: 2.3 PRINCIPLE & LEARNING STRATEGY OF DIR
text: Interventional Distribution. However, it is difficult to recover the oracle rationale from the joint distribution over the inputs and labels — that is, the causal and non-causal relations are hardly distinguished from each other. We get inspirations from invariant learning (Arjovsky et al., 2019; Krueger et al., 2021; Chang et al., 2020), which constructs different environments to infer the invariant features or predictors. To obtain the environments, previous studies mostly partition the training set by prior knowledge (Teney et al., 2020) or adversarial environment inference (Creager et al., 2021; Wang et al., 2021b). Different from partitioning the training data, we do not assume prophets about environments but introduce the interventional distribution (Tian et al., 2006; Pearl et al., 2016) instead to model the DIR task. Specifically, on the top of our SCM, we generate s-interventional distribution by doing intervention do(S = s) on S, which removes every link from the parents PA(S) to the variable S and fixes S to the specific value s. By stratifying different values S = {s}, we can obtain multiple s-interventional distributions.
With interventional distributions, we propose the principle of discovering invariant rationale (DIR) to identify a rationale C̃ whose relationship with the label Y is stable across different distributions.
Definition 1 (DIR Principle) An intrinsically-interpretable model h satisfies the DIR principle if it
1. minimizes all s-interventional risks: Es[R(h(G), Y |do(S = s))], and simultaneously
2. minimizes the variance of various s-interventional risks: Vars({R(h(G), Y |do(S = s))}), where the s-interventional risk is defined over the s-interventional distribution for specific s ∈ S.
Guided by the proposed principle, we design the learning strategy of DIR as:
minRDIR = Es[R(h(G), Y |do(S = s))] + λVars({R(h(G), Y |do(S = s))}), (4)
whereR (h(G), Y | do(S = s)) computes the risk under the s-interventional distribution, which we will elaborate in Section 2.4. Var(·) calculates the variance of risks over different s-interventional distributions; λ is a hyper-parameter to control the strength of invariant learning.
Justification. We theoretically justify the DIR principle’s ability to discover invariant rationales. Specifically, Theorem 1 shows that the oracle model fY respects the DIR principle. Moreover, we suggest that C can be inferred by making the intrinsically interpretable model h conform to the DIR principle under the uniqueness condition (cf. Corollary 1). We leave the detailed proofs in Appendix C due to the limited space. By making the distribution-relevant risks indifferent while pursuing low risks, the DIR principle is able to discover the invariant rationales C̃ as the approximation of the oracle rationales C, while encouraging hŶ approaching the oracle model fY .

heading: 2.4 DIR-GUIDED IMPLEMENTATION OF INTRINSICALLY-INTERPRETABLE GNNS
text: With the DIR principle and objective, we present how to implement the intrinsically-interpretable GNNs. We summarize the key notations of this section in Appendix A for clarity. Following Equation 2, a model h with intrinsic interpretability consists of two modules: h = hŶ ◦ hC̃ , where hC̃ is to extract a possible rationale, and hŶ is to make prediction based on the rationale. Moreover, to establish the s-interventional distributions, we design an additional module to do the interventions. In a nutshell, our framework consists of four components, as Figure 3 shows.
Rationale Generator. It aims to split the input graph instance g into two subgraphs: causal part c̃ and non-causal part s̃. Specifically, given an input graph instance g = (V, E) with the node set V
Published as a conference paper at ICLR 2022
𝒄�
𝒔�
Rationale Generator 𝒉𝑪 𝑮
Shared Encoder 𝒉𝜽 𝒁𝑪 𝒁𝑺
𝒀�𝑪…… …
𝒚�𝒔�
𝒀�
Classifier 𝚽 Shortcut Prediction
Causal Prediction
Intervener
𝓡��� 𝓡𝑺�
𝒀
No BP
…~ ~ ~
~ ~
~
~
~𝒉𝑰 𝑯𝑺~ 𝑯𝑺~~
~
𝒈 RationaleGenerator Shared GraphEncoder
Rationale Generator 𝒉𝑪
Shortcut Classifier
Causal Classifier
𝒚�𝒄� 𝒚� 𝒚
Distribution Intervener
Invariant Rationale Causal Prediction
No BP
Figure 3: DIR Implementation on GNNs, which includes a rationale generator, a distribution intervener, an encoder and two classifiers. For the inference, we only use ŷc̃ as the prediction.
and the edge set E , its adjacency matrix is A ∈ {0, 1}|V|×|V|, where Aij = 1 denotes the edge from node i to node j, and Aij = 0 otherwise. The rationale generator first adopts a GNN to generate the mask matrix M ∈ R|V|×|V| on A, where mask Mij indicates the importance of edge Aij :
Z = GNN1(g), Mij = σ(Z>i Zj), (5)
where σ(·) is the sigmoid function and Z ∈ R|V|×d summarizes the d-dimensional representations of all nodes. The generator then selects the edges with the highest masks to construct the rationale c̃ and collects c̃’s complement as s̃, as follows:
Ec̃ = Topr(M A), Es̃ = Top1−r((1−M) A), (6) where Ec̃ and Es̃ are the edge sets of c̃ and s̃, respectively; Topr(·) selects the top-K edges with K = r × |E|, and r is the hyper-parameter (e.g., 40%); is the element-wise product. Having obtained the edge sets, we can distill the nodes appearing in the edges to establish c̃ and s̃.
Distribution Intervener. It targets at creating interventional distributions. Formally, it first collects the non-causal part of all the instances into a memory bank as S̃. It next samples a memory s̃i ∈ S̃ to conduct the intervention do(S = s̃i), replacing the complement of the critical subgraph c̃j at hand and constructing an intervened pair (c̃j , s̃i), where i, j are indices.
Graph Encoder & Classifiers . Here we represent hŶ as a combination of a graph encoder and two classifiers. Specifically, it employs another GNN encoder on c̃ to generate node representations Zc̃ ∈ R|V|×d, and then combines them as graph representation Hc̃ ∈ RD via a global pooling operator, e.g., average pooling. Then it uses a classifier Φc to project the graph representation into a probability distribution over class labels ŷc̃. More formally, the process is as follows:
Zc̃ = GNN2(c̃), Hc̃ = Pooling(Zc̃), ŷc̃ = Φc(Hc̃). (7)
Analogously, we can obtain ŷs̃ for s̃ via the shared encoder and another classifier Φs. ŷc̃ is the prediction based merely on the causal part c̃, while ŷs̃ measures the predictive power of the intervened part s̃. Inspired by Cadène et al. (2019), we formulate the joint prediction ŷ under the intervention do(S = s̃) as ŷc̃ masked by ŷs̃:
ŷ = ŷc̃ σ(ŷs̃), (8) where the sigmoid function adjusts the output logits of c̃ to compensate for the spurious biases. In Appendix E, we present examples of how this operation helps discover the causal part.
Optimization. Having established the prediction ŷ of an instance g under the intervention do(S = s̃), we are capable of getting the s̃-interventional risk similar as Equation 4 as follows:
R(h(G), Y |do(S = s̃)) = E(g,y)∈O,S=s̃,C=hC̃(g)l(ŷ, y), (9)
where (g, y) ∈ O is a pair of graph instance g and its ground-truth label y from the training set O; l(·) denotes the loss function on a single instance. Moreover, we define the loss for Φs module as:
RS̃ = E(g,y)∈O,s̃=g/hC̃(g)l(ŷs̃, y) (10) Specifically, RS̃ is only backpropagated to the classifier Φs and we set apart the other components from its backpropagation to avoid interference with representation learning. Thus, this loss promotes the S̃-only branch to learn spurious biases given the non-causal features only. Overall, we can jointly optimize these components via the DIR objective and shortcut loss, i.e.,
min φsRS̃ + min γ,θ,φcRDIR. (11) where γ, θ and (φc, φs) are the parameters of the generator, encoder and two classifiers. While in
the inference phase, we yield c̃ and ŷc̃ as the causal rationale and the causal prediction of a testing graph g, which exclude the influence of the non-causal part s̃.

heading: 3 EXPERIMENTS
text: In this section, we conduct extensive experiments to answer the research questions: • RQ1: How effective is DIR in discovering causal features and improving model generalization? • RQ2: What are the learning patterns and insights of DIR training? Especially, how does invariant
rationalization help to improve generalization?

heading: 3.1 SETTINGS
text: Datasets. We use one synthetic dataset and three real datasets of graph classification tasks. Different GNNs are used in different datasets to achieve DIR and early stopping is exploited during training. Here we briefly introduce the datasets, while the details of dataset statistics, deployed GNNs, and training process are summarized in Appendix D. • Spurious-Motif is a synthetic dataset created by following Ying et al. (2019), which involves
18, 000 graphs. Each graph is composed of one base (Tree, Ladder, Wheel denoted by S = 0, 1, 2 respectively) and one motif (Cycle, House, Crane denoted by C = 0, 1, 2, respectively). The ground-truth label Y is determined by C solely. Moreover, we manually construct false relations of different degrees between S and label Y in the training set. Specifically, in the training set, we sample each motif from a uniform distribution, while the distribution of its base is determined by P (S) = b × I(S = C) + 1−b2 × I(S 6= C). We manipulate b to create Spurious-Motif datasets of distinct biases. In the testing set, the motifs and bases are randomly attached to each other. Besides, we include graphs with large bases to further magnify the distribution gaps.
• MNIST-75sp (Knyazev et al., 2019) converts the MNIST images into 70, 000 superpixel graphs with at most 75 nodes each graph. The nodes in the graphs are superpixels, while edges are the spatial distance between the nodes. Every graph is labeled as one of 10 classes. Random noises are added to nodes’ features in the testing set.
• Graph-SST2 (Yuan et al., 2020; Socher et al., 2013) Each graph is labeled by its sentence sentiment and consists of nodes representing tokens and edges indicating node relations. Graphs are split into different sets according to their average node degree to create dataset shifts.
• Molhiv (OGBG-Molhiv) (Hu et al., 2020; 2021; Wu et al., 2017) is a molecular property prediction dataset consisting of molecule graphs, where nodes are atoms, and edges are chemical bonds. Each graph is labeled according to whether a molecule inhibits HIV replication or not.
Baselines. We thoroughly compare DIR with Empirical Risk Minimization (ERM) and two classes of baselines:
• Interpretable Baselines: Graph Attention (Veličković et al., 2018) and graph pooling operations including ASAP (Ranjan et al., 2020), Top-k Pool (Gao & Ji, 2019) and SAG Pool (Lee et al., 2019). We use their generated masks on graph structures as rationales. We also include GSN (Bouritsas et al., 2020), a topologically-aware message passing scheme which enriches GNNs with interpretable structural features.
• Robust/Invariant Learning Baselines: Group DRO (Sagawa et al., 2019), IRM (Arjovsky et al., 2019), V-REx (Krueger et al., 2021). This class of algorithms improves the robustness and generalization for GNNs, which helps the models better generalize in unseen groups or out-ofdistribution datasets. We use random groups or partitions during the model training.
We also include an ablation model of DIR, DIR-Var, which sets λ = 0, i.e., discards the variance term inRDIR, to show the effectiveness of the variance regularization in the DIR objective. Metrics. We use ROC-AUC for Molhiv and ACC for the other three datasets. Moreover, for Spurious-Motif dataset, we use the precision metric to evaluate the coincidence between model rationales and the ground-truth rationales, and validate the interpretability ability quantitatively.

heading: 3.2 MAIN RESULTS (RQ1)
text: To fairly compare the methods, we train each model under the same training settings as described in Appendix D. The overall results are summarized Table 1, and we have the following observations:
1. DIR has better generalization ability than the baselines. DIR outperforms the baselines consistently by a large margin. Specifically, for MNIST-75sp dataset, DIR surpasses ERM by 7.65% and ASAP by 4.82%. Although structure features are shown to be helpful in mitigating feature distribution shift, DIR still performs better than GSN. For Graph-SST2 and Molhiv, DIR achieves the highest performance with low variance. For Spurious-Motif, DIR outstrips IRM averagely by 4.23% and SAG by 3.16% across different degrees of spurious bias. Such improvements strongly validate that DIR can generalize better in various environments.
2. DIR is consistently effective under different bias degrees, while the baselines easily fail. For interpretable baselines, Attention fails to make salient improvements when bias exists, and pooling methods also fall through under severe bias. This is empirically in line with our presumption that GNNs are easily biased to latch on spurious relations or non-causal features and thus generalize poorly in OOD data. For robust/invariant learning baselines, IRM underperforms ERM when b is small. This evidence is accordant with the conclusion in Ahuja et al. (2021) that IRM is guaranteed to be close to the desired OOD solutions when confounders exist, while it has no obvious advantage to ERM under covariate shift. Moreover, Group DRO and V-REx follow a similar pattern. In contrast, DIR works well in various scenarios. We credit such reliability to the rationales discovery from which the causal features C are potentially extracted, and the relation C → Y learned by the GNNs is invariant across the distribution changes in the testing set.

heading: 3. Data augmentation by intervention is beneficial while the variance regularization further
text: boosts model performance. Interestingly, the ablation model DIR-Var has already exceeded some of the baselines. We attribute such improvement to data augmentation via interventional distributions. On top of DIR-Var, DIR improves the model performance by averagely 1.57% in Spurious-Motif and 2.62% in MNIST-75sp. This suggests that the variance regularization demands a stronger invariance condition and is instructive for searching causal features.
4. DIR has better intrinsic interpretability than the baselines. In Table 2, we report intrinsic interpretable models’ performance w.r.t. Precision@5. From the consistent improvements over the baselines, we find DIR has an advantage in discovering causal features. And the performance gap between DIR and the baselines becomes more significant when the bias increases.
③① ②
adaption fitting adaption fitting
③
①
②
③① ②
(a) The first two subfigures show the training curves w.r.t. variance penalty and precision, on Spurious-Motif. The last three subfigures present the rationale distributions of the inspection points, which are visualized by t-SNE (van der Maaten, 2008).
③① ②
adaption fitting adaption fitting
③
①
②
③① ②

heading: 3.3 IN-DEPTH STUDY (RQ2)
text: We empirically analyze the DIR’s properties which hopefully give insights into its mechanisms and can be instructive for the existing training paradigms of deep models.
Rationale Visualization. Towards an intuitive understanding of DIR, we first present some cases of the discovered rationale for Graph-SST2 in Figure 4. DIR is able to emphasize the tokens that directly result in the sentences’ positive or negative sentiments, which are reliable and faithful rationales. Specifically, DIR highlights the positive words “majestic achievement” and “astonishing grandeur” in Figure 4a and underscores the negative words “worst dialogue” in Figure 4b as the rationales, which are clearly salient for the positive and negative sentiments, respectively. Furthermore, DIR can focus persistently on the causal features for OOD testing data. For example, it selects surprisingly engrossing and “admittedly middling” in Figures 4c and 4d, respectively. This again validates the effectiveness of DIR: (1) hC̃ is well-learned to distinguish causal and non-causal features under various interventional distributions; and (2) hỸ conducts message-passing on the highlighted rationales, extracts the graph representations, and finally outputs the predictions with high accuracy. See Appendix F.1 for more examples in Graph-SST2 and Spurious-Motif datasets.
Two-stage Training Dynamics. As Figure 5a displays, we find a pattern from the Var-Time curve — during training DIR, the variance penalty (i.e., Vars in Equation 4) first increases and then decreases to almost zero. Moreover, there exists an interesting correlation between the variance penalty and the precision metrics — that is, the precision rises dramatically as the penalty increases while growing slowly as the penalty decreases. To probe this learning pattern, we further visualize the rationale distribution in three turning points: (1) the start, (2) the middle, and (3) the end of training. Interestingly, the rationale distribution at the middle point is highly similar to that at the ending point. This illustrates two stages, adaption and fitting, in the patterns. By “adaption”, we mean that the exhibition of hC̃ , i.e., learning to select salient feature C̃, is mainly conducted during the initial training stage. Since the penalty value can be seen as the magnitude to violate the invariance condition, this stage explores the rationales that satisfy the DIR principle. Correspondingly, hỸ adapts
quickly with the input of varying rationales generated by hC̃ . By “fitting”, we mean that, in the later training process, hC̃ only makes small changes, resulting in the substantially unchanged rationales compared to the initial training process, which is learned from the rationale generator to conform to the DIR principle. This could also imply that based on the well-learned rationales, DIR mainly optimizes hỸ to consolidate the functional relation C̃ → Y until model convergence. Moreover, we compare the learning patterns of IRM and DIR in Figure 5b, where the penalty term of IRM (the gradient norm penalty in IRMv1 (Arjovsky et al., 2019)) follows a similar pattern to the DIR penalty. Notably, in MNIST-75sp, while IRM consistently outperforms DIR w.r.t. Training ACC, it does not improve and even degrades the performance in the testing dataset due to overfitting. However, DIR shows the solid resistance for over-fitting, partly thanks to the valid rationales exhibited in the adaption stage. For Molhiv, DIR outperforms IRM as the rationales filter out irrelevant or spurious structures bootless for classification tasks and are beneficial for generalization.
Sensitivity Analysis. We conduct a sensitivity analysis of model performance w.r.t. λ in Appendix F.2, which shows that DIR surpasses the best baselines under a relatively large range of λ.

heading: 4 RELATED WORKS
text: Inherent Interpretability of GNNs. We summarize two classes of the existing methods to build deep interpretable GNNs, (i) Attention (Vaswani et al., 2017; Veličković et al., 2018), which can be broadly interpreted as importance weights on representations.(ii) Pooling (Lee et al., 2019; Knyazev et al., 2019; Gao & Ji, 2019), which selectively performs down-sampling on representations. We include it in this category when it involves selection importance. However, the mechanisms to generate the rationales could be epistemic, as they only reflect the probabilistic relations between data and predicted labels (Pearl, 2000), which may not hold true in all data distributions. Thus, the rationales could fail to align with causal features and even degrade model performance due to being “fooled” by spurious features (Chang et al., 2020).
Invariant Learning. Backed by causal theory, invariant learning assumes the causal relation from the causal factors C to the response variable Y remains invariant unless we intervene on Y . As the most prevailing formulation, IRM (Arjovsky et al., 2019) extends the invariance assumption from feature level to representation level and finds a data representation Φ such that Ω ◦ Φ matches for all environments, where Ω is the classifier. However, concerns about its feasibility (Rosenfeld et al., 2021; Ahuja et al., 2021) and optimality (Kamath et al., 2021) have been discussed recently. Besides IRM, variance penalization across environments is shown to be effective for recovering invariance (Krueger et al., 2021; Xie et al., 2020; Teney et al., 2020). Notably, the existing methods generally require accessing different environments, thus additionally involving environment inference (Creager et al., 2021; Wang et al., 2021b). Similarly motivated as ours, Chang et al. (2020) discover rationales Z by minimizing the performance gap between environment-agnostic predictor f(Z) and environment-aware predictor f(Z,E). In graph domain, Bevilacqua et al. (2021) construct graph representations from subgraph densities and use attribute symmetry regularization to mitigate the shift of graph size and vertex attribute distributions.

heading: 5 CONCLUSION & FUTURE WORK
text: In this work, we rigorously study the intrinsic interpretability of Graph Neural Networks from a causal perspective. Our concerns are towards the exhibition of shortcut features when generating the rationales. And we proposed an invariant learning algorithm, DIR, to discover the causal features for rationalization. The core of DIR lies in the construction of environments (i.e., interventional distributions) and thus distilling the salient features as rationales that are consistently informative and uniform across these environments. Such rationales serve as the probing towards model mechanisms and are demonstrated to be effective in generalization. In the experiments, we highlight an adaption-fitting training dynamics for DIR to reveal its learning pattern. In the future, we will build more reliable and expressive interpretable models that are feasible under various assumptions, which potentially calls for high-level interpretability. We recommend interested readers go to the open discussion in Appendix G for the detailed description.

heading: ACKNOWLEDGMENT
text: This work was supported by the National Key Research and Development Program of China (2020AAA0106000), the National Natural Science Foundation of China (U19A2079), the SeaNExT Joint Lab, and Singapore MOE AcRF T2.

heading: ETHICS STATEMENT
text: In this work, we propose a novel algorithm for intrinsic interpretable models, where no human subject is related. This synthetic dataset is made available in the anonymous link (cf. Section 3.1). We believe the exhibition of rationales is beneficial for inspecting and eliminating potential discrimination and fairness issues in deep models for real applications.

heading: REPRODUCIBILITY STATEMENT
text: We summarize the efforts made to ensure reproducibility in this work. (1) Datasets: We use one synthetic dataset which is made available (cf. the anonymous link in Section 3.1), and three public datasets where the processing details are included in Appendix D. (2) Model Training: We provide the procedure of training in Algorithm A and the training details (including hyper-parameter settings) in Appendix D which are consistent with our implementation in the code (cf. the anonymous link in Section 3.1). (3) Theoretical Results: All assumptions and proofs can be referred to Appendix C.

heading: A NOTATIONS & ALGORITHM
text: Key Notations in the Main Paper.
Symbol Definition
g graph instance c / s ground truth causal or confounding subgraph c̃ / s̃ generated rationale or complement of rationale instance C / S variables in the causal graph S / S̃ space of the ground truth or identified spurious features ŷc̃ / ŷs̃ causal or spurious prediction ŷ joint prediction hC̃ rationale generator Φ1 / Φ2 causal or spurious classifier
Algorithm 1 Pseudocode for DIR in training interpretable Graph Neural Networks (Batch Version) Require: Training data distribution Ptr(G); number of classes Q; Stepsize α; hyper-parameter λ
1: Randomly initialize the parameters of generator hC̃ , encoder hθ (includes GNN2 and Pooling layer), two classifiers Φ1 and Φ2, which are denoted as γ, θ, φ1, φ2, respectively. 2: while not converge do 3: Sample graphs {(gi, yi)}Bi=1 from Ptr(G) 4: Generate each rationale and its complement: (c̃i, s̃i)← hC̃(g
i), for i = 1, . . . , B. 5: for each s̃i do 6: Intervener hI operates do(S = s̃i) 7: Model forward: ŷs̃ = Φ2(hθ(s̃i)) ∈ R1×Q, {ŷc̃}Bi=1 = Φ1(hθ({c̃i}Bi=1)) ∈ RB×Q 8: # block BP of DIR risk to shortcut branch Obtain joint prediction ŷ = [ŷ1, . . . , ŷB ], where ŷj = ŷc̃ σ(ŷjs̃).detach() 9: Compute and record risk R(ŷs̃, yi)
10: Compute and record s̃i-interventional risk. 11: end for 12: ComputeRDIR via Eq. 4 andRS̃ via Eq. 11 13: Update parameters: φ2 = φ2 − α∇φ2RS̃ ; φ1 = φ1 − α∇φ1RDIR; γ = γ − α∇γRDIR; θ = θ − α∇θRDIR 14: end while
B INSTANTIATED CAUSAL GRAPHS
We instantiate possible causal graphs in Figure 2a. Specifically, we use the example of Base-Motif graphs, whose labels are determined by the motif types. We use C = 0, 1, 2 to denote cycle, house, crane, respectively; And use S = 0, 1, 2 to denote ladder, tree, wheels, respectively.
• C |= S: Base graphs and motif graphs are independently sampled and attached to each other. • C → S: Type of each motif respects to a given (static) probability distribution. According to the
value of C, the probability distribution of its base graph is given by
P (S) = { 0.6 if S = C 0.2 otherwise (12)
• S → C: Similar to the example for C → S. • S ← E → C: Suppose there is a latent variable E takes continuous value from 0 to 1. Then the
probability distribution of S and C s.t. S ∼ B(3, E) C ∼ B(3, 1− E) (13)
where B stands for binomial distribution, i.e., for variable X , if X ∼ B(n, p), then we have P (X = k | p, n) = ( n k ) pk(1− p)n−k

heading: C THEORY

heading: C.1 ASSUMPTION
text: We phrase the SCM in Figure 2a as the following assumption:
Assumption 1 (Invariant Rationalization (IR)) There exists a rationale C ⊆ G, such that the structural equation model Y ← fY (C, Y ) , Y |= C and the probability relation S |= Y | C hold for every distribution P̃ over P(G, Y ), where S denotes the complement of C. Also, we denote fY as the oracle structural equation model.
By “oracle”, we mean that fY is the perfect structure equation model, which, when C is available, predicts the response variable with the minimum expected loss over any distribution P̃ . Or formally,
fY := arg min f R(f) = arg min f E(G,Y )∼P̃, Y [l(f(C, Y ), Y ), Y )]. (14)
where l is the task-specific loss function and we ignore the exogenous noise Y in fY ’s input except as otherwise noted.
Next, we argue that the assumption is commonly satisfied. For example, for sentences labeled by sentiment, C can represent the positive/negative words that cause the sentiment, while S includes the prepositions and linking words. For molecule graphs labeled by specific properties, C and S can represent the functional groups and carbon structures, respectively. Note that IR assumption enables and calls the introduction of interpretability, highlighting salient features and exhibiting human accessible checks. More importantly, it guarantees the model performance under possible feature reduction, i.e., C ⊂ G. We also see cases going beyond the IR Assumption. For example, G could be a generic function of S and C, instead of a simple joint. We use a toy example to elaborate this point. Following the Spurious-Motif dataset, we assume each graph has multiple motifs (house, cycle, crane) with only one type and is labeled by the motif type. Thus, the causal feature C will be the motifs. Let the spurious feature S be ”the way we connect the motifs”. For example, we can place the house motifs in a queue sequence and connect the adjacent motifs, thus forming the graph in a ”line” shape. Or we can place the houses in a cycle order and connect them into a ring. We further make such graph structures strongly correlated with the motif types. Thus, individual S and C may be intractable individually in the feature level. For example, if we separate the cycle-shaped houses into two lines, the spurious pattern could be broken while the part of the causal feature would be lost. In other words, S and C are dependent variables. Thus, they can’t be extracted and modeled separately, which goes out of the scope of our work.
Given that S and C are separable, we further make the following assumption to avoid the confusion of S and C:
Assumption 2 (Feature Induction) Define power set operation as P∗(·). For data G = S ∪ C and label Y , if S |= Y | C holds for any distribution P̃ over P(G, Y ), then it implies that for any induced feature S′ ∈ P∗(S), we have S′ |= Y | C holds for the distribution P̃ .
This assumption also implies that C could not be induced by S when |C| ≤ |S|. Thus, any feature subset C ′ except for C would violate the conditional independence condition. For images, this assumption is natural for the splicing of S doesn’t typically change its semantics. For example, the splicing of land background would still be divided land. While for graphs, here we assume the causal subgraph’s uniqueness among the induced complement graphs.

heading: C.2 PROOFS
text: Theorem 1 (Necessity) Suppose S → C does not exist, then the oracle function fY satisfies the DIR Principle (where C is given) over every distribution P̃ ∈ P(G, Y ).
Proof: We first prove the fact that P (Y = y | do(S = s)) = P (Y = y) for distribution P̃ . Specifically, we use P (s)I to denote the s-interventional distribution.
• If C → S,
P (Y = y | do(S = s)) by definition======== P (s)I (Y = y | S = s) = ∑ c P (s) I (Y = y | S = s, C = c)P (s) I (C = c | S = s)
given C→S ======== P
(s) I (Y = y | S = s, C = c)P (s) I (C = c)
given (Y |= S|C)P̃============= ∑ c P (s) I (Y = y | C = c)P (s) I (C = c)
given invariance condition =============== ∑ c P (Y = y | C = c)P (C = c)
= P (Y = y)
• If C |= S,
P (Y = y | do(S = s)) by definition======== P (s)I (Y = y | S = s) given S has no endogenous parent
=================== P (Y = y | S = s) givenC |= S========= ∑ c P (Y = y | C = c, S = s)P (C = c)
= ∑ c P (Y = y | C = c, S = s)P (C = c | S = s)
given (Y |= S|C)P̃============= ∑ c P (Y = y | C = c)P (C = c)
= P (Y = y)
• If C ← E → S,
P (Y = y | do(S = s)) by definition
======== P (s) I (Y = y | S = s)
givenE→S ======== ∑ e P (s) I (Y = y | S = s, E = e)P (s) I (E = e)
= ∑ e ∑ c P (s) I (Y = y | S = s, E = e, C = c)P (s) I (C = c|S = s, E = e)P (s) I (E = e)
given (Y |= {S,E}|C) and (C |= S|E)======================== ∑ e ∑ c P (s) I (Y = y | C = c)P (s) I (C = c|E = e)P (s) I (E = e)
= ∑ e ∑ c P (Y = y | C = c)P (C = c|E = e)P (E = e)
= ∑ e ∑ c P (Y = y | C = c, E = e)P (C = c|E = e)P (E = e)
= P (Y = y)
As P (Y = y | do(S = s)) = P (Y = y) holds true for every distribution P̃ , which is invariant w.r.t. iterative variable S. Moreover, we have P (C = c | do(S = s)) = P (s)I (C = c) = P (C = c). This
indicates that the intervention on S leave the causal structure C → Y untouched. Thus, we have
Var ({R(fY | do(s)) | s ∈ S}) = Var ({
E (G,Y )∼P (s)I (G,Y ),C⊂G
[l(fY (C), Y )] | s ∈ S })
= Var ({ E(C,Y )[l(fY (C), Y )] | s ∈ S })
= 0
Finally, taking the definition of f , we have
fY = arg min f Es∈S [ E (G,Y )∼P (s)I (G,Y ),C⊂G [l(f(C), Y )] ]
= arg min f
Es∈S [ E(C,Y )[l(f(C), Y )] ] = arg min
f Es∈S[R(f | do(s))]
Hence, fY takes the minimum penalty and satisfies the DIR Principle.
Notably, if S → C, then Var ({R(fY | do(s)) | s ∈ S}) may not equal to zero since c ∼ P
(s) I (C|S = s). In such case, fY is not necessarily satisfied to DIR Principle. That is, although fY still minimizes R(f | do(S)), we can’t be sure whether it reaches the lower bound of Var ({R(fY | do(s)) | s ∈ S}) without knowledge about the specific data distribution. Thus, we only consider the cases of C → S, C |= S and C ← E → S in the following discussion.
Theorem 2 (Uniqueness) Suppose l is a strict loss function and there exists one and only one nontrivial subset C, then there exists a unique structure equation model fY s.t. it satisfies the DIR Principle.
Proof: Since fY exists and satisfies the DIR Principle, we only need to prove its uniqueness under the given conditions. Otherwise, suppose we have another structure equation f ′Y 6= fY satifies the DIR Principle. Specifically, there exists a datum (g, y) s.t. f ′Y (c) 6= fY (c). Thus, we have l(f ′Y (c), y) > l(fY (c), y). Given that Var ({R(f ′Y | do(s)) | s ∈ S}) ≥ 0 = Var ({R(fY | do(s)) | s ∈ S}), we haveRDIR(f ′Y ) > RDIR(fY ).
In reality, there could be multiple candidates of C, e.g., Ci, Cj s.t. RDIR(f (Ci)Y ) = RDIR(f (Cj) Y ), where f (Ci)Y is the structure equation corresponds to Ci. Thus, it calls for the selection of C to avoid the learning of suboptimal fY . Inspired by Occam’s Razor, we define
C∗ = arg min |C| (15)
as the preferred rationale, or rationale of parsimony. We argue that rationales are not to be extended beyond necessity, which poses simpler hypotheses about causality. As the search of C∗ is NPhard (the worst time complexity is exponential), we use fixed size for the learned rationales in our experiments and leave a better optimization to future work.
Corollary 1 (Necessity and Sufficiency) Suppose l is a strict loss function and there exists one and only one non-trival subset C, then any structure causal model f ′Y s.t. it satisfies the DIR Principle iff. f ′Y = fY .
This is directly obtained from Theorem 2. Thus, under the unique constraint of C, we can approach the oracle fY by optimizing the DIR objective, which maintains the invariant causal relation between the causal feature and the response variable Y . In another way, based on the uniqueness of the feasible rationale, the optimization of the DIR Principle on the intrinsic interpretable model h (where C is exhibited inside of h) pushes the approach to C with rationales C̃. Then, fY can also be approached as an invariant predictor based on the learning from C̃.

heading: D SETTING DETAILS
text: Datasets We summarize dataset statistics in Table 3, and introduce the node/edge features and the preprocessing in each datasets:
• Spurious-Motif. We use random node features and constant edge weights in this dataset.
• MNIST-75sp. The nodes in the graphs are superpixels, and node features are the concatenation of pixel intensities (RGB channels) and coordinates of their mass centers. Edges are the spatial distance between the superpixel centers, while we filter the edges with a distance less than 0.1 to make the graphs sparser.
• Graph-SST2. We use constant edge weight and filter the graphs with edges less than three. We initialize the node features by the pre-trained BERT (Devlin et al., 2018) word embedding.
• OGBG-Molhiv. We use the official released dataset in our experiment.
GNNs. We summarize the backbone GNNs for each dataset in Table 3. The number of neurons in the sequent layers (in forwarding order) is reported. We use ReLU as activation layers and different global pooling layers. In OGBG-Molhiv, we adopt one fully connected layer for the prediction layers while using two fully connected layers for the models in other datasets. For baselines with node pooling/node attention, we add one node pooling/attention layer in the second convolution layer.
Training Optimization & Early Stopping. All experiments are done on a single Tesla V100 SXM2 GPU (32 GB). During training, we use Adam (Kingma & Ba, 2015) optimizer. The maximum number of epochs is 400 for all datasets. We use Stochastic Gradient Descent (SGD) for the optimization on Graph-SST2 and OGBG-Molhiv and Gradient Descent (GD) for the other two datasets. Also, we exhibit early stopping to avoid overfitting of the training dataset. Specifically, in MNIST-75sp, Graph-SST2 and OGBG-Molhiv, each model is evaluated on a holdout in-distribution validation dataset after each epoch. While for Spurious-Motif, we use an unbiased validation dataset (i.e., without spurious relations compared to the training dataset). If the model’s performance on the validation dataset is without improvement (i.e., validation accuracy begins to decrease) for five epochs, we stop the training process to prevent increased generalization error.
Hyper-Parameter Settings. We set the causal feature ratio and λ as (r = 0.8, λ = 10−4), (r = 0.25, λ = 10−2), (r = 0.6, λ = 102), (r = 0.8, λ = 10−3) for MNIST-75sp, Spurious-Motif, Graph-SST2 and OGBG-Molhiv respectively. For other baselines, we adopt grid search for the best parameters using the validation datasets.
Model Selection. We select each model based on its performance on the corresponding validation dataset. We repeat each experiment at least five times and report the average values and the standard errors in the paper.

heading: E UNIMODAL ADJUSTMENT
text: We follow Cadène et al. (2019) to demonstrate how the shortcut prediction can help to remove model bias. For clarity, we refer to the model parameters except for Φ2 as the main branch, i.e., except for the S-only branch.
Given a house-tree graph as the input graph, we suppose the shortcut prediction ŷs̃ of the tree subgraph leans towards the house class. Then after reweighting σ(ŷs̃) on ŷc̃, the softmax readout on the house class in the joint prediction ŷ will be magnified, which results in a smaller loss backpropagated to the main branch and prevents ŷc̃ from inductive bias.
In another situation where a house-wheel graph is given as the input, we similarly suppose the shortcut prediction ŷs̃ of the wheel subgraph leans towards other classes except the house, say, the circle class. Then after reweighting σ(ŷs̃) on ŷc̃, the softmax readout on the house class in the joint prediction ŷ will be reduced, which results in a larger loss back-propagated to the main branch and encourages the model to learn from these examples.
Furthermore, we offer the causal- and information-theoretical justifications: (1) From the perspective of causal theory (Pearl, 2000; Pearl et al., 2016), the element-wise multiplication enforces the spurious prediction to estimate the pure indirect effect (PIE) of the shortcut features, while the causal prediction captures the natural direct effect (NDE) of the causal patterns (VanderWeele, 2013); (2) From the perspective of information theory (Kullback, 1997), the element-wise multiplication makes the causal prediction reflect the conditional mutual information between the causal patterns and ground-truths, conditioning on the complement patterns.

heading: F MORE EXPERIMENTAL RESULTS
text: F.1 VISUALIZATION
We provide more visualization cases in Graph-SST2 dataset as shown in Figure 6 and Figure 7. The rationales are highlighted in deep colors.

heading: F.2 SENSITIVITY ANALYSIS

heading: MNIST-75spSpurious-Motif Graph-SST2 OGBG-Molhiv
text: We analyze the performance of DIR w.r.t. the hyper-parameter λ. As shown in Figure 10, with λ→ 0, DIR degrades to optimize the performance in each environment only, without explicitly penalizing the shortcuts’ influence on the model predictions. We also see that all testing performances drop sharply if λ is too large. Since a large weight on the variance term emphasis on the invariance condition while leading to the overlook on the performance loss, it could fail to exhibit fỸ correctly. Notably, such a trade-off in the DIR objective is commonly shared among all the datasets.

heading: F.3 STUDY OF THE SPURIOUS CLASSIFIERS
text: Here we provide more observations about the predictions of the learned spurious classifier, which sheds light on the designed model mechanism. We first look into the confidence of predictions and define
ν = E(g,y)∈O,s̃=g/hC̃(g)H (Softmax(ŷs̃)) (16)
where H is the entropy function, and a lower ν indicates higher confidence. We report the results for the trained spurious classifiers in Table 4. Thus, the results demonstrate the marked tendency of the spurious predictions and validate the design of the S−only branch. However, we show that spurious classifiers are over-confident and potentially overfit to spurious features, which fails to generalize out-of-distribution. In Table 5, we evaluate the spurious classifiers (taking non-causal features as inputs) on the testing sets. We argue that the performance degradation is caused by (i) feature-level problem: it could be theoretically inadequate to infer the label given the non-causal features, and (ii) paradigm-level problem: minimizing the empirical risk only can hardly exhibit stable relations between the features and labels.

heading: F.4 COMPARISON OF POST-HOC EXPLANATIONS AND INTRINSIC RATIONALES.
text: Here we aim to compare the explanations generated by GNNExplainer (Ying et al., 2019) and the rationales exhibited by DIR. Specifically, we generate post-hoc explanations from GNNExplainer for Spurious-Motif, where we use the models trained under ERM as the models to explain. We compute the precision of the explanations in Table 6.
The explanations generated by GNNExplainer reflect the models’ inner mechanism, which backs that deep models easily learn from data bias (especially when b is large), being at odds with the true reasoning process that underlies the task. Moreover, even when spurious correlations do not exist, the precisions of rationales generated by DIR still outperform the precisions of the post-hoc explanations, showing the effectiveness of DIR when identifying causal features.

heading: G OPEN DISCUSSIONS
text: Based on this work, we provide open discussions and future directions for the research community, which are inspired by the insightful comments of the ICLR reviewers.

heading: G.1 EXPRESSIVENESS OF RATIONALE GENERATORS
text: High expressiveness of the rationale generators could be beneficial for the identification of causal features. Therefore, we have offered additional techniques in our implementation to improve the expressiveness of the graph encoder. Specifically, we incorporate distance encoding measures (Li et al., 2020) like shortest-path distances as the extra node features for better structural representation learning. Also, more powerful graph encoders like RingGNN (Chen et al., 2019) and 3WLGNN (Maron et al., 2019) can be used as the graph encoders to distinguish different substructures better.

heading: G.2 GENERALIZATION TO UNSEEN SPURIOUS PATTERNS
text: In our implementation, the memory bank only contains the spurious patterns seen in the training set, while it could possibly fail to unseen spurious patterns. And we provide discussions and solutions to solve this limitation:
• Attribute level perturbation. When the spurious patterns in the testing are different from those in training set only on the attribute level, we can perturb the node/edge attributes of the subgraphs before intervention. And such perturbation is expected to improve the model’s robustness during inference.
• External knowledge base. When the spurious patterns also change on the structure level, for example, a star-shaped unseen base graph appears in the testing set of the Spurious-Motif, one potential solution is to resort to prior knowledge. We can enrich the memory bank with possible spurious patterns, e.g., tree (seen) and star (unseen) base graphs. With the external knowledge base, the model can be trained to recognize these possible spurious patterns and be well generalized to the testing dataset.
• Subgraph matching. In a more tricky scenario when the external knowledge base is not available, we can integrate our model with subgraph matching algorithms in the inference. For example, we can extract the training rationales into another bank C̃ and use them to query the testing graphs, i.e., checking if similar patterns exist in the testing graphs. The match results may assist the rationale generator in highlighting the causal features and avoiding unseen spurious features.

heading: G.3 HIGHER LEVEL INTERPRETABILITY
text: The interpretability of GNNs in the feature level implicitly demands the separability of a graph into causal and non-causal features. At the same time, we see cases going beyond such assumption (cf. Appendix C.1). We believe we could resort to higher level interpretability. For example,
• Interpretability of representations (Wang et al., 2021a; Chan et al., 2021). Instead of highlighting important features for the model decisions, the general goal of representation interpretability answers “What’s the information encoded by the i-th element of the embedding in the j-th layer?”.
• Interpretations on top of disentangled variables. Each disentangled latent variable reveals one independent generative factor in the data (Bengio et al., 2013). By generating importance score on these variables, we could possibly obtain more semantically rich interpretations than featurelevel interpretability.
Wherein, we believe there are fewer constraints on the separability of features. Thus, the models equipped with higher level interpretability could be applied to a broader range of data-generating assumptions.

Review summaries:
This paper proposes a novel algorithm DIR that allows the learned GNN model to make predictions based on causal patterns. The whole framework consists of four components where (1) the rationale generator splits the input into causal and spurious parts; (2) the distribution intervener replaces the spurious part; (3) the graph encoder embeds the graph based on the spurious and causal parts, respectively; (4) the shortcut and causal classifiers are applied to the spurious and causal embeddings, respectively. This algorithm is highlighted by its objective which encourages the predictions accurate while invariant to the spurious part. On both synthetic and real-world datasets, DIR outperforms the related baselines consistently. As a novel end2end rationalization method, As a novel end2end rationalization method, the underlying DIR principle is proved to be sufficient and necessary for satisfying the oracle function.

This paper proposes to address the limitation of current interpretable graph neural networks over out-of-distribution data. In this work, the intrinsically interpretable GNN is investigated by identifying the invariant rationales corresponding to environment-invariant causal patterns. The proposed model is evaluated on the graph classification task to demonstrate its effectiveness as compared to state-of-the-arts.