Title: Dissecting the Failure of Invariant Learning on Graphs
Venue: NeurIPS 2024
Accepted: yes
Keywords: Machine Learning

Out-of-distribution Generalization

Invariant Learning

Graph Machine Learning
Abstract: Enhancing node-level Out-Of-Distribution (OOD) generalization on graphs remains a crucial area. In this paper, we develop a Structural Causal Model (SCM) to theoretically dissect the performance of two prominent invariant learning methods--Invariant Risk Minimization (IRM) and Variance-Risk Extrapolation (VREx)--in node-level OOD settings. Our analysis reveals a critical limitation: these methods may struggle to identify invariant features due to the complexities introduced by the message-passing mechanism, which can obscure causal features within a range of neighboring samples. To address this, we propose Cross-environment Intra-class Alignment (CIA), which explicitly eliminates spurious features by aligning representations within the same class, bypassing the need for explicit knowledge of underlying causal patterns. To adapt CIA to node-level OOD scenarios where environment labels are hard to obtain, we further propose CIA-LRA (Localized Reweighting Alignment) that leverages the distribution of neighboring labels to selectively align node representations, effectively distinguishing and preserving invariant features while removing spurious ones, all without relying on environment labels. We theoretically prove CIA-LRA's effectiveness by deriving an OOD generalization error bound based on PAC-Bayesian analysis. Experiments on graph OOD benchmarks validate the superiority of CIA and CIA-LRA, marking a significant advancement in node-level OOD generalization.

Dissecting the Failure of Invariant Learning on Graphs

Abstract

Enhancing node-level Out-Of-Distribution (OOD) generalization on graphs remains a crucial area of research. In this paper, we develop a Structural Causal Model (SCM) to theoretically dissect the performance of two prominent invariant learning methods-Invariant Risk Minimization (IRM) and Variance-Risk Extrapolation (VREx)-in node-level OOD settings. Our analysis reveals a critical limitation: due to the lack of class-conditional invariance constraints, these methods may struggle to accurately identify the structure of the predictive invariant ego-graph and consequently rely on spurious features. To address this, we propose Crossenvironment Intra-class Alignment (CIA), which explicitly eliminates spurious features by aligning cross-environment representations conditioned on the same class, bypassing the need for explicit knowledge of the causal pattern structure. To adapt CIA to node-level OOD scenarios where environment labels are hard to obtain, we further propose CIA-LRA (Localized Reweighting Alignment) that leverages the distribution of neighboring labels to selectively align node representations, effectively distinguishing and preserving invariant features while removing spurious ones, all without relying on environment labels. We theoretically prove CIA-LRA's effectiveness by deriving an OOD generalization error bound based on PAC-Bayesian analysis. Experiments on graph OOD benchmarks validate the superiority of CIA and CIA-LRA, marking a significant advancement in nodelevel OOD generalization. The codes are available at https://github.com/ NOVAglow646/NeurIPS24-Invariant-Learning-on-Graphs .

1 Introduction

Generalizing to unseen testing distributions that differ from the training distributions, known as OutOf-Distribution (OOD) generalization, is one of the key challenges in machine learning. Invariant learning, which aims to capture predictive features that remain consistent under distributional shifts, is a crucial strategy for addressing OOD generalization. Numerous invariant learning methods have been proposed to tackle OOD problems in computer vision (CV) tasks [Arjovsky et al., 2020, Krueger et al., 2021, Bui et al., 2021, Rame et al., 2022, Wang et al., 2019, Mahajan et al., 2021, Zhang et al., 2021a, Wang et al., 2022a, Yi et al., 2022, Wang et al., 2022b, Xin et al., 2023]. While in recent years, enhancing OOD generalization on graph data is an emerging research direction attracting increasing attention. In this work, we focus on the challenge of node-level OOD generalization on graphs.

Straightforwardly adapting the above methods to node-level graph OOD scenarios presents several challenges: 1) the prediction of a node's label depends on its neighbored samples in an ego-subgraph, causing a gap from conventional CV OOD scenarios where samples are independently predicted; and 2) environment labels in node-level tasks are often unavailable [Wu et al., 2021, Li et al., 2023a,

Liu et al., 2023], rendering invariant learning methods based on environment partitioning infeasible. To illustrate the failure of directly adapting traditional invariant learning to graphs, we evaluate two representative OOD approaches, Invariant Risk Minimization (IRM [Arjovsky et al., 2020]) and Variance-Risk Extrapolation (VREx [Krueger et al., 2021]), in OOD node classification scenarios. We choose IRM and VREx for two reasons: 1) Numerous node-level graph OOD strategies [Zhang et al., 2021b, Wu et al., 2021, Liu et al., 2023, Tian et al., 2024] utilize VREx as invariance regularization (details in Appendix A.2). Therefore, analyzing VREx can cover a significant portion of graph-OOD methods; and 2) IRM and VREx are two prominent OOD methods that we can theoretically prove to be effective on non-graph data (Proposition 2.2). By testing their performance on graph data, we can better reveal the differences between graph and non-graph data. We choose real-world graph datasets: Arxiv, Cora, and WebKB; synthetic datasets: CBAS and a toy graph OOD dataset with spurious correlations between node features and labels for evaluation. From Table 1, we observe that IRM and VREx offer marginal or no improvement over ERM on both real-world and synthetic node-level graph OOD datasets. This naturally raises some questions here:

On graphs, why do traditional invariant learning methods fail? How to make them work again?

To theoretically analyze their failure modes, we build a Structural Causal Model (SCM) to model the data generation process under two types of distributional shifts: concept shift and covariate shift, and gain a high-level understanding of the challenges in node-level OOD generalization: To correctly predict a node's label, the structure of a predictive invariant neighboring egograph (which we call it a causal pattern ) and their invariant node features must be learned. However, identifying the correct structure of the causal pattern presents additional optimization requirements (compared to CV scenarios) for Graph Neural Networks (GNNs) since they must adjust the aggregation parameters (such as the attention weights in GAT [Veliˇ ckovi´ c et al., 2018]) to achieve this. IRM and VREx lack class-conditional

invariance constraints, which causes insufficient supervision for regularizing the training of these aggregation parameters, leading to non-unique solutions of GNN parameters and potentially resulting in the learning of spurious features. (detailed analysis is in Section 2). To overcome this, we propose Cross-environment Intra-class Alignment ( CIA ) that further considers class-conditional invariance to identify causal patterns 2 . We theoretically demonstrate that by aligning node representations of the same class and different environments, CIA can eliminate spurious features and learn the correct causal pattern, as same-class different-environment samples share similar causal patterns while exhibiting different spurious features. Table 1 shows CIA's empirical gains. To leverage the advantage of CIA and adapt it to scenarios without environment labels, we further propose CIA-LRA (Localized Reweighting Alignment), utilizing localized label distribution to find node pairs with significant differences in spurious features and small differences in causal ones for alignment, to eliminate the spurious features while alleviating the collapse of the causal ones. Our contributions are summarized as follows:

By constructing an SCM, we provide a theoretical analysis revealing that VREx and IRM could rely on spurious features when using a GAT-like GNN (Section 2.2) in node-level OOD scenarios, revealing a key challenge of invariant learning on graphs.

Wepropose CIA and theoretically prove its effectiveness in learning invariant representations on graphs (Section 3.1). To adapt CIA to node-level OOD scenarios where environment labels are unavailable, we further propose CIA-LRA that requires no environment labels or complex environmental partitioning processes to achieve invariant learning (Section 3.2), with theoretical guarantees on its generalization performance (Section 4).

We evaluate CIA and CIA-LRA on the Graph OOD benchmark (GOOD) [Gui et al., 2022] on GAT and GCN [Kipf and Welling, 2016]. The results demonstrate CIA's superiority over

non-graph invariant learning methods, and CIA-LRA achieves state-of-the-art performance (Section 5.2).

We leave comparisons of our method and existing node-level OOD works in Appendix A.3.

2 Dissecting Invariant Learning on Graphs

̸

For OOD node classification, we are given a single training graph G = ( A,X,Y ) containing N nodes V = { v i } N i =1 from multiple training environments e ∈ E tr . A ∈ { 0 , 1 } N × N is the adjacency matrix, A i,j = 1 iff there is an edge between v i and v j . X ∈ R N × D are node features. The i -th row X i ∈ R D represents the original node feature of v i . Y ∈ { 0 , 1 , ..., C -1 } N are the labels, C is the number of the classes. Denote the subgraph containing nodes of environment e as G e = ( A e , X e , Y e ) , which follows the distribution p e . Let A e ∈ { 0 , 1 } N e × N e and D e be the adjacency matrix and the diagonal degree matrix for nodes from environment e respectively, where D e ii = ∑ j =1 A e ij , N e is the number of samples in e . Denote the normalized adjacency matrix as ¯ A e = ( D e + I N e ) -1 2 A e ( D e + I N e ) -1 2 , I N e is the identity matrix. Let ˜ A e = ¯ A +( D e + I N e ) -1 2 I N e ( D e + I N e ) -1 2 . Suppose the unseen test environments are e ′ ∈ E te. The test distribution p e ′ = p e ∀ e ′ ∈ E te , ∀ e ∈ E tr. OOD generalization aims to minimize the prediction error over test distributions.

To understand the obstacles in invariant learning on graphs, we start by examining whether IRMv1 (practical implementation of the original challenging IRM objective, proposed by Arjovsky et al. [2020]) and VREx can be successfully transferred to node-level graph OOD tasks. Their objectives are as follows:

where w and ϕ denote the classifier and feature extractor, respectively. L is the cross-entropy loss. β is some hyperparameter.

2.1 A Causal Data Model on Graphs

Data generation process. We construct an SCM to characterize two kinds of distribution shifts: concept shift (Figure 1a) and covariate shift (Figure 1b). C and S denote unobservable causal/spurious latent variables that affect the generation of the graph G , and dashed E are environmental variables usually unobservable. We consider a simple case that each node v in environment e has a 2-dim feature [ x 1 v , x 2 v ] ⊤ , x 1 v , x 2 v ∈ R . Denote the concatenated node features of all nodes in e as X 1 ∈ R N e × 1 and X e 2 ∈ R N e × 1 corresponding to x 1 v and x 2 v , respectively. For the SCM in Figure 1a 3 , the data generation process of environment e is

𝐺

𝑌

𝐶

𝑆

𝐸

(a)

concept

shift

𝐺

𝑌

𝐶

𝑆

𝐸

(b)

covariate

shift

where k ∈ N + represents the "depth" (number of hops) of the causal pattern, and m ∈ N + is the depth of the ego-graph determining the spurious node features. n 1 ∈ R N e × 1 and n 2 ∈ R N e × 1 represent random Gaussian noise. ϵ e stands for an environmental variable, causing the spurious correlations between X e 2 and Y . A detailed description of the model is in Appendix F.1.

How the proposed model considers both node feature shifts and structural shifts? X 1 represent invariant node features causing Y e . X e 2 denotes spurious node features that vary with environments. As for structural shifts, we consider an environmental ˜ A e in Equation (2), which means the structure can vary with environments. For example, there could be a spurious correlation between certain structures and the label; or, the graph connectivity or size may shift [Buffelli et al., 2022, Xia et al.,

2023]. We model the invariant structural feature as the structure of a node's k -layer neighboring ego-graph. See Appendix F.2 for more discussions of the structural shifts.

We also have the following assumption about the stability of the causal patterns across environments:

Assumption 2.1. (Stability of the causal patterns) The k -layer causal pattern in Equation (2) is invariant across environments for every class c .

A simple multi-layer GNN. Consider a L -layer GNN f parameterized by Θ = { θ 1 , θ 2 , θ 1 1 ( l ) , θ 2 1 ( l ) , θ 1 2 ( l ) , θ 2 2 ( l ) } , l = 1 , 2 , ..., L -1 :

where θ i and θ i j ( l ) are scalars for i, j ∈ { 1 , 2 } , ∀ l . H ( l ) i ∈ R N × D are GNN representations.

Remark. In this GNN, we keep only the top-layer weight matrix [ θ 1 θ 2 ] ⊤ , and let the weight matrix of lower layers 1 , ..., L -1 be an identity matrix. This architecture resembles an SGC [ ? ]. θ 1 and θ 2 are for invariant/spurious features, respectively. θ 1 1 ( l ) , θ 1 2 ( l ) are weights for aggregating features from neighboring nodes and θ 2 1 ( l ) , θ 2 2 ( l ) are weights for features of a centered node, this setup can be seen as a GAT. When all lower-layer parameters equal 1, the GNN degenerates to a GCN (see Appendix F.2 for justification of the choice of the GNN).

We consider a regression problem that we aim to minimize the MSE loss over all environments E e [ R ( e )] = E e [ E n 1 ,n 2 [ ∥ f Θ ( A e , X e ) -Y e ∥ 2 2 ]] . The optimal invariant parameter set Θ ∗ is

In Equation (4), the GNN parameters for spurious features (line 2) is zero, which means it removes spurious node features. Also, it learns the correct depth k of the causal pattern ˜ A k X 1 (line 3-4).

2.2 Intriguing Failure of VREx and IRM on Graphs

Now we are ready to present the failure cases in this node-level OOD task: optimizing IRMv1 and VREx induces a model that relies on spurious features X e 2 to predict, leading to poor OOD generalization. To illustrate that this failure arises from the graph data, we first prove that IRMv1 and VREx can learn invariant features under the non-graph version of SCM of Equation (2).

Proposition 2.2. (IRMv1 and VREx can learn invariant features for non-graph tasks, proof is in Appendix G.1.1.) For the non-graph version of the SCM in Equation (2),

VREx and IRMv1 can learn invariant features when using a linear network: f ( X ) = θ 1 X 1 + θ 2 X 2 .

Now we will give the main theorem revealing the failure of VREx and IRMv1 on graphs.

̸

Theorem 2.3. (IRMv1 and VREx will use spurious features on graphs, informal) Under the SCM of Equation (2), the IRMv1 and VREx objectives have non-unique solutions for parameters of the GNN (3), and there exist solutions that use spurious features, i.e. θ 2 = 0 .

Intuitive illustration of the failure. From Theorem 2.3, we find that the main reason for the failure lies in the message-passing mechanism in representation learning. Let's provide some key steps in the proof of the IRMv1 case as an illustration. For the non-graph OOD task Equation (5), we can verify that when IRMv1 objective is solved, i.e. ∇ w R ( e ) = 0 for all e , the invariant solution θ 2 = 0

leads to ( θ 1 ) 2 X ⊤ 1 X 1 -θ 1 X ⊤ 1 X 1 = 0 , which can be satisfied when θ = 1 . However, in the graph case, θ 2 = 0 leads to

̸

where ( ˜ A e ) s X 1 is the learned representation of the GNN, 0 < s ≤ L . k is the depth of the causal pattern. Now we explain why the invariant solution may not hold on graphs. When the depth of the learned aggregation pattern s = k , Equation (6) cannot hold for a fixed θ 1 (since θ 1 will depend on e then). This means that identifying the underlying structure of the causal pattern imposes additional difficulty for invariant learning. Moreover, even if the GNN can learn representations of different depths (e.g. GAT) 4 , the proof in Appendix G.1.3 shows that IRM failed to provide sufficient supervision to optimize the aggregation parameters θ i j , i, j ∈ { 1 , 2 } such that s = k . A similar analysis holds for VREx. In general, successful invariant learning on graphs requires capturing both invariant node features and the structure of the causal pattern, while methods like IRM and VREx that solely enforce a cross-environment invariance at the loss level 5 may not be able to achieve these goals. The formal versions and proof are in Appendix G.1.3 (IRM) and G.1.2 (VREx).

3 The Proposed Methods

3.1 Cross-environment Intra-class Alignment

Inspired by the examples of VREx and IRMv1, we aim to introduce additional invariance regularization to guide the model in identifying the underlying invariant node features and structures. We propose CIA (Cross-environment Intra-class Alignment), which aligns the representations from the same class across different environments. Intuitively, since such node pairs share similar invariant features and causal pattern structures while differ in spurious features, aligning their representations will help achieve our targets. Denote the representation of node i as ϕ Θ ( i ) and the classifier parameterized by θ h as h θ h CIA's objective is:

̸

̸

where Ω e,e ′ c = { ( i, j ) | i = j ∧ Y e i = Y e ′ j = c ∧ E i = e, E j = e ′ } is the set of nodes with same label c and from two different environments. L is the cross-entropy loss. D is some distance metric and we adopt L-2 distance. Now we prove that CIA can learn invariant representations regardless of the unknown causal patterns:

Theorem 3.1. Under the SCM of Equation (2) and Assumption 2.1, optimizing the CIA objective will lead to the optimal invariant solution Θ ∗ in Equation (4) for parameters of the GNN (3).

The proof is in Appendix G.1.4. By enforcing class-conditional invariance, which is not considered in VREx and IRMv1, CIA overcomes the above obstacles and eliminates spurious features. As long as a GNN has the capacity to adaptively learn the true depth of the causal pattern (such as the one considered in Equation (3)) or a GAT), CIA can identify the invariant causal pattern.

Remark. One might note that the objective of CIA is analogous to MatchDG [Mahajan et al., 2021]. However, we are the first to adapt such an idea to node-level OOD tasks and theoretically reveal its advantage. In Appendix A.1, we compare our extension and the original MatchDG in detail.

3.2 Localized Reweighting Alignment: an Adaptation to Graphs without Environment Labels

So far, we have theoretically and empirically validated CIA's advantage on graphs, but it still requires environmental labels that are challenging to obtain in most node classification tasks [Wu et al., 2021, Liu et al., 2023, Li et al., 2023a]. In this section, we propose CIA-LRA (Localized Reweighting Alignment) that realizes CIA's objective without using environment labels by identifying node pairs with significant/minor differences in spurious/invariant features and then aligning their representations. As illustrated in Figure 2, CIA-LRA mainly incorporates three components:

̸

Localized alignment

Representation

alignment

𝑡 = 2

GNN

encoder

𝜙Θ

Training

graph

𝐺(𝐴, 𝑋)

Representations

of all nodes

𝜙 ∈

ℝ 𝑁×𝐷

𝑀𝑖,𝑗

= Sigmoid(𝜙𝜃𝑚

𝑖

⊤ 𝜙𝜃𝑚

(𝑗))

𝐴𝑚 = 𝐴 ⊙ 𝑀 ∈ ℝ 𝑁×𝑁

Invariant

Subgraph

Extractor

𝜙𝜃𝑚

Classifier

ℎ𝜃ℎ

CIA-LRA loss:

ቚ

ℒ𝐶𝐼𝐴-𝐿𝑅𝐴 =

𝔼

𝑐

𝔼

𝑖,𝑗∈Ω 𝑐 (𝑡)

𝒘𝒊,𝒋

𝜙Θ

𝑖

-𝜙Θ 𝑗 ቚ

2

•

𝜙Θ

𝑖

= 𝜙Θ

𝐴𝑚,𝑋 [𝑖, : ]

representation of

node

𝑖

•

Reweighting alignment:

𝑤𝑖,𝑗

•

Localized alignment:

Ω𝑐(𝑡)

Cross-entropy loss:

ℒ𝑐𝑒

= ℒ(ℎ𝜃ℎ

∘ 𝜙Θ

𝐴𝑚,𝑋 ,𝑌)

Nodes of

different

classes

A significant discrepancy in

Homo.

NLD

A minor discrepancy in

Hetero.

NLD

Pair 1

Pair 2

Pair 3 Pair 4

Pair 5

Pair 6

…

60% 10% 30%

60% 40% 0%

A significant discrepancy in

Hetero.

NLD

A minor discrepancy in

Homo.

NLD

Reweighting alignment

Weights of different node pairs for alignment

Neighborhood of

𝑖

Neighborhood of

𝑗

Total loss:

ℒ𝑐𝑒

+𝜆 ℒ𝐶𝐼𝐴-𝐿𝑅𝐴

Class 1

Class 2

Class 3

Localized reweighting alignment

Reweighted representation alignment

No alignment for non-local pairs

The Representation of a node

𝜙 𝑖

∈ ℝ 1×𝐷

𝐺(𝐴𝑚,𝑋)

̸

Localized alignment. To avoid learning a collapsed representation of invariant features, it is crucial to align node pairs that share similar invariant features. To achieve this, we align nodes close to each other (about 2 to 6 hops). This is based on two observations. First, we observe that spurious features tend to exhibit larger changes within local graph areas than invariant ones, and nodes from the same class that are too distant may differ more in their invariant features than the closer ones (evidence in Appendix D.4). This is because invariant features are generally more stable than spurious ones, according to Chen et al. [2022], Schölkopf et al. [2021]. Second, we empirically find that alignment over an extensive range or too many nodes yields only marginal performance improvements, or even leads to performance degradation (see Appendix D.1), while increasing computational costs. This may also be attributed to the feature collapse caused by excessive alignment of too many node pairs. Formally, the local pairs are defined as Ω c ( t ) = { ( i, j ) | i = j ∧ Y i = Y j = c ∧ d ( i, j ) ≤ t } , where d ( i, j ) represents the the shortest path length from node v i to v j , t ∈ N + is a hyperparameter. Also, we propose to assign smaller weights to pairs more distant away from each other.

̸

Reweighting Alignment. Since environment labels are unavailable, we need a metric to reflect the distribution of the spurious and invariant representations so that node pairs with significant/small differences in spurious/invariant features can be identified. Since we assume the causal patterns of the same class are similar (Assumption 2.1), the label distribution of homophilic (i.e., sameclass) neighbors directly affects the invariant features aggregated to the centered node (empirical evidence in Appendix D.5.2). Therefore, pairs with smaller differences in the ratio of homophilic neighbors should be assigned larger weights for alignment. The ratio discrepancy can be calculated as follows: r same ( c ) i,j = ∣ ∣ r c i -r c j ∣ ∣ , where r c i is the ratio of the neighbors of v i of class c within L hops ( L is the number of layers of the GNN). As for spurious features, we utilize Heterophilic (i.e., different-class) Neighborhood Label Distribution (HeteNLD) as a measurement, as it affects the two kinds of main distributional shifts on graphs: 1) environmental node feature shifts, and 2) Neighborhood Label Distribution (NLD) shift (both empirically verified in Appendix D.5.1). HeteNLD determines the first kind of shift when correlations exist between labels and spurious node features, e.g., concept shift. The second kind, NLD shift, which is affected by HeteNLD, can be regarded as a structural shift as the discrepancy in neighborhood distribution will induce a gap in the aggregated representations (Theorem 4.4 shows this shift increases OOD error). Although aligning the representations significantly differing in homophilic neighbor ratio mitigates these two kinds of shifts, it also leads to the collapsed invariant representations and suboptimal performance (Table 4 shows this effect). Therefore we assign larger weights to the pair with a larger discrepancy in HeteNLD when alignment. The discrepancy in HeteNLD is calculated as follows: r diff ( c ) i,j = ∑ c ′ = c ∣ ∣ ∣ r c ′ i -r c ′ j ∣ ∣ ∣ .

Invariant Subgraph Extractor. In Section 2.1, the structural invariant features are defined as the k -hop neighboring ego-graph for ease of analysis. However, in practice, the invariant structure may merely be a subgraph of the neighborhood nodes. To better capture the invariant ego subgraph, we train an invariant subgraph extractor inspired by Li et al. [2023a]. Concretely, we learn an auxiliary GNN encoder ϕ θ m (parameterized by θ m ) to predict an soft edge mask M ∈ R N × N , and then apply it during training and test:

Now we are ready to present the formal objective of CIA-LRA:

In Equation (9), L is the cross-entropy loss, Ω c ( t ) , r diff ( c ) i,j , r same ( c ) i,j and d ( i, j ) are defined in the above analysis. In practice, we use CIA-LRA as a regularization term added to the cross-entropy loss with a weight λ as a hyperparameter. The detailed implementation of CIA-LRA is in Appendix E.

4 Theoretical Justification: an OOD Generalization Error Bound

Now will derive an OOD generalization error bound to show that optimizing CIA-LRA can minimize OOD error. To achieve this, we adopt a PAC-Bayesian framework following Ma et al. [2021] and establish a Contextual Stochastic Block Model (CSBM, [Deshpande et al., 2018]) for OOD multi-classification. The proposed CSBM-OOD is as follows (more discussions are in Appendix F.4):

̸

Definition 4.1. (CSBM-OOD). For node i of class c from environment e , its node feature x i ∈ R D consists of two parts, x i = [ x ⊤ inv ; x ⊤ sp ] ⊤ , where x inv ∈ R D 2 sampled from the Gaussian distribution N ( µ c , σ 2 I ) is the invariant feature and x sp ∈ R D 2 sampled from the N ( µ e c , σ 2 I ) is the spurious feature. 6 Suppose { µ c } and { µ e c } for all c and e form sets of orthonormal basis. We use p hm i to denote the homophilic ratio of node i 's one-hop neighbors and use p ht i ( c ′ ) to denote the heterophilic ratio of node i 's one-hop neighbors of class c ′ ( c = c ′ ). We assume Pr ( y i = c ) are the same for all classes c .

The GNN model used for deriving the error bound (following Ma et al. [2021]): The GNN model has a 1-layer mean aggregation g that outputs the aggregated feature g i ∈ R D for node i . The GNN classifier h on top of g is a ReLU-activated L -layer MLP with W 1 , ..., W L as parameters for each layer. h is from a function family H . The prediction for node i is h i ∈ R C with h i [ c ] representing the predicted logit for class c . Denote the largest width of all the hidden layers as b .

Notations. Denote nodes of environment e as V e . We consider the error of generalizing from a mixed training environment e tr to any test environment e te ∈ E te, where V e tr := ∪ e ∈E tr V e represents all training nodes. To guarantee the generalization, we need to characterize the distance between V e te and V e tr : define ϵ e te ,e tr = max j ∈ V e te min i ∈ V e tr ∥ g i -g j ∥ 2 as the aggregated feature distance between the training and test subgroup. Define the number of nodes in environment e as N e . We consider the margin loss of environment e that is used by Ma et al. [2021], Mao et al. [2023]: ̂ L γ ( h ) := 1 e ∑ 1 [ h i [ y i ] ≤ γ +max c = y i h i [ c ]] .

̸

Now we introduce some assumptions adapted from Ma et al. [2021] that are used in our proof.

Assumption 4.2. (Equal-sized and disjoint near sets, adapted from Assumption 2 of Ma et al. [2021]) For each node v i ∈ V e tr , define V ( i ) e te := { j ∈ V e te | ∥ g i -g j ∥ 2 ≤ ϵ e tr ,e te } . For any test environment e te , assume V ( i ) e te of each v i ∈ V e tr are disjoint and have the same size N e te ∈ N + .

Assumption 4.3. (concentrated expected loss difference, adapted from Assumption 3 of Ma et al. [2021]) Let P be a distribution on H , defined by sampling the vectorized MLP parameters from

N ( 0 , σ 2 I ) for some σ 2 ≤ ( γ/ 8 ϵ e te ,e tr ) 2 /L 2 b ( λN -α e tr +ln2 bL ) . For any L layer GNN classifier h ∈ H with model parameters W h 1 , . . . , W h L , define T h := max l =1 ,...,L ∥ W l ∥ 2 . Assume that there exists some 0 < α < 1 4 satisfying

Now we present the node-level OOD generalization bound (proof in Appendix G.3):

˜

Theorem 4.4. (Subgroup OOD Generalization Bound for GNNs, informal) . Let h be any classifier in a function family H with parameters { ˜ W l } L l =1 . Under Assumption 4.2 and 4.3, for any e te ∈ E te , γ ≥ 0 , and large enough N e tr , there exist 0 < α < 1 4 with probability at least 1 -δ , we have

̸

̸

where p ht i ( c ′ | c ) is the ratio of heterophilic neighbors of class c ′ when y i = c , B e te = max i ∈ V e tr ∪ V e te ∥ g i ∥ 2 is the maximum feature norm, V ( i ) e te := { j ∈ V e te | ∥ g i -g j ∥ 2 ≤ ϵ e tr ,e te } . const is a constant depending on α , δ , and γ .

The observations from Theorem 4.4 is summarized as follows: Term (a) reflects the separability of the original features of different classes | [( µ c -µ c ′ ) ⊤ ; ( µ e te c -µ e te c ′ ) ⊤ ] | and the distance of the aggregated features between the training and test set ϵ e te ,e tr . The former factor is the nature of the dataset itself. Term (b) is the distributional discrepancy between the training and test subgroups, caused by the distribution shifts in spurious features. When there exist correlations between labels and spurious features, CIA-LRA can minimize this term by minimizing the representation distance of node pairs with large discrepancy in the label distribution of heterophilic neighbors 7 . Term (c) measures the shift in HeteNLD between the training and test subgroups, representing the OOD error caused by the shift in the aggregated features of the same class. CIA-LRA minimizes this term by enforcing stronger alignment on pairs with greater HeteNLD differences.

5 Experiments

5.1 Experiment Setup

We run experiments using 3-layer GAT and GCN on GOOD [Gui et al., 2022], a graph OOD benchmark. We reported the results on both covariate shift and concept shift. The detailed experimental setup and hyperparameter settings are in Appendix C. We compare our methods with the following algorithms: ERM [Vapnik, 1999]; traditional invariant learning methods: IRM , VREx , GroupDRO [Sagawa et al., 2019], Deep Coral [Sun and Saenko, 2016], IGA [Koyama and Yamaguchi, 2020]; graph OOD methods: EERM , SRGNN , CIT [Xia et al., 2023], CaNet [Wu et al., 2024]; graph data augmentation: Mixup [Wang et al., 2021a], GTrans [Jin et al., 2022].

5.2 Main Results of OOD Generalization

Table 2 reports the main OOD generalization results. The observations are summarized as follows: 1) CIA-LRA improves the best baseline methods by 2.44% and 3.23% on GAT and GCN, respectively,

Table 2: OOD test accuracy (%). Our methods are marked in bold . The best and second-best results are shown in bold and underlined, respectively. The values in parentheses are standard deviations. 'OOM' denotes Out Of Memory. Results of EERM on Cora GCN marked by '*' are from Gui et al. [2022] since we got OOM during our running.

achieving state-of-the-art performance. 2) CIA outperforms IRM and VREx on all splits, which validates our theoretical findings in Section 2. Notably, it performs best among the non-graph-specific methods. 3) CIA-LRA improves CIA in most cases. This suggests that our reweighting strategy can enhance generalization on graphs even without environment labels. 4) MatchDG outperforms IRM and VREx on 12 out of 16 splits but underperforms CIA on average (averaged over 16 splits except on Arxiv, CIA: 57.56, MatchDG: 56.73).

5.3 CIA can be Integrated into and Improve other Graph-OOD Methods

We replace VREx with CIA in the loss function of EERM to show that CIA can improve generalization in a plug-andplay manner. Table 3 shows that this improves original EERM by a large margin or has comparable performances, indicating the performance of node-level OOD algorithms can be limited by VREx.

5.4 Empirical Understanding of the Role of CIA-LRA

Table 3: By replacing VREx in EERM with CIA (marked as EERM-CIA ), the performance is significantly improved.

A synthetic dataset. We construct a synthetic dataset (mentioned in Section 1) to validate the role of each module in CIA-LRA in eliminating spurious features and preventing the collapse of invariant representations. We generate a random graph and create a 4-class OOD classification task. Each node has a 4-dim feature, with the first/last two dimensions representing invariant/spurious features (details in Appendix C.3), so we can disentangle the learned invariant and spurious representations.

of CIA-LRA. Results are

Concept

0

50

100

150

2

5

0.001

2

5

0.01

2

5

0.1

2

CIA

CIA-LRA

CIA-LRA w/o r same

Epoch

Invariant rep. var.

0

50

100

150

0.25

0.3

0.35

CIA

CIA-LRA

CIA-LRA w/o r same

CIA-LRA w/o r diff

Epoch

OOD acc. (%)

0

50

100

150

0

5

10

CIA

CIA-LRA

CIA-LRA w/o r diff

Epoch

Spurious rep. norm

Covariate

0

50

100

150

0.25

0.3

0.35

0.4

CIA

CIA-LRA

CIA-LRA w/o r same

CIA-LRA w/o r diff

Epoch

OOD acc. (%)

0

50

100

150

5

0.001

2

5

0.01

2

5

0.1

2

CIA

CIA-LRA

CIA-LRA w/o r same

Epoch

Invariant rep. var.

0

50

100

150

0

100

200

300

400

CIA

CIA-LRA

CIA-LRA w/o r diff

Epoch

Spurious rep. norm

Algorithms

Acc. (%)

IRM

61.14

VREx

61.32

no

r

diff

(

c

)

i,j

63.22

no

1

r

same

(

c

)

i,j

63.91

no

1

d

(

i,j

)

63.64

no

M

63.39

use

r

same

(

c

)

i,j

in numerator

62.70

full CIA-LRA

65.42

Table 4:

Ablation study

0.35

0.25

0

50

100

Epoch

1) Aligning the large discrepancy in HeteNLD helps to eliminate spurious features on concept shift and improves generalization. As evident from the right column, incorporating r diff diminishes the norm of spurious features under concept shift. For covariate shift, while r diff will not remove environmental spurious features due to their independence from labels, it still helps generalization since it reduces the error caused by shifts in HeteNLD as predicted by Theorem 4.4. 2) CIA-LRA alleviates collapse of causal representation that CIA may suffer when adopting a substantial λ . When using a large λ ( = 0 . 5 ), the performance of CIA deteriorates to the level of random guessing (25%) after approximately 50 epochs. In contrast, CIA-LRA sustains its accuracy at a high level because it avoids excessive alignment by aligning only local pairs and reweighting (further evidence in Appendix D.2). The mid column shows that the invariant features learned by CIA progressively collapse, even if CIA removes most spurious features (right column). 3) Maintaining the discrepancy in homophilic neighboring label distribution r same helps keep the variance of the invariant representation, slightly improving performance.

Ablation study. We also conduct ablation studies on CIA-LRA. Table 4 shows that removing any module causes a significant performance drop, demonstrating the effectiveness of each module.

5.5 Effects of the Hyperparameters of CIA-LRA

This section analyzes the effect of λ and t of CIA-LRA. Figure 4 shows that the test accuracy increases with λ when λ ≤ 0 . 5 . Too small t leads to a sub-optimal performance due to insufficient regularization from aligning only a few pairs. Also, most parameter combinations outperform the baseline methods, indicating that CIA-LRA leads to consistently superior performance. Additional studies of the effects of λ and t are in Appendix D.2 and D.3, respectively.

6 Conclusion

0.5

0.1

0.05

0.01

0.005

2

3

4

5

6

Hops

t

64.67

63.66

63.44

62.24

61.96

65.48

64.27

63.62

62.43

62.10

65.97

64.83

63.62

62.29

61.83

66.51

65.08

63.53

62.10

61.99

67.08

65.27

63.36

62.32

62.07

Test acc. (%)

61.5

62.0

62.5

63.0

63.5

64.0

64.5

65.0

65.5

0.5

0.1

0.05

0.01

0.005

2

3

4

5

6

Hops

t

56.58

56.04

55.63

55.06

54.91

56.94

55.88

55.70

55.29

55.06

56.71

56.73

55.80

55.17

55.07

57.75

56.88

56.30

55.60

55.11

57.95

57.18

56.18

55.79

55.12

Test acc. (%)

55.0

55.5

56.0

56.5

57.0

57.5

58.0

In this work, by theoretically dissecting the failure of IRM and VREx in node-level graph OOD tasks, we attribute it to the difficulty in identifying the graph-specific causal pattern structures. To address this, we propose CIA with additional class-conditional invariance constraints and its environmentlabel-free variant CIA-LRA tailored for graph OOD scenarios. Further theoretical and experimental results validate their efficacy. Notably, CIA can be incorporated in other graph OOD frameworks, serving as a better invariant learning objective than the widely-used VREx on graphs.

150

References

Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. In ICML , 2020.

David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapolation (rex). In ICML , 2021.

Manh-Ha Bui, Toan Tran, Anh Tran, and Dinh Phung. Exploiting domain-specific features to enhance domain generalization. In NeurIPS , 2021.

Alexandre Rame, Corentin Dancette, and Matthieu Cord. Fishr: Invariant gradient variances for out-of-distribution generalization. In ICML , 2022.

Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In ICCV , 2019.

Divyat Mahajan, Shruti Tople, and Amit Sharma. Domain generalization using causal matching. In ICML , 2021.

Dinghuai Zhang, Kartik Ahuja, Yilun Xu, Yisen Wang, and Aaron Courville. Can subnetwork structure be the key to out-of-distribution generalization? In ICML , 2021a.

Ruoyu Wang, Mingyang Yi, Zhitang Chen, and Shengyu Zhu. Out-of-distribution generalization with causal invariant transformations. In CVPR , 2022a.

Mingyang Yi, Ruoyu Wang, Jiacheng Sun, Zhenguo Li, and Zhi-Ming Ma. Breaking correlation shift via conditional invariant regularizer. In ICLR , 2022.

Qixun Wang, Yifei Wang, Hong Zhu, and Yisen Wang. Improving out-of-distribution generalization by adversarial training with structured priors. In NeurIPS , 2022b.

Shiji Xin, Yifei Wang, Jingtong Su, and Yisen Wang. On the connection between invariant learning and adversarial training for out-of-distribution generalization. In AAAI , 2023.

Qitian Wu, Hengrui Zhang, Junchi Yan, and David Wipf. Handling distribution shifts on graphs: An invariance perspective. In ICLR , 2021.

Haoyang Li, Ziwei Zhang, Xin Wang, and Wenwu Zhu. Invariant node representation learning under distribution shifts with multiple latent environments. ACM Transactions on Information Systems , (1):1-30, 2023a.

Yang Liu, Xiang Ao, Fuli Feng, Yunshan Ma, Kuan Li, Tat-Seng Chua, and Qing He. Flood: A flexible invariant learning framework for out-of-distribution generalization on graphs. In SIGKDD , 2023.

Shengyu Zhang, Kun Kuang, Jiezhong Qiu, Jin Yu, Zhou Zhao, Hongxia Yang, Zhongfei Zhang, and Fei Wu. Stable prediction on graphs with agnostic distribution shift. arXiv preprint arXiv:2110.03865 , 2021b.

Qin Tian, Wenjun Wang, Chen Zhao, Minglai Shao, Wang Zhang, and Dong Li. Graphs generalization under distribution shifts. arXiv preprint arXiv:2403.16334 , 2024.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In ICLR , 2018.

Qitian Wu, Fan Nie, Chenxiao Yang, Tianyi Bao, and Junchi Yan. Graph out-of-distribution generalization via causal intervention. arXiv preprint arXiv:2402.11494 , 2024.

Xiner Li, Shurui Gui, Youzhi Luo, and Shuiwang Ji. Graph structure and feature extrapolation for out-of-distribution generalization. arXiv preprint arXiv:2306.08076 , 2023b.

Shurui Gui, Xiner Li, Limei Wang, and Shuiwang Ji. Good: A graph out-of-distribution benchmark. In NeurIPS , 2022.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.

Davide Buffelli, Pietro Liò, and Fabio Vandin. Sizeshiftreg: a regularization method for improving size-generalization in graph neural networks. In NeurIPS , 2022.

Donglin Xia, Xiao Wang, Nian Liu, and Chuan Shi. Learning invariant representations of graph neural networks via cluster generalization. In NeurIPS , 2023.

Yongqiang Chen, Yonggang Zhang, Yatao Bian, Han Yang, MA Kaili, Binghui Xie, Tongliang Liu, Bo Han, and James Cheng. Learning causally invariant representations for out-of-distribution generalization on graphs. In NeurIPS , 2022.

Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE , (5):612-634, 2021.

Jiaqi Ma, Junwei Deng, and Qiaozhu Mei. Subgroup generalization and fairness of graph neural networks. In NeurIPS , 2021.

Yash Deshpande, Subhabrata Sen, Andrea Montanari, and Elchanan Mossel. Contextual stochastic block models. In NeurIPS , 2018.

Haitao Mao, Zhikai Chen, Wei Jin, Haoyu Han, Yao Ma, Tong Zhao, Neil Shah, and Jiliang Tang. Demystifying structural disparity in graph neural networks: Can one size fit all? In NeurIPS , 2023.

Vladimir N Vapnik. An overview of statistical learning theory. IEEE Transactions on Neural Networks , (5):988-999, 1999.

Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731 , 2019.

Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In ECCV Workshop , 2016.

Masanori Koyama and Shoichiro Yamaguchi. When is invariance useful in an out-of-distribution generalization problem? arXiv preprint arXiv:2008.01883 , 2020.

Yiwei Wang, Wei Wang, Yuxuan Liang, Yujun Cai, and Bryan Hooi. Mixup for node and graph classification. In WWW , 2021a.

Wei Jin, Tong Zhao, Jiayuan Ding, Yozen Liu, Jiliang Tang, and Neil Shah. Empowering graph representation learning with test-time graph transformation. arXiv preprint arXiv:2210.03561 , 2022.

Yuge Shi, Jeffrey Seely, Philip HS Torr, N Siddharth, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. arXiv preprint arXiv:2104.09937 , 2021.

Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In ICLR , 2020.

Junchi Yu, Jian Liang, and Ran He. Mind the label shift of augmentation-based graph ood generalization. In CVPR , 2023.

Haoyang Li, Ziwei Zhang, Xin Wang, and Wenwu Zhu. Learning invariant graph representations for out-of-distribution generalization. In NeurIPS , 2022a.

Qi Zhu, Natalia Ponomareva, Jiawei Han, and Bryan Perozzi. Shift-robust gnns: Overcoming the limitations of localized graph training data. In NeurIPS , 2021.

Yongqiang Chen, Yatao Bian, Kaiwen Zhou, Binghui Xie, Bo Han, and James Cheng. Rethinking invariant graph representation learning without environment partitions. In ICLR Workshop , 2023a.

Fredrik D Johansson, David Sontag, and Rajesh Ranganath. Support and invertibility in domaininvariant representations. In AISTATS , 2019.

Ling Yang, Jiayi Zheng, Heyuan Wang, Zhongyi Liu, Zhilin Huang, Shenda Hong, Wentao Zhang, and Bin Cui. Individual and structural graph information bottlenecks for out-of-distribution generalization. IEEE Transactions on Knowledge and Data Engineering , 2023.

Ying-Xin Wu, Xiang Wang, An Zhang, Xiangnan He, and Tat-Seng Chua. Discovering invariant rationales for graph neural networks. arXiv preprint arXiv:2201.12872 , 2022a.

Nianzu Yang, Kaipeng Zeng, Qitian Wu, Xiaosong Jia, and Junchi Yan. Learning substructure invariance for out-of-distribution molecular representations. In NeurIPS , 2022.

Tianrui Jia, Haoyang Li, Cheng Yang, Tao Tao, and Chuan Shi. Graph invariant learning with subgraph co-mixup for out-of-distribution generalization. arXiv preprint arXiv:2312.10988 , 2023.

Haoyang Li, Xin Wang, Ziwei Zhang, and Wenwu Zhu. Ood-gnn: Out-of-distribution generalized graph neural network. IEEE Transactions on Knowledge and Data Engineering , 2022b.

Yongqiang Chen, Yatao Bian, Kaiwen Zhou, Binghui Xie, Bo Han, and James Cheng. Does invariant graph learning via environment augmentation learn invariance? In NeurIPS , 2023b.

Xuexin Chen, Ruichu Cai, Kaitao Zheng, Zhifan Jiang, Zhengting Huang, Zhifeng Hao, and Zijian Li. Unifying invariance and spuriousity for graph out-of-distribution via probability of necessity and sufficiency. arXiv preprint arXiv:2402.09165 , 2024.

Shurui Gui, Meng Liu, Xiner Li, Youzhi Luo, and Shuiwang Ji. Joint learning of label and environment causal independence for graph out-of-distribution generalization. In NeurIPS , 2024.

David Burshtein, Vincent Della Pietra, Dimitri Kanevsky, and Arthur Nadas. Minimum impurity partitions. The Annals of Statistics , pages 1637-1646, 1992.

Bernhard Schölkopf. Causality for machine learning. In Probabilistic and Causal Inference: The Works of Judea Pearl , pages 765-804. 2022.

Yao Ma, Xiaorui Liu, Neil Shah, and Jiliang Tang. Is homophily a necessity for graph neural networks? In ICLR , 2022.

Jincheng Huang, Ping Li, Rui Huang, Na Chen, and Acong Zhang. Revisiting the role of heterophily in graph representation learning: An edge classification perspective. ACM Transactions on Knowledge Discovery from Data , 2023.

Xinyi Wu, Zhengdao Chen, William Wei Wang, and Ali Jadbabaie. A non-asymptotic analysis of oversmoothing in graph neural networks. In ICLR , 2022b.

Huayi Tang and Yong Liu. Towards understanding the generalization of graph neural networks. In ICML , 2023.

Hao Zhu and Piotr Koniusz. Simple spectral graph convolution. In ICLR , 2021.

Yifei Wang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Dissecting the diffusion process in linear graph convolutional networks. In NeurIPS , 2021b.

Yong Lin, Lu Tan, Yifan Hao, Honam Wong, Hanze Dong, Weizhong Zhang, Yujiu Yang, and Tong Zhang. Spurious feature diversification improves out-of-distribution generalization. arXiv preprint arXiv:2309.17230 , 2023.

Appendix

Table of Contents

A Additional Related Works

A.1 Invariant Learning for OOD Generalization

Invariant learning seeks to find stable features across multiple training environments to achieve OOD generalization [Arjovsky et al., 2020, Krueger et al., 2021, Bui et al., 2021, Rame et al., 2022, Shi et al., 2021, Wang et al., 2020, Mahajan et al., 2021, Wang et al., 2022a, Yi et al., 2022, Wang et al., 2022b]. IRM [Arjovsky et al., 2020] and VREx are two of the most well-known methods. The goal of IRM is to learn a representation that elicits a classifier to achieve optimality in all training environments. VREx [Krueger et al., 2021] reduces the variance of risks across training environments to improve robustness against distribution shifts. Mahajan et al. [2021] argues that the invariant features can also change across environments. Hence, they proposed MatchDG to align the representations of the so-called same "object" of different environments. To the best of our knowledge, we are the first work to theoretically analysis the limitations of IRM and VREx in OOD node classification problems.

It's worth emphasizing that although the CIA objective is similar to MatchDG [Mahajan et al., 2021], our extensions of MatchDG on graphs are non-trivial:

1) The extensions from MatchDG to CIA are mainly theoretical.

We extend the idea of MatchDG to the node-level OOD task by providing a theoretical characterization of CIA's working mechanism on graphs (Theorem 3.1), revealing its superiority in node-level OOD scenarios for the first time.

We establish the connection between node-level OOD error and two kinds of distributional shifts: (1) environmental node feature shifts and (2) the heterophilic neighborhood label distribution shifts (see Theorem 4.4), giving further explanation of CIA and CIA-LRA's performance gain in node-level OOD generalization.

2) The methodological extensions of CIA-LRA:

It identifies the node pairs with significant differences in spurious features without using environment labels, providing a new perspective that the widely adopted environment partition paradigm Wu et al. [2021], Yu et al. [2023], Li et al. [2023b,a, 2022a], Liu et al. [2023] may not be necessary for node-level OOD generalization. One can remove spurious features by leveraging neighboring label distributions (analysis in Section 3.2), shedding light on the role of neighborhood label distribution as compensation for the absence of environment labels.

It's the first node-level OOD method explicitly considering the OOD error caused by shifts in heterophilic neighborhood label distribution, (pointed out by term (c) in Theorem 4.4). Such shifts can be regarded as a kind of structural shift that has been first observed in Mao et al. [2023].

It's the first node-level OOD method using homophilic neighborhood label distribution to reflect the find-grained distribution of the invariant features, avoiding the collapse of invariant features.

A.2 Graph-OOD Works Using VREx and Similar Variants

We summarize the graph-OOD works that include VREx or IGA as part of the training objectives below. Node-level works, which can be covered by our analysis:

EERM [Wu et al., 2021], Equation (5):

where g w ∗ k ( G ) is the generated k -th environment.

INL [Li et al., 2023a], Equation (8):

where E infer is the inferred environment label.

FLOOD [Liu et al., 2023], Equation (12):

Zhang et al. [2021b] Equation (10):

as they claimed: "We note that minimizing the pair-wise distance of losses as defined in Equation 10 is equivalent to minimizing the variance of losses."

GLIDER [Tian et al., 2024], Equation (9):

Graph-level works, which are not covered in our analysis:

LiSA

where g ∗ i are the augmented training subgraphs of representing different environments.

G-splice [Li et al., 2023b], Equation (6):

where E A are the augmented environments.

GIL [Li et al., 2022a], Equation (8):

where E infer is the inferred environment label.

A.3 Comparison with Existing Node-level OOD Generalization Works

Among the graph OOD methods, one line of research focuses on the node-level OOD generalization [Wu et al., 2021, Liu et al., 2023, Zhu et al., 2021, Li et al., 2023a, Xia et al., 2023]. We summarize the drawbacks of previous node-level OOD methods as follows. 1) it is hard for environmentinference-based methods to generate reliable environments. To generate environments, FLOOD [Liu et al., 2023] uses random data augmentation that lacks sufficient prior. EERM [Wu et al., 2021] generates environments by maximizing loss variance, which may not necessarily enlarge differences in spurious features across environments that have been proven to be crucial for invariant learning [Chen et al., 2023a]. Also, the adversarial learning of its environment generation process may lead to unstable performance (Table 2) and high training costs. INL [Li et al., 2023a] relies on an estimated invariant ego-graph of each node, whose quality could significantly affect performance. Moreover, all these methods need to manually specify the number of environments, which could be inaccurate. 2) previous node-level invariant learning objectives also have some limitations. For instance, Zhang et al. [2021b], Wu et al. [2021], Liu et al. [2023], Li et al. [2023a], Tian et al. [2024] use VREx Krueger et al. [2021] as their invariance regularization. However, we theoretically prove its potential failure on node-level OOD tasks in Section 2.2. SRGNN [Zhu et al., 2021] only aligns the marginal distribution p ( X ) between the biased training distribution and the given unbiased distribution, which has been proved to have failure cases [Johansson et al., 2019]. 3) some work are based on intuitive guidelines and lacks theoretical guarantees on the OOD generalization performance [Liu et al., 2023, Zhu et al., 2021, Yang et al., 2023]. Our proposed CIA-LRA achieves invariant learning without complex environment inference that could be unstable through a representation alignment manner. Additionally, we provide theoretical guarantees for our invariant learning objective and empirically validate its working mechanism.

Comparison between our Theorem 2.3 and Theorem 1 of Wu et al. [2021] It's worth mentioning that Theorem 1 of Wu et al. [2021] proves min Θ V e [ R ( e )] will min I ( y, e | z ) , where q ( z | x ) is the induced distribution by encoder ϕ . This seems to conflict with Theorem 2.3. This is because the upper bound derived in Wu et al. [2021] I ( y, e | z ) ≤ D KL ( q ( y | z ) ∥ E e [ q ( y | z )]) ≤ V e [ R ( e )] is not tight. Thus, minimizing the variance of loss across training environments does not necessarily minimize mutual information between the label and the environment given the learned representation.

A.4 Graph-level OOD Generalization

There has been a substantial amount of work focusing on the OOD generalization problem on graphs. The vast majority have centered on graph classification tasks[Chen et al., 2022, Wu et al., 2022a, Li et al., 2022a, 2023b, Yang et al., 2022, Yu et al., 2023, Chen et al., 2023a, Buffelli et al., 2022, Jia et al., 2023, Li et al., 2022b, Chen et al., 2023b, 2024, Gui et al., 2024]. Most works aim at identifying the invariant subgraph of a whole graph through specific regularization so that the model can use it when inference. Compared to node-level OOD generalization, the graph-level one is more akin to traditional OOD generalization, as the individual samples (graphs) are independently distributed. We focus on the more challenging node-level OOD generalization in this work.

B Additional Theoretical Results of the Covariate Shift Case

B.1 Theoretical Model Setup of the Covariate Shift Case

In this section, we will extend our theoretical model in the main text to the covariate shift setting. The causal graph of the covariate shift is shown in Figure 1b. For the covariate shift setting, spurious features are independent of Y , while X changes with environment e . Thus we can model the data generation process for environment e as

where the definition of n 1 and n 2 are the same as Section 2, ϵ e represents environmental spurious features. ϵ e i (each dimension of ϵ e ) is a random variable that are independent for i = 1 , ..., N e . We assume the intra-environment expectation of the environment spurious variable is E ϵ i ∼ p e [ ϵ i ] = µ e ∈ R since spurious features are consistent in a certain environment. We further assume the crossenvironment expectation E e [ ϵ e ] = 0 and cross-environment variance V e [ ϵ e i ] = σ 2 , i = 1 , ..., N e for simplicity. This is consistent with the covariate shift case that p ( X ) can arbitrarily change across different domains, and the support set of X may vary. Also, we require L ≥ k to ensure the GNN has enough capacity to learn the causal representations.

B.2 Theoretical Results of the Covariate Shift Case

Now we will present similar conclusions as the concept shift case. Even if VREx and IRMv1 can successfully capture invariant features in the non-graph task, they induce a model that uses spurious features. Still, CIA can learn invariant representations under covariate shift.

Proposition B.1. (VREx and IRMv1 learn invariant features for non-graph tasks under covariate shift, proof is in Appendix G.2.1) For the non-graph version of the SCM in Equation (19),

Optimizing VREx min Θ L VREx = V e [ R ( e )] and IRMv1 min Θ L IRMv1 = E e [ ∥∇ w | w =1 . 0 R ( e ) ∥ 2 ] will learn invariant features when using a 1-layer linear network: f ( X ) = θ 1 X 1 + θ 2 X 2 .

Proposition B.2. (VREx will use spurious features on graphs under covariate shift) Under the SCM of Equation (19), the objective min Θ V e [ R ( e )] has non-unique solutions for parameters of the GNN (3) when part of the model parameters { θ 1 1 ( l ) , θ 2 1 ( l ) , θ 1 2 ( l ) , θ 2 2 ( l ) } take the values

̸

0 < s < L is some positive integer. Specifically, the VREx solutions of θ 1 and θ 2 are the sets of solutions of the cubic equation, some of which are spurious solutions that θ 2 = 0 (although θ 2 = 0 is indeed one of the solutions, VREx is not guaranteed to reach this solution):

̸

Proposition B.3. (IRMv1 will use spurious features on graphs under covariate shift) Under the SCM of Equation (19), there exists s ∈ N + that satisfies 0 < s < L and s = k such that optimizing the IRMv1 objective min Θ E e [ ∥∇ w | w =1 . 0 R ( e ) ∥ 2 ] will not lead to the invariant solution θ 2 = 0 for parameters of the GNN (3) when { θ 1 1 ( l ) , θ 2 1 ( l ) , θ 1 2 ( l ) , θ 2 2 ( l ) } take the special solution:

Proposition B.4. Optimizing the CIA objective will lead to the optimal solution Θ ∗ :

C Detailed Experimental Setup

C.1 Basic Settings

All experimental results were averaged over three runs with different random seeds. Following Gui et al. [2022], we use an OOD validation set for model selection.

GAT experiments. For experiments on GAT, we adopt the learning rate of 0 . 01 for Arxiv, 0 . 001 for Cora, 0 . 003 for CBAS, and 0 . 1 for WebKB. The reason why we didn't use the default learning rate in Gui et al. [2022] is that since the original GOOD benchmark didn't implement GAT, so we chose to tune a learning rate for adapting GAT to reach a decent performance. The settings of batch size, training epochs, weight decay, and dropout follow Gui et al. [2022].

GCN experiments. For experiments on GCN, we follow the default settings of batch size, training epochs, learning rate, weight decay, and dropout provided by Gui et al. [2022].

It's worth mentioning that we choose different normalization strategies for the invariant edge mask of CIA-LRA to achieve better performance for GCN. In Equation (8), we use Sigmoid as normalization. However, we find it is better to use a Min-Max normalization for GCN on some of the datasets. Specifically, for experiments on GCN, we use Sigmoid normalization for CBAS, Arxiv (time concept); Sigmoid for training and Min-Max for testing for WebKB (covariate); Min-Max normalization for the other dataset splits. For GAT, we use Sigmoid during training and testing for all datasets.

C.2 Hyperparameter Settings of the Main OOD Generalization Results

Most hyperparameter settings are adopted from Gui et al. [2022], except that for EERM we reduce the number of generated environments from 10 to 7 and reduce the number of adversarial steps from 5 to 1 for memory and time complexity concerns. For each method, we conduct a grid search for about 3 ∼ 7 values of each hyperparameter. The hyperparameter search space is presented in Table 5.

C.3 Details of the Toy Dataset

Now we introduce the setting of the toy synthetic task of Figure 3. The synthetic dataset consists of four classes. Each node has a 4-dim node feature. The first/last two dimensions correspond to the invariant/spurious feature for each of the four classes, as shown in Table 6a and 6b. We artificially create both concept shift and covariate shift in this dataset.

To explicitly disentangle the learned invariant and the spurious components for quantitative analysis, we employ a 1-layer GCN. We take the output of the first/last two dimensions of the weight matrix as the invariant/spurious representation.

(a) The invariant part (the first two dimensions) of the node features in the synthetic dataset. (b) The spurious part (the last two dimensions) of the nodes features in the synthetic dataset.

D Additional Experimental Results

D.1 Excessive Alignment Leads to the Collapse of the Invariant Features.

One may wonder why CIA underperforms CIA-LRA even if CIA uses the ground truth environment labels. In this section, we will show that CIA may suffer excessive alignment, which will lead to the collapse of the learned invariant features and consequently hurt generalization. We use the intraclass variance of the representation corresponding to invariant features (averaged over all classes) to measure the degree of collapse of invariant features. Base on this measurement, the excessive alignment can be caused by:

1) Using a that is too large. Evidence: on the toy dataset of Figure 3, a larger λ leads to smaller intra-class variance of invariant representations. We also compute the intra-class variance of invariant representations at epoch 50 on the toy dataset,

2) Aligning the representations of too many nodes. Evidence: we show that aligning fewer node pairs can alleviate the collapse of invariant representation. By modifying CIA to align local pairs (same-class, different-environment nodes within 1 hop), termed "CIA-local", the results in Table 7b show that when by aligning local pairs instead of all pairs, CIA-local avoids the collapse that CIA suffers and achieves better performance.

D.2 CIA-LRA Alleviates Representation Collapse Caused by Excessive Alignment

In Figure 5, we illustrate the impact of λ on OOD accuracy. Both CIA and CIA-LRA experience a performance decline at λ = 0 . 5 , indicating that excessive alignment can hinder generalization. Furthermore, CIA shows an earlier and more pronounced performance drop than CIA-LRA. This suggests that the CIA-LRA method mitigates representation collapse by aligning fewer pairs and selectively focusing on pairs with smaller differences in invariant features.

Figure 3 at epoch 200.

0.0005

0.001

0.005

0.01

0.05

0.1

0.5

45

50

55

60

Test acc. (%)

CIA

CIA-LRA

0.0005

0.001

0.005

0.01

0.05

0.1

0.5

30

35

40

45

50

55

Test acc. (%)

CIA

CIA-LRA

0.0005

0.001

0.005

0.01

0.05

0.1

0.5

55

60

65

70

75

80

85

90

95

Test acc. (%)

CIA

CIA-LRA

0.0005

0.001

0.005

0.01

0.05

0.1

0.5

60

65

70

75

80

Test acc. (%)

CIA

CIA-LRA

The role of CIA-LRA in alleviating the collapse of the invariant features can also be reflected in Figure 6, in which the representation learned by CIA collapsed to a compact region. However, CIA-LRA does not exhibit such collapse, maintaining the diversity of the causal representation.

60

40

20

0

20

40

60

60

40

20

0

20

40

60

Class 0

Class 1

Class 2

Class 3

(a) ERM

60

40

20

0

20

40

40

30

20

10

0

10

20

30

40

Class 0

Class 1

Class 2

Class 3

(b) CIA

60

40

20

0

20

40

60

60

40

20

0

20

40

60

Class 0

Class 1

Class 2

Class 3

(c) CIA-LRA

60

40

20

0

20

40

60

40

20

0

20

40

60

Class 0

Class 1

Class 2

Class 3

(d) CIA-LRA w/o

r

same

D.3 Effect of the Number of Hops for Localized Alignment

In Figure 7, we plot the OOD accuracy curve of CIA-LRA against the number of hops t for localized alignment (with λ = 0 . 05 ). CIA-LRA achieves optimal performance within a local range of 6 to 10 hops. Performance is notably lower at smaller hops ( t = 2 ), due to limited regularization from aligning only a few pairs of representations. As t increases, performance gains diminish and can even degrade, particularly on the CBAS color covariate. This underscores the importance of localized alignment: optimal OOD performance is attained by aligning nodes within about 10 hops. Extending the alignment range further does not enhance performance significantly and may lead to performance drops and higher computational costs. These findings support the hypothesis in Appendix D.4 that invariant features distant on the graph differ substantially, and their alignment could induce invariant feature collapse, leading to a suboptimal generalization performance.

D.4 Discussion and Validation of the Assumption on the Rate of Change of Causal and Spurious Features w.r.t Spatial Position

To verify the intuition presented in Section 3.2 that spurious features exhibit larger changes within a local range (about 5 to 10 hops) on a graph compared to invariant features, we conduct experiments

2

4

6

8

10

15

20

25

30

35

hops

t

60.5

61.0

61.5

62.0

62.5

63.0

63.5

64.0

Test acc. (%)

CIA-LRA

ERM

IRM

VERX

2

4

6

8

10

15

20

25

30

35

hops

t

56.0

56.5

57.0

57.5

58.0

58.5

59.0

Test acc. (%)

CIA-LRA

ERM

IRM

VERX

2

4

6

8

10

15

20

25

30

35

hops

t

82

84

86

88

90

92

Test acc. (%)

CIA-LRA

ERM

IRM

VERX

2

4

6

8

10

15

20

25

30

35

hops

t

78

79

80

81

82

83

Test acc. (%)

CIA-LRA

ERM

IRM

VERX

(a) Cora degree concept

(b) Cora degree covariate

(c) CBAS color concept

(d) CBAS color covariate

on real-world datasets Arxiv and Cora. To extract invariant features, we use a pre-trained VREx model and take the output of the last layer as invariant features 8 . To obtain spurious features, we train an ERM model to predict the environment label and take the output of the last layer as spurious features. For each class, we randomly sample 10 nodes and generate corresponding 10 paths using Breadth-First Search (BFS). We extract invariant and spurious features of the nodes on each path and plot the L-2 distances between the node representations on the paths and the starting node. The results of Cora are in Figure 8 and 9, and the results of Arxiv are in Figure 10 and 11. We chose some of the classes to avoid excessive paper length; the results for the other classes are similar.

We observe that: despite the curve's slight fluctuations, the invariant feature difference shows a clear positive correlation with the distance from the starting point. Specifically, within about 5 ∼ 10 hops, the changes of spurious features grow more rapidly than those in invariant ones. This insight led us to align the representations of adjacent nodes to better eliminate spurious features and avoid the collapse of the invariant features. This also explains why we add a weighting term d ( i, j ) in our loss function to assign smaller weight node pairs farther apart. Additional experimental evidence supporting the importance of localized alignment is in Appendix D.3, which shows that alignment over a large range may lead to suboptimal performance and increasing computational costs.

This assumption aligns with those adopted in a series of previous works on causality and invariant learning [Chen et al., 2022, Burshtein et al., 1992, Schölkopf, 2022, Schölkopf et al., 2021]. These works assume that invariant features are better clustered than spurious features. In the node-level graph OOD scenario, we observe this phenomenon primarily within local parts of a graph. In some cases, when two nodes are too far apart, their invariant features can vary more than the spurious features, as seen in Figure 11 (a) path 1,2,4,6,9 and 10. Therefore, matching the representations in a local region helps alleviate the invariant feature collapse problem.

D.5 Discussion and Validation of the Assumption on the Feature Distance and Neighborhood Label Distribution Discrepancy

D.5.1 Heterophilic Neighborhood Labels Distribution Reflect Spurious Feature Distribution

In this section, we will empirically validate the key intuition of CIA-LRA: the label distribution of the neighbors from different classes (which we call Heterophilic Neighborhood Label Distribution , HeteNLD) reflects the spurious representation of the centered node. In node-level OOD scenarios, the distributional shifts of spurious features originate from two main sources: (1) the shifts in spurious node features associated with environments, and (2) the shifts in Neighborhood Label Distribution (NLD), which affects the aggregated representation of the centered node. The first type of spurious feature is analogous to those defined in Computer Vision (CV) OOD domains, while the second type is specific to graph structures. The NLD shift is a more general instance of the graph heterophily problem [Ma et al., 2022, Huang et al., 2023, Mao et al., 2023], where changes in the ratio of homophilic neighbors from training to test graphs can degrade performance. This occurs because the changes in the homophilic ratio lead to the distributional shift in the aggregated representation of the same-class nodes. Most previous methods [Ma et al., 2022, Huang et al., 2023, Mao et al., 2023]

(a) class 16 of Cora

(b) class 17 of Cora

Path 1

Path2

Path3

Path4

Path5

20.0

InvariantFeature

InvariantFeature

InvariantFeature

InvariantFeature

SpuriousFeature

SpuriousFeature

SpuriousFeature

17.5

SpuriousFeature

20

15.0

25

L2Distance from Start

L2 Distance from Start

L2 Distance from Start

Start

15.0

DistancefromStart

15

12.5

20

from

12.5

0.02

10.0

15

2Distance

10.0

0.00

10

7.5

7.5

5.0

10

2

5.0

-0.02

2.5

5

2.5

InvariantFeature

-0.04

o

0.0

0

0.0

SpuriousFeature

0

5

0

5

0

5

0

5

0

1

2

Path Length

Path Length

Path Length

Path Length

Path Length

Path6

Path7

Path8

Path9

Path10

20.0

20.0

InvariantFeature

InvariantFeature

InvariantFeature

20

SpuriousFeature

SpuriousFeature

17.5

SpuriousFeature

25

L2Distance fromStart

15.0

L2Distance from Start

DistancefromStart

15.0

DistancefromStart

15.0

L2Distance fromStart

15

12.5

20

12.5

12.5

10.0

15

10.0

10.0

7.5

7.5

7.5

10

5.0

5.0

5.0

2.5

5

2.5

InvariantFeature

2.5

InvariantFeature

00

00

SpuriousFeature

00

0-

SpuriousFeature

5

0

5

0

5

5

0

5

Path Length

Path Length

Path Length

PathLength

Path Length

Path1

Path2

Path3

Path4

Path5

30

35

InvariantFeature

InvariantFeature

InvariantFeature

InvariantFeature

25

SpuriousFeature

Spurious Feature

SpuriousFeature

SpuriousFeature

20

20

25

30

L2Distance fromStart

20

L2Distance from Start

L2Distance fromStart

L2 Distance from Start

L2Distance from Start

25

15

20

15

15

InvariantFeature

20

10

10

SpuriousFeature

15

10

10

10

5-

0

0

0

o

o

0

2

4

6

0

2

4

6

0

2

4

6

0

2

4

6

0.0

2.55.0

7.5

Path Length

PathLength

PathLength

PathLength

PathLength

Path6

Path7

Path8

Path9

Path10

OE

InvariantFeature

SpuriousFeature

30

InvariantFeature

SpuriousFeature

25

InvariantFeature

SpuriousFeature

25

InvariantFeature

SpuriousFeature

25

InvariantFeature

SpuriousFeature

25

25

L2DistancefromStart

20

L2 Distance from Start

20

L2Distance from Start

20

L2 Distance from Start

20

L2Distance fromStart

20

15

15

15

15

15

10

10

10

10

10

5

5

5

0

o

0

0

2

4

6

0

2

6

0

2

4

6

0

2

4

6

0

2

4

6

PathLength

PathLength

Path Length

PathLength

Path Length

10

(a) class 39 of Cora

(b) class 41 of Cora

Path1

Path2

Path3

Path4

Path5

30

InvariantFeature

InvariantFeature

InvariantFeature

InvariantFeature

SpuriousFeature

SpuriousFeature

SpuriousFeature

SpuriousFeature

30

25

L2 Distance fromStart

25

L2Distance fromStart

L2Distance from Start

L2Distance from Start

L2Distance fromStart

201

15

20

20

InvariantFeature

2

15

15

SpuriousFeature

10

15

10

10

10

10

5

5

5

0

o

0

5

0

5

10

0

5

10

0

5

10

0

5

10

Path Length

Path Length

Path Length

Path Length

Path Length

Path6

Path7

Path8

Path9

Path10

InvariantFeature

20.0

20

InvariantFeature

InvariantFeature

SpuriousFeature

17.5

20

17

SpuriousFeature

SpuriousFeature

L2 Distance from Start

L2Distancefrom Start

15.0

L2 Distance from Start

L2Distance from Start

15.0

L2Distance fromStart

15

15

12.5

12.5

10

10.0

10

10.0

7.5

7.5

5

5.0

5

5.0

2-

2.5

2.5

InvariantFeature

InvariantFeature

00

SpuriousFeature

o

SpuriousFeature

0.0

0

0

5

10

0

5

10

5

10

5

10

0

2

4

Path Length

PathLength

PathLength

PathLength

Path Length

Path1

Path2

Path3

Path4

Path5

InvariantFeature

25

InvariantFeature

40

InvariantFeature

InvariantFeature

35

InvariantFeature

SpuriousFeature

SpuriousFeature

SpuriousFeature

SpuriousFeature

SpuriousFeature

40

20

20

30

L2DistancefromStart

L2 Distance from Start

L2DistancefromStart

30

L2Distance from Start

L2 Distance from Start

25

OE

15

15

20

20-

10

20

10

15

10

10

0

0

0

o

0

5

10

0

5

10

5

10

0

5

10

0.0

2.55.07.5

PathLength

PathLength

PathLength

PathLength

PathLength

Path6

Path7

Path8

Path9

Path10

InvariantFeature

InvariantFeature

Invariant Feature

Invariant Feature

25

InvariantFeature

30

SpuriousFeature

25

SpuriousFeature

Spurious Feature

OE

SpuriousFeature

SpuriousFeature

20

L2DistancefromStart

25

L2Distance from Start

20

L2Distance fromStart

L2Distancefrom Start

25

L2 Distance from Start

20

20

15

15

20

15

15

10

15

10

10

10

10

5

5

5

5

o

0

0

5

10

0

5

10

5

10

0

5

10

0

5

10

Path Length

Path Length

PathLength

Path Length

Path Length

(a) class 25 of Arxiv

(b) class 29 of Arxiv

Path1

Path2

Path3

Path4

Path5

InvariantFeature

InvariantFeature

17.5

InvariantFeature

17.5

InvariantFeature

20

SpuriousFeature

20

SpuriousFeature

15.0

SpuriousFeature

15.0

SpuriousFeature

V

2.5

L2DistancefromStart

L2DistancefromStart

15

L2DistancefromStart

12.5

L2Distance fromStart

12.5

L2Distance fromStart

2.0-

15

10.0

10.0

1.5

InvariantFeature

10

10

7.5

7.5

1.0

SpuriousFeature

5.0

5.0

5

0.5

2.5

2.5

0

00

0.0

0.0

0.0

2.5

5.0

0

5

0.0

2.5

5.0

0

5

0

1

2

Path Length

Path Length

Path Length

Path Length

Path Length

Path6

Path7

Path8

Path9

Path10

12

InvariantFeature

SpuriousFeature

12

SpuriousFeature

InvariantFeature

10-

InvariantFeature

SpuriousFeature

20.0

17.5

InvariantFeature

SpuriousFeature

12

SpuriousFeature

InvariantFeature

L2DistancefromStart

10

L2DistancefromStart

10

L2 Distance from Start

8

L2Distance fromStart

15.0

L2 Distance from Start

10

8

6

12.5

10.0

7.5

4

4

5.0

2

2-

2.5

2

0

0

o

0.0

o

0.0

2.5

5.0

0

5

0

5

0.0

2.5

5.0

0.0

2.5

5.0

PathLength

Path Length

PathLength

Path Length

Path Length

Path1

Path2

Path3

Path4

Path5

20.0

InvariantFeature

20.0

口

InvariantFeature

InvariantFeature

O

SpuriousFeature

17.5

SpuriousFeature

SpuriousFeature

17.5

DistancefromStart

0.02

2DistancefromStart

15.0

2DistancefromStart

15.0

L2Distance fromStart

15

L2DistancefromStart

15

12.5

12.5

0.00

10.0

Invariant Feature

SpuriousFeature

10

10

7.5

7.5

0.02

2

5.0

L2

5.0

5-

5

-0.04

2.5

0.0

InvariantFeature

SpuriousFeature

2.5

00

N

0

0

0

1

2

5

10

5

10

0

5

10

0

5

10

PathLength

Path Length

PathLength

Path Length

PathLength

Path6

Path7

Path8

Path9

Path10

InvariantFeature

InvariantFeature

InvariantFeature

InvariantFeature

16

SpuriousFeature

17

SpuriousFeature

0.

SpuriousFeature

SpuriousFeature

14

L2Distancefrom Start

L2 DistancefromStart

15.0

2DistancefromStart

L2DistancefromStart

L2DistancefromStart

12

0.02

15

12.5

10.0

0.00

10

SpuriousFeature

InvariantFeature

7.5

6

-0.02

5

5.0

5+

4

2.5

-0.04

0

0.0

0

0

0

5

10

5

10

1

2

0

5

10

o

5

10

Path Length

Path Length

Path Length

PathLength

Path Length

(a) class 13 of Arxiv

(b) class 17 of Arxiv

Path1

Path2

Path3

Path4

Path5

InvariantFeature

InvariantFeature

17.5

14

InvariantFeature

SpuriousFeature

SpuriousFeature

SpuriousFeature

L2Distance fromStart

12

L2DistancefromStart

12

fromStart

15.0

L2 Distance from Start

12

L2Distance from Start

10

10

12.5

10

4

10.0

8

Distance

6

7.5

6

4

2

5.0

2.5

2

InvariantFeature

InvariantFeature

0

0

0.0

SpuriousFeature

-0

SpuriousFeature

0

0

10

0

5

10

0

5

10

0

5

10

0

2

4

Path Length

Path Length

Path Length

Path Length

Path Length

Path6

Path7

Path8

Path9

Path10

InvariantFeature

SpuriousFeature

14

SpuriousFeature

InvariantFeature

InvariantFeature

SpuriousFeature

SpuriousFeature

InvariantFeature

L2 Distance fromStart

20-

L2 Distance from Start

10

12

DistancefromStart

0.02

DistancefromStart

15

L2 Distance from Start

5

15

8

000

10

10

a

2

-0.02

L2

2

InvariantFeature

0.04

0

SpuriousFeature

0

0

0

10

0

5

10

0

1

2

0

10

0

2

4

PathLength

PathLength

Path Length

PathLength

PathLength

Path1

Path2

Path3

Path4

Path5

3.5

1.75

InvariantFeature

InvariantFeature

InvariantFeature

25

SpuriousFeature

20

SpuriousFeature

3.0

1.50

0.0

SpuriousFeature

L2Distance fromStart

L2Distance fromStart

L2 Distance from Start

2.5

2DistancefromStart

1.25

DistancefromStart

20

0.02

2.0

1.00

15

10

1.

InvariantFeature

SpuriousFeature

0.7

SpuriousFeature

InvariantFeature

0.00

10

5

1.0

2

0.50

-0.02

5

0.5

0.25

-0.04

0

00

0.00

0

5

10

0

5

0

2

2

1

2

PathLength

Path Length

Path Length

Path Length

PathLength

Path6

Path7

Path8

Path9

Path10

3.5

3.5

InvariantFeature

17.5

20

SpuriousFeature

15.0

3.0

20

3.0

L2DistancefromStart

2DistancefromStart

12.5

2DistancefromStart

2.5

2DistancefromStart

DistancefromStart

2.5

15

15

10.0

InvariantFeature

2.0

InvariantFeature

InvariantFeature

2.0

InvariantFeature

10

7.5

SpuriousFeature

1

SpuriousFeature

SpuriousFeature

1.

SpuriousFeature

5

5.0

2

1.0

5

2

1.0

2.5

0.5

0.5

0

00

0.0

00

0

5

0

-5

0

2

0

5

0

2

PathLength

Path Length

Path Length

Path Length

Path Length

only focus on the binary-classification setting, where changes in the homophilic neighbor ratio are equivalent to changes in the heterophilic neighbor ratio. However, we consider the more general multiclassification tasks. Therefore, we propose to use HeteNLD as a measurement, considering every class different from the central class and using their distribution to reflect shifts in the aggregated representation. Although the ratio of homophilic neighbors also affects environmental spurious features and NLD, it affects the invariant representation as well. Assigning larger weights to the pair with significant differences in the ratio of homophilic neighbors will simultaneously eliminate environmental spurious features and learn a collapsed invariant representation. As evidenced in Table 4, moving the r same ( c ) i,j to the numerator of Equation (9) will lead to a significant performance decrease. Hence we use 1 r same ( c ) i,j instead of r same ( c ) i,j in w ij .

In the following part, we will empirically validate our intuition that HeteNLD can reflect the two spurious representation distributions on concept shift , where p ( Y | X ) varies across environments, and covariate shift , where p ( X ) changes with environments, respectively. We will show that HeteNLD affects the spurious features of the centered node in different manners under concept shift and covariate shift.

Covariate shift. For covariate shifts on graphs, since spurious features are not necessarily correlated with labels, the environmental spurious features cannot be reflected by HeteNLD. However, we can still measure how HeteNLD affects the aggregated neighborhood representation. To obtain neighborhood representation, we train a 1-layer GCN that aggregates neighboring features and discards the features of the centered node. We hope to observe whether the gap of HeteNLD accurately reflects the distance of neighborhood representation. To ensure that the discrepancy in the aggregated neighboring feature is caused solely by heterophilic neighbors, we only use point pairs with the same number of homophilic neighbors. Specifically, we compute the L-2 distance between the neighborhood representations of two nodes with the same number of class-same neighbors, and plot its trend w.r.t. the distance of HeteNLD (according to the definition of r diff i,j in Equation (9), except that we didn't normalize by the node degree here). We run experiments on Cora to verify this. We evaluate on both word shifts (node feature shifts) and degree (graph structure shifts) for a comprehensive understanding. We show the results of the first 30 classes of Cora. The results in Figure 12 and 13 show a clear positive correlation between the neighborhood representation distance and HeteNLD discrepancy under covariate shifts, indicating HeteNLD discrepancy can reflect the distance of the aggregated representation.

Concept shift. As for concept shift, spurious features are correlated with labels, thus the label of a node contains information about spurious features correlated with this class. Hence, by observing HeteNLD, we can measure the distribution of the spurious feature. For concept shift, we train a GNN to predict environment labels to obtain spurious representations. Table 14 and 15 also show a clear positive correlation between spurious feature distance and HeteNLD discrepancy on concept shift, indicating that HeteNLD discrepancy can reflect the distance of the environmental spurious features.

D.5.2 Homophilic Neighboring Labels Reflect Invariant Feature Distribution

Now will validate that the ratio of the same-class neighbors reflects the aggregated invariant representation. We use VREx to approximately extract invariant features and compute their distance w.r.t. the discrepancy of the ratio of the same-class neighbors. We evaluate on 4 splits of Cora: word + covariate , word + concept , degree + covariate and degree + concept . For each data split, we randomly choose 5 classes with sufficiently large differences in homophilic neighbor ratios for visualization. The results in Figure 16 also show a positive correlation trend between the distance of the invariant representations and the difference in the ratio of same-class neighbors, indicating the latter can reflect the former.

D.6 Validation of the True Feature Generation Depth

For the theoretical model in Section 2, we assume that the number of layers of the GNN L is greater than the depth of the causal pattern k . In this section, we empirically verify how large k really is on real-world datasets. Specifically, we use GCN with different layers to predict the ground-truth label Y on Cora and Arxiv datasets respectively (results are in Table 9a and 9b). As mentioned above, since a GCN with L layers will aggregate features from L -hop neighbors for prediction, if the depth

ass 15

7.5

1l

1.00

520-050-50-

20.0

7.5

0.75

of the GCN is equal to the true generation depth, then the performance should be close to optimal. Therefore, we use the layer number that yields the optimal empirical performance (denoted as L ∗ ) to approximate k . We find that the L ∗ ≤ 4 in most cases. This indicates that our assumptions L ≤ k hold easily.

D.7 Time Cost of CIA-LRA

To show the running time of CIA-LRA, we show the time cost to reach the best test accuracy on our largest dataset Arxiv (with 50k 60k nodes). The results are in Table 8 below. The time cost of CIA-LRA is comparable to baseline methods.

E Detailed training procedure

Table 1 shows the detailed training procedure (pseudo code) of CIA-LRA. We use the same GNN encoder for the invariant subgraph extractor. Empirically, we add CIA or CIA-LRA after one epoch of ERM training.

Algorithm 1 Detailed Training Procedure of CIA-LRA

Require:

A labeled training graph G = ( A,X,Y ) , a GNN f Θ , and invariant subgraph generator GNN f θ m . The number of hops t , CIA-LRA weight λ , the number of classes C , total iterations T , model learning rate r 1 , invariant subgraph generator learning rate r 2 , the number of f Θ 's layers L

Ensure:

Updated model f Θ with parameter Θ .

1: for iterations in [1 , 2 , ..., T ] do

2: Randomly sample a subgraph A ′ ∈ R N × N from A ∈ R N 0 × N 0 .

3: Compute and apply the edge mask according to Equation (8) to obtain the masked adjacency matrix A m ← A ′ ⊙ M ∈ R N × N .

4: Initialize L CIA-LRA ← 0 .

6: ### Calculate A t , where the ( i, j ) -th element of A t equals the length of the shortest path from node i to j if the length is less than t else infinity:

5: Calculate the node representations ϕ ( A m , X ) ∈ R N × D .

̸

7: Initialize A t ( i, j ) ← Inf if i = j , A t ( i, i ) ← 1 , A tem ← A m

8: for hop h in [1 , 2 , ..., t ] do

9: A tem ← A m A tem

11: A t ( i, j ) ← h

10: if A tem ( i, j ) > 0 and A tem ( i, j ) < A t ( i, j ) then

12: end if

13: end for

14: ### Compute the ratio of neighbored nodes of each class:

16: Initialize the neighbored label ratio R ← Y ∈ R N × C , where R ( i, c ) is the ratio of node i 's neighbors of class c within a L -hop range, Y are the one hop labels.

15: Compute the normalized adjacency matrix ¯ A , where ¯ A 's i -th row ¯ A i ← A mi /D i , A mi is the i -th row of A m and D i ∈ R is the degree of node i .

17: for l in [1 , 2 , ..., L ] do

18:

R

¯

AR

19: end for

20: for c in [1 , 2 , ..., C ] do

21: Sample the nodes of class c from A t and form A t c . Use A t c to screen for pairs of nodes not exceeding a distance of t hops Ω c ( t ) .

23: L CIA-LRA ←L CIA-LRA + L c CIA-LRA

22: Compute CIA-LRA loss of class c : L c CIA-LRA according to Equation (9) using Ω c ( t ) , R , A t c and ϕ ( A m , X ) :

24: end for

25: Compute final loss L ← L ce ( f Θ ( A,X ) , Y ) + λ L CIA-LRA, L ce is the cross-entropy loss

26: Update model parameters Θ ← Θ -r ∇ Θ L , θ m ← Θ m -r ∇ θ m L

27: end for

F Additional Discussion of Theoretical Settings and Results

F.1 Detailed Setup of the Theoretical Model in Section 2

The proposed data generation process. In the theoretical model of Equation (2), each dimension of n 1 ∈ R N e × 1 and n 2 ∈ R N e × 1 are i.i.d, following a standard Gaussian distribution. ϵ e ∈ R N e × 1 is an environment spurious variable. ϵ e i (each dimension of ϵ e ) are independent random variables, i = 1 , ..., N e . We further assume the cross-environment expectation E e [ ϵ e ] = 0 and cross-environment variance E e [ ϵ e i ] = σ 2 , i = 1 , ..., N e for brevity.

The considered multi-layer GNN. In the analyzed GNN of Equation (3), we simplify the classifier to an identity mapping. Such simplification has been adopted by various previous theoretical works on graphs [Wu et al., 2022b, Tang and Liu, 2023]. We assume L ≥ k to ensure the model has enough capacity to learn invariant features. We verify this assumption by using GCNs with different numbers of layers to predict the ground-truth labels (see Appendix D.6).

F.2 Discussion of the Structural Feature Considered in the Theoretical Model and Justification for the Choice of the Analyzed GNN

Structural Features and Structural Shifts Considered in Section 2.1. To reflect reality as much as possible, it is necessary to consider both nodal and structural invariant and spurious features in the theoretical model. As mentioned in Section 2.1, we model the invariant structural feature as the structure of the k -hop ego-subgraph. A natural question is raised here:

Can we find other ways to define the invariant/spurious structural features?

The answer is yes. For example, the invariant structure can be modeled as the subgraph of the ego-graph of a node, following Li et al. [2023a]. However, it is fundamentally impossible for GNNs using mean aggregation (like GCN) to learn such causal structures. This is because such GNNs will assign fixed weights to each neighboring node feature, and they can't split the causal substructure from the neighbored ego-sgraph. Therefore, we make the causal structure feasible for GCN-like GNNs to learn by defining the causal structure as the whole k -hop neighboring ego-graph, rather than a subgraph, and show that OOD failure can still happen (Theorem 2.3). Then, under this setting, the remaining challenge becomes identifying the true k by optimizing the shallow layer GNN parameters. However, in real practice, the invariant causal pattern may still be an ego-subgraph. This can be reflected in the performance gain of the invariant subgraph extractor used in CIA-LRA.

Why Do We Choose Such a GNN in Section 2.1? From the above analysis, we show that such a choice is a compromise solution between the case of GCN (that can only extract a whole ego-graph) and GAT-like GNNs (that can extract a subgraph from an ego-graph). Although in this GNN each neighbored node is solely assigned the same weight, the shallow layer parameters can be optimized to realize the aggregation of different depths to capture the causal structures of different depths.

F.3 Discussion of the Failure Solution for GNNs of VREx and IRMv1

In Theorem G.2 and G.3 (the formal version Theorem 2.3), we show that VREx and IRMv1 could induce a model that uses spurious features. Now we'll give an intuitive explanation of this failure mode. When the lower-layer parameters of the GNN θ 1 1 ( l ) , θ 1 2 ( l ) , θ 2 1 ( l ) , θ 2 2 ( l ) take the specific solution Θ 0 in Equation (21), we have

holds for i = 1 , 2 , l = 1 , ..., L -1 and every environment e . Thus, we get

( ∗ ) is because of Equation (25). Therefore, ∂ L ∂θ 1 = 0 ⇒ ∂ L ∂θ i 1 ( l ) = 0 . The same is true for ∂ L ∂θ 2 and ∂ L ∂θ i 2 ( l ) . This means the solution of the top-level parameters θ 1 and θ 2 of the GNN will only be constrained by two equations, ∂ L ∂θ 1 = 0 and ∂ L ∂θ 2 = 0 , rather than be constrained by all gradient functions ∂ L ∂θ j i = 0 , i = 1 , 2 . By analyzing the specific loss of VREx and IRMv1, we conclude that they will induce a non-zero θ 2 .

Note that the failure solution Θ 0 here is not the unique one, we choose Θ 0 just for the elegant expression and to better convey the intuition. In effect, the conclusion ∂ L ∂θ 1 = 0 ⇒ ∂ L ∂θ i 1 ( l ) = 0 and

∂ L ∂θ 2 = 0 ⇒ ∂ L ∂θ i 2 ( l ) = 0 holds as long as the lower-layer aggregation parameters satisfy

i.e., we don't require the spurious branch of the GNN to be identity mapping I as equation (36) does. This failure mode can happen to GCN (all θ i = 1 , i = 1 , 2 , j = 1 , 2 ) and also for GAT.

j

F.4 The Superiority of the Proposed CSBM-OOD

Our CSBM-OOD in Section introduces several advancements over the conventional CSBMs [Ma et al., 2021, Mao et al., 2023]: 1) It supports multi-class classification, extending beyond the binary classification framework of traditional CSBMs; 2) It accommodates unique neighboring label distributions for each node, in contrast to the traditional models that assume a uniform class-shared homophily/heterophily ratio across all nodes; 3) our model integrates OOD shifts, while traditional ones don't.

F.5 Tightness of the Error Bound of Theorem 4.4

When there are no distributional shifts in spurious node features and heterophilic neighborhood distribution between training and test environments, the terms (a)-(d) in Eq. (109) becomes zero, and the upper bound becomes ̂ L γ e tr ( ˜ h ) + const = ̂ L γ e tr ( ˜ h ) + 1 N 1 -2 α e tr + 1 N 2 α e tr ln LC (2 B e te ) 1 /L γ 1 /L δ , i.e., our bound only larger than the ideal error ̂ L γ e tr ( ˜ h ) by a constant const . When the number of training samples N e tr is large, const will be small enough and can be negligible. Hence, the tightness of our bound is guaranteed.

G Proofs of the Theoretical Results

G.1 Proofs of the Concept Shift Case Presented in the Main Text

In this section, we give proof of the propositions of the concept shift model presented in the main text.

G.1.1 Proof of the non-Graph Success Case of VREx and IRMv1 under Concept Shift

We restate Proposition 2.2 as Proposition G.1 below.

Proposition G.1. For the non-graph version of the SCM in Equation (2),

VREx and IRMv1 will learn invariant features when using a 1-layer linear network: f ( X ) = θ 1 X 1 + θ 2 X 2 .

Proof. VREx. Denote X 1 θ 1 + X e 2 θ 2 -X 1 -n 1 as l e . The variance of loss across environments is:

Take the derivative of V e [ R ( e )] with respect to θ 1 :

Using the fact that E n 1 ,n 2 [ n 1 ] = E n 1 ,n 2 [ n 2 ] = 0 , E n 1 ,n 2 [ n ⊤ 1 n 2 ] = E n 1 ,n 2 [ n ⊤ 2 n 1 ] = 0 and E n 1 ,n 2 [ n ⊤ 1 n 1 ] = E n 1 ,n 2 [ n ⊤ 2 n 2 ] = N e if it is the noise from e , E n 1 ,n 2 [ n ⊤ 1 ϵ e ] = E n 1 ,n 2 [ n ⊤ 2 ϵ e ] = 0 and using the assumption that E e [( ϵ e i ) 2 ] = σ 2 , we have

and

Plug Equation (31) and (32) back into Equation (30), we have

Let ∂ V e [ R ( e )] ∂θ 1 = 0 , we have θ 2 = 0 .

Now we need to validate θ 2 = 0 is also a solution to ∂ V e [ R ( e )] ∂θ 2 = 0 . Let's calculate ∂ V e [ R ( e )] ∂θ 2 when θ 2 = 0 :

So far, we have proved θ 2 = 0 is the solution for VREx, hence it will learn invariant features. We finish the proof for VREx.

IRMv1. The objective of IRMv1 is E e ∥∇ w R ( e ) ∥ 2 2 . When IRMv1 loss is optimized to zero, we have ∇ w R ( e ) = 0 for all environments e .

To realize ∇ w R ( e ) = 0 for all e , we must let θ 2 = 0 (and consequently θ 1 = 1 ), otherwise the solution of θ 2 will include terms related to ϵ e and N e that vary with environments, and a single value θ 2 cannot fit all these values. Thus we finish the proof for IRMv1.

G.1.2 Proof of the Failure Case on Graphs of VREx under Concept Shift

We present the formal version of the VREx part in Theorem 2.3 as Theorem G.2 below.

Theorem G.2. (VREx will use spurious features on graphs under concept shift, formal) Under the SCM of Equation (2), the objective min Θ V e [ R ( e )] has non-unique solutions for parameters of the GNN (3) when part of the model parameters { θ 1 1 ( l ) , θ 2 1 ( l ) , θ 1 2 ( l ) , θ 2 2 ( l ) } take the values

̸

for some 0 < s < L . Specifically, the VREx solutions of θ 1 and θ 2 are the sets of solutions of the cubic equation, some of which are spurious solutions that θ 2 = 0 (although θ 2 = 0 is indeed one of the solutions, VREx is not guaranteed to reach this solution):

where c 1 = E e [( ˜ A e s X 1 ) ⊤ ( ˜ A e s X 1 )] , c 2 = E e [ N e ( ˜ A e s X 1 ) ⊤ ˜ A e s X 1 ] , c 3 = E e [( ˜ A e s X 1 ) ⊤ 1 ] , c 4 = E e [(( ˜ A e k X 1 ) ⊤ 1 ] , c 5 = E e [ N e (( ˜ A e k X 1 ) ⊤ ˜ A e s X 1 + tr (( ˜ A e k ) ⊤ ˜ A e k ) + N e (1 + σ 2 ))] , c 6 = E e [( ˜ A e s X 1 ) ⊤ ( ˜ A e k X 1 )] , c 7 = E e [ ϵ e ⊤ ϵ e ϵ e ⊤ ( ˜ A e s X 1 )] .

Proof. We will use some symbols to simplify the expression of the toy GNN. Denote ˜ A m n 1 + n 2 + ϵ as η . Use the following notations to represent the components of the L -layer GNN model:

(38)

C 1 , C 2 , Z ∈ R N × 1 . We use C e 1 , C e 2 , and Z e to denote the variables from the corresponding environment e . We further denote C e 2 = C e 2 ′ ˜ A e s X 1 , Z e = C e 2 ′ η .

Using these notations, the loss of environment e is

Denote the inner term C e 1 θ 1 +( C e 2 + Z e ) θ 2 -˜ A e k X 1 -n 1 as l e .

The variance of loss across environments is:

Take the derivative of V e [ R ( e )] with respect to θ 1 :

Calculate the derivative by terms:

Since n 1 and n 2 are independent standard Gaussian noise, we have E n 1 ,n 2 [ n 1 ] = E n 1 ,n 2 [ n 2 ] = 0 , E n 1 ,n 2 [ n ⊤ 1 n 2 ] = E n 1 ,n 2 [ n ⊤ 2 n 1 ] = 0 and E n 1 ,n 2 [ n ⊤ 1 n 1 ] = E n 1 ,n 2 [ n ⊤ 2 n 2 ] = N e if it is the noise from e . Also, since ϵ e and n 1 , n 2 are independent, we have E n 1 ,n 2 [ n ⊤ 1 ϵ e ] = E n 1 ,n 2 [ n ⊤ 2 ϵ e ] = 0 .

When

we have C e 2 ′ = I N e ∈ R N e × N e and C e 1 = ˜ A e s X 1 . Consequently, we get E n 1 ,n 2 [ Z e ⊤ n 1 ] = tr ( C e 2 ′ ˜ A e k ) = tr ( ˜ A e k ) , E n 1 ,n 2 [ Z e ⊤ Z e ] = tr ( ( ˜ A e k ) ⊤ ( ˜ A e k ) ) + N e + ϵ e ⊤ ϵ e .

Use the above conclusions and rewrite Equation (42) as (here we only plug in the value of C e 2 ′ ):

( ∗ ) and ( ∗∗ ) represent terms that are independent and associated with ϵ e , respectively. Additionally,

Multiplying Equation (44) and (45) and take the expectation on e , using the assumption that E e [( ϵ e i ) 2 ] = σ 2 ( ϵ e i is the i -th element of ϵ e ):

Next target is to compute 2 E e [ E n 1 ,n 2 [ l ⊤ e l e ]] and E e [ E n 1 ,n 2 [2 l ⊤ e C 1 ]] . Since ϵ e has zero mean, we have:

and

Use Equation (46) (47) and (48) and let ∂ V e [ R ( e )] ∂θ 1 = 0 , we have:

Now we start calculating the expression of ∂ V e [ R ( e )] ∂θ 2 :

Plug Equation (49) in (51), we reach:

we conclude that

As for the derivative respect to θ 1 1 ( l ) , θ 2 1 ( l ) , θ 1 2 ( l ) , θ 2 2 ( l ) , when they take the special value in Equation (43), we have ∂ V e [ R ( e )] ∂θ 1 = 0 ⇒ ∂ V e [ R ( e )] ∂θ 1 1 ( l ) = ∂ V e [ R ( e )] ∂θ 2 1 ( l ) = 0 and ∂ V e [ R ( e )] ∂θ 2 = 0 ⇒ ∂ V e [ R ( e )] ∂θ 1 2 ( l ) =

∂ V e [ R ( e )] ∂θ 2 2 ( l ) = 0 , l = 1 , ..., L . So we conclude the solution induced by Equation (53) is the solution of the objective, and θ 2 = 0 is not a valid solution.

G.1.3 Proof of the Failure Case on Graphs of IRMv1 under Concept Shift

We present the formal version of the IRM part in Theorem 2.3 as Theorem G.3 below.

̸

Theorem G.3. (IRMv1 will use spurious features on graphs under concept shift, formal) Under the SCM of Equation (2), there exists s ∈ N + that satisfies 0 < s < L and s = k such that optimizing the IRMv1 objective min Θ E e [ ∥∇ w | w =1 . 0 R ( e ) ∥ 2 ] will not lead to the invariant solution θ 2 = 0 for parameters of the GNN (3) when { θ 1 1 ( l ) , θ 2 1 ( l ) , θ 1 2 ( l ) , θ 2 2 ( l ) } take the special solution:

Proof. From the proof of non-graph IRMv1 case Appendix G.1.1 we know that when IRMv1 objective is optimized, we have ∇ w R ( e ) = 0 for all e . For the graph case, the expected risk of environment e is

where the definition of C e 1 , C e 2 and Z e follows Equation (38). Now let's check if the invariant solution θ 2 = 0 is a valid solution. If θ 2 = 0 holds, then the following equation must hold for every environment e :

̸

When s = k , we have θ 1 = (( ˜ A e ) k X 1 ) ⊤ (( ˜ A e ) s X 1 ) (( ˜ A e ) s X 1 ) ⊤ (( ˜ A e ) s X 1 ) . The value of this solution of θ 1 varies with environment e , and thus is not a valid solution.

However, now we will show that optimizing IRMv1 does not necessarily lead to lower-layer parameters such that s = k . To reveal this, by taking the derivative of L IRMv1 w.r.t. θ 1 and θ 2 and let them = 0 , we can get two cubic equations:

and

̸

From the analysis in Appendix F.3, we know that as long as the lower-layer parameters take any value that satisfies the form in Equation (27), even if s = k , we can get ∂ L IRMv1 ∂θ 1 = ∂ L IRMv1 ∂θ 1 1 = ∂ L IRMv1 ∂θ 2 1 and ∂ L IRMv1 ∂θ 2 = ∂ L IRMv1 ∂θ 1 2 = ∂ L IRMv1 ∂θ 2 2 . Thus, IRM cannot necessarily learn a s = k . At this time (when ∂ L IRMv1 ∂θ 1 = ∂ L IRMv1 ∂θ 1 1 = ∂ L IRMv1 ∂θ 2 1 = 0 and ∂ L IRMv1 ∂θ 2 = ∂ L IRMv1 ∂θ 1 2 = ∂ L IRMv1 ∂θ 2 2 = 0 but s = k ), from the form of Equations (57) and (58) we know that there exist solutions that θ 2 = 0 , and the solution of θ 1 and θ 2 both depend on E e ( F ( e )) , where F ( e ) is some random variable associated with e .

̸

̸

G.1.4 Proof of the Successful on Graphs Case of CIA under Concept Shift

Theorem G.4. Optimizing the CIA objective will lead to the optimal solution Θ ∗ :

Proof. For brevity, denote a node representation of C e 1 c as C i 1 and the one of C e ′ 1 c as C j 1 . The same is true for C i 2 and C j 2 . In this toy model, we need to consider the expectation of the noise, while in real cases such noise is included in the node features so taking expectation on e will handle this. Therefore, we add E n 1 ,n 2 in this proof, and this expectation is excluded in the formal description of the objective in the main paper.

̸

̸

Let ∂ L CIA ∂θ 1 = 0 , we have:

̸

̸

Also, we have:

̸

Further let ∂ L CIA ∂θ 2 = 0 , combining Equation (62) and using Assumption 2.1 we get

̸

or, if θ 1 = 0 and ∀ l ∈ { 1 , ..., L -1 } , the parameters of that layer l of the invariant branch of the GNN are not all zero: θ 1 1 ( l ) = 0 or θ 2 1 ( l ) = 0 , then we get

̸

̸

̸

(65) According to Cauchy-Schwarz inequality, F > 0 unless ∃ l ∈ { 1 , ..., L -1 } s.t. θ 1 2 ( l ) = θ 2 2 ( l ) = 0 . To ensure ∂ L CIA ∂θ 2 , we conclude that θ 2 = 0 or ∃ l ∈ { 1 , ..., L -1 } s.t. θ 1 2 ( l ) = θ 2 2 ( l ) = 0 .

In conclusion, to satisfy the constraint of CIA, no matter whether the invariant branch has zero output, the spurious branch must have zero parameters, i.e.,

Thus, CIA will remove spurious features.

Now we show that when CIA objective has been reached (the spurious branch has zero outputs), the objective of min Θ E e [ L ( f Θ ( A e , X e ) , Y e )] will help to learn predictive paramters of the invariant branch θ 1 , θ 1 1 ( l ) and θ 2 1 ( l ) . When Equation (66) holds,

Let ∂ E e [ L ( f Θ ( A e ,X e ) ,Y e )] = 0 , we get the predictive parameters

Plug the final solution back in ∂ L CIA ∂θ 1 1 ( l ) , ∂ L CIA ∂θ 2 1 ( l ) , ∂ L CIA ∂θ 1 2 ( l ) , ∂ L CIA ∂θ 2 2 ( l ) , ∂ E e [ L ( f Θ ( A e ,X e ) ,Y e )] ∂θ 1 1 ( l ) , ∂ E e [ L ( f Θ ( A e ,X e ) ,Y e )] ∂θ 2 1 ( l ) , ∂ E e [ L ( f Θ ( A e ,X e ) ,Y e )] ∂θ 1 2 ( l ) , ∂ E e [ L ( f Θ ( A e ,X e ) ,Y e )] ∂θ 2 2 ( l ) , we can verify that these terms are all when further letting ∃ l ∈ { 1 , ..., L -1 } s.t. θ 1 2 ( l ) = θ 2 2 ( l ) = 0 .

G.2 Proof of the Covariate Shift Case

G.2.1 Proof of the non-Graph Success Case of VREx and IRMv1 under Covariate Shift

We restate Proposition B.1 as Proposition G.5 below.

Proposition G.5. (VREx and IRMv1 learn invariant features for non-graph tasks under covariate shift, proof is in ) For the non-graph version of the SCM in Equation (19),

Optimizing VREx min Θ L VREx = V e [ R ( e )] and IRMv1 min Θ L IRMv1 = E e [ ∥∇ w | w =1 . 0 R ( e ) ∥ 2 ] will learn invariant features when using a 1-layer linear network: f ( X ) = θ 1 X 1 + θ 2 X 2 .

Proof. VREx. For covariate shift, denote X 1 θ 1 + X e 2 θ 2 -X 1 -n 1 as l e , the expected risk in environment e is R ( e ) = E n 1 ,n 2 ∥ θ 1 X 1 + θ 2 ( n 2 + ϵ e ) -( X 1 + n 1 ) ∥ 2 2 . Take the derivative of the VREx objective V e [ R ( e )] with respect to θ 1 :

Calculate these terms:

Hence,

Let ∂ V e [ R ( e )] ∂θ 1 = 0 , we have θ 2 = 0 .

When θ 2 = 0 and when θ 1 = 1 , we get

As a result,

In conclusion, θ 1 = 1 and θ 2 = 0 is the solution of the VREx objective in this non-graph covariate shift task.

IRMv1. The objective of IRMv1 is E e ∥∇ w R ( e ) ∥ 2 2 . When IRMv1 loss is optimized to zero, we have ∇ w R ( e ) = 0 for all environments e .

To realize ∇ w R ( e ) = 0 for all e , we must let θ 2 = 0 (and consequently θ 1 = 1 ), otherwise the solution of θ 2 will include terms related to ϵ e and N e that vary with environments, and a single value θ 2 cannot fit all these values. Thus we finish the proof for IRMv1.

G.2.2 Proof of the Failure Case on Graphs of VREx under Covariate Shift

We restate Theorem B.2 as Theorem G.6 below:

Theorem G.6. (VREx will use spurious features on graphs under covariate shift) Under the SCM of Equation (19), the objective min Θ V e [ R ( e )] has non-unique solutions for parameters of the GNN (3) when part of the model parameters { θ 1 1 ( l ) , θ 2 1 ( l ) , θ 1 2 ( l ) , θ 2 2 ( l ) } take the values

̸

0 < s < L is some positive integer, θ 1 and θ 2 have four sets of solutions of the quadratic equation, some of which are spurious solutions that θ 2 = 0 (although θ 2 = 0 is indeed one of the solutions, VREx is not guaranteed to reach this solution):

Proof. We will use some symbols to simplify the expression of the toy GNN. Denote n 2 + ϵ e as η . Use the following notations to represent the components of the L -layer GNN model:

(78)

C 1 , Z ∈ R N × 1 . We use C e 1 and Z e to denote the variables from the corresponding environment e . We further denote Z e = C e 2 ′ η .

Using these notations, the loss of environment e is

Denote the inner term C e 1 θ 1 + Z e θ 2 -˜ A e k X 1 -n 1 as l e . The variance of loss across environments is:

Take the derivative of V e [ R ( e )] with respect to θ 1 :

Calculate the derivative by terms:

Since n 1 and n 2 are independent standard Gaussian noise, we have E n 1 ,n 2 [ n 1 ] = E n 1 ,n 2 [ n 2 ] = 0 , E n 1 ,n 2 [ n ⊤ 1 n 2 ] = E n 1 ,n 2 [ n ⊤ 2 n 1 ] = 0 and E n 1 ,n 2 [ n ⊤ 1 n 1 ] = E n 1 ,n 2 [ n ⊤ 2 n 2 ] = N e if it is the noise from e . Also, since ϵ e and n 1 , n 2 are independent, we have E n 1 ,n 2 [ n ⊤ 1 ϵ e ] = E n 1 ,n 2 [ n ⊤ 2 ϵ e ] = 0 .

When

we have C e 2 ′ = I N e ∈ R N e × N e and C e 1 = ˜ A e s X 1 . Consequently, we get E n 1 ,n 2 [ Z e ⊤ n 1 ] = 0 , E n 1 ,n 2 [ Z e ⊤ Z e ] = N e + ϵ e ⊤ ϵ e .

Use the above conclusions and rewrite Equation (82) as (here we only plug in the value of C e 2 ′ ):

( ∗ ) and ( ∗∗ ) represent terms that are independent and associated with ϵ e , respectively. Additionally,

.

Multiplying Equation (84) and (85) and take the expectation on e , using the assumption that E e [( ϵ e i ) 2 ] = σ 2 ( ϵ e i is the i -th element of ϵ e ):

Next target is to compute 2 E e [ E n 1 ,n 2 [ l ⊤ e l e ]] and E e [ E n 1 ,n 2 [2 l ⊤ e C 1 ]] Since ϵ e has zero mean, we have:

and

Use Equation (86) (87) and (88) and let ∂ V e [ R ( e )] ∂θ 1 = 0 , we have:

Now we start calculating the expression of ∂ V e [ R ( e )] ∂θ 2 :

Plug Equation (89) in (91), we reach:

we conclude that

As for the derivative respect to θ 1 1 ( l ) , θ 2 1 ( l ) , θ 1 2 ( l ) , θ 2 2 ( l ) , when they take the special value in Equation (83), we have ∂ V e [ R ( e )] ∂θ 1 = 0 ⇒ ∂ V e [ R ( e )] ∂θ 1 1 ( l ) = ∂ V e [ R ( e )] ∂θ 2 1 ( l ) = 0 and ∂ V e [ R ( e )] ∂θ 2 = 0 ⇒ ∂ V e [ R ( e )] ∂θ 1 2 ( l ) =

∂ V e [ R ( e )] ∂θ 2 2 ( l ) = 0 , l = 1 , ..., L . So we conclude the solution induced by Equation (93) is the solution of the objective, and θ 2 = 0 is not a valid solution.

G.2.3 Proof of the Failure Case on Graphs of IRMv1 under Covariate Shift

We restate Theorem B.3 as Theorem G.7 below:

Theorem G.7. (IRMv1 will use spurious features on graphs under covariate shift) Under the SCM of Equation (19), there exists s ∈ N + that satisfies 0 < s < L and s = k such that optimizing the IRMv1 objective min Θ E e [ ∥∇ w | w =1 . 0 R ( e ) ∥ 2 ] will not lead to the invariant solution θ 2 = 0 for parameters of the GNN (3) when { θ 1 1 ( l ) , θ 2 1 ( l ) , θ 1 2 ( l ) , θ 2 2 ( l ) } take the special solution:

̸

Proof. From the proof of non-graph IRMv1 case Appendix G.1.1 we know that when the IRMv1 objective is optimized, we have ∇ w R ( e ) = 0 for all e . For the graph case, the expected risk of environment e is

where the definition of C e 1 and Z e follows Equation (78). Now let's check if the invariant solution θ 2 = 0 is a valid solution. Let θ 2 = 0 ,

If θ 2 = 0 is a feasible solution, then the following equation must hold for every environment e :

̸

When s = k , we get θ 1 = (( ˜ A e ) s X 1 ) ⊤ (( ˜ A e ) s X 1 ) (( ˜ A e ) s X 1 ) ⊤ (( ˜ A e ) k X 1 ) (we discard the trivial solution of θ 1 = 0 ). The value of this solution of θ 1 varies with environment e , and thus is not a valid solution. However, now we will show that optimizing IRMv1 does not necessarily lead to lower-layer parameters such that s = k . To reveal this, by taking the derivative of L IRMv1 w.r.t. θ 1 and θ 2 and let them = 0 , we can get two cubic equations:

̸

1 1 and ∂ L IRMv1 ∂θ 2 = ∂ L IRMv1 ∂θ 1 2 = ∂ L IRMv1 ∂θ 2 2 . At this time (when ∂ L IRMv1 ∂θ 1 = ∂ L IRMv1 ∂θ 1 1 = ∂ L IRMv1 ∂θ 2 1 = 0 and ∂ L IRMv1 ∂θ 2 = ∂ L IRMv1 ∂θ 1 2 = ∂ L IRMv1 ∂θ 2 2 = 0 but s = k ), from the form of Equations (98) and (99) we know that there exist solutions that θ 2 = 0 , and the solution of θ 1 and θ 2 both depend on E e ( F ( e )) , where F ( e ) is some random variable associated with e .

̸

G.2.4 Proof of the Successful Case on Graphs of CIA under Covariate Shift

Theorem G.8. Optimizing the CIA objective will lead to the optimal solution Θ ∗ :

From the analysis in Appendix F.3, we know that as long as the lower-layer parameters take any value that satisfies the form in Equation (27), even if s = k , we can get ∂ L IRMv1 ∂θ 1 = ∂ L IRMv1 ∂θ 1 = ∂ L IRMv1 ∂θ 2

̸

Proof. For brevity, denote a node representation of C e 1 c as C i 1 and the one of C e ′ 1 c as C j 1 . The same is true for C i 2 and C j 2 . In this toy model, we need to consider the expectation of the noise, while in real cases such noise is included in the node features so taking expectation on e will handle this. Therefore, we add E n 1 ,n 2 in this proof, and this expectation is excluded in the formal description of the objective in the main paper.

̸

̸

̸

Let ∂ L CIA ∂θ 1 = 0 , we have:

̸

Thus, we get two possible solutions of the invariant branch. The first valid solution is the optimal one:

The second valid solution is a trivial one:

Take the derivative of the objective w.r.t. θ 2 :

̸

Let ∂ L CIA ∂θ 2 = 0 , we get θ 2 = 0 . Thus, CIA will remove spurious features.

Now we show that when CIA objective has been reached (the spurious branch has zero outputs), the objective of min Θ E e [ L ( f Θ ( A e , X e ) , Y e )] will help to learn predictive parameters of the invariant branch θ 1 , θ 1 1 ( l ) and θ 2 1 ( l ) . When θ 2 = 0 :

Let ∂ E e [ L ( f Θ ( A e ,X e ) ,Y e )] ∂θ 1 = 0 , we get the predictive parameters

Plug the final solution back in ∂ L CIA ∂θ 1 1 ( l ) , ∂ L CIA ∂θ 2 1 ( l ) , ∂ L CIA ∂θ 1 2 ( l ) , ∂ L CIA ∂θ 2 2 ( l ) , ∂ E e [ L ( f Θ ( A e ,X e ) ,Y e )] ∂θ 1 1 ( l ) , ∂ E e [ L ( f Θ ( A e ,X e ) ,Y e )] ∂θ 2 1 ( l ) , ∂ E e [ L ( f Θ ( A e ,X e ) ,Y e )] ∂θ 1 2 ( l ) , ∂ E e [ L ( f Θ ( A e ,X e ) ,Y e )] ∂θ 2 2 ( l ) , we can verify that these terms are all 0.

G.3 Proof of Theorem 4.4

The following proof of Theorem 4.4 is adapted from Ma et al. [2021] and Mao et al. [2023]. We restate Theorem 4.4 as Theorem G.9 below.

Theorem G.9. (Subgroup Generalization Bound for GNNs). Let ˜ h be any classifier in a function family H with parameters { ˜ W l } L l =1 . Under Assumption 4.2 and 4.3, for any e te ∈ E te , γ ≥ 0 , and large enough N e tr , with probability at least 1 -δ , we have

̸

̸

where p ht i ( c ′ | c ) is the ratio of heterophilic neighbors of class c ′ when y i = c , const = 1 N 1 -2 α + 1 N 2 α e tr ln LC (2 B e te ) 1 /L γ 1 /L δ is a term independent of aggregated feature distance and the difference in neighboring heterophilic label distribution, where B e te = max i ∈ V e tr ∪ V e te ∥ g i ∥ 2 is the maximum feature norm.

To prove Theorem G.9, we need the following lemma that bounds the expected loss discrepancy between the train and the test node subgroups.

Lemma G.10. Given a distribution P over H , for λ > 0 , define the expected loss discrepancy between V e and V e ′ with respect to ( P, γ, λ ) as D γ e,e ′ ( P ; λ ) := ln E h ∼ P e λ ( L γ/ 2 e ( h ) -L γ e ′ ( h ) ) . Under Assumption 4.2, we have

̸

̸

Proof. According to Eq. (14) of Lemma 5 in Ma et al. [2021], under 4.2, we already have

In the following proof, the main goal is to bound the term ( Pr ( y j = c | g j ) -Pr ( y i = c | g i )) under the multi-classification OOD generalization setting. Using the Bayes theorem, we have

̸

̸

̸

Let consider the term Pr ( g j | y j = c ) Pr ( g i | y j = c ′ ) -Pr ( g i | y i = c ) Pr ( g j | y j = c ′ ) ( ∑ C c ′ =1 Pr ( g j | y j = c ′ ))( ∑ C c ′ =1 Pr ( g i | y i = c ′ )) for each c ′ = c , and bound all these terms respectively. When y i = c , denote µ i ( c ) ∈ R D × ( C -1) as the matrix composed of all the class means of node v i 's heterophilic neighbors, i.e. the columns of µ i ( c ) are µ ′ c , c ′ ∈ { 1 , 2 , ..., C } / { c } . The elements of p ht i ∈ R C -1 are corresponding p ht i ( c ′ ) , c ′ ∈ { 1 , 2 , ..., C } / { c } . According to Definition 4.1, we have

Further arranging the formula, we obtain:

Using Lagrange mean value theorem, we have

for x, y > 0 and ξ ∈ [ x, y ] (or ξ ∈ [ y, x ] ). Using Equation (116), we get

Let's bound (A), (B) and (C) respectively.

(

Using these results, we have

̸

̸

Now we can proceed with the proof of Theorem G.9.

Proof. Directly replace the result of Lemma 5 in Ma et al. [2021] with that of Lemma G.10 and following the proof of Lemma 6 and Theorem 3 in Ma et al. [2021], under Assumption 4.3, we finish the proof of Theorem G.9.

H Limitations

The theoretical analysis is limited to linear GNNs. However, there some justification for using a linear GNN. 1) Some recent works [Zhu and Koniusz, 2021, Wang et al., 2021b] observed that linear GNNs achieve comparable performance to nonlinear GNNs. Tang and Liu [2023] also theoretically proved that SGC can outperform GCN under some mild assumptions. 2) many recent works on the theoretical analysis of graphs/OOD generalization adopt linear networks [Lin et al., 2023, Wu et al., 2022b, Mao et al., 2023]. 3) Our theory matches the experimental results on the nonlinear GCN and GAT that CIA outperforms IRM and VREx.

We didn't implement some of the node-level OOD methods in our main experiments including Li et al. [2023a], Liu et al. [2023] because they didn't release the code. Another limitation is that we didn't provide an in-depth theoretical comparison between our method and existing node-level OOD methods, but only revealed the difficulty of invariant learning on graphs through the examples of VREx and IRM. However, this theoretical finding is sufficient to inspire the CIA solution and its enhanced version, CIA-LRA. We reserve a more comprehensive analysis of the failure of OOD methods on graphs and broader guidance for designing graph-OOD work for future research.

I Broader Impacts

Our work contributes to improving the node-level OOD generalization of GNN models. We believe our work could positively impact various fields by improving predictive accuracy in areas like healthcare, social networking, etc., potentially leading to better-personalized services and enhanced safety.

̸

̸

̸

̸

J Compute Resources

We use one NVIDIA GeForce RTX 3090 or 4090 GPU for each single experiments. All algorithms except EERM and GTrans can be executed on a single 24GB GPU when processing the largest dataset, Arxiv, without encountering out-of-memory.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction concisely state how we dissect the challenges of invariant learning on graphs and out strategies to tackle it. In the introduction, we list all our contributions and indicate the sections in which they are made.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We point out the limitations in Appendix H: we didn't compare the empirical results with some of the node-level OOD works because their codes are unavailable; we didn't theoretically analyze all invariant learning methods.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All supplementary assumptions, formal versions of the theorems, and full proofs can be found in Appendix G.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have provided all basic experimental settings and the hyperparameter search space in Appendix C to reproduce all our results. The detailed training procedure of CIA-LRA is provided in Appendix 1. We will release our implementation of CIA and CIA-LRA following the acceptance of our work.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: The datasets utilized in our paper are publicly accessible and remain unmodified. We will release our code following the acceptance of our work, complete with detailed instructions to ensure the faithful reproduction of the main experimental results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We adopt the default training details open-source GOOD benchmark, and we have stated some modifications of the setting in the main text and the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The improvements offered by CIA-LRA exceed the error bars of the best existing methods. Also, CIA significantly outperforms VREx and IRM across most dataset splits.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: In Appendix J, we report the type of GPU we use and the memory costs.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research conforms, in every respect, to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the potential impacts of out work in Appendix I.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: [TODO]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Yes. All assets used in our research, including code, data, and models, are properly credited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a novel regularization and alignment mechanism for invariant graph learning with the goal of obtaining better out-of-distribution (OOD) generalization.  Besides proving that existing methods tackling OOD generalization in other domains do not transfer to graphs, they propose CIA (environment-label dependent) and CIA-LRA (environment-label independent) to tackle the challenge of OOD generalization on graph data. Both, theoretical and empirical evidence is provided showing that the proposed methods provide benefits in OOD generalization.\n\nSTRENGTHS:\n- theoretical derivation of failure of IRM and VREx on graph data\n- theoretical analysis of CIA an CIA-LRA\n- empirical evidence that the proposed methods achieve OOD generalization significantly better than IRM and VREx\n- detailed discussion of further findings and proofs in appendix\n\nWEAKNESSES:\n## Methodology\n- in Fig. 1 it is not clearly described what the variables exactly refer to (e.g. $G$ is not mentioned in the text)\n- while I appreciate the theoretical findings, the GNN defined in Eq. 3 is a linear model, hence the theoretical analysis is somewhat limited. However, one should keep in mind that the analysis of non-linear and highly complex networks is hard\n- line 204: it is a bit counter-intuitive why invariant node features of nodes with the same class should significantly diverge when they are far apart in the graph. Also the empirical evidence in Fig. 7/8 does not really support this statement.\n- a brief definition/discussion of concept shift and covariate shift would be beneficial for clarity\n- in line 232: why is a sum operator employed here instead of a mean? Won't a simple sum be highly influenced by the graph size and connectivity/density?\n- line 248: inconsistent naming ($x_{cau}$ and $x_{inv}$)\n- in theorem 4.2 it is unclear what $\\alpha$ is and what the inequation $0 < \\alpha < \\frac{1}{4}$ means. Further a bit of a discussion on the goodness/tightness of the given bound would be good to assess the usefulness of that bound\n- proof G 1.1.: the proof only shows that $\\theta_2 = 0$ is **a possible** solution, but not a unique solution, isn't it? (Didn't check the proofs too deeply though)\n- Eq. 34:  the entire term does not depend on $\\theta_2$, so it is unclear to me why $\\theta_2 = 0$ is **the** solution to that equation. Wouldn't any arbitrary value be a valid solution then?\n- Eq. 35: in App. F.1 it is stated that the expectation of $\\epsilon^e$ is 0. Looking at Eq. 35, won't all mutliplicative terms entailing $\\epsilon^e$ cancel out in the expectation and thus in the IRM loss? This would make the IRM loss independent of the environment **in expectation** (not for some specific environment though). Thus, I don't see how the conclusion is reached that the IRM loss depends on $\\epsilon^e$ if, in expectation, it is 0.\n- line 995: which property are you referring to exactly?\n\n## Experiments\n- since the authors present an alignment-based invariant learning strategy for graph data, it would be good to see some more recent baselines taking a similar approach such as [1] and [2]\n- line 296: why does CIA-LRA improve upon CIA? To me it is counter-intuitive since CIA-LRA has no access to the environment labels, thus I'd expect this task to be harder\n\n## General notes\n- in the Introduction there should be more on related methods based on representation alignment such as [1] and [2]\n\n## References\n[1] Shift-Robust GNNs: Overcoming the Limitations of Localized Graph Training Data. Zhu et al. NeurIPS 2021.\n\n[2] Stable Prediction on Graphs with Agnostic Distribution Shifts. Zahng et al. KDD 2023.\n\nQUESTIONS:\n- how often did you repeat the experiments? (Maybe it's written somewhere and I didn't see it)\n\nLIMITATIONS:\n- major parts of the analysis are rather limited as they rely on the definition of the GNN in Eq. 3 which is a linear model\n- a comparison to some more recent baseline would increase the value of the paper", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigate the failure case of IRMv1 and VRex for graph data, and develop a novel approach CIA for node-level OOD generalization.  To adapt for the scenarios without environment labels, the authors propose CIA-LRA for the datasets without environment labels.\n\nSTRENGTHS:\n1. The paper is well-written and provides clear motivation.\n\n2. The paper is solid in both theoretical and empirical aspects.\n\n3. The evaluations are fair and convincing.\n\nWEAKNESSES:\n1. The author may want to clarify the reasoning behind the two SCMs corresponding to the two types of distribution shifts, as this connection is not entirely clear.\n\n2. In Equation 9, it should be $\\phi_\\theta(j)$.\n\n3. Assuming the sample label corresponds to the same invariant factors, it is unclear why $r_{\\text{same}}$ is necessary for reweighting.\n\n4. It appears that CIA-LRA requires all labels for reweighting alignment. However, the typical node-level classification setting is semi-supervised, which limits the applicability of this method.\n\n5. Line 231: \"to pairs with\" should be corrected to \"to pair with\".\n\n6. Line 854: In the proof of the non-graph case, $(A^e)^k$ appears, which seems to be an oversight.\n\n7. The general idea of sampling nodes from the same target label and different environments is similar to CIGA [1], which performs supervised contrastive learning to identify invariant subgraphs. The objective in CIGA may be a specific form of $d(\\cdot)$. The authors may want to compare and contrast their approach with CIGA.\n\n\n__reference__\n\n[1] Chen et al., Learning Causally Invariant Representations for Out-of-Distribution Generalization on Graphs, Neurips 2022.\n\nQUESTIONS:\n1. CIA-LRA learns masked subgraph, is the experiment inductive or transductive?\n\n2. In line 318-321, why CIA will be less performant than CIA-LRA, it seems not reasonable, as CIA will adopt environment labels while CIA-LRA doesn’t. \n\n3. $\\theta_m$ is separated from $\\theta$, how to ensure it will learn the invariant subgraph, under what assumptions?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses OOD generalization in node-level graph neural networks. The authors theoretically analyze why popular invariant learning methods like IRM and VREx fail on graph data, and propose two novel solutions: CIA and its environment-label-free variant, CIA-LRA. These methods enforce intra-class similarities to improve OOD generalization. The authors provide theoretical guarantees, including an OOD generalization error bound under the PAC-Bayesian framework. Experimental results on graph OOD benchmarks demonstrate the effectiveness of the proposed methods, particularly CIA-LRA, in improving node-level OOD generalization performance.\n\nSTRENGTHS:\n1. The paper provides a comprehensive theoretical investigation into why existing invariant learning methods struggle with graph data, supported by formal proofs.\n2. The proposed CIA and CIA-LRA methods offer innovative approaches to tackle node-level OOD generalization, addressing limitations of previous methods.\n3. The authors conduct comprehensive experiments on multiple datasets and compare against various baselines, demonstrating consistent performance improvements.\n\nWEAKNESSES:\n1. Limited comparison to recent node-level OOD method [1].\n2. Lack of visualization of the overall method architecture to help readers understand intuitively.\n3. While Appendix J briefly mentions GPU requirements, a more detailed analysis of the computational complexity and runtime comparisons of CIA and CIA-LRA against baselines would be valuable.\n4. Insufficient citations of recent methods, such as [2] [3].\n\n[1] Learning invariant representations of graph neural networks via cluster generalization\n\n[2] Joint Learning of Label and Environment Causal Independence for Graph Out-of-Distribution Generalization\n\n[3] Individual and structural graph information bottlenecks for out-of-distribution generalization\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes the failures of standard invariant learning techniques for node classification.\nThrough theoretical analysis, they argue that methods such as IRM and VREx will learn spurious features on graph data. Using this as motivation, they design a new method, CIA, that introduces additional invariance regularization to better guide the GNN to learn better representations. They verify the effectiveness of their method on multiple benchmark datasets.\n\nSTRENGTHS:\n1. I think the authors do a good job of showing the failures of methods such as IRM and VERx on graph data. In particular, the theoretical analysis is interesting.\n\n2. The proposed method is straightforward and follows naturally from the theoretical analysis. Furthermore, the theoretical justification in Section 4 is welcome.\n\n3. The performance of the proposed method is good compared to baselines.\n\nWEAKNESSES:\n1. The authors seem to leave out some baselines in their experiments. For example, see [Wu et al., 2024] and [Li et al., 2023b]. From Footnote 1 at the bottom of page 2, I surmise that this is because they are not \"general solutions\". However, I find this to be a weak reason for not including them as baselines. Why should that be a disqualifying measure? The authors should better motivate which baselines they compare against and which they don't.\n\n2. Similarly, the authors should include the performance when using MatchDG. In the paper, the authors note the similarity of their method to MatchDG. As such, it reasons that it should be included in the experiments as a baseline. If not, the authors should provide sufficient reasons why to not include it.\n\n3. There's a lack of discussion on the efficiency (time complexity) of the method. This is important, as the authors note that the performance of CIA-LRA peaks at around 6-10 hops (see line 674). I'm curious as to how efficienct it is to consider such a large number of hops, especially on larger graphs. Naively, this seems to me that it can be a drawback of applying this method on larger graphs.\n\nQUESTIONS:\n1. What's IRMv1? You begin by mentioning the failure of IRM but then start calling it IRMv1. What's the difference?\n2. On line 135, the authors mention that the GNN used in their analysis resembles SGC. I'm curious how this effects the analysis. How different would the conclusions be if we considered a more standard GNN like GCN? To be clear, I'm not asking for a full analysis here or anything, I'm just curious of the practical limitations of considering SGC in the analysis.\n3. Could you share the time complexity of your method?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}