DivIL: Unveiling and Addressing Over-Invariance for Out-of- Distribution Generalization
Authors: Jiaqi WANG
Yuhang Zhou
Zhixiong Zhang
Qiguang Chen
Yongqiang Chen
James Cheng
Venue: Accepted by TMLR ()

Abstract: Out-of-distribution generalization is a common problem that expects the model to perform well in the different distributions even far from the train data. A popular approach to addressing this issue is invariant learning (IL), in which the model is compiled to focus on invariant features instead of spurious features by adding strong constraints during training. However, there are some potential pitfalls of strong invariant constraints. Due to the limited number of diverse environments and over-regularization in the feature space, it may lead to a loss of important details in the invariant features while alleviating the spurious correlations, namely the over-invariance, which can also degrade the generalization performance. We theoretically define the over-invariance and observe that this issue occurs in various classic IL methods. To alleviate this issue, we propose a simple approach Diverse Invariant Learning (DivIL) by adding the unsupervised contrastive learning and the random masking mechanism compensatory for the invariant constraints, which can be applied to various IL methods. Furthermore, we conduct experiments across multiple modalities across 12 datasets and 6 classic models, verifying our over-invariance insight and the effectiveness of our DivIL framework. Our code is available at https://github.com/kokolerk/DivIL.

Published in Transactions on Machine Learning Research (02/2025)
DivIL: Unveiling and Addressing Over-Invariance for Out-of- Distribution Generalization
Jiaqi Wang'
jpuong288cse.cuhk.com edu.hk
The Chinese University of Hong Kong
Yuhang Zhou
ralph. whh:hhhu@@mdlc com zzzhang021 60gmmail com qgchen Gir hit. edu.cr
Zhixiong Zhang"
Qiguang Chen
Harbin Institute of Technology
Yongqiang Chen
yychen24 @graail com
CMU CLeaR Group
James Cheng The Chinese University of Hong Kong
jcheng www.kesss....cc.oo. edu.hk
Reviewed on OpenReview: https: opwnreviee......... net/ forum? id= 2Zon4ATYsh
Abstract
Out-of-distribution generalization is common problem that expects the model to perform well in the different distributions even far from the train data. A popular approach to ad- dressing this issue is invariant learning (IL), in which the model is compiled to focus on invariant features instcad of spurions features by adding strong constraints during training. However, there are some polential pit of strong invariant constraints. Due to the limited number diverse environments and over-regularization in the feature space, it may lead to loss of important details in the invariant features while alleviating the spurious correla- tions, namely the over-invariance, which can also degrade the generalization performance. We theoretically define the over-invariance and observe that this issue occurs in various classic IL methods. To alleviate this issue, we propose simple approach Diverse Invariant Learning (DivIL) by adding the unsupervised contrastive learning and the random mask- ing mechanism compensatory for the invariant constraints, which can be applied to various IL methods. Furthermore, we conduct experiments across multiple modalities across 12 datasets and 6 classic models, verifying our over-invariance insight and the effectiveness of DivIL framework. Our code is available in https:://ittai com htt///ooolkkkkiilll..
1 Introduction
Modern machine learning methods have achieved great success across various domains such as natural lan-
guage processing, computer vision, and graph neural networks (Kipf & Welling 2017 Devlin et al. 2019 Xu
ct al. 2019). However, these methods heavily rely on the assumption that training and testing data come from the independent and identically distribution(T... Dasspassppiii) (Liu et al. 2023 Peters et al. 2016c
When faced with out-of-distribution (OOD) data, almost all of these methods generalize poorly since they are prone to inherit data biases from the train as shortcuts (Koh al.) 2021
Gulrajani & Lopez-Paz
2021
Gui et al. 2022 Ji et al.) 2022 Xie et al. 2024h.
A canonical method for the OOD generalization is invariant learning (IL) based on the invariant principle
from causality AArjovsky et al. 2019 Ahuja et al. 2021 Peters el al. 2016b Krueger el. al. 2021 Yao et al.
* Equal contribution.

---

Published in Transactions on Machine Learning Research (02/2025)
Invariant
Encironment
environment
wheel
O-o
tree
Invariant
Spucliou
8
invariance
X
star
house
class
class 2
Pall Data
(a) Structural causal model.
(b) Illustration of the over-invariance issue in the graph field.
Figure 1: (a) shows the structural causal model of invariant and spurious features in relation to the invariance and the environments. (b) shows the over iveriinvrianne issue in the graph field. Each graph G consists of the invariant subgraph Ge (star, house) and the spurious subgraph G, (wheel, tree). Previous IL methods alleviate spurious subgraphs while sacrificing important details of invariant subgraphs (The circle is the invariant subgraph G. identified by the model.), causing the over-invariance issue.
2024 Chen et al.) 2024 Wang et al. 2024 Xu et al. 2025 As seen in Figure la the basic assumption of IL is that each data is determined by the invariant feature Za and the spurious feature 2 and only learning the invariant features can achieve the success of OOD generalization. Specially, the two variables are unobservable and the invariant one is stable across environments (Z T S(C) while the spurious one changes with environments (S). The key challenge of II, is how to learn the invariant features while alleviating the spurious features. To achieve this goal, various IL methods add regularization to the original Empirical Risk Minimization (ERM) loss, for example, IRMv1, VREx, and Fishr (Krueger et al.
2021
Rame et al.
2022
introduce gradients-induced losses and EIIL, EILLS, and CIGA (Creager al.
2021b Fan et al.
2023
Chen et al.
2022) adapt the environment irrironment--nduueed induced penalties. Others (Peters et al.)
2016b Ahuja et al,
2021
Wu et al.
2022 Sagawa* et al. 2020) apply complex invariant strategies during the training process
to extract invariance.
However, the rigorous invariance definition (Arjovsky et al. 2019 must (1) be Bayesian optimal across all environments and (2) completely abandon the spurious feature, which gives a strong restriction to the representation learning. Despite improvement in performance on the test set, most IL methods perform poorly compared to ERM on the train set (Creager et al. 2021b Kamath et al.) 2021a Krueger et al., 2021). Furthermore, extracting invariant features requires the train data from different environments which are often artificially divided or typically absent in real-world scenarios. Some studies (Lin el al.) 2022: Kamath et al.) (021b) have proved that in cases with insufficient environments in the train set, IL fails to distinguish the invariance and the spurious correlation. This reveals two critical dilemmas of IL to capture the invariance; while beneficial for OOD generalization, its over-regularization limits the representation and requires an infinite number of diverse environments.
In this paper, we highlight that during the pursuit of invariance, current IL methods tend to use fewer features to avoid any risk of violating invariance, referred to as the over invrrinvaee Figure Ib demonstrates example of over-invariance in the graph field where II. predicts the label only by the small subgraph of the invariant subgraph Ge and ignores other part of the graph. However, this may come at the cost of losing enough details and diversity of the invariant feature despite alleviating the spurious correlation, which also degrades the out-of-distribution generalization. Furthermore, we rigorously define the over-invariance and conduct simulation experiments on two classic IL methods, IRM (Arjovsky et al. 2019) and VREx (Krueger et al. 2021), verifying the existence of the over-invariance.
Built upon our observation, propose simple and novel method Diverse Invariant Learning (DivIL) with a focus on promoting richer and more diverse invariance. Since the quality of the invariant feature plays an essential role in IL, we consider striking a balance between the strong regularizers to alleviate spurious correlations and the detailed capture of the invariant features. We combine invariant penalties and unsupervised contrastive learning (UCL) with random data augmentation to extract domain-wise and sample-wise features, eliminating the reliance on the environments. Mcanwhile, we mask the front part of the UCT. feature as zero to reduce overfitting to spurious shortcuts (Jing et al. 2021). We evaluate DivIL
2

---

Published in Transactions on Machine Learning Research (02/2025)
extensive set of 12 benchmark datasets across natural language, computer vision, and graph domains with various distribution shifts, including challenging setting from AT-aided drug discovery (Ji et al. 2022). We demonstrate that DivIL can significantly enhance the performance of invariant learning methods, thereby reinforcing our insight of the over-invariance issue invariant learning. Our main contributions arc:
We discover and theoretically define the over-invariance phenomenon, i.e., the loss of important details in invariance when alleviating the spurious features, which exists in almost all of the previous IL methods.
We propose Diverse Invariant Learning (DivIL), combining both invariant constraints and unsuper- vised contrastive learning with randomly masking mechanism to promote richer and more diverse invariance.
Experiments conducted on 12 benchmarks, 4 different invariant learning methods across modali- lies (graphs; vision, and natural language) demonst that DivIL effectively enhances the out-of- distribution generalization performance, verifying the over-invariance insight.
2 Background
In this work, we focus on the OOD generalization in the classification task. Specifically, given set of datasets D = (D*). collected from multiple environ- ments Call, samples (X,,Y? YY*) E D' are considered as drawn independently from an identical distribution Ps. A model f w wo generically has a representa- tion function + X + H that learns the meaningful feature Z for each data and predictor to predict the label Y based on the feature Z. The goal of OOD generalization ion is to train the model on the train set D (sE.... that generalizes well to all (unseen) environments.
Z
Zc
Zz
7c
H - Y
(a) FIF
(b) PII
(c) MIlF
Figure Illustrations of three structural causal mod- els (SCMs).
It is known that OOD generalization is impossible without assumptions the environments Call- Thus formulate the data generation process with structural causal model and latent variable model Pearl (009
The generation of the observed data X and labels Y are controlled by set of latent causal variable C' and spurious variable S as suggested in Figure
la ie.,
Ze = Syen (C); 2. = Syen (5); 2= (Ze. Za)
X = ggen (Z); = f(z))-
Ze is the invariant feature determined by the causal variable C, Z. varies with the environment S, and label Y is determined by the casual variable C. Besides, based on the latent interaction among C, S and Y, SCM can further categorized into Full Informative Invariant Features (FILF) and Partially Informative Invoriant Features (PII).. Furthermore, PIIF and FIF shifts can be mixed together and yicld Mixed Informative Invariand Features (MII)), as shown in Figure We refer interested readers to Ahuja et al. (0021) for detailed introduction to the generation process.
Invariance Learning. The invariance learning (IL) approach tackles the OOD Generalization problem by predicting invariant features within the data. Considering a classification task, the objective of invariance learning is to find an extractor & such that ((X+) = Z. for s E eall. The learning objectives for and w are formulated as:
mFn RC (w (2.) s.l. Z. 2. $(X*).
where 12t is the risk of the function, which is implemented by the cross-entropy loss Loe = 1og))) Sea 1. is the strong restriction for the model that distinguishes the representation from the interventions
3

---

Published in Transactions on Machine Learning Research (02/2025)
from the environments, only obtaining the information about the invariance C. Besides, the basic assumption of the OOD generalization is the environments, which are usually not accessed in real scenarios. So there are essentially two distinct categories of Inverse Learning (IL) methods, depending on whether the environments are explicitly labeled in the training datasets. In this paper, we remove environmental restrictions and focus on environments not covered in the training dataset.
3
Over-invariance Issue
3.1 Invariant Features Derived from Invariance Principle
The theoretical guarantee to previous IL methods is the invariant principle, which defines what predictor is invariant in different environments. Following Arjovsky et al. (2019), we formally define the invariant principle follows:
Definition 3.1 (Invariance Principle) We define a data representation function & as eliciting an invari- ant predictor w ocross environments S if there is classifier simultaneously optimal for all environments. Specifically, this condition can be expressed as follows:
w E argming o(X))), As S.
2
The invariant principle gives the rigorous definition of the invariant model. A data representation function & elicits an invariant predictor across environments S if and only if, for all feature Z in the intersection of the supports of (X*)), we have [[(*)((X*)) = Z| = E|y ((x*)= 2], for all 8, E S. For more clarity, we further define the invariant feature derived from the invariant principle.
Definition 3.2 (Invariant feature) Let I = 10 represend selection function. An invariian feature of label Y under both train distribution per and test distribution pte is any subset Z 20 1 of the latent feature Z E Ie² that satisfies
E E 2 = Eptr [Y] Epic Z9 = EYtc.Y/Z].
(3)
An invariant feature, denoted Z", consists of features from X that carry predictive power for the target Y across both training and test cnvironments. In other words, Z provides as much information about Y as the full feature set X, ensuring that the predictive relationship is stable across environments S.
3.2 Rethinking the Effect of Invariant Features in OOD Generalization
In the above section, formally define the invariant feature. However, this definition imposes significant restriction on the feature space, potentially leading to a degradation in ont-of-distribution generalization although it alleviates spurious correlations. In this part, we attribute two potential risks of the invariant feature dilemma: 1) limited environmental diversity and 2) regulaegulaiization via the loss function.
Limited Numbers of Diverse Environments. A lack of sufficient environmental diversity in real-world scenarios fails meet the requirements of the invariance principle. According to Equation 2 the hypothesis of Invariant Risk Minimization (IRM) assumes that the environment labels are well-defined and that all environments Call must be represented in the expectation condition. However, even if the environments in the training set differ, they can still be significantly dissimilar to those in the test set, causing the model to learn shortcuts based on the training data.
Over-Reeullarzation via Implementation. In addition to the inherent limitations of the environment collection, there is also a gap between the theory based on the ideal assumption and the implementation in practice. For example, IRMv1 dArjovsky et al. 2019)) employs the l norm of the model gradients on the Empirical Risk Minimization (ERM) loss learn the invariance across training environments. This penalty-
based approach is also utilized by various IL met such VREx (Krueger et. al. 2021), Fishr (Rame)
4

---

Published in Transactions on Machine Learning Research (02/2025)
et al. 2022). and IB-IRM (Ahuja et al. 2021 All these methods share the common goal of constraining the rate of feature changes across different environments, preventing overfitting in a specific environment due to the rapid changes in gradients. This strategy aims to force the model to learn the invariant features by stable adjustments in train environments. However, the strong second-order regularization terms in the ERM loss restrict the diversity of invariant. features, thereby limiting the model's 's ability to capture a broad range of relevant features.
The above two reasons highlight the failure cases of the invariance principle, revealing that rough constraints may inadvertently harm valuable invariant features, a phenomenon refer to over over-invariance. Formally, due to the unavailability of test environments Pie, such an invariant principle could inadvertently overlook minor invariant characteristics, potentially misinterpreting them as mere hallucination tions of spurious features, dubbed as the over- iee-innvreannee issue. In particular, we define the over ver-invare issue as follows:
Definition 3.3 (Over-Invariant feature) Let 7o be the invoriant feature of Y, if there crist subset O of Z that satisfies
Ept [0000 E.. Ex,111199 [M]2 1os. * Epto [ (9]
(4)
then Of is the over-invarians feature.
An over-invariant feature is the subset of the invariant feature (O C Z") that performs similarly in the train environments. While Oo maintains predictive accuracy for Y in the training distribution ptr, it only contains part information of Y, may leading poor OOD performance in pte,
3.3 Theoretical Analysis
Since distinguishing between invariant and spurious information the hidden feature space is challenging in real-world scenarios, we create synthetic dataset to simulate various distributions, allowing us to further observe the existence of the over-invariance.
Definition 3.4 (Data Generation) Given the data (x. y, Us), y is the label and ys is the enviroument, y random variable taking value -1 with with probabilily and with with probability 1 s. The data x E RRd is composed of two components: the invariant feature ze and the spurious feature Iz, where sample is generated as follows:
is uniformly sampled from {-1.1} and ys Rad(s) x y where Rod(s)
IRd, Ie E Rd_ and d = de dg. Each
x [x TRdd where Iee N(HeH, o2),
N(HaVs:: 02).
Here, Pe IR and μa E Rd. represent the mean of the Gaussian distributions. oe E Rdoxdo and a xd, denotes the standard deviations that control the variability.
To analyze the preferences the invariant learning for different components of invariant features, we quantify the strength of the subset features, by masking the irrelevant data as and measuring their l2 norms of the learned representation. Intuitively, this measures how much information the model extracts from the specified dimensions collectively.
Definition 3.5 (Strength) Given subset of dimensions {m, +1, 1,...,n). n), we mask all other dimensions x as 0 and pass the masked data Xmn through the featurizer. Let x. be the representation function learned by the invariant learning. The strength of the selected feature subset is follows:
(tren)th(xmm.)) = ||00 (Xman)
(5)
this paper, we set the dimensions of both the invariant and spurious features to de = d, = 8. Let 1, the all-one vector of length and diag be the diagonal matrix. For the invariant feature Te. we set Mo 10 1d, and a. = diag(5, 5,3,3, 1, 0.1, 0.1), where different variances represent different important levels of invariance where high variance means more important. For the spurious features Vs, we set μz 10 1d,
5

---

Published in Transactions on Machine Learning Research (02/2025)
var
+ Var
V
var-5
V87=0.1
var-1
var
var-5
3:375
4.2
3:35
3:325
Strongth
4.1
3:275
3:25
3:225
3.175
3.9
3-15
1.5
3
(gg(spurious variance)
log(s lo(ssuuiiuus variance)
(a) feature strength using IRM
(b) feature strength using VREx
The strength on different subsets of the invariant feature, lower strength means less preference, verifying the existence of over-invariance issue. The X-axis represents the logarithm of spurious variance aa. The Y-axis shows the strength of the corresponding subset invariant features ((z) under varying invariant variances o, = {0.1, 1,3,5) For each configuration, we run 10 different seeds and report the average results.
Figure 3:
and uniformly sample dg from the range [10-3, 100.s] simulating noisy environments. We set 0.3 in the train set and s = 0.7 in the test set, reprosenting the OOD environments. We train a two-layer perceptron network featurizer & x - R' where each layer consists of linear transformation and a ReLU activation and a one-layer linear classifier predictor w. We take the classic invariant learning methods TRMv1 (Arjovsky et al. 2019 and VREx (Krueger et al. 2021) for example.
Figure g illustrates the strengths inside the invariant features. We separate the invariant data De into parts based different variances 10.1,1,3,5) and calculate their strengths varying with different spurious variances a, simulating the noisy euvironments. The results indicate that while all of the features are invariant, their strengths vary. IL methods are selective to invariant features with some invariant features being learned less effectively than others. The lower strength of the subset of the invariant feature suggests that IRMv1 and VREx may struggle with key parts of invariant features, leading to the over-invariance issue. Formally, we give the following informal proposition to further illustrate the over-invariance:
Remark 3.6 (Over-invariance issue) Our synthelic experiment indicates that, with high probubility, there erists subset of the invariant data Fer denoted as the over-invariant data oe, where the strength of the remaining invoriont data (z)) vary significantly, potentially approaching zero in extreme cases:
strength(z.(o.) > strength(o.) +0.
Thus, over-inoariance may occur at test time.
Over-invariance issuc in the real datascts. We give more visualization examples and discuss them in the interpretation architectures XGNN as shown Appendix C.2 We have two observations based on the above visualization examples. 1) Invariant learning methods tend extract the subgraph of the ground truth invariant subgraph, thus verifying the over-invariance issuc. 2) When confronted with multiple invariant subgraphs, these learning methods usually focus only of them rather than considering all the invariant subgraphs. The above two observations aligned with our synthetics experiments in Figure 3 measured by the strength, thus further verifying the occurrence of the over- ive-innvr issue.
4 DivIL: Diverse Invariant Learning
Built upon our analysis of the pitfall of the invariant feature and the observation about over- over-inarrianee issue, we propose a novel approach Diverse Invariant Learning (DivIL) that integrates unsupervised contrastive learning (UCT.) and the masking mechanism, which can be applied various IT. methods.
Enhancing the Environments by Random Data Augmentation As discussed in Section 3.2 one fundamental assumption of the invariant principle the requirement for infinite number of environments,

---

Published in Transactions on Machine Learning Research (02/2025)
Positive
Negative
random mask
Transformer Module
A person on a horse jumps... at camera...
Children smiling
Gs
Gc
edge remove node drop
Figure 4: Data augmentation of Lust in DivIL across multi-modals. Left up: random masking the figure to 0. Left down: edge removing and node dropping for the graph. Right: We feed the same input sequence to the encoder twice by applying different dropout masks to obtain the positive pair.
which is impractical in realdworld. To compensate for this limitation, we employ data augmentation to produce a wider range of samples. By introducing variations of data, we disrupt the spurious correlation between the labels and the environments from the train data, fostering the creation of diverse environments. We only use random data augmentation without careful designs which enough to show the benefits. As illustrated in Figure ! we use edge dropping, node dropping, and random subgraph extraction for graph following You et al. (2020) and Ding ct al. (2022); randomly masking the data ==xxx3 zero with a probability of for CV, and obtaining z;; 23 with dropout. masks on fully-connected layers as well as attention
with a probability of p for NLP (Vaswaní et al.. 2017 Gao et al. 2021
Alleviating the Restriction by Unsupervised Contrastive Learning To alleviate the spurious cor- relations, 1L methods usually add strong penalties the loss (Arjovsky et al., [2019] Krueger al.) 2021 Rame et al. 2022) which tends to suppress subtle yet important details, thus causing the over-invariance is- sues. Unsupervised learning (UT.), particularly unsupervised contrastive learning (UCI.) (Oord et al.
2018
provides a powerful mechanism for addressing this by learning the sample-level features, promoting the focus
on the minor details of the invariance (Xue et al.] 2023a; Jing el al. 2021
Chen el al. 2020a. Qin et al.
2022 Zhang et al. 2022b Formally, Luct is defined as follows:
exp(| - =11le)
Lud
log Ej exp(11 27112 (-) exp 211B
(6)
Here, z represents the feature vector of the original sample, z is the augmented version of the same sample, treated as positive pair, and zj is the different sample (negative pair).
Overall Training Objective of DivIL By merging our UCL loss with the conventional invariant loss, we aim to balance the diversity of invariant features while preserving the effectiveness of the prior invariant loss in reducing the influence of spurious features. The final objective function of Diverse Invariant Learning (DivIL) is follows:
LoiviL = Lpred + ALa + BLuel
(7)
Here, Lpred is the cross-cntropy loss, La is any invariant loss from IL methods, and Lud is our proposed unsupervised contrastive loss. The hyperparameters À > 0 and 3 > 0 control the trade-off between invariance and diversity, which can be tuned based on the task. As seen in Figure 5 we observe that after adding the UCL loss, the strength of invariant features increases across various IL penalties, verifying our analysis that incorporating unsupervised contrastive learning as complement to the invariant loss effectively enhances OOD generalization.

---

Published in Transactions on Machine Learning Research (02/2025)
URM URM
Vatx-dicl.mmthhe
Featurs Strength
Feature Strength
3/5
3:3
5
0.1
0.1
Invariant Feature's Variance
Invariant Feature's Variance
Figure 5: The increased strengths after incorporating UCL with IRMv1 and VREx. The X-axis represents different invariant variances o, = {5, 3, 1, 0.1J.The Y-axis shows the strength of the corresponding subset of invariant features ((x) before and after adding the UCL. For each configuration, we run different seeds and report the average results.
A Algorithm Overall Training Objective of DivIL
1: Input: Train datasel. Der = (D*) secu.Srua and test dataset Die = (sen for 4 ttc.
2: Output: Trained model f = w a +b
3: Function: Representation function 4 X + H, Invariant predictor w +
Hyperparametors: À 0, 8 > O, mask probability p, learning rate 7
// DivIL OOD Generalization Training
for each batch (X,, Y,) B from Dr do Calculate cross-cntropy loss: Lpred
7:
8:
Calculate invariant loss: L1
9:
Using data angmentation technique to get the 2 and 2'. masking the front p percent of the entire dimensions Calculate unsupervised contrastive loss: Luel in equation 6
10:
and z',
11:
12:
Calculate total loss: LoivIL Lpred AL1 + 3L
Compute gradients: V 0Casii V Olain
13:
14:
Update model parameters:
15:
+ 4 V
16:
w w w V
17:
end for
Remark 4.1 (Effectiveness of DivIL) Our syathelic experiment indientes that for the original over- invariant data oe illustrated in Remark 3.1, the strength of oe learned by DiVIL in equation is bigger than the original invariant learning:
strength (o.) strengthyyM//vRE...
Thus, DivIL effectively miligates the over- aner-inaariance issue at test time.
Enhancing Diversity of the Invariance via Random Masking Contrastive learning methods, by repelling negative samples, can alleviate the over-inarriane problem to extent. However, when faced with strong data augmentation or deep-layer implicit regularization, the model performance can also remain suboptimal (Jing et al. 2021). To further enhance feature diversity, we trained non-linear projector to scatter the representation space spectrum. ionally we introduced a random masking mechanism to the features to overcome over-invrriance. We set the first p dimensions of the contrastive learning feature dimension 0, that 21.p 0.
In conclusion, the detailed training procedure of DivIL is shown in Algorithm
8

---

Published in Transactions on Machine Learning Research (02/2025)
Table 1: Performance on real-world graph dataset, demonstrating better OOD generalization ability of DivIL. The blue, gray, and Underline highlight the first, second, and third best results, respectively. All results are reported with mean std across seeds {0, 3,
EC50 EC50-Assay Assay
FC50-Scaffold
Ric0
Ki-Assay
Ki-Scaffold
Ki-size
CMNIST-sp
Gmap--ST2
ERM
70.30 2.15
63.45
1.43
61.47
1.99
70.43 2.19
72.43 1.38
71.43 3.60
25.67 + 9.70
82.75 + 0.20
IRM
71.00 = 4.47
60.42 = 0.69
60.30 = 1.18
70.39
1.44
69.38 2
70.80 -
19.19 2.83
82.31 + 1.22
VREX
71.91
62.07 1 1.30
61.03
1.27
68.74
18.7441:4.11 4.13
70.51 3.13
70.34 1 4.26
14.91 1.85
82.40 0.63
EIIL
70.39 + 3.11
61.20 + 1.68
60.31 = 1.64
69.20 = 2.29
69.99 = 1.58
72.78
1178:300 3.08
22.37 + 7.35
82.31 = 1.50
IB-IRM
67.04
61.04 + 1.13
62.20 0.64
71.94
74.16 1.29
71.15
37.44 7.36
81.95 0.74
CNC
74.96 2.48
63.59 0.87
2.14
74.08
3.67
67.54 1.26
5.21
19.41 +.14
80.72 1.15
CNCP
2.674
62.05 1.22
60.53
1.33120 2.14 2.14
74.13
1 2.46
67.70 2.85
67.54 3.337
24.99 4.70
80.76 1 0.64
CIGA
76.63 + 1.16
66.25 1.49
63.66 1.15
71.55
1.84
2.48 71.54
74.52 3.09
41.35 5.56
82.89 + 0.97
DIVIL
77.00 + 1.52
67.41 + 0.55
64.33 + 0.83
72.64 + 2.78
73.38 0.91
75.99 + 2.46
46.29 + 11.2
83.30 + 0.91
5
Experiments
We evaluate DivIL and compare with IL methods on a range of tasks requiring OOD generalization, DivIL provides generalization benefits and outperforms II. methods on a wide range of tasks, including: 1) graph datasets such as the synthetic Spurious- Motif and drug discovery, 2) Colored MNIST (CMNIST) dataset, and 3) natural language datasets.
5.1 Experiments on Graph
Datasets We employed one synthetic dataset along with eight realistic datasets, including the Spurious- Motif datasets introduced in Wu et al. (2022). These datasets consist of three graph classes, each containing designated subgraph as the ground-truth explanation. Additionally, there spurious correlations between the remaining graph components and the labels in the training data, challenging the model's ability to differ- entiate between true and spurious features. The degree of these correlations is controlled by the parameter b, with values of 0.33, 0.6, and 0.9. Furthermore, to examine our method in real-world scenarios charac- terized by more complex relationships and distribution shifts, we incorporated the DrugOOD dataset (Ji et al. 2022) from Al-aided Drug Discovery, which includes Assay, Scaffold, and Size splits from the EC50 category (denoted EC50-*) and the Ki category (denoted as Ki-*). Additionally, we included tests the CMNIST-sp datasel, which consists of superpixel graphs derived from the ColoredMNIST datasel using the algorithm from Knyazev et al. 2219),, featuring distribution shifts in node attributes and graph size. We also tested on the sentiment analysis dataset Graph-SST2 (Yuan et al. 2020), which is formed by converting each text sequence from SST2 into a graph representation.
Baselines We compared DivIL with insspird invariant graph learning methods, such as IRM Ar- jovsky et al. (019), v-Rex Krueger et al. (2021) and IB-IRM Ahuja et al. (2021). Addilionnlly, we evaluated DivIL against methods like EIIL Creager et al. (2021b), CNC, CNCP Zhang et al. (2022a), and CIGA Chen ct al. [2022). all of which do not require environment labels. Notably, CNC, CNCP, and CIGA employ contrastive sampling strategies to address the OOD problem. We implemented Lu following the SOTA method CIGA, and for Luc,, we selected the best-performing DA techniques, such edge removal, node dropping, and subgraph extraction, based on You et al. (2020). We report classification accuracy for the Mottf, CMNIST-sp, and Graph-SST2 datasets, and ROC-AUC for the DrugOOD datasets. The evaluation is conducted five times with different random seeds ((1, 2, 3, 4, 5}), selecting models based on validation performance. We utilized the GCN backbone Kipf & Welling (0177) with sum pooling to enhance across all experiments.
DivIL outperforms previous IL methods. As demonstrated in Table 1 DivIL shows better general- ization ability than all baseline models on real-world datasets. Specifically, in the MNIST-sp dataset, DivIL surpassos CIGA by 5%. Furthermore, the ki-scaffold and ki-assay datasets, CIGA performs worse than ERM, while DivIL by implementing the Ca on CIGA achieves higher performance. The results not only highlight the competitive edge of DivIL over established baselines but also emphasize its generalization across

---

Published in Transactions on Machine Learning Research (02/2025)
Table 2: Performance on Spurious-Motif dataset with different GNN backbones. DivIL can enhance the per- formance on various architectures. The blue, gray, and Underline highlight the first, second, and third best re- sults, respectively. All results are reported with mean + std across seeds {0, 1, 2.3, 4).
assay - scaffold
size
oddge remove node drop
0.8
0.64
0.63
0.60
0.58
0.5
0.6
0.48
0.4
0.2
0 subgraph edge remove node (a) drugaod with ec-
SPMotif-D.33 SPMotif-0600 SPMotif-0.90
b-0:33
b-0.
(b) SPMotif
GNN
GIN La
47.8
49.21 + 4.2
44.11 = 5.5
50.67 3.83
50.39
+ 2.19
42.35 + 6.36
Figure 6: Ablation study of different graph data augmentations in DivIL. (a) compares the perfor- mance of subgraph extraction, edge removing, and node dropping on the ec-* category from DrD datasets. (b) illustrates the performance of edge moving and node dropping on SPMotif with different.
+DivII.
51.48 + 5.48
51.77 + 4.89
45.84 + 3.82
GCN
58.51 +: 2.84
50.51
4 4.75
44.67 + 3.5
+ha
67.53 = 1.35
39.96 + 8.59
47.66 + 6.08
+DivIL
67.81 = 3.33
62.79
+: 3.86
50.93 + 9.05
XCNN
+ CIIGI)
57.58 = 3.73
58.11 + 4.29
52.14
+DI (GIN)
63.86 +: 2.19
60.31 + 2.04
52.54 + 6.57
OOD shift biases.
+ (CCG
63.37 = 4.27
65. 45 + 4.91
59.64 + 4.64
DivIL (GCN)
63.90 +
70.03 + 3.66
66.85 + 6.61
varying datasets. Such muanced performance differentials underscore our capabilities of DivIL in navigating complex real-world datasets, positioning the over-invariance issues a crucial problem.
DivIL shows effectiveness on various backbones. Additionally, we incorporate XGNN, an inter- pretable GNN to extract the invariant subgraph Ge commonly used in graph OOD models (Wu et all Li et al. 2022 Chen et al. 2022). Specifically, a XGNN W, o 42 is with extractor o2 9 - 9 that w2(Ge) with a downstream classifier w : + V. Table shows that DivIL significantly outperforms Vanilla GNN and II, (we implement the methods with one of the SOTA graph IL methods CIGA (Chen et al. 2022 on Spurious- Motif under various backbones like GCN, GIN, and XGNN settings. Moreover, as the spurious bias increases, the perfor- mance of DivIL remains more stable, while the baselines and IL models tend to fail, like in the SP dataset DivIL improves performance from 65.45% to 70.03% and in the SPMotif-0.90 dataset from 59.64% to 66.85%.
2022
identifies an invariant subgraph Ge to help predict their labels
Sensitivity different graph data augmentations. Figure 6 G(a illustrates that different random augmentation methods, such as subgraph extraction, edge removing, and node dropping (You et al. 2020). yield similar performance in addressing graph out-of-distribution (OOD) challenges, echoing observations found in recent studics (Guo et al.) 2023). Additionally, in Figure b the comparison between edge remoing and node dropping methods in SPMotil under varying shift biases reveals a slight advantage of edge removing over node dropping at 6 0.33 and b 0.6. However, b = 0.9, node dropping surpasses edge removal, although the difference remains modest. This observation supports our insight that the data augmentation strategies employed may not significantly influence the graph OOD problems.
5.2 Experiments on CMNIST
We evaluate DivIL on the synthetic datasets ColoredMNIST following Arjovsky et al. 2019)) We compare DivIL with ERM, and various IL methods, including causal methods that focus on learning invariance (IRM, VREx) and gradient matching techniques (Fishr). As previously done in Fishr, we maintain all IL method implementations identical to the IRM implementation, notably the same MLP and hyperparameters, and just add the DivIL penalty to the loss. We use two-stage scheduling selected in IRM for the regularization strength X, which is low until epoch 190 and then jumps to a large value. Due to the varying degrees of over-invariance introduced by different IT. methods, we performed a simple search over / values of {0.01, 0.05, 0.1, 0.2), and mask probabilities p of (0.3, 0.5, 0.7}.
Table reports the accuracy averaged over 5 runs with standard deviation. Adding DivIL can achieve the trade-off between train and test accuracies, notably in test set. It reaches 69. 20% in the colored test set
10

---

Published in Transactions on Machine Learning Research (02/2025)
and 70.43% when digits are grayscale. In addition, DivIL improves the performance in all IL methods, verifying our understanding of the issue of over-invariance. Figure Fa) displays the results of DivIL using different invariant losses across various mask percentages p, demonstrating the robustness of DivIL to the hyperparameter p with minimal variance in accuracy across different values. Figure ((b) illustrates that increasing the weight 8 of the Luct lerm leads to improved performance IRMv1, VREx, and Fishr. This supports our insight that over-invariance issues exist in current incremental learning (II.) methods. By introducing diversity penalties Luct, we can mitigate this issue and enhance out-of-distribution (OOD) performance.
Table 3: Performance on CMNIST dataset. All results are reported with mean + std over 5 runs. DivIL improves the accuracy on the lest and the gray set, demonstrating its effectiveness in alleviating over- invariance issues in the visual domain.
train sct
test set
gray set
ERM
86.47 = 0.16
14.18 + 0.68
70.74 + 0.77
TRM
71.47 = 1.18
65.30 + 1.09
66.66
2.33
+ DivIL
70.93 = 0.29
66.40 + 1.39
66.97
1.85
VREx
72.14 = 1.49
67.05 + 0.84
68.96
2.03
DivIL
72.67 0.93
67.50
1.45
69.30
1.91
Fishr
71.34 = 1.27
69.18
0.80
70.35 1.14
DivIL
71.27 = 1.36
69.25 0.81
70.43 1.02
5.3 Experiments on Natural Language Inference
Table 4: Performance SNLI(inddomain) MNLI matched and mismatched (out-domain) dataset. DivIL effectively enhances out-of-distribution generalization while preserving in-domain test performance. Addi- tionally, it provides an ablation study demonstrating the necessity of both Luel and C components.
SNLI
MNLI
matched
mismatchod
ERM
77.7
54.4
54.7
IRM
77.7
55.0
55.5
DivIL
79.3
55.5
59.2
DivIL- Luct
77.6
54.5
56.6
DivIL- La
78.8
54.5
57.1
Inspired by Qin al. (2024). we also demonstrated the effectiveness of our method in NLP through a Natural Language Inference (NLI) (Dagan al.) (013)) task, which assesses the logical relationship between sentences: entailment, contradiction, neutrality. Our model was trained on subset of the SNLI (Bowman et al. 2015 training set and evaluated on selected cases from the SNLI validation set, as well as the match and mismatch subsets of the MNLI (Williams et al. 2017) validation set. While SNLI represents an in- distribution (ID) scenario, MNLI helps assess the generalization to out-of-distribution (OOD) data. The results show that our method performs well in both IID and OOD scenarios, validating its effectiveness.
In our experiment, we employed a pretrained GPT-2 model with randomly initialized classification head. We set the maximum loken length to 64 and trained the model for epochs using the AdamW optimizer. The learning rate was configured at 2e-5, with weight decay of 0.01 and a linear learning rate scheduler. We used training batch size of 32. To optimize our model, we implemented supervised contrastive loss Lu following Zhang et al. 2022a and explored various combinations of weights for À and B, choosing values from the set {0, 0, 0.3, 0.5, 0.7, 1.0}. Additionally, we fixed the projection mask probability at 0.7 and reported the results for the best perfforrfoning configuration.
Table shows the results of DivIL on the NLI task, where DivIL outperforms both IRM and ERM approaches wealdworld natural language datasets. Furthermore, ablation study reveals that removing either La
11

---

Published in Transactions on Machine Learning Research (02/2025)
Luel leads to decrease in OOD performance. However, the performance remains better than that of IRM or ERM, indicating that DivIL achieves a trade-off between strong regularization and feature diversity. Figure (c) illustrates the performance of DivIL with varying weights, denoted as 3, for the loss function Luct across the SNLI, MNLI-match, and MNLI-mismatch datasets. Unlike the findings in CMNIST, increasing B does not necessarily improve out-of-dist ributtin (OOD) performance. It's essential to choose an appropriate
weight, possibly due to the unique structure of natural language.
IRM
VREx
Fishr
IRM
VREX
Fishr
SNLI -
MNLI
MNLI
0.70
0.70
1.00
0.80
Acc
0.65
0.65
0.60
0.40
0.60
0.60
0.20
p=0.3
p=0.5
p=0.7
0.05
0.10
0.20
0.3
0.7
(a) CMNIST with projection mask
(b) CMNIST with Luct weight B (p=o.7)
(c) NLT with Luct weight B
(B 0.01)
(2 = 0.7)
Figure 7: Ablation study of DivIL on CMNIST and NLI datasets. (a) illustrates the performance of DivIL various mask percent p across different implementation of La. (b) and (c) illustrates the performance of DivIL on different weight of UCL 3 in CMNIST and NLI, respectively.
6 Related work
Out-of-Distribution (OOD) Generalization. Existing strategies to tackle OOD generalization can be broadly classified into three approaches (Yang et al.] (024).. Representation learning focuses on developing robust feature representations that generalize across various distributions, including unsupervised domain generalization (Mahajan et al. 2021 Zhang et al.] 2022b Chen et al. 2020b) and disentangled representations
(Bengio et al. 2013 Higgins al. 2017 Kim & Mnih 2018 Yang et al. 2021). Model-based approaches, such
as Invariant Learning Roseeneeld et al. 2020 Ganin & Lempitsky 2015 Li et al. 2018 Creager et al. 2021a)
and causal learning Peters et al. 2016a Pfister et al.) 2019)) aim to capture invariant relationships across
environments to enhance robustness against distributional shifts. Finally, optimization-based techniques seek to ensure strong worst-case performance under potential distributional changes
(Delage & Ye 2010
Namkoong & Duchi 2016 Duchi & Namkoong 2021 Duchi et al., 2023 Zhou et al.) 2022). safeguarding
models from uncertainties in the data.
Discussion on Different Invariant Losses Many invariant learning methods focus on learning stable features across environments by incorporating penalties into the loss function to ensure the consistent change rate (Arjovsky et al., 2019 Krueger al. 2021 Rame et al. 2022 Zhang et al. 2022a).. One series of methods relies explicit environment labels. For example, IRMv1 (Arjovsky et al. 2019h implements the theory of the invariance principle in practice by assuming the classifier as the constant and employing a gradient-based based penalty that requires the sum of the gradients of the model to remain small. The loss function Lirmel is defined as follows: Cirmel = Eoce, (o the loss across different environments defined Lorex = V or (Cprea Carea (f), Chread()) where k is the environment numbers in the training dataset.
VREx (Krueger et al. 2021) takes the variance of
There are also plentiful studies in invariant lcarning without environment labels. Creager et al. 2021b proposed a minmax formulation to infer the enviromment labels. Liu et al. (2021b) proposed hell-booliigg framework based on the estimated invariant and variant features. Liu et al. (2021a); Zhang et al. (022a)) proposed to infer labels based the prodictions of an ERM trained model. Other methods adopt the loss of supervised contrastive learning as La. like CNC Zhang al. (2022a) and CIGA Chen et al. (2022), using different heuristic strategies to choose the positive and negative samples. For example, the invariant, penalty
12

---

Published in Transactions on Machine Learning Research (02/2025)
of CIGA encodes the subgraph Ge and take the learned feature 24 from the same label, treated as the positive pair, and 2j is the from different labels (negative pair).
2020ap and MoCo et al. 2020 demonstrate how con- trastive objectives can improve feature robustness and help models generalize better to unseen environments. Additionally, (Wen & Li 2021) and (Ji et al. extract semantically meaningful features from data. Furthermore, (Xue et al. 2023b) conducts systematic experiments contrastive learning, revealing the effectiveness of combining supervised and unsupervised contrastive learning for feature learning. By clustering similar features and pushing apart dissimilar ones,
Contrastive Learning. ( Chen et al.
2023
demonstrate that contrastive learning can effectively
contrastive learning prevents feature collapse, even under strong regularization (Chen et al. 2022 Zhang
et al. 2022a
7
Conclusion
We shed light on the limitations of invariant constraints in addressing out-of-distribution generalization. While these constraints can mitigate spurious correlations, our research revealed the risk of over-invariance, potentially leading to the loss crucial details in invariant features and a subsequent decline in generalization performance. To tackle these challenges, we introduced Diverse Invariant Learning (DivIL), leveraging contrastive learning and random feature masking introduce uncertainty and diversity. Our comprchensive experiments spanning various modalities and models, underscored the efficacy of our proposed method in enhancing model performance.
References
Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, Jean-Christophe Gagnon-Audet, Yoshua Bengio, Toan- nis Mitliagkas, and Irina Rish. Invariance principle meets information bottleneck for out-of-distribution generalization. In Advances in Newral Information Processing Systems, 2021.
Martin Arjovsky, Lóon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprinl ar xrXio:1907002890 2019.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspec- tives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798 1828, 2013.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annolated corpus for learning natural language inference. In Conference on Empirical Methods in Natural Language Processing, 2015. CRL https: :/aps:///pp... http://pp...eaanicccccooarr.r.or wrw.:a/ps/oooooooorro.C. 1460045520
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597 1607. PMLR, 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXio preprint ar arXin:2003.0420 2020b.
Yongqiang Chen, Yonggang Zhang, Yatao Bian, Han Yang, Kaili Ma, Binghui Xie, Tongliang Liu, Bo Han, and James Cheng- Learning cansally invariant representations for out-of-distributioo generalization on graphs. In Advances in Neural Information Processing Systems, 2022.
Yonggiang Chen, Yatao Bian, Bo Han, and James Cheng. How interpretable are interpretable graph neural networks?, 2024.
Elliot Creager, Jörn-Henrik Jacobsen, and Richard Zemel. Environment inference for invariant learning, In International Conference on Machine Learning, pp. 2189-2200. PMLR, 2021a.
Elliot Creager, förn-Henrik Jacobsen, and Richard S. Zemel. Environment inference for invariant learning. Tn International Conference on Machine Learning, pp. 2189 2189-2200, 2021b.
13

---

Published in Transactions on Machine Learning Research (02/2025)
Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. Book reviews: Recognizing textual entailment: Models and applications by ido dagan, dan roth, mark sammons and fabio massimo zanzotto. In International Conference on Computational Logic, 2013. URL https://api. semantcsccoooalr UtL.ppp://api..eeattccccoooo
.rgg
CorpusID:61205942
Erick Delage and Yinyu Ye. Distributionally robust optimization under moment uncertainty with application data-driven problems. Operations reseorch, 58(3):595 612, 2010.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidi- rectional transformers for language understanding. In Conference of the North American Chapter of the Association for Compulational Lònguislies: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186 2019.
Kaize Ding, The Xu, Hanghang Tong, and Huan Liu. Data augmentation for deep graph learning: A survey. arXiv preprint arXi::2202.082: 2022.
John Duchi, Tatsunori Hashimoto, and Hongscok Namkoong. Distributionally robust losses for latent co- variale mixtures. Operalions Research, (2)):664 664, 2023.
John C Duchi and Hongseok Namkoong. Learning models with uniform performance via distributionally robust optimization. The Annals of Statistics, 49(3):1378-1406, 2021.
Jianqing Fan, Cong Fang, Yihong Gu, and T. Zhang. Environment invariant linear least squares. ArXiu, 0bs//2333.00092, 2023. URL http:////pp...mmmmmmccccccoorr wttp:/roo//oorss:::05333000
Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Interna- tional conference machine learning, pp. 1180 1189. PMLR, 2015.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In Empirical Methods in Natural Language Processing (EMNLP), 2021.
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew E. Peters, Michael Schmitz, and Luke Zettlemoyer. Allennlp: deep semantic natural language processing platform. arXiv preprint, arXiv:1803.07640, 2018.
Shurui Gui, Xiner Li, Limei Wang, and Shuiwang Ji. GOOD: A graph out-of-distribution benchmark. In Thirty-sizth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
Tshaan Gulrajani and David Topez-Paz. In search of lost domain generalization. In International Conference on Learning Representations, 2021.
Xiaojun Guo, Yifei Wang, Zeming Wei, and Yisen Wang. Architecture matters: Uncovering implicit mech- anisms in graph contrastive learning. In Thirly-seventh Conference on Neural Information Processing Systems, 2023.
Kaiming He, Haoqi Fan, Yuxin WI, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognilion, pp. 9729 9000,9738--88 2020.
Trina Higgins, Loie Matthey, Arka Pal, Christopher P Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. ICLR (Poster), 3, 2017.
Wenlong Ji, Zhun Deng, Ryumei Nakada, James Zou, and Linjun Zhang. The power of contrast for feature learning: A theoretical analysis. Journal of Machine Learning Research, 24(330):1 2023.
14

---

Published in Transactions on Machine Learning Research (02/2025)
Yuanfeng Ji, Lu Zhang, Jiaxiang Wu, Bingzhe Wu, Long-Kai Huang, Tingyang Xu, Yu Rong, Lanqing Li, Jie Ren, Ding Xue, Houtim Lai, Shaoyong Xu, Jing Feng, Wei Liu, Ping Luo, Shuigeng Zhou, Junzhou Huang, Peilin Zhao, and Yatao Bian. DrugOOD: Dut-of-Distribution (OOD) Dataset Curator and Benchmark for Al-aided Drug Discovery Focus on Affinity Prediction Problems with Noise Annotations. arXiv preprinl, arXiv:2201.0900 009677, 2022.
Li Jing. Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In International Conference on Learning Representations, 2021.
Pritish Kamath, Akilesh Tangella, Danica Sutherland, and Nathan Srebro. Does invariant risk minimization capture invariance? In International Conference on Artificial Intelligence and Statistics, pp. 4069-4077,
2021a.
Pritish Kamath, Akilesh Tangella, Danica Sutherland, and Nathan Srebro. Does invariant risk minimization capture invariance? In International Conference on Artificial Intelligence and Statistics, pp. 4069 4077. PMLR, 2021b.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International conference on machine learning, pp. 2649-2008 2658. PMLR, 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Confer- ence on Learning Representations, 2015.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017.
Boris Knyazev, Graham W. Taylor, and Mohamed R. Amer. Understanding atlention and generalization in graph neural networks. In Advances in Neural Information Processing Systems, pp. 4204-4214 2019.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubra- mani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara M. Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A benchmark of in-the-wild distribution shifts. In International Conference Machine Learning, pp. 5637 5664, 2021.
David Krueger, Ethan Caballero, Jörn-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Rémi Le Priol, and Aaron C. Courville. Out-of-distrrbutiin generalization via risk extrapolation (rex). In International Conference on Machine Learning, pp. 5815 11265526, 2021.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5400 5409, 2018.
Haoyang Li, Ziwei Zhang, Xin Wang, and Wenwu Zhu. Learning invariant graph representations for out-of- distribution generalization. In Advances in Neural Information Processing Systems, 2022.
Yong Lin, Shengyu Zhu, Lu Tan, and Peng Cui. ZLN: When and how to learn invariance without environment partition? In Advances in Neural Information Processing Systems, 2022.
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghumathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pp. 6781-6792, 2021a.
Jiashuo Liu, Zheynan Hu, Peng Cui, Bo Li, and Zheyan Shen. Heterogeneous risk minimization. In Inter-
national Conference on Machine Learning, volume 139, pp. 6804 6814, 2021b.
Jiashuo Liu, Zheyan Shen, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of- distribution generalization: A survey, 2023.
15

---

Dublin, URL Published in Transactions on Machine Learning Research (02/2025)
Divyat Mahajan, Shruti Tople, and Amit Sharma. Domain generalization using causal matching. In Inter- national conference on machine learning, pp- 1313-7324. PMTRR 2021.
David Mendez, Anna Gaulton, A. Patrícia Bento, Jon Chambers, Marleen De Veij, Eloy Felix, María Magariños, Juan F. Mosquera, Prudence Mutowo-Meullenet, Michal Nowotka, María Morrillo-Maraiión,, Fiona M. I. Hunter, Laura Junco, Grace Mugumbate, Milagros Rodriguez-López, Francis Atkinson, Nicolas Bosc; Chris J. Radoux, Aldo Segura-Cabrers Anne Hersey, and Andrew R. Leach. Chembl: towards direct deposition of bioassay data. Nucleic Acids Research, DeDaaaaaas-lsssee:DDDa D993099900,000000 2019.
Hongseok Namkoong and John C Duchi. Stochastic gradient methods for distributionally robust optimization with f-divergences. Advances in neural information processing systems, 29, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprinl urXiv: 03748, 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, perf-peerormance deep learning library. In Advonces in Newral Information Processing Systems, pp. 8024-8035, 2019.
Judea Pearl. Causality: Models, Reasoning and Inference. Cambridge University Press, USA, 2nd edition, 2009.
Jonas Peters, Peter Bühhmann, and Nicolai Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Stotistical Society Series B: Statistical Method- ology, (15):947711 2016a.
Jonas Peters, Peter Büihlmann, and Nicolai Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 78(5):947- 1012, 2016b.
Jonas Peters, Peter Büihlmann, and Nicolai Meinshausen. Causal Inference by using Invariant Predic- tion: Identification and Confidence Intervals. Journal of the Royal Statistical Society Series B: Sta- tistical Methodology, 7875)1947-1 10 2016c. ISSN 1369-7412. doi: 10.1111/rssb.2166.. URL
10. https://doi..o htt:://ooooro/////// http:///i..r//000111//////////// 10t11100011111rbbb...
12167
Niklas Pfister, Peter Bühlmann, and Jonas Peters. Invariant causal prediction for sequential data. Journal of the American Statistical Association, 114(527):1264-125 1276, 2019.
Libo Qin, Qiguang Chen, Tianbao Xie, Qixin Li, Jian- Maan-Guagg Lou, Wanxiang Che, and Min- Yen Kan. GL- CLcF: global-local contrastive learning framework for cross-lingual spoken language understanding. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. Ireland, May 2022. Association for Computational Linguistics. doi: https://aclanthollee 2ww.caannhhooogoorg22222222 acl wacc-long.......oo 191
Libo Qin, Qiguang Chen, Xiachong Feng, Yang Wu, Yongheng Zhang, Yinghui Li, Min Li, Wanxiang Che, and Philip S Yu. Large language models meet ulp: A survey. arXin preprint arXiv::405.. 02. 2024.
Alexandre Rame, Corentin Dancette, and Matthieu Cord. Fishr: Invariant. gradient variances for out-of- distribution generalization. In International Conference Machine Learning, 2022.
Elan Rosenfeld, Pradeep Kumar Ravikumar, and Andrej Risteski. The risks of invariant risk minimization. In International Conference on Learning Representations, 2020.
Shiori Sagawa*, * Pang Wei Koh*, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2020.
16

---

Published in Transactions on Machine Learning Research (02/2025)
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Iilia Polosukhin. Attention is all you need. In Neural Information Processing Systems, 2017.
URL http:::/api.eeenntiiiieeoo sew.nticsccchallrr :tt.:/acccocacooooooooooooiaaaaa 13756489
Qizhou Wang, Yong Lin, Yongqiang Chen, Ludwig Schmidt, Bo Han, and Tong Zhang. A sober look at the robustness of clips to spurious features, 2024.
Zixin Wen and Yuanzhi Li. Toward understanding the feature learning process of self-supervised contrastive learning. In International Conference on Machine Learnêng, pp. 11112-11122. PMLR, 2021.
Adina Williams, Nikita Nangia, and Samuel R. Bowmnan. A broad-coverage challenge corpus for sentence understanding through inference. In North American Chapter of the Association for Computational Lin- guistics, 2017. URL htttps://pp.. semanticscholar wwwphhollrrrrr/a/Cooe Pttg/oorpuIDD:3332800
Yingxin Wu, Xiang Wang, An Zhang, Xiangnan He, and Tat-Seng Chua. Discovering invariant rationales for graph neural networks. In International Conference on Learning Representations, 2022.
Binghui Xie, Yonggiang Chen, Jiaqi Wang, Kaiwen Zhou, Bo Han, Wei Meng, and James Cheng. Enhancing evolving domain generalization through dynamic latent representations. 2024.
Jiaxing Xu, Yongqiang Chen, Xia Dong, Mengcheng Lan, Tiancheng Huang, Qingtian Bian, James Cheng, and Yiping Ke. Brainood:
generalizable brain network analysis, 2025.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2019.
Yihao Xue; Siddharth Joshi, Eric Gan, Pin-Yu Chen, and Baharan Mirzasolciman, Which features are learnt by contrastive learning? on the role of simplicity bias class collapse and feature suppression. arXiv preprint XrXe::2305. 16536, 2023a.
Yihao Xue, Siddharth Joshi, Eric Gan, Pin- Yn Chen, and Baharan Mirzasoleiman. Which features are learnt by contrastive learning? on the role of simplicity bias in class collapse and feature suppression. Tn International Conference on Machine Learning, pp. 38938 38970. PMLR, 2023b.
Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized aut-of-distribution detection: A survey. International Journal of Computer Vision, pp. 1-28, 2024.
Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. Causalvae: Disentangled representation learning via neural structural causal models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9593-9602, 2021.
Tianjun Yao, Yongqiang Chen, Zhenhao Chen, Kai Hu, Zhiqiang Shen, and Kun Zhang. Empowering graph invariance learning with deep spurious infomax, 2024.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph rastivve learning with augmentations. Advances in neural information processing systems, 33:5812 5823, 2020.
Hao Yuan, Haiyang Yu, Shurui Gui, and Shuiwang Ji. Explainability in graph neural networks: A taxonomic survey. IEFE Transactions on Pattern Analysis and Machine Intelligence, 45:5782-577 5799, 2020. URL
http:://api. wtw..iisssmaaiiccoo....rrrr wwwpsoollrrrrrr//// wwwg//orggs/oo.o. :2:922224222222
Michael Zhang, Nimit Sharad Sohoni, Hongyang R. Zhang, Chelsea Finn, and Christopher Ré. Correct- n-contrast: A contrastive approach for improving robustness to spurious correlations. arXiv preprint, arXiv:2203.01517, 2022a.
Xingxuan Zhang, Linjun Zhou, Renzhe Xu, Peng Cui, Zheyan Shen, and Haoxin Liu. Towards unsupervised domain genoralization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4910 4920, 2022b.
Xiao Zhou, Yong Lin, Renjie Pi, Weizhong Zhang, Renzhe Xu, Peng Cui, and Tong Zhang. Model agnostic sample reweighting for dutto--disttibutionn learning. In International Conference on Machine Learning, pp. 277021. PML.R 2022.
17

---

Published in Transactions on Machine Learning Research (02/2025)
A Datasets
A.1 CMNIST
Colored MNIST is a binary digit classification dataset introduced in IRM (Arjovsky et al. (019)). Compared to the traditional MNIST, has 2 main differences. First, 0-4 and 5-9 digits are each collapsed into a single class, with a 25% chance of label flipping. Second, digits are either colored red or green, with a strong correlation between label and color in training. However, this correlation is reversed at test time. Specifically, in training, the model has access to two domains E (90%, 80%]: in the first. domain, green digits have 90% chance of being in 5-9; in the second, this chance goes down to 80%. In test, green digits have 10% chance of being in 5-9. Due to this modification in correlation, a model should idcally ignore the color information and only rely on the digits' shape: this would obtain a 75% test accuracy.
A.2 NLI
The natural language inference (NLI) task involves determining the logical relationship between pairs of sentences, typically categorized as entailment, contradiction, or neutral. In this task, model is presented with a premise sentence and a hypothesis sentence, and it must infer how the hypothesis relates to the premise as seen Table Table 7 NLI crucial in natural language understanding as tests a model's ability to comprehend and reason about language, making it a fundamental benchmark for evaluating the performance of language models and their ability to capture semantic relationships and contextual information within the
text.
We provide more details about the motivation and construction method of the datasets used in our exper- iments. Statistics of the datasets are presented in Table 5) We use about 8,000 examples in the train set from the SNLI Bowmman et al. (0015). from the Image Captions from the Flickr30k Corpus domains. We selected 1,000 examples from the validation-mattceed set of the MNL.I dataset (Williams al. 2017); sourced from the Fiction, Government, Slate, Telephone, and Travel domains. Additionally, we chose another 1,000 examples from the malidation-matched set of the MNLI dataset, taken from the 9/11, Face-to-Face, Letters, OUP, and Verbatim domains, to form out-of-domain (OOD) test set. Examples of SNLI and MNLI are shown in Table G and Table 7
Table 5: Statistics of our constructed OOD NLI Dataset.
Split
Genre
Examples
Partition
Data Domain
Metrics
IMAGE CAPTIONS THE FLICKи30K
Train set
SNLI
7992
Irain
ACC
CORPUS
IMAGE CAPTIONS THE FLICKR30K
SNLI
991
validation
ACC
CORPUS
Test set
FICTION, GOVERNMENT, SLATE, TELE-
1000
ioodatioon-matcee
ACC
MNLI
PHONE, TRAVEL
9/11, FACE-TO-FACE, LETTERS, OUP, ACC
1000
cnlidation-mismattele
VERBATIM
A.3 Graph
We provide more details about the motivation and construction method of the datasets used in our experi- ments. Statistics of the datasets are presented in Table g
Spurious-Motif We construct 3-class synthetic datasots based on BAMotif following Wu et al. (2022) where the model needs to tell which one of three motifs (House, Cycle, Crane) the graph contains. For each dataset, we generate 3,000 graphs for each class in the training set, and 1,000 graphs for each class in the validation set and testing set, respectively. We introduce the bias based on FIIF, where the motif and one of the three base graphs (Tree, Ladder, Wheel) are artificially (spuriously) correlated with a probability of various biases, and equally correlated with the other two. Specifically, given predefined bias b, the
18

---

Published in Transactions on Machine Learning Research (02/2025)
Table 6: NLI dataset samples from SNLI (IID).
Premise
Hypothesis
Label
Two men holding their mouths open.
Two men with mouths agape.
ENTAILMENT
Trying very hard not blend any yellow paint into the white.
Someone painting house. man on a small 4 wheeled vehicle flying through the air.
NEUTRAL
The man is on a bike. Two people in park walking Two power walkers walking beside one another in race.
CONTRADICTION
NEUTRAL
Women standing podium with crowd and building in the background.
woman stands at podium
ENTAILMENT
Table 7: MNLI dataset samples from the malichtion-matched (above) and validation-mismathhed (below) subsets (OOD).
Premise
Hypothesis
Label
pretty good newspaper uh-huh
I think this decent newspaper.
ENTAILMENT
Massive tidal waves swept over Crete, and other parts of the Mediterrancan, smashing buildings and drown- ing many thousands people.
The waves came with no warning in- habitants.
NEUTRAL.
For such governmentwide review, an entrance confer- ence generally held with applicable central agencies, such as the Office of Management and Budget (OMB) the Office of Personncl Management.
An entrance conference is held with special- ized agencies.
CONTRADICTION
As Figure 6.6 shows, the safety stock needed achieve a given customer service level is proportional to standard deviation of the demand forecast.
Figure 6.6 shows the safety stock needed to achieve given customer service level.
ENTAILMENT
Some of Bin Ladin's close comrades were more peers than subordinates.
There were three people who could be con- sidered peers of Bin Ladin.
NEUTRAL
Everything, except knowing that you are helping to protect Earth's natural re- sources.
Nothing except knowing that you are helping to pro- tect the Earth's precious natural resources.
CONTRADICTION
probability of specific motif (e.g., House) and specific base graph (Tree) will co-occur is b while for the others is (1 b)/2 (e-g., House-Tadder, House-Wheel) We use random node features in order to study the influences of structure level shifts.
CMNIST-sp To study the effects of PIIF shifts, select the ColoredMNIST dataset created in Ar- jovsky et al. (2019). We convert the ColoredMNIST into graphs using the superpixel algorithm introduced by Knyazev et al. 2019
Graph-SST2 Inspired by the data splits generation for studying distribution shifts on graph sizes, split the data curated from sentiment graph data [84), that converts sentiment sentence classification datascts Graph-SST2 (Yuan et al. 2020) into graphs, where node features are generated using BERT (Devlin et al. 2019) and the edges are parsed by Gardner et al. (2018). Our splits are created according to the averaged degrees of each graph. Specifically, we assign the graphs as follows: Those that have smaller or equal 50-th percentile while smaller than 80-th percentile are assigned to the validation set, and the left are assigned test set.
19

---

Published in Transactions on Machine Learning Research (02/2025)
DrugOOD datasets To evaluate the OOD performance in realistic scenarios with realistic distribution shifts, we also include three datasets from DrugOOD benchmark (Ji et al. 2022 DrugOOD is systematic OOD benchmark for Al-aided drug discovery, focusing on the task of drug target binding affinity prediction for both macromolecule (protcin target) and smallmolccule (drug compound). The molecule data and the nolations are curated from realistic ChEMBL database Mendez al. 2019). Complicated distribution shifts can happen on different assays, scaffolds and molecule sizes. In particular, we select DrugOOD--lap-core. ec50-assay, DruggOOD-bap-oor--ec0------ and DragOOD-Ibap-core core-kiizzzs,, from the task of Ligand Based Affinity Prediction which uses ic50 measurement type and contains core level annotation noises. We directly use the data files provided by Ji et al. (2022).
DrugOOO--bap--or---iie
Table 8: Graph dataset details. The number of nodes and edges are respectively taking average among all graphs.
Dataset
Training
Validation
Testing
Classes
Nodes
Edges
Metrics
SPMOTIF
9,000
3,000
3,000
3
44.96
65.67
ACC
CMNIST-SP
40,000
5,000
15,000
2
56.90
373.85
ACC
Graph-SST2
24,881
7,004
12,893
2
10.20
18.40
ACC
EC50-Assay
4,978
2,761
2,725
2
40.89
87.18
ROC-AUC
EC50-Scaffold
2,743
2,723
2,762
2
35.54
75.56
ROC-AUC
EC50-Size
5,189
2,495
2,505
2
35.12
75.30
ROC-AUC
Ki-Assay
8,490
4,741
4,720
2
32.66
71.38
ROC-AUC
Ki-Scaffold
5,389
4,805
4,463
2
29.96
65.11
ROC-AUC
Ki-Size
8,605
4,486
4,558
2
30.35
66.49
ROC-AUC
B Implement Details
During the experiments, we do not tune the hyperparametens exhaustively while following the common recipes for optimizing the models. Details follows. We will publish our code when the paper is accepted.
B.1 CMNIST Implements
the network is layers MLP with ReLu activation, optimized with Adamn (Kingma& & Ba (2015)). IRM selected the following hyperaarameters by random search over 50 trials: hidden dimension of 390, 12 regularizer weight of 0.00110794568,, learning rate of 0.000489853666644469944 penalty anneal iters (or warmup iter) of 190, penalty weight (A) 91257.186131 159118 501 epochs and batch size 25,000 (half of the dataset sizc). For the implementation of the invariant losscs(IRM, VREx and Fishr), we strictly keep the same hyperparameters values in our implementation and the code almost unchanged from httpp:///gl //gptbu com hoom//lexxam//iisht To account for the varying degrees of ever-ivarriance intro- duced by different IL methods, we performed straightforward scarch over 3 values of f0.01, 0.05, 0.1, 0.2) and projection mask probabilities of (0.3, 0.5, 0.7}, while keeping the random augmentation mask probability fixed at 0.2.
In the experimental setup in Section 5.2
B.2 NLI Implements
We employed pretrained GPT-2 model with randomly initialized classification head. We set the maximum token length to 64 and trained the model for 5 epochs using the AdamW optimizer. The lcarning rate was configured at 2e-5, with weight decay of 0.01 and a linear learning rate scheduler. We used training batch size of 32. To optimize our model, we explored various combinations of the invariant loss and unsupervised loss weights for À and 3, choosing the best from the {0, 0.1, 0.3, 0.5, 0.7, 1.0} according the validation set. Additionally, we fixed the projection mask probability at 0.7 and reported the results for the best-performing configuration.
20

---

Published in Transactions on Machine Learning Research (02/2025)
B.3 Graph Implements
For fair comparison, DivIL uses the same GNN architecture for GNN encoders the baseline methods. We use the GCN backbone and the sum pooling in Table 1 By default, we fix the temperature to be in the unsupervised contrastive loss, and merely search the penalty weight of the contrastive loss from {0.1, (0.1,0.2,0) 0.5, 1.2) according to the validation performances. We select the best of the random mask per- centage p from the {0.2, 0.3, 0.5, 0.7] according to the validation performances. For the implementation of graph data augmentation, we use the tool from You et al. (2020). We select the best percentage Pz of node dropping, edge removing, and subgraph extraction from the {0.05, 0.1, 0.15, 0.2) according to the validation performances to create the positive pair and keep P1 0 representing the sample itself. For the implemen- tation of our basclines, we take the code almost unchanged from https: thub. www. con//Phase/CIO,)
C More Ablation Study
C.1 Abitions on Random Masking
We introduced a random masking mechanism to overcome the issue of over-invariance, which differs from the traditional dropout method. Unlike dropout, which randomly drops out dimensions independently, we obscrved that fcatures tend to diversify in the carlier dimensions and become more invariant in the later dimensions. Therefore, we promote the diversity of characteristics by setting the first p dimensions of the contrastive learning feature space to O. This approach not only alleviates the over-invariance problem but also enhances flexibility in specific dimensions of the feature space.
To validate the effectiveness of this mechanism, we conducted ablation experiments on the CMNIST dataset. The experimental results showed that the model using the random masking method outperforms the model using traditional dropout in terms of out-of-distribution generalization. As shown in Table g the random masking mechanism improves the robustness of the model, particularly the face of complex data distribu- tions, effectively mitigating vvr-invariance and improving the overall performance of the model.
Table 9: Comparison of Random Masking and Dropout on the CMNIST Datasct.
Method
Train
Test
IRM
71.47 + 1.18
65.30 + 1.09
+ DivIL (dropout)
70.65 + 0.54
66.19 1.78
+ DivIL (random masking)
70.93 + 0.29
66.40
1.39
VERx
72.14 + 1.49
67.05 0.84
+ DivIL (dropout)
72.54 + 1.72
66.73 1.34
DivIL (random masking)
72.67 + 0.93
67.50 + 1.45
Fishr
71.34 + 1.27
69.18
0.80
DivIL (dropout)
71.36 + 1.41
69.20 - 0.77
DivIL (random masking)
71.27 + 1.36
69.25 + 0.81
C.2 Interpretation Visualization of Over-invariance on Graph
To demonstrate the over-invariance issue on the real datasets, we use the interpretable GNN architecture (XGNN) to extract Ge, it brings an additional demonstration that over-invariance risk occurs in the real- graph datasets, which may facilitate human understanding in practice.
We provide some visualizations of invariant subgraphs learned by Chen et al. 2002 First, we illustrate the examples in SPMotif under the biases of 0.6 and 0.9 shown in Figure and Figure We use pink to color the ground truth nodes in Ge, and denote the relative attention strength with edge color intensities. We also
21

---

Published in Transactions on Machine Learning Research (02/2025)
provide some interpretation visualization examples in DrugOOD datasets shown in Figure 10 to Figure 11 We use the edge color intensities to denote the attention of models that pay to the corresponding edge.
We have two observations based on the above visualization examples. 1) Invariant learning methods tend to extract the subgraph of the ground truth invariant subgraph(Figure over-invariance issue. 2) When confronted with multiple invariant subgraphs (Figure 110, these learning
thus verifying the
methods usually focus on only one of them rather than considering all the invariant subgraphs. In the fields of computer vision and natural language processing, do not visualize them here due to the ambiguous definitions and interpretations of invariance and spurious effects. The above two observations aligned with our synthetics experiments in Figure B mcasured by the strength.
SPMotif: y=0
SPMotif:
SPMotif:
Figure 8: Over-invariance visualizations of examples from SPMotif-Strue under bias= 0.6 CCen et al.
2022
SPMotif:
SPMotif:
SPMotif: y-2
Figure 9: Over-invariance visualizations of examples from SPMotif-Strue under bias= 0.9 CChn et al. 2022
drugood Ibap lc50 assay: y=1
drugood Ibap ic50 assay: y-1
drugood Ibap core_ ic50 assay: y-1
Figure 10: Over-invariance visualizations of activate examples (y = 1) from DrugOODD-Assa
CCen et al.
2022
Software and Hardware
We implement our methods with PyTorch (Paszke et al. 2019: and PyTorch Geometric (eey Lenssen
2019). We experiments Limux Servers installed with 3090 graphics cards and CUDA 10.2.
22

---

Published in Transactions on Machine Learning Research (02/2025)
drugood lbappcoreee_____eeeeeeaeeeeeeeee core ic50 assay: y=0
drugood lbap core, i0 assay: y=
drugaod seggood_bbapcooo core i:50 assay: y=0
Figure 11: Over-invariance visualizations of inactivate examples (y = 0) from DrugOOD-Assay (Chen et al. (2022))
23


Reviews:
[{"review_id": "oWK02JFUE9", "reviewer": "Reviewer_uhmW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Evaluation is comprehensive.\n- Besides images, the methods generalize to data in different domain, such as graph or natural language.\n- The idea of over-invariance is interesting.\n\nWeaknesses:\n- The addition of the contrastive term does not substantially advance the theoretical or practical understanding of invariant learning. There are existing works that already explore this area [1] [2]. While the proposed method uses a different augmentation strategy (similar to dropout), it feels incremental rather than a significant leap forward.\n- While the paper discusses an interesting idea of over-invariance, it relies heavily on synthetic experiments to validate its claims. This does not convincingly demonstrate the real-world impact of over-invariance.\n- Features are modeled as Gaussian distributions with clear separability between invariant and spurious components. Such assumptions may not hold in real-world data, limiting the applicability of the theoretical insights.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates the phenomenon of over-invariance in invariant learning methods, where overly restrictive invariant constraints lead to the loss of discriminative features and hurt the OOD generalization. The authors propose a new method, DivIL, to address this limitation. DivIL employs a contrastive loss, where the positive pair of a sample is obtained by random dropping a subset of it (e.g., masking a subset of pixels to 0 for images). DivIL is tested across 12 datasets spanning computer vision, natural language, and graph learning tasks, and shows improvements over existing invariant learning methods.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Explain in details how the proposed method extends beyond existing work in invariant learning and contrastive learning.\n- Revise the theoretical framework to reduce reliance on overly simplistic assumptions (e.g., Gaussian feature distributions, clearly separable spurious and invariant features). Or discuss how the proposed framework fits real-world data.", "broader_impact_concerns": "No concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MJGCKai8Mw", "reviewer": "Reviewer_wSGc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "S1: The provided code is well documented. The use of numerous README files makes it easy to understand how the code can be used. The appendix further expands on experiment details.\n\nS2: The theoretical definitions laid out in Definitions 3.2 and 3.3 builds off of previous work (Definition 3.1) to provide new insights into the structure of invariant features and the issue of over-invariance.\n\nS3: The extensive experiments benchmark multiple existing methods and the proposed method on multiple datasets across image, natural language, and graph modalities. The proposed method shows similar or increased performance in each evaluation.\n\n\nW1: Section 3.3 does not clearly follow from Section 3.2. Definition 3.3 states that there should not be a *feature subset* in $Z^c$ that changes the expected *class prediction*. Definition 3.5 instead values *embedding norms* under *masked input values*, where the masked values are also valid samples from the distribution. Why not use the decomposed logits, as in the equation above Theorem 3 in Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalization (Ahuja et al. 2021), to directly examine changes in class prediction?\n\nW2: How does Figure 3 relate to Remarks 3.6 and 4.1? Doesn’t Figure 3 show that the strength is nonzero without DivIL? Is averaging across values hiding the zeros? How would adding DivIL to the methods change the results in Figure 3?\n\nW3: Why should the proposed random masking method alleviate the over-invariance problem more than a more common method like dropout? The lack of an ablation further obscures the impact of this choice.\n\nW4: Are the results reported in the tables statistically significant? If so, under what test of significance? The lack of a large margin between some results makes this ambiguous.\n\nW5: Is there a paper that should be cited for the particular unsupervised contrastive learning loss function used? “Representation learning with contrastive predictive coding” (Oord et al., 2019) can be credited for the general formulation, but I’m not familiar with any uses of the negative squared norm in previous literature.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper aims to improve the state-of-the-art of out of distribution detection by improving invariant learning techniques. Invariant learning aims to learn features without spurious correlations, allowing classifiers to generalize to new environments. Previous works potentially suffer from over-invariance, where a subset of invariant features are discarded unnecessarily, degrading generalization. The authors build off of the previous theoretical definition of invariance to rigorously define invariant features, over-invariance, and over-invariant features. The feature norm under data masking is empirically investigated to try to identify over-invariance in practice. DivIL is proposed as a solution to over-invariance, adding contrastive learning and feature masking to existing methods. The experiments benchmark invariant learning methods on multiple datasets across multiple modalities. The proposed method shows similar or increased performance in each evaluation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "C1: Complete the README.md in CIGA_NC_latest.\n\nC2: I think “exceeded human-level performance” in the first sentence of the introduction is overstating previous results.\n\nC3: Definition 3.2 does not appear to be grammatically correct. Suggested rewording: “Suppose selection function $I = \\\\{0, 1\\\\}^k$, such that the subset $Z^c = Z ◦ I$ of the latent feature $Z ∈ R^k$ is an invariant feature of label $Y$ under both train distribution $P_{tr}$ and test distribution $P_{te}$. Therefore $Z^c$ satisfies”\n\nC4: Remark 4.1 cites Equation 6 when it should be 7.", "broader_impact_concerns": "I do not have any concerns about potential broader impacts of this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "powcvQ0V5I", "reviewer": "Reviewer_VXRv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The authors provide a very useful introduction section which helps bring readers up to speed on the subfield. Their figure 1 illustration is well designed and illuminating. Experiments which show how much benefit straightforward pragmatic interventions can offer in a particular setting will always be of interest to at least some TMLR readers. The authors' experiments on graphs show consistent (if not statistically significant) improvement over the strongest baseline method.\n\nPlease see the requested changes section for weaknesses.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this work, the authors offer novel insights into the well-studied out-of-distribution generalization problem from the standpoint of invariant learning (IL), whose core algorithm (IRM) adopts the assumption that *spurious* features (a) harm out-of-distribution generalization and (b) are inconsistent across environments (also sometimes referred to in this literature as groups). Several theoretical and empirical challenges to this idea have entered the literature since its introduction in 2019, to which the authors add another --  over-regularization in the feature space may lead to a loss of important details, an effect which they call *over-invariance*. The authors claim that over-invariance degrades generalization performance, propose Diverse Invariant Learning (DivIL), combining both invariant constraints and unsupervised contrastive learning with randomly masking mechanism to promote richer and more diverse invariance, and offer experiments on 12 benchmarks to validate the efficacy of their method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. You write \"known as the in-distribution assumption (IID assumption) (Liu et al., 2023; Peters et al., 2016c)\". I.I.D. stands for independent and identically distributed, per your citations and general use. Please modify this sentence to reflect this.\n2. Please add summaries of the important findings contained in each table to the captions provided for that table.\n3. Table 1 should be relocated to the experiments section.\n4. Your most robust experiments are on graphs; therefore, those results should be presented first.\n5. Please scope your third claim in the experiments section with greater precision. \"can attain state-of-the-art performance\"; from my reading of the paper, it appears that your method can claim state of the art only on graphs, and that even there, the differences are not statistically significant.\n6. The claim that the authors are the first to discover what they call *over-invariance* is overly broad and should be reworded. Similar phenomena have been extensively studied and documented under various names, such as *underfitting* (Goodfellow, Ian, et al. Deep Learning. MIT Press, 2016).\n7. The description of DivIL mentions at least three new mechanisms introduced that break with prior work; **Random Data Augmentation**, **Unsupervised Contrastive Learning** and **Random Masking**. The authors should provide ablations on the impact of each of all changes in isolation on at least some of their datasets (ideally in the appendix) and reference them in the main paper.", "broader_impact_concerns": "I see no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper was reviewed by three experts who provided valuable comments and insightful suggestions. After constructive discussion, two reviewers agreed to recommend the paper for acceptance to TMLR. Following a thorough review of the revised version and further discussion, I also recommend acceptance, as all reviewers concur that the experiments support the main claim regarding the over-invariance issue in real-world problems.

Moreover, please ensure the discussion with Reviewer uhmW is included in the camera-ready version, specifically: “Explain in detail how the proposed method extends beyond existing work in invariant learning and contrastive learning.”
Decision: Accept (poster)