Continuous Invariance Learning
Authors: LIN Yong
Fan Zhou
Lu Tan
Lintao Ma
Jianmeng Liu
Yansu HE
Yuan Yuan
Yu Liu
James Y. Zhang
Yujiu Yang
Hao Wang
Venue: ICLR 2024 poster (2024)
Keywords: Causality; Domain Generalization; Invariance Learning
Primary area: general machine learning (i.e., none of the above)

Abstract: Invariance learning methods aim to learn invariant features in the hope that they generalize under distributional shift. Although many tasks are naturally characterized by continuous domains, current invariance learning techniques generally assume categorically indexed domains. For example, auto-scaling in cloud computing often needs a CPU utilization prediction model that generalizes across different times (e.g., time of a day and date of a year), where `time' is a continuous domain index. In this paper, we start by theoretically showing that existing invariance learning methods can fail for continuous domain problems. Specifically, the naive solution of splitting continuous domains into discrete ones ignores the underlying relationship among domains, and therefore potentially leads to suboptimal performance. To address this challenge, we then propose Continuous Invariance Learning (CIL), which extracts invariant features across continuously indexed domains. CIL is a novel adversarial procedure which measures and controls the conditional independence between the labels and continuous domain indices given the extracted features. Our theoretical analysis demonstrates that CIL learns features that satisfy the invariant constraint with infinite samples. Empirical results on both synthetic and real-world datasets (including data collected from production systems) show that CIL consistently outperforms strong baselines among all the tasks.

Published as a conference paper at ICLR 2024
CONTINUOUS INVARIANCE LEARNING
Yong Lin 1-, Fan Zhou 2+, Lu Tan', Lintao Ma², Jiameng Liu', Yansu He,, Yuan Yuan 8.7, Yu Liu?, James Zhang?. Yujiu Yang". Hao Wang TTe Hong Kong University of Science and Technology. AAnt Group, 'Rutgers University, *Tsinghua University, 5 'Chinese University of Hong Kong, MITT CSAIL, 'Boston College
ABSTRACT
Invariance learning methods aim to learn invariant features in the hope that they gen- eralize under distributional shifts, Although many tasks are naturally characterized by continuous domains, current invariance learning techniques generally assume categorically indexed domains. For example, auto-scaling in cloud computing often needs a CPU utilization prediction model that generalizes across different times (c.g., time of day and date of year), where 'time' is continuous domain index. In this paper, we start by theoretically showing that existing invariance learning methods can fail for continuous domain problems. Specifically, the naive solution of splitting continuous domains into discrete ones ignores the underlying relation- ship among domains, and therefore potentially leads suboptimal performance. To address this challenge, we propose Continuous Invariance Learning (CIL), which extracts invariant features across continuously indexed domains. CIL is a novel adversarial procedure that measures and controls the conditional independence between the labels and continuous domain indices given the extracted features. Our theoretical analysis demonstrates the superiority of CIL. over existing invariance learning methods. Empirical results on both synthetic and real-world datasets (including data collected from production systems) show that CIL consistently outperforms strong baselines among all the tasks.
INTRODUCTION
Machine learning models have shown impressive success in many applications, including computer vision (He et 2016), natural language processing (Liu et al. 2019). speech recognition (Deng et al.) 2013), etc. These models normally take the independent identically distributed (IID) assumption and assume that the testing samples are drawn from the same distribution as the training samples. However, these assumptions can be easily violated when there distributional shifts in the testing datasets. This is also known as the out-of-distribution (OOD) generalization problem.
Learning invariant features that remain stable under distributional shifts is promising research area to address OOD problems. However, current invariance methods mandate the division of datasets into discrete categorical domains, which inconsistent with many real-world tasks that are naturally continuous. For example, in cloud computing, Auto-scaling (a technique that dynamically adjusts computing resources to better match the varying demands) often needs a CPU utilization prediction model that generalizes across different times (e.g., time of a day and date of year), where 'time" is continuous domain index. Figure [ illustrates the problems with discrete and continuous domains.
Consider invariant features that can be generalized under distributional shift and spurious features that are unstable. Let x denote the input, y denote the output, denote the domain index, and E' denote taking expectation in domain t. Consider the model composed of featurizer & and classifier w (Arjovsky et al. 2019). Invariant Risk Minimization (IRM) proposes to learn (xx) by merely using invariant features, which yields the same conditional distribution P'y (bx))) across all domains t, i..., Pyy 1 x) = P(y t (x:), t), which is equivalent to .L (.x) (Arjovsky et al. (211).. Existing
approximation methods propose to learn featurizer by matching E y <b (z) for different t (See
Section 2.1 2. for details).
'Theee authors contributed equally this work. Corresponding to Lintao Ma(lintao.nl mit @ antgroup.com).

---

Published as a conference paper at ICLR 2024
A case study on REx. However, in the continuous environment setting, there are only very limited samples in each environment. So the finite sample estimations E' [ax)] can deviate significantly
from the expectation EE' [a)]. In Section 2.2. We conduct a theoretical analysis of REx (Krueger et al. 2021), a popular variant of IRM. Our analysis shows that when there is large number of domains and limited sample sizes per domain (i.e.. in the continuous domain tasks), REx fails to identify invariant features with constant probability. This is in contrast to the results in discrete domain tasks, where REx can reliably identify invariant features given sufficient samples in each discrete domain.
The generality of our results. Other popular approximations of IRM also suffer from this lim- itation, such as IRMv1 (Arjovsky et al., 2019). REx (Krueger et al. (02T). IB-IRM (Ahuja
et al., 2021), IRMx (Chen et al., 2022). IGA (Koyama & Yamaguchi] 2020), BIRM (Lin et al.
2022a b IRM Game (Ahuja et al., 2020), Fisher (Rame et al. 2022) and SparseIRM (Zhou et al. 2022). They employ positive losses on each environment to assess whether É' [3/ [=x)
matches with the other environments, However, since the estimations É'' [x)] are inac- curate, these methods fail to identify invariant features. We then empirically verify the inef- fectiveness of these popular invariance learning methods in handling continuous domain prob- lems. One potential naive solution is to split continuous domains into discrete ones. How- ever, this splitting scheme ignores the underlying relationship among domains and therefore can lead to sub-optimal performance, which is further empirically verified in Section 2.2
Our methods. Recall that our task is to learn a feature extractor (.)) that extracts invariant features from i.e. learn & which elicits y 1 (x*). Previous analysis shows that it is infeasible align E' [y/ (x)] among dif- ferent domains in continuous domain tasks. Instead, we propose to align E ( (a=)] for each class y. We can start by training two domain regressors, i.e., h((())) and g (x=), 3). to regress over the continuous domain index t using L1 or L2 loss. Notably, the L1/L2 distance between the predicted and ground-trui domains captures the un- derlying relationship between continuous domain indices and can lead to similar domain index prediction losses between domains on both h and g if ((x)) only extracts invariant features. Specifically, the loss achieved by h, i.c., Elit - h(((x))11 and the loss achieved by g, i.c., Ellt- g (x), y) would be the same if (xx) only extracts invariant features. The whole procedure is then formulated into mini-max framework as shown in Section 3
Discrete
Continuous
uni
Figure 1: Illustration of distributional shifts in discrete and continuous do- mains (Wang et al. 2020).. Existing IRM methods focus on discrete domains, which is inconsistent with many real- world tasks. Our work therefore aims to extend IRM to continuous domains.
In Section 3.1. we emphasize the theoretical advantages of CIL over existing IRM approximation methods in continuous domain tasks. The limited number of samples in each domain makes it challenging to obtain an accurate estimation for E [y] (zx), leading to the ineffectiveness of existing methods. However, CIL does not suffer from this problem because it aims align (E (xr), which can be accurately estimated due to the ample number of samples available in each class (also see Appendix D for the discussion of the relationship between E'Iy [y/ (x)) and E [t (x)). In Section 4 we сarry out experiments on both synthetic and real-world datasets (including data collected from production systems in Alipay and vision datasets from Wilds-time (Yao et al., 2022)). CIL consistently outperforms existing invariance learning baselines in all the tasks. We summarize our contributions as follows:
We identify the problem of learning invariant features under continuous domains and theoretically demonstrate that existing methods that treat domains as categorical indexes can fail with constant probability, which is further verified by empirical study.
We propose Continuous Invariance Learning (CIL) as the first general training framework to address this problem. We also show the theoretical advantages of CIL over existing IRM approximation methods on continuous domain tasks.
We provide empirical results on both synthetic and real-world tasks, including an industrial application in A Alipay and vision datasets in Wilds-time (Yao et al. 2022). showing that CIL consistently achieves significant improvement over wate-of-the-ar art baselines.
2

---

Published as a conference paper at ICLR 2024
2 DIFFICULTY OF EXISTING METHODS IN CONTINUOUS DOMAIN TASKS
2.1 PRELIMINARIES
Notations. Consider the input and output (æ, y) in the space X' x V. Our task is to learn a function So E F:: XX + Y. parameterized by 0. We further denote the domain index as t E T. We have (at y. t) ~ P(x, y)t) x P((). Denote the training dataset as D == {(x+ ... t,. } fi=1- The goal of domain generalization is to learn a function fo on D that can perform well in unseen testing dataset Dic (This is different from continuously indexed domain adaptation (Wang et al. 2020). as setting does not have access to target-domain data.) Since the testing domain tt differs from t" P(z: y (*)) is also different from F(x: y y)t")) due to distributional shift. Following Arjovsky et al. (2019) we assume that æ is generated from invariant feature v, and spurious feature 2, by some unwnown function Ç. i.e., x = E( ).). ). By the invariance property (more details shown in Appendix
we
have y 1 t E for all E T. The target of invariance learning is to make fo only dependent on D..
Invariance Learning. To learn invariant features, Invariant Risk Minimization (IRM) first divides the neural network / into two parts, i.e., the feature extractor & and the classifier w (Arjovsky et al. 2 The goal of invariance learning is to extract ((x) which satisfies y 1 t ())()).
Existing Approximation Methods. Suppose we have collected the data from set of discrete domains, T = {t,, ..., twr)- The loss in domain is R' () == E ww (()))) y)] Since is hard to validate y 1 (xx) in practice, existing works propose to align E. [3/ (a:) for all / as an approximation. Specifically, if ((xz) merely extract invariant features, EE' y ((x)) (r)
would be the same in all t. Let w denote the optimal classifier for domain t, i.c., w := arg min y R' [ww
②). We have w" w'' (((x)) ) = EE' [y| [<x)] hold if the function space of w is sufficiently large (Li et al. 2022). So if ((x)) relies
invariant features, w² would be the same in all t. Existing approximation methods try to ensure w that is the same for all environments to align E² [)]])(( (Arjovsky et al. 2019 Lin
al. 2022a Ahuja al. 2021 Further variants also include checking whether (w, 4 the same Krueger et al. [02T] Ahuja et al. 2020) Zhou et al., al. 2022 Chen et al.) 2022), or the gradient is 2020). For example, REx (Krueger
the same for all (Rame et al.) 2022 Koyama Yamaguchi]
et al. 2021 penalizes the variance of the loss in domains:
CREX EIET (w, D) X/T Var(R 4)).
(I)
where Var(R" 4)) is the variance of the losses among domains.
Remark: The REx loss is conventionally CRES Eeer R' (w, 1) + AVr Q)). We re-scale both terms by [7] for ease of presentation, which does not change the results.
2.2 THE RISKS OF EXISTING METHODS ON CONTINUOUS DOMAIN TASKS
Existing approximation methods propose to learn a feature extractor by matching (F'gg) ((x)) for different t. However, in the continuous environment setting, there are a lot of domains and each domain contains limited amount of samples, leading to noisy empirical estimations E' [g)[(())) Specifically, existing methods employ positive losses on each environment t to assess whether E'(y/ (b (x:) matches the other environments. However, since the estimated E2' [y] (xx) can deviate significantly from E' [b))) these methods fail to identify invariant features. In this part, we first use REx as an example to theoretically illustrate this issue and then show experimental results for
other variants.
Theoretical analysis of REx on continuous domain tasks. Consider a simplified case where æ a concatenation of spurious feature Ig and an invariant feature x.. i.c., æ [g]. Further, & is feature mask, i.c., 4b E {0, 1 Let o and Φ, denote the feature mask that merely select spurious and invariant feature, i.e., 4. 0] and a, = [0, 1]. respectively. Let R'(w, 4) nf Ers (x(x))) y1) denote the finite sample loss on domain t where n' is the sample size in domain t. Denote PREX ()) as the finite-sample REx loss in Eq. equation 1 We omit the subscript 'REx' when it is clear from the context in this subsection. REx can identify the invariant feature mask
L if and only if Z( < L()), v E {0, 1)2. Suppose the dataset S that contains [7] domains. For simplicity, we assume that there are equally n' = n/(T] samples in each domain.
3

---

Published as a conference paper at ICLR 2024
Assumption 1. The expected loss of the model using spurious features in each domain, (ww, Dg) follows a Gaussian distribution, i... R' (w, (w,D,) ~ (N((R, D,), 6R), where the mean R(w (w, D 2) is the average loss over all domains. Given feature mask d { P.), the loss of individual sample (x. y) from environment 1. deviates from the expectation loss (of domain t) by a Gaussian, fly, w (())) B' [L(y, w ((((())))] (O, o The variance as is drawn from a hyper exponen- tial distribution with density function p(a4; A) = eexp( (oo0(-)0)))))))) for each D.
Remark the Setting and Assumption. Recall that R'(w. 1 represents the expected loss of (w, L) in domain t. When & = 2,, the loss is the same across all domains, resulting in zero penalty. However, when T = ... the loss in domain deviates from the average loss, introducing penalty that increases linearly with the number of environments. In this case, REx can easily identify the invariant feature if we have an infinite number of samples in each domain. However, if have multiple environments with limited sample sizes in each, REx can fail with a constant probability. This limitation will be discussed further in the following parts.
Let g- [o. 1] + [0, oc) denote the inverse of the cumulative density function: g = P(= S t) of the distribution whose density is p(z) 1 1/2 that REx fails when there are many domains but with a limited number of samples in each domain.
for O. Proposition P [] shows
Proposition 1. If [T + oc, with probability approaching 1, we have B[C(D,)) < [E((( where the expectation is taken over the random draw of the domain and each sample given da, However, if the domain number [T] is comparable with n, REx can fail with a constant probability. For example, oaya then with probability at least 1/4, E[C( > E(C(O.))
if
(1/T)
Proof Sketch and Main Motivation. The complete proof is included in Appendix F When there are only few samples in each domain (i.e., [T] is comparable to n), the empirical loss R'(w, b) deviates from its expectation by a Gaussian variable e (O, o m2). ). Afte taking square, we have [[(e]] o /2.. There are [7] domains and na n/ So we have Σ, E[{E]] o n, indicating that the data randomness can induce an estimation error that is quadratic in [T] Note that the expected penalty (assuming we have infinite samples in each domain) grows linearly in [T]. Therefore the estimation error dominates the empirical loss L with large [T]. which means the algorithm selects features based on the data noise rather than the invariance property. Intuitively, existing IRM methods try to find feature &b which aligns É' [Y g (x)] among different t. Due to the finite-samll estimation error, È' [Y [=x) can be far away from E' (x))), leading to the failure of invariance learning. One can also show that other variants, e.g., IRMv1, also suffer from this issue (see empirical verification later in this section).
Implications for Continuous Do- mains. Proposition I shows that if we have a large number of domains and each domain contains limited data, REx can fail to identify the invariant feature. Specifically, as long as [T] is larger than O (n),, REx can fail even if each domain contains (vv)) samples.
The performance invariant learning methods
0.70
0.65
0.60
Test Accurscy
0.55
When we have continuous domain in real-world applications, each sam- ple (m:; 31) has a distinct domain in- dex, i.c., there is only one sample in each domain. Therefore when TT] R n, REx can fail with probability 1/4
0.50
IRMv1
0.45
REX
CIL
0.40
10
10a
10a
Number Domains
Figure 2: Empirical validation of how the performance of IRM deteriorates with the number of domains while the total sample size is fixed. The experiments are conducted on CMNIST (Arjovsky et al. (219)) with 50,000 samples. We equally split the original 2 domains into more domains. Since CMNIST only contains classes, 50% test accuracy is close to random guessing. Notably, the data with continuous domains can contain infinite number of domains with only one sample in each domain.
when n
3g
(1/4
Empirical verification. We conduct experiments on CMNIST (Arjovsky et al. 2019) to corroborate ical analysis. The original CMNIST contains 50,000 samples with 2 do-
theoret-
4

---

Published as a conference paper at ICLR 2024
mains. We keep the total sample size fixed and split the 2 domains into 4, 8, 16..., and 1024 domains (more details in Appendix [L). The results in Figure 2 show that the testing accuracy REx and IRMv1 decreases the number of domains increases. Their accuracy eventually drops to 50% (close to random guessing) when the number of domain numbers is 1,024. These results show that existing invariance methods can struggle when there are many domains with limited samples in each domain. Refer Table for results on more existing methods.
Method
IRMv1
REx
IB-IRM
IRMx
IGA
InvRat-EC
BIRM
IRM Game
SIRM
CIL (Ours)
(c)(%)
50.4
51.7
55.3
52.4
48.7
57.3
47.2
58.3
52.2
67.2
Table 1: OOD performance of existing methods continuous CMNIST with 1024 domains.
Further discussion on merging domains. A potential method to improve existing invariance learning methods in continuous domain problems is to merge the samples with similar domain indices into a 'large'' domain. However, since we may have no prior knowledge of the latent heterogeneity of the spurious feature, merging domains is very hard in practice (also see Section for empirical results on merged domains).
OUR METHOD
We propose Continuous Invariance Learning (CIL) as a general training framework for learning invariant features among continuous domains.
Formulation. Suppose (xx) successfully extracts invariant features, and we have y L (a) according to Arjovssyy et al. (019). The previous analysis shows that is difficult to align E' & (x)) for each domain since there are only very limited samples in in the continuous domain tasks. This also explains why most existing methods fail in continuous domain tasks. In this part, we propose to align E (xx) for each class y (see Appendix D for the discussion on the relationship between EE' yy| (xx) and E (x))))(x))) Since there is a sufficient number of samples in each class y. we can obtain an accurate estimation E" it ( (x) ... We perform the following steps to verify whether [t [xx) is the same for each y : we first fit function h H. to predict t based on (xx). Since t is continuous, we adopt L,2 loss to measure the distance between & and h()(2))) i.e., E [[]]((((=)) =||3|_ We another function 9 predict based on ((x)) and y. and minimize (l)g(((x), y) 2. y 1. (x: holds, does not bring more information to predict t when conditioned (xx). Then the loss achieved by g (x)()) y) would be similar to the loss achieved by
h()(x)), i.e. Blllg( (x)), y) - [|
(((())) (112).
In conclusion, solve the following framework:
min (u, 8,50
s.t. min max [[[h(((()) - tlž (g(( y) - -t NEW gEG
In practice, we can replace the hard constraint in equation 2 with soft regularization term as follows:
min max Q(w, g.g. Ex.yt (l A(1)(((()) t| (1)()()).)) +1())
u. g
where À E R is the penalty weight.
Algorithm. We adapt Stochastic Gradient Ascent and Descent (SGDA) to solve equation 2 SGDA alternates between inner maximization and outer minimization by performing one step of gradient each time. The full algorithm is shown in Appendix A.
Our method is indeed based on the assumption that the class variable is discrete. Invariance learning methods have primarily been applied to classification tasks where the class variable is typically discrete, which aligns with our requirement. It is worth noting that previous methods often assume that the domains are discrete, which may not be applicable in many applications where the domains are continuous.
Remark:
5

---

Published as a conference paper at ICLR 2024
3.1 THEORETICAL ANALYSIS OF CONTINUOUS INVARIANCE LEARNING
In this section, we assume & is one-dimensional scalar to ease the notation for clarity. All results still hold when is vector. We start by stating some assumptions on the capacity of the function classes.
Assumption 2. H contains h", where (2) == E[t = ((x)]
Assumption 3. G contains g", where 9 (z.y) == E[t ()))
Then we have the following results:
andi Lemma 1. Suppose Assumption and hold, h² and g" minimize the following losses given a fixed o we have h" (-) = arg min E (a..s) (1( - t)², .nnd g"() = arg min geg ((=.).)(9((((2) y)
t)² where the proof is shown in Appendix
Theorem 2. Suppose Assumptions 2 and hold. The constraint of equation 2 is satisfied if and only E[t (x=)] (()), 2) holds for Vy
Proof in Appendix H The advantage of CIL is discussed as follows:
The advantage of CIL in continuous domain tasks. Consider a 10-class classification task with an infinite number of domains and each domain contains only one sample, i.e., 11 00, and n/|T| 1, Proposition
T shows that REx would fail to identify the invariant feature with constant probability. Whereas, Theorem aarooremmshwwe shows that CIL can still effectively extract invariant features. Intuitively, existing methods aim to align E' (x)) across different values. However, in continuous environment
settings with limited samples per domain, the empirical estimations É (x)) become noisy. These estimations deviate significantly from the true E' [ (xx)). rendering existing methods ineffective in identifying invariant features. In contrast, CIL proposes to align E9 t (<x)). which can be accurately estimated as there are sufficient samples in each class.
We have shown that the definite advantage of CIL based on Proposition 1 and Theorem 2. In the following part, we are going to show the finite sample property of our CIL for completeness, This finite sample analysis is standard analysis of mini-max formulation based on the results Lei et al. (202T). We consider the empirical counterpart of the soft regularization version (equation 2 for finite sample performance, i.e.,
min max Qww, g. == Ex.y. 0,, 9
((w(((()) =Be.u.(((w(((())) 4) ((1)(((()) t12 (l9(*(x).)) -
+1
where É is the empirical counterpart of expectation E. Suppose that we solve equation 2 with SGDA (Algorithm and obtain (w,, $ g, h), which is (21,c2 optimal solution of equation Specifically, we have
Q(w.6,h.0) inf QWw, h c1, Q(w, 2. h)) 2 sup Q(u h,) w.
(2)
In the following, we denote Q' (w, 4,h) := sup, Q(u, 4 h, We can see that small Q* (w, &, h) indicates that the model (w, 4 h) achieves a small prediction loss of Y well small invariance penalty.
Proposition 2. Suppose we solve equation 2 by SGDA as introduced in Algorithu D using a training dataset of size n and obtain an (c1, solution (w, &, appendix, we then have with probability at least 1 that
h). Under the assumptions specified in the
Q" (τ. (1, S (1 (1+n))
inf Q" (w, 2 h, 1.
1 1+ (e +21 (214))
where Ô absorbs logarithmic and constant variables which are specified in the Appendix
Empirical Verification. We apply our CIL method on CMNIST (Arjovsky et al. 2019) to validate our theoretical results. We attach a continuous domain index for each sample in CMNIST. The 50,000 samples of CMNIST are simulated to distribute uniformly on the domain index t from 0.0 to 1000.0 (Refer to Appendix M for detailed description). As Figure 2 shows, CIL outperforms REx and IRMv1 significantly when REx and IRMv1 have many domains.

---

Published as a conference paper at ICLR 2024
Eav. Type
Method
Linear
Sine
Split Num
OOD
Split Num
OOD
None
ERM
86.38 (0.19)
(0.22)
87.25 (0.46)
16.05 (6.03
IRMv1
51.02 (0.86)
49.72 (0.86)
16
49,74 (0.62)
50.01 (0.34)
82.05 (0.67)
(2.55)
(0.91)
54,97
Discrete
GROupDRO
16
99.16 (0.46)
30.33 (0.30)
2
99.23 (0.04)
30.20 (0.30)
11BNet
63.25 (19.01)
(16.63)
(16.81)
36.41 (15.87)
IRMv1
49.57 (0.33)
48.70 (2.65)
50.43 (1.23)
49.63 (15.06)
Continuous
REx
78.98 (0.32)
41.87 (0.48)
79.97 (0.79)
42.24 (0.74)
Diversify
50.03 (0.04)
50.11 (0.09)
50.07 (0.06)
50.27 (0.21)
CIL (Ours)
57.35 (6.89)
57.20 (6.89)
69.80 (3.95)
59.50 (8.67)
Table 2: Accuracy on Continuous CMNIST for Linear and Sine (t). The standard deviation in brackets is calculated with 5 independent runs. The Env. type "Discrete" means that manually create by equally splitting the raw continuous domains. The environment type "Continuous" indicates using the original continuous domain index. "Split Num" stands for the number of domains manually create and we report the best performance among spilt {2, 4, 8, 16). Detailed results in Appendix L
EXPERIMENTS
To evaluate our proposed CIL method, we conduct extensive experiments on two synthetic datasets and four real-world datasets, the synthetic logit dataset is presented in Appendix K
we compare CIL with 1) Standard Empirical Risk Minimization (ERM), 2) IRMvl proposed in Arjovsky et al. (2019), 3) REx in equation [ proposed in Kruueer et al. (202I), and 4) GroupDRO proposed in [Sagawa et al.) (2019) that minimize the loss of worst group(domains' with increased regularization in training. 5) Diversify proposed in Lu al. (2022) and 6) IBNett proposed in Li et al. (2022), adding invariant information bottleneck (IIB) penalty in training. Note that while CIDA (Wang et al. (2020) and its variants (Xu al.) 2022 2023 Liu al. 2023) also handle continuously indexed domains, they
domain adaptation methods and therefore not included as baselines (see Appendix for details), For IRMv1, REx, GroupDRO, and IIBNet, we try them the original continuous domains as well as manually split the dataset with continuous domains into discrete ones (more details in separate subsections below). All the experiments are repeated least three times and we report the accuracy with standard deviation on each dataset.
4.1 SYNTIETIC DATASETS
4.1.1 CONTINUOUS CMNIST
Setting. We construct continuous variant of CMNIST following Arjovsky et al. (2019). The digit the invariant feature x. and the color is the spurious feature x,, Our goal is to predict the label of the digit, y. We generate 1000 continuous domains. The correlation between o. and the label y is pv. = 75%, while the spurious correlation (t) changes among domains t, whose details are included in the Appendix L Similar to the previous dataset, we try two settings with (t) being linear and Sine function.
Results. Table 2 reports the training and testing accuracy of methods on CMNIST in two settings. We also tried different domain splitting schemes for IRMv1, 1, REx, GroupDRO, and IIBNet with the complete results in the Appendix ERM performs very well in training but worst in testing, which implies ERM tends to rely on spurious features. GroupDRO achieves the highest accuracy training but the lowest in testing except for ERM. Our proposed CIL outperforms all baselines on two settings by at least 8% and 5%, respectively.
4.2 REAL-WORLD DATASETS
4.2.1
HOUSEPRICE
We also evaluate different methods on the real-world HousePrice dataset from Kaggld Each data point contains 17 explanatory variables such as the built year, area of living room, overall condition
https: : www. kaggle. cww.c//ouse-eeeeeoooo//occ.coe.a pri ces -aaa-addacced- regress intrrsssion-echheee on-chhniiees

---

Published as a conference paper at ICLR 2024
HousePrice
Insurance Fraud
Alipay Auto-scaling
Env. Type
Method
ID
OOD
ID
0OD
ID
OOD
None
ERM
(1.42)
73.94 (5.04)
79.98 (1.17)
(1.44)
89.97 (1.35)
57.38 (0.64)
IRMv1
84.29 (1.04)
73.46 (1.41)
(1.84)
67.28 (0.64)
88.31 (0.48)
66.49 (0.10)
REx
84.23 (0.63)
71.30 (1.17)
78.71 (2.09)
73.20 (1.65)
89,90 (1.08)
65.86 (0.40)
Discrete
GroupDRO
85,25 (0.87)
74.76 (0:98)
86,32 (0.84)
(1.30)
91.99 (1.20)
59.65 (0.98)
NeB
52.99 (10.34)
47.48 (12.60)
73.73 (22.96)
69.17 (18:04)
61.88 (13.01)
52.97 (12.97)
IntRAT
83.33 (0.12)
74.41 (0.43)
82.06 (0.72)
73.35 (0.48)
89.84 (1.38)
57.54 (0.47)
IRMv1
82.45 (1.27)
75.40 (0.99)
54,98 (3.74)
52.09 (2.05)
88.57 (2.29)
66,20 (0.06)
REx
83.59 (2.01)
68.82 (0.92)
72.90 (1.64)
72.90 (0.46)
89.94 (1.64)
63,95 (0.87)
Diversify
81.14 (0.61)
70.77 (0.74)
(3.14
80.16 (0.24)
59.81 (0.09)
IeB
62.29 (4.40)
53.93 (3.70)
76.34 (5.20)
72.01 (6.99)
80.49 (8.30)
58.89 (5.53)
InvRAT
82.29 (0.76)
77.18 (0.41)
80.63 (1.04)
72.07 (0.74)
88.74 (1.54)
60.58 (3.22)
Continuous
EIIL
82.62 (0.42)
76.85 (0.44)
80.60 (1.36)
72.44 (0,58)
91.34 (1.50)
(3.14(0.74)
HRM
84.67 (0.62)
77.40 (0.27)
(1.11)
73.52 (0.46)
89.84 (1.20)
55.44 (0.35)
ZIN
84.80 (0.60)
77.54 (0.30)
81.93 (0.73)
73.33 (0.43)
90.56 (0.91)
58,99 (0.87)
CIL (LI)
83,41 (0.75)
77.98 (1.02)
82.39 (1.40)
76.54 (1.03)
81.44 (1.77)
68.51 (1.33)
CIL (1.2)
82.51 (1.96)
79.29 (0.77)
80.30 (2.06)
75.01 (1.18)
81.25 (1.65)
71.29 (0.04)
Table 3: Accuracy of each method on three real-world datasets with standard deviation in brackets. Each method takes 5 runs independently. The details of the settings for HousePrice, Insurance Fraud, and Alipay Auto-scaling can be found in Section 1, 4.2.2 and 4.2.3 respectively. CIL is our method. L1 or L.2 means we use the L1 or L.2 loss. We adopt L.2 loss by default in other tables.
rating, etc. The dataset is partitioned according to the built year, with the training dataset in the period [1900, 1950] and the test dataset in the period (1950, 2000]. Our goal is to predict whether the house price is higher than the average selling price in the same year. The built year is regarded the continuous domain index in CIL. We split the training dataset equally into 5 segments for IRMv1, REx, GroupDRO, and IBBNet with a decade in each segment.
3 GroupDRO performs the best across all baselines both on training and testing, while IIBNet seems unable to learn valid invariant features in this setting. REx achieves high training accuracy but the lowest testing accuracy except for IIBNet, indicating that learns spurious features. Our CIL outperforms the best baseline by over 5% on testing accuracy, which implies the model trained by CIL relies more on invariant features. Notably, CIL also enjoys much smaller variance on this dataset.
Results.
The training and testing accuracy is shown in Table
4.2.2 INSURANCE FRAUD
This experiment conducts a binary classification task based on vehicle insurance fraud detection dataset on Kaggle After data preprocessing, each insurance claim contains 13 features including demographics, claim details, policy information, etc. The customer's age is taken as the continuous domain index, where the training dataset contains customers with ages between (19, 49) and the testing dataset contains customers with ages between (50, 64). We equally partition the training dataset into discrete domains with 5 years in each domain for existing methods dependent on discrete domains. Results in IRMvl inferior to ERM in terms of both training and testing performances. REx only slightly improves the testing accuracy over ERM. Table 3 that CIL performs the best across all methods, improving by about 2% compared to other methods.
4.2.3 ALIPAY AUTO-SCALING
Auto-scaling QQia et al. 2022) is an effective tool in clastic cloud services that dynamically seale computing resources (CPU, memory) to closely match the ever-changing computing demand. Auto- scaling first tries to predict the CPU utilization based on the current internet traffic. When the predicted CPU utilization exceeds threshold, Auto-scaling would add CPU computing resources to ensure a good quality of service (QoS) effectively and economically. In this task, we aim to predict the relationship between CPU utilization and the current running parameters of the server in the Alipay cloud. Each record includes 10 related features, such as the number of containers, and network flow, etc. We construct a binary classification task to predict whether CPU utilization is above the 13% threshold or not, which is used in the cloud resource scheduling to stabilize the cloud system.
https: / /ww. k kaggle com http://ieee.i.iisssttoooorrrrees www.:ssseennnnnnnnnnnneee
8

---

Published as a conference paper at ICLR 2024
Method
ID OOD
Fine tuning
81.98 69.62
EWC Kirkpatrick et 2017
80.07 66.61
LZnke 2017
78.70 65.18
A-GEM (opez- & Kanzato 2017
81.04 67.07
ERM
79.50 63.09
GroupDRRO- Sagawa et
77.06
Distribution shift over time
mixupp Zhang el al
2017a
83.65 58.70
CORAL.
T Tuan & Kaenkn
2016
77.53 668.53
IRM-T AArovsky a.
2019
80.46 59.34
SimCLR Chen
2020
78.59 64,42
Swav aron et ai
4020
78.38 60.15
SWA Izmailov et al.
2018
84.25 67.90
Time
CIL (Ours)
82.89 71.22
1930s
1980s
2000s
2010s
The accuracy the worst test OOD domain of each method on Yearbook dataset on Wild-time. The performance of baseline methods is copied from Yao et al. (2022).
Figure 4:
Figure 3: An illustration Yearbook Yao al. 2022).
Images taken from Yao et al. (2022).
We take the minute the day as the continuous domain index. The dataset contains 1440 domains with 30 samples in each domain. We then split the continuous domains by consecutive 60 minutes as the discrete domains for IRMv1, 1, REx, GroupDRO, and IIBNet. The data taken between 10:00 and 15:00 are the testing because the workload variance in this time period is the largest and exhibits obviously unstable behavior. All the remaining data serves as the training set. Table 3 reports the performance of all methods. ERM performs the best in training but worst in testing, implying that ERM suffers from the distributional shift. On the other hand, IRMv1 performs the best across all existing methods, exceeding ERM by 9%. CIL outperforms ERM and IRMv1 by 13% and 5%, respectively, indicating its capability to recover more invariant model achieving better OOD generalization.
4.2.4 WILDTIME-YEARBOOK YEARBOOK
We adopt the Yearbook dataset in Wildtime benehmark (Yao 2022 which gender classification task on images taken from American high school students as shown in Figure 3 The Yearbook consists of 37K images with the time index as domains. The training set consists of data collected from 1930-197 1970 and the testing set covers 1970-2013. We adopt the same dataset processing, model architecture, and other settings with the WildTime. To keep consistent with Yao al. (2022). 5 adopt the same baseline methods with Yao al. (2022) and directly copy the performance of baseline methods from Yao et al. (2022). (1122).For For the details of these baselines, we refer the reader to the Appendix B of Yao al. (2022). Table 4hoowssswwwws that achieve the best OOD performance of 71 71.22% improving about 1.5% over the previous SOTA methods (marked with underline).
CONCLUSION AND DISCUSSION
We proposed Continuous Invariance Learning (CIL) that extends invariance learning from discrete categorical indexed domains to natural continuous domain in this paper and theoretically demonstrated that CIL is able to learn invariant features on continuous domains under suitable conditions. However, learning invariance would be more challenging with larger DNNs due IRM's inherent sensitivity to over-fitting (Lin et al. 2022a Zhou et al. (022)). Recent works has shown the effectiveness of called spurious feature diversification (Lin al.] (023a), which has shown very promising performance even on modern large language models (Lin et al.) 2023b). It would be an interesting future direction explore feature diversification on continuous domains.

---

Published as a conference paper at ICLR 2024
REFERENCES
Kartik Ahuja, Karthikeyan Shanmugam, Kush Varshney, and Amit Dhurandhar. Invariant risk minimization games. In International Conference on Machine Learning, pp. 145-155. PMLR, 2020.
Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, SannCChrittophe Gagnon-Audet, Yoshua Bengio, Ioannis Mitliagkas, and Irina Rish. Invariance principle meets information bottleneck for out-of- distribution generalization. Advances in Neural Information Processing Systems, 34:3438-344 3450, 2021.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:/907.022: 2019.
Adeleh Bitarafan, Mahdieh Soleymani Baghshah, and Marzieh Gheisari. Incremental evolving domain adaptation. IEEE Transactions on Knowledge and Data Engineering, 28(8):2128-214 2016.
Andreea Bobu, Eric Tzeng, Judy Hoffman, and Trevor Darrell. Adapting to continuously shifting domains. 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912-9924, 2020.
Shiyu Chang, Yang Zhang, Mo Yu, and Tommi Jaakkola. Invariant rationalization. In International Conference on Machine Learning, pp. 1448 1458. PMLR, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Yongqiang Chen, Kaiwen Zhou, Yatao Bian, Binghui Xie, Kaili Ma, Yonggang Zhang, Han Yang, Bo Han, and James Cheng. Pareto invariant risk minimization. arXiv preprint arXiv:2206.07666. 2022.
Elliot Creager, Jörn-Henrik Jacobsen, and Richard Zemel. Environment inference for invariant learning. In International Conference Machine Learning, pp. 2189-2200. PMLR, 2021.
Li Deng, Geoffrey Hinton, and Brian Kingsbury. New types of deep neural network learning for speech recognition and related applications: An overview. In 2013 IEEE international conference on acousties, speech and signal processing, pp. 3599-8603. IEEE, 2013.
Farzan Farnia and Asuman Ozdaglar. Train simultaneously, generalize better: Stability of gradient- based minimax learners. In International Conference on Machine Learning, pp. 3174-3185. PMLR, 2021.
Trygve Haavelmo. The probability approach in econometrics. Econometrica: Journal of the Econometric Society, pp. iii-115, 1944.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Judy Hoffman, Trevor Darrell, and Kate Saenko. Continuous manifold based adaptation for evolv- ing visual domains. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 867-874, 2014.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson, Av- eraging weights leads to wider optima and better generalization. arXiv preprint arXiv: / 1803.05407, 2018.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
10

---

Published as a conference paper at ICLR 2024
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Domain extrapolation via regret minimization. arXiv preprint arXiv:2006. 03908, 2020.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for robustness to spurious correlations, arXiv preprint 2200::2200002933, 2022.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Barnskka-aariikka,... et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526, 2017.
Masanori Koyama and Shoichiro Yamaguchi. Out-of-distribution generalization with maximal invariant predictor. 2020.
David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapola- tion (rex). In International Conference on Machine Learning, pp. 9815-5826. PMLR, 2021.
Yunwen Lei, Zhenhuan Yang, Tianbao Yang, and Yiming Ying. Stability and generalization of stochastic gradient methods for minimax problems. In International Conference on Machine Learning, pp. 6175-61851 6186.. PMLR, 2021.
Bo Li, Yifei Shen, Yezhen Wang, Wenzhen Zhu, Dongsheng Li, Kurt Keutzer, and Han Zhao. Invariant information bottleneck for domain generalization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 7399-7407, 2022.
Shaojie Li and Yong Liu. High probability generalization bounds with fast rates for minimax problems. In International Conference on Learning Representations, 2021.
Yong Lin, Hanze Dong, Hao Wang, and Tong Zhang. Bayesian invariant risk minimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16021-16030, 2022a.
Yong Lin, Shengyu Zhu, and Peng Cui. Zin: When and how learn invariance by environment inference? arXiv preprint arXiv:2203.05811 2022b.
Yong Lin, Lu Tan, Yifan Hao, Honam Wong, Hanze Dong, Weizhong Zhang, Yujiu Yang, and Tong Zhang. Spurious feature diversification improves out-of-distribution generalization. arXiv preprint arXiv:2309.172: 2023a.
Yong Lin, Lu Tan, Hangyu Lin, Zeming Zheng, Renjie Pi, Jipeng Zhang, Shizhe Diao, Haoxiang Wang, Han Zhao, Yuan Yao, et al. Speciality vs generality: An empirical study on catastrophic forgetting in fine-tuning foundation models. arXiv preprini urxiv:2309.06256 2023b.
Tianyi Liu, Zihao Xu, Hao He, Guangyuan Hao, Guang-He Lee, and Hao Wang. Taxonomy-strun structured domain adaptation. In ICML, 2023.
Yinhan Liu. Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanow. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: arXiv: 1692, 2019.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Wang Lu, Jindong Wang, Xinwei Sun, Yiqiang Chen, and Xing Xie. Out-of-distribution representation learning for time series classification. In The Eleventh International Conference on Learning Representations, 2022.
Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two- layer neural networks. Proceedings of the National Academy of Sciences, 115633))7766557777) 2018.
Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society. Series B (Statistical Methodology), pp. 947-1012, 2016.

---

Published as a conference paper at ICLR 2024
Huajie Qian, Qingsong Wen, Liang Sun, Jing Gu, Qiulin Niu, and Zhimin Tang. Robustscaler: Qos-aware autoscaling for complex workloads. arXiv preprint arXiv:2204.07199 2022.
Alexandre Rame, Corentin Dancette, and Matthieu Cord. Fishr: Invariant gradient variances for out- of-distribution generalization. In International Conference on Machine Learning, pp. 18347-18377.
PMLR, 2022.
Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. Domain-adjusted regression or: Erm may already learn features sufficient for out-of-distriutioo generalization. arXiv preprint WrXiv:2202.06836,, 2022.
Dominik Rothenhäusler, Nicolai Meinshausen, Peter Bühlmann, and Jonas Peters. Anchor regression: heterogeneous data meets causality. XrXiv preprint arXiv:/801.06229, 2018.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08733 2019.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In Computer Vision-ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14, pp. 443-450 Springer, 2016.
Hao Wang, Hao He, and Dina Katabi. Continuously indexed domain adaptation. In JCML, 2020.
Markus Wulfmeier, Alex Bewley, and Ingmar Posner. Incremental adversarial domain adaptation for continually changing environments. In 2018 IEEE International conference on robotics and automation (ICRA), pp. 4489 4495. IEEE, 2018.
Chuanlong Xie, Fei Chen, Yue Liu, and Zhenguo Li. Risk variance penalization: From distributional robustness to causality. arXiv e-prints, pp. arXiv-2006, 2020.
Zihao Xu, Guang-He Lee, Yuyang Wang, Hao Wang, al. Graph-relational domain adaptation. In ICLR, 2022.
Zihao Xu, Guangyuan Hao, Hao Hc, and Hao Wang. Domain indexing variational bayes: Interpretable
domain index for domain adaptation. In ICLR, 2023.
Huaxiu Yao, Caroline Choi, Bochuan Cao, Yoonho Lee, Pang Wei W Koh, and Chelsea Finn. Wild- time: A benchmark of in-the-wild distribution shift over time. Advances in Neural Information Processing Systems, 35: 10309-10344 2022.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International conference machine learning, pp. 3987-3995. PMLR, 2017.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint
20177.
2017a.
Junyu Zhang, Mingyi Hong, Mengdi Wang, and Shuzhong Zhang. Generalization bounds for stochastic saddle point problems. In International Conference on Artificial Intelligence and Statistics, pp. 568-576. PMLR, 2021.
Kun Zhang, Biwei Huang, Jiji Zhang, Clark Glymour, and Bernhard Schölkopf. Causal discovery from mmnstationaryhheerogeeeee data: Skeleton estimation and orientation determination. In IJCAI: Proceedings of the Conference, volume 2017, pp. 1347. NIH Public Access, 2017b.
Xiao Zhou, Yong Lin, Weizhong Zhang, and Tong Zhang. Sparse invariant risk minimization. In International Conference on Machine Learning, pp. 27222-27244 PMLR, 2022.
12

---

Published as a conference paper at ICLR 2024
A LIMITATION AND SOCIAL IMPACT
Limitation Our framework uses min-max strategy which acquires the suboptimal solution in specific scenarios, and takes enough discrete time points continuous domains in experiments without true continuous environments with infinity domain indices.
Social Impact We extend invariant learning into the continuous domains which is more natural to real-world tasks. Specifically, would be helpful to solve the OOD problems related to time domains, e.g. it has been applied Alipay Cloud achieve the auto-scaling of server resources.
B RELATED WORK
Causality, invariance, and distribution shift. The invariance property is well known in causal literature back to early 1940s (Haavelmo) 1944), showing the conditional of the target given its direct causes invariant under intervention any node in the causal graph except for the target itself. In 2016, Invariant Causal Prediction (ICP) is first proposed in Peters et al. (2016) to utilize invariance property to identify the direct causes of the target. Models relying are robust to interventions. From the perspective of causality, the distributional shift in the testing distribution is due to the interventions on the causal graph (Arjovsky et al.) (019). So model with invariance property can hopefully generalize even in the existence of distributional shifts. Anchor regression (Rothenhäusler al., 2018) build the connection between distributional robustmess with causality, and demonstrates that suitable penalty in anchor regression based on the causal structure equivalent to ensure a certain degree of robustness distributional shift. Notably, |Peters al.) (2016); RRothenhäusler et al. (2018) both assume the inputs æ are handcrafted meaningful features, which limits their applications machine learning and deep learning where the input can be raw images.
the target's direct causes
Arjovsky et al. (2019) proposed the first invariance learning method, invariant risk minimization (IRM), which extends ICP (Peters et al. 2016) to deep learning by incorporating feature learning. Invariance learning has gained its popularity in recent years and inspired a line of excellent works, where variety of variants have been proposed. To name few, Krueger et al. (2021); Xie al. (2020) penalize the variance of losses in domains: /Ahuja et al. (2020) incorporates game theory into invariance learning; Chang et al. (2020) estimates the violation of invariance by training multiple independent networks; Jin et al. (2020) proposes an invariance penalty based on regret. Notably, these works all require discretely indexed domains. Another line of works try to learn invariant features when explicit domain indices are not provided (Creager et al. 202T). However, Lin et al. (2022b) theoretically shows that it is generally impossible learn invariance without environmental information. Some recent works (Lin et al., 2022a; /hou et al. (022) show that invariance learning methods are sensitive to overfitting caused by the overparameterization of deep neural networks. These works are orthogonal to our study.
Distribution Shift with Continuous Domains. There are few methods considering domain shifts with continuous domain indexes. [Bobu et al. (2018); Hoffman et al. (2014); Wulfmeier et al. (2018); Bitarafan et al. (2016) consider the distribution shifts incremenaally over time, while trying to perform domain adaptation sequentially. Continuously indexed domain adaption (CIDA) (Wang al. 2020 the first to adapt across (multiple) continuously indexed domains simultancously. Note that these works all assume that the input of the samples in the testing domains are available, and are therefore not applicable for the OOD generalization tasks considered in this paper. Zhhng et al. (2017b) tries to discover causal graph based the continuous heterogeneity, but assumes that the features given (cannot be learned from raw input), making it not applicable in our setting either.
C INVARIANCE PROPERTY IN CAUSALITY.
Figure 5 shows an example (similar to Figure of ICP (Peters et al. 2016)) of an causal system with nodes (2) 1 22, Ia, .4). Suppose our task is to build model based on a subset of {21, 22, 3D, x to predict y. under the distributional shifts in different domains caused by interventions. In our example, there interventions node 2y in domain 2, and interventions nodes 23 and 24 in domain 3. Notably, we do not allow interventions on target itself (indicating that the noise ratio of y should be constant among domains). The invariance property shows that the conditional probability
13

---

Published as a conference paper at ICLR 2024
of y given its parents remains the same on interventions any nodes except for the y itself. In this example, P(y 22, rs) remains the same under all the three domains. However, one can check that ((y( and P(y(z4) changes in domain 2 and 3, respectively. Therefore it is safe to build the model on (I2, to predict y. In contrast, 21 and 444 are unreliable because the conditional distribution is unstable under distributional shifts. In this example, By is (33) and æ, is (T4)
crnfrommnne =
enstronmente
Figure 5: An illustration of the invariance property in causality (similar to Figure in ICP (Peters iet al. 2016)). This figure shows a causal system with five nodes, y, 21, 22, I3, and x4. Our task to predict y based on the z's. There are different interventions in different domains, leading to distributional shifts. Intervention on node can be simply interpreted as changing the node value. The changes can propagate to the descendants of the intervened node. The invariance property shows that P(y 22, Fz) remains the same in all three domains. In contrast, P(y|z) and P(y[æ4) changes in domain and 3 due interventions, respectively. So it is safe to build model 22 and my to predict y, which is expected to be stable under novel testing distribution.
D DISCUSSIONS ON THE FIRST MOMENTS
To encourage conditional independence 1 (a)( existing IRM variants try to align (x)))((x))
(see Section 2.1 and our CIL proposes to align [ [t| (xx) (see Section
D.1 DISCRETE DOMAIN CASE
For discrete domain problems, when there are sufficient number of samples in each domain t and class y. ES ["[t] (x))] and E' /yy [xx) performs similarly (both reflecting the first moment of y (x)), although from different perspectives), and neither of them has a clear advantage over the other. Consider the following example, which has gained popularity (Lin et al.) (2022b): We have
binary classification task, y E {-1, 13, containing 2 domains {i, 2}. one invariant feature x and one spurious feature T.. Furthermore, we have
1, with prob 0.5,
Evs with prob 0.8, -. with prob 0.2,
y
hold for both domains.
1,, with prob 0.5,
And
y, with prob 0.5, In domain 1, x, y, in domain 2, x, - y. with prob 0.5,
In other words, the correlation between the invariant feature x. and the label is always 0.8. The correlation between æ, and y is I in domain and 0.5 in domain 2. By simple calculation, we have
E y x. 1]
EE y 1] 0
E [y]a I -1]
E [y/x,
=
E
Iylan a 1] = 1, E E x = = =
[ x. = 1] = 1, E B [2 = -
14

---

Published as a conference paper at ICLR 2024
and
B== 2 1] = 4/3, 3y=-l[tz 1] =
E mm. = =-1]= =
R [t F. -1] = =4/3
11 = =1 =1|=3 =3/2,
E== 1 =3/2
E
[t = - E [t/ = --1 =
So we can observe that [=[t/=,]] is the same for all and E [y| is the same for all t. Additionally, E [t/as] is different for y other words, if we aim to select a feature from E {mo, vt to align either |E* [4|ar] or E' ['(g(a..) we would choose the invariant feature Ty in both cases. So our CIL method and existing methods performs similarly in this example with 2 domains.
and y 1, while EE' [/y/a is different for / = 1 and 2. In
D.2 CONTINUOUS DOMAIN CASE
Consider 10-class classification task with infinite number of domains and each domain contains only one sample, i.e., n - oc, and n/[7] = Proposition [ shows that REx would fail to identify the invariant feature with constant probability. Whereas, Sreeoremsshor effectively extract invariant features. Intuitively, existing methods aim to align E [y [(x]] across different values. However, in continuous environment settings with limited samples per domain, the empirical estimations E' [x)) become noisy. These estimations deviate significantly from the true E' (y] ( (x)), rendering existing methods ineffective in identifying invariant features. In contrast, CIL proposes to align E (x)),, which can be accurately estimated as there are sufficient sample in each class.
that CIL still
E ALGORITHM
Algorithm CIL: Continuous Invariance Learning
Input: Feature extractor Ob label classifier w, domain index regressor and g; The training dataset D [(x; yi, t;) Output: The learned @b, classifier w, the domain regressor h and g.
Initialize Φ w. and g.
2: while Not Converge do
3: Sample batch B from D
4: Obtain the loss for on the batch B.
LB( .) = B/ (x (x), y), tl2
5: Perform one step of gradient ascent on La( g) w.r.t g, i.e., g + g g VV
6: Obtain the loss for (æ,w,,) w, h) Al]h(((xx), t|13)
on B. Ls(w, h
1/|B] Ex,,EE (e(w(*(x)),
7: Perform one step of gradient descent on La(w, 4 w.r.t (w, 2, 1,,, i.c.
(w, &, /)) + (w, 2 h)) + v (w. (w( & 4.4)
8: end while
9: return (w, h.)
F PROOF OF PROPOSITION
Recall that R$ (' ② E ((((() y)] denotes the lose of (w, 4)) in domain where is the classifier & is the feature selector, and is the lose function. If there are many domains (i.c., T is large) and each domain only contains limited sample (i.e., n is limited), then the empirical REx
15

---

Published as a conference paper at ICLR 2024
loss is follows:
[((())) Σ Rd (w, @) + V/T//ar(R" (w; +))
1ET
(w, Φ)+ + [w,(w))) 1 - (w, 1))
1T
VET
1ET
1/7/ (- (-(R'(us, (R, () - (w, 2) + (R(w, R(w, + (Ri D) - R(w, 2)) 2
Σ
(w, 1) + A Σ
(R' +) R(w. 4)) 2
1ET
IET
Ao
R² $) - Rw, 4)) + Σ
(w, b) R',W,, 2
Σ
(R(w, b) - RR,,, Φ))
+
AET
SE
A
A2
Ag
Σ
(R; Q)- - R² (w, 8)) ( (R(w, b) R(w,, $))
IET
Σ
RR' D)- Rw 1)) (R' D) R(w, 4))
1ET
As -
(R(w,)-RR(u, B(w (R' の) - R(W,,))
IET
We then have
(TT() if = 4.
Ag
0, if +- $
if d- 4 nall
nhe
A
ΣΣ
4 V/V(011
IET
Re
A2 Σ
0/ [T)x((
Aa Σ
3)) 0/ (Vx((IT)
1=1
A4 2
Σ
R', (w, 2) - () Re' 0))
(R(w,, 2) R(w,, 1
NET
-2
E 2 V x(1)
i=1
i=1
Ag ~ / + x - /22/ + + xxx((T)
Ag -2 (RW, R(w, 1) Σ
(R' (w, 1) - RR(,, 1)) = )) 0 0
1ET
By taking account nee n/ we have
EIET R2 2.) + (2+ 17 77)
Bl((((x)) o,, Te, Tv, Cmull, anl =
EIET (., + (2+ |T| + 1733
ELEr R (ws; @ + سسي (2 + 7 + 1T13)
16

---

Published as a conference paper at ICLR 2024
Denote Q (EiET (w, Fo) - EvET R' (w,
assume 8x/T|
Q.
have
(T))o, ov, oRl > E (2)) vn, RRl
(0, -,, n VulaRi T + 1Q)
valon 7 (2 (7 +
1T
Since 82, 8, are independently drawn from a hyper exponential distribution where the density function Then if
P(x; A) X exxp(-Az), so P(d.- 6, S 2) 1
DRYR
AG-
REx is unable identify the invariant feature with a probability of least 1/4.
G
PROOF FOR LEMMA
Proof.
arg min E (b - t)²
MEH
min ((ott) ((((()))) t)²
MEH
arg min Bng (b()()))))))))()))) (h(((x)) 0)²
Because
(M 1)a
=h(b(=))² 2h( (=)) + [()))(z))
we then know minimum achieved at 4( (x)()) = [[([(())] (*(())) by solving this
quadratic problem. The minimum loss achieved by h (+)
Ex(a)
([(())()))) E ([t
We can prove the result for (-) a similar way, and the minimum loss achieved by g-()) is (g (v))) vl].
H PROOF FOR THEOREM
Proof. The penalty term of Equ equation is
min max Ex,yt NEW SEg
[((((())) t) (x()(())))) v) t)
= Eo(w [V]] Eo(x) M ( (z) vil
= Eo())
| az) |
E (
( ( 18 4 (z), y))
20.
The last inequality is due to Jensen's 's inequality and the convexity of the quadratic function. The inequality is achieved only when [(([(((=)) = E[t (xx), vy y.
1 PROOF FOR PROPOSITION 2
Denote 0 = [w. Tb. h] and 9 w क. Ål. We further use q(0, g; denote the loss of Eqn 2 on single sample z : [x, y, z]. We start by stating some common assumptions in theoretical analysis as follows: Our goal is to analyze the performance of the approximate solution (ι,, ②., h, g) on the training dataset with finite samples.
17

---

Published as a conference paper at ICLR 2024
Assumption ((Li & Liu 2021)). Let D be the dataset generated by replacing data point in the training dataset DD" with another data point drawn independently from the training distribution. We assume SGDA is e-argument-stabe if for any D such that
|0SGDA (D") - OSGDA (D")I e,
l1gsGDa (D") ISGDA (oCDA(D")\lE= S
(5)
Assumption 5. ÍiLi & Liu 2021i] Denoting (w, 2, h) as 0 and Qww, D, h, g) as Q(0, g) for short, Q(0,g) g) is p-strongly convex in U, i.e.,
Q(01,9) Q(02,9) VoQ(02.9) (0: 02) 1 1101 02112.
and ju-strongly concave in g. i.e.
Q(01, Q(02.9) VoQ(02,9)T 02) 110 0. 02112-
The convex-concant assumption for the minimax problem popular in existing literature (Li Liu 2021: Lei et 2021; Farnia & Ozdaglar 2021 Zhang al., 202I). simply because non-convex- non-concave minimax problems are extremely hard to analyze due to their non-unique saddle points. When w and h (the classifier for y and regressor for t) are linear, it easy to verify that convex in them. Furthermore, recent theoretical studies show that the overparameterized neural networks (NN) behave like convex systems and training large NN is likely to converge to the global optimum (Jacot et al., 2018: Mei et al., (018).
Assumption (Lipschitz continuity (Li & Liu 2021 ). Let L > Assume that for any 9 and z, have
|Vo gi || S L
and |V Vgg(O, g; <.
Assumption 7 (Smoothness
Li & Liu 2021)). Let B > 0. Assume that for any 01, O2. 91, 92 and
we have
Vof(f11 (0, (1))
(of (00(0,, 92; z)
5 (0) - 02
91 92
We define the strong primal-dual empirical risk as follows:
( 9) sup Q(0.g) - inf ((0.9)
We first restate Theorem 2 as follows:
Theorem 3. Assume we solve equation by SGDA as introduced Algorithm 7 and obtain (E1,
solution (Ó, ii). Further, suppose Assumption 5 holds and SGDA e-argument stable as described in Assumption 4 then for any ö >> 0. 17 0, we have with probability at least 8
Q' (0) 5(1+ inf Q" (0) + + M
log 3 ++++1) Le log, n log x + c1 + 62
where μ is the strongly-convexity, L is the Lipschitz, is the stability, M is is the sample size of the training dataset, Ô absorbs logarithmic and constant variables which is specified in the appendix.
Proof. By the suboptimality assumption of â and ÿ. we have
ASCDA(0,0) ssup Q(0.g) inf Q(0,9)
8
<Q + (Q(0.0) -))
=6 + 62
Denote d = arg ming (f) and /" = arg max, Q(0.g) g). We can decompose
Q' inf (0) Q* Q (0) + Q Q(00 .0) Q(0 .9) -(00,,)) 2(0" Q"
As
We bound A A4 respectively:
18

---

Published as a conference paper at ICLR 2024
From Eqn (22) of Li & Liu
2021), we have
2M (3/6)
3 + μ
A I
+ 50v
log, n3og(3e/6)+
(6)
3n
LMQ(0.0)+1/2(0//11 1)3122 32m(3/2 1)2 19/ (og])))))) log
n
(7)
we have
A sup ((0,g) =
((0)
(8)
<sup Ssup@(0,n) ((0,g) inf ((0,9)
(9)
Sel +
(10)
By Eqn (10) of & Liu (2021).
2M (3/6)
A3 10000 .9) Q0000"0000000 .9)
50v 5ee log, log(3e/6)
(11)
3n
(4MQ(0+,)) 22/2 + 32nc² log log(3/0)
+
n
(12)
At last
Q(0", 9) (0*) Q(0" sup Q(0" .9) s O.
(13)
Putting these together with some rearrangement, we finally have
(0) + (0)+cl+ 1++ 7
1 8 + 6 + + 7 n 8 + (e +1 Le log, n log
M log
9 S(1 + 7) Qn Q' (0) where C' is a constant. the description of the CIL method in the introduction section is hard to understand.
J DETAIL DESCRIPTION ABOUT SETTING OF EMPIRICAL VERIFICATION
In this section, we show the spurious relationship ps (t) varies across the domains shown in Figure
6
Empinical anelysis
0.55
0.90
Figure 6: Spurious Relation of Empirical Verification 3.1
K LOGIT DATASET EXPERIMENT
Setting. We generate the first synthetic dataset with the invariant feature x, E IR?, spurious feature I, E 20 and target f0, 1). The continuous domains t E [0, 100). The conditional distribution of x, given y varies along t as follows:
= (0 w.p. 0.5.
x,, ~ [N(y.0"), w.p. Pos
(N(v.09)
W. ~ [N(y. 02), w.p. Pt()), [N(-y,0)) w.p. 1 - P. (t),
1, w.p. 0.5.
(N(- w.p. 1 - Po,
19

---

Published as a conference paper at ICLR 2024
linear
sine
0.90
0.50
0.85
0.00
0.70
0.70
8
8
100
80
200
Figure 7: Spurious correlation of logit dataset
Env. Type Split Num
Method
Linear
Sine P=
None
ERM
25.33(2.01)
36.18(3.06)
IRMv1
55.39(6.87)
69.91(2.40)
8
REx
46.46(7.50)
155.51)
Discrete
2
GroupDRO
46.68(1.80)
60.95(2.23)
16
IIBNet
51.50(18.60)
49,6 (21.8))1887)
IRMv1
25.90(2.85)
41.28(3.60
REx
42.73(11.47)
63.33(5.05)
Continuous
Diversify
53.39(2.96)
53.30( 1.93)
CIL
60.95(6.59)
76.25(4.99)
Table 4: Comparison on the synthetic Logit data. The metrics including accuracy and standard deviations in parenthesis calculated on three independent runs. The environment type "Discrete" indicates that we manually create by equally splitting the raw continuous domains. The environment type "Continuous" indicates we use the original continuous domain index. "Split Num" stands for the number of domains we manually create and we report the best performance among spilt {2, 4, 8, 16).
where and Pa (t) are the probabilities of the feature's agreement with label
Notably, ps (t) varies among domains while Pe stays invariant. The observed feature con- catenation of æ. and C., i.c., x == [ v.). We generate 2000 samples as the training dataset {(x, Vi, 2000. Our goal is to learn a model fxx to predict y that solely relies on E.. We set pe
to be 0.9 in all domains. (t) varies across the domains as in Figure
We can see that x, exhibits high but unstable correlation with y. Since existing methods need discrete domains, we equally divide the continuous domain into different numbers of discrete domains, ie, (2, 4,8, 16).
Results. Table shows the test accuracy of each method. ERM performs the worst and the large gap implies that ERM models heavily depend on spurious features. IRMv1 improves by 25% on average compared to ERM. CIL outperforms ERM by over 30% average. CIL also improves by 5-7% over all existing methods based on discrete domains on the best manual discrete domain partition, which indicates that CIL can Iearn invariant features IRMv1 and REx changes with the split number is shown in Table
effectively. The trend of how the performance of
L ADDITIONAL EXPERIMENT RESULTS FOR
Details on continous CMNIST. The original CMNIST dataset consists of two domains with varying spurious correlation values: 0.9 in one domain and 0.8 in the other. To simulate continuous problem, randomly assign time indices 1-512 to samples in the first domain and indices 513-1024 to the second domain. The spurious correlation only changes at time index 513, as shown in Figure on Page 16 of the Appendix file. In this case, each of the 1024 domains comprises approximately 50
20

---

Published as a conference paper at ICLR 2024
Env. Type Split Num Method Linear
Sine
None
ERM
25,33(2.01)
36. 18.10())06
2
IRMvl
53.11
66. 16(4.2)))
55.39(6.87)
(2.91(2.40)
5
39.35(11.70)
55.98(3.91)
8
47.90(2.44)
59.03(5.32)
16
30.66(4.45)
56.90(4.98)
50
27.32(0.99 97) 42.73(11.47)
100
25.90(2.85) 2.85)
41.28(13.66)
Discrete
2
REx
(4).6)
59.76(1.22)
4
$5.80012.499
64.05(11.29)
5
50.77(11.99) _669 65.15(5.77)
46.46(0.75)
68.15(5.51)
16
(4.05(
67.78(6.60)
50
47773(11.47)
61.18(2.78)
100
(1.87(00.88
63.33(5.05)
Continuous
CIL
60.95(6.59) 76.25(4.99)
Table 5: Comparison on the synthetic Logit data. The metric is accuracy and standard deviations in parenthesis are calculated on three independent runs. Split Num stands for the number of domains we manually create by equally splitting the raw continuous domains. We try settings where the spurious correlation (t) changes by linear and Sine functions, respectively.
Env. Type Split Num Method
Train
Test
2
IRMv1
49.74(0.62) (0.04
51.02(0.86) 49.72(0.86)
51.45(3.53) 49.466(0.44)
16
(2.31 49.71(00.90)
100
50.21(1 .6) 49. (0.05(
49.56(0.74) 47.0 (2.02(2.72)
1000
49.67(0.33) 48. 78(655)
Discrete
83.09(0.40) 45.50 12)
82.75 47. 47.14().22)
8
82.05(0.67) 49.31(2.55)
16
82.32(0.45) 47 (7.66()))) 1.54)
80,54(0.78) 49, 12(0.20)
500
(9.21(0.45) 42.55(1.055
1000
78. 78.98(0.32) 41.87(0.48)
Table 6: Accuracy on the continuous CMNIST with (t) as a linear function with different split number in (2, 4, 8, 16, 100, 500, 1000).
samples. Based on the results in Figure 2 in the main part of the manuscript, REx and IRMv1 display testing accuracies close random guessing in this scenario, Similar constructions are made for .... 512 domains.
In this section, we provide the complete experiment results Table and Table 7 for IRMv1, REx, GroupDRO, and IIBNet with other numbers of splits ("split num ") on the continuous CMNIST dataset. The settings are the same as in Section 4.1.I where Table 2 show the best results for each method with one corresponding "split num".
Results. Aable/aand report the training and testing accuracy of different domain splitting schemes (2, 4, 8, 16) for both IRMv1 and REx on the continuous CMNIST dataset with two p, (t) settings. Under linear p. (t), REx performs the best with split number 8 by improving around 4% over the worst one. However, the results fairly close under sine p. (t). On the other hand, IRMv1 with all testing accuracy around 0.5 does not seem to be able to extract useful features for the task with either linear (t) sine (t).
M ADDITIONAL RESULTS FOR THE EXPERIMENTS
The result across different split number is shown in Table g
21

---

Published as a conference paper at ICLR 2024
Env. Type Split Num Method
Train
Test
2
IRMvl
49.47(0.21)
49.44(0.57)
49.77(0.78) 49.96(0.55)
8
49.60(0.65) 49.85(0.26)
16
49.74(0.62) 50.01(0.34)
100
49.62(0.58) 49 49.83(0.63)
500
50,28(0.87)
(6.21(10.1L)
1000
50.43(1.23)
49.63(15.06)
Discrete
2
REx
(2,95(0.47) 54.17(1.37)
4
81.93(0.91) 54.97(1.71)
54.96(2.10)
16
81.60(0.69 69) 54.07(1.79)
100
80.94 78) 48.66(0.79)
81.48(0.67) 43.14(0.97)
1000
79.97(0.79) 42.24(0.74)
Table 7: Accuracy the continuous CMNIST with pt(t) as Sine function with different split number in (2, 4, 8, 16, 100, 500, 1000).
Env. Type Split Num Method HousePrice Insurance Fraud
IRMv1
72.68(0.36)
71.52(1.35)
73.46(1.41)
68.4((1.33)
10
72.97(1.88)
67.28(1.64)
25
74.42(2.10) 10)
54.25(1.75)
75.40(0.99)
52.09(2.05)
Discrete
REx
70.97(0.06)
74.42(1.32)
71.30(1.17)
74.42(0.87)
10
70.46(0.25)
73 (0())5)
(1.30)
72.96(0.24)
(20.99)))
72.90(0.46)
Continuous
CII. 60.95(6.59)
76.25(4.99)
Table 8: Comparison the real world dataset: HousePrice and Insurance. The metric is accuracy and standard deviations in parenthesis are calculated on three independent runs. Split Num stands for the number of domains we manually create by equally splitting the continuous domains.
N ABLATION STUDY
In this section, we tried different penalty set up in equation on the auto-scaling dataset to validate the robustness of CIL. The approximating functions (x)())),, ), (x),, y) implemented by 2-Layer MLPs. We evaluate the performance changes by increasing either the hidden dimension of the MLPs (Table or penalty weight À (Table 10)
Hidden Dimension ID Accuracy OOD Accuracy
32
84.37(5.95)
64.24(5.47)
64
84.17(2.58)
10.40(1.18)
128
81.25(1.65)
71.29(0.04)
256
85.57(2.67)
68. (8.12((225)
512
85.28(2.27)
68.77((.60
Table 9: ID and OOD accuracy on the auto-scaling dataset across different MLP setups for h (((())) o( (ar), y) in equation Standard deviation in brackets is calculated with 3 independent runs. Other settings are kept the same as Section O
Results. The and OOD accuracy of different MLP setup (hidden dimension) for h((())) (((()) y) under CIL on the auto-scaling dataset are shown in Table CIL performs the best when the hidden dimension 64, and the performance is stable with even higher dimensions. However, the testing accuracy drops to 64% when the dimension is reduced to 32, as the model is probably too simple to able to extract enough invariant features. But it is still better than ERM shown in Table 3
22

---

Published as a conference paper at ICLR 2024
Penalty Weight
Train
Test
100
86.91(1.31) (..6))
1000
84.47(2.57) 70.01(1.54)
10000
84.17(2.58) 70.40(1.18)
100000
83.95(2.73) 70.42(1.16)
1000000
83.95(2.73) 70.35(1.24)
Table 10: ID and OOD accuracy on the auto-scaling dataset across different penalty weights À in cquation Standard deviation in brackets is calculated with 3 independent runs. Other settings are kept the same as in Section O
Dataset
I.R
OLR Steps Penalty Step Penalty Weight
Logit (lincar)
0.001
0.001
1500
500
10000
Logit (sine)
0.001 0.001
1500
500
10000
CMNIST (linear)
0.001 0.001
1000
500
8000
CMNIST (sine)
0.001 0.001
1000
500
8000
HousePricc
0.001 0.01
1000
500
100000
Insurance
0.001 0.01
1500
500
10000
Auto-scaling
0.01
1000
10000
HildTime-Yeurrood 0.00001 0.001
1000
500
100
Table 11: The running setup for our CIL on each Dataset
Similarly, Table 10 reports the training and testing accuracy for different penalty weights A. They all outperform ERM and the discrete methods in Table 9 All accuracy is close to 70% when the penalty weight larger than or equal to 1000, which shows the model is robust to different penalty weights. The performance improvement lowers to 67% when the penalty is reduced to 100, but still outperforms ERM by 10%. The above experiments prove the robustness of our CIL, which can improve the model performance in any setup case.
O SETTINGS OF EXPERIMENTS
In this section, we provide the training and hyperparameter details for the experiments. All exper- iments are done server base on Alibaba Group Enterprise Linux Server release 7.2 (Paladin) system which has 2 GP100GL [Tesla P100 PCIc 16GB] GPU devices.
LR: learning rate of the classification model (x))) e.g. le-3.
OLR: learning rate of the penalty model (((())), g (x)) y), e.g. 0.001
Steps: total number of epochs for the training process, e.g. 1500
Penalty Step: number of epochs when to introduce penalty, e.g. 500
Penalty Weight: the invariance penalty weight, e.g. 1000
We show the parameter values used for each dataset in Table
P EXPERIMENT ON HEART DISEASE
We evaluate method the real-world Heart Disease dataset from Kaggle This dataset contains records related to the diagnosis of heart disease in patients. Each record consists of features including patient demographicx(e.gg g., age, gender), vital signs(e.g.. resting electrocardiogram, resting heart rate, maximum heart), symptoms(c.g, chest pain), and potential risk factors associated with heart
https: www. kaggle. com http//atattteaaaaaaaaaa am www.diaaboooooooooccoooooooooo e wwt./heattccce sease
23

---

Published as a conference paper at ICLR 2024
conditions. Our target is to determine the presence absence of heart disease in the patient. The Cholesterol value is taken as the continuous domains index, where the training dataset contains patients with Cholesterol value between (60.0, 220.0] and the testing dataset between (220.0, 421.0). The training dataset is equally split into discrete domains with 10 in each domain for existing methods dependent discrete domains. Results in Table [2 show that all existing methods are inferior to ERM in terms of in-distribution training performance. However, all methods except IIBNet achieve a higher accuracy than ERM on OOD testing. Our CIL performs the best across all methods, improving by about 2% compared to other methods.
Env. Type
Method
ID
OOD
None
ERM
88. (8.77)))55) 80.58(2.10)
GroupDRO
86.98/0.880 80) (6)
Discrete
IIBNct 81.64(0.55) 77.67(1.21)
IRMv1
87.24(2.07) 83.17(1.66
REx
87.76(1.87) 82,85(0.92)
Continuous
Diversify
87.24(1.21) 82.52(2.86 86)
EIIL. 87.36(0.48) (2.13()225)
HRM 88. 10(,91)) 81.92(1.72)
CIL.
86.23(1.25) 84.79(0.92)
Table 12: Comparison on the HeartDisease datasets
Q ON THE CORRELATION OF Y
is possible that the label is not independent of the domains. Taking the Heart Disease dataset as an example, can visualize the proportion of positive labels among subgroups of patients with different Cholesterol values. The distribution of Y is shown as Figure [] In this case, where the distribution of Y changes with the domain, we have observed that our method consistently outperforms ERM (Empirical Risk Minimization) and other competitive invariance learning methods. Additionally, we have explored another approach that involves re- weighting the samples to balance the Y ratio within each subgroup of the training data, where the subgroups are defined by Cholesterol intervals of 20.
For instance, let's consider a subgroup with a positive ratio 0.33. We reweight the samples from this subgroup by a factor of 0.5/0.33. After the reweighting process, the Y ratio in each subgroup becomes 0.5. We have found that combining this reweighting technique with our CII. method achieves slightly better ID performance but slightly worse OOD performance. Notably, the OOD performance of reweighted CIL is still consistently better than existing methods when we compare Table I3 with Table 12
0.6
Heart Dioease Ratio
0.5
0.4
0.3
0.2
0.1
0.0
2.1
70
150
190
270
350
Cholesterol
Figure 8: How the ratio changes with the Cholesterol value
24

---

Published as a conference paper at ICLR 2024
Method
ID
0OD
88.77(1.25)
80.58(2.10)
CIL
86.23(1.22 (1))5
84.79(0.92)
CiL(re-weight)
86.91(0.79)
84.12(1.36)
Table 13: Comparison of re-weighted CIL with vanilla CIL
R ADDITIONAL EXPERIMENT ON CONTINUOUS CMNIST
The results in Table in Section 4 shows large variance Continuous CMNIST. We conjecture that
it is due to that fact that invariance learning methods are prone to over-fittting in densely connected DNNs (Lin et al. 2022a Zhou et al. 2022). To further improve and stabilize the performance of CIL on Continuous CMNIST, we try to incorporate methods in (Rosenfeld et al. 2022. Kirichenko et al. 2022).
We have conducted additional experiments on the Continuous CMNIST dataset. For feature extraction, utilized fixed pretrained ResNet18 model the CMNIST images. In other words, used the extracted features as inputs for our CII. Our approach involved training only the linear layer on top of the fixed pre-trained features. This technique has been widely adopted in existing literature, where researchers have found that training just the last layer is sufficient because the pre-trained model already captures enough invariant and spurious features (Rosenfeld al. 2022] Kirichenko et al., 2022). This method can significantly alleviate the overfitting issue. The results presented in Table demonstrate that our CIL significantly benefits from being trained based the fixed feature extracted by the pre-trained features, they exhibit exceptional ID) and OOD performance with low variance. Notably, the baseline methods are all trained on the feature extracted by fixed pre-trained model and the other settings are the same with Section
Linear
Sine
Type
Split Num
ID
OOD
Split Num
ID
OOD
None
ERM
84.84(0.01)
10.60(0.08)
85.1700.01)
10:58(0.8)
IRMv1
75.68(0.77)
52.06(1.18)
2
76.20(0.15 15)
52.35(0.45)
REx
78.42(0.73)
39.30/4,00)
70.19(0.03
62.22/0.20)
Discrete
GroupDRO
84.73(0.01)
12.04/0277)
85.00(0.01)
12.28/0.15)
[1BNet
74.93(0.16)
41.60(0.63)
61.30(1.69)
45.73(1.58)
IRMv1
77.28(0.11)
46.95(0.48)
77.37(0.67)
(8.02((
Continuous
Diversify
REx
78.07(0.40)
83.29(0.19)
46.95(1.83)
30.92(1.10)
77.03(0.46)
(0:331
41.36(1.233
46 1e.11(1669
CIL
70.33(0.63)
62.15(1.37)
72.47(0.43)
67.80(0.40)
Table 14: Accuracy on Continuous CMNIST for Linear and Sine ps (t). The standard deviation in brackets is calculated with 5 independent runs. The Env. type "Discrete" means that we manually create by equally splitting the raw continuous domains. The environment type "Continuous" indicates using the original continuous domain index. "Split Num" stands for the number of domains manually create and we report the best performance among spilt (2, 4, 8, 16).
25


Reviews:
[{"review_id": "zFmi19HEXF", "reviewer": "Reviewer_Zzch", "summary": "This paper proposed the Continuous Invariance Learning(CIL) method that extends invariance learning from discrete categorical indexed domains to continuous domain in this paper.  Empirical results on both synthetic and real-world tasks demonstrated that CIL achieves improvements over baselines.", "questions": "1. The real-world datasets implemented in this paper, such as the HousepPrice dataset, take the built year as the continuous domain index. While in real life, adjacent years are separated by 1 year. Can this be regarded as a continuous domain? Same question with the Insurance Fraud datasets.\n\n2. In the paragraph *Existing Approximation Methods*, the authors let wt denote the optimal classifier for domain *t*. For the continuous domains, are there are continuous and infinite classifiers for each domain? How does such an assumption work in practice?\n\n3. In the paragraph ‘Formulation’, the authors say that since if $y ⊥ t|Φ(x)$, the loss achieved by $g(Φ(x),y)$ would be similar to the loss achieved by h(Φ(x)). If the loss of $g(Φ(x),y)$ is similar to the loss of  $h(Φ(x))$, then will $y ⊥ t|Φ(x)$ hold？\n\n4. In the paragraph ‘Empirical Verification’, the samples are simulated to distribute uniformly on the domain index $t$. According to figure 5, there is a step up of the ps(t). Please explain the reason.\n\n5. As for the experiments on real-world dataset, I am wondering if it is more convincing to expand the verification on some mechanically related data sets. Because we usually think of information such as speed and speed as continuous rather than discrete.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "investigating invariant learning for continuous domains is an interesting idea. This paper presents some innovative contributions and can extend the invariance learning among discrete domains to continuous domains, which has certain reference significance for other studies. As a theoretical rooted work, it first proves that the existing method fails in the continuous domain through theoretical derivation, and then proves the effectiveness of its own method. Also, the empirical studies verify the effectiveness of the work.", "weaknesses": "Although the contributions of this work is worth noting, it still has some limitations in terms of problem definition and presentation. First is about the problem setting: in my view, when we talk about *continuous* in machine learning, it will reflect some time-series issues, i.e., the continual learning or lifelong learning framework. However, it seems in this work the notion of *continuous* is related to *many* domains. It’s more like we have several intermediate domains between two discrete domains. I’m wondering whether the notion of *continuous* is accurate here.\nSecond is about the presentation of the paper. Some fonts of the tables and figures are to small, making them hard to read. This article can be further optimized in terms of visualization. Besides, there are also many grammar errors, which makes the paper less readable, e.g., under Eq. 1, “…in among…”, in the subsection of *theoretical analysis of REx*, ‘Eq. equation 1’", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JpkoKzJXzq", "reviewer": "Reviewer_MYYo", "summary": "This paper addresses the problem of generalization in continuous domains. Specifically, they aim to learn invariant features in settings characterized by continuous domain. Previous methods are restricted to work with the discrete indexed domains. \n\nThe authors demonstrate theoretically that when the number of domains is large and the number of samples from each domain is finite, existing methods can fail. Thus, discretizing continuous domains may not be a good solution. The authors then propose a min-max objective to learn invariant features by aligning domain distribution for each class of given features. To align the distributions, they use two domain predictors --- one uses invariant features, and the other uses label and invariant features. The goal is to learn features such that knowledge of labels does not increase the information about the domain. \n\nAuthors tested their method on several toy and real-world datasets with continuous domains. Their approach outperforms in all the cases.", "questions": "## Hyperparameters \n- What is the size of invariant features, i,e., $\\phi (x)$? Is the method sensitive to the size of these features?\n- What NN architecture & hyperparameters were used for training?\n\n \n## On linear scaling of penalty  \n- In the \"Remark on the settings and assumption\" paragraph, how does the penalty scale linearly with the number of environments when the spurious mask is used? We have assumed $\\mathcal R^t(w, \\phi) \\sim \\mathcal N (\\mathcal R(w, \\phi), \\delta_R)$. So, shouldn't the variance penalty be $\\delta_R$ when the spurious mask is used?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The arguments in the paper flow well, and the problem formulation is interesting. This paper could be a benchmark (for datasets & settings) for future work exploring generalization over continuous domains. \n- Authors provide theoretical evidence to explain the failure of existing methods. This complements the empirical results that demonstrate the same.\n- The authors demonstrate the superior performance of their method across various real-world and toy datasets", "weaknesses": "- In almost all the datasets considered in the paper, the ground truth labels are independent of domains. However, it is possible to have domains where these are correlated --- different amounts of correlation in different domains. Why was such a toy environment not considered? It would be interesting to see how the proposed method performs in more correlated settings.\n\n- The proposed approach relies on classes being discrete, whereas the prior method relies on the domain being discrete. This limitation should be highlighted in the paper. \n\n\n------- \nMinor Typo Issues:\n- Incorrect use of braces for citations (not an exhaustive list; please check the paper carefully): \n  - Sec 1. First para: use bracketed citations for He et al. (2016) etc,\n  - Sec 2. .. following (Arjovsky et al., 2019) ... \n- Sec 1, Para 3 last line, incorrect brackets\n- Sec 2: Invariance learning Para: ... to extracted Φ(x) ... -> to extract\n- Sec 2: ...variance of the losses in among domains -> ...variance of losses among domains.\n- Sec 3: Formulation para: Shouldn't soft regularization eqn have \"- t\" in the last two terms?\n- Tables 3 and 4: Some values are too high in ID columns", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IFhSUtv6kj", "reviewer": "Reviewer_dGY4", "summary": "This paper extends the invariant risk minimization (IRM) from discrete environment index to continuous environment index, in which there is no explicit environment partition and the domain variable is continuous. Authors first identify that some typical invariant representation learning algorithms, such as REx and IRMv1, may fail when the number of environments is large and there are only limited samples in each environment. Authors then propose a new regularization term by making $y$ is independent to $t$ (the domain index) given the extracted features $\\Phi(X)$, and uses a min-max scheme to approximate the degree of independence by training two regression functions to fit \n$p(t|\\Phi(X))$ and $p(t|\\Phi(X),y)$.", "questions": "Please see above weaknesses point 2 and minor point 2.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The movation of extending invariant representation learning from discrete environment to continuous environment is nice and useful. \n2. Applications on Alipay and Wilds-time demonstrate the practical usages of the new method.", "weaknesses": "Frankly speaking, I really like the motivation of this paper. However, I just feel the main strategy (especially the way to measure conditional independence) is not novel to me, which has been used in previous extensions of IRM, such as InvRat (Chang et al., 2020) and IIBNet (Li et al., 2022). Moreover, the proof to Proposition 2 seems largely rely on result of (Li & Liu, 2021), including their assumptions.\n\n1. The estimation on the degree of independence between y and t given $\\Phi(X)$ does not seem novel to me. In fact, for discrete environment index $e$, both InvRat (Chang et al., 2020) and IIBNet (Li et al., 2022) use the same strategy to measure\n$I(y;e|\\Phi(x))$ ($I$ is the conditional mutual information) by minimizing the maximum difference of two prediction losses: 1) use $\\Phi(X)$ to predict $y$; 2) use  $(\\Phi(X),e)$ to predict $y$.\n\nThe only difference is that this paper changes the role of $y$ and $e$ and estimate $I(e;y|\\Phi(x))$ by minimizing the maximum difference of two new prediction losses: 1) use $\\Phi(X)$ to predict $e$; 2) use  $(\\Phi(X),y)$ to predict $e$. \n\nGiven the fact that the conditional mutual information is symmetric, i.e., $I(y;e|\\Phi(X))=I(e;y|\\Phi(X))$, and it is equivalent to write this condition with either $p(y|\\Phi(X))=p(y|\\Phi(X),e)$ or $p(e|\\Phi(X))=p(e|\\Phi(X),y)$, it does not make a big difference or novelty to change the prediction target. Although I admit that in continuous regime, learning two functions to predict e makes more sense.\n\n2.  I would like to see some comparisons with environment partition approaches (see below), in which there is no explicit discrete environment index. In principle, those approaches should perform slighly better than manually environment splitting.\n\n[1] Creager, Elliot, Jörn-Henrik Jacobsen, and Richard Zemel. \"Environment inference for invariant learning.\" International Conference on Machine Learning. PMLR, 2021.\n\n[2] Liu, Jiashuo, et al. \"Heterogeneous risk minimization.\" International Conference on Machine Learning. PMLR, 2021.\n\nSome other points:\n1. Section I should be proof to Proposition 2, rather than Theorem 2?\n2. How to manually split environments for IRMv1, IIBNet, etc? By clustering? or did I miss something in the supplementary material?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SucjRpIiD8", "reviewer": "Reviewer_9c4M", "summary": "The paper proposed Continuous Invariance Learning (CIL), a robust optimization algorithm targeting the continuous-domain setting. Through theoretical analysis, the authors show that existent methods fail to identify the spurious features when $\\mathcal{T} \\ge O(\\sqrt{n})$, where $n$ is the sample size and $\\mathcal{T}$ is the number of domains, whereas CIL can handle even the $\\mathcal{T} \\in O(n)$ case. Empirical experiments on both synthetic and real-world datasets indicate that CIL achieves SOTA performance.", "questions": "What if I apply your method to a discrete-domain and discrete-label setting? How well does Appendix D.1's conclusion generalize to other cases? Does that depend on the relative cardinality of $\\mathcal{Y}$ and $\\mathcal{T}$?\n\nIn the introduction, you wrote *to regress over the continuous domain index $t$ using L1 or L2 loss*, but you only mention L2 loss in Section 3. What about the L1 loss?\n\nIs $O(\\sqrt{n})$ the slowest rate $|\\mathcal{T}|$ can grow with respect to $n$ for REx/IRM to fail with a non-zero probability?\n\nThe proof in Appendix F is unclear: (i) the notations are not defined; (ii) the assumption that *each domain only contains one sample* contradicts *assuming we have infinite samples in each domain* in the main text; (iii) last sentence is confusing: did you mean *if $|\\mathcal{E}| \\ge \\frac{\\sigma_R \\sqrt{n}}{\\Delta \\mathcal{G}^{-1}(1/4)}$ > 1/4, then REx can not identify the invariant feature with probability 1/4*?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "CIL is motivated by a \"duality\" between domain and label due to the condition independence, which is an insightful observation. As a result, the method can target either continuous-domain discrete-label or discrete-domain continuous-label settings (the paper focuses on the former). It enjoys a strong theoretical guarantee and works well in practice, as indicated by the comprehensive experiments in which it outperforms all baselines.", "weaknesses": "While the method is titled \"Continuous Invariance Learning\", the theoretical analysis assumes countably many domains. For example, you can never have a domain with an irrational index like $\\pi$ or $\\sqrt{2}$. The authors may want to extend the theorems to cover the truly continuous setting. It is easy to see that the method itself applies to the truly continuous setting though.\n\nAccording to Tables 3 and 4, CIL seems to perform poorly regarding in-domain prediction, with suboptimal accuracy and large variance. Moreover, I believe CIL has the potential to perform well in the discrete-domain continuous-label setting, but the authors didn't highlight or conduct experiments to demonstrate that.\n\nLesser issues:\n1. Both $\\mathcal{T}$ and $\\mathcal{E}$ are used as the notation for domain indices, which is a little confusing.\n2. In Table 5, you consider SWA as a baseline but didn't mention it in the main text.\n3. The tables need some cleanup. In Table 4, you are claiming the accuracy for continuous IRMv1 under ID Alipay autoscaling is 885.7%. There are other issues like extra paratheses, missing space around paratheses, a missing period in the Table 3 caption, and an uncapitalized *method* in the Table 5 header.\n4. In the paragraph starting with \"Existing Approximation Methods\": *Since is hard to validate* should be *Since it is hard to validate*.\n5. In the paragraph just above subsection 4.2.4, *All The remaining data* should be *All the remaining data*.\n6. In Appendix D.1: *Consider the following example, which has gained popularity* is repeated twice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)