Integrated Latent Heterogeneity and Invariance Learning in Kernel Space
Authors: Jiashuo Liu
Zheyuan Hu
Peng Cui
Bo Li
Zheyan Shen
Venue: NeurIPS 2021 Poster (2021)
Keywords: invariant learning
latent heterogeneity
spurious correlation
out-of-distribution generalization
robust learning

Abstract: The ability to generalize under distributional shifts is essential to reliable machine learning, while models optimized with empirical risk minimization usually fail on non-$i.i.d$ testing data. Recently, invariant learning methods for out-of-distribution (OOD) generalization propose to find causally invariant relationships with multi-environments. However, modern datasets are frequently multi-sourced without explicit source labels, rendering many invariant learning methods inapplicable. In this paper, we propose Kernelized Heterogeneous Risk Minimization (KerHRM) algorithm, which achieves both the latent heterogeneity exploration and invariant learning in kernel space, and then gives feedback to the original neural network by appointing invariant gradient direction. We theoretically justify our algorithm and empirically validate the effectiveness of our algorithm with extensive experiments.

Kernelized Heterogeneous Risk Minimization
Jiashuo Liu Zheyuan Hu²* Peng Cuil Bo Li and Zheyan Shen' Department of Computer Science & Technology. Tsinghua University, Beijing, China Department of Computer Science, National University of Singapore, Singapore School of Economics and Management, Tsinghua University, Beijing, China Muujiashu77,2zyuu200 gmmil.com, cuip@ cuip@ssnnghu..cdu.. libo @ sem. wwwgssm..ttttsds.....uuu shenzy 1 mails. www.mmml.ssiinsuuaa...mmmm tsinghua.cdu.cn
Abstract
The ability to generalize under distributional shifts essential reliable machine learning, while models optimized with empirical risk minimization usually fail on non-i.i.d testing data. Recently, invariant learning methods for -uito--distribution (OOD) generalization propose to find causally invariant relationships with multi- environments. However, modern datasets are frequently multi-sourcess without explicit source labels, rendering many invariant learning methods inapplicable. In this paper, we propose Kernelized Heterogeneous Risk Minimization (KerHRM) algorithm, which achieves both the latent heterogeneity exploration and invariant learning kernel space, and then gives feedhack appointing invariant gradient direction. We theoretically justify our algorithm and empirically validate the effectiveness of our algorithm with extensive experiments.
original neural network by
Introduction
Traditional machine learning algorithms which optimize the empirical risk often suffer from poor generalization performance under distributional shifts caused by latent heterogeneity or selection biases that widely exist in real-world data 9 220. How to guarantee machine learning algorithm with good generalization ability on data drawn out-of-distribution is of paramount significance, especially in high-stake applications such financial analysis, criminal justice and medical diagnosis, etc. 13 18] which known as the Dut-of-ddistriuttionOOOD generalization problem
To ensure the OOD generalization ability, invariant learning methods assume the existence of the causally invariant correlations and exploit them through given environments, which makes their performances heavily dependent the quality of environments. Further, the requirements for the environment labels are too strict to meet with, since real-world datasets are frequently assembled by merging data from multiple sources without explicit source labels. Recently, several works 5 15) to relax such restrictions have been proposed. Creager et al. [5] directly infer the environments according to given biased model first and then performs invariant learning. But the two stages cannot be jointly optimized and the quality of inferred environments depends heavily on the pre-provided biased model. Further, for complicated data, using invariant representation for environment inference is harmful, since the environment-speccfee features are gradually discarded, causing the extinction of latent heterogeneity and rendering data from different latent environments undistinguishable. Liu er al. [15) design a mechanism where two interactive modules for environment inference and invariant learning respectively can promote each other. However, it can only deal with scenarios where invariant and variant features are decomposed on raw feature level, and will break down when the decomposition can only be performed representation space(e.g.. image data).
"Equal Contributions CCoresponding Author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

This paper focuses on the integration of latent heterogeneity exploration and invariant learning on representation level. In order to incorporate representation learning with theoretical guarantees. we introduce Neural Tangent Kernel(NTK[I] [TO) into our algorithm. According to NTK theory 10]. training the neural network is equivalent to linear regression using Neural Tangent Fea)ures(NTF), which converts non-linear neural networks into linear regression in NTF space and makes the integration possible. Based on this, our Kernelized Heterogeneous Risk Minimization (KeHHRM) algorithm is proposed, which synchronously optimizes the latent heterogeneity exploration module Me and invariance learning module Mp in NTF space. Specifically, we propose our novel Invariant Gradient Descent(IGD) for Mp. which performs invariant learning in NTF space and then feeds back to neural networks with appointed invariant gradient direction. For MM. we construct an orthogonal ceooommnnt-speccfiie features and to further accelerate the
heterogeneity-awee aware kernel to capture the heterogeneity exploration. Theoretically, we demonstrate our heterogeneity exploration algorithm for Mc with rate-distortion theory and justify the orthogonality property of the built kemel, which jointly can illustrate the mutual promotion between the two modules. Empirically, experiments on both synthetic and real-world data validate the superiority of KerHRM in terms of good -ittr---iitribttion generalization performance.
2 Preliminaries
Following 13.. we consider data D = (ee jeEsupp(E..) with different sources data D (X", Y*) collected from multiple training environments Etr- Here environment labels are unavailable as in most of the real applications. E,r is a random variable on indices of training environments and Pe is the distribution of data and label in environment The goal of this work is to find a predictor f() : X y with good out-of-distribution generalization performance, which is formalized as:
arg min max L(fle) f eEsupp(E)
where [(fle) = ETX(X" Y=] represents the risk of predictor f on environment e, and ((. -): Y x + R+ the loss function. Note that & is the random variable on indices of all possible environments such that supp(Err) (Err) C supp(E). Usually, for all e E supp(E) \ supp(E>>), the data and label distribution p P=(X, Y) can be quite different from that of training environments Eyr- Therefore. the problem in equation [ is referred to as problem [.] Since impossible to characterize the latent environments & without any prior knowledge or structural assumptions, the invariance assumption is proposed for invariant learning: Assumption 2.1. There exists random variable (X) such that the following properties hold: a. Invariance property: for all e, e' E supp(E), we have P (X))) = pe' (X)) holds. h Sufficiency property: Y = f(V3) +e, e e
(OOD) Generalization
This assumption indicates invariance and sufficiency for predicting the target Y using V/ which is known as invariant representations with stable relationships with Y across E. To acquire such v a branch of works 41 15) proposes to find the maximal invariant Predictor(MIP)) of an invariance set, which are defined as follows:
Definition 2.1. The invariance set I with respect & is defined as: (()s(X)) (Vs(X): (X))] = (X)))((
Ie (V=(Xx)
where H-) is the Shannon entropy of a random variable. The corresponding maximal invariant predictor (MIP) of Ie is defined as S arg max4ez, I(Y; b), where II(:-)) measures Shannon mutual information between two random variables.
Firstly, we propose that using the maximal invariant predictor S of Ig can guarantee OOD optimality in Theorem 2.1 The formal statement is similar to 15] and can be found in Appendix A.3.
Theorem 2.1. (Optimality Guarantee, informal) For predictor (X(X) satisfying Assumption 2.1
V is the maximal invariant predictor with respect to E and the solution to OOD problem in equation 7 is Ey [ ] = arg miny supeesuppte) E[[(f)]e]
on finding MIP solutions rely on the availability of data from multiple training environments Exr, which is hard to meet with in practice. Further, their validity is highly determined by the given Etr. Since Ig C Ie,,, : the invariance regularized by & is often too large and the learned MIP may contain variant components and fails to generalize well. Based on this,
However, recent works

---

Step 0: Initialization.
Step 3: Heterogenelty Exploration.
MLP model fu() with Initia! We. Neural Tangent Feature ( =P P.,,,(00)
Neural Tangent Feature Space
Generate E with clustering korad R. E (0()),
[0]
(t) learn Mtp
in followingh,
k
M
M
Clustering Kernel xy) ((x)"*((x))
See Soction 3.3
(x.Y%
Step 1: Invariant Prediction.
Step 2: Variont Component Decomposition.
Auep
inseriant model parameters @ Fleam NTF space, and neural network fw() with o w
Update clustering kernel < (111) with ou in NTF space.
(E+1)
Kc
W inv
Skop
See Section 3.2
See Section 3.1
Figure 1: The framework for KerHRM. The middle block diagram shows the overall flow of the algorithm, which consists of two modules named heterogeneity exploration module M, and invariant prediction module Mp. The whole algorithm runs iteratively between M. and My, where one
iteration consists of three steps, which we illustrate in section 3.1 3.2 and 3.3 respectively.
Heterogeneous Risk MinimizaaiionHHRM[15 proposes to generate environments &,, with minimal [Ze... and to conduct invariant prediction with learned Etr. However, the proposed HRM can only deal with simple scenarios where X = [Vr3, V on raw feature level (V''s are invariant features and V variant ones), and will break down where X h(V' v (h(-,-) is an unknown transformation function), since the decomposition can only be performed in representation space. In this work, we focus the integration of latent heterogeneity exploration and invariant learning in general scenarios where invariant features are latent in X, which can be easily fulfilled in real applications.
Problem 1. (Problem Setting) Assume that X = h S. VV) E IR, where V satisfies Assumption 2.1 hu() is unknown transformation function and V I Vi (following functional representation lemma/Z]) given heterogeneous dataset D = / Esupp(Euuten)) without environment labels, the task is to generate environments Etearn with minimal [Eraaral and meanwhile learn invariant models.
3 Method
Remark. Following the analysis in section to generate environments Eteorn with minimal [T is equivalent generate environments with as varying PYY) as possible, to exclude variant parts Vy from the invariant set Igearn
In spite of such insight, the latent V Vi make it impossible to directly generate Etearn. In this work, we propose our Kernelized Heterogeneous Risk Minimization (KerHRM) algorithm with two interactive modules, the frontend Me for heterogeneity exploration and backend M, for invariant prediction. Specifically, given pooled data, the algorithm starts with the heterogeneity exploration module Ma with learned heterogeneity- awwee kernel K2 to generate Elearn- The learned environments are used by MM to produce invariant direction O.n in Neural Tangent Feature(NTF) space that captures the invariant components s. and then 0:n is used to guide the gradient descent of neural networks. After that, we update the kernel to orthogonalize with the invariant direction 0ive so as to better capture the variant components v and realize the mutual promotion between Me and Mp iteratively. The whole framework is jointly optimized, so that the mutual promotion between heterogeneity exploration and invariant learning can be fully leveraged. For smoothness we
Algorithm Kernelized Heterogeneous Risk Minimization (KerHRM) Algorithm
Input: Heterogeneous dataset = {D = (X YY))eee,
Initialization: MLP model (-) with initialized Wa, Neural Tangent Feature ((X)= = Verwu (X (X)(fixed in
following), clustering kernel initialized as Ric (x1,72) = x x2 for t= to T do 1. Generate Elerrn with clustering kernel Ke. (t-1); Elarn Mc((((X), Y), (()
e(()
Learn invariant model parameters gft). with Elearn in NTF space: p(t) (g(t)
3. Feedback to Neural Network (:)()) with gft) wine arg min, C(w; X) + Reg(w, )(0)
4. Update the clustering kernel (0) with g(0) Re
Orthogonal Transform( (t- g
end for

---

begin with the invariant prediction step to illustrate our algorithm, and the flow of whole algorithm is shown in figure T
3.1
Mpi Invariant Gradient Descent with Elearn (Step 1)
For our invariant learning module Mp. we propose Invariant Gradient Descent (IGD) algorithm. Taking the learned environments Eiearn input, IGD firstly performs invariant learning in Neural Tangent Feature (NTF[1O) space to obtain the invariant direction 8ine. and then guides the whole neural network (-) with ine to learn the invariant model(neural network)'s parameters Wine-
Neural Tangent Feature Space The NTK theory [T]] shows that training the neural network is equivalent to linear regression using non-linear NTFs Φ(x), as in equation 4 For each data point
I E Rd, where d is the feature dimension, the corresponding feature is given by δ(x) = fwx( e RP, where p is the number of neural network's ' parameters. Firstly, we would like to dissect the feature components within d(x) by decomposing the invariant and variant components hidden in (x)). Therefore, we propose to perform Singular Value Decomposition (SVD) on the NTF matrix:
4(X) ~
where p >>>> n2 2 k
(3)
ReXP
Baxk
B
R
Intuitively, in equation 3 each row j.: of V represents the j-th feature component of IR and we take such different feature components with the top k largest singular values to represent the data, and the rationality of low rank decomposition is guaranteed theoretically [2]] [6] and empirically [2]. Since SVD ensures every feature components orthogonal, the neural tangent feature of the i-th data
point can be decomposed into d(x;)²² ~ Σÿ= ULJ Sj.s ViJ where Ut,- Sjyj denotes the strength of the j-th feature component in the i-th data. However, since neural networks have millions of parameters, the high dimension prevents from learning dircetly on high dimensional NTFs (XX..
Therefore, we rewrite the initial formulation of linear regression into:
(x) x fwo (X) ((X)" (w wo) = fwo (X) +SVV (w we)
= f= (X) v(X) (v"(w- wn)) fwm (X) (XX)
(5)
where we let 0 = VT (w- E R²* which reflects how the model parameter w utilizes the k feature components. Since V is orthogonal, fitting w - Wo with features ((X) is equivalent to fitting using reduced NTFs (X). In this way, convert the original high-ddmensional regression problem into the low-dimensional one in equation since in wide neural networks, we have p >> n 2 k.
Invariant Learning with Reduced NTFs (XX) We could perform invariant learning on reduced NTFs L (X) in linear space. In this work, we adopt the invariant regularizer proposed in [ to learn 0 = V (w- Wo) due to its optimality guarantees, and the objective function is:
Oine arg min
(0, + Vars (VoL")
(6)
eEficarn
Guide Neural Network with invariant direction Pine With the learned fin... it remains to feed back to the neural network's parameters W. Since for neural networks with millions of parameters whose x 108, it is difficult to directly obtain w as w = wo + V0 Therefore, we design a loss function to approximate the projection (w-wn/ Co--w//VVl (...). Note that (X) = fm (X)+USVV
Wo) = fmo (X) + U S8... we have
S S (fw(X) - fwo (X)) P.c
(7)
Therefore, we can ensure the updated parameters w satisfy that S (-11""(((()) - Swn (X)) E R is parallel to fine. which leads to the following loss function:
Winv argmin L(w; X,Y) + (1 1(0mn, S "0"
(X)))
(8)
[ 1UT SUTTUU fwm (X))
where C(w; X, Y) is the empirical prediction loss over training data and the second term is to force the invariance property of the neural network.
3.2 Variant Component Decomposition with O.n (Step 2)
The core of our KerHRM is the mutual promotion of the heterogeneity exploration module Me and the invariant learning module Mp. From our insight, we should leverage the variant components By

---

to exploit the latent heterogeneity. Therefore, with the better invariant direction 8ive learned by Mp that captures the invariant components in data, remains to capture better variant components V so as to further accelerate the heterogeneity exploration procedure, for which we design a clustering kernel KR on the reduce NTF space of TR with the help of inn learned section 3.1 Recall the NTF decomposition in equation 3] the initial similarity of two data points 2; and 23 can be decomposed as:
10) (0),;)) = o(x,)) p(a;) = (U,S, U,5)
(9)
With the invariant direction nents used by 0 via
0 inn learned by MP in iteration t, we can wipe out the invariant compo-
(zs) + U,S - (U.S, o!..) /fft//1 /100
(10)
which gives a new heterogeneity-avare kernel that better captures the variant components V as (++1) Ric (Iz; = v (+ (a.) ( (x)).
3.3 MM: Heterogeneity exploration with Hic (Step 3)
partition Etearn for invariant prediction module Mp. and we implement it as clustering algorithm with kernel regression given the heterogenneity-awan variant components in data. Following the analysis above, only the variant components Vi should be leveraged to identify the latent heterogeneity, and therefore use the kernel K. well Wy (X) learned in section 3.2 to capture the different relationship between Vy and Y, for which we use v as the clustering centre. Specifically, we assume the j-th cluster centre Peş (Y (X)) to be Gaussian around f(O,; Vv as:
Me takes one heterogeneous dataset as input, and outputs a learned
Zz V (x;))"y y V that captures the
(V)VV(X), = Pe, (Y (X)) = (V2TT)- exp( - (();)))((X)))) (x)))))
(11)
For the given N EcesupptEncenn) |D| data points D (vv y: the empirical distri-
bution can be modeled as PN w Ei= Sep(z)). Under this setting, we propose one convex clustering algorithm, which aims at finding mixture distribution in distribution set Q defined as:
QQ-
Σ (Vv Y). e Ax}
(12)
SE(K]
to fit the empirical data best. Therefore, the original objective function and the simplified one are:
min DKL min
L. =
Σ M
log
Σ [E(K]
(13)
QER
N
Note that our clustering algorithm differs from others since the cluster centres are learned models parameterized with Ô. As for optimization, we use EM algorithm to optimize the centre parameters O and the mixture weights g iteratively. Specifically, when optimizing the cluster centre model f(O):.), we use kernel regression with n)(:s)) avoid computing V (X) and allow large For generating the learned environments Etearn. we assign i-th point to j-th cluster with probability P,, = hy (vvv (vv((z)) 9t)/ ElEK/] (2:()v(z)))00 9:).
4 Theoretical Analysis
In this section, we provide theoretical justifications of the mutual promotion between M, and Mp- Since our algorithm does not violate the theoretical analysis in [T] and [10] which proves that better Elearn from Me benefits the MIP learned by Mp. to finish the mutual promotion, we only need to justify that better One from Mp benefits the learning of Elearn in Mc.
1. Using V benefits the clustering. Firstly, we introduce Lemma from using V benefits the clustering in terms of larger between-cluster distance.
15 to show that
Lemma 4.1. For Gj E supp(Eratent). ), assume that X satisfying Assumption 2.1 then under reason-
able assumption (5)) we have Dxi(Pe: (Y)xX) Ps: (Y]X)) & Dea V. Ps: (Y\V)) (T))), )).
Then similar to 14 we use the rate-distortion theory to demonstrate why larger DKL. between cluster centres benefits our convex clustering well as the quality of Eiearn.
5

---

Theorem 4.1. (iate-Distortion) For the proposed convex clustering algorithm, we have:
min DKL (xx(Px]1Q) = mi: + (r, (z.), ys. θ3) + Const QEQ
(14)
where rij (m;), is a discrete random variable over the space {1, 21,2,.... N} x (1,2....,K) which denotes the probability of i-th data point belonging to j-th cluster, I.J are the marginal distribution of random variable rsj respectively, d( (I,), Vs. 2,) (fe, (fhv (2:)) yA and T(;;-) the Shannon mutual information. Note that the optimal can be obtained by the optimal Q and therefore we only minimize the r.h.s with respect to O
Actually d models the conditional distribution v. If in the underlying distribution of the empirical data P(Y v differs a lot between different clusters, the optimizer will put more efforts in optimizing Er,r (. (1:(()()))))))))))))))))) Ve: 0 to avoid inducing larger error, resulting in smaller efforts put on optimization of [[Z; J) and a relatively larger J). This means data sample points / have larger mutual information with cluster index J. thus the clustering prone be more accurate.
Orthogonality Property: Better Oiee for better Vy. Firstly, we prove the orthogonality property between (equuatut 6) and parameters θ of clustering centres f.)
Theorem 4.2. (Orthogonality Property) Denote the data matrix of j-th environment X and Vi (X))),, then for each Ø₁ [K] (( - yTy, have Span(@) S Ker(Oine) and Span( (V) S. Ker(Oine)), where Span denotes the column space and Ker the null space.
Theorem 4.2 justifies that the parameter space for clustering model fe(-) well as the space of learned variant components Vy is orthogonal to the invariant direction Oines which indicates that better invariant direction Osn regulates better variant components Vy and therefore better heterogeneity. Taking (1) and (2) together, we conclude that better results(O of M, promotes the latent heterogeneity exploration in Me because of larger between-cluster distance. Finally, we use linear but general setting for further clarification.
Example. Assume that data points from environments e E & are generated as follows:
x= Y + 3.VV) N E I
(15)
where Y = +1 with equal probability, the coefficient Be varies across environment e, V TR. is the invariant feature and following Donriingfunction functional representation lemma [Z) VY is the variant feature with Vy 1 TV's E Rd and its relationship with the target Y relies on the environment-specipe Be-
Remark. example 4 when Mp achieves optimal, we have Onne = Vs, which is the mid vertical hyperplane of the two Gaussian distribution. Then following equation 10 have By X (X" im 0 = Y Be V which directly shows that in the next iteration, M, uses solely variant components V in X to learn environments Etearn with diverse X(Y|X) PYY ). which by lemma 4.1 and theorem 4.1 gives the best clustering results.
5 Experiments
In this section, we validate the effectiveness of our method on synthetic data and real-world data.
Baselines
We compare our proposed KerHRM with the following methods:
Empirical Risk Minim; ming Er, X,
Distributionally Robust Optimization(DRO 6 mine sup pup
Environment Inference for Invariant Learning 55)
min max Σ 1 Σ (e(e)((w@((x)) N.
((())(()))()))) wi)l2 (16)
wi==10 N.
EEE
Heterogeneous Risk Minimization(HRM [5]
Invariant Risk Minimization(IRM ( with environment E,, labels:
min Σ L + (w 2)
(17)
3

---

Regression with Selection Bias We choose one typical m of DRO as DRO is another main branch of methods for OOD generalization problem of the same setting with us (no environment labels). And HRM and EIIL are another methods for inferring environments for invariant learning without environment labels. We choose IRM as another baseline for its fame in invariant learning. but note that IRM is based on multiple training environments and we provide E,, labels for it, while the others do not need. Further, for ablation study, we run KerHRM for only one iteration without the feedback loop and denote it as Static KerHRM(KeHHRM)
For all experiments, we use a two-layer MLP with 1024 hidden units.
Evaluation Metrics To evaluate the prediction performance, for task with only one testing environ- ment, we simply use the prediction accuracy of the testing environment. While for tasks with multiple environments, we introduce Mean Error defined as Mean Error
|Ereas| EeeErare 2" Std Error
Mean Mean_Errortt Error)?, which are mean and standard deviation eπrоr across Etest. And we use the average mean square error for L
defined Std Error V
|Eeer| Eeeeco.. (C
5.1 Synthetic Data
Classification with Spurious Correlation
Following [19]. we induce the spurious correlation between the label Y E {+1, -1} and a spurious attribute A E {+1, -1}. Specifically, each enviromment is characterized by its bias rate E (0, 1]. where the bias rate represents that for 100 * r% data, A = Y, and for the other 100 * (1 - r)% data, A = -Y. Intuitively, measures the strength and direction of the spurious correlation between the label Y and spurious attribute A, where larger |r- - 0.5] signifies higher spurious correlation between Y and A, and sign(r 0.5) represents the direction of such spurious correlation, since there is no spurious correlation when 0.5. We assume X H(S, V Rdd where S R is the invariant feature generated from label Y and V the variant feature generated from spurious attribute A:
S|Y NYY1, Ta), VIA ~ (A1,,
(18)
and H e 2d is an random orthogonal matrix to scramble the invariant and variant component, which makes it more practical. Typically, we set o 2 of to let the model more prone to use spurious V since V is more informative.
In training, we set d and generate 2000 data points, where 50% points are from environment el with number K = 2. In testing, we generate 1000 data points from environment 63 with ra = 0.1 to induce distributional shifts from training. In this experiments, we vary the bias rate 72 of environment 62 and the scrambled matrix which can be an orthogonal or identity matrix (as done in ), and results after 10 runs are reported in Table I
0.9 and the other from environment 62 with 72. For our method, we set the cluster
From the results, we have the following observations and analysis: ERM suffers from the distribu- tional shifts between training and testing, which yields the worst performance in testing. DRO can only provide slight resistance to distributional shifts, which we think is due to the oves-pess problem(8) 8). EIIL achieves the best training performance but also performs poorly in testing. HRM outperforms the above three baselines, but its testing accuracy is just around the random guess(0.50) which is due to the disturbance of the simple raw feature setting in 113. IRM performs better when the heterogeneity between training environments is large(r2 is small), which verifies our analysis in section that the performance of invariant learning methods highly depends the quality of the given Egy. Compared to all baselines, our KerHRM performs the best with respect to highest testing accuracy and lowest (Train Acc Test AAc), showing its superiority to IRM and original HRM.
Further, we also empirically analyze the sensitivity to the choice of cluster number K of our KerHRM. We set = 0.80 and test the performance with K = {2, 3, 4, 5) respectively. Results compared with IRM are shown in Table [ From the results, we can see that the cluster number of our methods does not need to be the ground truth number(ground truth is 2) and our KerHRM is not sensitive to the choice of cluster number K. Intuitively, we only need the learned environments to reflect the variance of relationships between P(Y Nr but do not require the environments to be ground truth. However, we notice that when K is far away from the proper one, the convergence of clustering algorithm is much slower.
In this setting, we induce the spurious correlation between the label Y and spurious attributes V through selection bias mechanism, which is similar to that in [2]. We assume X = HSS VJT
7

---

Table I: Results in classification simulation experiments of different methods with varying bias rate 12, and scrambled matrix H, and each result is averaged over ten times runs.
0.70
0.75
0.80
Methods
Thain Aoe
Test Acc
Train Arc
Test Ace
Train Acc
Test Ane
ERM
0.850
0.400
0.862
0.325
0.875
0.254
DRO
0.857
0.473
0.870
0.432
0.883
0.395
EIIL
0.927
0.523
0.925
0,946
0.463
HRM
0.836
0.832
0.519
0.852
0.488
IRM(with e,, label)
0.836
0.606
0.853
0,544
0.877
0.401
KerHRM
0.764
0.671
0.782
0.632
0.663
0.619
KerHRM
0.759
0.724
0.760
0.686
0.741
0.693
Table 2: Ablation study on the cluster number K.
Each result is averaged over ten times runs.
KerlIRM
KerlIRM
KerlIRM
KerlRRM
IRM
HRM
(K = 2)
(K (K=3)
(K= 4)
(K =
Train Acc
0,877
0.852
0.741
0.758
0.756
0.753
fest_Ace
0.401
0.488
0.693
0.687
0.698
0.668
R2 and Y = ((S) + e, where (() is non-linear function such that P(Y|S) remains invariant across environments while P(Y(V) changes arbitrarily. For simplicity, we select data (Ii, Mi) with probability P(z,, 21) according to certain variable V,2 E V:
P(r., Ma) = |r| --1l:
(19)
where |r| > 1. Intuitively, eventually controls the strengths and direction of the spurious correlation between V/, and Yi.e. if > 0, data point whose Vú, close to its y is more probably to be selected.). The larger value of |r| means the stronger spurious correlation between V,, and Y, and means positive correlation and vice versa. Therefore, here we use r to define different environments.
In training, we generate 1000 points from environment e1 with predefined and 100 points from 62 with r -1.1. In testing, to simulate distributional shifts, we generate data points for 6 environments with [-9. -2.,.... 119).. We compare our KerHRM with ERM, DRO, EIIL and IRM. We conduct experiments with different settings on r and the scrambled matrix H.
From the results in 3 we have the following analysis: ERM, DRO and EIIL performs poor with respect high average and stability error, which is similar to that classification experiments(Table D. The results of HRM are quite different in two scenarios, where Scenario 1 corresponds to the simple raw feature setting(H = /) in [5] but Scenario violates such simple setting with random orthogonal H and greatly harms HRM. Compared to all baselines, our KerHRM achieves lowest average error 5/6 settings, and its superiority is especially obvious our more general setting(Scenario 2).
Colored MNIST
To further validate our method's capacity under general settings, we use the colored MNIST dataset, where data X are high-dimensional non-linear transformation from invariant features(digits Y) and variant features(color C). Following [ we build synthetic binary classification task, where each image is colored either red or green in a way that strongly and spuriously correlates with the class label Y. Firstly, binary label Y is assigned to each images according to its digits: Y = 0 for digits 0~4 and Y for digits 5m Secondly, we sample the color id C by flipping with probability and therefore forms environments, where = 0.1 for the first training environment, the second training envirouments and e = 0.9 for the testing environment. Thirdly, we induce noisy labels by randomly flipping the label Y with probability 0.2.
0.2 for
We randomly sample 2500 images for each environments, and the two training environments are mixed without environment label E,r for ERM. DRO, EIIL, HRM and HRM, while for IRM, the Esr labels are provided. For IRM, we sample 1000 data from the two training environments respectively and select the hyper-parameee parameters which maximize the minimum accuracy of two validation environments. Note that we have no access to the testing environment while training, therefore we cannot resort to testing data to select the best one, which is more reasonable and different from that in [. For the others, since we have no access to labels, we simply pool the 2000 data points for validation. The results are shown in Table 4 where Perfect Inv. Model represents the oracle
8

---

Table 3: Results in selection bias simulation experiments of different methods with varying selection
bias r, and scrambled matrix H, and each result is averaged over ten times runs.
Scenario 1: Non-Scrambled Setting (H I, varying
L55
1.9
2.3
Methods
Mean Frror
Std Error
Viean Error
Std Eiror
Mean Error
Str. Error
ERM
5.056
0.223
5.442
0.204
5,503
0.234
DRO
4.571
0.205
4.908
0,180
5.081
0.209
EIIL
5.006
0.211
5.252
0.172
5.428
0.205
HRM
3.625
0.057
3.901
0.050
4.017
0.082
IRM(with & lahel)
3.873
0.176
4.536
0.172
4.509
0.194
KerHRM
4.384
0.191
3.989
0.195
3,527
0.178
KerHRM
4.112
0.182
3.659
0.186
3.409
0.174
Scenario 2: Scrambled Setting (random orthogonal II, varying Y)
y=
1.5
r= 1.9
2.3
Methods
Mean Error
Std Error
Mean Error
Std Error
Mean Error
Std Error
HRM
5.059
0.229
5.285
0.207
5.478
0.211
DRO
4.494
0.212
4.717
0.175
4.978
0.207
EIIL
4,945
0.215
5.207
0.187
5,294
0,220
HRM
4.397
0.096
4.801
0.142
4.721
0.096
IRM(with Err lahel)
4.269
0.218
4.477
0.174
4.392
0.178
KerHRM
4,379
0.205
3.543
0.169
3.571
0.164
KerHRM
4.122
0.195
3.375
0.163
3,473
0.160
results that can be achieved under this setting. We run each method for 5 times and report the average accuracy, and since the variance of all methods are relatively small, we omit it in the table.
Table 4: Colored MNIST results. The first row indicates whether each method needs the environment label, The Perfect Inv. Model represents the oracle results that be achieved. The Generalization Gap is defined as (Test Accuracy Train Accuracy).
Perfect Inv. Model
Method
ERM
DRO
EMIL
HRM
IRM
KccHRM
KccHRM
Need E,, Label?
X
X
X
X
V
X
X
Train Accuracy
0.845
0.644
0.777
0.835
0.766
0.802
0.654
0,800
Tost Accuracy
0.6
0.419
0.542
0.282
0.468
0.29%
0.648
0.800
Generalization Gap
0.739
0.223
00.23
-0.533
-0.298
-0.506
-0.006
From the results, our KerHRM generalize the HRM to much more complicated data and consistently achieves the best performances. KerHRM even outperforms IRM significantly in an unfair setting where we provide perfect environment labels for IRM, which shows the limitation of manually labeled environments. Further, to best show the mutual promotion between Me. and MMp we plot the training and testing accuracy as well as the KL-divergence DEL of P(Y between the learned Elearn over iterations figure From figure 2 we firstly validate the mutual promotion between M, and MM since DRI. and testing accuracy escalate synchronously over iterations. Secondly, figure Zcoorespondss to our analysis section that the performance of invariant learning method highly correlated to the heterogeneity of &xr, which sheds lights to the importance of how to leverage the intrinsic heterogeneity in training data for invariant learning.
5.2 Real-world Data
In this experiment, we test our method on a real-world regression dataset (Kaggle) of house sales prices from King County, US where the target variable is the transaction price of the house and each sample contains 17 predictive variables, such as the built year, number of bedrooms, and square footage of home, etc. Since it is fairly reasonable to assume the relationships between predictive variables and the target vary along the time (for example, the pricing mode may change along the time), there exist distributional shifts in the price-prediction task with respect the build year of houses. Specifically, the houses in this dataset were built between 1900 ~ 2015, and we divide the whole dataset into 6 periods, where each contains a time span of two decades. Notice that the later periods have larger distributional shifts. We train all methods on the first period where built year [1900, 1920) and test on the other 5 periods and report the average results over 10 runs in figure For IRM, we further divide the period I into two decades for the E,, provided.

---

10-
10
165
Train
De
ERM
KL-divergence between Essur
Test
EIIL
08-
Prediction ErroF
IRM
KorHIL
KorHiL
0.
04
Whole-iteration
Environment
Figure 2: Results for the Colored MNIST task. We plot the training and testing accuracy, as well as the KL-divergence between learned Eir.
Figure 3: Results for the real-word regression task. We train all methods 61 and on the others, and report the average results over 10 runs.
Analysis The testing enrors of ERM and DRO increase sharply across environments, indicating the existence of the distributional shifts between environments. IRM performs better than ERM and DRO, which shows the usefulness of environment labels for OOD generalization and the possibility of learning invariant predictor from multiple environments. The proposed KerHRM outperforms EIII. and HRM, which validates its superiority of heterogeneity exploration. KerHRM even outperforms IRM, which indicates the limitation of manually labeled environments in invariant learning and the necessity of latent heterogencity exploration.
6 Limitations
Although the proposed KerHRM a competitive method, has several limitations. Firstly, since in Me. we take the model parameters cluster centres, the striet convergence guarantee for clustering algorithm Me is quite hard to analyze. And empirically, we find when the pre-defined cluster number K is far away from the ground-truth, the convergence of Me will become quite slow. Further, such restriction also affects the analysis of the mutual promotion between M. and Mp. which we can only empirically provide some verification. Besides, although Tangent Kernel to deal with data beyond raw feature level, how to deal with more complicated data still remains unsolved. Also, how reduce the randomness induced by Neural Tangent Kernel and how to incorporate deep learning with the mutual promotion between the two modules needs further investigation, and we left for future work.
incorporate Neural
Conclusion
In this paper, propose the KerHRM algorithm for the OOD generalization problem, which achieves both the latent heterogeneity exploration and invariant prediction. From our thcoretical and empirical analysis, we find that the heterogeneity of environments plays key role in invariant learning, which consistent with some recent analysis [7] and opens a new line of research for OOD generalization problem. Our code is available at http:/ggiihbboomomsSsSts////eeee
Acknowledgements
This work was supported National Key R&D Program of China (No. 201 AAAA012200)))
10

---

References
[1] Martín Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk mini- mization. CoRR, abs/1907 .229002883.. 2019.
[2] Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning. ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 322 332.. PMLR, 2019.
[3] Shiyu Chang, Yang Zhang, Mo Yu, and Tommi S. Jaakkola. Invariant rationalization. In Proceedings of the 37th International Conference on Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 1458. PMLR, 2020.
[4] Shiyu Chang, Yang Zhang, Mo Yu, and Tommi S. Jaakkola. Invariant rationalization. In Proceedings of the 37th International Conference on Machine Learning. ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 1448-1458, PMLR, 2020.
[5] Elliot Creager, Hen-Henrik Jacobsen, and Richard Zemel. Environment inference for invariant learning. In ICML Workshop on Uncertainty and Robustness, 2020.
[6] John C. Duchi and Hongseok Namkoong. Learning models with uniform performance via distributionally robust optimization. Forthcoming in Annals of Statistics, abs/1810.08750, 2018.
[7] Abbas El Gamal and Young-Han Kim. Network information theory. Network Information Theory, 12 2011.
[8] Charlie Frogner, Sebastian Claici, Edward Chien, and Justin Solomon. Incorporating unlabeled data into distributionally robust learning. CoRR, abs/1912.07729, 2019.
[9]
Hal Daumé III and Daniel Marcu. Domain adaptation for statistical classifiers. J. Artif. Intell. Res., 26:101-,26, 2006.
[10] Arthur Jacot, Clément Hongler, and Franck Gabriel. Neural tangent kernel: Convergence and generalization in neural networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Anmual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages
2018.
[11] Masanori Koyama and Shoichiro Yamaguchi. Out-of-distribution generalization with maximal invariant predictor. CORR, abs/2008.01 01883, 2020.
[12] Kun Kuang, Ruoxuan Xiong, Peng Cui, Susan Athey, and Bo Li. Stable prediction with model misspecification and agnostic distribution shift. In The Moirty-Fouth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 4485-4492. AAAI Press, 2020.
[13] Matjaz Kukar. Transductive reliability estimation for medical diagnosis. Artif. Intell. Medicine, (9(1-2):81-10 2003.
[14] Danial Lashkari and Polina Golland. Convex clustering with exemplar-basee models. In John C. Platt, Daphne Koller, Yoram Singer, and Sam T. Roweis, editors, Advances in Neural Information Processing Systems 20, Proceedings of the Twenty-First Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada, December 3-6, 2007, pages 825 832. Curran Associates, Inc., 2007.
[15] Jiashuo Liu, Zheyuan Hu, Peng Cui, Bo Li, and Zheyan Shen. Heterogeneous risk minimization. In Proceedings of the 38th International Conference on Machine Learning, ICMI 2021, Virtual Event, Proceedings of Machine Learning Research. PMLR, 2021.
11

---

[16] Samet Oymak, Zalan Fabian, Mingchen Li, and Mahdi Soltanolkotabi. Generalization guar- antees for neural networks via harnessing the low-rank structure of the jacobian. CoRR. abs/1906.05392, 2019.
[17] Elan Rosenfeld, Pradeep Kumar Ravikumar, and Andrej Risteski. The risks of invariant risk minimization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[18] Cynthia Rudin and Berk Ustun. Optimized scoring systems: Toward trust in machine learning for healthcare and criminal justice. Interfaces, 48(5):4494466, 2018.
[19] Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameterizattion exacerbates spurious correlations. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13- 18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 8346 8356. PMLR, 2020.
[20] Antonio Torralba and Alexei A. Efros. Unbiased look dataset bias. In The 24th IEEE Conference Computer Vision and Pattern Recognition, CVPR 2011, Colorado Springs, CO, USA, 20-25 June 2011, pages 1521-1528. IEEE Computer Society, 2011.
[21] Madeleine Udell and Alex Townsend. Why are big data matrices approximately low rank? SIAM J. Math. Data Sci., 1(1):144-166) 2019.
12


Reviews:
[{"review_id": "pdlzUzSXF3d", "reviewer": "Reviewer_w5jL", "summary": "This paper proposes a novel optimization framework for Out-Of-Distribution (OOD) generalization problems. OOD generalization problem is one of the most important problems in today's machine learning,  and it is highly non-trivial to investigate it at the optimization framework level.\nThe proposed framework focuses on the integration of latent heterogeneity discovery and invariant learning on representation level, by introducing neural tangent kernel (NTK). Theoretical analysis is also conducted to verify the mutual promotion of these two modules. Experiments show that the proposed method can significantly outperform the commonly used and latest optimization algorithms.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well-written overall with a clear positioning with regard to the related research literature. I am convinced that this problem is relevant and critical to the ICML community. Although the overall logic follows the Heterogeneous Risk Minimization proposed in [15], this paper makes an important step forward by weakening the assumption that invariant and variant features can be decomposed on raw feature level. I like in particular how the proposed framework is theoretically justified. The heterogeneity exploration part is analyzed with a rate-distortion perspective, and the orthogonality property of kernels is guaranteed.  Experiment detail is clear in the paper and appendix on dataset details, baseline methods, and how to choose the hyperparameters for the proposed method. The results are satisfactory overall, and some of them are quite impressive.\n\nThe weakness of the proposed method would be the requirement of the hyperparameter to control the cluster number. I would like to see how the number of clusters affects the performance. Also, the authors are encouraged to provide some failure case analysis. For example, whether the mutual promotion between these two modules is sensitive to initialization. \n\nI believe this paper has merits that outweigh flaws and I would like to vote for acceptance.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "VVHSZH2_AGh", "reviewer": "Reviewer_Rd4w", "summary": "The paper proposes KerHIL, a new invariant learning method for out-of-distribution (OOD) generalization. KerHIL both explores the latent heterogeneity and learns the invariance in kernel space by iterating between two modules M_c and M_p, respectively. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Despite some typos and references errors (in checklist), the paper is well-written. The authors explained and demonstrated the method clearly, and gave theoretical proofs for the theorems. Experiments are also done in a good manner. However, there are still some points I am curious about. \n\nFirst of all, the authors stressed the difficulty brought by multi-source training datasets. However, this issue is only discussed empirically using training dataset with only two sources. It would be better if the authors could extend the training sources to three or more. \n\nSecondly, it would be more intuitive if the authors provide explanations or even guesses about how predicting training environment labels aid the training process. And how accurate the prediction need to be in order to fully help the training.\n\nFor the experiments conducted, experiment details are well listed. However, the parameter settings for different baselines and the proposed KerHIL are not demonstrated. For example, the specific value for the regularization term in Equation (8) is not included. What’s more, it would be good if the authors add ablation studies so as to explore more about the properties of KerHIL. This can help to identify what is the bottleneck (which module) for KerHIL and may provide intuitions about how to extend the method to more complicated datasets and deeper neural networks. \n\nOverall, I think the paper gives a great direction for future research while the current efforts seem somehow preliminary, and need further improvement in both theoretical and experimental parts.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "None.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "Px13Y6-w0I", "reviewer": "Reviewer_Rqt2", "summary": "This paper studies the out-of-distribution (OOD) generalization problem. It proposes a Kernel heterogeneity & Invariance Learning method, which jointly learns both the latent heterogeneity exploration and invariant learning in kernel space. This paper provides strong theoretical guarantees for the proposed method. We know the proposed method makes clear improvements over related baselines in OOD scenarios from the empirical results.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed studied in this paper is an important and promising direction. This paper first tries to jointly learn latent heterogeneity environments and invariance learning modules on representation level. It is important for applying invariance learning on unstructured data (e.g., image).\nBy introducing Neural Tangent Kernel (NTK) into the proposed algorithm, the authors convert non-linear neural networks into linear regression in NTK. Thus, mutual promotion will be well guaranteed. Moreover, a novel invariant gradient descent method is proposed to learn an invariant classifier. Meanwhile, the environment-specific features are captured by the orthogonal heterogeneity-aware kernel to accelerate the heterogeneity exploration. Thus, the proposed method is both novel and theoretically sound.\nThe paper conducts extensive experiments on synthetic and real-world data to validate the effectiveness of the proposed method. The proposed outperforms baselines with a large margin on OoD datasets.\nQuestions:\n1.\tHow to select the hyperparameter lambda in Eq.(8)?\n2.\tCould the proposed framework integrate the deep model (e.g., deep CNN) as the backbone to deal with a more complex image dataset?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "M69dBv74aT", "reviewer": "Reviewer_iPzu", "summary": "The authors present an algorithm for invariance learning under distributional shifts which greatly improves OOD generalization. This is an extremely important problem, and the authors give both good analysis and empirical results. ", "questions": "", "limitations": "", "rating": 7, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Section 2 is a really present read. Unfortunately, section 1 is almost unreadable to readers without this prior knowledge. I would suggest the authors restructure, such that the \"related work\" section comes after you have defined what you mean by invariant learning etc. \n\nAt line 165 you lose me. What is big theta_j ? Does delta denote Dirac's delta?\n\nHow is it determined how many clusters (or sources) there are?\n\nCan the authors expand why KerHIL^s is of importance?\n\nDo you think the results on Colored MNIST are affected by you using an MLP for your predictor? Would the color have as much influence if a CNN was used?\n\nOverall, I think the results are really good and you test out all the relevant hypotheses with a positive outcome. Very nice!\n\nI did not verify mathematical proofs.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I have big no comments here, but I hope the authors will include more limitations in the final version of the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the OOD generalization. The paper proposes kernel heterogeneity and the invariance learning method, which enable to learn the latent heterogeneity and invariant learning in kernel space.  The proposed method and theory provide novel and interesting directions for the OOD problem.  The empirical results are also good.   This work deserves to be presented in NeurIPS.  
Decision: Accept (Poster)