Invariance Learning based on Label Hierarchy
Authors: Shoji Toyota
Kenji Fukumizu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Invariance
out-of-distribution generalization
deep learning

Abstract: Deep Neural Networks inherit spurious correlations embedded in training data and hence may fail to predict desired labels on unseen domains (or environments), which have different distributions from the domain to provide training data. Invariance Learning (IL) has been developed recently to overcome this shortcoming; using training data in many domains, IL estimates such a predictor that is invariant to a change of domain.  However, the requirement of training data in multiple domains is a strong restriction of using IL, since it demands expensive annotation. We propose a novel IL framework to overcome this problem. Assuming the availability of data from multiple domains for a higher level of classification task, for which the labeling cost is lower, we estimate an invariant predictor for the target classification task with training data gathered in a single domain.  Additionally, we propose two cross-validation methods for selecting hyperparameters of invariance regularization, which has not been addressed properly in existing IL methods.  The effectiveness of the proposed framework, including the cross-validation, is demonstrated empirically. Theoretical analysis reveals that our framework can estimate the desirable invariant predictor with a hyperparameter fixed correctly, and that such a preferable hyperparameter is chosen by the proposed CV methods under some conditions. 

Invariance Learning based on Label Hierarchy
Shoji Toyota
The Graduate University for Advanced Studies Tokyo 190-8562, Japan shoji@ism ac ac. jp
Kenji Fukumizu
The Institute of Statistical Mathematics
The Graduate University for Advanced Studies Tokyo 190-8562, Japan f fukumizu@0 am.. ac.
Abstract
Deep Neural Networks inherit biased correlations embedded in training data and hence may fail to predict desired labels on unseen domains (or environments), which have different distributions from the domain to provide training data. In- variance Learning (IL) has been developed recently to overcome this shortcoming; using training data in many domains, IL estimates such predictor that is invari- ant to a change of domain. However, the requirement training data in multiple domains is strong restriction of using II., since it demands expensive annotation. We propose a novel IL framework to overcome this problem. Assuming the avail- ability of data from multiple domains for classification task at higher level, for which the labeling cost is lower, we estimate an invariant predictor for the target classification task with training data gathered propose two cross-validation methods for selecting hyperparameters of invariance regularization, which has not been addressed properly in existing IL methods. The effectiveness of the proposed framework. including the cross-validation, is demonstrated empirically. Theoretical analysis reveals that our framework can estimate the desirable invariant predictor with a hyperparameter fixed correctly. and that such preferable hyperparameter is chosen by the proposed CV methods under some conditions.
single domain. Additionally, we
Introduction
Training data used in machine learning may contain features that are spuriously correlated to the la- bels of data. Deep Neural Networks (DNNs) often learn such biased correlations embedded in train- ing data and hence may fail to predict desired labels of test data generated by different distribution from one to provide training data. In classification of animal images, DNNs tend to misclassify cows on sandy beaches, since most training pictures are taken in green pastures and DNNs inherit context information in training [3, 9]. Another example is learning from medical data. Systems trained with data collected in one hospital do not generalize well to other hospitals; DNNs unintentionally extract environmental factors specific to a particular hospital in training [16-18].
Invariance Learning (IL) a rapidly developed approach overcome the issue of biased correlation, which is caused by some bias in the distribution of a training dataset [10-15, 21, 33-36, 54J. In this paper, we use the term domain specify the bias in the distribution of dataset. IL is thus method for removing the influence of domain shits. Using training data from multiple domains, II.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

estimates a predictor invariant to the change of domains, aiming at keeping good performance in unseen domains as well as in the training domains.
While the IL. approach has attracted much attention, requiring training data from multiple domains may hinder wide applications in practice; preparing training data in many domains often involves expensive data annotation. In real-world data, labels may be missing [42-45, 56-58] or incomplete; in some cases, data may only specify classes to which the image does not belong [46-48]. Such data with insufficient annotation are not directly applicable to the standard IL methods; they must be re-annotated accurately, often great financial or human expense. The high cost drives a strong need to establish new IL framework without or with lower annotation costs.
To mitigate the problem of annotation cost, we propose a novel II. framework for the situation where the training data of target classification is given in only one domain, while the task of higher label hierarchy, which needs lower an- notation cost, has data from multiple domains. Here, the task of higher la- bel hierarehy means a classification task with coarser labels than those of the tar- get classification. Figure 1 shows an example of label hierarchy. Consider the case where target classification has 300 labels .bird1gn0 snake snake100, turtle1, ...: turtle corresponding to 300 species. Then, the binary labels (bird, reptile) are an example of labels in the higher hierarchy. The decrease in the number of classes reduces annotation time per image. Moreover, annotation of the binary labels does not require any expert knowledge, while annotation of the original 300 labels would requires expert knowledge about birds, snakes, and turtles. Hence, the new IL framework significantly re- duces the annotation cost in comparison with previous IL methods; we need exhausting annotation of 300 classes only for one domain and just binary labels for other domains.
bird
reptile snake100
bird1
bird100
snakel
turtle1
turtle100
Figure 1: Example of Label hierarchy. Blue labels are in higher level of label hierarchy than red labels.
Another important issue in IL is hyperparameter selection. Most IL methods involve some hyper- parameters to balanee the classification accuracy and the degree of invariance. As [21] and [19] point out, in the literature of II., the best performances of invariance had often been achieved by selecting the hyperparameters using test data from unseen domains. Moreover, [19] numerically demonstrated that, without using test data, simple methods of hyperparameter selection fail to find preferable hyperparameter. demonstrates a strong need for establishing an appropriate method of hyperparameter selection for IL.
We propose two methods of vooss-vali (CV) for hyperparameter selection in our new frame- work. Since we assume training data of single domain for the target task, it is impossible to esti- mate the deviation of the risks over the domains. Our CV methods mitigate the difficulty by using additional data from multiple domains in the higher label hierarchy. Theoretical analysis proves that our methods select a hyperparameter correctly under some conditions.
The main contributions of this paper are as follows:
We establish novel framework of invariance learning, which estimates an invariant predictor from a single domain data, assuming additional data from multiple domains for a classification task at higher level.
Under the framework, we propose two methods of cross-validation for selecting hyperparameters without accessing any samples from unseen target domains.
Experimental studies verify that the proposed framework extracts invariant predictor more effectively than other existing methods.
We mathematically prove that our framework can estimate a correct invariant predictor with a hyperparameter fixed correctly and that such preferable hyperparameter is selected by the pro- posed CV methods under some settings.

---

2 Invariance Learning based on Label Hierarchy
Notations Throughout this paper, the space of input features and finite class labels are denoted by X' and V; respectively. For given predictor f X' + y and random variable (X, Y) on X' x y with its probability Px,y. (f) denotes the risk of f on (X, Y); i.e., (f) :=
[ [=)) yd)...y where : 3 . - R is a loss function. For m No0. [m]] denotes the set [1,.,,m] m.. J. For finite set A, |A| E N denotes the number of elements in A.
2.1 Review of Invariance Learning
Following [10], to formulate the out-of iitto--distribution (o.o.d.) generalization, assume that the joint distribution of data (X", ))) depends on the domain (or environment) E E, and consider the are given training datasets yx ii.d. from multiple domains Eir c E. The final goal of the o.o.d. problem is to predict desired label Y E y from X X for larger target domains & Erp. To discuss the o.o.d. performance, [10] introduced the o.o.d. risk
dependence of predictor on the domain variable e. Suppose
D {22 of
max R (f), Ro.o.d. (f)
(1)
.Eē
where R ( :- ((()))))))) This is the worst case risk over E, including unseen domains & Ear--
To solve (1), [10] estimates a predictor f that performs well in unseen domains & \ &er as well as training domains Etr, namely a predictor invariant to a change of domains. The invariant predictor is composed of two maps and w; that is, f w a & holds. feature map & X - H. which is often called an invariance, realizes a feature of x E X' in the feature space H. with biased correlations estimates the label of feature (xx). The estimation of an
in x removed. predictor H invariant predictor is implemented by solving the following optimization problem:
Σ cEE..
Rs (w 2o0))))) D)
where Ie is the set of invariances captured by Ucee. D":
Ie 1 1 X H| P(Ye) (D( P( (Xxx))) for any e1, 02 E E
This invariance is based on conditional independence as [14, 15, 40), while [10, 11] use a different type invariance based on argmin,, R (wr b)) instead of invariance using the difference among Etr, assuming the availability of multiple training domains.
All IL methods estimate the
2.2 Invariance estimation by higher label data
Our goal is to make invariant predictor from single training domain E,r = {e") }. In this case, (2) is reduced to the empirical risk minimization min, Re (f) on e", and therefore the standard IL framework is not able to extract an invariance.
In this paper, we introduce assumption that additional data D for an another task (Xe, 2o), which is the higher label hierarchy than (X Y*), is available with respect to multiple domains Ead C ε. The task of higher hierarchy (Z) has a coarser label Z E Z than Y E )) more formally, Z represented as Z g(Y*) with a surjective label mapping 9 y + from the lower to the higher level in the label hierarchy. The example in Section is formalized by a surjevtive function g as, setting y == (bird),-. bird turtle).... turtle snake,, soake and Z == [bird, reptile), g(y) == bird if y bird, (i 21, .... 1003) and g(v) == reptile else.
By making use of (Di.) (DaabesEaa our objective for the invariant prediction is given by
mineez... Rd (w o $).
where Iad is the set of invariances:
Iad = { Φ X + H6 P(g((=))((X=)) P(g(Y=)) for any e,, 02 E End).
Note that (3) evaluates the risk with a single training domain while the invariances are given by additional data of multiple domains.
3

---

2.3 Construction of objective function
Among several candidates of the loss and model design, we focus a probabilistic output case and evaluate its eлror by the cross entropy loss; that is, we model w by Pe H. - Py, where Py denotes the set of probabilities on y and denotes model parameter. The risk is then written by
Rs (po o 1) /
log po (Y (x
We aim to solve (3) by minimizing the following objective function:
Objective(O, ④) :- R². (po ②)
+ (Dependence measure of P(g(Y"))((X)))
End)-
(4)
Here, Re. (po b) denotes the empirical risk of po O the training domain & {e"} evaluated by D= : (po o @) := p E(se" """")"""" log Po (b(x")): While we can consider some
variations of invariance regularization, we adopt the one used in [10] and construct an objective
function
Objective(O,,A.. Oud, +) = R (po o $) + Σ
IV Vonu Ban-Bad
R(x- 22 (p
ZIH Puu
$)1
(5)
eEEad
Here, Pg Z|H H Pe is the linear logistic regression model same [10), + is nonlinear neural network, and RX :2 2|H
2|H ( o b) Dial [(e")")EE log Pead (z")))>))
It is not obvious if the regularization term in (5) is valid as a dependence measure of P({(())(((X)) since it was proposed for another type of invariance based on argmin,, R Rw
b). The next lemma shows that these notions of invariance are the same in the current setting.
Lemma 1. When modeling w by conditional probabilities, the following statements are equivalent:
P(Z (X*)) does not depend on e <> argmin z1n R(X" (R(""22")( 22) 2|H o oD) does not depend on e,
Po.d
where model Z(H in the right hand side runs over all probability densities. Pe..
While our objective function (5) is similar to the ones in [10, 21] in that they are composed of an empirical risk and an invariance regularization, the correctness has not been fully discussed so far. In Section we will mathematically prove the correctness of (5) under some settings.
3 Hyperparameter selection method
3.1 Hyperparameter Selection in Invarance Learning
The objective function (5) has a hyperparameter À select, as is often the case with methods. The hyperparameter selection in IL has special difficulty, however, because the o.o.d. problem needs to predict y* on unseen domains, must be chosen without accessing any data in such unseen domains. It was reported in [19, 21] that the success of IL methods depends strongly on the careful choice of hyperparameters, and some the results even used data from unseen domains in the choice. [19] reported also experimental results of various IL methods with two CV methods, training-domain validation (Tr-CV) and leave-one-domain-our validation (LOD-CV), and showed that the CV meth- ods failed to select preferable hyperparameters. In the Colored MNIST experiment, for example, the accuracy of Invariant Risk Minimization [10] is 52.0% at best, which is about random guess level.
The failure of the CV methods is caused by the improper design of the objective function for CV; they do simulate the o.o.d. risk, which is the maximum risk over the domains. Tr-CV splits data in each training domain into training and validation subsets, and takes the sum of the validated risks over the training domains. Obviously, this is not an estimate of the o.o.d. risk. LOD-CV holds out one domain among the training domains in turn and validates models with the average of the validated risks over the held-out domains. Again, this average does not correspond to the o.o.d. risk. In summary, the problem we need to solve is answering the following question: how can we construct an evaluation function of the o.o.d. risk from validation data? In the sequel, we will propose two methods of CV, which are summarized in Algorithm 1.

---

Algorithm CV methods. If CORRECTION = True. is selected by method II and if False, I.
Require: Splitt DE
Ded Dadi.... ..., Ded into K parts. Set the hyperparameter candidates A. - -Dal
Require: (2))
where D == ((1,2) E P la 2n) for all Ead and x" E 2
1: for À E A do
2: for k = 1 to K do
3:
Learn 0/
[-x] by using [(-x]-
kjj D odd[-1): 1J:
Dea ad.
4:
(A) (A)
(2(6) Σ (we"
y0 EED( log Pot B)
(1" 100 1- -x(xx²))
//Risk estimation on c*.
5:
R2 1= (A)
(Mlo (n.))ED -lo-llogPp log (y) / (z).g(Y) ==)) for in Z
6:
for e E End do
7:
(A( (ay E(wr log Pop p0 (= (-(0011()))) (x")). // Risk estimation on e.
8:
CORRECTION then
9:
R2 (A) + Ex'ez (=-)) . R (A)
|/ Correction term addition.
10: end if
11:
end for
12: Rgnd
(A) + (ax (1)
|| o.o.d. risk estimation.
13: end for
14: Roo.dd (A) + K Σ Rgad (A)
// Final o.o.d. risk estimation.
15: end for
16: Select X argminxex Rood(()) (A)
3.2 Method I: using data of higher level task
We divide each of D= Dr ..., Dis into K parts where |Eadl n, and use the k-th sample (D)) D "d]
" Pad,ng) and the rest {Di- D -et----eee-- [d][- "". Den xd] for validation and training.
respectively. To approximate the o.o.d. risk of the trained predictor Por_.: + we wish to
timate (e(o R(-1) for Ead U {e"} by the validation set. For e", use the standard empirical estimate Ríja (Pop o o For e E Ead. we substitute unavailable Y with Z and use (X* .T" RM
(Po
o - [P E(er, log Pop (2 / (2-=)(²*)))
3.3 Method II: using correction term
Method I can be improved by correcting the replacement R = R(X" with R(x",z") for Ead- We use the following theorem for the correction:
Theorem 2. Let Z == { E Z (z"((2)|= > 1}. For any map & : X H. po : + Py, and random variable (X, Y) X V, the following equality holds:
R(X.Y) (po ))- R( (X))s())) (po o 1)
Σ (P(()) =²) R (XX,)) (po o } 2"E2"
Here, R(X,)))=* (po b) = Pog ( (X) gYY 2 dd where P(x,))19())= denotes the conditional distribution of (X,Y) given the event g(Y) = x,, and po(y)0(z),g(() 2²)= = Evea-t(a Poly (n))
p=())((2))
The ir is given in Appendix A. The theorem shows that, estimate the correction term, we need estimate (Y) =) and (ii) R(X Y'))=" (Pop (D(-)) for every E 2
(i) is naturally estimated even on E Endi P(ze =)= (x, E De == (ii) is not easily estimable; while direct simulation of the integration ddpixe "=)))("))==- demands data from ye) ~ yx, our available data De
De where De De Te
End is from Px',g(Ve), not from yx,,y.. To solve the non-availability of data from Px,,ye, we use the training data De Pxxe ye instead. Namely, (ii is estimated by
5

---

R(x0 RTi
1(x="8= X 11 (pot b)
o x = 10f E (a..e) log pop s) - =),,
where Dij ((r.)) E lo(1) = c Díg- In Algorithm the above risk estimate is abbreviated by R
(A) for notation simplicity.
Theoretical analysis
Throughout this section, to avoid discussing the non-trivial effects of nonlinear D, we focus on the simplified case of variable selections, where the feature map & is chosen from the projections of to subset of its components, For example, & may be ( T2, F3) = (I1, Iz3 when æ is three- dimensional. This type of IL appears practically
causal inference [14, 13] and regression [40]. Let A' := X1 X2 where A := R and 2 == R with n1, E N. For a projection +, let p denote the Re-component of (i 1, 2). If + has a Xy-component, we write Im * 0. For simplicity of analysis, the domain set is defined by all the probability distributions with the fixed marginal distribution Pxj,y= of (X1, Y); namely,
{(xe, Y = (X,Y) Y): random variable on X x y/ POxI(X), |
= Pxt,y1 - }
(+)
In this case, for any E & the variable Y*) satisfies (i) Pyojo equals to Pyt x and (ii) the marginal distribution (x) of the invariant feature T (X) equals to Px7- The above setting and definition persist through Section 4.
4.1 Theoretical analysis of our objective function
The following theorem ensures that, neglecting estimations and under some conditions, minimum of our objective function (5) with careful hyperparameter choice also minimizes the o.o.d. risk (1): Theorem 3 (o.o.d. optimality of our objective function). Under the setting (*), additionally assume that the following condition holds:
(A) For any variable selection &b with Im + 0, there exist two domains 02) C End such that + (X*)))
Then, there exists X* E TR such that any minimizer (At, ot ad: $1) of (5).
R(x 22 (ot, Or Φt) argmiin( R (po 4) + Σ 8 1( 0.0.0
(p +)113)
eEEnd
is o.o.d. optimal, i.e.,
Pgt o + E argmiinpo:x-- po: Ro.o.d (po),
where models pe and Poy in ming.ewa. run all the probability density functions, and T runs all the variable selections. The gradient Vo should be understood as the functional derivative on the space of probability density functions.
21tt
For the proof, see Appendix B. Condition (A) means that Ead has sufficient variation to capture the desirable invariance x
4.2 Theoretical analysis of our cross validation methods
In Sections 3.2 and 3.3, we approximate R(X*, ([x")") using label Zo of higher level. While the approxi- mation is not exact, we will prove that the proposed CV methods still select correct hyperparameter under some conditions. We will also elucidate the difference of the two CV methods. Given hyper- parameter X, minimizing (5) over the model yields the feature map (variable selection) denoted by Φ + Ra* (nx I ni +22). For simplicity theoretical analysis, we assume that the minimiza- tion of (5) achieves perfectly the conditional probability density function of (xA("")> denoted by p" - (> (x)). Then, neglecting estimation errors, the approximated o.o.d. risk of p a T used in Methods and II are represented by the following R' (A) and R! (X), respectively:

---

R*(A)= == max
max R(x
(Xx (((x)((()))))) ((""))(" Φh),, R(x-" yn) (p (
(6)
REEud
(o 1oo
(x) == max (ECacU(e")
R(x (000))(00))) -
yo' 115 yo E P(z = R 2M
"00)""(0"" (p
(pri
We have the following theoretical justification of our CV methods: the chosen gives minimizer of the correct CV criterion. For the proofs, see Appendices C and D.
Theorem 4 (Correctness of Method 1). Under the setting of variable selection (*), assume further that the following conditions (i) and (ii) hold:
(i) Among a A of hyperparameteer candidates, there exists X E A such that Tp² -T.i. (ä) Let pf be the probability density function of Pxc g(Y=")) Then, for any X with Im + 0. there is ex E Ead such that
(2,2) Pxex
(( ,(Y*x) satisfies pf (2) S Se -E with probability 1.
Here, E Ryo some sufficient small positive real number (that is, 0 < < and B H(Y ( (Xe")) )) the conditional entropy of (Φ (xe)). ).Y").
Then, we have
argminxes (*) argmin Rood (p TT))
Theorem (Correctness of Method II). Under the setting of variable selection (#), assume that, in addition to (i) in Theorem 4, the following condition (ii)) holds:
(ii)' for any À with Im + V, there is ex E End such that
2x ~ ) satisfies pd (z)x>(x)) Se holds with probability 1. Here, E is some sufficiently small positive real number and
Bx = H|Y" 1 x (X*)))
Σ P()(Y = R(x y )1 (p- o }.)).
Then, under the setting (*), have
argminxex Rd (A) C argmin Rooo o $2).
The conditions (ii) and (ii)' impose that, for at least one ex E Ead, the two domains and e* are different in the following meaning. X fails to remove domain-specinf factors (i.e., Im * (), for some 6) Edd, 2x) ox*x,)(X>x) yields low pd (=)))(x) with high probability. On the other hand, (x, z) ~ Pxe* g(x=e)) yields high of (=)(>(x)) with high probability: that is, e* and ex are different.
The theoretical analysis shows, while Method simpler to implement than Method II, Method more applicable. Noting that B 2 3x and hence, e than (ii). Recalling that (ii) and (ti)' impose the discrepancy between End and e as discussed in the last paragraph, relaxation of conditions from (ii) to (ii)' implies that method II be applied even when domains End and e" have smaller discrepancy than the condition for Method I. The difference of these two methods will be demonstrated in Section 6. The assumptions (ii) and (ii)' are discussed in Appendix F.
e - -, the condition (ii' is milder
feasibility of the
5 Related work
The proposed framework uses additional data from multiple domains as well as the training data for the target task. It is relevant to Transfer learning (TL) [22-24] and meta-learning [30, 41], which realize fast and accurate learning for new target task based on a model pre-trained with additional data sets. For example, after initial learning with large data set, fine tune [22, 23]
Fine-tuning
7

---

re-trains the model with the target task, while frozen feature [24] fixes the pre-trained model and tunes a head network. Although they show advantages in many learning problems, they may not work effectively in the current setting; in the fine-tuning with the target task (X"*, Y²)), the model tends to learn biased correlation in the data set and does not generalize to unseen domains. Some fine-tuning methods will be compared with the proposed approach in Section
Domain adaptation by deep feature learning Domain adaptation strategies by deep feature learning [25-27, 31, 32] assume that we can access input data on test domain in advance, and try to obtain data representation & x)) that follows the same distribution for the training and test domains. While the strategies lead to high predictive performance on a test domain similar to a training domain, such & does not function by discarding environmental factors from X E X as noted in [10]. Experimental comparisons will be shown in Section 6.
Distributionally robust optimization The proposed work tries to minimize the worst risk among risks on distributions perturbated from training distribution [69, 68, 64). The perturbated dis- tributions are formalized as small e-ball centered at the training distribution evaluated by some divergence. In our setting, the change of distributions in training and testing is necessarily small; if the background changes drastically, it is expected that its corresponding distribution also changes drastically.
Few-shot and zero-shot learning Some methods of few-shot and zero-shot learning [70, 71] try to generalize to new classes not seen in the training set, given only a small number of examples of each new class given no examples of each new class. Their problem setting is relevant to ours in that both of them have restrictions on the data obtained in training. [70, 71] train prototype representations of each class, which enable us generalize to new classes not seen in the training set. While these methods are useful for generalizing new classes, they do not intend to remove biased correlations embedded in all training data and hence, are unsuitable for our problem setting.
Other strategies [65] tries to obtain a de-biased feature following the independence ((X) 1 F, seeing & as a random variable E. Recently, [66] considers the setting where there exists some f in the model that f & fooodd where fu.o.d is an estimator with high prediction performance on both training and test domain, and that fx)) fo.o.d (x) for a sample x from training domains. Under the setting, they derive an upper bound of the risk on a test domain and propose a method for decreasing the upper bound. As a debias method, [67] uses two NNs; the first NN learns biased mapping by the standard ERM, while the second one is trained with the samples that have large errors by the first NN. This method is based on the idea that the training with samples with large errors by the first NN mitigates data bias.
6 Experiments
We study the effectiveness of the proposed framework and CVs through experiments, comparing them with several existing methods: empirical risk minimization (HRM), fine-tuning methods, and deep domain adaptation strategies. For fine-tuning, we employ two typical methods of transfer learning: fine tune (FT) and frozen feature (FF) [22-24]. As a deep domain adaptation technique, we adopt the state-of-the-art method DSAN [31]. We also compare our two CVs (CVI and CVII) with conventional CVs: training-domain validation (Tr-CV) and ceave-one-domain- (LOD-CV) [19]. We have two hyperparameters to be selected by CV. In the training with (5), we set := Abetore when the training epoch is less than a certain threshold f, and À := Aater if the epoch is larger than f. It is known that these two hyperparameters are critical for IL methods to achieve good results. From a set of candidates, each of the CV methods selects a pair (t. Aateer. To know the best possible performance among the candidates, we also apply the test-domain validation (TDV) [19], which selects the hyperparameters with the unseen test domain, and thus not applicable in practical situations. Additional experiments and experimental details can be found in Appendices G and H. respectively. The code is available in Supplementary Material.
cross-validation
Colored MNIST We apply our framework to Colored MNIST [10] with )2 = [10] and 2 := [2]. We aim to predict Ye E y from digit image data X E 22222X211 The label Ye changed randomly to one of the rest uniformly with probability of 25%. All digits in images are colored
8

---

ImageNet y
ImageNet
ImageNet
Dataset
CMNIST
33
y = [17].
Best possible
.750
random guess
.100
.333
.133
.059
Oracle
.715 (.00)
J43 (.018)
.749 (.008)
.708 (,010)
ERM
.433 (.004)
.47 (.016)
.507 (.020)
.357 (.020)
(.020)
.46((030)
.409 (.020)
.361 (.011)
.248
.482 (.127)
.226 (.046)
.162 (.011)
DSAN
.033
.278 (.004)
.293 (.008)
.060 (.007)
Ours CVI
6066 (.051)
.652 (.028)
.622 (,011)
.556 (.004)
Ours CV II
.688 (.018)
.666 (.027)
.622 (.011)
.556 (.004)
Ours TT-CV
500 (:006)
641 (.03))
.122 (.012)
.44 (.013)
Ours LOD CV
460 (.200)
.25(,028)
.522 (.022)
$28 (.019)
Ours TDV
657 (.008)
.673 (.035)
.634 (.033)
.56 (.004)
Table 1: Average Test Accuracies and SEs of Colored MNIST and ImageNet (5 runs): Oracle show the result of ERM with graysc MNIST (CMNIST) and of training with both e, and e2 (ImageNet). TDV selects À that yields the highest performance e2. The best scores are bolded.
Dataset
CVI
CVII
Tr-V
L.DDCVV
CMNIST
.051 (,053)
.039 (.017)
.163 (.006)
.197 (.205)
ImageNet: y
.027 (.029)
.013 (.020)
(.021)
.170 (.041)
ImageNet: y [7]
.012 (.001)
.012 (.001)
.018 (.015)
.054 (.024)
ImageNet: [17]
.0000 (.000)
.000 (.000)
.001 (.002)
.025 (.021)
Table 2: Means and SEs of {(Accuracy of TDV on e2) - (Accuracy of Each CV on 62) (5runs).
The lowest errors are bolded.
red or green. The domain [0, controls the color of digits; the digits Y >> 4 and ye < colored in red and green, respectively, with probability e. In training, D= Pxe-1,y yo is drawn with sample size n² 5000, and testing. Ye is predicted from X for e2 == 0.9. Regarding the higher level Z', the task is to predict Z 0 for X in 0 and Z = for 5 - 9 (that is, g( = Y > and else, g(Y*) = 0). The label Z is swapped randomly with 25%. We set Ead (0.1, 0.5, 0.7, 0.9} with n 5000 for each
Eod. We model & by 3 3-layer neural net. Setting the maximum epoch 500 and Anefore == 1.0, we select (t, Aateer from x 7 candidates with E {0, 100, 200, 300}, Auter E {100, 10 ..... 10%] by each of the CVs.
Table shows the test accuracies for 2000 random samples in the domain 62. The results, together with additional ones in Appendices G.1 and G.2, demonstrate that the proposed methods signifi- cantly outperform the others for e2. Among the two proposed methods, CV yields higher test accuracies on 62. Table 2 shows the accuracy gain of each CV from TDV with the same data sets for domain e2. These results, together with Appendices G. and G.2, concur with the theory Section 4 suggesting that CVII succeeds in wider situations, resulting in smaller errors.
ImageNet To see the performance of the pro- posed methods for more practical data, they are applied to the ImageNet [53] with its label re- annotated imitating BREEDS [52], which proposes a method for re-annotating ImageNet to create an o.o.d. benchmark. The target task here to pre- dict labels Y E y of images X E x We conduct three experiments with | 3, 7, 17. For each experiment, we prepare image datasets in different manners 61 and 62. The datasets con- sist of images belonging to one of the classes ) 2, 4, and 8 classes out of 3, 7, and 17 classes, re- spectively, are composed of different subtypes be- tween éj and e2; for example, the images of class bird in ej composed of ruffed grouse and in-
Dataset el
Dataset e2
ruffed grouse indigo albatross
water outel
laggerhead
Leatherback
box turtle
mud turtle
turtle
lighthouse
fountain
structure
Figure 2: ImageNet experiment dataset.

---

digo bunting, and the bird images on 62 are composed of albatross and water ouzel (Figure 2). In detail, show Appendix H.3. In training, D ~ Px ye is drawn, and in testing, Y is predicted from X on e2. The coarser label Z is binary (that is, Z [2]), and the sample the higher level Did of (Xx, 2²) drawn from both ej and ez- Here, D is the same as D but with labels re- annotated by g. We model & by ResNet50 [29]. Setting the maximum epoch 32 and Abctere == 0.1, we select (t, Aater) from the CVs.
candidates with {10, 20, 303, Aater E {0, 1, 10, 100} by each of
Table shows the test accuracies 62. We see that the proposed framework succeeded in predicting on 62, while the other methods failed. Table 2, which shows the difference between accuracies by TDV and each CV, verify that CVI and II selects À with the smallest error.
Comparison of two CV methods To highlight the difference between the proposed two CVs, compare them regarding the discrepancy between the additional domains of higher level Edd and the domain for training of the target task e*. We used synthesized data with X' = R2 y [10] and
2 = [2). preparing ten distributions [ on IR? which include a domain-specific factor in the second component depending on E Z (see Appendix H.2 for explicit representations of [2
The task to predict the distribution label {1, ..... 10). Setting e* := 20 with n² 60000, the test task is to predict the label for domain = 20. Regarding the task with label of higher level, use g(y) 0 if 4 and g(v)
else. We draw Dad Px=,z- (n 20000) from Ead {eud: 40}, where Cad ranges from 9 to 1. As ead increases, ead approaches to e". The model 3-layer neural net. We set the maximum epoch 500 and 0, and select Aater from candidates Aater E {0, 0.001, 80, 100} by each CV method.
Fuol
Fod
Faal
Foot 1757
TDV
396 (078)
621 (036)
63010413
.95 (061)
.901007)
A21 (.09)
5641071
552 (056)
.2000120
.529 (.128)
.555 (.11)
.355
346 (.172)
502 (.17)
335
(100
.173
527
315
(.22
(.11)
386
(4511.16)
.437 (317))
.821..140)
Table 3: Comparison of Two CVs: Average Test ACCs and SEs of the estimates (10runs).
Table 3 shows the test accuracy on e = - with 2000 random samples (zz, y) ~ Px--".y--e From the results, case where the variation among the domains is smaller as ead approaches to e". This accords with the theoretical results in Theorems and 5, which show that CVII finds correct hyperparameter in smaller discrepancy between End and e than CVI.
that CVII tends select better hyperparameters than CVI, especially in the
7 Conclusion
We have proposed new framework of invariance learning: assuming the availability of datasets for another relevant task in higher label hierarchy, obtain invariant predictor for the target classification task using training data in a single domain. We have also proposed two CV methods for hyperparameter selection, which has been outstanding problem of previous methods for invariant learning. Theoretical analysis has revealed correctness of our methods, including CVs, and the experimental results have demonstrated the effectiveness of the proposed framework and CVs.
Limitations and potential societal impact In Theorem it is ensured that objective function gives a minimizer the o.o.d. risk, only at least two domains have different distributions. In general, different domains do not necessarily have different distributions. Judging the discrepancy a further important problem. As a positive impact, our method will enable us to estimate predictors that don't 't use discriminatory factors such gender. There may be some negative aspects in that our method removes important information for a prediction as well as unnecessary ones.
Acknowledgements
The research was supported by Grant-in-Aid for JSPS Fellows 20J21396, JST CREST JP- MJCR2015, and JSPS Grant-in-Aid for Transformative Research Areas (A) 22H05106.
10

---

References
V. Vapnik. Principles of risk minimization for learning theory. In NIPS, 1992.
[2] L. Adolphs, J. Kohler, and A. Lucchi. Ellipsoidal trust region methods and the marginal value of Hessian information for neural network training. arXiv: 092001 (version 1), 2019.
S. Beery, G. V. Horn, P. Perona. Recognition in terra incognita. ECCV, 2019.
J. R. Zech, M.A. Badgeley, M. Liu, A. Costa, J. Titano, E. K. Oermann. Variable generalization performance a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study. LoS Medicine 15, e1002683 2018.
[5] T. Niven, H.-Y Kao. Probing neural network comprehension natural language arguments. Proceedings of the 57th Anmual Meeting of the Association for Computational, 2019.
[6] S. Gururangan, S. Swayamdipta, O. Levy, R. Schwartz, S. R. Bowman, N. A. Smith. Annotation artifacts in Natural Language Inference data. Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics, 2018.
[7] J. Dastin. Amazon scraps secret Al recruiting tool that showed bias against
women.
2018.
[8] A. Ilyas, S. Santurkar, D. Tsipras, Engstrom, B. Tran, A. Madry. Adversarial examples bugs, they are features. In NeurlPS, 2019.
[9] Shane. Do neural nets dream of electric sheep? http://biwerddessscc/////////110 2018.
[10] Arjovsky, L. Bottou, Gulrajani, and Lopez-Paz. Invariant Risk Minimization. arXiv:1907.0283
2019.
[11] K. Ahuja, K. Shanmugam, Varshney, and A. Dhurandhar, Invariant risk minimization games. In ICML, 2020.
[12] D. Rothenhausler, P. Buhlmann, N. Meinshausen, and J. Peters. Anchor regression: heterogeneous data meets causality. JRSS B, 83(2):215-2 24-2266, 2018.
[13] C. Heinze-Deml, J. Peters, and N. Mcinshausen. Invariant causal prediction for nonlincar models. Journal of Causal Inference, 6(2), 2018.
[14] J. Peters, P. Bühlmann, and N. Meinshausen, Causal inference using invariant prediction: identification and confidence intervals. JRSS 6(2): 947-1012, 2016.
[15] M. Koyama, S. Yamaguchi. When is invariance useful in arXiv, 2008.0 0188,, 2020.
Generalization problem?
[16] F. A. AlBadawy, A. Saha, and M. A. Mazurowski. Deep learning for segmentation of hrain tumors: Impact of cross-institutional training and testing. Medical physics, 45(3): 1150-1155 1158,118,,11 2018
[17] Perone, Ballester, R. Barros, and Cohen-Adad. Unsupervised domain adaptation for medical imaging segmentation with self-ensembling. Neurolmage, 194(1): 1-11, 2019.
[18] W. D. Heaven. Google's medical Al was super accurate a lab. real life was a different story. MIT Technology Review, 2020.
[19] Gulrajani, D. Lopez-Paz. In Search of Lost Domain Generalization. In ICLR, 2021.
[20] P. Kamath. Does Invariant Risk Minimization Capture Invariance? In AISTATS, 2021.
[21] D. Krueger, E. Caballero, Jacobsen, A. Zhang, J. Binas, D. Zhang, R. Priol, A. Courville, Out-of- Distribution Generalization via Risk Extrapolation. In ICML, 2021.
[22] S. J. Pan, and Q. Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22(10):1345-1355 2009.
[23] Q. Yang. Y. Zhang, W. Dai, and S. J. Pan. Transfer Learning. Cambridge University Press, 2020.
[24] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson. How transferable are features in deep neural networks? In NIPS, 2014.

---

[25] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. March, and V. Lempitsky. Domain-adversarial training of neural networks. Journal of Machine Learning Research, 17(59):1-35, 2016.
[26] S. Ben-David, J. Blitzer, K. Crammer, and F. Pereira. Analysis of representations for domain adaptation. In NIPS, 2007.
[27] G. Louppe, M. Kagan, and K. Cranmer. Learning pivot with adversarial networks. In NIPS, 2017.
[28] S. Sagawa, P. W. Koh, T. B. Hashimoto, P. Liang. Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for worst-case generalization. In ECCV, 2019.
[29] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
[30] C. Finn, P. Abbcel, and S. Levine. Model-agnostic meta- learning for fast adaptation of deep networks. In ICML, 2017.
[31] P. Stojanov, Z. Li, M. Gong, Ruichu Cai, J. G. Carbonell, K. Zhang. Domain Adaptation with Invariant Representation Learning: What Transformations to Learn? In NewrIPS, 2021.
[32] Y. Zhang, H. Tang, K. Jia, M. Tan. Domain-Symmetric Networks for Adversarial Domain Adaptation. In CVPR, 2019.
[33] J. Liu, Z. Hu, P. Cui. B. Li. Z. Shen. Heterogeneous Risk Minimization. In ICML, 2021.
[34] J. Liu, Z. Hu, P. Cui, B. Li, Z. Shen. Kerneiized Heterogeneous Risk Minimization. In NeurlPS, 2021.
[35]
Creager. Jacobsen, Zemel. Environment Inference for Invariant Learning. In ICML, 2021.
[36] G. Parascandolo, A. Neitz, A. Orvieto, L. Gresele, B. Scholkopf. Learning explanations that hard vary. ICLR, 2021.
[37] D. Kingma, J. L. Ba. Adam: Method for Stochastic Optimization. In ICLR, 2015.
[38] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-200-200 200-2011 dataset. Technical report, California Institute of Technology. 2011.
[39] B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(6): 1452- 1464, 2017
[40] Rojas-Carulla, Carulla, B. Scholkopf, R. Turner, and Peters. Invariant models for causal transfer learning. Journal of Machine Learning Research, 1933,),-34, 2018
[41] Andrychowiezl, Denill, S. Colmenarejol, N. de Freitas. Learning to learn by gradient descent by gradient descent. In NIPS, 2016
Hoffmanl, Pfaul, T. Schaul, Shillingford,
[42] Pham. Dai, Q Xie, Q Le. Meta Pseudo Labels. CVPR, 2021
[43] D. Lee. Pseudo-Label The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks. In ICML Workshop, 2013
[44] Z, Zheng, L. Zheng, Y. Yang. Unlabeled Samples Generated by GAN Improve the Person Reidentification Baseline vitro. In ICCV, 2017
[45] X. Gu, Sun, Z. Xu. Spherical Space Domain Adaptation With Robust Pseudo-Label Loss. In CVPR. 2020
[46] T. Cour, B. Sapp, and B. Taskar. Learning from partial labels. Journal of Machine Learning Research, 12(42):150--1536, 2011.
[47] Y. Yan, Y. Guo. Partial Label Learning with Batch Label Correction. AAAI, 2021.
[48] N. Xu, J. Lv, X.Geng. Partial label learning via label enhancement. AAAI, 2019.
[49] Y. Katsura, M. Uchida. Bridging Ordinary-Labsl Learning and Complementary-Label Learning. In ACML, 2020.
[50] L. Feng. T. Kaneko, B. Han, Ga. Niu, B. An, M. Sugiyama. Learning with Multiple Complementary Labels. In ICML, 2020.
12

---

[51] T. Ishida, G. Niu. A. Menon, M. Sugiyama. Complementary-Labe Learning for Arbitrary Losses and Models. In ICML, 2019.
[52] S. Santurkar, D. Tsipras, and Madry, Breeds: Benchmarks subpopulation shift. In ICLR, 2021.
[53] J. Deng, A. Berg, S. Satheesh, H. Su, A. Khosla, L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[54] C. Lu, Y. Wu, J. Hemández-Lobatto, B. Scholkopf. Invariant Causal Representation Learning For duttof-distribution generalization. ICLR, 2022.
[55] M. Takada, H. Fujisawa, T. Nishikawa. HMLasso: Lasso with High Missing Rate. In IJCAI, 2019.
[56] C. K. Enders. Imputation of missing data in industrial databases, Applied intelligence. Psychological methods, 8(3):322, 2003.
[57] K. Lakshminarayan, S. A. Harp. and T. Samad. Imputation of missing data in industrial databases. Applied intelligence, 11(3):259-273 1999.
[58] H. Tan, G. Feng, J. Feng. W. Wang, Y. Y. Zhang, F. Li. tensor-based method for missing traffic data completion. Transportation Research Part Emerging Technologies. 28:15-27, 2013.
[69] S. Sagawa, P. W. Koh, B. Hashimoto, P. Liang. Distributionally Robust Neural Networks In ICLR, 2020.
[68] W. Hu, G. Niu, Sato, and M. Sugiyama. Does distributionally robust supervised learning give robust classifiers? In ICML, 2018.
[64] Y. Oren, S. Sagawa, T. Hashimoto, and P. Liang. Distributionally robust language modeling. In EMNLP, 2019.
[65] H. Bahng, S. Chun, Yun, J. Choo, S. Oh Learning De-hiased Representations with Biased Represen- tations. ICML, 2020.
[66] H. Wang, Z. Huang, H. Zhan, Y. J. Lee, E. P. Xing Toward Learning Human-aligned Cross-domain Robust Models by Countering Misaligned Features. In UAI, 2022.
[64] Y. Oren, Sagawa, T. Hashimoto, and P. Liang. Distributionally robust language modeling. In EMNLP, 2019.
[65] H. Bahng, S. Chun, S. Yun, J. Choo, S.J. Oh Learning De-hiased Representations with Biased Represen- tations. In ICML, 2020.
[66] Wang, Huang. Zhan, Y. Lee, P. Xing Toward Learning Human Hlmann-aligne Cross-doman Robust Models by Countering Misaligned Features. In UAI, 2022.
[67] J. Nam. H. Cha, S. Ahn, J. Lee, J. Shin Learning from Failure: Training Debiased Classifier from Biased Classifier. In Neurips, 2020.
[68] W. Hu, Niu, Sato, and M. Sugiyama. Does distributionally robust supervised learning give robust classifiers? In ICML, 2018.
[69] S. Sagawa, P. W. Koh, T. B. Hashimoto, P. Liang. Distributionally Robust Neural Networks In ICLR, 2020.
[70] Snell, K. Swersky, R. Zemel. Prototypical Networks for Few-shot Learning In Neurips, 2017.
[71] O. Vinyals, C. Blundell, T. Lillicrap, k. kavukcuoglu, D. Wierstra. Matching networks for one shot learning. In Neurips, 2016.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contri- butions and scope? [Yes]
(b) Did you describe limitations your work? [Yes] Section 1, we note that data with target classification label needs from least one domain. In Section 4, we theoretically reveal conditions for our objective function and CV methods to succeed.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] It included in Section 7.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 4.
(b) Did you include complete proofs of all theoretical results? [Yes] They are included in Appen-
dices.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material URL)? [Yes] We put the code in the supple- mental material.
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were cho- sen)? [Yes] They are specified in Scction 6 and Appendix H.4.
(c) Did you report error bars (e.g.. with respect to random seed after running experiments mul- tiple times)? [Yes]
(d) Did you include total amount of compute the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] They are included in Appendix H.4.
If you using existing assets (e.g., code, data, models) curating/relcasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'l us-
ing/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable infor- mation or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you descrihe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "o4Mt4NR5jYE", "reviewer": "Reviewer_QPnu", "summary": "\nThis paper proposes a novel framework of invariance learning that enables invariant predictor to be estimated in single domain data by assuming additional data from multiple domains for a higher level of classification task, and proposes two methods of cross-validation for selecting hyperparameters without accessing any samples from unseen target domains. In the given experiments setting and with 2 datasets, the proposed framework shows improvment over other existing methods. In addition, the paper carries out a detailed theoretical analysis.", "questions": "\n* How to ensure that data in different environments are in different domains in higher level of classification task？\n* Is it possible to validate the method on another out of domain dataset?\n* Does the method just do a normal invairant learning by reducing the fine-grainedness of the labels?\n", "limitations": "The author discusses a little in the limitation section, and lacks discussion on scalability, efficiency, and future research directions.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* Invariance learning in a single domain data is a novel idea that can save cost of annotation and training.\n* The paper proposes two methods that can select suitable hyperparameters without accessing the unseen domain data, and gives a sufficient theoretical analysis.\n* The theoretical proof of the paper is solid.\n* The paper is well-written in general.\n\nWeaknesses:\n\n* Using a higher level of label hierarchy will lose a certain degree of fine-grainedness.\n* In the experimental part, only two datasets are used, and more experiments need to be supplemented.\n* For the experiments on ImageNet, the target task only predicts three labels, which is not convincing.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "TBtdciueNgv", "reviewer": "Reviewer_GyeR", "summary": "The paper introduces an interesting extension of IRM, which leverages the higher-level classes to learn the invariant classifier, thus it releases the data annotation efforts in many cases when the previous methods require the detailed annotation of classes. ", "questions": "This is an interesting paper with sound statistical effort contributing to expanding the areas of IRM, also with a realistic aim to remove fine-grained label dependence. The statistical efforts are sufficient, but my main questions are about the real-world implications of these statistical efforts. \n\n  - multiple theorems rely on several additional assumptions, it will be much more convincing if the real-world implications of these assumptions are discussed and how easy/hard these assumptions will be met in the applications. \n     - e.g., premise (A) at line 177 (although there is an explanation on line 182 but the explanation does not link to the reality well enough) seems to suggest that \"there must be enough domains so that the desired function is the only one that is shared with any two domains\" what are the corresponds interpretations for premises (I)(ii) for theorem 4 and 5? \n  - related to Theorem 3, the proof in Appendix is presented in a non-conventional way, and line 482 seems to introduce additional assumptions that are not presented in the theorem statement in the main manuscript. \n  - about the proofs of other theorems, there seem no clean proofs but a continued introductions of additional theorems. please clear all the numbering issues if there are. \n  - the empirical work is not very convincing either, there is not one selection method that can stand out from others, thus, in reality, we only know there exists a better method than previous ones, but not sure which one exactly to deploy to the real-world. \n      - is ACC on e1 not considered important for evaluation, if so, then CV-II methods are probably considered good enough? \n  - the imagenet results are a little misleading, while the authors keep using \"ImageNet\" to refer to the dataset, what's used in fact is a slice of the whole imageNet. \n      - why to use a creation of OOD imagenet where there are multiple OOD imageNet already? (imageNet-A, imageNet-sketch)\n  - too many baseline methods along the debiasing literature (OOD robustness) are missing, only to name a couple [1,2]\n\n[1] Learning De-biased Representations with Biased Representations\n\n[2] Toward learning human-aligned cross-domain robust models by countering misaligned features", "limitations": "there is a devoted paragraph regarding the topic. ", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "  - Strengths\n     - the paper aims to release the requirements of find-grained class annotations, which is an important aim toward real-world usage of the algorithm. \n     - the paper involves reasonable sound and comprehensive statistical efforts, which might be valuable to the statistics community. \n  - Weakness\n     - the link between the statistical work and the real-world implications seems missing (see below questions for details)\n     - the empirical evidence of the strength of the method is not convincing enough. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PCvgOv-apTX", "reviewer": "Reviewer_Lj4R", "summary": "This work proposes an invariance learning framework by using a training dataset of a single domain and additional data with coarser annotations. Two cross-validation methods are further proposed for the hyperparameter selection. The authors provide theoretical analyses of the proposed methods, and experiments on several datasets show the effectiveness of the methods.", "questions": "Questions:\n\n1. The settings of the datasets adopted in this work seem to be simple. Even though the authors claim that the ImageNet dataset is more practical, it only contains 3 classes with 2 high-level classes. It is better to see the performance on practical datasets with more classes, which can help to show how effective the proposed method is.\n\n2. In Line 154, what does the $n$ mean? Why will there be many different classes even for the addional data? If we want to reduce the annotation costs, the number of coarser labels should be controlled to a relatively small number. So what will be a typical value for the $n$ here to perform Method I?", "limitations": "The authors have discussed and addressed some potential limitations in the paper.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1. The idea of adopting coarser labels to learn the invariance among different domains is interesting. This is a good exploration on how to reduce the annotation costs in invariance learning. \n\n2. This work provides sound proof of why the proposed method will work.\n\n\nWeakness:\n\n1. The method proposes to adopt coarser labels to reduce the annotation costs. However, the authors may give a more strict explanation on how \"coarse\" the higher label hierarchy is. The examples provided in this work seem too simple, which makes me confused about how applicable this method can be to other more practical datasets or situations.\n\n2. The authors should conduct experiments on benchmark datasets as those in previous works like CIFAR-10. Now the settings of the datasets adopted in this work seem to be simple, which makes it not easy to understand the effectiveness of the proposed method. Also, it will help if the authors can compare the proposed method with previous works including but not limited to [1,2].\n\n3. The writing of this paper can be further improved. The authors should be more careful and proofread the paper on the typos and grammar errors, especially in the Introduction section. And some notations should be explained to help the readers to better follow the work, e.g. $e^*$ in Line 119.\n\n[1]. Gregory Benton, et al. Learning Invariances in Neural Networks\n\n[2]. Alexander Immer, et al. Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximations", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5b_1xkJ082T", "reviewer": "Reviewer_LVLW", "summary": "This paper studies the invariant learning problem for out-of-distribution generalization. The main efforts focus on the new training framework for saving the expensive annotations and hyperparameter selection strategies for searching optimal hyperparameters under appropriate settings. Both the training framework and selection strategies are supported by the theoretical results, where the main results show the proposed method ensure better generalization and optimal hyperparameter under some assumption. Overall, this paper is well-written and generally easy to follow while there are still some concerns regarding clarity, assumption, and rigor.", "questions": "1. In lines 129 and 179, an assumption ‘run all the probability density functions’ is made on posterior distribution (i.e., the concept/predictor). As far as I understand this assumption, it also implicitly admits that the training distributions cover all possible distributions/samples. which seems to conflict with the setting of out-of-distribution generalization. More explicitly, if the training distributions cover all possible cases, the target distribution can be a standard test distribution without shift. More justifications for this assumption are appreciated.\n\n2. In the proposed framework, the complexity of manual annotations seems to be unchanged since the meta(coarser)-labels on all samples are still necessary. Specifically, assuming that $|\\mathcal{Y}|=k_1$ and $|\\mathcal{Z}|=k_2$, it is clear that $k_2\\leq k_1$. If the complexity of manually annotations for target task is $\\mathcal{O} (k_1 n)$, then the complexity of annotations for propose framework is $\\mathcal{O} (k_2 n)$. Though $ k_2 n \\leq k_1 n$, the complexity of meta(coarser)-labels is still linear w.r.t. the sample size. I suggest more rigorous discussions on the cost of label hierarchy framework since one of the most important contributions is the reduction of labeling cost.\n\n3. Clarity of definition and notation.\n-  3.1) The claim ‘$P$ does not depend on $e$’ in definitions of ‘set of invariances’ could be more rigorous. In my understanding, it can be defined as conditional independence on $e$ by introducing $e$ as a variable of domain, or just defined as ‘$P(Y^{e_i}) = P(Y^{e_j})$ for any $e_i, e_j \\in \\mathcal{E}$’.\n-  3.2) The notation ‘(ii)’ is used without definition. Does it imply (ii) in Thm.2 (i.e., (ii) in line 169)?\n-  3.3) It is not straightforward to understand the claim in line 172 (a paragraph). Specifically, the claim ‘composed of all possible distributions of $X_2$’ is stated while the equivalent properties are given w.r.t. the invariant component $X_1$ (i.e., (i) and (ii)). More explanations will be appreciated.\n-  3.4) There seems to be no definition for $\\Lambda$ in line 197. Besides, what is the purpose of $\\Lambda$ in algorithm?\n-  3.5) The notation $-$ for complement operation could be confused with other operations on set or analysis. I suggest using $A\\backslash B$ instead of $A-B$.\n\n4. The term ‘spurious correlations’ seems to be misleading while the ‘biased correlations’ could be more appropriate. Though the model learns on source domain does may be biased on the target domain, the embedding is still useful on source domain and generally characterize most of the task information. Besides, according to my understanding of the ‘spurious correlations’ with the example provided in line 24, this problem still cannot be guaranteed to be addressed when target domain or multiple domains are available. Thus, revision or justification on this term will be appreciated.\n", "limitations": "Overall, this paper is well-written and the merits are significant. Thus, if most of the concerns (i.e., **Question**) are addressed, it can be a good paper towards a more applicable out-of-distribution generalization model.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n+ This paper takes a further step towards a more applicable out-of-distribution generalization model with appropriate and reliable hyperparameter selection. Thus, the technical novelty is significant.\n+ The methodology on meta(coarser)-label and theoretical results do provide a guaranteed way to deal with the dilemma of annotation cost on multiple source domains.\n+ The overall organization of this paper is good, and the claims are self-contained.\n\n**Weaknesses:**\n\n  *Major*\n  - Some assumptions seem to be strong (Q1).\n  - The essential cost (complexity) of annotations seems to be unchanged. (Q2)\n  *Minor*\n  - Clarity on definition and notation could be improved.\n  - The term ‘spurious correlations’ may be misleading.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept