Can Variance-Based Regularization Improve Domain Generalization?
Authors: Chuanlong Xie
Ruichen Li
Qishi Dong
Liwei Wang
Zhenguo Li
Venue: NeurIPS 2022 Submitted (2022)
Keywords: Variance-Based Regularization
Domain Generalization
Robustness

Abstract: If there is no prior information, domain generalization with only access to multi-domain training data relies on guessing what the test data is.  In this work, we consider mild assumptions that there is a distribution over domains and the out-of-distribution data is generated by the shift of the domain distribution. We study a domain-level variance-based regularizer. We show that the variance-regularized method can locally approximate the group distributionally robust optimization and embed the local information into the objective function as a weighting scheme. By taking the empirical domain distribution as an anchor of the location, we propose a weighting correction scheme and provide theoretical guarantees of in-distribution generalization. Compared to the Empirical Risk Minimization, we prove the potential benefits of our proposed method but do not observe consistent improvements in general.

31 respectively. The distribution shift between training and test data is characterized by the change of Q, Can Variance- Regularization Improve Domain Generalization?
Anonymous Author(s)
Affiliation
Address
email
Abstract
Without prior information, domain generalization with only access multi-domain training data relies on guessing what the test data is. In this work, we consider mild assumptions that there a distribution over domains and the out-of-di of-distributiin data is generated by the shift of the domain distribution. We study domain-leve variance-based regularizer. We show that the variance-regularrzze method locally approximates the group distributionally robust optimization and embeds the local information into the objective function as a weighting scheme. By taking the empirical domain distribution an anchor of the location, we propose weighting correction scheme and provide guarantees in-distribution generalization. Com- pared to the Empirical Risk Minimization, we prove the potential benefits of our proposed method but not observe consistent improvements general.
5
11
Introduction
12
24 is out-of-distribution (OOD) generalization problem and has drawn 38 30). Some recent works consider an ambitious goal that generalizes to "absolutely" unseen domain by learning
Domain generalization 12
14 much attention recently 33
features. From the perspective of theory, 16 the price of such invariant learning methods is the requirement for harsh assumptions or strong prior information, which is necessary to guarantee that the invariance exists and is identifiable. In this 18 work, we assume that there exits a distribution of domains and the OOD test data is generated by the shift of the domain distribution. Then domain generalization is formulated into distributionally robust optimization problem (DRO, ( IT (0]).
20
21 Let = (x,y) y) be data point consisting of an input vector E X and the target label E y. Suppose the training data is structured with respect to a latent domain label:
22
Djr {2,
m} {(Z.g) 1 S < m;), sisn) n),
where is the total sample size m, is the sample size of the i-th domain and n is the number of 24 domains. We assume that the training domains are randomly drawn from possible domains with domain distribution Q, i.e. E,, = {er, , Cn C E with e; ~ Q and the data points under domain e is sampled from the distribution P.. Let H be the hypothetical space and E H be a model that maps
26
[0, M] measures 28 how poorly the output ý = hi(x) predicts the target y. Denote F as the collection of the functions
X' to l(x) E y. The loss function E(ý,y) y
((h()),-) + [0, .M] with h H. The in-domain expected risk and its sample average
29
30
approximation (29) are denoted by
m f(4.,)
R(fles) = Ex-re (x()) and R(fle:) m.
32 while the data distributions P.. e are fixed.
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

33 We study the group distributionally robust optimization problem (group DRO, 18 26 28 )
min max [z)r([())] s.t. P f= P.Q(de), D₂(Q)Q0) S p.
(3)
JEF Q
34 where Qo is a selected domain distribution, D+(-I-) stands for the didivergence (13 14)) and the
35 tuning parameter p modulates the distribution shift. Throughout this paper, D-(-Il-) is the x².
36 divergence, i.e., @(t) = 2t - 1) Sagawa* et [28] consider the empirical optimization problem.
Σ g: ([(e)))))))))))) with A. = - ((......) : JEF qEan
min max
(n)
O
37 Here A, the (n -i-dmennsinaal probability simplex. this case, the parameter p fixed 38 and sufficiently large. For more ambitious goals, Krueger et al. [22] propose the minimax risk extrapolation (MM-REx) that extends the uncertainty region Dn into
39
{ q (q1..... :
An(a)
4
40
where the parameter a E (-00, 1/nn] modulates the uncertainty region. The negative value of a extrapolates risks and encourages robustness to large distribution shifts.
42 At the sample level, the DRO loss can be asymptotically approximated by the sum of the ERM loss 2 and a variance-basee regularizer [16]. where the negligible error term converges to zero almost surely. Section 7 in [16] gives general results when the DRO objective is a Hadamard differentiable functional to P and F is Po-Donsker class. From the perspective of generalization, the upper bound of the prediction risk may also have a variance-based regularization term that trades between approximation error and estimation error [3 13 21 Sample variance penalization 2 replaces the variance-basee based regularization with its empirical estimator and gives theoretical guarantees on the prediction performance. To address the computationally intractable problem caused by the non- convexity of the regularizer, Namkoong and Duchi [25] and Duchi and Namkoong [15] investigate
44
46
48
50
51 the robustly regularized risk, that provides convex surrogate for aariance-regllarized loss, and
52 prove finite-sample and asymptotic results characterizing prediction performance. Back to domain generalization problem, Krueger et al. 22 develop variance-regularized empirical loss (V-REx):
R(f) WVone (f), where
54
n
Σ R(fles) and R(F) 71
Voui(f) =
Σ (R(fle)) -R()))))(((())-)))))))))))))))))
Xie et al. (35 prove that with high probability, optimizing the regularized loss R() + > Vone(S) equivalent to solve a MM-REx problem.
55
56
In this work, refine R(f) and Vour based on the intuitive understanding of generalization
58 and distribution estimation. Recall the problem in (3). In general, Qo is the ground-truth domain
59 distribution and Q belongs to neighborhood of Qo- Therefore, the empirical version of (3) should
replace Qo with its empirical approximation over Etr, i.e., Mp
60
(ğ1, 12. .1. (m,)
m
61 However, the existing variance- methods directly replace Qo with a discrete uniform
82 distribution (the center of (a)) without considering a consistent and efficient estimator q. In the
63 sample variance penalization, this problem does not exist because the discrete uniform distribution
64 on sample points (no tie), i.e. the empirical distribution, is a consistent estimator of the ground-truth
65 data distribution. Consider new uncertainty region:
2a ④- Ân(o) {q: Do(all)) s p}.
Specifically, any (gi....,gh. 9n) Qo (g) satisfies
a N a
qi 1,
Σ
- S
2 46

---

67
we prove that with high probability, the MM-REx problem on ca can be uniformly equivalent to minimize the variance-regulariiee empirical loss R(ʃ) XV V (f) where
In Section 3.2
68
2 R(f) Σ (ffe)) and Vout (f) Σ & (R(fle)) - R(f))²
Comparing to R(f) and Voue (f), the two terms R(()) and Voue (f) just introduce a weighting scheme 70 derived from the empirical domain distribution q. addition, the term it(f) is the exact ERM loss
69
32
In Section 3.1. we investigate the generalization guarantee of the variance-regulariized estimator,
72
f arg min R()) + X Voue()) JEF
(5)
via the covering number of the function class F. Appendix C also provides version of the general-
74 ization guarantee with localized Rademacher complexities, which may provide tighter generalization In Section 4 consider a general uncertainty region Qo.)(g)). where the 76 choice of qo represents a kind of prior knowledge. Similar to the arguments in Section 3 we can also write qo as weight assignment and embed into the variance-regularizze loss function. We present
bounds in some
77
78 a general form of the proposed method and prove that the optimization equivalence in Section 3.2
79
still holds when we replace 2a (q) with Qa.p(go).
80
Our results clearly show that
81
From the perspective of generalization, we propose a weighting correction scheme for variance-regularizee domain generalization methods. The proposed method can outperform ERM under some cases, which shows the potential competitive edge of the proposed weighting correction method.
83
84
We do not observe that method consistently improves ERM under general cases.
86
The proposed method is robust to the change of the domain distribution Q. From an optimization perspective, it is equivalent to solve group DRO problem.
87
2
Preliminaries
88
In this section, present the rationale for using variance-based regularization to improve the 90 robustness generalization. Section 2.1] gives two domain adaptation examples that the test data is known. We prove that the standard deviation of risk can bound the generalization gap between 82 training and test data. In Section 2.2 we formulate an invariant learning principle as a hypothesis testing problem. We point that penalizing the risk variance can protect the null hypothesis: the model is invariant across domains.
34
2.1 Risk variance bounds generalization gap
96
We present two simple examples to show that penalizing the standard deviation of risk is a natural strategy improve robustness to the domain distribution shift.
96 Risk Interpolation. the first example, we assume the test distribution belongs to the convex hull of training domains. This is typical risk interpolation case. Let pa be the test distribution. Suppose there exists q" = ....... : (% E An such that p* = -'= 13 Paçs where Pays SiS i training domains. Then the generalization gap between the training and test data is
100
e Σ 9f R(feee Ea.((lee) Eli ( Ea. (
where q: Q(de:)/Q(de)))) the proportion of the training domain e; in the training data. We write q = (qı,-... ). By the Cauchy-Schwarz. Schwarz inequality, we have
102
103
erry S ////222000 (2 x VVnu())
(6)
3

---

104
where Voue (f) is the between-domee risk variance over the training domains:
72
Σ di (((fle)) Σ R((((e)))
Vout (f)
Notice that (f) only depends the training data. Therefore, is natural to penalize v Vone (f) to obtain tight upper bound of the test error. The principle here is that if for Ve; E Etr, R(f (e;) is a constant that only depends on f. i.e. Vour = 0, then changes from to q cannot cause any generalization gap.
105
107
109
Sub- lub-population Shift. Recall that the training data in (T structured with respect to latent domain
110
label. In this example, the domain label is the class label y. Therefore, the marginal distribution of y is different in the training and test data, and the conditional distribution P(xly) is the same, Let
111
112
) = . K}. Then the generalization gap between the training and test data is
K
erry
=
Blf(z)ly = (Pe = =k) Pely = k))
k=1
1 v (P (o)
113 where
K
K K Σ
Vou(J) = Σ P = ()(y=))(B\((e)) k)
B[(())ly = k))²
114
the between-class risk variance over the training data. Therefore, the generalization gap is also bounded above by the retween-domail risk variance. If the in-class risks are equal, i.e., (zf y (zf(z)ly K), Yk, K y, then the sub-population shift cannot cause generalization gap.
115
116
117
2.2
Penalizing risk variance protects invariant models
118
In this section, we heuristically discuss the relationship between variance-based regularization and invariant learning. The REx principle (22) presents training goals: Reducing training risks and Increasing the similarity of training risks. Krueger et al. [2 heuristically explain the utility of
119
120
121
V-REx as enforcing the equality of training risks in the limit case + +oo. In some experiments. V-REx with small À also shows robust generalization and may outperform ERM. Here we understand this phenomenon by extending the REx principle the population level:
122
123
Minimizing the expected risk R(f);
125
Cannot reject the null hypothesis of the test:
Ho R(fle) R(fee))))))))))))))))C)) Ve,e e' E vS Hi R(fle) + Be, E (7)
126 In general, Principle (i) is achieved by minimizing the ERM loss. Next we show that variance-based
regularization related to the hypothesis testing problem Principle (ii). Under regular assumptions, one can use the one-way ANOVA F-test to check the hypothesis testing in (7). The F-test statistic is 129 the ratio of the between-domain variance to the in-domain variance, i.e.,
127
Voue (f)
mi;
F V(J- (J) VIne with V()
((())
Here Vone(f) and R(f) are defined in (4. If larger than threshold, e.g. the (1 -quntiile of a F' distribution. one should reject the null hypothesis. Here 5% is the significance level. If the in-domain variance of well-trained model is approximately stable, then penalizing Vone (f) is equivalent to constraint that Hg cannot be rejected. Therefore our proposed method that penalizes
130
132
133
Vone()) (f) is consistent with the REx principle and the regularization term Vout (f) is generalized
194
version of V-REx.

---

3 Variance-Based Regularization
136
137
we study a variance-based regularization method for domain generalization, which minimizes the following empirical loss function:
Motivated by Section 2
138
R(()) + X Voue(f) (f),
(8)
where À is tuning parameter and (f) is empirical estimator of the between-domain risk variance. The proposed loss (8) directly optimizes the ERM principle R(f), which is different to
140
the recent invariant learning methods that minimize R()). e-g. Invariant Risk Minimization [].
141
142
The regularization term is slightly different to V-REx: (i) The square-root operator is derived from generalization gap; (ii) Different to the empirical variance of R[fle), we penalize the between-domain variance of (z).
143
145
We consider Qa in (3) the training domain distribution and denote the training distribution as Po = P. (dde). To proceed further, we denote more notations as follows:
148
R(J) = Ee-Qo [R([]e)] Ex-Pa [z)),, V() = Ez Vin (fle) Ea-P. [[[(2) R((le)) (Ju = R..qq [(([]e)) R(J))
147
where R() is defined in 2 Here (f)
(f) is the netween-domain variance and Vin (fle) is the in-domain variance of the domain E. According to the decomposition of the total variance, have
149
V()) = Var(B[[(22))) [[Var(][ee] Vous()) + Be-Qo (Von (fle)]-
When Qo and P. are replaced by the corresponding empirical distributions, we rewrite V(J), Vin (ele) and (f) as (f), (fle) and Vou((f) respectively. In the finite-sample setup, the decomposition the total variance also holds:
150
152
V()) Voue (f)
+
m V (Jle). Σ 1=1
m
3.1 Generalization
154
Since the empirical loss ( is derived from generalization bounds, we present two versions of the generalization guarantee. The first result depends on the covering number of the function class In the appendix, we also derive version of the generalization bound with localized Rademacher complexities, which can provide more refined uniform generalization bounds in some cases.
155
156
157
158
We start with the definition of the covering number. Let J. be a collection of bounded functions y [0, M). Suppose F is subset of metric space with norm || -|- We say collection f: , / C F is an e-cover of F if for each f E F, there exists f" such that []ʃ l'll S The
159
160
181
covering number Fis
N(F = inf (N E N: there exists collection (f'...., /N
which is an c-cover of F with respect to || ||- 11].
182 In the following, we use the 8o norm: |lf - glooo = sup lf(z) g(2)). Now we are ready to present the following theorem:
184 Theorem
Let n 2 and {z.,j S S n, si S m²} is an i..d sample drawn from Po- Suppose (22 E M. any I F and E X y and the function class F has the over number:
186
N. = N(F,ee,, (xxx))))))
< 8 < and
(n + 2)N
2f
t = log
X
5
1
5

---

187
Then we have, with probability at least 1 - 8.
(m, 1)Vin( (fes)
B() S R() XV Vona())
m
(m 1)m
(4m 1)M22
Vm(m= - 1)
3m
==1
(mg 1)
+ +22+
188
holds for every F.
The proof of Theorem is presented in the Appendix B. In some cases, the covering number- based analysis cannot provide a tight generalization bound 4 5 31.. Therefore, we also use the local Rademacher complexity [5] to present the generalization of the proposed variance-based regularization. The details and proof are postponed into Appendix C.
170
172
Why study In-Distribution generalization? Theorem [ provides the generalization guarantee for the in-distribution (ID) generalization rather than the OOD generalization. But its result gives important insights into the OOD generalization. First, the ID error provides a lower bound for the worst-case OOD error since E,r is a subset of E. Second, some empirical studies of OOD generalization have observed a linear relationship between the ID and OOD test error [7 19
174
176
178
Third, some OOD generalization bounds are derived from a domain adaptation framework e.g.
179
36
OOD error S. ID error + error gap + O(-),
which starts from the ID enrrr and then depicts the error gap. Most recent works focus minimising the error gap and ignore how their robust (or invariant ) methods increase the ID test error. Fourth, our assumptions are mild and general. We do not impose strong constraint on the test data, e.g. structured generative mechanism, and only assume the domain distribution shift. Therefore, we analyze the ID error of the proposed robust method under mild assumptions.
181
183
We denote f* as the optimal function and let f be a solution:
185
E arg min R(f) + X Voue
JEF
Next study the excess risk of f. According to Theorem
obtain the following result.
187
Corollary 2 Suppose the assumptions in Theorem 1 hold. Let 0 < d < 1 and
2., +2
2t
t t= log 8
X =
V m- 1
188
Then, with probability at least 5,
R(F*) 1 2X1
(m ())(S*
m:
Vin (fles)
m
m
(2
+ X
n
m4
444 1)M
m
3m
189
Parametric Example. Suppose the hypothetical space F is a class of parametric functions:
F= [fo(z) x E [
190 where the parameter set θ is bounded. Further, for any data point z, fo(z) a L-Lipschitz function
181 of 0 with respect to e norm on O. Then the covering number is bounded above:
N. < (1 diam(O) L L. 1)) with diam(e) sup 10 0' 1|2-
0

---

192
Then we take
1
log m
log N. = O m), x 0(1
m
193
Therefore, by Corollary
with probability at least 1 - 6,
(m (f)
n
my Van (fle) + O
log m m
R(f) R(f*) & 2
Σ
V
(10)
m
m
194
Potential competitive edge. The second term on the RHS of 1TP contains the empirical in-domain variance Vin (f(e) For per-parameterized model, the empirical in-domain variance of f can be close to zero. If there exists an optimal function f* arg min, R(f) such that V(f*) O, then the term (log m m/m) dominants the convergence rate of the excess risk For ERM, the convergence rate of the excess risk is 1/, which is slower than log m n/m. m. Due to the fast convergence rale, our proposed method can outperform ERM when the sample size m is large enough.
195
197
198
199
200
Cannot consistently outperform ERM. If there is no optimal function f E arg min, R(f) satisfies
201
v( = 0, then the first term on the RHS (10) can dominant the excess risk. this case, the convergence rate of the the excess risk of our method is /log m/m, which is slower than ERM. This implies that if V(**) for V E arg min, R(f), ERM can outperform our method when m is large enough.
202
OOD generalization. According to Eq. (9. OOD error can be rewritten as sum of ID error and error gap. The distance between the training and test domain distribution can determine the error gap term under setup. Furthermore,
206
only assume that the training and test domain distributions but different, and do not impose any structured generative models, such as structural equation models probabilistic graphical models (20). In other words, do not introduce prior information and use mild and general assumptions. Due to the uncertainty of the test data, the error gap should be the worst-case error gap for the domain distribution shift and hypothetical space, which is independent of
close
208
34
210
212
the estimator. This implies that without prior information, the ID error is a reliable metric to infer the OOD) error.
214
convexiy, Similar to the Sample Variance Penalization (23], the proposed objective function 8 is in general non-convex and computationally intractable. The proposed regularization term is non-convex even if the loss function is convex. is still unclear how to actually minimize the mriance-regularizee objective function. Krueger et al. [22] use a penalty annealing scheme to obtain a good pre-train model. In the Appendix, we empirically show that our method can use random
216
218
219
initialization without dropping generalization performance.
220
3.2
Optimization
221
In this section, we show that minimizing 8 is equivalent to solving a group DRO problem con- ceriing local neighbourhood of the empirical domain distribution. Let q = (g1, .., qu.) be discrete distributions defined on the domain set &,y {ei, "e....,,,,, We consider the following optimization problem that minimizes
222
223
224
max g
Σ q. R(fee))
(11)
225
which is slightly different to group DRO problem because Q. (q. p) is not centered at the uniform discrete distribution. We denote X = 22p and rewrite the empirical loss in (8) as
226
C(f:p) = R(f) + V22Vouu(
(12)
The following theorem shows that the objective [I] 228 in the form of 12)
is bounded by two variance-regullarized functions
'The factor log m comes frome the covering number N.. If the hypothetical space F only contains finite models, No constant and independent to Then the convergence rate of the excess risk is 1/m.
7

---

229 Theorem Suppose the training dataset DD and a function f E F are given. Let p+ be the largest 230 distance between á and q 2a, and
min: 1)2 (o (f)
p- 2( min, R(ffes)) R(f)))
then we have
-(S:P))
max too (4)
2a S ((S:+))-
(13)
232
This second inequality in 13 implies that the optimization problem (11 with p +00 always bounded above by the variance-regularized loss with the tuning parameter p+. On the other hand, we
233
234
can also derive tuning parameter p- depends on the training data and a given model f. and then
235
prove that C(f: p-) is lower boundary of [1 According to the proof of Theorem 3 one can find
236
that the equality holds:
Σ 1=1
R(f R(f) 2pVoue(f))
max qEQ
237 when the radius P satisfies p S p_. If Vous(J) is nonzero and p given, the equality holds if and only if Ve, E Etr,
S q V(V
2p ( (oue(f)
(14)
239 Therefore, the parameter a and the radius govern each other.
Sketch of Proof: We start with preliminary result: for any and p.
(..e()l s ((Si)).
((F:p)
max GEQ.. Cu (4)
which is directly derived from the Cauchy- Sehhwar- inequality. By checking the conditions for the
242
equality, we obtain the constraints in
14].
Note that Qa. (q) C x...
p. (q) since p+ is the largest
243
distance between a and q 20 qq. Hence the second inequality (13 is trivial. Let q" be
q
arg qE g
Σ R(((e))))
244 Furthermore, qº E Qa.p- (q) C Qa (q). Hence the first inequality in (13 holds.
245
Theorem 3 shows the equivalence between group DRO and the variance- regularization in 12). However, the lower bound p-f:p-) still depends the training data and model. Next
247
concentration inequalities and the covering numbers of F to derive the uniform results.
Theorem Suppose that a is a non-positive scalar and V (f) = V(f) (fveee O.
248
For each training domain, both and qi larger than 0. We write
V (f)
2
(n 1)8 -
16M2
250 Let > and 0 < 1) 1 be two constants. Define
Fin (JEF:V()2 and Vin(fles) V(J)
Sn, Ve, E Eir).
251 For any f E Fr.m the following expansion uniformly holds:
=
max
9 g
8

---

with probability at least 1 - N7.n x p. where NT. = N N(Fr.m V (1 - -n)), (1 is the
252
covering number Fr.n and
m(1 T
1
m(1 - 11)2T²
p
exp
32M
16
n1 -
exp
M
m (1 - 17)
+ 417 2
+ exp
2M2m;
+ 37
1=1
4
General Version
255
Recall the uncertainty region in (3:
(2(Q)) S p}. Section 3 Qo is the ground-truth (Q
domain distribution. In fact, it can be a selected anchor distribution closed to the target test domain. The choice of Qo can be regarded as a kind of prior knowledge and the hyperparameter p represents how strong is the confidence in the prior. We formulate the finite-sample optimization problem
257
Σ R(fe())
(15)
Q (TO)
259
where qo is the conditional distribution of given E Etr, which is derived from Qo. In this problem, the uncertainty region 2a.p is centered at discrete distribution qo rather than the uniform distribution or the empirical distribution q. Therefore, we can manually select qo to introduce the prior information.
261
According to the proof of Theorem in the Appendix, the optimization equivalence Section 3.2 also holds when we replace (á) with (g)) To proceed further, we rewrite L(S; Qo)
263
R(f. qo) + V 2ppVoue go) with
285
R(f. qo) Σ go R(fles and Vout(f, qo) = - qu. (R(fle)) R(f.qu))
Then restate Theorem as the following general version.
Theorem Given the training dataset and a function f F, then for any distribution qo. the inequality always holds:
268
Σ (lle))) S C(::p.))..
q (q0)
i=1
If the between- variance Vonc (f. do) is non-zero, the equality holds if and only if Ve, E Exc.
289
a S .59.. so:0(
(4) 2p ( R()) + +).
Vo
270 On the other hand, o is fixed, the equality holds when the radius of 2x. satisfies
mine(a/ 2min, R(((e:) - f((S{go)))
271
This result shows the equivalence between the optimization problem (15) and the variance-regularizee loss L(ʃ: p, qo). Therefore, for unbalanced domains and any given prior qo. can still use the variance-based regularization to approximate the DRO problem. Please refer to the Appendix for the
272
273
complete proof of Theorem 5
5
Conclusion
276
In this work, we study variance- based regularization method for domain generalization. We prove the guarantees for in-distribution generalization and figure out the potential benefits of our proposed method compared ERM. Our proposed objective function non-convex and the optimization procedure is computationally intractable. The learnt model can be highly dependent on initialization or pretraining. In future work, we will consider combining generalization bounds with specific optimization algorithms to seek fine-grained generalization guarantees.
277
278
280

---

References
282
[1] Kartik Ahuja, Karthikeyan Shanmugam, Kush Varshney, and Amit Dhurandhar. Invariant risk minimization games. In International Conference on Machine Learning, pages 155-155. PMLR, 2020.
284
Isabela Albuquerque, João Monteiro, Mohammad Darvishi, Tiago Falk, and Ioannis Mitliagkas, Adver- sarial target-invariant representation learning for domain generalization. 2020.
286
287
[3] Syed Mumtaz Ali and Samuel D Silvey. general class of coefficients of divergence of one distribution from another. Journal of the Royal Statistical Society: Series B (Methodological), 28(1) (8)):1311144, 1966.
288
289
[4] Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463 482, 2002.
290
291
[5] Peter Bartlett, Olivier Bousquet, Shahar Mendelson, et al. Local rademacher complexities. The Annals of Statistics, 13975374 2005.
292
293
[6] Peter I. Bartlett, Michael Jordan, and Jon D McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473): 2006.
294
295
77] Shai Ben-David, John Blitzer, Koby Crammer, Fernando Pereira, et al. Analysis of representations for domain adaptation. Advances in neural information processing systems, 19:137, 2007.
296
297
[8] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79(1):551-175, 2010.
298
299
[9] Aharon Ben-Tal, Laurent El Ghaoui, and Arkadi Nemirovski. Robust optimization,, volume 28, Princeton University Press, 2009.
300
301
[10] Aharon Tal, Elad Hazan, Tomer Koren, and Shie Mannor. Oracle-based robust optimization via online learning. Operations Research, 63(3):628-638 2015.
303
[11] Dimitris Bertsimas, Vishal Gupta, and Nathan Kallus. Data-driven robust optimization. Mathematical Programming, 167(2):235 2202)222222222222222 2018
[12] Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classification tasks to new unlabeled sample. Advances in neural information processing systems, 24:2178-2186, 2011.
[13] Stéphane Boucheron, Olivier Bousquet, and Gábor Lugosi. Theory of classification: A survey of some recent advances. ESAIM: probability and statisties, 9:323-375, 2005.
308
[14] Imre Csiszár. Information-type measures of difference of probability distributions and indirect observation. studia scientiarum Mathematicarum Hungarica, 2:229-318, 1967.
310
[15] John Duchi and Hongseok Namkoong. Variance-based regularization with convex objectives. The Journal of Machine Learning Research, 20(1):2455 (K(1):24502504 2019.
312
313
[16] John C Duchi, Peter W Glynn, and Hongseok Namkoong. Statistics of robust optimization: generalized empirical likelihood approach. Mathematics of Operations Research, 2021.
314
315
[17] Dan Hendrycks. Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai. Tyler Zhu Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of nut-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340- 8349, 2021.
316
317
319
[18] Weihua Hu, Gang Niu, Issei Sato, and Masashi Sugiyama. Does distribationally robust supervised learning give robust classifiers? In International Conference Machine Learning, pages 2018
PMLR.
321
[19] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: benchmark in- the-wild distribution shifts. In International Conference on Machine Learning, pages 5637-5664. PMIR, 2021.
323
324
325
326
[20]
Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.
327
328
[21]
Vladimir Koltchinskii et al. Local rademacher complexities and oracle inequalities in risk minimization. The Annals of Statistics, (4(6):2593-2655 2006.
10

---

330
[22] David Krueger, Ethan Caballero. Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang. Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapolation (rex). In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 5815-558 5826. PMLR, 18 24 Jul 2021. URL httpttttt://ppoeedleeee
332
335
334
mlr. www.pprrssvvvviee http//vv1/////u/ V1399/ruuger22a. html
335
[23] Andreas Maurer and Massimiliano Pontil. Empirical bernstein bounds and sample variance penalization. arXiv preprint arXiv:0907.37400 2009.
336
[24] Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In International Conference on Machine Learning, pages 10-18, 2013.
338
339
[25] Hongseok Namkoong and John C Duchi. Variance- regularization with convex objec- tives. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gamett, editors, Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL Paper pdf
342
wwwprip..ccppppppee wttppappe////////ieeeee. /207//iiee/
343
5a1 4444225666155ff01fafttbb///
[26]
Yonatan Oren, Shiori Sagawa, Tatsunori Hashimoto, and Percy Liang. Distributionally robust language modeling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1118-422,222 2019.
345
346
347
348
[27] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pages 5389-5400 PMLR, 2019.
319
350
351
[28] Shiori Sagawa*, Pang Wei Koh*, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2020. URL.ltttps:/pppereviiie het/forummid--ryxxuuttt
352
353
354
[29] Alexander Shapiro. Darinka Dentcheva, and Andrzej Ruszezyúskki Lectures on stochastic progranming:
355
modeling and theory. SIAM, 2014.
[30] Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of-distribution generalization: A survey. arXiv preprint arXiv:2108.1 13244, 2021.
357
358
[31]
Nathan Srebro, Karthik Sridharan, and Ambuj Tewari. Smoothness, low noise and fast rates. In Advances 2010.
in neural information processing systems, pages
360
[32] Vladimir Vapnik. Principles of risk minimization for learning theory. In Advances in neural information processing systems, pages 831 838, 1992.
381
382
[33] Jindong Wang. Cuiling Lan, Chang Liu. Yidong Ouyang, and Tao Qin. Generalizing to unseen domains: A survey on domain generalization. In Zhi- Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, JJCAI-21, pages 4627-4635. International Joint Conferences on Artificial Intelligence Organization, 8 2021. doi: 10. 10.24963/jjai.2022 2021//6 URI. WRRpshttps://aa. org/10 11963//jcai22021/688 2021/628 Survey Track.
363
364
366
387
[34]
Sewall Wright Correlation and causation. 1921.
368
[35] Chuanlong Xie, Haotian Yc, Fei Chen, Yue Liu, Rui Sun, and Zhenguo Li. Risk variance penalization. arXiv preprint (xXiv:000600544 2020.
369
[36] Haotian Ye, Chuanlong Xie, Tianle Cai, Ruichen Li, Zhenguo Li, and Liwei Wang. Towards theoretical framework of oul-of-distriiution generalization. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
372
373
[37] Guojun Zhang, Han Zhao, Yaoliang Yu, and Pascal Poupart. Quantifying and improving transferability in domain generalization. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
374
375
376
[38] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. arXiv preprint 20X3v:2003,02503, 2021..
11

---

Checklist
378
379
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
381
382
(b) Did you describe the limitations of your work? [Yes] We describe the limitation of our work in Section 3.1 and Conclusion.
383
384
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We anticipate negative consequences of our work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
387
388
2. If you are including theoretical results...
389
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
390
(b) Did you include complete proofs of all theoretical results? [Yes] See the Appendix.
391
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or as a URL)? [No] We use the DomainBed benchmark and introduce the implementation of our method in the Appendix. We plan to open the source code to reproduce the experimental results later.
393
395
396
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See the Appendix.
387
398
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] We report error mean and standard deviation. See the Appendix.
400
401
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No]
402
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
404
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
407
408
(d) Did you discuss whether and how consent was obtained from people whose data you'le 're using/curating? [N/A]
409
410
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
412
5. If you used crowdsourcing or conducted research with human subjects...
413
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
414
415
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
416
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
418
12


Reviews:
[{"review_id": "vEP5gNn6Z-E", "reviewer": "Reviewer_eL3F", "summary": "This paper addresses the connection between domain generalization and group distributionally robust optimization. Variance-regualarized ERM is approximated by Group DRO with $\\phi$-divergence ball. Unlike previous works, they take empirical domain distribution as an anchor of uncertainty sets of DRO instead of the uniform distribution, and provides guarantees of in-distribution generalization. They argue that with some conditions, their method can outperform ERM for domain generalization.", "questions": "I cannot fully understand the benefits of using empirical domain distribution as a anchor distribution of $phi$-divergence ball, instead of the uniform distribution. can it be  shown through some experiments?", "limitations": "The author have addressed the limitations.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* This paper did a focused job of generalizing and analysizing the connection between varizance-based regualarization and domain generalization. Their theoretical result for generalization bound for in-distribution seems solid.\n\nWeaknesses\n* Their work seems similar to [1]. The work [1] showed that group DRO formulation with unifrom distribution as a anchor is bounded with risk variance and the equality can be established with some condition. In this paper, the study is extended into general anchor including empirical distribution. However, theorem 3 and 4 can be straightforwardly extended from the results of [1]. \n\n* They argue that a weighting correction scheme can improve generalization in some cases. It would be better to show some experiments to make their arguments valid.\n\n1. Xie, et al. Risk variance penelization, 2020.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "l3bQH4UST-h", "reviewer": "Reviewer_nws1", "summary": "This paper investigated the variance-based regularization method for domain generalization. In particular, the authors originally study how the empirical efficient estimator ($\\hat{\\mathbf{q}}$) effects the generalization performance for a convex hull of domains. The authors first give an generalization bound for in-distribution generalization based on the in-domain variance and out-of-domain variance (Thm. 1). Then, they study the relationship of empirical efficient estimator and the group DRO objectives, and show the equivalence between group DRO and the variance-based regularization (Thm. 3).\n\nOverall, this paper gives some new insights for the variance-based domain generalization, such as the effects of empirical efficient estimator and the in-distribution generalization bound based on variance. However, I still think the theorical results in this paper is insufficient for answer the question: \"Can Variance-Based Regularization Improve Domain Generalization?\".  Besides, the main contribution of studying the effect of empirical efficient estimator seems a bit trivial, since there exists work on variance-based domain generalization [1][2].\n\n[1] Krueger et al.. Out-of-distribution generalization via risk extrapolation (rex). ICML 2021.  \n[2] Xie el tal., Risk variance penalization. arXiv preprint arXiv:2006.07544, 2020.\n", "questions": "I hope the authors to clarify my concerns, but if I am wrong, please correct me.", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. some new insights for the variance-based domain generalization, such as the effects of empirical efficient estimator and the in-distribution generalization bound based on variance.  \n2. Although there exists work on variance-based generalization [3]. the study of variance-based domain generalization is interesting to me.\n3. The theoretical derivations are complete.\n\n[3] Duchi et al., Variance-based regularization with convex objectives 2017\n\n**Weaknesses**\n1. The main concern to me is the significance of the theoretical results. Thm 1 gives an in-domain generalization result (bound the generalization risk of training domains), which is far away from the setting of domain generalization, where the target domain is unseen during training and the discrepancy between the target domain and training domains can be large. Although the authors have discussed this problem in Eq. (9), I still think they should give a generalization bound on the standard setting of domain generalization.\n\n2. Another major concern is the novelty of this paper. To my knowledge, Xie et al. [2] have proposed the relationship between group DRO objective and the variances. I think the novelty of this paper is the study of empirical efficient estimator for variance-based domain generalization, which seems a bit minor.\n\n3. For experimental results in Tab. 1, I notice the results on PACS are amazingly strong (96.9%). The details of experimental setting and analysis for the results is missing, but I think the experiment is important for this work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "A-L9sxdXEu", "reviewer": "Reviewer_QGcy", "summary": "In this paper, the author studied the problem of whether the variance-based regularization approach can improve domain generalization. With a mild assumption of the out-of-distribution data is generated by the shift of the domain distribution, the paper proposed a weighting correction scheme based on the previous variance-regularized domain generalization methods. They prove the guarantees for in-distribution generalization and show the potential advantages of the proposed method when compared to the Empirical Risk Minimization. Experimental results show improvements in some cases.", "questions": "\n1) Although the proposed method has a different assumption compared to invariant learning methods, is there any connections between these two approaches?\n\n2) In Line 18-20, do you assum that the OOD test data is generated by domain shift from one single domain or a mixture of multiple domains?\n", "limitations": "NA", "rating": 5, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\n1) The paper is well-written with strong mathematical support.\n\n2) The paper studies the question of whether variance-based regularization can improve domain generalization. Based on strong theoretical support, they proposed a simple weighting correction scheme and provide guarantees of in-distribution generalization. \n\nWeakness\n\n1) It could be better to include the experiments in the main paper. The main paper did not mention that there are experiments in the Appendix. More experiments can be done to support the proposed idea.\n\n2) Based on the description from Line 194 to 204, the sample size m is critical to the proposed approach. In Line 199, it looks like a larger m could benefit the proposed method while Line 203 shows that the m should not be too large.  An empirical experiment might be required to demonstrate the impact of the sample size m.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2ReNegmqCGQ", "reviewer": "Reviewer_18XN", "summary": "This work studied a variance-based regularization method based on weighting correction for domain generalization. It also theoretically analyzed the potential benefits of the proposed method over ERM, and the connection between the proposed variance-based regularization method and the group DRO problem from an optimization perspective.", "questions": "(1) In section 4, how would the selection of the anchor distribution affect the generalization results?\n(2) More empirical comparison between the proposed method and ERM can be better for supporting the observations in section 3.1.", "limitations": "The authors showed that no potential negative societal impact was observed in this work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n(1) This paper proposed a weighting correction scheme for variance-regularized domain generalization methods inspired by the generalization error bound.\n(2) It theoretically compared the proposed method with ERM.\n(3) It demonstrated that the objective function of the proposed method is equivalent to a group DRO problem from an optimization perspective.\n\nWeaknesses:\n(1) More experiments need to be provided to validate the observations of this paper.\n(2) It is unclear how the generalization gap between the training and test data in Line 101-102 is derived.\n(3) The computational complexity of variance-regularized domain generalization method with weighting correction scheme is not analyzed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject