Pareto Invariant Risk Minimization: Towards Mitigating the Optimization Dilemma in Out-of-Distribution Generalization
Authors: Yongqiang Chen
Kaiwen Zhou
Yatao Bian
Binghui Xie
Bingzhe Wu
Yonggang Zhang
MA KAILI
Han Yang
Peilin Zhao
Bo Han
James Cheng
Venue: ICLR 2023 poster (2023)
Keywords: Out-of-Distribution Generalization
Optimization
Multi-Objective Optimization
Causal Invariance

Abstract: Recently, there has been a growing surge of interest in enabling machine learning systems to generalize well to Out-of-Distribution (OOD) data. Most efforts are devoted to advancing optimization objectives that regularize models to capture the underlying invariance; however, there often are compromises in the optimization process of these OOD objectives: i) Many OOD objectives have to be relaxed as penalty terms of Empirical Risk Minimization (ERM) for the ease of optimization, while the relaxed forms can weaken the robustness of the original objective; ii) The penalty terms also require careful tuning of the penalty weights due to the intrinsic conflicts between ERM and OOD objectives. Consequently, these compromises could easily lead to suboptimal performance of either the ERM or OOD objective. To address these issues, we introduce a multi-objective optimization (MOO) perspective to understand the OOD optimization process, and propose a new optimization scheme called PAreto Invariant Risk Minimization (PAIR). PAIR improves the robustness of OOD objectives by cooperatively optimizing with other OOD objectives, thereby bridging the gaps caused by the relaxations. Then PAIR approaches a Pareto optimal solution that trades off the ERM and OOD objectives properly. Extensive experiments on challenging benchmarks, WILDS, show that PAIR alleviates the compromises and yields top OOD performances.

Published as a conference paper at ICLR 2023
PARETO INVARIANT RISK MINIMIZATION: TOWARDS MITIGATING THE OPTIMIZATION DILEMMA IN OUT- OF-DISTRIBUTION GENERALIZATION
Yonggiang Chen'; Kaiwen Zhou', Yatao Bian", Binghui Xie Bingzhe Wu² 'The Chinese University of Hong Kong Tencent AI Lab 'Hong Kong Baptist University {yachen, kwzhou, bhxi e21 hyang, k ma, jcheng) @cse. cuhk edv. hk
Yonggang Zhang", Han Yang', Kaili Ma Peilin Zhao², Bo Han", James Cheng! (yatao. bian, wwbingzher wubingzheagent @ m com masonzhaole htwp://hhweeeenetttcccmmm com (csygzhang, bhanml] comp. hkbu. edu. hk
ABSTRACT
Recently, there has been a growing surge of interest enabling machine learning systems to generalize well to Out-of -itt--f--Dititttiinn (OOD) data. Most efforts are devoted to advancing optimization objectives that regularize models capture the underlying invariance; however, there often are compromises in the optimizution process of these OOD objectives: i) Many OOD objectives have to be relaxed as penalty terms of Empirical Risk Minimization (ERM) for the ease of optimization, while the relaxed forms can weaken the robustness of the original objective; ii) The penalty terms also require careful tuning of the penalty weights due the intrinsic conflicts between ERM and OOD objectives. Consequently, these com- promises could easily lead to suboptimal performance of either the ERM OOD objective. To address these issues, we introduce a multi-objective optimization (MOO) perspective to understand the OOD optimization process, and propose a new optimization scheme called PAreto Invariant Risk Minimization (PAIR). PAIR improves the robustness of OOD objectives by cooperatively optimizing with other OOD objectives, thereby bridging the gaps caused by the relaxations. Then PAIR approaches a Pareto optimal solution that trades off the ERM and OOD objectives properly. Extensive experiments challenging benchmarks, WILDS, show that PAIR alleviates the compromises and yields top OOD performances.
INTRODUCTION
The interplay between optimization and generalization is crucial to the success of deep learn- ing (Zhang et al., 2017; Arora et al., 2019; Allen-Zhu et al., 2019; Jacot et al., 2021; Allen-Zhu & Li, 2021). Guided by empirical risk minimization (ERM) (Vapnik, 1991), simple optimization algorithms can find uneventful descent paths in the non-convex loss landscape of deep neural networks (Sagun et al., 2018). However, when distribution shifts are present, the optimization is usually biased by spurious signals such that the learned models can fail dramatically in Out-of-Distribution (OOD) data (Beery et al., 2018; 88 DeGrave et al., 2021; Geirhos al., 2020). Therefore, overcoming the OOD generalization challenge has drawn much attention recently. Most efforts are devoted to proposing better optimization objectives (Rojas-Carulla et al., 2018; Koyama& & Yamaguchi. 2020; Parascandolo et al., 2021; 1; Krueger et al., 2021: Creager et al., 2021; Liu et al., 2021; Pezeshki et al., 2021; Ahuja al., 10; Wald et al., 2021; Shi et al., 2022; Rame et al., 2021: Chen et al., 2022b) that regularize the gradient signals produced by FRM, while it has been long neglected that the interplay between optimization and generalization under distribution shifts has already changed its nature.
In fact, the optimization process of the OOD objectives turns out to be substantially more challenging than ERM. There are often compromises when applying the OOD objectives practice. Due to the optimization difficulty, many OOD objectives have to be relaxed as penalty terms of ERM in
Work done during an internship at Tencent AI Lab.
Code is available https: //g wtwb.ccom..cc Phhass/PA IR.

---

Published as a conference paper at ICLR 2023
LooD
1.02
OCO setutio
365
362
164
003
164
CLOVE
-052
-0.35
310
-p14 (a) Theoretical failure case.
CERM
64
(b) Gradient conflicts. (c) Unreliable opt. scheme. (d) Exhaustive tuning.
Figure I: Optimization issues in OOD algorithms. (a) OOD objectives such as IRM usually require several relaxations for the ease of optimization, which however introduces huge gaps. The ellipsoids denote solutions that satisfy the invariance constraints of practical IRM variant IRMv1. When optimized with ERM, IRMv1 prefers fi instead of fIRM (The predictor produced by IRM), (b) The gradient conflicts between ERM and OOD objectives generally exist for different objectives at different penalty weights (x-axis). (c) The typically used linear weighting scheme to combine ERM and OOD objectives requires careful tuning of the weights to approach the solution. However, the scheme cannot reach any solutions in the non-convex part of the Pareto front. In contrast, P PAIR finds an adaptive descent direction under gradient conflicts that leads to the desired solution. (d) Due the optimization dilemma, the best OOD performance (c.g.. IRMv1 w.r.t. modified COLOREDMNIST from Sec. 5) usually requires exhaustive tuning of hyperparameters (g-axis: penalty weights; z-axis: pretraining epochs), while P PAIR robustly yields top performances by resolving the compromises.
practice (Arjovsky et al., 2019; Koyama & Yamaguchi, 2020; Krueger et al., 2021; Pezeshki et al., 2021; Ahuja et al., 202 a; Rame et al., 2021), but the relaxed formulations can behave very differently from the original objective (Kamath et al.. 2021) (Fig. ((a)). Moreover, due to the generally existing gradient conflicts between ERM and OOD objectives (Fig. 1(b)). trade-offs among ERM and OOD performance during the optimization are often needed. Sagawa* al. (2020); Zhai al. (2022) suggest that ERM performance usually needs to be sacrificed for better OOD generalization. On the other hand, it usually requires careful tuning of the OOD penalty hyperparameters (Zhang et al., 2022a) (Fig. (d)), which however either weakens the power of OOD objectives or makes them too strong that prevents models from capturing all desirable patterns. Consequently, using the traditional optimization wisdom to train and select models can easily lead to suboptimal performance of either ERM or OOD objectives. Most OOD objectives remain struggling with distribution shifts or even underperform ERM (Gulrajani & Lopez-Paz, 2021; Koh et al., 2021). This phenomenon calls for better understanding of the optimization in OOD generalization, and raises a challenging question:
How can one obtain a desired OOD solution under the conflicts of ERM and OOD objectives?
To answer this question, we take a multi-objective optimization (MOO) perspective of the OOD optimization. Specifically, using the representative OOD objective IRM (Arjovsky et al., 2019) as an example, we find that the failures in OOD optimization can be attributed to two issues. The first one is the compromised robustness of OOD objectives due to the relaxation in the practical variants. In fact, it can even eliminate the desired invariant solution from the Pareto front w.r.t. the ERM and the OOD penalty (Fig. ((a)). Therefore, merely optimizing the ERM and the relaxed OOD penalty can hardly approach the desired solution. On the other hand, when the Pareto front contains the desired solution, as shown in Fig. (c)), using the traditional linear weighting scheme that linearly reweights the ERM and OOD objectives, cannot reach the solution if it lies in the non-convex part of the front (Boyd & Vandenberghe, 2014). Even when the OOD solution is reachable (i.c., lies in the convex part), it still requires careful tuning of the OOD penalty weights to approach the solution, as shown in Fig. I(d).
To address these issues, we propose new optimization scheme for OOD generalization, called PAreto Invariant Risk Minimization (P (PAIR), which includes new optimizer (PAIR-o) and new model selection criteria (PAIR-S). Owing to the MOO formulation, PAIR-O allows for cooperative optimization with other OOD objectives to improve the robustness of practical OOD objectives. Despite the huge gaps between IRMv1 and IRM, we show that incorporating VREx (Krueger et al., 2021) into IRMv1 provably recovers the causal invariance (Arjovsky et al., 2019) for some group of problem instances (Sec. 3.2). When given robust OOD objectives, PAIR- finds a descent path with adaptive penalty weights, which leads to Pareto optimal solution that trades off ERM and OOD performance properly (Sec. 4). In addition, the MOO analysis also motivates PAIR-S, which facilitates the OOD model selection by considering the trade-off's between ERM and OOD objectives.
We conducted extensive experiments on challenging OOD benchmarks. Empirical results show that PAIR-g successfully alleviates the objective conflicts and empowers IRMvl to achieve high perfor-
2

---

Published as a conference paper at ICLR 2023
mance in 6 datasets from WILDS (Koh al., PAIR-S effectively improves the performance of selected OOD models up to 10% across datasets from DOMAINBED (Gulrajani & Lopez- Paz. 2021), demonstrating the significance of considering the ERM and OOD trade-offs in optimization.
2 BACKGROUND AND RELATED WORK
We first briefly introduce the background of our work (more details are given in Appendix B.1.
Problem setup. The problem of OOD generalization typically considers supervised learning setting based on the data D = {De) JeeE collected from multiple causally related environments Eall, where a subset of samples D {X{, ) from single environment E Eall are drawn independently from an identical distribution P- (Peters et al., 2016). Given the data from training environments (D*) JeE the goal of OOD generalization is find a predictor f X + Y that generalizes well to all (unseen) environments, i.e., to minimize maXecEa (f),, where Le is the empirical risk under environment e. The predictor w &p is usually composed of featurizer yp X + Z that learns to extract useful features, and classifier w: : Z + y that makes predictions from the extracted features.
Existing solutions OOD generalization. There exists a rich literature aiming overcome the OOD generalization challenge, which usually appear as additional regularizations of ERM (Vapnik, 1991). Ganin al. (2016); Sun Saenko (2016); Li et al. (2018); Dou et al. (2019) regularize the learned features to be domain-invariant. Namkoong & Duchi (2016); Hu et al. (2018); Sagawa* et al. (2020) regularize the models to be robust to mild distributional perturbations of the training distri- butions, and Zhang et al. (2022c); Liu et al. (2021); Zhang et al. (2022b); Yao et al. (2022) improve the robustness with additional assumptions. Recently there is increasing interest in adopting the causality theory (Pearl, 2009; Schölkopf et al., 2021) and introducing the causal invariance to representation learning (Peters et al., 2016; Arjovsky et al., 2019; Creager et al., 2021; Parascandolo et al., 2021; Wald et al., 2021; Ahuja et al., 202 They require yp to learn causally invariant representations such that a predictor w acting on minimizes the risks of all the environments simultaneously. This work focuses on resolving the optimization issue in learning the causal invariance. In addition, Koyama & Yamaguchi (2020); Krueger et al. (2021); Shi et al. (2022); Rame et al. (2021) implement the invariance by encouraging agreements various levels across environments. However, they mostly focus on developing better objectives while neglecting the optimization process of the objectives.
Optimization dilemma in OOD generalization. Along with the development of OOD methods, the OOD optimization dilemma is gradually perceived in the literature. Guirajani & Lopez-Paz (2021) find hard to select a proper model in OOD generalization given ERM performance at different environments. Sagawa* et al. (2020); Zhai et al. (2022) find the ERM performance needs to be sacrificed for satisfactory OOD performance. Some initial trials are proposed. Lv et al. (2021) use the guidance of the data from similar distributions with the test environment in MOO to resolve gradient confficts and achieve better performance in domain adaption. Zhang et al. (2022a) propose to construct diverse initializations for stabilizing OOD performance under the dilemma. However, why there exists such a dilemma in OOD generalization and whether we can resolve it remain elusive.
Objettve Optimization (MOO). MOO considers solving objectives w.r.t. {L,) 2 losses, i.e., ming ((0) = (L)( ..., Lm (0)) (Kaisa, 1999). A solution 0 dominates another 0, i.e., L(0)= < L(O), if (0) < (0) for all i and L(O)= + LO)) A solution 0* called Pareto optimal no other 9 dominates A" The set of Pareto optimal solutions is called Pareto set (P) and its image is called Pareto front. In practice, it is usual that one cannot find a global optimal solution for all objectives, hence Pareto optimal solutions are of particular value. Although MOO has been widely applied to improving multi-rask learning (Sener & Koltun, 201 8), it remains underexplored on how to model and mitigate objective conflicts in OOD generalization from the MOO perspective.
3 OPTIMIZATION CHALLENGES IN IRM AND ITS EFFECTIVE FIX
This work focus on one of the most representative OOD objectives in learning the causal invariance IRM, to show how we can understand and mitigate the optimization dilemma through the MOO lens.
3.1 DRAWBACKS OF IRM IN PRACTICE
We first introduce the basics of IRM and the drawbacks of its practical variants, and leave theoretical details in Appendix C. Specifically, the IRM framework approaches OOD generalization by finding
3

---

Published as a conference paper at ICLR 2023
an invariant representation y,, such that there exists classifier acting ip that is simultaneously optimal in Eg. Hence, IRM leads to a challenging bi-level optimization problem as
min 00
Σ (w o (p), s.t. w E arg min L o (p), Ve E Ey.
w:2-y
eEEr
Given the training environments Eu, and functional spaces W for w and @ for y, predictors = o up satisfying the constraint in Eq. 1 are called invariant predictors, denoted as Z(En). When solving for invariant predictors, characterizing I(Er) is particularty difficult in practice, hence it is natural to restrict W to be the space of linear functions on = Rdd (Jacot et al., 1021). Furthermore, Arjovsky al. (2019) argue that linear classifiers actually do not provide additional representation power than scalar classifiers, i... d = 1, W- = S = RR. The scalar restriction elicits a practical variant IRMş as
min Σ (p)) V Velu=ICe(ur (w: 4) 0, Ve E Ei-- 3
Since Eq. 2 remains constrained programming. Arjovsky et al. (2019) further introduce soften- constrained variant, called IRMv1, as the following
min C.) + + Wolww=/Lo(w x))?
(3)
Theoretical failure of practical IRM variants. Although the practical variants seem promising. the relaxations introduce huge gaps between IRM and the practical variants, so that both IRMg and IRMvI can fail to capture the invariance (Kamath et al., 2021). The failure case is illustrated by the two-bit environment with de. 3e E [0, 1]. Each environment D. = {Xe, Y is generated following
Y := (o.5, X = (Xf. X²), Xi= == Y Rad( X5 == Y Rad(
(4)
where (a) is random variable taking value- -1 1 with probability and +1 with probability 1 1- Each environment is denoted as E. = {aa 3e) < 3e < 1] where X is the invariant feature as a is fixed for different environment e, and X is the spurious feature as 3e varies across different
Let Is( (Ext) denote the set of invariant predictors elicited by the relaxed constraint in IRMs. It follows that ((E,) S (Ew). Consequently, there exist some undesired predictors but considered "invariant" by IRMg and IRMv1. For example, in & = {00.1, 1, 0.11). (0.1.0.4)), the solutions satisfying the constraint in IRMs are those intersected points in Fig. 1(a) (The ellipsoids are the constraints). Although f1, JIRM E (Ew), both IRMs and IRMv1 prefer f1 instead of fIRM (the predictor produced by IRM), as fi has the smallest ERM loss. In fact, Kamath al. (2021) show that the failure can happen in a wide range of environments even given infinite amount of environments and samples, demonstrating the huge gap between the practical and the original IRM variants.
Empirical drawback of practical IRM variants. In addition, the optimization of IRMv1 introduces more challenges due the conflicts between the IRMvl penalty and ERM objective. As shown in Fig. 1(d), it often requires significant efforts to tune the hyperparameters such as pretraining epochs and penalty weights in Eq. 3. Otherwise, the IRMv1 penalty could be either too weak to enforce the invariance as required by IRM, too strong that prevents ERM from learning all desirable patterns.
3.2 PARETO OPTIMIZATION FOR IRM
As shown that both IRMs and IRMvl fail to properly trade off between ERM and IRM objectives, we switch to new perspective, i.e., the lens of MOO, to understand the failures of IRM in practice.
Understanding the IRM failures through the MO0 perspective. To begin with, it is natural to reformulate the practical IRM problem (Eq. 3) as an MOO problem:
min(CERM, LIRM)
(5)
where LERM- = I Exee, L. denotes the ERM loss, and LIRM Jet Ee V Vullo== (w 12) denotes the practical IRMv1 loss. To un- derstand the behaviors of solutions to Eq. 5, We visualize the Pareto front w.n.t {L, JeEE, using the previous failure case in Fig. 1(a).
Figure 2: Pareto front of ERM losses w.r.t. environments.
Let P(L, (8), (0), .... Lm (0) denote the set of Pareto optimal solutions w.n.. (C), L. (0)) As shown in Fig. 2, at first, we can find that FIRM P(L,, f2). In other words, solving any
4

---

Published as a conference paper at ICLR 2023
environmenttreweigguee ERM losses cannot obtain fIRM- Moreover, together with Fig. I(a), the failure remains even combined with the IRMg or IRMv1, i.c., FIRM & C2, LIRM), hence matter how carefully control fIRM PCCERMM LIRM). as fIRM is dominated by ft. Therefore, the optimization process, we cannot obtain fIRM by merely minimizing the objectives in Eq. 5. This is essentially because of the weakened OOD robustmess of IRMs and IRMv1 caused by the relaxations. Thus, choosing robust objectives for optimization is of great importance to OOD generalization. The ideal objectives should at least constitute Pareto front that contains the desired OOD solution.
Improving OOD robustness of practical IRM variants. In pursuit of proper optimization objectives, we resort to the OOD
extrapolation explanation of IRM (Bottou et al., 2019). A solution that is simultancously optimal to all training environments (i.e.,
satisfying the original IRM constraints) is also stationary point of ERM loss w.r.t. some OOD distribution:
0C,/0fIMM = 0, L. E 1EEE.
LE dehe EE
Ae 1 1]. (6)
-25
KL--------LD
where L1 is the ERM loss under the OOD distribution. Different Figure 3: Variance distribution.
from Distributionally Robust Optimization approaches (Namkoong & Duchi, 2016), Eq. 6 allows for some negative Å, and hence its solutions are expected to extrapolate better (Bottou et al., 2019).
The previous failure case implies that both IRMs and IRMv1 fail in the extrapolation due to the relaxations, nevertheless, we can introduce additional objectives to directly improve the OOD extrapolation power of the practical IRM variants. To this end, we introduce the REx objective to IRMv1, which is derived by directly minimizing the worst case ERM loss under all OOD distributions up to a certain distance from the training distributions (Krueger et al., 2021). More formally, REx minimizes the worst case L, under an additional constraint of (Ae)eee, 2 -3 in Eq. 6. For the ease of optimization, they also propose an alternative objective as LVrEs := var( [Le (oE,)) ). In Fig. 3, we plot the distribution of LVrFs in the the failure case of Fig. ((a). It can be found that, JIRM lies in the low variance region. Similarly, in Fig. 2, the zero variance solutions (shown the purple line at middle) points out the underlying fIRM beyond the Pareto front. Therefore, incorporating CVRES in Eq. 5 can relocate fIRM into the Pareto front, which implies the desirable objectives as the following
(IRMX)
min (LERM, LIRM LVREX)"
By resolving a large class of failure cases of IRMs and IRMv1 (Kamath et al., 2021), solutions to Eq. are more powerful than those to IRMs and IRMv1 in OOD extrapolation. In fact, we have Proposition 1. (Informal) Under Setting A (Kamath et al. (2021)), for all a E 1), let E = f(a, Bc) Be E (0, 1)} be any instance of the two-bit environment (Eq. 4), Ix denote the invariant predictors produced by Eq. 7, it holds that Ix( I(E).
The formal description and proof of Proposition 1 are given in Appendix E. Proposition I implies that Eq. 7 are the ideal objectives for optimization. However, Eq. can even add up the difficulty of OOD penalty tunning. It introduces front more complicated for the linear weighting scheme to find the desired solution.
more penalty to the overall objective that makes the Pareto
Pareto optimization for IRMX. Ideally, the set of Pareto optimal solutions is small such that each I E P( (PEERMM LIMM LVRES) satisfies the invariance constraints of IRMv1 and VREx, i.e., fIRM = 0 and LVRES = 0, and with minimal LERM, thereby eliciting the desired OOD solutions. However, the ideal constraints might be too strong to be achieved when there are noises among invariant features and labels (Duchin et al., 2020; Ahuja et al., 2021b), which will future enlarge the of Pareto optimal solutions. Therefore, it is natural to relax the constraints as LIRM S fIRM and LVREX S éVREX- When GIR + 0, EVREx - 0, recovers the ideal invariance. To obtain desired solution under these circumstances, the optimization process is expected to meet the following two necessities:
(i). The additional objective in IRMX can make the Pareto front more complicated such that the desired solutions are more likely to appear in the non-convex part, which are however not reachable by the linear weighting scheme (Boyd & Vandenberghe, 2014). Therefore, the optimizer needs to be able to reach any Pareto optimal solutions in the front, e.g.. MGDA algorithms (Désidéri, 2012).²
RReaders might be interested in the necessities of keeping IRMv1 in the objectives. Proposition I considers only the ideal case, we additionally provide more empirical reasons in Appendix C.2; Our results can also be extended to multi-class following typical machine learning theory practice. We leave more sophisticated Pareto front exploration methods (Zhang & Golovin, 2020; Ma et al., 2020) to future investigation.
5

---

Published as a conference paper at ICLR 2023
-2
(a) Ground truth.
(b) IRMv1.
(c) VREx.
(d) PAIR.
Figure 4: Recovery of causal invariance. The causal invariance (Definition. 3.1) requires the model predictions to be independent of the spurious features within the overlapped invariant features. In this example, intuitively it requires the colored belts be perpendicular to z-axis within [-2, It can be found that PAIR succeeds out of IRMvl and VREx in recovering the causal invariance.
(ii.. When both EIRM, ÉVREX > O, there can be multiple Pareto optimal solutions while there are few desired OOD solutions. Hence preference of ERM and OOD objectives is usually needed. As the optimality of each OOD objective usually appears as a necessary condition for satisfactory OOD performance, the preferences for OOD objectives are expected to higher.
Given the requirements, we leverage preference-aware MOO solver solve IRMX for the desired Pareto optimal solution (Mahapatra & Rajan, 2020). We summarize the overall solution as PAreto Invariant Risk Minimization (PAIR). When assigning high preference LIRM and L in IRMX (Eq. 7), PAIR approaches Pareto optimal solution that minimizes the OOD losses while not sacrificing the ERM performance too much, and has good OOD performance, shown as in Table.
3.3 RECOVERY OF CAUSAL INVARIANCE
To better understand how PAIR bridges the gaps between the practical and original IRM objectives, we examine to what extent PAIR can recover the causal invariance specified by Arjovsky et al. (2019) in more difficult case. More formally, the causal invariance is defined as follows.
Definition 3.1. (Causal Invariance) Given predictor f := yp the representation produced by the featurizer is invariant over Eall if and only for all 61, C2 E Ealls it holds that
ED, (X) = =]
ED., (Y|+())= = =],
for all z 1 n 2 where 2 = {x(X))(XX
(upp((D.)
Following Definition 3.1, we construct regression problem. As shown in Fig. 4, Y = sin(x)) + is solely determined by X,, i.e., the values of the z-axis, while X₂ the values y-axis and does not influence the values of Y. Different colors indicate different values of Y. In this problem, the invariant representation op should only take X₃ and discard X2. We sampled two training environments as denoted by the ellipsoids colored in red, among which the overlapped region of the invariant features X1 is [-2, 2, 2]. Hence the prediction produced by the invariant predictor following Definition 3. is expected to be independent of X2. In other words, the plotted belts need be perpendicular the x-axis within the overlapped invariant features [-2, 2]. More details can be found in Appendix C.3.
We plot predictions with the best MSE losses of IRMvl and VREx in Fig. 4(b) and Fig. 4(c), respectively. Although both IRMv1 and VREx fail to achieve the causal invariance as expected, perhaps surprisingly, PAIR almost recovers the causal invariance, as shown in Fig. 4(d).
PARETO INVARIANT RISK MINIMIZATION
The success of PAIR in empowering unrobust IRMvl to achieve the causal invariance of IRM demonstrates the significance of considering the trade-offs between ERM and OOD objectives in the optimization. In the next, we will summarize our findings and elaborate PAIR in more details.
4.1 METHODOLOGY OUTCOMES
Key takeaways from the IRM example. To summarize, the failures of OOD optimization can be attributed to: i) Using unrobust objectives for optimization; ii) Using unreliable scheme to approach the desired solution. Nevertheless, we improve the robustness of the OOD objectives by

---

Published as a conference paper at ICLR 2023
introducing additional guidance such that the desired solution is relocated in the Pareto front w.r.t the new objectives. After obtaining robust objectives to optimize, we then leverage a preference- aware MOO solver to find the Pareto optimal solutions that maximally satisfy the invariance constraints by assigning the OOD objective a higher preference while being aware of retaining ERM performance.
More formally, let food be the desired OOD solution and F be the functional class of fund: a group of OOD objectives Lood = (food) 1 i== are robust if their composite objective Lood satisfies that
(ood) 3 ood (f), VJ + food E F,
(8)
When given robust OOD objective Lood, target is to solve the following MOO problem
miny (CERM, Lood)"
where Lodd corresponds to Cood-rllxxe invariance constraint Lood Lood (ooa(()) (f), Vf * faod E F. Denote the <inv as empirical loss of using the underlying invariant features to predict labels, then the optimal values of the desired OOD solution w.r.t. Eq. 9 are (finv, food)" )" = (CERM((aoa) Lood(food)
Good
which corresponds to an ideal preference (or OOD prefer- ence) for the objectives, that is Pond = (fime sood) The optimal solutions of Eq. 9 that satisfy the exact Pareto optimality, i.ed Li Poody j.jz V VLi, L L, are expected recover sood in Eq. 8.
PAIR-O as an optimizer for OOD generalization. To find a desired Pareto optimal solution specified by Pood, we adopt a 2-stage optimization scheme, which consists of two phases, i.e., the "descent" and the "balance" phase, following the common practice (Gulrajani & Lopez-Paz, 2021).
In the "descent" phase, we train the model with the ERM loss such that it approaches the Pareto front by merely minimizing LERM first. Then, in the "balance" phase, we adjust the solution to maximally satisfy the exact Pareto optimality specified by Pood. We adopt the off-the-shelf preference-awe MOO solver EPO (Mahapatra & Rajan, 2020) to find the desired Pareto optimal solutions with the given Pood- Specifically, at each step: Pood implies descent direction gb that maximally increase the satisfaction to the exact Pareto optimality. Then, we will find an objective weight vector to reweight both the ERM and OOD objectives (thus their gradients), such that the reweighted descent direction Jase has a maximum angle with gb. Meanwhile, to avoid divergence, Idsc also needs to guarantee that it has a positive angle with the objective that diverges from the preferred direction most. We provide detailed descriptions and theoretical discussions of the algorithm in Appendix D.I.
PAIR-3 for OOD model selection. Model selection in OOD generalization is known to be challeng- ing. the validation data used to evaluate the model performance is no longer necessarily identically distributed to the test data (Gulrajani & Lopez-Paz, 2021). The IRM example also implies that the traditional model selection methods that merely depends on the validation performance, i.e., the ERM performance, can easily compromise OOD performance due to the conflicts with ERM objective, especially when the validation set has large gap between the test set (cf. CMNIST in Table 3).
When given no additional assumption, we posit that the OOD loss values can serve as a proxy for OOD performance, which essentially corresponds the underlying prior assumed in the OOD methods. It naturally resembles PAIR optimization therefore motivates PAIR-s. s. PAIR-S jointly considers and trades off the ERM and OOD performance in model selection, and select models that maximally satisfy the exact Pareto optimality. We leave more details and discussions in Appendix D.2.
4.2 THEORETICAL DISCUSSIONS AND PRACTICAL CONSIDERATIONS
Essentially both PAIR- o and PAIR- aim to solve Eq. 9 up to the exact Pareto optimality. However, in practice, the ideal preference is usually unknown and the exact Pareto optimality could be too strict to achieve. Therefore, we develop an e-approximated formulation of Eq. 9, i.ee.//oos; P Cyl S VLC,, Lj E L, which might be of independent interest. Built upon the relaxed variant, we analyze the OOD performance of PAIR in terms of sample complexity, given the empirical risk and imprecise OOD preference, and prove the following Theorem in Appendix E.2.
Theorem 4.1. (Informal) For E (0,1) 1) and any 8,8 > O, J is a finite hypothesis class, both ERM and OOD losses are bounded above, let IPAR be the index of all losses, Pmax max P; and
Lenax = maxic Irame L;, if the number of training samples D 2 (32Lmm /8") log[2(m - + 1)]Fl/n [/7), then with probability at least 7, PAIR-O and PAIR-S yield an 6-approximated solution of food-
Practical Considerations. Theorem 4.1 establishes the theoretical guarantee of PAIR-O and PAIR-S given only an imprecise OOD preference. Empirically, we find that assigning a large
7

---

Published as a conference paper at ICLR 2023
enough preference to the OOD objectives is generally sufficient for PAIR-o to find desired OOD solution. For example, in most experiments PAIR-O yields a satisfactory OOD solution with a relative preference of (1, 1010, 1e12) for ERM, IRMv1, and VREx. For PAIR-S, the empirical upper bounds of (Cinv. Food) from the running history and adjust OOD preference to be slightly larger. We provide detailed discussion on the preference choice in practice in Appendix D.3.
estimate
Besides, the requirement of whole network gradients in PAIR-O can be a bottleneck when deployed models that have a prohibitively large number of parameters (Sener & Koltun, 2018). To this end, we can use only the gradients of classifier w to solve for the objective weights, or freeze the featurizer after the "descent" phase to further reduce the resource requirement (Zhang al., 2022a). We discuss more practical options and how PAIR can be applied to other OOD methods in Appendix D.4.
5 EXPERIMENTS
We conduct extensive experiments on COLOREDMNIST, WILDS and DOMAINBED to verify the effectiveness of PAIR-O o and PAIR-s S in finding better OOD solution under objective conflicts.
Proof of concept on COLOREDMNIST. In Table I,
Table I: OOD Performance on COLOREDMNIST
compare PAIR-o implemented with IRMX to other strong baselines on COLOREDMNIST (CM- NIST) and the failure case variant (Kamath al., 2021) (CMNIST-m), We follow the evaluation setup as in IRM (Arjovsky et al., 2019) and report the re- sults from 10 runs We assign relative preference 1e01, le12) to ERM, IRMv1 and VREx objec- tives, respectively. It can be found that PAIR-O significantly improves over IRMv1 across all environ- ment settings, while IRMX using the linear weighting scheme performs worse than PATR-o, confirming the effectiveness of PAIR-OO Interestingly, using only the gradients of the classifier vr in PAIR-O can yield competitive performance as that uses f or şo, while the former has better scalability. Therefore, we will use PAIR- ow in the following experiments. More details are given in Appendix F.I.
Method
CMNIST
CMNIST-m
Avg.
ERM
17.1 + 0.9
73.3 + 0.9
45.2
IRMvl
67.3 t 1.9
76.8
72.1
V-REx
68.6
1.
75.8
IRMX
65.8
81.6
73.7
PAIR-Oy
68.6
83.7 1.2
76.2
PAIR-OP
68.6 0.8
83.7 1.2
76.2
PAIR-O.
0.7
83.7 + 1.2
76.5
Oracle
72.2 1: 0.2
86.5 + 0.3
79.4
Optimum
75
90
82.5
Chance
50
50
50
Table 2: OOD generalization performances on WILDS benchmark.
CAMELYON17
COMMENTS
FMoW
IWILDCAM
POVERTYMAP
RXRXI
Avo. RANKLL)
Avg. acc. (%)
Worst acc. (%)
Worst acc. (%)
Macro F1
Worst Pearson
Avg. acc. (%)
ERM
70.3 (16-))
(6.06)))))
32.3 (1125)))
30.8 (17.0)
0.45 (2000)
4.50
CORAL
(4727)
06.6 (1.1.)
31.7 (11.20
32.7 (1:0.)
(2
28.4 (22.f)
5.50
GnapDRO
68.4 (t7.9)
70.0 (::2.)
30.8 (z:08)
23.8 (#:2.)
0.39 (=20.6)
23.0 (:20.)
6.83
IRMv1
(+8.1)
66.3 (#2.f)
350.0 (::1.77
15.1 (#:49)
(=00077)
(::0.)
V-RES
71.5 (+:.3
64.9 (+:12)
27.2 (+:078)
27.6 (1t0.7
0.40 (=0006)
7.5 (1:0.)
7.00
Fish
(427)
(1:02)
31.6 (#:051)
21.8 (1:07)
(=0005)
10.1 (#:7.5)
4.33
LISA
74.7 (+67)
(+:.0)
333.5 (+0.70)
24.0 (+0.)
0.48 (=0007)
31.9 (#00.5
2.67
IRMX
(160)
(1:38)
33.7 (1:92))
26.6 (1:09)
0.45 (2M0M)
28.7 (2m2)
4.00
PAIR-O
(+7.0)
(
35.5 (1:111)
27.9 (#a07)
0.47 (20.06)
28.8 (=0.))
2.17
"Averaged rank is reported because of the dalasel belerogeneity. lower runk better.
Can PAIR-O effectively find better OOD solutions under realistic distribution shifts? We evaluate PAIR-o implemented with IRMX on 6 challenging datasets from WILDS benchmark (Koh al., 2021), and compare PAIR-o with other state-of-the-art OOD methods from different lines (Sec. 2). including CORAL (Sun & Saenko, 2016), GroupDRO (Sagawa* et al., 2020), IRM (Arjovsky et al., 2019), V-REx (Krueger et al., 2021), Fish (Shi et al., 2022) and an advanced importance- data augmentation method LISA (Yao et al., 2022). By default, we assign a relative preference (1, 10 12) to ERM, IRMv1 and VREx objectives, respectively, and restrict the search space of the preference. Our implementation and evaluation protocol follow the exact configuration previous works (Koh et al., 2021; Shi et al., 2022; Yao et al., 2022). Details can be found in Appendix F.3.
Table 2 shows that PAIR-O substantially improves over IRMv1 as well IRMX and yields top- ranking OOD performance among all tfate-of-the-a methods across different realistic distribution shifts, demonstrating the effectiveness and significance of resolving the optimization dilemma in OOD generalization. Besides, the advances PAIR over IRMX also confirm the effectiveness of PAIR-O in finding better trade-off between ERM and OOD objectives.
8

---

Published as a conference paper at ICLR 2023
l0g 1010
0.85
ERM
0.84
0.90
Acc TD
V124
1w0
0.75
3.82
163-
1e7
00.0
weights
100
200
300
400
5,0
0.00
0.50
1e5-
BRM
0.78
165
60.55
000
0.50
(a) PAIR v.s. IRMX.
(b) Pemalty weights trajectory.
(c) Normalized losses.
(d) Preference sensitivity.
Figure (a) Each point is the best performed IRMX among corresponding pretraining epoch (r-axis),
the IRMvl penalty weights (y-axis) and all possible VREx penalty weights. Despite the substantial tunning efforts, IRMX performs no better than PAIR. That because (b) PAIR can adaptively adjust the penalty weights during the optimization process, and leads to (c) Pareto optimal solution. (d) The robustness of PAIR-O o to different preference choices enables it adaptable to various scenarios.
How can PAIR-O mitigate the objective conflicts? We conduct ablation studies with the modified COLOREDMNIST (More details and results are given in Appendix F.2). First, shown in Fig. 5(a), PAIR-O effectively finds a better solution than exhaustive tuning of penalty weights in IRMX. That because PAIR can adaptively adjust the penalty weights (Fig. 5(b)), which leads to Pareto optimal solution that has lower OOD losses while not compromising the ERM loss too much (Fig. 5(c)). The other reason is that, PAIR-o is generally robust different choices of preference choices (Fig. 5(d)), which makes it adaptable to various scenarios, confirming our discussions in Sec. 4.2.
Can PAIR-s effectively select better OOD solutions under realistic distribution shifts? To verify the effecctiveness of PAIR-S, we apply PAIR-s s to multiple representative OOD methods as discussed in Sec. 2, and examine whether PAIR-S can improve the model selections under rigorous hyperparameters tunning (Gulrajani & Lopez-Paz, 2021) on COLOREDMNIST (Kamath et al., 2021), PACS (Li et al., 2017) and INCOGNIT (Beery et al., 2018). Intuitively, models selected merely based on ERM performance tend to have high preference or better performance on environments that have a similar distribution of the corresponding validation set, which will lead to higher variance of performances at different environments or lower worst environment performance. Hence we use training-domie domain validation accuracy for COLOREDMNIST and TERRAINCOGNITS and test-domain validation accuracy for PACS validate the existence of this issue under different scenarios (Teney et al., 2021). More details and results are provided in Appendix G.
Table 3: OOD generalization performances using DOMAINBED evaluation protocol.
COLOREDMNIST
PACS 1
TErrAINcoGNITA
PAIR-s
+90%
+80%
10%
A WT.
A
C
P
8
A Wr.
I.100
1,38
144
1.66
A wr.
71.0
73.4
87.2
79.5
76.9
46.7
57.4
39.7
ww..............................
DANN
71.0
73.4
10.0
86.5
79.9
97.1
75.3
46.1
41.2
56.7
35.6
DANN
71.6
73.3
10.9
+0.9
87.0
81.4
96.8
77.5
+2.2
43.1
41.1
55.2
38.7
+-31
GrouDRRO
72.6
87.7
79.6
48.4
40.3
57.9
40.0
GRou
72.7
73.2
13.0
+3.1
86.7
83.2
97.8
81.4
+1.8
45.4
40.3
57.9
40.0
+0.0
IRMvI
72.33
72.6
9.9
82.3
80.8
95.8
78.9
48.4
35.6
05.4
40.1
IRMv1
67.4
64.8
24.2
14.3
85.3
81.7
97.4
79.7
10.8
40.4
38.3
48.8
37.0
1.4
www.............................
www.............................
www.............................
................................
Fisha
72.2
81.6
49.2
10.6
57.9
10.4
Fishr
09..
70.9
22.6
+ 12.7
87.4
82.6
97.5
82.2
+0.6
51.0
40.7
58
40.8
+0.3
UUsin the training domain validation uecuracy. Ussin the test domain validation accuracy.
Table shows that there is a high variance in the performances different environments of the models selected only based on the validation accuracy. In contrast, by jointly considering and trading off the ERM and OOD performances in model selection, PAIR-S substantially mitigates the variance by improving the worst environment performance of all methods under all setups up to 10%. It could serve as strong evidence for the importance of considering ERM and OOD trade-offs.
CONCLUSION
In this work, we provided new understanding of optimization dilemma in OOD generalization from the MOO perspective, and attributed the failures of OOD optimization to the compromised robustness of relaxed OOD objectives and the unreliable optimization scheme. We highlighted the importance of trading off the ERM and OOD objectives and proposed new optimizer PAIR-O and new model selection criteria PAIR-S s to mitigate the dilemma. We provided extensive theoretical and empirical evidence to show the necessity and significance of properly handling the ERM and OOD trade-offs.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
We thank the reviewers for their valuable comments. This work was supported by CUHK direct grant 4055146. YZ and BH were supported by the NSFC Young Scientists Fund No. 62006202, Guangdong Basic and Applied Basic Research Foundation No. 2022A151501 RGC Early Career Scheme No. 22200720, and Tencent AI Lab Rhino-Binn Bird Gift Fund.
ETHICS STATEMENT
Considering the wide applications and high sensitivity of deep neural networks to distribution shifts and spurious correlations, it is important to develop new methods that are able to generalize to OOD data, especially for some cunan-eentered AI scenarios such as autopilot and social welfare. By understanding and mitigating the optimization dilemma in OOD generalization, our work could serve as an initiate step towards new foundation of optimization for OOD generalization, with the hope for building more trustworthy and AI systems facilitate broader AI applications and social benefits. Besides, this paper does not raise any ethical concerns. This study does not involve any human subjects, practices to data set releases, potentially harmful insights, methodologies and applications, potential conflicts of interest and sponsorship, iisseimination/biisfaaaane concern,, privacy and security issues, legal compliance, and research integrity issues.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of theoretical results, provide detailed proofs for our proposi- tions and theorems in Appendix E. To ensure the reproducibility of our methods and experimental results, provide detailed deseription of the IRM in Appendix C.I, the algorithms D, and the experimental setting in Appendix F, in addition to the main text. Besides, we will further provide link to anonymous repository that contains the source codes for reproducing the results in our paper during the discussion phase.
10

---

Published as a conference paper at ICLR 2023
REFERENCES
Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, Sean-Christophe Gagnon-Audet, Yoshua Bengio, Ioannis Mitliagkas, and Irina Rish. Invariance principle meets information bottleneck for out-of- distribution generalization. In Advances in Neural Information Processing Systems, 2021a. (Cited on pages 1, 2, 3, 19, 23, 31, 33, 40 and 42)
Kartik Ahuja, Jun Wang, Amit Dhurandhar, Karthikeyan Shanmugam, and Kush R. Varshney. Empirical or invariant risk minimization? sample complexity perspective. In International Conference on Learning Representations, 2021b. (Cited on pages and 30)
Zeyuan Allen-Zhu and Yuanzhi Li. Feature purification: How adversarial training performs robust deep learning. In IEEE Annual Symposium on Foundations of Computer Science, pp. 977-988, 2021. (Cited on page 1)
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. Advances in Neural Information Processing Systems, pp. 6155-6166, 2019. (Cited on page 1)
Martín Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Pazz Invariant risk minimization. arXiv preprint 1rXiv: 2019. (Cited pages 2, 3, 4. 6, 8, 19, 21, 22, 23, 24, 30, 31, 32, 36,3 37 and 46)
Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. International Conference on Machine Leurning, pp. 322-332, 2019. (Cited on page 1)
Péter Bándi, Osear Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, Quanzheng Li, Farhad Ghazvinian Zanjani, Svitlana Zinger, Keisuke Fukuta, Daisuke Komura, Vlado Ovtcharov, Shenghua Cheng, Shaoqun Zeng. Jeppe Thagaard, Anders B. Dahl, Huangjing Lin, Hao Chen, Ludwig Jacobsson, Martin Hedlund, Melih Çetin, Eren Halici, Hunter Jackson, Richard Chen, Fabian Both, Jörg Franke, Heidi Küsters- Vandevelde, Willem Vreuls, Peter Bult, Bram van Ginneken, Jeroen van der Laak, and Geert Litjens. From detection individual metas- tases to classification of lymph node status at the patient level: The CAMELYON17 challenge. IEEE Trans. Medical Imaging, 38(2)5500-560 2019. (Cited on page 43)
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Computer Vision European Conference, Part XVI, volume 1 11220, pp. 472-489, 2018. (Cited on pages 1, 9 and 46)
Sara Beery, Elijah Cole, and Arvi Gjoka. The iwildcam 2020 competition dataset. arXiv preprint arXiv:2004. 10340, 2020. (Cited on page 44)
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion of The 2019 World Wide Web Conference, pp. 491-500, 2019. (Cited on page 43)
Léon Bottou, Martin Arjovsky, Ishaan Gulrajani, and David Lopez-Paz. Learning representations using causal invariance. Keynote in International Conference on Learning Representations, 2019. URL hwtps:tp////. http://eeooooooooooooeoo org atte wwwiiiiaiinncoo ances. (Cited on page 5)
Stephen P. Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, 2014. (Cited on pages& and 5)
Yongqiang Chen, Han Yang, Yonggang Zhang, Kaili Ma, Tongliang Liu, Bo Han, and James Cheng. Understanding and improving graph injection attack by promoting unnoticeability. In International Conference Learning Representations, 2022a. (Cited on page 20)
Yongqiang Chen, Yonggang Zhang, Yatao Bian, Han Yang, MA KAILI, Binghui Xie, Tongliang Liu, Bo Han, and James Cheng. Learning causally invariant representations for out-of-distributte generalization on graphs. In Advances in Neural Information Processing Systems, 2022b. (Cited pages 1, 19 and 31)
11

---

Published as a conference paper at ICLR 2023
Gordon A. Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 6172-6180, 2018. (Cited page 43)
Elliot Creager, Jörn-Henrik Jacobsen, and Richard S. Zemel. Environment inference for invariant learning. In International Conference on Machine Learning, volume 139, pp. 2189-2200, 2021. (Cited on pages 1, 3, 19 and 31)
Alex J. DeGrave, Joseph D. Janizek, and Su-In Lee. Al for radiographic COVID-19 detection selects shortcuts over signal. Nature Machince Intelligence, 3(7):610-619, 2021. (Cited on page I)
Qi Dou, Daniel Coelho de Castro, Konstantinos Kamnitsas, and Ben Glocker. Domain generalization via madel-agnostic learning of semantic features. In Advances in Neural Information Processing Systems, pp. 6447-6458, 2019. (Cited on pages 3, and 31)
John C. Duchin, Tatsunori Hashimoto, and Hongseok Namkoong. Distributionally robust losses for latent covariate mixtures. arXiv preprint arXiv:2007. 2020. (Cited on page 5)
Jean-Antoine Désidéri. Multiple-gradient descent algorithm (mgda) for multiobjective optimization. Comptes Rendus Matheattque,, 350(5)313-313, 2012. (Cited on pages and 20)
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor S. Lempitsky. Domain-adversarial training of neural networks. Journal of Mache Learning Research, 17:59:1-59:35, 2016. (Cited on pages 3, 18, 29, 31 and 46)
Robert Geirhos, Jörn- Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge. and Felix A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):65--67, 2020. (Cited on page 1)
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations, 2021. (Cited on pages 2, 3, 18, 19, 20, 24, 26, 28, 37, 45, 46 and 47)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016. (Cited on pages 44 and 46)
Weihua Hu, Gang Niu, Issei Sato, and Masashi Sugiyama. Does distributionally robust supervised learning give robust classifiers? In International Conference on Machine Learning, pp. 2034-2044 2042, 2018. (Cited pages 3, 19 and 31)
Gao Huang. Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 2261-2266 2017. (Cited on pages 30, 43 and 44)
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: convergence and generalization in neural networks. In Annual ACM SIGACT Symposium Theory of Computing, pp. 6, 2021. (Cited on pages 1, and 21)
Miettinen Kaisa. Nonlinear Multiobjective Optimization. Springer, 1999. (Cited on pages 3, 20 and 45)
Pritish Kamath, Akilesh Tangella, Danica Sutherland, and Nathan Srebro. Does invariant risk minimization capture invariance? In International Conference Artificial Intelligence and Statistics, pp. 4069-4077, 2021. (Cited on pages 2, 4, 8, 8, 9, 21, 22, 23, 31, 32 and 33)
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations, 2015. (Cited pages 24 and 37)
12

---

Published as a conference paper at ICLR 2023
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsub- ramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara M. Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A benchmark of in-the-will wiil distribution shifts. In International Conference Machine Learning. pp. 5637-5664, 2021. (Cited on pages 2, 3, 8, 18, 39, 43 and 44)
Masanori Koyama and Shoichiro Yamaguchi. duittf-distribution generalization with maximal invariant predictor. arXiv preprint arXiv:2008.0/883 2020. (Cited on pages 1, 3, 19, 23, 31 and 40)
David Krueger, Ethan Caballero,
Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Rémi Le Priol, and Aaron C. Courville. Out-of-distribution generalization via risk extrapolation (rex). In International Conference on Machine Learning, pp. 5815-5826, 2021. (Cited on pages 1, 2, 2, 19, 23, 29, 31, 32, 36 and 42)
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE,
1998. (Cited on page 46)
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Deeper, broader and artier domain generalization. In International Conference Computer Vision, pp. 5543-5551, 2017. (Cited on pages and 46)
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. European Conference on Computer Vision, pp. 647-663, 2018. (Cited on pages 3, 18 and 31)
Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qingfu Zhang, and Sam Kwong. Pareto multi-task learning. In Advances in Neural Information Processing Systems, pp. 12037-12047, 2019, (Cited pages 20, 29 and 30)
Yong Lin, Hanze Dong, Hao Wang, and Tong Zhang. Bayesian invariant risk minimization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16008 2022a. (Cited on page 19)
Yong Lin, Shengyu Zhu, and Peng Cui. ZIN: when and how learn invariance by environment inference? arXiv preprint 20Xhv:2203.058/8, 2022b. (Cited on page 19)
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pp. 6781-6792, 2021. (Cited on pages 1, 3 and 19)
Suyun Liu and Luis Nunes Vicente. The stochastic multi-gradient algorithm for multi-objective optimization and its application to supervised machine learning. Annals of Operations Research, 2021. (Cited on page 20)
Fangrui Lv, Jian Liang, Kaixiong Gong, Shuang Li, Chi Harold Liu, Han Li, Di Liu, and Guoren Wang. Pareto domain adaptation. In Advances in Neural Information Processing Systems, 2021. (Cited on pages 3, 19 and 29)
Kaili Ma. Haochen Yang, Han Yang, Tatiana Jin, Pengfei Chen, Yongqiang Chen, Barakeel Fanseu Kamhoua, and James Cheng. Improving graph representation learning by contrastive regularization. arXiv preprint, 1rXiv:2101.11525 2021. (Cited page 20)
Pingchuan Ma, Tao Du, and Wojeiech Matusik. Efficient continuous pareto exploration multi-task learning. In International Conference Machine Learning, Proceedings of Machine Learning Research, pp. 6522-6531, 2020. (Cited on pages 5 and 20)
Divyat Mahajan, Shruti Tople, and Amit Sharma. Domain generalization using causal matching. In International Conference on Machine Learning, volume 139, pp. 7313-7324, 2021. (Cited on page 19)
13

---

Published as a conference paper at ICLR 2023
Debabrata Mahapatra and Vaibhav Rajan. Multi-task learning with preferences: Gradient descent with controlled ascent in pareto optimization. In International Conference on Machine Learning, pp. 6597-6607, 2020. (Cited pages 6, 7, 20, 26, 27, 28, 29, 30, 34 and 38)
Hongseok Namkoong and John C. Duchi. Stochastic gradient methods for distributionally robust optimization with f-divergences. In Advances in Neural Information Processing Systems, pp- 2208-2216, 2016. (Cited on pages 3, 5, 19 and 31)
Giambattista Parascandolo, Alexander Neitz, Antonio Orvieto, Luigi Gresele, and Bernhard Schölkopf. Learning explanations that are hard to vary. In International Conference on Learning Representations, 2021. (Cited on pages 1, 3, 19 and 31)
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,o Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, pp. 8024 8035, 2019. (Cited on page 45)
Judea Pearl. Causality. Cambridge University Press, 2 edition, 2009. (Cited on pages 3 and 19)
Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 78(5):947-1012, 2016. (Cited on pages 3, 18, 19, 23 and 31)
Jonas Peters, Dominik Janzing, and Bernhard Schlkopf. Elements of Causal Inference: Foundations and Learning Algorithms. The MIT Press, 2017. ISBN 0262037319. (Cited on page 19)
Mohammad Pezeshki, Sékou-Oumar Kaba, Yoshua Bengio, Aaron Courville, Doina Precup, and Guillaume Lajoie. Gradient starvation: A learning proclivity in neural networks. In Advances in Neural Information Processing Systems, pp. 1256-1272, 2021. (Cited on pages 2, 23, 31 and 40)
Alexandre Rame, Corentin Dancette, and Matthieu Cord. Fishr: Invariant gradient variances for cuitof-distribution generalization. arXiv preprint arXiv:2109,02934, 2021. (Cited on pages 1, 2, 3, 18, 19, 23, 29, 31, 40, 42, 46 and 47)
Mateo Rojas-Carulla, Bernhard Schölkopf, Richard Turner, and Jonas Peters. Invariant models for causal transfer learning. Journal of Machine Learning Research, 19(36):1-34, 2018. (Cited on pages 1, 19 and 31)
Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. Domain-adjusted regression or: Erm may already learn features sufficient for out-of-distribution generalization. arXiv preprint TrXiv:2202.06856. 2022. (Cited pages 18 and 30)
Bashir Sadeghi, Sepehr Dehdashtian, and Vishnu Boddeti. On characterizing the trade-off in invariant representation learning. Transactions on Machine Learning Reseurch, 2022. ISSN 2835-8856. (Cited on page 19)
Shiori Sagawa*, Pang Wei Koh*, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2020. (Cited on pages 3, 19, 29, 31 and 46)
Levent Sagun, Utku Evci, V. Ugur Güney, Yann N. Dauphin, and Léon Bottou. Empirical analysis of the hessian of parametreee neural networks. In International Conference on Learning Representations, Workshop Track Proceedings, 2018. (Cited page
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert. distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019. (Cited on pages 30, 43 and 44)
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEFF, 109(5);612- (255)661-3344 634, 2021. (Cited on pages and 19)
14

---

Published as a conference paper at ICLR 2023
Ozan Sener and Vladlen Koltun. Multi-task learning multi-objective optimization. In Advances in Neural Information Processing Systems, pp. 2018. (Cited on pages 3, 8, 20, 29 and 30)
Ozan Sener and Vladlen Koltun. Domain generalization without excess empirical risk. In Advances in Neural Information Processing Systems, 2022. (Cited on page 19)
Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning From Theory to Algorithms. Cambridge University Press, 2014. (Cited on page 35)
Yuge Shi, Jeffrey Seely, Philip Torr, Siddharth N, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. In International Conference on Learning Representations, 2022. (Cited on pages 1, 3, 8, 19, 29, 31, 44 and 45)
Baochen Sun and Kate Saenko. Deep CORAL: correlation alignment for deep domain adaptation. In European Conference on Computer Vision, volume 9915, pp. 443 453-500,45 2016. (Cited on pages 8, 18 and 31)
Iyya Sutskever, James Martens, George E. Dahl, and Geoffrey E. Hinton. On the importance of initialization and momentum deep learning. In International Conference on Machine Learning, pp. 1139-1147, 2013. (Cited on page 38)
James Taylor, Berton Earnshaw, Ben Mabey, Mason Victors, and Jason Yosinski. RxIxl: An image set for cellular morphological variation across many experimental batches. In International Conference Learning Representations, 2019. (Cited on page 44)
Damien Teney, Ehsan Abbasnejad, Simon Lucey, and Anton den Hengel. Evading the simplicity bias: Training a diverse set of models discovers solutions with superior OOD generalization. arXiv preprint orXiv:2105.05612. 2021. (Cited on pages 46 and 47)
Damien Teney Yong Lin, Seong Joon Oh, and Ehsan Abbasnejad. ID and OOD performance are sometimes inversely correlated on real-world datasets. arXiv preprint arXiv:2209.0065 2022. (Cited on page 19)
Vladimir Vapnik. Principles of risk minimization for learning theory. In Advances in Neural Information Processing Systems, pp. 831-838, 1991. (Cited on pages 1, 18 and 46)
Yoav Wald, Amir Feder, Daniel Greenfeld, and Uri Shalit. On calibration and out-of-domain generalization. In Advances in Neural Information Processing Systems, pp. 2215-2227, 2021. (Cited on pages 3, 19, 23, 28, 31 and 40)
Sang Michael Xie, Ananya Kumar, Robbie Jones, Fereshte Khani, Tengyu Ma, and Percy Liang, In- n-out: Pre-training and self-training using auxiliary information for out-of--istribun robustness. In International Conference on Learning Representations, 2021. (Cited on page 19)
Huaxiu Yao, Yu Wang. Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn. Improving out-of-distribution robustness via selective augmentation. In International Conference on Machine Learning, pp. 25407-25437, 2022. (Cited on pages 8, 19 and 44)
Christopher Ych, Anthony Perez, Anne Driscoll, George Azzari, Zhongyi Tang, David Lobell, Stefano Ermon, and Marshall Burke. Using publicly available satellite imagery and deep learning to understand economic well-being in africa. Nature Communications, 11(2583), 2020. (Cited on page 44)
Runtian Zhai, Chen Dan, J. Zico Kolter, and Pradeep Ravikumar. Understanding why generalized reweighting does not improve over ERM. arXiv preprint arXiv:2201. 12293, 2022. (Cited on pages 2, and 19)
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understand- ing deep learning requires rethinking generalization. In International Conference on Learning Representations, 2017. (Cited page 1)
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. International Conference on Learning Representations, 2018. (Cited on pages 29 and 44)
15

---

Published as a conference paper at ICLR 2023
Jianyu Zhang, David Lopez-Pazz Paz, and Léon Bottou. Rich feature construction for the optimization- generalization dilemma. arXiv preprint arXi::2203.15516, 2022a. (Cited on pages 2, 3, 8, 19, 30 and 40)
Michael Zhang, Nimit Sharad Sohoni, Hongyang Zhang, Chelsea Finn, and Christopher Ré. Correct-n-contrast: a contrastive approach for improving robustness to spurious correlations. In International Conference on Machine Learning, volume 162, pp. 26484-26516, 2022b. (Cited on pages 3 and 19)
Richard Zhang and Daniel Golovin. Random hypervolume scalarizations for provable multi-objective black box optimization. In International Conference on Machine Learning, pp. 11096-11105, 2020. (Cited pages 5 and 20)
Yonggang Zhang, Mingming Gong, Tongliang Liu, Gang Niu, Xinmei Tian, Bo Han, Bernhard Schölkopf, and Kun Zhang. Causaladv: Adversarial robustness through the lens of causality. 2022c. (Cited on page
Han Zhao, Remi Tachet des Combes, Kun Zhang, and Geoffrey J. Gordon. On learning invariant representations for domain adaptation. In International Conference on Machine Learning, pp.
7523-7532, 2019. (Cited on page 19)
Han Zhao, Chen Dan, Bryon Aragam, Tommi S. Jaakkola, Geoffrey J. Gordon, and Pradeep Raviku- mar. Fundamental limits and tradeoffs in invariant representation learning. arXiv preprint arXiv:2012.10713, 2020. (Cited on page 19)
Peilin Zhao and Tong Zhang. Stochastic optimization with importance sampling for regularized loss minimization. In International Conference Machine Learning, pp. 1-9, 2015. (Cited on pages 27 and 38)
Kaiwen Zhou, Fanhua Shang, and James Cheng. A simple stochastic variance reduced algorithm with fast convergence rates. In International Conference on Machine Learning, pp. 5980-5989, 2018. (Cited on page 27)
Kaiwen Zhou, Yanghua Jin, Qinghua Ding, and James Cheng. Amortized nesterov's momentum: A robust momentum and its application to deep learning. In Conference on Uncertainty in Artificial Intelligence (UAI), pp. 211-220, 2020. (Cited on pages 27 and 38)
Xiao Zhou, Yong Lin, Weizhong Zhang, and Tong Zhang. Sparse invariant risk minimization. In 39th International Conference on Machine Learning, pp. 27222-27244 2022. (Cited on page 19)
16

---

Published as a conference paper at ICLR 2023
Appendix of "Pareto Invariant Risk Minimization"
CONTENTS
Notations
18
B More Discussions on Background and Future Directions
18
B.1 Background and related work
18
B.2 Limitations and future directions
20
More Details on IRM Failures and Fix
20
C.1 More detail about failure case of IRM
21
C,2 Discussions of objectives in PAIR
................................
C,3 More details on the extrapolation example
23
D More Details on the Implementations of PAIR
25
D.I Detailed description of PAIR-o for OOD optimization
26
D.2 Detailed description of PAIR-s for OOD model selection
27
D.3 Discussion the practical choices of OOD preference
28
D.4 Discussion on the use of PAIR in practice
29
Theoretical Discussions
31
E.1 Proof for Proposition
31
E.2 Proof for Theorem 4.1
34
F More Details Experiments
37
F1
More details on COLOREDMNIST experiments
37
F2 More details about ablation studies
38
F3 More details about experiments on WILDS
................................
43
F.4 Software and hardware
45
More Details of Model Selection Results on DOMAINBED
45
G.I Introduction of difficult model selection in DOMAINBED
45
G.2 Training and evaluation details
46
G.3 Full DOM Results
47
17

---

Published as a conference paper at ICLR 2023
A
NOTATIONS
We first list the notations for key concepts in our paper.
Table 4: Notations
X' = R"
the input space
y IR
the label space the latent space
Z = R"
the featurizer yp X' - Z learns a latent representation for each input example
w
the classifier w 2 y
f F
the predictor f = w o sp + y is composed of featurizer and classifier when w is linear. f can be simply represented via dot product w
Eall
the set indices for all environments
Ew
the subset indices of training environments
the index set of specific environment
D", De
the datasct from environment containing samples {X; Y? considered as i.i.d. from P the overall dataset containing data from all environments, D = {D9) JeEE,
D
I(E)
the set of invariant predictors w.r.t. some OOD objectives (e.g.. IRM) and environments & the empirical risk calculated based on D" e.g., square loss or logistic loss
Le
the vector losses {C. mm considered in m objectives from MOO problem,
shared a set parameters
P(L)
the set Pareto optimal solutions w.r.t objectives
Poot
the vector of objective preference
G e Rm
the matrix of gradients w.r.t. m objectives and parameters E Rd each objective L; corresponds to gradient vector E R"
gm+1
the m-simplex corresponding OOD objectives, {B uT+ Dm+ Bs 1}
B MORE DISCUSSIONS ON BACKGROUND AND FUTURE DIRECTIONS
B. BACKGROUND AND RELATED WORK
In this section, we provide more details of the backgrounds and closely related works to ours, in complementary Sec. 2.
The problem of OOD generalization. The problem of OOD generalization typically considers a supervised learning setting based the data D {D*) JesEa collected from multiple causally related environments Eall, where a subset of samples D = (X;,Y,9 from a single environment E Eall drawn independently from an identical distribution IPe (Peters al., 2016). Given the data from the goal of OOD generalization is to find predictor f : X' + y
training environments that generalizes well all (unseen) environments, i.e., to minimize max (f), where C. is the empirical risk (Vapnik, 1991) under environment e, X' and ) are the input and labeling spaces, respectively. The predictor f = w e yp usually composed of featurizer ş : X ~ 2 that learns to extract useful features, and classifier w Z + y that makes predictions from the extracted features. In practice, ip is commonly implemented as deep feature extractor, while w is generically implemented as simple dense linear classifier (Gulrajani & Lopez-Paz, 2021; Koh et al., 2021; Rame et al., 2021; Rosenfeld et al., 2022).
Existing solutions to OOD generalization. There exists rich literature aiming overcome the OOD generalization challenge, which usually appear as additional regularizations of ERM (Vapnik, 1991). The first line is the Domain Generalization works (Ganin al., 2016; Sun & Saenko, 2016; Li
18

---

Published as a conference paper at ICLR 2023
et al., 2018: Dou et al., 2019) that tries regularize the learned features to be domain-invariate However, Zhao et al. (2019) show that the domain invariant features solely are not sufficient for guaranteed good OOD generalization. We refer readers to Gulrajani & Lopez-Paz (2021) for more details of the literature about Domain Generalization. Moreover, Namkoong & Duchi (2016); Hu al. (2018); Sagawa* et al. (2020) aim to regularize the models to be robust to mild distributional perturbations of the training distributions such that the models are expected to perform well in unseen test environments. Following the line of distributional robustness, Liu al. (2021); Zhang et al. (2022b); Yao et al. (2022) further propose advanced strategies to improve the robustness by assuming that models trained with ERM have strong reliance to spurious features.
Recently there is increasing interest in adopt theory of causality (Pearl, 2009; Peters et al., 2017; Schölkopf et al., 2021) and introduce the causal invariance to the learned representations (Peters et al, 6016; Rojas-Carulla et al., 2018; Arjovsky et al., 2019). The causal invariance is inspired by the assumption of Independent Causal Mechanism (ICM) causality (Peters et al., 2017). ICM assumes that conditional distribution of each variable given its causes (i.e., its mechanism) does not inform or influence the other conditional distributions (Pearl, 2009; Peters et al., 2017). Peters et al. (2016) introduce the concept of environments which are generated by different interventions on certain variables involved in the underlying data generation process of (X, Y). Despite of the changes to the intervened variables, the conditional distribution of intervened variables (they usually are the direct parents of Y in the underlying causal graph) and Y is invariant. Therefore, the invariant relationship can be leveraged to predict Y and generalize to different environments. We refer interested readers to Peters et al. (2016): Schölkopf et al. (2021); Ahuja et al. (2021a) for more details. Inspired by the causal invariance principle, Arjovsky et al. (2019) propose the framework of Invariant Risk Minimization (IRM) that allows the adoption of the causal invariance in neural networks. It further inspires plentiful invariant learning works (Parascandolo et al., 2021; Mahajan et al., 2021; Creager et al., 2021; Wald et al., 2021; Ahuja et al.. 2021a; Chen et al., 2022b; Lin et al., 2022b). At the heart of these works is the intuition that: When a predictor w acting on yp minimizes the risks in all of the environments simultancously. & is expected to discard the spurious signals while keeping the causally invariant signals. Additionally, there can be more definitions and implementations of the invariance (Koyama & Yamaguchi, 2020; Krueger et al., 2021; Shi et al., 2022; Rame al., 2021) which further encourage agreements at various levels across different environments. We refer interested readers Rame et al. (2021) for detailed comparison and discussion. As shown that most of the existing approaches encounter the optimization dilemma when learning the causal invariance, this work mainly focus on resolving the optimization issue in learning the causal invariance defined by the framework of Invariant Risk Minimization (Arjovsky et al., 2019), which is different from the literature of IRM variants or other OOD objectives that focus on proposing better objectives to learn the causal invariance.
Optimization Dilemma in OOD Algorithms. Along with the developments of OOD methods, the optimization dilemma in OOD generalization is gradually perceived in the literature, and raises new puzzles to the community. In fact, several recent works also notice the optimization dilemma in OOD algorithms, specifically, the trade-off between discovering the statistical correlations (i.e., ERM) and preventing the usage of spurious correlations (e.g., IRM). Empirically, Gulrajani & Lopez-Paz (2021) observe that, with careful hyperparameter tuning and evaluation setting. many OOD algorithms cannot outperform ERM in domain generalization, demonstrating the difficulties of properly mitigating the trade-offss between OOD and ERM objectives in practice. Moreover, Sagawa* et al. (2020); Zhai et al. (2022) find that, regularization on ERM, or sacrificing ERM performance, is usually needed for achieving satisfactory OOD performance. A similar phenomenon has also been observed by Zhao et al. (2020); Xie et al. (2021): Sadeghi et al. (2022); Sener & Koltun (2022); Teney et al. (2022), which aligns with our findings through Pareto front as shown in Fig. 6(a) and Fig. 7(a). Besides, Lin et al. (2022a) find that IRM can easily overfit and learns unexpected features when applying IRM on large neural networks. Zhou et al. (2022) propose to alleviate this problem by imposing sparsity constraints. Orthogonal to Lin et al. (2022a); Zhou et al. (2022) that focuses on the optimization consequences, we focus on the optimization process of OOD objectives. In addition, Zhang et al. (2022a) find that, the performance of OOD algorithms largely relies on choosing proper pretraining epochs which aligns with our findings in Fig. ((d), hence propose to construct ready-to-use features for stable OOD generalization performance. Orthogonal to Zhang et al. (2022a), we focus on developing better optimization scheme for OOD algorithms, including choosing the proper objectives and the achievability of the invariant predictors. Besides, Lv et al. (2021) propose ParetoDA to leverage MOO to resolve the gradient conflicts amon the objectives
19

---

Published as a conference paper at ICLR 2023
in Domain Adaption. ParetoDA uses the guidance of validation loss based on the data that has the identical distribution to test distribution, to trade-off the conflicts in domain adaption objectives. However, there can be multiple test domains, and the data that has identical distribution with the test domain is usually unavailable in OOD generalization. Therefore, ParetoDA is unsuitable for general OOD generalization methods. Despite the increasing literature that perceives the OOD optimization dilemma, it remains an open problem on why there exists such dilemma, and how to effectively mitigate the conflicts of ERM and OOD objectives and obtain OOD generalizable solution.
Further implications by the OOD optimization dilemma. In addition to preventing finding a proper OOD solution, the OOD optimization dilemma also raises significant challenges for model selection of OOD algorithms. Guirajani & Lopez-Paz (2021) highlight this challenge with rigorous evaluation of OOD algorithms. Similar to PATR-o, PAIR-S resolves the dilemma by leveraging the OOD loss values and explicitly considering the trade-offs of ERM and OOD performance. We present more details in Sec. G.I.
Multi-Objective Optimization (MOO) and its applications in Multi-Tash Learning, MOO mm
considers solving m objectives, w.r.t. { losses, i.e., ming ((0) - (O) (.n()))))))) (Kaisa, 1999). A solution 0 dominates another 0, i.e., L(0) 3 L(0), if (,) S (0) for all i and L( + L(0). A solution 0 is called Pareto optimal if there exists no other solution that dominates 8". The set of Pareto optimal solutions is called Pareto set, denoted as P. and its image is called Pareto front. As is usual that we cannot find a global optimal solution for all objectives in practice, hence Pareto optimal solutions
(0))²
of particular value. The multiple-gradient descent algorithm (MGDA) is one of the commonly used approaches to efficiently find the Pareto optimal solutions (Désidéri, 2012) but limited to low-dimensional data. Sener & Koltun (2018) then resolve the issue and apply MGDA to high-dimensional multi-task learning scenarios, where the objective conflicts may degenerate the performance when using linear scalarization. As pure MGDA cannot find a Pareto optimal solution specified by certain objective preferences, Lin et al. (2019); Zhang Golovin (2020); Ma et al. (2020) propose efficient methods to explore the Pareto set. Mahapatra & Rajan (2020) propose EPO to find the exact Pareto optimal solution with the specified objective preferences. Although MOO has gained success in mitigating the task conflicts in multi-task learning, it remains undcrexplored on whether and how we can leverage the MOO to model and resolve the ERM and OOD conflicts. Without a proper set of objectives and preference guidance, the existing MOO solvers are unable to obtain desired solution for OOD generalization.
B.2 LIMITATIONS AND FUTURE DIRECTIONS
Although PAIR is shown to effectively mitigate the objective conflicts and boost the OOD perfor- mance via better optimization and model selection, the performance gain sometimes can decrease given the limitations of PAIR. We believe future works can be built upon resolving the limitations of PAIR, as detailed below.
From the optimizer perspective, the improvements of PAIR-O can decrease on some datasets. We hypothesize it is because of the inevitable stochastic gradient bias in all MGDA MOO solvers (Liu Vicente, 2021), and potentially large variance in estimating the IRMv1 penalties (e.g.. RxRx1 where both IRMvl and VREx are shown to perform poor ), as we discussed Appendix D.4.2.
For PATR-s, discussed in Sec. 4 that PAIR-S can mitigate the drawbacks of selecting models using an unreliable validation set (has large gap from the test domain), the improvements will be bit smaller when the gaps narrow down (e.g.. PACS using test domain validation accuracy). Besides, the estimation of satisfaction to Pareto optimality in PAIR- s can also be affected by the variances in estimating loss values in stochastic setting (e.g., TERRAINCOGNITA,
discussed in Appendix D.2.
Additionally, PAIR can also be applied to scenarios where gradient conflicts exist, such as the tradcoff between adversarial power and unnoticeability of the attacks (Chen et al., 2022a), as well as improving the quality of representations in contrastive learning (Ma et al., 2021).
C MORE DETAILS ON IRM FAILURES AND FIX
In this section, we provide more details about the failure case of IRM and its effective fix from the perspective of MOO, in complementary to Sec. 3.
20

---

Published as a conference paper at ICLR 2023
C.I MORE DETAIL ABOUT FAILURE CASE OF IRM
We follow Kamath et al. (2021) to discuss the failure case of IRM. Specifically, given the problem setup as in Sec. B.I, we are interested in the linear classifcatioon/eerrssson following the setting. The loss values are measured as population loss in each environment.
Setting (identical (Kamath et al. (2021))): y
R. y S R. / either the square loss lag (9: y) = (Ø y)². or the logistic loss Elog (ū. y) = log (1 + exp (-p(-91) (-)y)) when y = {-1,1}
(binary classification).
IRM approaches the problem by finding an invariant representation y X + 2, such that there exists a predictor w Z - J/ acting on ş that simultaneously optimal among Eaul Hence, IRM leads to challenging bi-level optimization problem (Arjovsky et al., 2019) as
min [x)
nEEF
(10)
E arg (10 pp), Ve En. 122-y
Given the training environments E and functional spaces W for w and & for y, predictors w yp satisfying the constraint are called invariant predictors, denoted as I(Ex). When solving Eq. 10, characterizing I(Eu) is particularly difficult practice, given the access only from small subset of environments. It is natural to introduce a restriction that W is the space of linear functions on 2 Rdd (Jacot et al., 2021). Furthermore, Arjovsky et al. (2019) argue that linear predictors actually do not provide additional representation power than scalar predictors, i... 1, W = S R1. The scalar restriction on W elicits a practical variant IRMs as
finite samples
min Ce()),.s. V Vnlw= 5) 0, Ve Ex-- BE
(11)
Let (Eu) denote the set of invariant predictors elicited by the relaxed constraint in IRMg. It follows that ((E)) S (E(Eg) (Kamath et al., 2021). Yet, Eq. 11 remains a constrained programming. Hence, Arjovsky et al. (2019) introduce a soft-constrained variant IRMv1
min E Ce()))+ ja i (w up)
(12)
AEE
Theoretical Failure Practical IRM Variants. Although the practical variants seem promising, Kamath al. (2021) show there exists huge gaps between the variants and the original IRM such that both IRMs and IRMv1 can fail to capture the desired invariance, even being given the population loss and infinite amount of training environments. The failure case, called two-bit environment (Kamath et al., 2021), follows the setup of ColoredMNIST in IRM (Arjovsky et al., 2019), and defines environments with two parameters 0e; 3e [0, 1). Each D. is defined
: R.d X1 Y (a.), X2 Y (ad)),))
(13)
where Rad(o) (a) is a random variable taking value -1 with probability o and +1 with probability 1 - a. We denote an environment e with (a,, 3c) for simplicity. The setup in IRM can be denoted as E. ((), 0=)00< B. <1} where X, the invariant feature as fixed for different e.
fne
4
(a) Pareto Front under MSE loss.
(b) Failure case under MSE loss.
(c)
Variance distribution under MSE
loss.
Figure Counterparts Fig. I(a), Fig. 3 and Fig. 2 implemented MSE loss.
21

---

Published as a conference paper at ICLR 2023
In the example given by Arjovsky et al. (2019), i.e., Eu {(0.25, 0.1). (0.25, 0.2)). IRMs and IRMv1 are shown to be able to learn the invariant predictor fIRM as the original IRM despite of the relaxation. However, due to I(err S Is(Er), Kamath al. (2021) show that the set of "invariant predictors" produced by IRMs and IRMv1 is broader than our intuitive sense. For example, when given & = { 0.11), 0.1.), (0.1, ,, the solutions satisfying the constraint in IRMs are those intersected points in Fig. ((a) (The ellipsoids are the constraints). Although fo, fi, f2, fIRM E (E both IRMs and IRMv1 I prefer has the smallest ERM loss. In fact, Kamath et al. (2021) prove that, the failure can happen in wide range of environments with a 0.1464 and > 0.8356, even being given infinite number of additional environments, under MSE loss. It follows that I(Eur IsE(E)))) In other words, the relaxation in IRMs and IRMv1 will introduce additional "invariant predictors" which however do not satisfy the original IRM constraint. Both IRMs and IRMv1 will prefer those "invariant predictors" when they have lower ERM loss than fIRM. demonstrating the significant theoretical gap between the practical variants and the original IRM.
instead of fIRM (the predictor elicited by the original IRM), fi
600
Asu
-2"
50
4
(a) Pareto Front under Logistic (b) Failure case under Logistic loss.(c) Variance distribution under Logis- loss.
tic loss.
Figure 7: Counterparts of Fig. I(a), Fig. 3 and Fig. 2 implemented in Logistic loss.
More visualization results of the failure cases. In the main paper, we visualize the Pareto front, ERM loss distribution, and the variance distribution of the failure case given MSE losses, given the environment setup of & == {00.1, 0.11), (0.1, 0.4)]. We plot Fig. ((a) and Fig. 3 based on the Mathematica code provided by Kamath et al. (2021), where we focus on the odd predictors -,,(-1, 1) (1-1,-1), 1)). Since Fig. (a)), Fig. and Fig. are implemented MSE loss, for completing the discussion under Setting A (Kamath et al., 2021), also give their logistic counterparts as in Fig. 7.
due to the symmetry in two-bit environments. i.e., predictors satisfying (11,-1)
and (11, (-1-1,
0 od
0 $
1
0.8
1e2
1e2
OOD
1e3
0.5
1e3
1e4
1e4
performances
1e5
0.4
1e5
0.6
1e6
1e6
1e7
0.3
1e7
0.5
1c8
1e9
0.2
1e9
0.4
46668
5.334 Meights
76655
(a) IRMvl in the original CMNIST.
(b) IRMvl in CMNIST-m.
(c) Detailed performance of IRMv1.
Figure 8: Performances of IRMv1 I in CMNIST and CMNIST-m under different hyperparameters.
Practical Drawback of Practical IRM Variants. In addition to the theoretical gap, the optimization of IRMv1 is also difficult due to the conflicts between the IRM penalty and ERM penalty Eq. 12. It often requires significant efforts for choosing proper hyperparameters such as pretraining epochs and IRM penalty weights, i.e., X. Otherwise, IRMv1 may not enforce the constraint in IRMs, hence will lead to unsatisfactory performance, shown in Fig. (dd) We argue that the gradient conflicts generally exist in OOD optimization for various objectives, in Fig. 1(b), we visualize the cosine similarity between the gradients produced by ERM and OOD objectives, which is averaged from 50
22

---

Published as a conference paper at ICLR 2023
epochs after the pretraining. It can be found that, all of the OOD objectives (Arjovsky al., 2019; Krueger et al., 2021; Ahuja et al., 2021a; Koyama & Yamaguchi, 2020; Rame et al., 2021; Wald et al., 2021: Pezeshki et al., 2021) tend to yield gradients that have lower cosine similarity with those of ERM. The generally existed conflicts can further lead to suboptimal performances of these OOI) objective in practice even with exhaustive parameter tunning.
In complementary to Fig. I(d), we provide full results in Fig. 8, where we show the results of IRMv1 under different penalty weights (y-axis) and pretraining epochs (æ-axis) COLOREDMNIST (Ar- jovsky et al., 2019) (CMNIST) as well as the failure case (Kamath et al., 2021) (CMNIST-m), or E == {(0.1, 0.2), (0.1, 0.25)} described in two-bit environment. mances of IRMv1 are highly dependent on proper tuning of pretraining epochs and the penalty weights. The dependence grows stronger when IRMv1 is shown to unrobust on CMNIST-m. We also provide more detailed results of IRMv1 on CMNIST-m in Fig. 8(c), where the dependence can be clearly observed. contrast, PAIR performs robustly well under different pretraining epochs, using default preference (1, 10, 1e1 to ERM, IRMv1 and VREx objectives, respectively. In Sec. 5, we provide more evidences to demonstrate the power PAIR-O.
can be found that the perfor-
C.2 DISCUSSIONS OF OBJECTIVES IN PAIR
In Sec. 3.2, we derive group of ideal objectives for improving the robustness of IRMv1, shown as the following IRMX (IRMX)
60
min( (GRM, LIMM LVREx)*
(14) We prove in Proposition 2 that IRMX is able to solve a large number of failure cases of MRMs and IRMv1, and recovers the set of invariant predictors produced by the original IRM. However, motivated readers might be interested in the reasons for keeping IRMv1 in IRMX, since VREx solcly could resolve the two-bit environment failure case.
test
3.025 E:50 6425 0.33 0.22 0150 6.15 0.20
beta
Figure 9: Drawbacks of V-REx in practice.
Theoretically, Proposition 2 requires also the invariant predictors produced by IRMs. i.e., [E)) to recover the invariant predictors yielded by IRM. Nevertheless, considers only the ideal case. In the next, we elaborate on detailed discussion from the empirical side.
Drawbacks of Robust Minimization in Practice. After showing REx (Krueger al., 2021) can help avoiding the failure cases of IRMs, natural question is that, does LIRM remain necessary? We find the answer is "Yes". In Fig. 9, we use a modified example of & = {0.25, 0.1), (0.25, 3)} with ColoredMNIST (Arjovsky et al., 2019), where we change the variance between two environments through different 3. It can be found that, as the variance between two environments getting closer, the performance of REx (Krueger et al., 2021) (denoted vrex) drops more sharply than IRMv1 (denoted as irmvl). The main reason is that, as the variation of spurious signals in two enviromments tends to be smaller, the gradient signal of var((CeJeee, tends vanish, while the signals from LIRM maintains. This issue can be more serious in stochastic gradient descent where the estimates of the variance of {Ce Jeee, in minibatches tend to be noisy, leading to weaker signals.
C.3 MORE DETAILS ON THE EXTRAPOLATION EXAMPLE
In this section, we provide more details and results about the extrapolation example that examines the recovery of causal invariance, in complementary to Sec. 3.3.
We first restate the definition of causal invariance specified by Peters et al. (2016); Arjovsky et al. (2019); Kamath et al. (2021) as in Definition C.I.
Definition C.1. (Causal Invariance) Given a predictor f o s,, the representation produced by the featurizer up invariant over Eatl if and only for all e1, 62 E Eutl, holds that
ED., [Y (XX) = ED [Y]p(X)= = 2],
for all z E 2 0 1 where 2 = (x(X) E (D.)).
Then, we construct a regression example from X' RR + y : R. The input X is a two dimensional inputs, i.e., X (X1, X2). X, is designed to be the invariant feature, i.e., Y sin(X,) + 1, while
23

---

Published as a conference paper at ICLR 2023
(a) Uniform.
(b) ERM.
Gaussian.
(d) ERM.
Figure 10: Recovery of causal invariance via PAIR. (a), (c) We adopt two sampling methods where sample the training data (mainly) from the regions marked red, and evaluate the predictions across all region from (-4, -4) to (4, 4). The predictor following the invariance defined in IRM (Arjovsky et al., 2019) requires the predictions to be independent of spurious features within the overlapped invariant features. In this example, intuitively it requires the colored lines be perpendicular to
a-axis within |-2, 2.2, 2]. (b) and (d) show the performances of ERM under two sampling methods, can be found that ERM fail recover the causal invariance and incurs high MSE loss.
X₂ is designed to be the spurious feature that can be controlled to be spuriously correlated with label Y. The environments are synthesized according to different sampling methods.
Shown as Fig. 10, we leverage two sampling methods: i) Uniform sampling and ii) Gaussian sampling, where the latter is more difficult than the former. For Uniform sampling, uniformly sample the rectangle regions {(-3, -3), (-2, })} as environment and {(-1,2), 2). (3) as environ- ment 2, shown the red regions marked in Fig. 10(a). For Gaussian sampling, two Gaussian distributions: the first one has the center as (-0.9, - 22.2) with the covariance matrix has the center as (1, 2) with the covariance matrix as
sample from
as (0.99, 0.11), (0. 0.1) the second (-0,3, J. shown as the red regions marked in Fig. 10(c).
Therefore, in these two examples, the invariant representation yP should only take Xı and discard the spurious features X₂ under the overlapped invariant features, i.e., [-2, 2]. As we use different colors denote, the prediction produced by the invariant predictor following Definition C.1 expected be independent of X2. In other words, the plotted lines need to be perpendicular to the r-axis within overlapped invariant features 2]
We implement the predictor with a 3-layer linear perceptron that has hidden dimension of 128. We use the MSE loss and Adam (Kingma Ba, 2015) optimize the neural network. We sample 2500 training data points from each environment and evaluate with 1000 data points uniformly sampled across all regions. For fair comparison, we train all algorithms 10000 epochs until con- verge. Following the common practice (Gulrajani & Lopez-Paz, 2021), tions of the OOD penalties for all methods as 150. For IRMv1, VREx and IRMX, we search the penalty weights from 1e to 1e and find they generically perform well when with the penalty weights of le 2 to lel. While for PAIR, we search the relative preferences across 6 choices (1, le4, le16), (1, 1e4, 1e12), (1, 1e6, 1e8), (1, 1e8, 1e4), (1, 1e4, 1e4), (1, 1e8, 1e8), and find (1, 1e4, le12), (1, 1e8, le4), (1, le4, le4), (1, 1e8, 1e8) have lower validation losses.
anneal itera-
Vale Roic ENGTRAA
wiTe Jo OE
Vell Joi 7755575555705577
(a) Uniform.
(b) Uniform.
Gaussian.
(d) Gaussian.
Figure 11: Recovery of causal invariance via IRMv1.
24

---

Published as a conference paper at ICLR 2023
vile Jese 2200110220272000
(a) Uniform.
(b) Uniform.
(c) Gaussian.
(d) Gaussian.
Figure 12: Recovery of causal invariance via VREx.
(a) Uniform.
(b) Uniform.
(c) Gaussian.
(d) Gaussian.
Figure 13: Recovery of causal invariance via IRMX.
(a) Uniform.
(b) Uniform.
(c) Gaussian.
(d) Gaussian.
Figure 14: Recovery of causal invariance via PAIR.
We plot predictions with the best MSE losses IRMv1, VREx, IRMX and PAIR in Fig. 11, Fig. 12, Fig. 13, and Fig. 14 respectively. We also plot the validation loss at the top of the image while does not necessarily indicate a better recovery of causal invariance. It can be found that, when given the uniform sampled environments, the unrobust IRMv1, VREx and IRMX of the causal invariance, while when switching to the Gaussian sampled environments, they can fail dramatically as expected. In contrast, for both uniform sampling and Gaussian sampling, PAIR manage to recover the causal invariance almost perfectly. Perhaps even more surprisingly, PAIR achieve lower extrapolation loss up to 0.06 and 0.32, which are essentially beyond the extrapolation requirement issued by the causal invariance. Hence we believe it is an interesting and promising future direction probe the extrapolation ability within and beyond causal invariance.
part
D MORE DETAILS ON THE IMPLEMENTATIONS OF PAIR
In this section, we provide more details about the implementation PAIR as a optimizer and model selection criteria, in complementary Sec. 4.1.
25

---

Published as a conference paper at ICLR 2023
Key takeaways from the IRM example. Recall that the key takeaways from the failures of OOD optimization can be attributed to: i) using unrobust objectives for optimization: ii) using unreliable scheme to approach the desired solution. Nevertheless, objectives by introducing additional guidance such that the desired solution can be relocated in the Pareto front w.r.t. the new objectives. After obtaining robust objectives to optimize, we then leverage a preference-awane MOO solver to find the Pareto optimal solutions that maximally satisfy the invariance constraints by assigning the OOD objective higher preference while being aware of retaining ERM performance.
can improve the robustness of the OOD
More formally, let food be the desired OOD solution, a group of OOD objectives Lood {Lindd m are robust if they satisfy that
Lood) 3 (f), + Foot E F,
(15)
where F denotes the functional class of possible predictors. When given a robust OOD objective Lood, our target is solve the following MOO problem
miny (CEnM: Lood)"
(16)
where Lod corresponds a Eood codd relaxed invariance constraint as ooot (foca) = Cood 3 Lood (f), Vf *
food E F. Denote the <inv as empirical loss of using the underlying invariant features to predict labels, then the optimal values of the desired OOD solution are (finv; food) (LERMM( Lana( (ront )²²
which corresponds to an ideal OOD preference for the objectives that Pood = (ei c ood
T. Then
the solution of Eq. 9 needs to maximally satisfy the OOD preference, i.e., maximize L(f)² Pood.
D.I DETAILED DESCRIPTION OF PAIR- O FOR OOD OPTIMIZATION
To find a Pareto optimal solution that satisfies the OOD preference Pood, we leverage the preference- aware MOO solver (Mahapatra & Rajan, 2020). Different from Mahapatra & Rajan (2020), we adopt an explicit 2-stage "descent" and "balance" scheme, following the common practice in OOD generalization (Gulrajani & Lopez-Paz, 2021).
Illustrated as in Fig. 15, in the "descent" phase, we train the model to minimize the ERM loss such that it approaches the Pareto front by merely minimizing LERM first. Then, the "balance" phase, we adjust the solution to maximally satisfy the OOD preference Pood-
L0o
OOD solution
Deseent Phane
Init. point
Mcanwhile, to avoid divergence from the Pareto front, at each step, the descent direction gdss not only needs to maximize Pund, but also needs to avoid ascending all the loss val- ues. More formally, let G denote the gradient signals produced by L, step of the "balance" phase, it solves the following LP for the objective weights 3*,
L(f)T P(md)
EERM
8 arg max
(GB) gb.
Figure 15: PAIR-O.
Illustration of
s.t. (G 2 g G,, J-
(GB) 2, eje
0,
(17)
where gm+1 = (B E RT D B = 1}. 9b is the adjustment direction that leads to the
preferred Pareto optimal solution by Pood J = [j gb > 0} are the indices of objectives which
donot conflict with go while J { Gf gb 00 are those have conflicts with go. J LilL, maxy (LLyPo))y))))))) is the index of the objective which diverges from the preference most.
Specifically, Mahapatra & Rajan (2020) show that using the following go could provably lead the solution converge to the desired preferred Pareto optimal solution, which is defined as follows
gb p C (log((m- + +)) p(L));
(18)
where @ is the element-wise product operator, L(L) is the quantitative divergence of the current solution from the preferred direction, calculated through the losses at the current step, as follows
m+1
(LL) = (L((Im/m) = Σ L, log(mL.),
(19)
26

---

Published as a conference paper at ICLR 2023
Algorithm Pseudo code for PAIR-O.
1: Input:
Training data Drr = {X,, YiJē= with environment partitions Dir {De Jees, learning rate 77 batch size b; number of sampled environments d: OOD preference Pood for ERM loss CERM and m OOD losses Lood o 1: pre- wee-trainnng epochs epi maximum training epochs for "balance" phase eg: Trainable parameters at the "balance" phase A:
2: Randomly initialize parameters in the model f = yp;
3: for 1 to ep do
4: Sample batches of data (X,,Y)), Y2)
5: Make predictions with f: (8,) = [(XX)
6: Calculate the empirical loss LERM with (8, 1وزيーー
7: Update parameters of f with the empirical loss LERM using the learning rate n);
8: end for
9: for = 1 to EG do
10: for D É "permute((D")
Ex) do 11: Sample batch of the data from D {Xj, Yf D";
Make predictions with f: 11 = f((x})}-1):
13:
end for
Calculate empirical and OOD losses LERM and Lond and obtain the overall losses L;
15:
Obtain gradients G = aL/30;
Calculate the OOD divergence [() using Eq. 19;
17:
Obtain the adjustment direction go using Eq. 18;
18:
Obtain the index sets J, JJ", required by Eq. 17;
19:
Solve Eq. 19 for the loss weights 8":
20:
Update parameters gi+1
21:
end for
where L is the normalized loss
m+11
L, = Poods Li/ Σ p,L
j
Then, we elaborate the detailed algorithm of PAIR-o implemented via the EPO solver (Mahapatra & Rajan, 2020) as in Algorithm 1.
We now state a informal version of the convergence guarantee.
Theorem D.1. (Informal) Given LERM along with m differentiable OOD losses Londs at each step in the "balance" phase (line 9 to line 21 in Algorithm 1), there exists a step size tj such that, the set of new loss values L (++1) = (LERM, Li.,. Lm) with the updated parameters (+++) by any
n E [0, m,), denoted as At has the following properties:
(i). A' contains the exact Pareto optimal solution satisfying the OOD preference vector, i.e. L' E A':
(ii). A' grows monotonically smaller and smaller.
From (i) and (ii) in Theorem D.I, it suffices to know that as the optimization continues, A' converges to the losses of the exact Pareto optimal solution, hence for the parameters. The proof for Theorem D.1 simply follows the Theorem 1 to Corollary 1 in Mahapatra & Rajan (2020). Note that PAIR-O provides a general framework to find better OOD solution that properly trades off ERM and OOD objectives. In experiments, we find that using the simply modified variant of EPO solver (Mahapatra & Rajan, 2020) in PAIR-O can effectively find descent path under the gradient conflicts that leads to better OOD solution. Nevertheless, more sophisticated preference-aweee MOO solver can be developed and integrated into the framework of PAIR-O, which we believe is a promising future direction (Zhao& & Zhang, 2015; Zhou et al., 2018: 8; 2020).
D.2 DETAILED DESCRIPTION OF PAIR-S FOR OOD MODEL SELECTION
In this section, we provide a detailed description of PAIR-S a for OOD model selection for Sec. 4.1. Before start, we also provide detailed description of the critical reasons for designing PATR-s
27

---

Published as a conference paper at ICLR 2023
in Appendix G.1. From the IRM example, it is obvious that traditional model selection methods that merely use validation performance, i... ERM performance, are not suitable to select a desired solution for OOD generalization. Otherwise, the OOD performance would be easily compromised due to its conflicts with ERM objective. This issue is more serious when the validation set has a large gap between the test set (ef. Training-domain validation set selection for CoLORRDMNNSS in Table 3). Intuitively, models selected merely based on ERM performance tend to have a high preference or better performance on environments that have similar distribution of the corresponding validation set, which will lead to higher variance of performances at different environments or lower worst environment performance. Therefore, it is natural to jointly consider the ERM and OOD performances in model selection. Specifically, the selected model is expected to maximally satisfy the exact Pareto optimality.
Since our focus of PAIR-S simply incorporate the PAIR score as an additional model selection criteria. More specifically, given OOD preference Pood, we can calculate the PAIR selection score as
s. is mainly to validate the existence of previous mode selection issues, we
SPAIR
L' Pood:
(20)
where Pood is the normalized OOD preference Pood/ Ei= Poody- With the PAIR score,
then can apply it into the DOMAINBED model selection algorithms (Gulrajani & Lopez-Paz, 2021). Specifically, the model selection in DOMAINBED aims select models from several rigorous hyperparameter trials according to the validation accuracy. For the model selection in each run, one can obtain all training domain validation accuracies but only one test domain validation accuracy for fairness.
The algorithm detailed as in Algorithm 2. The PAIR score is mainly used select models among the logged steps within one run. To avoid trivial cases, we expect the models participated into the selection are converged. To this end, we heuristically use a threshold e to filter out the first steps and find it empirically effective. To select models from different runs, we will first use the validation accuracy to filter out some unreliable cases, and then adopt the PAIR to finalize the model selection. The only exception is the test domain validation accuracy, where the test domain validation accuracy is more likely to be a reliable indicator than the PAIR score.
The main limitation of the PAIR estimation is about the estimation of the loss values. In stochastic gradient descent, one could only obtain a stochastic estimate of loss values based on a minibatch sample of D... When the stochastic estimates of the loss values are unbiased, the PAIR is unbiased, too. However, there can exist certain variances in the stochastic estimates, which can severely affect the precision of the score thus the comparison of different models. Although Theorem E.1 establishes certain theoretical guarantees that allows for some degree of uncertainties, the variances are usually unavoidable. A instant fix for the issue is that one could afford some additional evaluation time to obtain a better estimate of the loss values. Besides, one could also jointly consider the uncertainty of the estimation and derive a more accurate model selection (Wald et al., 2021), which we leave for future work.
D.3 DISCUSSION ON THE PRACTICAL CHOICES OF OOD PREFERENCE
Essentially, the performances of both PAIR-O and PAIR-S have certain dependence on the quality of the OOD preference Pood. however, it is often the case that the ideal OOD preference is usually unknown. It desirable to analyze the performances of PAIR- and PAIR-S a under a imprecise OOD preference. Mahapatra & Rajan (2020) discussed a bit that when the exact Pareto optimal solution under the preference does not exist, the EPO solver can still find a Pareto optimal solution that is closest to the preferred direction. We discuss it in more general way by developing new MOO formulation of Eq. 16 under a approximated preference up to some approximation error of c. The theoretical discussion can be found in Sec. E.2. In this section, we focus on the practical side of the choice of Pood-
We first discuss some heuristics that can be leveraged to obtain proper O0D preference under two scenarios:
(). one has little-to-no knowledge about the OOD loss values;
(ii). one has the access to some running histories that one has some empirical knowledge about the
OOD loss values;
28

---

Published as a conference paper at ICLR 2023
Algorithm 2 Pseudo code for PAIR-S.
I: Input: Running history H from R runs, where each running history is consist of loss history
of (m + 1) losses, i.c. LERM and Lood = {food) and training and validation accuracy history A { (,, A from logging steps; OOD preference Pood: Convergence step c; Validation accuracy percentile y;
L ( L2 " (1++1))i= Y
2: for = 1 to R do
3: Calculate PAIR score using Pood for all 7 steps as S {s ===1 using Eq. 20;
4: Filter out the first e steps to avoid trivial cases and get S {s']T Jt=ei
5: Store the step with maximum PAIR score as 8, = arg max, S:
6: end for
7: Obtain the selected steps from R runs as S = |
8: Obtain the validation accuracies for all selected steps Aval (A
9: Calculate the validation selection bar as Aval (max Ayal min Aval) p + min Aval
10: Filter out all runs that have a validation accuracy lower than Aval and obtain Hi
11: Find the with highest PAIR score as r. arg max,eR 8y:
12: Return associated history ry:
In practice, i) mostly fits to PAIR-O while ii) mostly fits to PAIR-S.
When i) one has little-to-no knowledge about the OOD loss values, one can leverage certain theoretical inductive biases about the OOD losses. In fact, it is usual the case that the theoretical conditions for the optimality of OOD objectives do not hold in practice (Ganin et al., 2016; Sagawa* et al., 2020; Krueger et al., 2021; Shi et al., 2022; Rame al., 2021). In this case, minimizing the OOD losses acts more like a necessary condition for a satisfactory OOD solution. Therefore, one could assign sufficiently larger preference to OOD objectives than ERM objective. For example, throughout all experiments in the paper, we mostly assign (1, 1e10, 1e12) to ERM, IRMv1, 1, and VREx losses, which works under many scenarios.
Besides, among different OOD objectives, one could easily know which is more likely to be optimized than another. Therefore, to ensure all OOD losses are equally maximally optimized, we could assign the -ossiyooptimizalle OOD objectives higher preference. For example in IRMX, VREx tends to be easier to optimize than IRMv1 therefore we assign a higher preference to VREx. Moreover, if one could know the performances of different OOD objectives, it is natural to assign higher preference to those which solely perform better.
When ii) one has the access to some running histories that one has some empirical knowledge about the OOD loss values, one could obtain a empirical estimate of the OOD loss values w.r.t. ERM loss values convergence. Since the estimate is obtained under gradient conflicts, one could expect the ratios of OOD loss w.r.t. ERM loss should be higher when one could resolve the gradient conflicts properly. Therefore, one could assign slightly higher preference to OOD losses than the empirically estimated ratios. In the model selection experiments, we directly increase the ratio by le2 and find it works well expected.
In fact, both i) and ii) are discussed under minimal assumption about the external knowledge of the optimization process, the task and the data. We expect better estimate of the OOD preference could be obtained when more external inductive biases are incorporated. For instance, PAIR-o generalize ParetoDA (Lv et al., 2021) when one could obtain validation set that has similar distribution the test data. Even under the case that such data is not available, one could also adopt some techniques such as Mixup (Zhang et al., 2018) obtain an approximation. We believe that obtaining better estimate of the ideal OOD preference would be a promising future development based on our work.
D.4 DISCUSSION ON THE USE OF PAIR IN PRACTICE
D.4.1 SCALABILITY
Similar to other MOO algorithms (Sener & Koltun, 2018; Lin et al., 2019; Mahapatra & Rajan, 2020), PAIR-O requires full gradients of the predictor to make an accurate derivation of the objective weights 8", which could be bottleneck when deployed to large-scale networks, usually involves
29

---

Published as a conference paper at ICLR 2023
prohibitively massive number of parameters. Sener & Koltun (2018) develops an approximation of the full gradients using the gradients w.r.t. the latent representation produced by the featurizer, i.., (X)) However, it requires strong assumption on the structure of the data and the model. Moreover, when it involves complex network architectures such as DenseNet (Huang et al., 2017) or DistilIBERT (Sanh et al., 2019) in WILDS,, the approximation even the full gradients be even imprecise, as the gradients of the complex neural networks can not be directly concatenated as those of simple linear networks.
To this end, we develop another approximation that takes only the gradients of the classifier, which usually appears as linear classification layer in the predictor. Interestingly, we empirically find aL can even produce more useful signals for OOD generalization than the gradients w.r.t. classifier, shown as in Table
When considering a more resource restricted scenarios, such as the iWildCam and RxRx1 in WILDS, we freeze the featurizer after the "descent" phase, which can further resolve the memory and computation overheads. It also aligns with some recent discoveries that the featurizer trained merely with ERM may already discovery all useful patterns (Rosenfeld et al., 2022). Zhang et al. (2022a) also find the technique useful in Camelyon17 dataset of WILDS.
D.4.2 Loss VALUE ESTIMATION
Similar to other MOO algorithms (Sener & Koltun, 2018; Lin et al., 2019; Mahapatra & Rajan, 2020), PATR-o is described and analyzed in full batch setting, i.e., full gradient descent. However, in practice, stochastic setting tends to appear more often than vanilla gradient descent due to the scalability considcrations. As also discussed in Sec. 4.1, variances are unavoidable no matter the estimated values are biased or unbiased. Fortunately, the robustness of PAIR- to the preference can partially mitigate the issue.
The another potential limitation in PAIR-O could be the possibly negative estimate of some OOD losses, such as the stochastic estimates of IRMv1, since general MOO algorithms together with PAIR-O only accept non-negative loss values as the inputs. To this end, we will use IRMv1 as an example to explain how one could handle the potentially negative values in loss value estimation.
We will first introduce the unbiased empirical estimator of IRMv1, following Arjovsky et al. (2019); Ahuja et al. (2021b). More specifically, considering the IRMv1 objective,
min Σ Cx( + V wlw=1Lo 3²
(21)
3E
Observe that
OE [l(w p(Xx), Yo)
Ol(w x(X)), Yo)
V 1. (w)
E
Ow
w=1.0
Iw
w=1.0
and
(0E' 0E l(w (ooo,(((X) v(x"),
2
V Wulw=1.ofo(w =
Ow
w=
(22)
-(1=
(EE "El(w ((x), Y9)
Ow
for which the simplification is derived by taking the derivative inside the expectation, using the Leibniz integral rule. Obviously, the stochastic estimate of Eq. 22 is biased.
To obtain unbiased estimate of IRMv1 penalty, observe that E[X]² = E[AB],
if A, and X are i.i.d. random variables w.r.t. the same distribution X. Equipped with this observation, we can further write Eq. 22 as
dl(w p(x)), Ye)
Ollw ((Xx), Pe)
V Vww/==1.oool 2|]l2 E.
Dw
1= dw
al(w ((X)), Yo) Iw
Öl(w . ((80)), Ye)
E
ge
w=1.0
Iw
1.0
(23)
30

---

Published as a conference paper at ICLR 2023
where (X", Y ~ Pe and (X" Y Pe are i..d. samples from Pe of the environment e. As E. (1((pp(()))))) 8w
and E (((m)((X"))) 8w
can separately be estimated in mini- batches without bias, Eq. 23 essentially provides practical unbiased estimator of IRMv1.
However, different from IRMv1, Eq. 23 does not have any guarantees for its non-negativity, though the expectation of Eq. 23 is non-negative. To this end, we propose two heuristics to mitigate the issue.
The first heuristic is to add all minibatch estimates E' (H())(("")))))
by a sufficiently
der
w=10/
large constant C, such that the minimum value of E [d((==((X)))))
+ C is non-negative.
8he he=10
Moreover, as the constant does affect the calculation of the gradients, when IRMv1 is minimized to 0, E (e(w=p(X")) 8w
is also optimized to C.
The other heuristic to multiply the negative minibatch estimates 1E (B((u)o(X*))))
by proper negative constant -C, which will make all estimations non-negative. On the other hand, however, it can dramatically affect the variances in the cstimations. Essentially, this multiplication will enlarge the expectation of the estimated IRMv1, and may cause instability of the training, due to the unrobustness of IRMv1. Therefore, we can heuristically search the values C from 1 to le 4 by observing the early training dynamics. If the training unstable, then we heuristically tune C' to be smaller by le - 2.
dw
1w=100/
Although both of the heuristics above can not rigorously recover non-negative estimate of IRMv1 penalty (which is essentially impossible for the formulations like IRMv1), we empirically find them effective, for which we hypothesize because of the robustness of PAIR-o to the preference in OOD generalization.
D.4.3 GENERALIZING TO OTHER OOD METHODS
As shown in Fig. 1(b), the gradient conflicts between ERM and OOD objectives generally exist (Ar- jovsky et al., 2019; Krueger et al., 2021; Wald al., 2021; Pezeshki et al., 2021; Rame et al., 2021). implies that, on the one hand, the optimization dilemma generally exist for all OOD objectives. Meanwhile, both PAIR-O and PAIR-s are generically applicable all OOD methods. In experi- ments (Sec. 5), we validate the generality of PAIR-s s only for several OOD methods from the four main lines discussed in related works (Sec. B. though. PAIR-O essentially has similar generality as PAIR-S s, for whose performances at real world datasets, we will leave for future verification due the limited computational resources. Nevertheless, we can theoretically discuss the implementation options about how P PAIR-O can be applied to different OOD methods.
First, for Domain Generalization based methods (Ganin et al., 2016; Sun & Saenko, 2016; Li et al., 2018; Dou et al., 2019), such as DANN (Ganin et al., 2016), PAIR-o can directly take the domain classification loss and the label classification loss as the inputs.
Second, for Distributionally Robust Optimization methods (Namkoong & Duchi, 2016; Hu et al., 2018; Sagawa et al., 2020), PAIR-o can take the worst group loss or some more sophisticated regularizations and the ERM loss the inputs.
Third, for the causal invariance based methods (Peters et al., 2016; Rojas-Carulla et al., 2018;
Arjovsky et al., 2019; Creager et al., 2021; Parascandolo et al., 2021; Wald et al., 2021; Ahuja et al., 202 la; Chen et al., 2022b) and agreement based methods (Koyama & Yamaguchi, 2020; Krueger al., 2021; Shi et al., 2022; Rame et al., 2021), they can be handled by PAIR-o similarly IRMX.
E THEORETICAL DISCUSSIONS
E. PROOF FOR PROPOSITION
We first restate the proposition with formally defined Setting by Kamath al. (2021).
Setting A (identical to Kamath et al. (2021)): Considering the task of linear classifica-
tion/regression X' y where the quality of predictors X 5 is measured by population
31

---

Published as a conference paper at ICLR 2023
losses : 5 x -2. R.R'R,, is either the square loss (9. == 2 y)² the logistic loss (og(0.2) y) = log + ex+e (-ÿy)) when y = {-1,1} (binary classification).
Proposition 2. Under Setting A (Kamath et al. (2021)), for all (0, 1), let E == {(a, 3e)
Be E (0,1)) 1)) be any instance of the two-bit environment (Eq. 13), Ix denote the invariant predictors produced by Eq. 7, holds that Isnx (E) I[E).*
Our proof is proceeded by discussing the set of invariant predictors elicited by an ideal V- REx (Krueger al., 2021) objective Ix(E) (in more general way), and then incorporating Ix (E) into that elicited by IRMs or IRMv1 (Arjovsky et al., 2019) [E( for the two-bit failure case (Eq. 13).
We now first discuss the invariant predictors produced by the invariance constraints ideally elicited by V-REx. Recall that V-REx (Krueger al., 2021) aims to minimize the variances of ERM losses at different environments:
LVrex var( (CCe)eEL))
Therefore, when CVRES is minimized, we have Le, = Lee, Yei, 62 E Eg. Then, we can define the invariant predictors produced by V-REx, as the following.
VREx;: Define [E) {f X D/ (e) (e,( E}. VREx, is the objective:
min (E) GEEN
E
Then, we characterize the set of Ix through the following lemma. Lemma 1. Under Setting A, let f = & be the predictor elicited by I(E) and (Xe, Yo) ~ De.
If Se lxq: EDD [Y²]] ] identical, the distribution of sp(X.) identical (or = for all E, é = Loog and H (Ye (X))) is identical
then I(E) (E)
Proof. For any f = w o o I(E), using Observation in (Kamath et al, 2021), we have that ED., [Y (x() = 2) = ED.,, [Y x(X)= = =],
(24)
for all E 62 E and for all x Z,5
(i) For square loss Lag.
Ca( } 2 (D. - -)1
Ep. - Y
1
Fo. [ED, [w x - 2w o yp (X)Y [(x]] + B2, (V?).
where w is the simultaneously optimal classifier for all EE
Then, note that for all = E Z, it holds that
Ep, [ur(=)² 2x) (xX) = 2) = w=z)² En, (Y x(X) = =].
Using equation 24 and the assumptions that Ep, [ is identical and the distribution (x()) is
identical (or f = 0) for all e E E, we can conclude that for all e1, E &, Le, (f) = Ce,(f),
(ii) For logistic loss llog. note that the simultancously optimal w has the form
x(X) = =x] Prp, Y = -1 (xx) = 21 = log
Prp. |Y
1 Ep. [Y ((X) = 2]
w(z) = logg
1 Ep, (Y ((X) = 2)
for all E E and all Z. We can thus conclude that in this case, L. (f) Ep. (H 2)] (Y (x)).. which completes the proof.
Mootivated readers might he interested the necessities of keeping IRMv1 in objectives, for which we provide details in Appendix C.2.
We assume that the support of (xX (denoted as Z) is identical each environment for simplicity.
32

---

Published as a conference paper at ICLR 2023
Remarks. We formulate Lemma I in general setting that covers Two-Bit-Eny special case. It can be easily verified that the assumptions in this lemma are all satisfied in Two-Bit-Emy (Eq. 13). Moreover,
show that other environment settings (e.g., those in IB-IRM (Ahuja et al., 2021a)) also satisfy the assumptions.
Proposition 3. Under Setting for all a E (0,1), les & = [(a, Be E (0, and f be an odd (or linear) predictor. It holds that Lx (E) Is(E) I(E).
Kamath et al. (2021), we know that there are only two predictors in I(E'): The zero predictor fo = 0 (for both Eay and blog) and fIRM(Z1 T2) (1 - 20.) (for é = f fIRM (x1, x2) = log 1- (for c = logg)
Proof. From the proof of Proposition
(f())2 2f(X)Y + +Y Note that in Two-Bit-Env, Y 1. Thus, in this case, E (E) implies that Ep, (f(x2 - 2f(X)) is identical for all E. Moreover,
(i) For square loss Esg. (f(
I E Is(e) V V (J) 0 for all E
ED, ED. (f) E,, (x()Y for all e E.
We can conclude that for any f E Ix (E) n (E))), holds that
Ep, Ep, and ED. I (X)x) are identical for all e E, (oo[[(())) ED (XX)) for all E
(25)
(26)
Denote f(1.)) f(X1 1, 1). and f(.-1); ((-,) similarly defined. For
condition equation 25,
1 (Pa.)) (in)+((11a ((-1,-1) + (Pa.-1) P-
Ep.(((X)
Bo(1 20x)
(-f(..)) f(-1.-1) + fi,-11 + f PL-1.))
+
2
(27)
ED.1((X)M - a (J(1,)) ((-1,-1) + (S(-1.1) f(1--1))
Be
2 (f(.. f((11-11 + f(-11)) f(.-11)
To enforce condition equation 25 for any a. 3e (0, 1), it is required that
(ayy) - f(-1)) - f -Pa. - f-1--1 fa.-1 f(-1)) =.
0,
fa.) f(-1.-1) (f(-1,1)- fa.-1),
(a. + ---1-11-
= -fi + +f-111
In this case, condition equation 26 implies that f(..)) + P2-1--1 (1 (1)) (f(1.)) f(- Without restricting f to be an odd predictor (or equivalently, linear predictor), this constraint is circle passing through fo and fIRM. Requiring that f is odd, i.c., f(.1) = - -f--1-11 and fa.-1) = -f(-1.1), we can conclude that there are only two predictors left in Ix (E) n Is(E), which are f(.1) = f(-1.11) fa,-11 f(-1,1) and
f(,1)) 2a,
f(-1-11) 2a
3 f(11)22) = (1-2 x1-
f(,-1)=1 11 2a,
f(111) =2a 1.
(ii) For logistic loss bog. Le(f) = ED. log exp +=xx(((((X))))) Similarly, f E (I(()
nIs(E)
implies that
Ep, [oo (1 + exp (- (-f(X)Y)) is identical for all E,
(28)
-((J)Y 0.
ED.
(29)
cxp (f(XY))
33

---

Published as a conference paper at ICLR 2023
From condition equation 28 and that f is an odd predictor (f(.1) -f(-1,-1) and f(1-1)
- f(),), we can conclude that
(1 + 1lo.))))
(1 + cf
f(,1) f0,-1).
(1 e 32-2
- -
+1+
which is due that (1+e")a In this case, condition equation 29 can be simplified
(1+e-0)a-a is one-to-one function.
eaa fq.1ja -1.1)a-aav)(1 - a) = - fa.1) = f(11) = log 1
O
Thus, the only predictors in (8( are fo and SIRM-
Corollary 1. Under Setting A. for all a E (0. 1) and & = {(a, Be,) (a, Rea)) for any two distinct Beg Beg E (0, 1), Ix (Et) Ix [x(E) ( (E(E).
Proof. This directly follows from the observation that in the proof of Proposition 3, enforcing condition equation 25 and equation 28 for two distinct 9es Peg impose the identical constraints f.
E.2 PROOF FOR THEOREM 4.1
We first restate the informal version of the theorem as the following, while the formal description of Theorem E. will be given in Theorem E.4 with more formal definitions.
Theorem E.1. (Informal) For 7 E (0, 1) and any e, 5 > O, if F is a finite hypothesis class, both ERM and OOD losses are bounded above, let IAAR be the index of all losses, Pmax maX Pi and
Lmax = max Li, if the number of training samples [D] > s2linguums log 2(m+1)111 then with probability at least 7/ PAIR-O and PAIR-S yield c-approximated solution of food-
The proof for Theorem 4.1 is also theoretical discussion on the performances of PAIR-O and PAIR-S under an approximated OOD preference. Essentially, the performances of both PAIR-O and PAIR-s s have certain dependence the quality of the OOD preference Pood, however, it often the case that
ideal OOD preference usually unknown. is desirable to analyze the performances of PAIR-O and PAIR-S under imprecise OOD preference. Mahapatra & Rajan (2020) discussed a bit that when the exact Pareto optimal solution under the preference does not exist, the EPO solver We discuss it in a more general way by developing a new MOO formulation of Eq. 16 under an approximated preference up some approximation error of
still find Pareto optimal solution that is closest the preferred direction.
Without loss of generality, given a OOD preference Pood (PERM, PITERMPl,,-.. Pl, Pm)
cood the ERM loss LERM and m OOD losses Lood (C f L.m )T Eq. 16 can be reformulated
fPAIR arg min [FRM(J) fEF
(30)
s.t.
PERM =
p1 Pi p2
Pm Cma (f).
We remark that under the ideal OOD preference, the optimal solution of Eq. 30, is also the optimal solution to Eq. 16 (i.e., the unconstrained version). In other words, fPAI food. We will use fPAR differentiate from the solution the unconstrained version. We focus Eqq. 30 for the reason that it is more convenient to establish the discussion on the approximated OOD preference, from the perspective of optimization constraints.
Exactly enforcing the above preference constraint is too restrictive both practically and theoretically, instead we incorporate the approximation by relaxing the constraint of the loss values w.r.t. the OOD preference. The c-approximated problem of Eq. 30 is as the following
fAAR == arg min (JRM())
JEF
(31)
s.t.
Vi,j E IPAIR, i j.. (p.C.()) -
34

---

Published as a conference paper at ICLR 2023
where IPAI == {ERM, ood1, oodg. ................................ oodm) } is the index set of overall losses. We denote the relaxed constraint set in Eq. 31 as PAII = {f Vi,j E IAAI,, + j. -r c).
Clearly, it holds that the solution sets satisfy SPAR = fPAIR-
Then we define the empirical version of the c-approximated problem Eq. 31 with preference vector Pood as follows.
fPAI == arg min JEF
LERM(S)
(32)
Vi.j E TAAIR, i f. -P,
s.t.
Similarly, we denote the above constraint set as PFAIR = {f Vi.j E IPAIR # j. (p. PfCr(f) S et.
Assume finite hypothesis class F and define
min JEF Jemm |Ip:C.(f) -,,
First, we recall the definition of v-representative sample from Shalev-Shward & Ben-David (2014). Definition E,2. "Shalev-Swwart & Ben-David (2014)) A training set S is called u-representative (W.r... domain X, hypothesis F, loss E and distribution D) if
of F, F,(()) - C(f)) V,
where C(f) == E(x,))= and Z(f) 1sT Eix (ff())))
Equipped with this definition, we can now characterize the condition under which the constraint sets in equation 31 and equation 32 contain exact the same predictors.
Lemma 2. For any O, assuming 8 >> O and denoting Pmax = maxie Ima Pi. if the training set -representative w.r.t. domain X, hypothesis F, distribution 'D and all the ERM and OOD Dir is 4ymax losses (LERM, Lond), } then PRAI P PAIR
Proof. We first show that PEXIR S PÊAIR By the definition ō, for all f E F, and Vi,J E IPAIR / + j we have
(p:Le()) 1if1 S - or (mtC,()) - (f) 2 E
(33)
Using this property, for any f PAAIR- we can conclude that Vi,j E TPAR + j.
(p:C(()- piC;(0) = -,C(() S
This inequality further implies that
IMCC(()) - S6-8
|In(C(()) 12 -,C((( 5c-8
= -In[C.()) Sc- -,,,0000-.(((,(,,202- - -0)
-Z,()) (2) -
which is based on the triangle inequality of the absolute value function.
From the definition of 4pmax representative, we have (a(s) Êx()) <
Vi E TPAIR- Substi-
4pmax
tuting this in the above inequality, we obtain
D.8
4pmax
4ptax
8
S
2
which implied that f P PAIR-
35

---

Published as a conference paper at ICLR 2023
Then, prove that PPAIR S PAAR-
For any f E PAIR it holds that Vi,j E IAIR, i
i +
lp.E.(f) - 2,2,(
(P.C((f) P,S) -,,1))-))(-))0,)))), (10 -1,
- (,4,0)--1121 -xCU
se
= IPIC((()) - P,5,00 S S - -.0)) - -,00
p&d
(PIC,()) - -,5,, s+ +
4pmax 4pmax
sc+ 2'
which is again based on the triangle inequality the absolute value function and the definition of 4pmax -epresentative. Together with equation 33, conclude that (p:C.(f) CyC S 5 =
PARR- which implies PAAIR S PPAIR-
Based on the above discussion, we have proven that PAIR
PAAIR
Assumption E.3. For all f E F.X X.Y y, the ERM loss is bounded, i.e., (e(f(X), Y) S LERM < oc, and all the OOD objectives Lond can be written as the expectation of some bounded loss functions, i.e., Vi [m], flned()) E(X,Y)~ [f (X),Y) Y)] and (J Y)1 S Iod < co.
We remark that the assumption is natural and generally holds for many OOD objectives including IRMv1 (Arjovsky al., 2019) and VREx (Krueger al., 2021).
Theorem E.4. For any > 0, (0, if Assumption E.3 holds and 5 O, denoting Pmax = max Isua Ps and Lmax = max,e Li if the number of training samples |Piel 2 321 faaax log 2(m+1)/Fl, then with probability at least 7, we have for any fPAIR fPAIR and 8
fiAR E fÉAR: (frun) S CERM(ffAnn S CERM (f(AAR)
2pmax
Proof. We proceed by first assuming that the training set 1) is
4pmax -representative
domain
X, hypothesis F, distribution and the ERM and OOD losses (CERM, Lood), and then we establish the sample complexity required for this condition. From Lemma 2, we know that given this condition and the assumptions in the theorem, PAAI PAAR- Then, since the training set Der is representative w.r.t. the ERM loss LERM, we have for any SPAIR E SPAIR and fÍAIR E SÉAI.
4paax
LERM _ERMI(ÉARR) S
4prmax
(GERM(JAAn É <
4pmax
Moreover, based on the optimality of problem equation 32, we can conclude that
8
8
SEROa(fEAme
I CERM S PERM (fFAmR) S LERM +
4pmax
4pmax
8
LEROA (ixo)) S CERMI(FAAI)
2pmax
Then, using the optimality of problem equation 31, it holds that
5
LERMI < LERM( < LERM(FÉAUR) 2pmax
It remains to analyze the sample complexity of ensuring that the training set Der is 8 representative w.r.t. X, F, DD and all the ERM and OOD losses {LERM, Land).
4pmax
36

---

Published as a conference paper at ICLR 2023
For any E IPAR, based Assumption E.3, write Li(f) B(x (((((()..)) and (f) = [ف] Exx,,Y ((f((;)) Y)) with (4(f(x),,
& L, I Imax Vf, X,Y. Using
Hoeffding's s inequality, we can conclude that for any EF,
Pr 12,00- - 2.(f) 2
5
D 0
S exp
4pmax
32 mux P
max
Thus, for any E (0, 1), if we require
32
2
P
2(m + F
|D| 2
max
log
82
holds that
Pr 3f F. -2(0))
8
S
Σ
Pr
|2,(1) -c(f))-((f))
6
7
4pmax
4pmax
m
FEF
Thus,
8
Pr 3i e PAIR 3f E FF f.(f)
4pmax
Pr Pr -
4pmax
VETPAIR
Finally, we can conclude that with probability least 1 y, Vi E IPAIR Af E F.
8
[2,(0) -2(f) 5
4pmax
which completes the proof.
The e-approximated formulation has a close relationship to another relaxation as the following.
Remarks.
fPAIR arg min LERM(F) fer
s.t. CAAT (f) S G, [m).
Essentially, both the c-approximated formulation and the above formulation are natural relaxation of the original problem (Eq. 30 Eq. 16). As the the optimal solution food- In this work, since we focus on the approximations on the preference, e-approximated formulation essentially provides convenient touch which could be of independent interests for future discussions.
foody, the above formulation also yields
F MORE DETAILS ON EXPERIMENTS
In this section, we provide more details about the experiments (Sec. 5) in the main paper.
F.1 MORE DETAILS ON COLOREDMNIST EXPERIMENTS
In the proof-of-concept experiments with COLORED we follow the evaluation settings as IRM (Arjovsky et al., 2019) and the test-domain selection as DomainBed (Gulrajani & Lopez- Paz, 2021). Specifically, we use 4-Layer MLP with hidden dimension of 256. By default, we use Adam Kingma & Ba (2015) optimizer with learning rate of le - 3 and weight decay of le to train the model with 500 epochs and select the last epoch as the output model for each hyperparameter setting. We choose the final model from different hyperparameter setups as the one that maximizes the accuracy on the validation that share the same distribution as test domain. We then do grid search for the corresponding hyperparameters. For pretraining epochs, we search from {0, 50, 100, 150, 200, 250}. For OOD penalty, we search from {1e1, 1e2, 1e3, 1e4, 1e5). We evaluate each configuration of hyperparameters 10 times and report the mean and standard deviation of the performances. Besides, for IRMv1, we will refresh the history in Adam optimizer when the pretraining finishes, following the practice in Gulrajani & Lopez-Paz (2021). We also empirically
37

---

Published as a conference paper at ICLR 2023
0g
40 00
0.875
0..05
FRY 200
I5M
FRI
IAM
EPN
9.80
VS2
VREX
0.550
161
0.625
0.75
0,000
0.70
104
0.575
0.65
165
0.550
0.60
VR7a
VBEx
0.525
0.55
0,500
0.50
(a) CMNIST.
(b) CMNIST-M.
(c) CMNIST losses.
(d) CMNIST-m losses.
Figure 16: (a). PAIR can effectively find better solution than exhaustive tuning of penalty weights in IRMX. That is because PAIR can adaptively adjust the penalty weights during the optimization process, and leads to Pareto optimal solution, as shown in (c).(d).
find that refreshing the optimizer after pretraining can bring better performance of IRMv1 in COLOREDMNIST. While for VREx, we find the refreshing
needed.
For the implementation of IRMX, we change the penalty to be the sum of IRMv1 and VREx losses and conduct the same hyperparameter search as for IRMv1 1 for fair comparison. As for the implementation of PAIR, we use SGD with a momentum of 0.9 (Sutskever et al., 2013) after pretraining, to avoid the interference of Adam to the gradient direction and convergence of EPO (Mahapatra & Rajan, 2020) solver. Moreover, we also empirically find that SGD requires larger learning rate (we search over two choices, i..., 0.01 and 0.1) for approaching the direction. This is because of the design in EPO solver that it first fits to the preference direction then does the "pure" gradient descent, while the intrinsically conflicting directions pointed by the objectives can make the loss surface more steep. We will leave in-depth understanding of the above phenomenon and more sophisticated optimizer design in more complex tasks and network architectures to future works (Zhao & Zhang, 2015; Zhou et al., 2020).
F.2 MORE DETAILS ABOUT ABLATION STUDIES
Comparison between PAIR-O and the linear weighting scheme under exhaustive parameter search. In the main paper, to investigate how PAIR-o can find a better OOD solution under objective conflicts, we first conduct a ablation study to compare the OOD performances of PAIR-O and the exhaustive tuned IRMX. Specifically, we tune both IRMv1 and VREx penalty weights from a substantially larger scope, i.e., {1, lel, 1e2, 1e3, 1e4, le5, 1e6). As for pretraining epochs, we search from (0, 50, 100, 150, 200, 250). The results of IRMX in COLOREDMNIST and the modified COLOREDMNIST are shown as Fig. 16(a) and Fig. 16(b), respectively. Each point represents the best performed IRMX with the configuration of the corresponding pretraining epoch, the IRMv1 penalty weight and different VREx penalty weights from {1, lel, le2, le3, le4, le5, 1e6).
We also present a full exhaus- tive hyperparameter tunning study based on linear weight- ing scheme for IRMX, shown in Fig. 17, where further enlarge the search space of penalty weights from le6 to le12 to better compare
05008380850
0.8
le1
le1-
1e2
le3-
0.65
seoornnnnce 000
1e2
1e3
0.7
0OD performances
1e4
1e4-
le5
0.60
1e5
with IRMX optimized via
1e6
1e6
0.6
1e7
1e7
PAIR-O. Similar to Fig. 16(a)
1e8
1e8
and Fig. 16(b), each point
1e9
0.55
1e9
0.5
in Fig. 17 is selected from
1e10
1e10
lel1
lel1
best performed models trained
le12
0.50
1e12
0.4
with the corresponding IRMv1 penalty weights, and pretrain- ing epoch, and all possible vREx penalty weights from (1, le1, 1e2, 163, 1e4, le5, 1e6, 1e7, 168, 1e9, 1e10, 1e11, 1e12).
pair
pair
(a) CMNIST.
(b) CMNIST-m.
Figure 17: Full exhaustive hyperparameter tunning study
38

---

Published as a conference paper at ICLR 2023
Compared to IRMv1 shown as in Fig. 8, IRMX substantially improve the OOD performances in both COLOREDMNIST and the modified COLOREDMNIST, confirming our theoretical results. However, the OOD performances of IRMX turn out to be upper bounded by that optimized with PAIR-O at each pretraining epochs. In other words, PAIR-o requires substantially less parameter tuning efforts to achieve the top OOD performances, confirming the advances of PATR-O. In more complex tasks where the exhaustive parameter tunning is prohibitively expensive, such as in the experiments with WILDS (Koh et al., 2021), IRMX performs worse than PAIR, which further validates the effectiveness of PAIR-O.
To better demonstrate the advantages of PAIR-O over linear weighting scheme, we replicate the previous study in two datasets from WILDS, i.e., COMMENTS and FMoW. Due to the computa- tional resource limits, we limit the search scope of IRMv1 and VREx to {le - 2, 1, 1e2), respectively, It can be found that, even with broader hyperparameter search space, IRMX optimized via linear weighting scheme remain under-performed than PAIR-O.
Table 5: Comparison between linear weighting scheme and PAIR- in WILDS.
CIVI.COMMENTS VRE.
12-2
1e2
FMoW
IVREX
1e-2
le2
1e-2
12.5(=20))
73.8
T3. (0007)
1--2
34 34 +3.6
31. 43
43
70.3 14
30.23
38.70 2.3)
33.7 33.7/1870
le2
72.3-033)
74.35030)
18.14/2600
2A13-o
70.2(207)
Loss values distribution at convergence. As for the loss distribution experiments (Fig. 16(c), 16(d). we plot the IRM, and VREx loss values at convergence of best performed algorithms. The plotted values are in log-scale and normalized to [0, 1]. It can be found that PA PAIR-O effectively find a better solution in terms of IRMv1 and VREx losses, while not generating the ERM performances too much, which confirms our motivations for the design of PAIR.
Penalty weights trajectory. To examine whether PAIR-O can effectively adjust the penalty weights of ERM and OOD objectives, especially when the model has not arrived at the Pareto front (i.e., the gradient conflicts are expected to be more intense), we plot the trajectories of penalty weights generated by PAIR-O in both CMNIST and CMNIST-m, shown in Fig. 18.
It can be found that the whole training process can be divided
riting
Adaption
Gasenalation
1.0
fiting
Adaption
Ganerallaaiin
Accuract
0.75
Accuracy
into three phases: "Fitting" phase;
0.50
0.5
train
"Adaption" phase; and "General- ization" phase. In the "Fitting"
test
test
Epech
Epoch
phase, the model is trained with
Penalty weights
100
200
300
400
500
Penaly weights
100
200
300
400
500
only the ERM objectives and is expected to approach the Pareto
8:9
ERM
ERM
000
000
front first (cf. Fig. 15). It also cor- responds to the "descent" phase in the PAIR-- o algorithm, hence the penalty weight for ERM objective is while for OOD objective O. Then, when PAIR-O enters into the "balance" phase, PAIR-O begins to yield high weights to OOD objectives, while not diminishing the weights to ERM objectives. That is the "Adaption" phase, where PAIR-O begins to adjust the solution towards the Pareto front well as the preferred direction. When the solution is close to the Pareto front, then PAIR-O enters into the "Generalization" phase. That is to incorporate the invariance into the features by assigning high weights the OOD objectives.
0.0
10"
(a) CMNIST.
(b) CMNIST-m.
Figure 18: Penalty weights trajectory
Preference sensitivity analysis under strict hyperparameter configuration. Another reason for the high performance of PAIR-o both COLOORDMMISTT and realistic datasets from WILDS is because of its robustness to different preference choices. In complementary to the theoretical discussion in Theorem E. we also conducted preference sensitivity analysis experiments under strict hyperparameter configurations. In other words, the hyperparameter search space is restricted to single point, i.e., learning rate of 0.01, and a pretraining epoch of 150. The results are shown in Fig. 19 for both the original and the modified COLOREDMNIST dataset. It can be found that, PAIR-O maintains high performance and robustness to different preference choices.
39

---

Published as a conference paper at ICLR 2023
It also aligns with our discussion on preference choice in practice (Sec. D.3), that we need to as- sign a higher preference to robust, and more easy-to-optimize objec- tives, i... VREx. When the rela- tive preferences are given within reasonable scale, PAIR-O easily yields top OOD performances.
0.84
161
160
0.6
163
0.2
164
169
165
0.4
166
187
168
129
1c30
lwll
lu12
(a) CMNIST.
(b) CMNIST-m.
Figure 19: Preference sensitivity under strict hyperparameter configuration. æ-axis is the preference for VREx while y-axis is the preference for IRMv1
Additional ablation study on COI.OREDMNIST with "per- fect" initialization. We also con- duct experiments with "perfect" initializations for different methods, to check whether the OOD constraints can enforce the invariance, following Zhang et al. (2022a). Besides the OOD methods used in the paper, we also include another OOD method IGA (Koyama & Yamaguchi, 2020) to give a more comprehensive overview of their performances with "perfect" initialization. We also introduce another variant of ColoredMNIST, i.e., CMNIST-11: {0.255, 0.10), (0.25, 0.20)} to complement more details. All methods are initialized with a ERM model learned on gray-scale ColoredMNIST data which is expected to learn to use digit shapes in the image to make predictions. The learning rate is 1e- 3 and the penalty weight is 1e5. Different from Zhang et al. (2022a), we use SGD to optimize the models, as Adam would generate larger step sizes when the gradients continue to be within a small range under the "perfect" initialization. Results are shown as in Fig. 20.
0.728
3.85
0.725
3.82
175
3.52
2006
<006
5006
9006
11300
3200
4008
1200
6630
15000
2200
4030
1200
$230
10000
(a) "Perfect" init. CMNIST-10.
(b) "Perfect" init. CMNIST-IL. (c) "Perfect" init. CMNIST-25.
Figure 20: OOD performances with "Perfect" initializations.
can be found that, in CMNIST-10, 10, IRM, IRMx and IGA cannot enforce the invariance while V-REx and PAIR maintain the invariance, which is consistent our previous findings. Moreover, IGA fails to maintain the invariance in CMNIST- 11 and CMNIST-25, demonstrating the relatively low robustness of IGA objective. Besides, V-REx consistently maintain the invariance even in CMNIST-11, for the reason that the gradient signals of variance in "perfect" initialization tend to vanish. In contrast, PAIR improve over both IRM and IRMx maintain the invariance, confirming the effectiveness of PAIR.
Additional ablation study on the performance of PAIR-O and PAIR-s s with more OOD ob- jectives and their composite with IRMv1. Besides VREx, we conduct additional ablation studies of PAIR with IB (Ahuja et al., 2021a), Fishr (Rame et al., 2021), CLOvE (Wald et al., 2021), IGA (Koyama & Yamaguchi, 2020) and SD (Pezeshki et al., 2021), based on COLOREDMNIST and the modified COLOREDMNIST. We focus on the cases with no less than 2 OOD objectives, one could simply obtain a low OOD loss for single OOD objective, where linear weighting scheme is likely to approach the desired OOD solution as the Pareto front is simpler. However, it is often the case that single OOD objective is not sufficiently robust to locate the desired OOD solution to the Pareto front.
In experiments, we follow the same evaluation protocol previous experiments on COLOREDM- NIST. Due to the resource limits of NVIDIA RTX 3090Ti used for the original COLOREDMNIST experiments in previous sections, switch the hardware and software platform to Linux servers with NVIDIA V100 graphics cards with CUDA 10.2, hence the results in Table and Table are not directly comparable with those in Table 1. Similar previous experiments, for the stability
40

---

Published as a conference paper at ICLR 2023
of MOO solver under heterogeneous objectives, search learning rate for VREx and Fishr from {0.01, 0.02, 0.04, 0.1, 0.2) at stage 2 while a larger scope {0.1, 0.4, 0.8, 1 for other objectives. Note that even considering the learning rate into the hyperparameter search space, PAIR still uses smaller scope than that of linear weighting scheme. Besides, we follow our previous discussion in Appendix D..3 to set up the preference of different OOD objectives. Specifically, for Fishr, a larger preference of le12 than that for IRMv1 (1e8), since the agreements based methods tend to have a smaller loss than IRMv1. While for the other objectives, we use a smaller preference of 1e8 than that for IRMv1 1 (1e12). Note that this is only a heuristic setup and the performance of PA PAIR can be further improved if the preferences can be tuned.
Table 6: Generality study of PAIR for IRMv1 with other objectives in COLOREDMNIST.
IRMvl
PAIR-O
PAIR-8
CMNIST
CMNIST-M
Avg.
AAvg.
ERM
17.14(10.73)
73.30(
45.22
IRMv1
67. 07.29( (..99)
76.89(1..29)
72.09
+0.00
IB
55.48((13.67)
76.01(10.58)
65.75
56. 36.09((+2.00)
75.66(+10.6)
65.88
6621
61. 61.12(++2.85)
83.30(+.1.00)
72.21
+0.12
60. 60.69(++..6))
83.70(+1..9)
72.20
+0.11
VREx
68 68.62(1.073)
83.52( 12.52)
76.07
1.41
81.75(+1.68)
73.97
+1.88
68 (..1))
83.80/+11.00
76.35
+4.26
69.16(40.76)
83 1.6))
76.56
+4.47
Fishr
69 69.38(+10.9)
77. 77.29(+
73.34
66.20(
81.07(+.3.98)
73.63
+1.54
68 68.90(+0.56)
82.70(+1.00)
75.80
+2.49
68.78(+0.78)
84.02(+1137)
76.40
+3.31
CLOvE
55.55(+9.97)
74.20(++2:99)
64.88
66.35(+1.51)
+7.70(+1..00
72.02
00.07
64.99(12.29)
75.70(11.05)
70.35
1177
65.55(+2.17)
77.29(11.55)
71.42
0.67
IGA
58 7..67()7.69)
76.27(+1.01)
68.97
51.22(+33.77)
+4.20(+2245)
62.71
-9.38
66.17(12.34)
81884(11.09
74.01
+1.91
66.51(10.8)
82.12(13.04)
74.32
+2.23
SD
6..31
76.73/+0..0)
69.52
62.48 12.48(+11.25
61.24(+0.69)
71.86
-0.23
19.52(+6112)
62.82(+0064)
71.17
-0.92
65.54(109))
83.57(10..1
74.56
+22.77
Oracle
72.08(1028)
86.53(10.1)
79.31
79.31
The results are given in Table. 6. It can be found that, not all OOD objectives can improve IRMv1 performance. For the OOD objectives that can enhance the OOD robustness when incorporated into IRMvI, PAIR can further improve over the combined OOD objectives optimized via linear weighting scheme. While for unrobust combinations, intuitively it is hard to improve the OOD performance for the following reasons:
(). When the new objective combination is unrobust, the desired solution may not lie in the new Pareto optimal front;
(i). Eventhough the desired solution lies in the new Pareto optimal front, the weakened OOD robustness introduces more local minimals that have low OOD losses while worse OOD generalization performance;
(iii). As an extra objective is involved, the OOD preference used in PAIR tends to have higher divergence from the ideal one;
Therefore, given unrobust OOD objective combinations, the performance gain of PAIR is not theoretically guaranteed. Nevertheless, PATR-o still improve of the unrobust objective
41

---

Published as a conference paper at ICLR 2023
combinations, demonstrating its robustness. Notably, PAIR-S further improve the performance of PAIR-O at most cases, demonstrating the generality of PAIR.
To study what OOD objectives are suitable to be combined with IRMv1 and whether using more OOD objectives can bring more performance improvements, additionally, we conduct experiments with all possible composites of IRMv1 and IB (Ahuja et al., 2021a), Fishr (Rame al., 2021) and VREx (Krueger et al., 2021). In experiments, similar as in previous study, PAIR-O adopts slightly broader learning rate search scope of {0.01, 0.02, 0.04, 0.2) at stage 2, order to prevent divergence. Note that even considering the learning rate into the hyperparameter search space, PAIR still uses a smaller search scope than that of linear weighting scheme. PAIR-S adopts the training domain validation accuracy to perform the model selection. Both PAIR-O and PAIR-s adopts heuristic preference setup that uses a decreasing preference from 1e12 to 1e8 by a step size of 1e2 for more objectives. For example, in the composite of IB, IRMv1 and VREx, we adopt the preference of (1e8, le10, 1e12) for the OOD objectives. The choice of preference follows previous discussion in Appendix D.3.
Table 7: Generality study of PAIR for composite objectives in COLOREDMNIST.
IB
VREx
Fishr
CMNIST
CMNIST-M
Avg.
A Avg.
ERM
17.14(10.73)
73.30(
45.22
IRMv1
67.29(=0.99)
76.89(#322)
72.09
+0.00
Linear
56.09(=2.04)
75.66(+110.)
65.88
-6.11
+PAIR-O
61.12(=2.85)
83.30(
72.21
+0.12
+PAIR-O +PAIR-S
60.69(+2.26)
83.70(
72.20
+0.11
Linear
66.19(
81.75(11.66))
73.97
+188
+PAIR-O
68. 68.89(=1.13)
83.80(#1.60))
76.35
+4.26
+PAIR-O +PAIR-S
69.16/(10.7)
$3.96(#1.6)
76.56
+4447
Linear
66.20(=2.81)
81.07(
73.63
+1.54
+PAIR-O
66.45/20.90)
82.70( 1.00)
74.58
+249
+PAIR-O +PAIR-S
67.57(=0.8)
83.22( 12.10)
75.40
+3331
Linear
52.61(11.56)
(3.84(1..00
58.23
- 13.9
+PAIR-O
68.35(=1.73)
81.25(+3.08)
74.80
+2.71
+PAIR-O +PAIR-S
69.05(+0.76)
83.11(#1.46)
76.08
+3.99
Linear
51.91(#1.26)
68.88( 1.222)
60.39
11.7
4PAIR-O
19.77(
74.59( +1.11
67.15
4.94
+PAIR-O +PAIR-S
66.98(=2.66)
75.91(+3.50)
71.45
-0.65
Linear
22.95)
79.34(15.77)
72.09
+0.00
+PAIR-O
4
67.96(=1.60)
81.44(+2.24)
74.70
+2.61
+PAIR-O +PAIR-S
18.19(=1
81.89(
75.04
+2.95
Linear
50.00( 10.32)
69.60 12.35)
59.80
12.3
+PAIR-O
1..0)(
83.46( 13310)
75.18
+3.08
+PAIR-O +PAIR-S
68. 68.59(-1.29)
85.30(+0.64)
76.95
+4.85
Oracle
72.08(=0.24)
86.53(
79.31
The results are shown in Table 7. The best and second best results are in bold and underline. respectively. It be found that incorporating more OOD objectives does not necessarily bring more performance improvements into IRMv1. The linear weighting scheme can further exacerbate the performance drops of unrobust OOD objective combinations. For example, when incorporating IB objective into IRMv1, the OOD performance drops, since IB is proposed to mitigate a specific type of distribution shifts instead of directly improving learning the invariance in the original IRMv1 setting. In contrast, it can be found that incorporating Fishr can bring performance increases at most cases. The reason is that minimizing Fishr loss can approximately minimizes the VREx loss, as shown by Rame al. (2021). Therefore, could be that more objectives will make the Pareto front more complicated, and also lead to higher divergence of the OOD preference (since are less likely know the ideal preference given more objectives). Hence, the preferred composition of the objectives are preferred to those that have theoretical guarantees and are concise as possible.
suspect that the reason for the performance drop
42

---

Published as a conference paper at ICLR 2023
Interestingly, we also find that, although incorporating more objectives in PAIR-o does not neces- sarily bring performance increase, a combination of PAIR-o and PAIR-s can further improve the OOD performance, despite of the simple implementation of PAIR-o. It serves strong evidence for the generality and significance of PAIR.
F.3 MORE DETAILS ABOUT EXPERIMENTS ON WILDS
In this section, we provide more details about the WILDS datasets well as the evaluation setups in the experiments.
F.3.1 DATASET DESCRIPTION.
We select 6 challenging datasets from WILDS (Koh et al., 2021) benchmark for evaluating PAIR-O performance in realistic distribution shifts. The datasets cover from domain distribution shifts, subpopulation shifts and the their mixed. A summary of the basic information and statistics of the WILDS datasets can be found in Table. Table. 9, respectively. In the following, we will give a brief introduction to each of the datasets. More details can be found in the WILDS paper (Koh et al., 2021).
Table 8: A summary of datasets information from WILDS,
Dataset
Data
Class information
Domalns
Metric
Architecture
CAMELYON17
Tissuc slides
Tumor (2 classes)
hospitals
Avg. acc.
DenseNet1221
CIVE.COMMENTS
Online comments
Texicity (2 classes)
demographic groups
Wr. group are.
DistilBERT
FMOW
Satellite images
Land use classes)
16 years regions
group
DenseNat-121
(WILDCAM
Photos
Animal species (186 classes)
locations
Macro Fl
ResNei-50
POVERTY
Sacelite images
Asset freal valuod)
23 countrics
Wr. group Pearson (f)
Resnci-18 18
RxRXI
Cell images
Genetic treatments (1,139 classes)
51 experimental batchas
Arg.
ResNet-50
Table 9: A summary of datasets statistics from WILDS.
# Examples
Domains
Dataset
train
val
test
train
val
test
CAMELYON17
302,436
34,904
85.054
3
1
COMMENTS
269.038
45,180
133,782
FMOW
76,863
19.915
22,108
11
3
2
IWILDCAM
129,809
14,961
42,791
POVERTY
10,000
4,000
4,000
13-14
4-5
4-5
RXRXI
40,612
9,854
34,432
4
14
Camelyon17. We follow the WILDS splits and data processing pipeline for the Camelyon17 dataset (Bándi et al., 2019). It provides 450, 000 lymph-node scans from 5 hospitals. The task ( is to take the input of 96 96 medical images to predict whether there exists tumor tissue in the image. The domains d refers to the index of the hospital where the image was taken. sampled from the first 3 hospitals where the OOD validation and test data
in Camelyon17
The training data are sampled from the 4-th and 5-th hospital, respectively. We will use the average accuracy as the evaluation metric and a 12 121 (Huang et al., 2017) the backbone for the featurizer.
CivilComments. We follow the WILDS splits and data processing pipeline for the CivilComments dataset (Borkan al., 2019). provides 450, 000 comments collected from online articles. The task is to classify whether an online comment text is toxic or non-toxic. The domains d are defindd according the demographic features, including male, female, LGBTQ, Christian, Muslim, other religions, Black, and White. CivilComments is used to study the subpopulation shifts, here we will use the worst group/domain accuracy as the evaluation metrie. As for the backbone of the featurizer, we will use DistillBert (Sanh et al., 2019) following WILDS (Koh et al., 2021).
FMoW. We follow the WILDS splits and data processing pipeline for the FMoW dataset (Christie et al., 2018). It provides satellite images from 16 years and 5 regions. The task in FMoW is to classify the images into 62 classes of building or land use categories. The domain is split according to the year that the satellite image was collected, as well as the regions in the image which could be Africa, America, Asia, Europe or Oceania. Distribution shifts could happen across different years and regions. The training data contains data collected before 2013, while the validation data contains
43

---

Published as a conference paper at ICLR 2023
images collected within 2013 to 2015, and the test data contains images collected after 2015. The evaluation metric for FMoW is the worst region accuracy and the backbone model for the featurizer is a DenseNet-121 121 (Huang et al., 2017).
iWildCam. We follow the WILDS splits and data processing pipeline for the iWildCam dataset (Beery et al., 2020). It is consist of 203, 029 heat motion- acttvvtec photos of animal specifies from 323 different camera traps across different countries around the world. The task of iWildCam is to classify the corresponding animal specifies in the photos. The domains is split according the locations of the camera traps which could introduce the distribution shifts. We will use the Macro Fl as the evaluation metric and a ResNet-50 (He al., 2016) as the backbone for the featurizer.
PovertyMap. We follow the WILDS splits and data processing pipeline for the Mav dataset (Yeh et al.. 2020). It consists of satellite imagery and survey data 19, 669 villages from 23 African countries between 2009 and 2016. Different from other datasets, the task in PovertyMap regression task that asks the model to predict the real-valued asset wealth index computed from Demographic and Health Surveys (DHS) data. The domain is split according to the countries that the image was taken and whether the image is of an urban or rural area. The relative small size of PoverMap allows for using cross-fold evaluation, where each fold defines different set of OOD countries (Koh et al., 10.1). We will use the Pearson correlation of the worst urban/rural subpopulation as the evaluation metric and ResNet- 18 (He et al., 2016) as the backbone for the featurizer.
RxRx1. We follow the WILDS splits and data processing pipeline for the RxRxI dataset (Taylor et al., 2019). The input is an image of cells taken by fluorescent mieroscopy. The cells can be genetically perturbed by siRNA and the task of RxRx1 1 is to predict the class of the corresponding SiRNA that have treated the cells. There exists 1, 139 genetic treatments and the domain shifts are introduced by the experimental batches. We will use the average accuracy of the OOD experimental batches as the evaluation metric and a ResNet-50 (He et al., 2016) as the backbone for the featurizer.
F.3.2 TRAINING AND EVALUATION DETAILS.
We follow previous works to implement and evaluate our models (Koh et al., 2021; Shi et al., 2022; Yao et al., 2022). The information of the referred paper and code is listed as in Table. 10.
Table 10: The information of the referred paper and code.
Puper
Commit
Code
WILDS (Koh al., 2021)
v2.0.
hiLppss//wiiee stanford. wow..oduuueuuuu... httppp////gttuub..o/////ooiiioip
Fish (Shi et al., 2022)
5d2a99
com wwwgeee/geeeeeeeeeoo /f
LISA (Yao al., 2022)
The general hyperparemter setting inherit from the referred codes and papers, and are shown as in Table 11. We use the same backbone models to implement the featurizer (He et al., 2016; Huang et al., 2017: Sanh et al., 2019). By default, we repeat the experiments by 3 rums with the random seeds of 0, 1. While for Camelyon17,
follow the official guide to repeat 10 times with the random seeds from 0 to 9, and for PovertyMap, we repeat the experiments 5 times with the random seeds from 4. Note that the PovertyMap use cross-fold validations hence each run will use different training and evaluation splits, following the WILDS official guideline.
For the evaluation of baselines, we refer the previous results from the literature (Koh al., 2021; Shi et al., 2022; Yao et al., 2022) by default, while we rerun Fish (Shi et al., 2022) and LISA (Yao et al., 2022) to validate the reported results. Since the original implementation of Fish does not support the evaluation of the updated PovertyMap dataset, we mildly adjust the hyperparameter settings to reproduce the corresponding results as shown in Table. 11. We also reduce the batch size on FMoW due to the memory limits and we find it does not affect the reproducibility of Fish and LISA. Besides, since the original implementation of LISA A does not support PovertyMap Map, which differentiates as regression task that could be not suitable with Mixup (Zhang et al., 2018), however we find the "group by label" strategy in LISA works particularly well and reaches to the state of the art. For IRMX, we implement as the simple addition of IRMv1 and VREx penalties based on the Fish implementation (Shi et al., 2022), and search the penalty weights using the same space as for other objectives (Koh al., 2021) ensure the fairness. Besides, since previously reported results did not
44

---

Published as a conference paper at ICLR 2023
cover the performance of VREx in iWildCam and PavertyMap, implement VREx and report the results based on the Fish implementation (Shi et al., 2022).
Table 11: General hyperparameter settings for the experiments WILDS.
Dutaset
CAMELYON17
Civn.COMMENTS
FMow
IWILDCAM
POVERTYMAP
RXRXI
Num. of seeds
10
3
3
Learning rate
lc-4
20-6
le-4
le-4
le-4
1c-3
Weight decay Scheduler
0
le-5
n/a
n/a
n/a
n/a
Cosine Warmup
Batch size
32
16
32
16
64
72
Architecture
DenseNet121
DistilBert
DenseNet121
ResNet50
ResNet18
ResNet50
Optimizer
SGD
Adam
Adam
Adam
Adum
Adam
Pretraing Step
10000
20000
24000
24000
5000
15000
Maximum Epoch
2
200
90
For PAIR-o, we implement it based on the Fish code (Shi et al., 2022). The detailed algorithm can be found in Algorithm. 1. We leverage the same number pretraining steps as in Fish to fulfill the first "descent" phase in PAIR-O algorithm. Then, during the "balance" phase, at each training step, sampled & batches of data from different domains, calculate loss and conduct the back- pack-propagation. By default, we only the gradients of the classifier to solve for the objective weights during the "balance" phase. Except for iWildCam and RxRx1 datasets, due the memory limits, as discussed in Sec. 1, we use the freeze technique ensure the consistency of batch size and number of sampled domains as in Table. 11. Moreover, as discussed in Sec. D.4.2, the unbiased stochastic estimate of IRMv1 penalties can not guarantee the non-negativity of the estimated loss values, which are however not compatible with MOO theory (Kaisa, 1999) (thus the same for PAIR-o). Therefore, we will manually adjust the negative values to be positive, by multiplying it with a adjustment rate (short in Neg. IRMvl adj. rate in Table. 12). The adjustment rate is tuned from 1 to le 4 with step size of le 2 to avoid the training divergence and instability. Following the discussion as in Sec. D.3, tune the OOD relative preference by merely varying the preference for IRMv1 objective from the default choice (1, Ie10, 1e12) by a step size of le2. We find the performances IRMv1 I and VREx highly correlate the corresponding relative preference weights. We list hyperparameters of PAIR-O in Table 12. Although we did not tune the hyperparameters heavily, we find that PAIR-O generically works well across different challenging datasets and realistic distribution shifts WILDS, As discussed in Sec. D.3, there could be more sophisticated approaches to further improve the search and estimate of OOD preference, which we will leave for future developments based on PAIR
Table 12: Hyperparameter settings of PAIR-s for the experiments on WILDS.
Datasct
CAMBLYON17
CIVI.COMMENTS
FMoW
(WILDCAM
POVERTYMAP
RxRx]
Gradients from
Classilier
Classilier No
Classifier
Classilier
Classifier
Classifier Yes (e.1)8
Freeze featurizer
No
No
Relative Preference
(e.))
(1.1
(I.le12,1e12)
(1.ee10,ee12)
(1.1)8,
Neg, IKMvl ad), rite
1e4
1e-2
Te-2
Group
Hospitals
Demographies toxicity
regions
Trap locations 10
Countries 5
Experimental batches 10
Sampled domains
F.4 SOFTWARE AND HARDWARE
We implement our methods with PyTorch (Paszke et al., 2019). For the software and hardware configurations, we ensure the consistent environments for each datasets. Specifically, we run COL- OREDMNIST experiments on Linux Servers with NVIDIA RTX 3090Ti graphics cards with CUDA 11.3, 40 cores Intel(R) Xeon(R) Silver 4114 CPU 2.20GHz, 256 GB Memory, and Ubuntu 18.04 LTS installed. While for WILDS and DOMAINBED experiments, we run on Linux servers with NVIDIA V100 graphics cards with CUDA 10.2.
G MORE DETAILS OF MODEL SELECTION RESULTS ON DOMAINBED
G.1 INTRODUCTION OF DIFFICULT MODEL SELECTION IN DOMAINBED
DOMAINBED is proposed by Gulrajani & Lopez-Paz (2021) to highlight the importance of model selection in OOD generalization. Specifically, they empirically show that, under rigorous hyperpa-
45

---

Published as a conference paper at ICLR 2023
rameter tunning, ERM (Vapnik, 1991) achieves the state-of-the-an performances. Although recently progress are made to outperform ERM under the rigorous DOMAINBED evaluation protocol (Rame et al., 2021), whether there exists proper model selection for OOD algorithms remains clusive.
The difficulty proper model selection for OOD algorithms is mainly because of: We lack the access to validation set that have similar distribution with the test data. Therefore, Gulrajani & Lopez-Paz (2021) provide 3 options to choose and construct a validation set from: training domain data; leave-one-out validation data; test domain data. However, all three validation set construction approaches have their own limitations, as they essentially posit different assumptions on the test distribution (Gulrajani & Lopez-Pate Paz, 2021; Teney al., 2021; Rame et al., 2021).
PAIR-s s tries to address the limitations caused by the difficulty of finding a proper validation set for model selection in domain generalization, by leveraging the prior assumed within the OOD algorithm. Essentially, different lines of OOD algorithms discussed in Sec. B.1 adopt different prior and assumptions on the causes of the distribution shifts. The main purpose of the OOD evaluation is to validate the correctness of the posed assumptions. To this end, the selected models should properly reflect the preferences implied by the assumptions, i.e., the OOD loss values. When considering the loss values during the model selection, it is natural to leverage the MOO perspective and explicitly consider the trade-offs between ERM and OOD performance. The detailed description, implementation options, and potential leverages of PAIR-s provided in Appendix D.
G.2 TRAINING AND EVALUATION DETAILS
Since main purpose of the DOMAINBED experiments is to validate the existence of the problem and the effectiveness of PAIR-S, we apply PAIR-s to the representative methods of the four discussed OOD solutions in Sec. B.1. Specifically, choose the following four methods of all implemented algorithms DOMAINBED (https: : //tpt/gg.tttt/t///// thub com/ wacebookresearreoooooooccceeeeee Domain
ERM: Empirical Risk Minimization (Vapnik, 1991)
IRM: Invariant Risk Minimization (Arjovsky et al., 2019)
GroupDRO: Group Distributionally Robust Optimization (Sagawa* et al., 2020)
DANN: Domain Adversarial Neural Network (Ganin al., 2016)
Fishr: Invariant Gradient Variances for OOD Generalization (Rame et al., 2021)
Due to the limits of computational resources, we select out of 7 datasets from DOMAINBED. We refer Rame et al. (2021) to prescribe the detail, listed follows:
1. Colored MNIST (Arjovsky et al., 2019) is a variant of the MNIST handwritten digit classifi- cation dataset (Lecun et al., 1998). Domain E (90%, 80%, 10%} contains a disjoint set of digits colored: the correlation strengths between color and label vary across domains. The dataset contains 70,000 examples of dimension (2, 28, 28) and classes. Most importantly, the network, the hyperparameters, the image shapes, etc. are not the same as in the IRM setup for COLOREDMNIST experiments.
2. PACS (Li et al., 2017) includes domains d E fart, cartoons, photos, sketches), with 9,991 examples of dimension (3, 224, 224) and 7 classes.
3. Terralncognita (Beery et al., 2018) contains photographs of wild animals taken by cam- era traps at locations E {L100, L38, L43, L46), with 24,788 examples of dimension (3, 224, 224) and 10 classes.
Note that CMNIST dataset in DOMAIN BED use convolutional neural network the backbone for the featurizer, which is not the same MLP for COLOREDMNIST experiments. By default, all real datasets leverage ResNet-50 (He al., 2016) pretrained ImageNet, with dropout layer before the newly added dense layer and fine-tuned with frozen batch normalization layers.
During the training, strictly follow the evaluation protocol in DOMAINBED. Note that the hyperparameter configurations of Fishr have some differences from the default configurations hence refer the configuration tables by Rame al. (2021) directly, shown follows.
46

---

Published as a conference paper at ICLR 2023
Table 13: Hyperparameters, their default values and distributions for random search (Gulrajani & Lopez-Paz, 2021; Rame et al., 2021).
Condition
Parameter
Default value
Random distribution
PACS/
learning rate
0.00005
Uninieemm (00/uloemm-6,33... -6,335)
TERRAINCOGNTA
batch size weight decuy dropout
32
2 20asfomm if not DomainNet else 25nixem(3.5)
10Unifeemt-6-- 66-2)
RandomChoice (0,00.00.5))
COLOREDMNIST
learning rate
0.001
10 Uniiorm(m4...--300
butch size weight decuy
64
2 (3.9fom(3.9)
All
steps
5000
5000
regularization strength À ema
1000
10Unilor(1..4
Fishr
0,95
(niform((.9.) 0.99)
warmup iterations
1500
Uniform(O, 5000)
As for the construction of the validation set, we with training domain validation set and test domain validation set, as leave-one-out domain selection requires more runs and more computational resources that are out our limits. Specifically, to construct the validation set, the data from each domain will be first splitted into 80% (for training and evaluation) and 20% (for validation and model selection). For training domain validation set, the validation data is consist of the 20% split from each training domain. While for the test domain validation set, the validation data is consist of the 20% split from each test domain.
The whole evaluation will be repeated 3 times where in each repeat, there will be 20 samplings of hyperparameters from the distribution shown Table 13. Therefore, there will be 20 runs in each repeat and there will be 1 model selected from the 20 runs.
For the implementation of PAIR-s, s, we follow the algorithm as in Algorithm 2. Since training domain validation accuracy tends to be ie., has a worse reflection of the OOD generalization performance due to the high similarity with the training data (Teney al., 2021), during the selection within each run, filter out the models before the last steps in COLORED During the selection within one repeat (across different runs), Algorithm 2 and finalize the selection according the PAIR score. Except for GroupDRO and DANN of which the objective value tend have higher variance and relatively low OOD robustness, we aggregate the models within each repeat by the validation accuracy. In contrast, for the test domain validation accuracy, we filter the models before the last steps for DANN while 10 steps for others according the robustness of the objectives during the selection within each run. During the selection within one repeat (across different runs), we directly adopt the validation accuracy to finalize the model selected. Note that Guirajani & Lopez-Pazz (2021) argue that test domain validation more likely to be a invalid benchmarking methodology. since requires access to the test domain which is usually inaccessible in realistic applications.
unreliable indicator than test domain validation accuracy,
and the last 10 steps in PACS and TERRAINCOGEA
use a percent of 50% for step 9 in
For the selection loss values L, we use the values reported solely at each logging step, which is evaluated every 100 steps with minibtach of the training data, listed follows:
ERM: N/A.
IRM: ERM and IRMv1 (nll, penalty).
GroupDRO: Worst group ERM loss (Losses.min()). ()).
DANN: Weighted ERM and domain discrimination loss loss).
Fishr: ERM and Fishr penalty (n:11, penalty),
G.3 FULL DOMAINBED RESULTS
In this section, we provide full results of the DOMAINBED experiments. To begin with, we first present the overall results of the three datasets, including the averages and the improvements of the worst
47

---

Published as a conference paper at ICLR 2023
domain accuracies, in Table. 14 and Table. 15. From results seed that PAIR-S consistently improves the OOD performance across all datasets and validation set options. Remarkably, in the most challenging setting that uses train domain validation set COLOREDMNIST, PAIR-s improves the worst domain performances IRMv1 and Fishr by large margin up 14.3%. In the realistic dataset PACS, PATR-S s improves the worst domain performances of IRMv1 by large margin up to 7.3%. In TERRAINCOGNITA PAIR-S improves the worst domain performances of DANN by large margin up 3.15 1. Besides the worst domain performance, PAIR-s s improves the average domain performances up to 1.0% and empower the OOD methods to reach new state-of-the ttate-of-the-arts.
When using the test domain validation set, since the validation set itself could reflect the OOD generalization performance, therefore the improvements could be lower. When comes to OOD objectives that have relatively low robustness, the worst domain performance could be lower.
We also report the detailed results at each domain with the variance in the next section.
G.3.1 OVERALL RESULTS
Table 14: Overeall OOD generalization performances using training domain validation accuracy.
COLOREDMNIST
PACS
TERRAINCOGNIA
Overall
PAIR-8
Avg. acc
Wr. acc
Avg. acc
Wr. acc
Avg. ace
Wr. ace
Avg. acc
ERM
51.4 + 1.0
84,8 + 0.3
44.6 + 1.1
60.2
DANN
51.5 0.1
82.5 1 0.8
44.9 1: 0.9
59.6
DANN
51.9 1..9::00 0.1
+0,9
83.3 - 0.5
+0.7
44.5 + 1.5
+3,1
59.9
GRo
51.8 1: 0.0
84.1 + 0.8
46.6 1: 1.1
60.8
GROupDRO
53.0 + 0.4
+3.1
84.4 + 0.7
+1,1
46.6 + 1.1
+0.0
61.3
IRM
51.6
44,9 - 0.3 0.3
60.0
IRM
52.2 1.2:0..5 0.5
+14.3
85.1 0.9
+7.3
41.1 + 3.8
+1.4
59.5
Fishr
51.8 0.1
85.6
47.0 + 1.4
61.5
Fishr
54. 100
+12.7
85.6
+1.1
47.7 1.711111
+0.3
62.5
Table 15: Overeall OOD generalization performances using test domain validation accuracy.
COLOREDMNIST
PACS
TERRAINCOGNITA
Overall
PAIR-8
Avg.
A
Avg.
Avg.
A
Avg.
ERM
57.8 + 0.2
87.0 + 0.1
52.9 + 0.9
65.9
DANN
57.4 0.8
84.7 = 0.5
50.8 0.3
64.3
DANN
56.2 1.1
-2.6
85,7 0.2
+2,2
50.7 0.5
+0.4
64.2
DRO
61.3 0.4
86.9 + 0.0
52.5 1 0.2
66.9
GRoupDRO
60.1 0.7
.43
87.3
+1.8
52.0 12.0-::00 0.7
+0.6
66.4
IRM
68.1 1.6
84.4 + 0.5
49.2 0.6
67.2
69.0 1.1
86.0 =
+0.8
50.7 0.9
+0.4
Fishr
1:00::0
87.5 +
0.2
Fishr
68.2 3.0
87.4 1
+0.6
52.1 0.7
48

---

Published as a conference paper at ICLR 2023
G.3.2
TRAINING DOMAIN VALIDATION SET
Table 16: OOD generalization performances with training domain validation set COLOREDM- NIST.
Algorithm PAIR-s
+90%
+80%
-90%
Avg
A wr. acc
ERM
71.0 + 0.5
73.4 = 0.1
10.0 0.1
51.5
DANN
71.0 0.3
73.4 = 0.1
10.0
51.5
DANN
V
71.6 + 0.3
73.3 + 0.2
10.9
51.9
+0,9
GroupDRO
72.6 0.2
73.1 +.0
9.
51.8
GroupDRO V
72.7 + 0.2
73.2 =
13.0
53.0
+3.1
IRM
72.3 0.3
72.6 + 0,4
9.9 0.1
51.6
IRM
/
67.4 + 2.6
64.8 + 1.4
24.2 1.6
52.2
+14.3
Fishr
72.2 0.6
73.1 = 0.3
9.9 0.2
51.8
Fishr
v
69.1 + 2.9
70.9 = 1.7
22.6 2 1.4
54.2
+12.7
Table 17: OOD generalization performances with training domain validation set on PACS.
Algorithm
PAIR-s
S
Avg
A wr. acc
ERM
82.6 + 1,6
79.2 +
97.2 = 0.5
74.9 + 2.6
83.5
DANN
84.7 + 1.8
75.8 0,9
=.1
72.3
1.0
1.0
82.5
DANN
86.5 + 0.9
77.0 1.8
97.0
73.0
83.3
+0.7
GRoupDRO
83.4 1.7
0.3
9
78.2 1.3
84.1
GroupDRO
/
83.4 1.7
78.3 0.3
97.6 0.2
78.2 + 1.3
84.4
+1.1
IRM
82.9 2.6
81.4
96.7 0.6
73.1 3.1
83.5
IRM
22.42:2
80.5
97.2 = 0.2
80.4 + 1.3
85.1
+7.3
Fishr
85.3 + 1.1
80.3 0 1.1
97.9 0.3
79.1 = 1.7
85.6
Fishr
85.4 + 1.4
80.2 1.2 0.8
96.2 = 0.7
80.5 - 0.8
85.6
+1.1
Table 18: OOD generalization performances with training domain validation TERRAINCOG-
NITA.
Algorithm
PAIR-s
L100
L38
L43
L,66
Avg
A wr. acc
ERM
46.7 + 3.5
41.8 + 1.0
57.4
1.0
39.7 + 0.2
46.4
DANN
46.1 3.5
41.2 = 1.0
56.7
0.9
35.6
1.1
44.9
DANN
43.1 + 3.8
41.1
+
55.2 2.1
38.7 + 1.9
44.5
+3.1
GroupDRO
48.4 + 2.9
40.3 3.1
57.9
2.2
40.0 + 0.5
46.6
GroupDRO
48.4 + 2.9
40.3
$7.9
40.0 0.5
46.6
+0.0
IRM
48.4 + 3.8
35.6
1 2.9
55,4 = 0.9
40.1 - 1.4
44.9
IRM
40.4 + 7.3
38.3 + 2.5
48.8
6.3
37.0 0.9
41.1
+1.4
Fishr
49.2 + 4.4
40.6
1.4
57.9 = 1.1
40.4 1.2
47.0
Fishr
/
51.0 + 3.3
40.7 + 1.3
58.2
0.1
40.8 + 1.2
47.7
+0.3
49

---

Published as a conference paper at ICLR 2023
G.3.3 TEST DOMAIN VALIDATION SET
Table 19: OOD generalization performances with test domain validation set COLOREDMNIST.
Algorithm PAIR-S
+90%
+80% %
-90%
Avg
A
ERM
71.7 + 0.2
72.7 + 0.2
28.8 + 0.8
57.8
DANN
73.0 1.2
73.3 - 0.1
25.8 + 1.7
57.4
DANN
V
72.1 + 0.3
73.2 + 0.3
23.2 3.8
56.2
-2.6
GRO
73.4 - 0.4
72.4 + 0.0
38.1
0.8
61.3
GroupDRO
V
73.2 + 0.2
73.3 - 0.3
33.8 - 2.3
2.3
60.1
-4.3
IRM
72.3 0.3
72.5 = 0.4
59.4 5.3
68.1
IRM
V
71.7 0.4
73.1 0.1
62.3 3.1
69.0
+2,9
Fishr
73.8 0.5
73.6
+ 0.1
56.7
8.6
68.0
Fishr
V
73.7 + 0.6
73.5 = 0.2
57.3 - 8.4
68.2
+0.6
Table 20: OOD generalization performances with test domain validation set on PACS.
Algorithm
PAIR-S
A
S
Avg
A wr. acc
ERM
86.6 + 0.7
82.5
0.8
97.3 0.5
81.8 + 0.7
87.0
DANN
86.5 + 0.8
79.9 + 0.4
97.1
0.1
75.3 + 1.1
84.7
DANN
87.0 + 0.2
81.4 + 0.7
96.8
0.5
77.5 1.3
85.7
+2.2
GROupDRO
87.7 + 0.4
82.1 + 0.7
98.0)
0.2
79.6 + 0.7
86.9
GroupDRO
86.7 + 0.3
83.2 + 1.1
97.8
0.1
81.4 + 0.5
87.3
+1.8
IRM
82.3 + 1.5
80.8
0.7
95.8
1.3
78.9 + 1.4
84.4
IRM
85.3 + 0.3
81.7 + 0.9
97.4 = 0.3
79.7 + 1.8
1.8
86.0
+0.8
Fishr
88.4 + 0.4
82.2 + 0.7
97.7
0.5
81.6
1 0.4
87.5
Fishr
87.4 + 0.8
82.6 0.5
97.5
0.6
82.2 + 0.0
87.4
+0.6
Table 21: OOD generalization performances with test domain validation set on TERRAINCOGNITA.
Algorithm
PAIR-S
L100
L38
L.43
1.6
Avg
A wr.
ERM
58.7 + 1.7
51.3 + 1.8
59.9 0.6
41.7 + 1.0
52.9
DANN
+ 0.5
47.4 +
59.0
42.9 + 0.3
50.8
DANN
54.4 1.3
46.9 + 1.2
58.1 = -.2
43.3 0.0
50.7
+0.4
GRoupDRO
57.3 0.4
50.4
59.7 -0.7
42.8
52.5
GroupDRO
55.9 3.2
5.7
57.9 0.4
4.
52.0
+0.6
IRM
53.6 + 0.5
47.91.9
54.1 0.9
41.3 0.6
49.2
IRM
59.3 1.8
45.5
56.4
41.7
50.7
+0.4
Fishr
60.7 0.8
4.7
59.5 0.5
45.0
53.7
Fishr
58.9 + 1.0
46.4
58.6 0.7
44.5 0.8
52.1
-0.5
50


Reviews:
[{"review_id": "bzJLOHfke-", "reviewer": "Reviewer_eVsN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors provides many instances of optimization dilemma in ERM vs OOD from\nthe MOO perspective, and attributed the failures of OOD optimization to the compromised robustness\nof relaxed OOD objectives and the unreliable optimization scheme.", "summary_of_the_review": "The paper provided an interesting theoretical understanding of OOD losses and the invariance principle, as well as an advocation for using MOO to balance conflicting objectives. However, the lack of novelty in MOO and clarity in the necessity of their MO framework make this paper a weak reject. ", "strength_and_weaknesses": "The paper provides theoretical instances of tradeoffs between optimizing for ERM vs OOD objectives from a Pareto optimality lens and illustrates their empirical techniques with some theoretical justifications. They also provide better alternative optimization algorithms to perform this optimization than the main baselines.\n\nThe main gains are derived from two observations: the first being that using OOD vs ERM objectives \"can even eliminate the desired invariant solution from the Pareto front\". This is very interesting and they provide a theoretical example in Figure 1, although the model is not explained very clearly. More importantly, it is unclear why the invariance property is even a desired property given that selecting for such solutions will provide an inferior solution in terms of OOD/ERM. If the invariance property is indeed necessary, it is then unclear why it should not be incorporated as an alternative objective. A clear explanation of the inadequacies of the current approach should be elucidated further.\n\nThe second gain is from \"using the traditional linear weighting scheme that linearly reweights the ERM\nand OOD objectives, cannot reach the solution if it lies in the non-convex part of the front\" and they propose an alternative method of multiobjective optimization that finds a \"best\" point on the Pareto frontier. It would be good to explore if optimizing with  non-linear scalarizations (https://arxiv.org/abs/2006.04655) that can provably explore the Pareto frontier could help. Furthermore, the proposed method seems to only find 1 \"best\" point in the Pareto frontier, which almost degrades to single-objective optimization since there is no utilization of multiple Pareto optimal points (for ensembling for example). It becomes unclear why a MOO approach is needed, other than to perhaps inspire the right \"weight\" tuning? \n\nIn the experiments section, there seems to be no clear \"tradeoff\" between algorithm objectives that were considered theoretically. Rather it seemed like the different objective were actually the quality of optimization on different environments. Why are these not used as objectives instead? Furthermore, it seems like PAIR-O calls a packaged optimizer (EPO solver), so it does not seem the algorithm was novel.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "a9kW4fw63V", "reviewer": "Reviewer_hPqV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This submission analyzes and extends previous invariant risk minimization (IRM) formulations to a multi-objective optimization scheme, named as PAreto Invariant Risk minimization (PAIR) for finding models generalizing well and with good OOD performance. The authors motivated the work by analyzing how IRM may fail to achieve good generalizability and OOD performances and providing practical and theoretical failures of the IRM variants (relaxations). They discussed the importance of the trade-offs between ERM and OOD objectives and choosing desired solutions in Pareto optimal frontier and  developed PAIR-o, as an optimizer for OOD generalization and PAIR-s for model selection by leveraging the previous multi-objective optimization (MOO) solver EPO from the literature. Experimental results on several OOD benchmarks demonstrated the efficacy of the proposed PAIR framework. \n\n\n", "summary_of_the_review": "This submission analyzes and extends previous invariant risk minimization (IRM) formulations to a multi-objective optimization scheme, named as PAreto Invariant Risk minimization (PAIR) for finding models generalizing well and good OOD performance, with both solid theoretical and empirical results. The authors leveraged the existing MOO solver for PAIR and it may be more interesting if new solution algorithms can be derived. The presentation may need to be significantly improved as detailed in the above concerns. ", "strength_and_weaknesses": "\nThe elaborated discussion on the practical and theoretical failures of the IRM variants (with relaxations IRMs, IRMX, VREX) is clear and motivating the proposed development. Illustrations, proof-of-concept as well as empirical benchmark results have also demonstrated the potential of the PAIR framework. \n\nDue to the limited review time, this reviewer may have misunderstood some points but here are some potential issues which the authors may consider addressing. \n\n1. Mathematical notations may need to be revised to be consistent. For example, the operators changed from $\\circ$ in equation (1) to $\\cdot$ in equations (2) and (3). Some notations were not defined throughout the submission. Probably due to the page limit, however, Section 4.1 is the part where the details of the methods were described but since the algorithm is also in the Appendix, maybe better description for the methods can help improve readability of the submission. \n\n2. PAIR considers a composite OOD loss in the MOO setting. What other OOD objective combinations can be made,  which will lead to better performance? Can we make a structured analysis of which objectives should be combined in MOO?\n\n3. It appears that the theoretical analysis for classification was mostly on binary classification problems. While empirical experiments on benchmarks are with multi-class classification problems. The authors may want to state clearly about the generalizability of the theoretical results. \n\n4. The discussion in Appendix C.3 may also be mentioned in the main paper to clarify where to use PAIR-o and PAIR-s. Can optimization and model selection ideas be combined? Will it be useful in any settings or provide superior performance?\n\n5. The empirical results, in particularly in Tables 2 and 3, did not always show consistent improvements and the improvements are not with a huge margin, if any. More discussions on the observed trends may also help better understand possible gaps. \n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "N3ULGub57K2", "reviewer": "Reviewer_1jAA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper motivates the need for Pareto optimal solutions from the fact that most domain generalization methods can be seen as regularized ERM problems that struggle to outperform ERM.\nThe paper suggests that this is due to two key problems. One is the approximation of the original OOD objective (often encoded as a regularization). The other is due to using linear penalty methods for approximating a Pareto optimal solution, which is known to only work if the frontier is convex.\nThus, the paper proposes to formulate domain generalization methods as multi-objective optimization problems with ERM as one objective and the OOD objectives as additional objectives.\nThen, multiple gradient descent method for multi-objective optimization is applied.\nThe empirical results demonstrate the benefits of the proposed approach.\n", "summary_of_the_review": "Overall, the perspective of multi-objective optimization for domain generalization is a fresh take on domain generalization and provides practical performance benefits. The clarity of writing and illustrations are very well done and will aid in the works impact. The work could be improved by clarifying its contributions with respect to prior work in IRM variants and MOO. Also, it would be interesting to apply this more generally to other OOD methods.\n", "strength_and_weaknesses": "*Strengths*\n- The paper motivates its work via two key problems with domain generalization methods, most of which can be seen as regularized versions of ERM training. Namely, that the surrogate objectives may be inappropriate and that the regularized solutions do not find Pareto optimal solutions.\n\n- The examples and illustrations are well-constructed and instructive for understanding some of the key issues.\n\n- The results demonstrate the promise of using the proposed PAIR-o and PAIR-s schemes for better domain generalization. Particularly, PAIR-s can be applied non-IRM DG methods.\n\n*Weaknesses*\n- (Minor weakness) It seems odd that this paper focuses on IRM, as IRM has been often shown to be suboptimal and not necessarily SOTA.  It seems that the overall framework of Pareto optimization might be better applied to more SOTA domain generalization methods.\n\n- (Minor weakness) The novelty It seems that the problems with domain generalization and MOO methods are parts of prior work. Is the main novelty in synthesizing these two areas? I do think it's useful but it seemed the \"key problems\" (i.e., poor IRM approximations and pareto efficiency) and the MOO method were not novel by themselves. Perhaps this is just a problem of distinguishing the paper's contributions and prior work.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JxWiOd5gvL", "reviewer": "Reviewer_pR4m", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper illustrates the desired IRM solution $f_{\\text{IRM}}$ does not belong to the Pareto front of the IRMv1/IRMs loss included MOO problem in the environment (Kamath et al., 2021) described. This paper includes the VERx objective by (Krueger et al., 2021) into the MOO problem (IRMX) and shows in Proposition 1 that IRMX enjoy the same set of the invariant predictors as the original IRM in the ideal case. This paper illustrates PAIR recovers the causal invariance (Arjovsky et al., 2019) better than IRMv2 and VREx in Figure 4. This paper proposes to solve the IRMX problem by a MGDA-based optimizer with preference-awareness (Mahapatra & Rajan, 2020) and a 2-stage descent and balance scheme (Gulrajani & Lopez-Paz, 2021). Specifically, in the descent step, the PAIR-o algorithm minimizes the ERM loss. In the balance step, the algorithm follows the EPO search algorithm by (Mahapatra & Rajan, 2020). This paper also proposes a model selection method based on the highest PAIR score on the validation set. To relax the computational burden, this paper proposes only using the gradients of the classifier $w$ and choose needs an empirically large enough preference to the OOD objective, $p_{ood} = (1,1e10, 1e12)$. This paper also consider solving IRMX using linear weighting scheme but it achieves suboptimal generalization vs PAIR. The ablation study shows IRMX and PAIR are picking opposite trade-off between VREx and IRMv1 (Fig 5 (c)). Experiments using DOMAINBED evaluation protocol demonstrate PAIR-s can help worst case performance. ", "summary_of_the_review": "While I appreciate the engineering dedication and detailed discussion, my biggest concern is misleading/over-claimed statement, unfair comparison and originality.", "strength_and_weaknesses": "Strength:\n1. The motivation of studying the IRMX objective is well discussed. Firstly, this paper has state-of-art literature on OOD generalization well discussed. Early examples (e.g. Figure 2 and Figure 4) help to understand the drawbacks of IRM variants and the effectiveness of PAIR. Discussions in Appendix B also validates the necessity of keeping IRMv1 in the objective. \n2. The proposed model selection criterion PAIR-s achieves consistent worst environment improvement therefore it seems to be a good baseline for follow-up research. I also appreciate the detailed experimental setup.\n\nWeaknesses:\n1. I think the experimental setup between IRMX and PAIR-o is not fair enough. Most importantly, I think the one to one correspondence between $p_{ood}$ and linear weighting scheme can be established due to same number of objectives. If this is true, why they have different tuning range? IRMX is from 1 to 1e6 while PAIR is from 1e8 to 1e12 as specified in Fig 5 (d).  Furthermore, IRMX seems can not set different weights between IRMv1 and VREx while PAIR-o can. Is this intended? If IRMX is allowed to tuned as PAIR, I wonder if loss landscapes between these two will match each other in Fig 5(c). Right now they are almost opposite. \n2. I am not quite sure about two context reasons provided at Page 5 on why PAIR-o is preferable than IRMX. For (i), how exactly linear weighting scheme can not reach non-convex Pareto front? For (ii), why is $p_{ood}$ considered as a preference between ERM and OOD but linear weighting scheme is not considered a preference?\n3. I think the proposed advantage \"PAIR-o can avoid exhaustive tuning\" can be over claimed. I do not see clear reason the tuning over preference is less heavy than other algorithms such as linear weighting scheme. \n4. I think Theorem 4.1/D.4 is misleading. Firstly, the formal Theorem D.4 actually describes the ERM generalization between empirical $\\hat{f}^\\epsilon$ and population $f^\\epsilon$ specified by gap $\\delta$. It shouldn't be claimed as \"PAIR yield an $\\epsilon$-approximated solution of $f_{ood}$\" In Theorem 4.1. Secondly, the context before this theorem states \"ideal preference is usually unknown\" but I doubt the theorem can provide any guidance on choosing p. In particular, it tells us that p should have an upper bound depending on $\\delta$ - more accurate solution (i.e. small $\\delta$) needs smaller $p$. Is this contradicting the choice $p_{max} = 1e12$ in the experiment?\n5. The scalability seems to be a honest issue. While I do not want to emphasize this one too much, I encourage the author to look for alternatives. For example, stochastic algorithms seem an option for IRMX with linear weight scheme.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster