Exchanging Lessons Between Algorithmic Fairness and Domain Generalization
Authors: Elliot Creager
Joern-Henrik Jacobsen
Richard Zemel
Venue:  (2021)
Keywords: algorithmic fairness
domain generalization
representation learning
invariance

Abstract: Standard learning approaches are designed to perform well on average for the data distribution available at training time. Developing learning approaches that are not overly sensitive to the training distribution is central to research on domain- or out-of-distribution generalization, robust optimization and fairness. In this work we focus on links between research on domain generalization and algorithmic fairness---where performance under a distinct but related test distributions is studied---and show how the two fields can be mutually beneficial. While domain generalization methods typically rely on knowledge of disjoint "domains" or "environments", "sensitive" label information indicating which demographic groups are at risk of discrimination is often used in the fairness literature. Drawing inspiration from recent fairness approaches that improve worst-case performance without knowledge of sensitive groups, we propose a novel domain generalization method that handles the more realistic scenario where environment partitions are not provided. We then show theoretically and empirically how different partitioning schemes can lead to increased or decreased generalization performance, enabling us to outperform Invariant Risk Minimization with handcrafted environments in multiple cases. We also show how a re-interpretation of IRMv1 allows us for the first time to directly optimize a common fairness criterion, group-sufficiency, and thereby improve performance on a fair prediction task.


Under review as a conference paper at ICLR 2021
EXCHANGING LESSONS BETWEEN ALGORITHMIC FAIRNESS AND DOMAIN GENERALIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Standard learning approaches are designed perform well on average for the data distribution available at training time. Developing learning approaches that are not overly sensitive to the training distribution is central to research on domain- out-of-distribution generalization, robust optimization and fairness. In this work we focus on links between research on domain generalization and algorith- mic fairness performance under distinct but related test distributions is studied- and show how the two fields can be mutually beneficial. While domain generalization methods typically rely on knowledge of disjoint "domains" or "en- vironments", "sensitive" label information indicating which demographic groups are at risk of discrimination is often used in the fairness literature. Drawing in- spiration from recent fairness approaches that improve worst-case performance without knowledge of sensitive groups, we propose novel domain generaliza- tion method that handles the more realistic scenario where environment parti- tions are not provided. We then show theoretically and empirically how different partitioning schemes can lead to increased or decreased generalization perfor- mance, enabling us to outperform Invariant Risk Minimization with handcrafted environments in multiple cases. We also show how re-interpretation of IRMv1 allows us for the first time to directly optimize a common fairness criterion, group- sufficiency, and thereby improve performance fair prediction task.
INTRODUCTION
Machine learning achieves super-human performance on many tasks when the test data is drawn from the same distribution as the training data. However, when the two distributions differ, model performance can severely degrade to even below chance predictions (Geirhos et al.] 2020). Tiny perturbations can derail classifiers, as shown by adversarial examples (Szegedy al. 2014) and common-coree corrutoosrsrutionss in image classification (Hendrycks & Dietterich) 2019). Even new test sets collected from the same data acquisition pipeline induce distribution shifts that significantly harm performance (Recht et al. 2019; Engstrom et al. 2020). Many approaches have been proposed to overcome model brittleness the face of input distribution changes. Robust optimization aims to achieve good performance
any distribution close to the training distribution (Goodfellow et al.] 2015 Duchi et al., 2016) Madry et al. 20188). Domain generalization on the other hand tries to go one step further, to generalize to distributions potentially far away from the training distribution.
The field of algorithmic fairness meanwhile primarily focuses on developing metrics to track and mitigate performance differences between different sub-populations or across similar individu- als (Dwork et al., [0112 [orbett--aviin & Goel, (2088: [Couldechova & Roth 22018). Like domain generalization, evaluation using data related to but distinct from the training set is needed to charac- terize model failure. These evaluations are curated through the design of audits, which play a central role in revealing unfair algorithmic decision making (Buolamwini & Gebru 2018 Obermeyer al. 2019).
While the ultimate goals of domain generalization and algorithmic fairness are closely aligned, little research has focused on their similarities and how they can inform each other constructively. One of their main common goals can be characterized as:
Learning algorithms robust changes across domains or population groups.

---

Under review as a conference paper at ICLR 2021
Method
Handcrafted Environments
Train aces
Test aces
ERM
x
86.3 + 0.1
13.8 + 0.6
IRM
/
71.1
0.8
65.5 + 2.3
EIL+IRM
X 73.7 + 0.5
68.4 + 2.7
Table I: Results on CMNIST, digit classification task where color is spurious feature correlated with the lahel during training but anti-correlated at test time. Our method Environment Inference for Invariant Learning (EIIL, taking inspiration from recent themes in the fairnsss literature, augments IRM to improve test set performance without knowledge of pre-specified environment labels, by instead finding worst-case environments using aggregated data and a reference classifier.
Achieving this not only allows models to generalize to different and unobserved but related distri- butions, also mitigates unequal treatment of individuals solely based on group membership.
In this work we explore independently developed concepts from the domain generalization and fair- ness literatures and exchange lessons between them to motivate new methodology for both fields. Inspired by fairness approaches for unknown group memberships (Kim al. 2019 Hébert-Johnsee
cct al. 2018 Lahoti et al. 2020), we develop a new domain generalization method that does not require domain identifiers and yet can outperform manual specification of domains (Table Lever- aging domain generalization insights a fairness context, we show the regularizer from IRMv1 (Ar- jovsky et al. 2019) optimizes fairness criterion termed "group-sufficince" which for the first time enables us to explicitly optimize this criterion for non-convex losses in fair classification.
The following contributions show how lessons be exchanged from the two fields:
We draw several connections between the goals of domain generalization and those of algorithmic faimesss suggesting fruitful research directions in both fields (Section 2
Drawing inspiration from recent methods inferring worst-case sensitive groups from data, we propose a novel domain generalization algorithm Environment Inference for Invariant Learning (EIIL) for cases where training data does not include environment partition labels (Section 3) Our method outperforms IRM on the domain generalization benchmark ColorMNIST without access environment labels (Section 4
We also show that IRM, originally developed for domain generalization tasks, affords differentiable regularizer for the fairness notion of group sufficiency, which was previously hard to optimize for non-convex losses. On variant of the UCI Adult dataset where con- founding bias is introduced, we leverage this insight with our method EIIL to improve group sufficiency without knowledge of sensitive groups, ultimately improving generaliza- tion performance for large distribution shifts compared with a baseline robust optimization method (Section 4
We characterize both theoretically and empirically the limitations of our proposed method, concluding that while EIIL can correct baseline ERM solution that uses spurious feature or "shortcut" for prediction, it is not suitable for all settings (Sections 3 and
2 DOMAIN GENERALIZATION AND ALGORITHMIC FAIRNESS
Here we lay out some connections between the two fields. Table 2 provides a high-level comparison of the objectives and assumptions of several relevant methods. Loosely speaking, recent approaches from both areas share the goal of matching some chosen statistic across a conditioning variable e, representing sensitive group membership algorithmic fairness or an environment/domain indica- tor in domain generalization. The statistic in question informs the learning objective for the resulting model, and is motivated differently in each case. domain generalization, learning informed by the properties of the test distribution where good generalization should be achieved. In algorithmic fairness the choice of statistic motivated by a context-speciie fairness notion, that likewise en- courages particular solution that achieves "fair" outcomes (Chouldechova & Roth 2018). Empty spaces in Table 2 suggest areas for future work, and bold-faced entries suggest connections we show in this paper.
2

---

Under review as a conference paper at ICLR 2021
Statistic to match e
known?
Dom-Gen method
Fairness method
K[F|c]
yes
REx Krneger al. 2020
CVaR Fairness
Group DRO (Saaaw et a. 2019
(Williamson & Menon 2019
E/ele] no
DRO Duchi et al. 2016
Fairness without Demographies
Hashimoto al. 2018 Lahoti et 2020
yes
Score-hased IRM
Group sufficiency Chouldechova, 2017 Liu al. 2019
Ely)
yes
IRM AArovvkky et al. 2019
IRM across sensitive groups
Ely/((rr).e
EIIL (ours)
EIIL (ours)
Bly)((a), e] ((s),e)
no
Multicalibration (Hberr-Johnson et al. 2018
Multiaceuracy Kim al. 2019
Myy # ((+))) - 1,c]
no
Fairesss gerrymandering Kearns et al 2018
Domain Generalization (Dom-Gen) and Fairness methods can be understood as matching some statistic across conditioning variable e, representing "environment" or "domains" in Dom-Gen literature and "sensitive" group membership in the Fairness literature. We boldface new connections highlighted in this work, with blank spaces suggesting future work.
Table 2:
Notation Let X' be the input space, & the set of environments (a.k.a. "domains"), y the target space. Let æ, y, ~ prh y. e) be observational data, with I X. y E y. and E E. H denotes a representation space, from which a classifier w a (that maps to the pre-image of A()) via a linear map w) can be applied. T. X' > H denotes the parameterized mapping or "model" that we optimize. We refer to (xx) E as the "representation" of example x. ý E y denotes a hard prediction derived from the classifier by stochastic sampling or probability thresholding.
E: H x y + I denotes the scalar loss, which guides the learning.
The empirical risk minimization (ERM) solution is found by minimizing the global risk, expressed as the expected loss over the observational distribution:
CRRM
a = (x..e (((()),))) v)]
Domain Generalization Domain generalization is concerned with achieving low error rates on unseen test distributions. One way to achieve domain generalization is by casting it as robust op- timization problem (Ben-Tal et al., 2009) where one aims to minimize the worst-case loss for every subset of the training set, or other well-defined perturbation sets around the data (Duchi et al.) 2016) Madry et al.] 2018). Other approaches tackle domain generalization by adversarially learning rep- resentations invariant (Zhang et al.) 2017: Hoffman et al. 2018 Ganin et al. 2016) conditionally invariant (Li et al.) 2018) to the environment.
Distributionally Robust Optimization (DRO) (Duchi et al. 2016), seeks good performance for all nearby distributions by minimizing the worst-case loss sup, : Eg [[] s.t. D(gllp) < c, where D denotes similarity between two distributions (e.g. x" divergence) and is a hyperparameter. The objective can be computed an expectation over p via per-example importance weights y,- p(a..)))
2(m:.1))
Recently, Invariant Learning approaches such Invariant Risk Minimization (IRM) (Arjovsky eet al. 2019) and Risk Extrapolation (REx) (Krueger et al., 2020) were proposed to overcome the limitations of domain invariant representation learning (Zhao et al. 2019) by discovering invariant relationships between inputs and targets across domains. Invariance serves as a proxy for causal- ity, as features representing "causes" of target labels rather than effects will generalize well un- der intervention. In IRM, a representation ((x) is learned that performs optimally within each environment-aae and is thus invariant to the choice of environment e E-tth the ultimate goal of generalizing to an unknown test dataset p(z. ). Because optimal classifiers under standard loss functions can be realized via a conditional label distribution (x) = Ely[z)). then an invariant representation (zx) must satisfy the following Environment Invariance Constraint:
Eiy (x) = h, e Efy ((x) = e2] Vh E H Ver E.
(EI-CONSTR)
Intuitively, the representation (zx) encodes features of the input x that induce the same conditional distribution over labels across each environment.
3

---

Under review as a conference paper at ICLR 2021
Because trivial representations such as mapping all onto the same value may satisfy environment invariance, other objectives must be introduced to encourage the predictive utility of tb Arjovsky et al. 2 propose IRM way to satisfy (EI-CONSTR) while achieving good overall risk.
As a practical instantiation, the authors introduce IRMv1, a gradient-penalty regularized objective enforcing simultaneous optimality of the same classifier @ in all environments] Denoting by
R E (eode the per-environment risk, the objective to be minimized is
CLRM ()) =
Σ 3EE
RE + w (w +)1
(2)
2020) propose the related Risk Extrapolation (REx) principle, which dictates a stronger preference to exactly equalize R A e (e.g. by penalizing variance across e), which is shown to improve generalization in several settings
Krueger et al.
Fairness Early approaches to learning fair representations (Zemel ct al. 2013 Edwards & Storkey 2013 Louizos et al. 2015 Zhang et al. 2018 Madras et | (018)) leveraged statistical indepen-
dence regularizers from domain adaptation (Ben-David et al., [010 Ganin al. 2016), noting that
marginal or conditional independence from domain to prediction relates to the fairness notions of
demographic parity ú .. (Dwork al. 2012 and equal opportunity
ely (Hardt et al. (016)).
Recall that (EI-CONSTR) involves an environment-sppecfic conditional label expectation given a c]. Objects of this type have been closely studied in the fair machine learning literature, where e now denotes a "sensitive" indicating membership in a pro- tected demographic group (age, race, gender, etc.), and the vector representation (xx) is typically replaced by scalar score Siz) IR. Ely]S(x), e] can now be interpreted as a calibration curve that must be regulated according to some fairness constraint. [Chouldechova (2017) showed that equalizing this calibration curve across groups is often incompatible with a common fairness con- straint, demographic parity, while Liu et al. 2019) studied "group sufficiency" Safiiiinee""sstssiiiie satisfied when B[y[S(x),,) c| (1r), é'Jve, e'-of classifiers with strongly convex losses, concluding that ERM naturally finds group sufficient solutions without fairness constraints.
data representation Elyl
Because Liu et al. (019)) consider convex losses, their theoretical results do not hold for neural net- work representations. However, by noting the link between group sufficiency and the constraint from EI-CONSTR). observe that the IRMv1 regularizer (applicable to neural nets) in fact minimizes the group sufficiency gap in the case of a scalar representation ((x) C R. and when e indicates sen- sitive group membership. It is worth noting that AArjovsky et al. (2019) briefly discuss using groups
as environments, but without specifying a particular fairness criterion.
Reliance sensitive demographic information is cumbersome since it often cannot be collected without legal or ethical repercussions. [éébertJJhhnson et al. (2018) discussed the problem of mit- igating subgroup unfairness when group labels are unknown, and proposed Multicalibration way of ensuring a classifier's calibration curve is invariant to efficiently computable environment splits. Since the proposed algorithm requires brute force enumeration over all possible environ- ments/groups, Kim et al. (2019) suggested more practical algorithm by relaxing the calibration constraint to an accuracy constraint, yielding Multiaccuraate classifier The goal here is to boost the predictions of a pre-trained classifier through multiple rounds of auditing (searching for worst- case subgroups using an auxiliary model) rather than learning an invariant representation.
A related line of work also leverages inferred subgroup information to improve worst-case model performance using the framework of DRO. Hashimoto et al.] (2018) applied DRO to encourage long-term fairness in a dynamical setting where the average loss for subpopulation influences their propensity to continue engaging with the model. Lahoti et al. (2020) proposed Adversarially Reweighted Learning, which extends DRO using auxiliary model to compute the importance weights " mentioned above. Amortizing this computation mitigates the tendency of DRO to overfit its reweighting strategy noisy outliers. |Wang et al.) (2020) proposed group DRO method for adaptively estimating soft assignments to groups suitable for the setting where group labels are noisy.
wwe & yields classification decision via linear weighting on the representation features.
"Analogous to REx, Williamson & Menon 2019) adapt Conditional Variance at Risk (CVaR) (Rockafellar
& Uryasev 2002) to equalize risk across demographic groups.
Kearns al. 2018 separately utilize boosting equalize subgroup errors without sensitive attributes.
4

---

Under review as a conference paper at ICLR 2021
Limitations of generalization-fiin fairness One exciting direction for future work is to apply methods developed in the domain generalization literature to tasks where distribution shift is related to some societal harm that should be mitigated. However, researchers should be wary of blind "so- lutionism", which can be ineffectual or harmful when the societal context surrounding the machine learning system is ignored (Selbst et al.] 2019). Moreover, many aspects of algorithmic discrimina- tion are not simply matter of achieving few errors on unseen distributions. Unfairness due to task definition or dataset collection, as discussed in the study of target variable selection by Obermeyer et al. (019), may not be reversible by novel algorithmic developments.
3 INVARIANCE WITHOUT DEMOGRAPHICS OR ENVIRONMENTS
In this section we draw inspiration from recent work on fair prediction without sensitive labels (Kearns et al. 2018: Hebert-JJhnson et al. 2018 Hashimoto et al., 2018 Lahoti et al., 2020) to propose a novel domain generalization algorithm that does not require a priori domain/environment knowledge. To motivate the study of this setting and show the fairesss and invariance considerations at play. consider the task of using a high dimensional medical image x to predict target label y E {0, 1} indicating the presence of COVID-19 19 in the imaged patient. DeGrave et al. (2020) describe the common use of a composite dataset for this task, where the process of aggregating data across two source hospitals e E 1 H2 leads to a brittle neural net classifier w & ((z) that fixates on spurious low-level artifacts in predictive features.
Now we will consider a slightly different scenario. Consider a single hospital serving two different demographic populations
{P1, P2]. While P1 has mostly sick patients at time t = 0 due to the prevalence of COVID-19 in this subpopulation, P2 currently has mostly well patients. Then p(a,, y/e = Pi, 0) and p(z, yle P2, 0) will differ considerably, and moreover classifier using a spurious feature indicative of subpopulation membership-eitee either a low-level image artifact or attribute of the medical record-may achieve low average error on the available data. Of course such a classifier may generalize poorly. Consider temporal distribution shifts: suppose at time t = 1, due to the geographic density of the virus changing over time, P1 has mostly well patients while pa- tients from P2 are now mostly sick. Now the spurious classifier may suffer worse worse-than Ailan-chance error rates and imply unfair outcomes for disadvantaged groups. In reality the early onset and frequency of exposure to COVID-19 has been unequally distributed along many social dimensions (class, race, occupation, etc.) that could constitute protected groups (Tai et al. 2020), raising concerns of addi- tional algorithmic discrimination.
Learning to be invariant to spurious features encoding demographics would prevent errors due to such temporal shifts. While loss reweighting in DRO/ARL can upweight error cases, without an explicit invariance regularizer the model may still do best on average by making use of the spurious feature. IRM can remove the spurious feature in this particular case, but method for discovering environment partitions directly may occasionally be Sredeed[Thi need is clear when demographic makeup is not directly observed and method to each example into the maximally separating the spurious feature, i.c. inferring populations {PI, J. is needed for effective invariant learning.
3.1 ENVIRONMENT INFERENCE FOR INVARIANT LEARNING
We now derive a principle for inferring environments from observational data. Our ex- position extends IRMv1 (Equation 2 but we emphasize that our method EIIL is applica- ble more broadly to any environment-basee
learning objective. We begin by introducing indicator of the hand-crafted environment assignment per-example. Noting that Ne := Σ, (e) represents the number of examples in environment e, we can re-express this objective to make its dependence on environment labels explicit
u;(e') pa (c' [z; v1) = 1 = e')
CRRM (0,
Σ Σ (,(e)(((()))) CEE
V +2> 3EE
Ne [uj(e)((w ((x)). mn)
No
(3)
'In variant of the first example where hospitals e {H1, H2) are known, the given environments could he improved by a method that sorts whether spurious artifacts are present, i.e. inferring equipment type.
5

---

Under review as a conference paper at ICLR 2021
Our general strategy is to replace the binary indicator u,(e), with probability distribution g(e/2,,3) yt), representing a soft assignment the i-th example to the e-th environment. q(e/2 9t)
should capture worst-case environments w.r. the invariant learning objective; rewriting ((e y/) as (i(e) for consistency with the above expression, we arrive at the following bi-level optimization:
min max CIRM (Φ,q)
(EILL)
We leave the full exploration of this bi-level optimization to future work, but for now propose the following practical sequential approach, which call ElILv1 (See Appendix A foror pseudocode):
1. Input reference model e;
2. Fix & + Φ and fully optimize the inner loop of EIIL
to infer environments qi(e)
((e|r; 1/);;
3. Fix q + q and fully optimize the outer loop to yield the new model Φ.
Instead of requiring hand-crafted environments, we instead require a trained reference model ē; which is arguably easier to produce and could be found using ERM on pobs(z, y). for example. In our experiments we consider binary environment and explicitly parameterize the q(e|z, y) as a vector of probabilities for each example in the training data
3.2 ANALYZING THE EIII. SOLUTION
To characterize the ability of EiILvv to generalize to unseen test data, we now examine the inductive bias for generalization provided by the reference model क. We state the main result here and defer the proofs to Appendix(B Consider dataset with some feature(s) 2 which are spurious, and other(s) which are valuable/causal w.r.t. the label y. Our proof considers binary features/labels and two environments, but the same argument extends to other cases. Our goal is find model & whose representation (v is invariant w.r.t. z and focuses solely
Theorem Consider environments that differ in the degree to which the label y agrees with the spurious features z: Py((y = 2)) * P(1(y = =)ee2): then a reference model 4 S that invariant to valuable features and solely focusing on spurious features z maximally violates the Invariance Principle (EI-CONSTR). Likewise, consider the case with fixed representation + that focuses the spurious features: then choice of environments that maximally violates CONSTR is E =
= =)} and e2 = {v, 2, 2, [y # =)}.
If ennvirromenns are split according agreement of and z, then the constraint from (EI- satisfied by representation that ignores 2: (xx) 1 Unfortunately this requires priori knowledge of either the spurious feature or reference model Øspurious that extracts it. When the wrong & Spuriious is not a priori known, it can sometimes be recovered directly from the training data; for example CMNIST we find that &ERM approximates &Color This allows EIIL to find environment partitions providing the starkest possible contrast for invariant learning.
solution
Even if environment partitions are available, may be possible to improve performance by inferring new partitions from scratch. It can be shown (see Appendix B.2) that the environments provided in the CMNIST dataset (Arjovsky et al. 2019) do not maximally violate (EI-CONSTR) for a reference model &attor and are thus not maximally informative for learning to ignore color. Accordingly, EIIL improves test accuracy for IRM compared with the hand-crafted environments (Table
'The theoretical analysis of IRM suggests that the more (statistically independent) environments the better in term of generalization guarantees. This suggests in the setting where these analyses apply. extending ElIL to find more than two environments (with term promote diversity amongst inferred environments) may further help out-of-domain generalization, which we leave for future investigation.
'Note that under this parameterization. when optimizing the inner loop with fixed & the number of rameters equals the number of data points (which is small relative to standard neural net training). We leave amortization of q future work.

---

Under review as a conference paper at ICLR 2021
Accuracy
[RM IRM/EIL
RMiggs
Label noise 6
Label noise 6
(a) Train accuracy.
(b) Test accuracy
Figure 1: CMNIST with varying label noise Oy. Under high label noise (0, > 22), where the spurious feature color correlates to label more than shape on the train data, IRM(EguL) matches or exceeds the performance of IRM(CHH)
the test set without relying hand-crafted environments. Under medium label noise (.1 < By < .2), IRM(CEu)) is worse than IRM(CHc) but better than ERM, the logical approach if environments where color is less predictive than shape at train time, ERM performs well and IRM((rm.) fails. GRAYSCALE indicates the oracle solution using shape alone, while Color uses color alone.
available. Under low label noise (0y .1),
4 EXPERIMENTS
We defer a proceed with the established domain generalization benchmark ColorMNIST, and then discuss a variant of the algorithmic fairness dataset UCI Adult. We note that benchmarking model perfor- mance on a shifted test distribution without access to validation selection-is a difficult open problem, solution which is beyond the scope of this paper. Ac- cordingly we use the default IRM hyperparameters wherever appropriate, and otherwise follow a recently proposed model selection strategy (Gulrajani & Lopez-Paz, 2020) (see Appendix D
synthetic regression experiment Appendix E for lack of space. We
ceaplees-ssecialllll during model
4.1 COLORMNIST
ColorMNIST (CMNIST) is a noisy digit recognition task where color is a spurious feature that correlates with the label train time but anti-correlates at test time, with the correlation strength
at train time varying across two pre-specified environments (Arjovsky et al. 2019). Crucially, label noise is applied by flipping y with probability Oy: the default setting (0, 0.25) implies that shape (the correct feature) is marginally less reliable than color in the train set, so naive ERM ignores shape to focus on color and suffers from below-chance performance at test time.
We evaluated the performance of the following methods: ERM: A naive MLP that does not make use of environment labels e, but instead optimizes the average loss on the aggregated environ- ments; IRM(CHc): the method of Arjovsky et al. (2019) using hand-crafted environment labels; IRM(CEM)): : our proposed method (a.k.a. ElILv1) I) that infers useful environments (not using hand- crafted environment labels) based on the naive ERM, then applies IRM to the inferred environments.
After noting that EllLvI-denoed IRM((Fm)) above outperforms IRM without access to envi- ronment labely in the default setting (See Tables [ and (6. we examine how the various methods perform as a function of fy. This parameter influences the ERM solution since low 6,, implies shape more reliable than color in the aggregated training data (thus ERM generalizes well), while the op- posite trend holds for high fy. Because EIILv1 relies on reference model Φ its performance is also affected when Φ- ERM (Figure []]. We find that IRM((emL) generalizes better than IRM(EHC) with sufficiently high label noise 6, > .2. but generalizes poorly under low label noise. This is precisely due to the success of ERM this setting, where shape is a more reliable feature in the training data than color. We verify this conclusion by evaluating IRM(()mL) when T &Color i.e. hand-coded color-based predictor as reference. This does relatively well across all settings of fg. approaching the performance of the (oracle) baseline that classifies using grayscale inputs.
T Following the suggestion of Gulrajani & Lopez-Paz (2020). we note Section 4.2 contains "oracle" results that are overly optimistic for cach method (see Appendix D for model selection details). MNIST digits grouped into (0. 1, 2, 43 and {5, 6, 9) the CMNIST target label y is binary.

---

Under review as a conference paper at ICLR 2021
4.2 CENSUS DATA
We now study a fair prediction problem using a variant the UCI Adult dataset Whichh 48, 842 individual census records collected from the United States 1994. The task commonly used as an algorithmic fairness benchmark is to predict a binarized income indicator (thresholded $50, 000) as the larget label, possibly considering sensitive attributes such as age, sex, and race. Because the original task measures in-iistibution test performance, we instead construct vari- of this dataset suitable for measuring out-of offooii-iitiiuttion test performance, which we call Con- foundedAdult.
which comprises
Train accs
Test aces
Baseline
92.7 + 0.5
31.1 + 4.4
ARL (Lahoti al. 2020)
72.1 + 3.6
61.3 + 1.7
E
69.7 + 1.6
78.8 + 1.4
Table Accuracy on ConfoundedAdull, a variant of the UCI Adult dataset where some sensitive subgroups correlate to the label at train time and reverse this correlation pattern test time.
Lahoti et al. (2020) demonstrate the benefit of per-example loss reweighting on UCI Adult using their method ARL to improve predictive performance for undersampled subgroups. Following La- hoti al. 2020), we consider the effect of four sensitive subgroups subgroups-definet defined by composing bina- rized race and sex labels-on model performance, assuming the model does not know priori which features are sensitive. However, we focus on a distinct generalization problem where a pernicious dataset bias confounds the training data, making subgroup membership predictive of the label on the training data. At test time these correlations are reversed, so a predictor that infers subgroup mem- bership to make predictions will perform poorly test time (see Appendix C for details). [wwork] et al. (2012) described a similar motivating scenario where the conditional distribution mapping Teatures to target labels varies across demographic groups due cultural differences, the most predictive predictor for one group may not generalize to the others. The large distribution shift of our test can be understood as worst-case audit to determine whether the classifier uses subgroup information in its predictions.
Using EIILv1 first infer worst-case environments then ensure invariance across them performs favorably on the audit test set, compared with ARL and a baseline MLP (Table 3}. We also find that, without access sensitive group information, using the IRMv1 penalty on the EIIL environments improves subgroup sufficiency (Figure 2). Appendix F..3 provides an ablation showing that all com- ponents of the ElILv1 approach are needed to achieve the best performance.
sulepouo
sllan
selegrouy callbration
brinttioal cafiiattinn
Lime
mex
46.4
50
450
confoeree
Carforne
(a)
(b)
(c)
(d)
(c)
Figure 2: We examine subgroup Seffiiiency-whtteee calibration curves match across demographic subgroups- on the ConfoundedAdull dataset. Whereas ARL is not subgroup-sufficien fers worst-case environments and regularizes their calibration to be similar (b), ultimately improving subgroup sufficiency (c). This helps EIII. generalize better to a shifted test set (e) compared with ARL. (d). Note that neither method uses sensitive group information during learning.
(a), EIIL in-
https://archi iss uci http.eemmmmm/daatttss////d/
8

---

Under review as a conference paper at ICLR 2021
5 CONCLUSION
We discussed the common goals of algorithmic fairness and domain generalization, compared re- lated methods from each literature, and suggested how lessons can be exchanged between the two fields to inform future research. The most concrete outcome of this discussion was our novel domain generalization method, Environment Inference for Invariant Learning (EIIL). Drawing inspiration from fairness methods that optimize worst-case performance without access to demographic infor- mation, EIIL improves the performance of TRM on CMNIST without requiring a priori knowledge of the environments. On variant of the UCI Adult dataset, EIIL makes use of the IRMv1 regularizer to improve group sufficiency- fairness criterion previously difficult to optimize for non-convex losses without requiring knowledge of the sensitive groups.
REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. 2019.
arXiv preprint
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory of learning from different domains. Machine learning, 79(1-2)1111175,, 2010.
Aharon Ben-Tal, Laurent El Ghaoui, and Arkadi Nemirovski. Robust optimization, volume 28.
Princeton University Press, 2009.
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commer- cial gender classification. Conference on fairness, accountability and transparency, pp. 77-91, 2018.
Alexandra Chouldechova. Fair prediction with disparate impact: A study of bias recidivism prediction instruments. Big data, 5(2):153-163, 2017.
Alexandra Chouldechova and Aaron Roth. The frontiers fairness in machine learning. arXiv preprint arXiv:1810.088/00 2018.
Sam Corbett-Davies and Sharad Goel. The measure and mismeasure fairness: A critical review of fair machine learning. arXiv preprint (eXiv:88080000) 2018.
Alex J DeGrave, Joseph D Janizek, and Su-In Lee. Ai for radiographic covid-19 detection selects shortcuts over signal. medRxiv, 2020.
John Duchi, Peter Glynn, and Hongseok Namkoong. Statistics of robust optimization: A generalized empirical likelihood approach. arXiv preprint arXiv:1610.03425, 2016.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science confer- ence, pp. 214-226, 2012.
Harrison Edwards and Amos Storkey. Censoring representations with an adversary. In International Conference on Learning Representations, 2015.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Jacob Steinhardt, and Alek- sander Madry. Identifying statistical bias in dataset replication. arXiv preprint arXiv:2005.09619, 2020.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François training of neural net-
Laviolette, Mario Marchand, and Victor Lempitsky. works. The Journal of Machine Learning Research, 17(1):2096-033), 2016.
Robert Geirhos, Jöm-- Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1):554-67) 673, 2020.

---

Under review as a conference paper at ICLR 2021
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434 2020.
Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. In Advances in neural information processing systems, pp- 3315-3323, 2016.
Tatsunori B Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. Fairness without demographics in repeated loss minimization. In International Conference on Machine Learning, 2018.
Ursula Hébert-Johnson, Michael P Kim, Omer Reingold, and Guy Rothblum. Calibration for the ceamputttionaall-ddennfffiil masses. In International Conference on Machine Learning, 2018.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. In International Conference on Learning Representations, 2019.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International conference on machine learning, pp. 1989-1998, 2018.
Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. Preventing faimess gerryman- dering: Auditing and learning for subgroup fairness. International Conference on Machine Learning, pp. 2564-2572, 2018.
Michael P Kim, Amirata Ghorbani, and James Zou. Multiaceuracy: Black-box post-processing for fairness in classification. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pp. 247-254, 2019.
David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Remi Le Priol, and Aaron Courville. cittr--distribution generalization via risk extrapolation (rex). arXiv preprint arXiv:2003.0066 2020.
Preethi Lahoti, Alex Beutel, Jilin Chen, Kang Lee, Flavien Prost, Nithum Thain, Xuezhi Wang, and Ed H Chi. Faimess without demographics through adversarially reweighted learning. In Neural Informational Processing Systems, 2020.
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 624 224-639, 2018.
Lydia T Liu, Max Simchowitz, and Moritz Hardt. The implicit faimess criterion of unconstrained learning. International Conference on Machine Learning, 2019.
Christos Louizos, Kevin Swersky, Yujia Li, Max Welling, and Richard Zemel. The variational fair autoencoder. In International Conference Learning Representations, 2015.
David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel. Learning adversarially fair and transferable representations. In International Conference on Machine Learning. 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. 2018.
Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. Dissecting racial bias in algorithm used to manage the health of populations. Science, 366(6464):447-453, 2019.
Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant pre- diction: identification and confidence intervals. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 78(5):947-001,, 2016.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar, Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, 2019.
10

---

Under review as a conference paper at ICLR 2021
R Tyrrell Rockafellar and Stanislav Uryasev. Conditional value-at- for general loss distributions. Journal of banking & finance, 26(7):1443-1477) 2002.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. In International Conference on Learning Representations, 2019.
Andrew D Selbst, Danah Boyd, Sorelle A Friedler, Suresh Venkatasubrammnnian and Janet Vertesi. Fairness and abstraction in sociotechnical systems. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59-68, 2019.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014.
Don Bambino Geno Tai, Aditya Shah, Chyke A Doubeni, Irene G Sia, and Mark L Wieland. The disproportionate impact of covid-19 on racial and ethnic minorities in the united states. Clinical Infectious Diseases, 2020.
Serena Wang, Wenshuo Guo, Harikrishna Narasimhan, Andrew Cotter, Maya Gupta, and Michael Jordan. Robust optimization for fairess with noisy protected groups. In Neural Informational Processing Systems, 2020.
Robert C Williamson and Aditya Krishna Menon. Fairness risk measures. 2019.
Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair representations. In International Conference on Machine Learning, pp. 325-333, 2013.
Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adver- sarial learning. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pp. 335-340, 2018.
Yuan Zhang, Regina Barzilay, and Tommi Jaakkola. Aspect-augmented adversarial networks for domain adaptation. Transactions of the Association for Computational Linguistics, 5:515-528, 2017.
Han Zhao, Remi Tachet des Combes, Kun Zhang, and Geoffrey J Gordon. On learning invariant representation for domain adaptation. International Conference on Machine Learning. 2019.
A EIILV1 PSUEDOCODE
Algorithm The first stage of EllLv1 infers two environments that maximally violate the IRM objective. The inferred environments are then used to train an IRM solution from scratch.
Input:
Reference model + dataset D {a;, y1} with Fı pobe iid, loss function e, duration
Nateps
Output: Worst case data splits D1, D2 for use with IRM.
Randomly initialize e E R! as vectorized logit of posterior with o(e,) == g(e|o,, v.). for n
Nsteps do
R E, ((( v:):
// risk // D1 invariance
G V Ww/w=ll (((() ((e) ((.)).
regularizer
R² E.Il-o(e E, ((1l-(()))(((zz)..))))
// D2 risk
G V Vw/w=ill -. E, ((; (
// D2 invariance
regularizer
1 Eee(1 X0"
c + poptimUpdate(e V.L)
end
é ~ Bernoull((o(e));
11 sample splits 1/ split data
Dj t (I. = 1}, D2 + (on, = 0}:
11

---

Under review as a conference paper at ICLR 2021
B PROOFS
B.1 PROOF OF THEOREM 1
Consider a dataset with some feature(s) which are spurious, and other(s) v which are valu- able/causal w.r.t. the label y. This includes data generated by models where v - y + 2, such that P(y/e, 2) P(ylv). Assume further that the observations x are functions of both spurious and valuable features: z == f(e, 2). The aim of invariant learning to form a classifier that predicts from that focuses solely on the causal features, i.e., is invariant to and focuses solely on a.
Consider classifier that produces a score (ax) for example æ. the binary classification setting is analogous to the model o while the score ((z) is analogous to the representation (z)). To quan- tify the degree to which the constraint in the Invariant Principle (EI-CONSTR] holds, we introduce measure called the group sufficiency gap
((S,e) (()(()(()), (()))()).e2)1
Now consider the notion of an environment: some setting in which the æ + relationship varies (based on spurious features). Assume single binary spurious feature We restate Theorem I as follows:
Claim: If environments are defined based on the agreement of the spurious feature and the label y. then classifier that predicts based on 2 alone maximizes the group-sufficiency gap (and vice versa a classifier predicts y directly by predicting z. then defining two environments based on agreement of label and spurious feature e1 = {v, 2, yl (y = =)} and the gap).
{v, 2, yl I (y # =)} maximizes
We can show this by first noting that if the environment is based spurious feature- agreement,
then with e E {0, 1} we have (y = z). classifier predicts z, i.e. S(x) = 2, then we have
(SS,e) EEyy ( (g 2)] Elyl=(x), 1 + =)]l
For each instance of x either
Now we note that when z = 1 we have E(ylz, 1 = =))= =1 and Elylz 1 # =))= =0, while when Elylz, fly = =]) = 0 and Elyl 1( + x)] =1 Therefore for each example [x())(()). 1l = 2)) + 1(y + 2) 1, contributing to an overall A(S,e) 1,
or z
which is the maximum value for the sufficiency gap.
B.2 GIVEN CMNIST ENVIRONMENTS ARE SUBOPTIMAL W.R.T. SUFFICIENCY GAP
The regularizer from IRMv1 encourages a representation for which sufficiency gap is minimized be- tween the available environments. Therefore when faced with a new task it is natural to measure the natural sufficiency gap between these environments, mediated through naive or baseline method. Here show that for CMNIST, when considering naive color-based classifier as the reference model, the given environment splits are actually suboptimal w.r.t sufficiency gap, which motivates the proposed EMIL approach for inferring environments that have more sever sufficiency gap for the reference model.
We begin by computing A(S, e). the sufficiency gap for color- classifier 9 the given train environments {e1, }- We introduce an auxiliary color variable which is not observed but can be sampled from via the color based classifier g:
p(vlg(z) x', Eptt (((l2,,,,').)
10 was previously used in fairness setting by Liu et al. 2019 measure differing calibration curves across groups
12

---

Under review as a conference paper at ICLR 2021
Denote by GREEN and RED the set of green and red images, respectively. Le. we have G iff 2 = and r E GREEN iff z(x)) = 1. The the sufficiency gap is expressed as
e( Ep(z,e)
Eplule,e ( ei] Bpt pl (Vlg(z).
(okulk.on)lll,.cc p [u|
1
Bptulzer) |8/ e1] - ( e2]
2
se SE (GRBEN,
1
(|E[ylz GREEN, ei] Elyl= GREEN, 02/ |Byy]2 RED, ei] Elylz RED, e2]))
2
1
(10.1 -0.2| 10.9 0.8))
2
10
The regularizer in IRMv1 is trying to reduce the sufficiency gap, in some about this gap as a learning signal for the IRM learner. natural question would be whether a different set of environment partition {e} can be found such that this learning signal is stronger, i.e. the sufficiency gap is increased. We find the answer is yes. Consider an environment distribution qel|,,, y, =) that assigns each data point to one of environments. Any assignment suffices far as its marginal matches the observed data: f= Sa ((r, y; 2, e) = = p y)-
think
We express the sufficiency gap (given color-based classifier g) function of the envi- ronment assignment
ASS 9) = By(a,e) ( ei] (vlg(() e2]] ( (agge (x(l)
e2ll] e2ll
Where we use the same change variables trick as above replace g(x) with samples from p(=|a> (note that this is the color factor from the generative process p according with our assumption that matches this distribution).
We want to show that there exists a q yielding higher sufficiency gap than the given environments. Consider that yields the conditional label distribution
[1(y =2) ife C1 11 + if e2.
( ((yle,z) ())lee)))
This be realized by encoder/auditor q(e/2, y, that ignores image features in and parti- tions the example based on whether or not the label and color x agree. We also note that z is deterministically the color of the image in the generative process: p(=[2r) [ RED)
13

---

Under review as a conference paper at ICLR 2021
Now compute the sufficiency gap:
A (S, q)
Beg.eelllBeeelssccaaatttttll - Be
ei] Bgul
EZEGREEN
lulz, en] - Bogjlkeeeeeneeeel))bbll..
2
EEEERAD
y ( = =) ( = =)) - -E yy 1ly 2) [(g(x) = =))1)
2
y
2
+ EREGGRENN
y * (1 = ((((x) = =z]) Σ
Σ
yy * 1 (y 2) (((()) = =))1)
2
y
2
EarERAD (
Σ
(y (1y = 1) * 1(x E RED)) E(v (v 11 + 1) 1 RED))))
2
+ ERE
Σ wy 1y= 1xx E GREEN)) - ΣΣSr wv 1(y + 1(x E GREEN))[)
2
ク 10
1
ERERRDD - 0I] + E E
2 =1
Note that 1 the maximal sufficiency gap, meaning that the described environment partition maxi- mizes the sufficiency gap w.r.t. the color-based classifier g.
C DATASET DETAILS
Constructing the ConfoundedAduut dataset To create semi-synthetic dataset, called Con- foundedAdult, we start by observing that the conditional distribution over labels varies across the subgroups, and in some cases subgroup membership very predictive of the target label. We con- struct a test set (a.k.a. the audit set) where this relationship between subgroups and target label is reversed.
The four sensitive subgroups are defined following the procedure of Lahoti et al. (2020), with sex (recorded binary: Malc/Female) and binarized race (Black/non-Black attributes compose to make four possible subgroups: Non-Black Males (SG1), Non-Black Females (S2), Black Males (SG3), and Black Females (SG4).
We start with the observation that each subgroup has a different correlation strength with the tar- get label, and in some cases subgroup membership alone can be used to achieve relatively low enror rates in prediction. As these correlations should be considered "spurious" to mitigate unequal treatment across groups, we create semi-synthetic variant of the UCI Adult dataset, which we call ConfoundedAAuult, where these spurious correlations are exaggerated. Table 4 shows various conditional label distributions for the original dataset and our proposed variant. The test set for Confounded Adult revereses the correlation strengths, which can be thought of as worst-case audit to ensure the model not relying on subgroup membership alone in its predictions. We generate samples for ConfoundedAduit using importance sampling, keeping the original train/test splits from UCI Adult as well as the subgroup sizes, but sampling individual examples underlover-sampled Con.JoundeuAddtt
according importance weights procraai
D EXPERIMENTAL DETAILS
Model selection Krueger et al. (2020) discussed the pitfalls achieving good test performance on CMNIST by using test data to tune hyperparameters. Because our primary interest is in the properties of the inferred environment rather than the final test performance, we sidestep this issue in the Synthetic Regression and CMNIST experiments by using the default parameters of IRM without further tuning. However for the ConfoundedAdult dataset a specific strategy for model selection is needed.
14

---

Under review as a conference paper at ICLR 2021
Subgroup (SG)
ply 1[S
UCIAdult
ConfoundedAdult
Train
Test
Train
Test
0.31
0.30
0.94
0.06
2
0.11
0.12
0.06
0.94
3
0.19
0.16
0,94
0,06
0.06
0.04
0.06
0.94
Table 4: AonfoundedAAult is variant of the UCI Adult dataset that emphasizes test-time distrbu- tion shift.
We refer the interested reader to Gulrajani Lopez-Paz (2020) for an extensive discussion of pos- sible model selection strategies. They also provide large empirical study showing that ERM is difficult baseline to beat when methods are put on equal footing w.r.t model selection.
In our case, we use the most relaxed model selection method proposed by Gulrajani & Lopez-Paz (2020), which amounts to allowing each method at random from reasonable range, with the best hyperparameter setting selected for each method. While none of the methods is given an unfair advantage the search over hyperparameters, the basic model selection premise does not translate real-world applications, since information about the test-time distribution is required to select hyperparameters. Thus these results can be understood as being overly optimistic for each method, although the relative ordering between the methods can still be compared.
test evaluations using hyperparameter chosen
CMNIST IRM is trained these two environments and tested holdout environment con- structed from 10, 000 test images in the same way as the training environments, where colour is predictive of the noisy label 10% of the time. So using color as a feature to predict the label will lead to accuracy of roughly 10% on the test environment, while it yields 80% and 90% accuracy respectively on the training environments.
To evaluate IRM(CEI)) we remove the environment identifier from the training set and thus have one training set comprised of 50, 000 images from both original training environments. We then train an MLP with binary cross-entropy loss the training environments, freeze its weights and use the obtained model to learn environment splits that maximally violate the IRM penalty, When optimizing the inner loop of EIIL, we use Adam with learning rate 0.001 for 10, 000 steps with full data batches used to computed gradients.
The obtained environment partitions then used to train new model from scratch with IRM. Following Arjovsky et al. (2019), we allow the representation to train for several hundred annealing steps before applying the IRMvl penalty.
Census data Following Lahoti et al. (020)), we use a two-hidden-layer MLP architecture for all methods, with 64 and 32 hidden units respectively, and a linear adversary for ARL. We optimize all methods using Adagrad; learning rates, number of steps, and batch sizes chosen by the model selection strategy described above (with 20 test evaluations per method), as are penalty weights for IRMv1 regularizer and standard weight decay. For the inner loop of EIIL (inferring the environ- ments), we use the same settings as in CMNIST. We find that the performance sensitive to the number of steps taken with the IRMv1 1 penalty applied. To limit the number of test queries needed during model selection, we use an early stopping heuristic by enforcing the IRMv1 penalty only during the final 500 steps of training, with the previous steps serving annealing period to learn baseline representation to be regularized. Unlike the previous datasets, here we minibatches compute gradients during IRM training (for consistency with the ARL method, which uses minibatches). However, full batch gradients are still used for inferring environments in EIII..
EIIL somewhat
15

---

Under review as a conference paper at ICLR 2021
Causal MSE
Noncausal MSE
ERM
0.827 0.185
0.824 = 0.013
ICP(enc)
1.000 + 0.000
0.756 = 0.378
IRM((Hc)
0.666 0.073
0.644 = 0.061
IRM(eEm))
0.148 0.185
0.145 0.177
Table 5: IRM using EllL-discovered environments (EEIL) outperforms IRM in a synthetic regres- sion setting without the need for hand-crafted environments (erc). This is because the reference representation PERM uses the spurious feature for prediction. MSE standard deviation across 5 runs reported.
E ADDITIONAL EMPIRICAL RESULTS
E. SYNTHETIC DATA
We begin with regression setting originally used as a toy dataset for evaluating IRM (Arjovsky al. 2019). The features E R comprise a "causal" feature v RN/2 concatenated with
"non-causal" feature R N/2 [v, z]. Noise varies across hand-crafted environments
V =
No 25)
V + fy
"y e²
Z y + Cz
6 ~(0,1).
We evaluated the performance of the following methods:
ERM: A naive regressor that does not make of environment labels but instead opti-
mizes the average loss on the aggregated environments;
IRM(eHc): the method of Arjovsky et (2019) using hand-crafted environment labels;
ICP(CHс): the method of Peters et al. 2016) using hand-crafted environment labels;
(eM(e our proposed method (which does use hand-crafted environment labels) that infers useful environments based on the naive ERM, then applies IRM to the inferred vironments.
The regression methods fit scalar target y = 1f via a regression model ~ w x to minimize ly - b/1 w.r.t. w, plus an invariance penalty as needed. The optimal (causally correct) solution is w = [], Given solution [W., w.2] from one of the methods, report the mean squared error for the causal and non-causal dimensions as l|w, 1/3 and l|w 0 (Table Because v is IRM using hand-crafted environments, denoted IRM(eHc), exploits variability in noise level in the non-causal feature (which depends on the variability of σy) achieve lower error. Using EIILv] instead of hand crafted environments yields an improvement on the resulting IRM solution (denoted IRM((Em)) by learning worst-case environments for invariant training.
marginally noisier than ERM focuses the spurious
We show in a follow-up experiment that the EIlLvv solution is indeed sensitive to the choice of reference representation, and in fact, can only discover useful environments (environments that allow IRM(Cnn)) to learn the correct causal representation) when the reference representation encodes the incorrect inductive bias by focusing on the spurious feature. We can explore this dependence of EIILvI on the mix of spurious and non-spurious features in the reference model by constructing that varies in the degree it focuses on the spurious feature, according to convex mixing parameter E [0, 1). a 0 indicates focusing entirely on the correct causal feature while a indicates focusing on the spurious feature We refer to this variant as IRM(eEuL) - = o SPcROOU)) and measure its performance as function of (Figure Environment inference only yields good test- time performance for high values of a, where the reference model captures the incorrect inductive bias.
16

---

Under review as a conference paper at ICLR 2021
155
IPM (cseened
(FM) FIL. Rpxu)
110
MSB
103
IPMYs
033
(a) Causal error
(b) Non-cause error
Figure 3: MSE of the causal feature v and non-causal feature 2. IRM( Japplied to the ERM solution (Black) out-performs IRM based the hand-crafted environment (Green Blue). To examine the inductive bias of the reference model t hard code model & SPSRRIOO where controls the degree of spurious feature representation in the reference classifier; IRM(C out- performs IRM((HH) when the reference & focuses on the spurious feature, egg. with as ERM or a-SPURIOUS for high a.
Train accs
Test aces
ERM
86.3 = 0.1
13.8 0.6
IRM(enc)
71.1 + 0.8
65.5 + 2.3
IRM(eEML ④ 4 FRRM)
73.7 + 0.5
68.4 + 2.7
IRM(erm. 1 Color
75.9 + 0.4
68.0 + 1.2
Color
85.0 = 0.1
10.1 + 0.2
GRAYSCALE
75.3 = 0.1
72.6 0.6
ARL
88.9 = 0.2
20.7 + 0.9
Table Accuracy across ten runs with label noise 0,, = 0.25 GRAYSCALE hard codes out the color feature and thus represents oracle solution to CMNIST.
F.2 COLORMNIST
Table expands the results from Table Ty adding additional methods discussed in Section 3
In Table 7 we measure the performance of some alternative strategies for optimizing the bi-level problem from Equation (EIIL). In particular, we consider alternating updates the representation & and environment assignments g, as well as solving the inner/outer loop of EIIL multiple times. On the CMNIST dataset, none of these variants offers a performance benefit above the method used in Section
EIIl -lops=k indicates that the inner and outer loops of the EIIL objective in Equation (EIIL) successively optimized k times, with & = 1 corresponding to IRM(eem)). the method studied in the main experiments section. In other words, Pioops=1 is solved using IRM(erm.). ) then this rep- resentation used as a reference classifier to find $ Pioops=kt1 = IRM(egIIL $ = @toope=k) the
Train aces
Test accs
Moops=1
73.7 + 0.4
68.8 + 1.8
10ops=2
85.4 + 0.4
10.0 : 0.4
1oops=3
75,6
69.0 - 0.9
1oops=1
84.8 0.2
10.1 1.3
E
Moops=5
76.2 + 0.5
67.6 + 0.7
EIII
AttUipatts
82.3
0.4
24.0 + 1.1
Table 7: We measure performance CMNIST of various altemattve approaches optimizing the EIIL objective, ultimately concluding that none of the alternatives out-performs the method studied in Section See text for details
17

---

Under review as a conference paper at ICLR 2021
Train aces
Test accs
BIILv1
68.7 1.7
79,8 + 1.1
EIILv1 (no regularizer)
78.6 + 2.0
69.2 + 2.8
IRM (random environments)
94.7 0.1
17.6 + 1.6
Table Our ablation study shows that both ingredients of E (finding worst-case environments and regularizing invariance across them) are required to achieve good test-time performance on the ConfoundedAdull dataset.
next "loop" of computation. This also means that the training time needed is A times the training time of IRM((euuL) As we expect from our theoretical analysis, using the IRM(C solution as a reference classifier for another round of EIIL is detrimental: since the reference classifier already relies on the correct shape feature, environments that encourage invariance to this feature are found in the second round, so the Elll 2topss=2 classifier uses color rather than shape.
A1U consists of optimizing Equation uuiin alternating steps to + and q. Unforatun- tely, whereas this strategy works well for other bi-level optimization problems such as GANs, it seems to do poorly in this setting. This method outperforms ERM but does not exceed chance-level predictions on the test set.
E.3 CENSUS DATA
Ablation Here we provide ablation study extending the results from Section to demon- 4.2
strate that both ingredients in the ElILv1 solution fintiion-inding worst-case environment splits and reg- ularizing using the IRMv1 penalty- necessary achieve good test-time performance the Adul dataset.
that ARL perform favorably compared with DRO (Hashimoto et al. 2018) in adaptively computing how much each example should contribute to the overall loss, Le. computing the per-example ,/8 C Ez..u<p I (x:)) y:)]. Because all per-environment risks in IRM are weighted equally (see Equation 2). and each per-environment risk comprises an average across per-example losses within the environment, each example contributes its loss the overall objective in accordance with the size of its assigned environment. For example with two environments e1 and e2 of sizes |eal and |e2). we implicitly have the per-example weights of % 1 for E and % Jeg] for 62, indicating that examples in the smaller environment count [eg]
From Lahoti al. (020))
more towards the overall objective. Because EIILv1 is known to discover worst-case environments of unequal sizes, we measure the performance of EIILy1 using only this reweighting, without adding the gradient-norm penalty typically used in IRM (i.e. setting À 0). To determine the benefit of worst-case environment discovery, we also measure IRM with random assignment of environments. Table & shows the results, confirming that both ingredients required to attain good performance using EILvv..
F ADDITIONAL THEORETICAL RESULTS
F.I OPTIMAL SOFT PARTITIONS MAXIMIMALLY VIOLATE THE INVARIANCE PRINCPLE
We want to show that finding environment assignments that maximize the violation of the softened version of the regularizer from Equation also maximally violates the invariance princple. Because the invaraince principle Y (xX),, e] = E|Y (X),, e']ve, e' is difficult to quantify for continuous e'))eeeVe,, ((X), we consider a binned version of the representation, with b denoting the discrete index of the bin in representation space. Let q: E [o. 1] denote the soft assignment of example to environment 1, and - 4 denote its converse, the assignment of example to environment 2. Denote by y: {0. 1} the binary target for example i, and ý E [0, 1] as the model prediction on this example. Assume that l represents cross entropy or squared emror loss that Vi y) (ú (x))
18

---

Under review as a conference paper at ICLR 2021
Consider the IRMvl regularizer with soft assignment, expressed
D(q)=
Σ
V w| Ne Σ
(i(e)(0: y1 ((:))l
Ci - -.) +
Σ/ 16 (1- 9:(00 -
01)0(2;)11
Σ' 1/
D; 4:0:002a
E. (Ii) Ex W
E1(1 q: (1)0:((a)))
E.(1 9: (xi))
12 +
Σx 9u
Σ. 1 9r
Σv 9e
(4)
Now consider that the space of ((X) is discretized into disjoint bins b over its support, using 2i,6 {0, 1) to indicate whether example falls into bin b according to its mapping ((x:). Thus we have
Σ 2.69:00. Ep 24.
E, (1.000110(zz)) Ev 21,690
E, 21,0 - (.)s, Ex 24,0 (w)
Σ 21,611 - 9: (:));(((z)))
D(g) Σ(1 6
1|
113)
Σx 21,( gr)
(5)
The important point is that within bin, all examples have roughly the same ((x,) value, and the
same value for J as well. So denoting K.( := Σ. (( and K,"2 == En (,((1-g)) as the relevant constant within-bin summations, we have the following objective to be maximized by EIIL:
E. #2,0%,
D())= = Σ (IIK ELZM
E, - (1.)
1|2 + +I|!!
Ex ALMME
Ev 2i,6 49
One way to maximize this to assign all y; = values to environment (q: = 1 for these examples) and all y of the examples except the i-th one have been assigned this way, and then that the loss is maximized by assigneing the i-th example according this rule.
the other environment (q: 0). We can show this is maximized by considering all
Now we want to show that the same assignment maximially violates the invariance principle (show- ing that this soft ElIL solution provides maximal -on-invariance). Intuitively within each bin the difference between E[yle = 1] and Elyle = 2] is maximized (within the bin) if one of these ex- pected label distributions is while the other values to the first environment and the = 0 values to the second.
This can be achieved by assigning all the y;
Thus global optimum for the relaxed version of EIIL (using the IRMvl regularizer) also maximally violates the invariance principle.
19


Reviews:
[{"review_id": "op6_KJK-yGd", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### POST-REVISIONS ###\nThanks for the revisions made to the theoretical results. I still find parts of the discussion in Appendix F to be unclear. \n\nFirstly, how do you derive eq. (5) from eq. (4)? In eq. (4), the denominators \\sum_i q_i are independent of \"\\Phi(x_i)\", but in eq. (5), they have a dependence on \\Phi through z_{i,b}. I think the change in normalization important to show the invariance principle holds (as the invariance principle requires a conditioning on each value \\Phi takes), but am unable to follow your derivation.\n\nSecondly, I'm not convinced that the maximizing partition for eq (5) assigns all examples with y=1 to one group, and those with y=0 to another group. Wouldn't the maximizing partition also depend on what \\hat{y} evaluates to for those examples?\n\nOverall, I'm able to see what the authors are trying to get at with this example, but unfortunately the revisions aren't sufficient to address all of my concerns regarding the theoretical results.\n********************\n\nThe paper presents an approach for training models that generalize well to out-of-distribution (OOD) samples, particularly when the source of domain shift (e.g. spurious correlations or sensitive groups) is not known before hand. The paper combines ideas from two prior papers from the domain generalization and fairness literature: (i) invariant  risk minimization for OOD generalization (Arjovsky et al.) and (ii) adversarially reweighting for fairness without protected groups (Lahoti et al.). \nAt a high level, the proposed approach seeks to minimizes the average classification loss across the worst-case partitioning of the dataset into two groups. Experimental results on datasets with synthetically generated spurious features show that the proposed approach is able to generalize better to OOD samples in the high noise regime, without having knowing aprior which features are spuriously correlated with the labels.\n\nPros:\n- The question tackled is practically important: how one can generalize to OOD samples without knowing the exact source of discrepancy between train and test data. \n- The experimental results look encouraging\n\nCons:\n- The paper lacks a clear theoretical motivation for the specific optimization objective that the authors end up using (eq 3). In particular, do we know (at least in some a simple setting) that maximizing this objective over soft-group memberships \"u_i\" will identify the partitioning of the data that maximally violates the Invariant Constraint? I elaborate on this next.\n\nRelaxed training objective lacks strong theoretical backing:\nThe authors directly adapt the training setup of Arjovsky et al., where the goal is to train a model which learns the same conditional label distribution for any given input \"x\" across a set of known partitioning of the training data, dubbed as the invariance constraint . Each of these partitions, referred to as 'environments', represent a different training distribution, and the goal is to train a model that performs equally well across all of them. Arjovsky et al. show that for the special case of linear invariant predictors, the training problem can be relaxed into an unconstrained objective with a regularization penalty. \n\nThe present paper extends the setup of Arjovsky et al. to problems where the environments are not a prior known, and seeks to minimize the average classification loss over a partitioning of the data that maximally violates the invariance constraint. However, they do not explicitly solve this optimization problem, and instead simply minimize the worst-case value of the \"relaxed training objective\" of Arjovsky et al. over all (soft) partitioning of the data.\n\nIs the relaxation that Arjovsky et al. employ with known environments still relevant to your problem formulation, where you would like the invariance constraint to hold for all possible partitioning of the data?\n\nAt the very least, this requires a discussion. Ideally, it would be nice to see a derivation of the relaxation for some simple special cases: e.g. like Arjovsky et al., can you show that for linear predictors, \"finding a partition that maximally violates the invariant constraint\" is equivalent to \"maximizing the relaxed unconstrained objective in eq. 3 over partitions\"?\n\nOther comments:\n- Eq 3:  I think \"w\" is a scalar here (otherwise evaluating the gradient at w = 1.0 doesn't make sense). Please make that explicit and also provide some intuition for why this regularization penalty with a scalar \"w\" makes sense for your problem set up.\n- I am not entirely sold on the general theme of this paper of exchanging lessons between fairness and domain generalization. The authors are definitely correct in crediting a prior fairness paper for the idea of  the idea of adversarially re-weighting examples with a soft groups model, but as they themselves point out this idea has existed in different forms in the domain generalization literature (e.g. DRO). So my reading is that the paper seems to slightly over-emphasize the connection to the fairness literature, but this is a personal take. Having said this, the paper does provide (in Sec 2) a nice literature overview of similar problems tackled by the domain generalization and fairness communities.\n- In the color MNIST experiments, you observe \"IRM(eEIIL) generalizes better than IRM(eHC) with sufficiently high label noise\". If I understand correctly, IRM(eHC) has access to the true environments, whereas IRM(eEIIL) uses environments inferred from data. Wouldn't we expect the former method to have an advantage over the latter?\n- Additional baseline: Would it make sense to compare with (a form of) DRO for the color MNIST task (e.g. ones cited in Table 2)?  You do mention in another experiment that Lahoti et al. compare with DRO for their particular fairness application, but do those observations also apply yo the tasks you consider in this paper.\n- Iterative training: I think a natural extension of your approach (which you've probably already thought about) is to solve (EIIL) using an iterative technique that alternates between maximizing over \"q\" and e.g. performing gradient descent updates on \"\\Phi\". Iteratively performing full optimizations over both sets of parameters may not in general have good convergence properties.\n- Might be a relevant citation for the use of soft partition assignments for fairness: https://arxiv.org/pdf/2002.09343.pdf\n- Fig 1: Would be nice if the plots were color blind friendly :)\n- References: Might be good to mention the conference venue wherever available: e.g. Hashimoto et al. appeared in ICML 2018.", "extra_scores": {}, "extra_text": {}}, {"review_id": "_rN8KFzJMmb", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The main contribution of the paper is to highlight the similarity between two active areas in ML namely \"domain generalization\" and \"fairness\". Further, the paper proposes an approach inspired by recent developments in the fairness literature for domain generalization. The high-level idea is that similarly to the way that fair algorithm are able to improve the worst-case accuracy of predictors across different groups without knowing the sensitive attributes, perhaps we can use these ideas to domain generalization when environment partitions are not known to the algorithm. In some sense, in both of these research areas the goal is to design robust algorithms. Similarly, the paper uses the idea from domain generalization to design fair algorithms w.r.t. a notion called \"group sufficiency\". The idea is to somehow infer the \"worst-case\" subgroup (i.e., the one that our algorithm has the worst accuracy on it) and then using a round of auditing improve the performance of the algorithm across all subgroups.\n\nThe authors have supported their approach with empirical evaluations. In particular, I find the result on CMNIST quite interesting where the new algorithm as opposed to the standard approach like ERM will not be fooled by the spurious feature and can infer the useful environment.     \n\nWhile the paper has introduced (to best of my knowledge) a new concept, it seems that are many interesting questions that could show the applicability of the connection better are not yet answered (e.g., bi-level optimization EIIL). This could also help the paper to be supported with more provable guarantees. In general the paper is exploring a new connection between two areas and has shown its efficacy in practice and I believe it can lead to further works on this topic.   \n\n\nMinor comments:\n- define the notion of \"group sufficiency\" explicitly in the paper. I could not find the definition of the notion in words till in the caption of Figure 2 on page 8 and is formally defined on page 12!\n-page 5: poorly. . Consider -> poorly. Consider\n-page 6: generalizattion -> generalization\n-page 7: graysacle ->grayscale\n-page 14: exagerated -> exaggerated\n-page 14: orginal -> original\n-page 17: implicily -> implicitly", "extra_scores": {}, "extra_text": {}}, {"review_id": "A8vA0SBZUIV", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThis paper studies the connections between algorithmic fairness and domain generalization. As discussed in Section 2, the “environment” in domain generalization plays a similar role as the “group membership” in algorithmic fairness. The paper shows in Table 2 that the methods of each field can apply to the other field. \n\nThe paper develops its own algorithm EIIL which extends the Invariant Risk Minimization (IRM) of domain generalization to work in the situation when the prior knowledge of environments is not available. And this extension is mainly based on the idea from algorithmic fairness literature which considers the worst-case environments and solves a bi-level optimization. \n\nThe paper shows empirically that their algorithm EIIL outperforms IRM with handcrafted environments in terms of test accuracy on CMNIST.\n\nStrength:\n(1) The connection between domain generalization and algorithmic fairness shown by the paper is interesting.\n(2) The paper demonstrates the performance of EIIL via empirical results.\n\nWeakness:\n(1) Other than the high level intuitions and examples, the paper does not provide any theoretical analysis of the performance of the EIIL for domain generalization. What guarantees can EIIL get in terms of the test error and how does it compare to IRM (when making reasonable assumptions about the training and test distributions)?\n(2) Similarly, the paper does not provide any theoretical analysis of EIIL for algorithmic fairness.\n(3) On top of page 6, after explaining the bi-level optimization, the paper switches to the sequential approach (EIILv1) without much explanation. Why is the bi-level optimization not practical? How well can the proposed sequential approach approximate the bi-level optimization results and how does this affect the performance of EIILv1?\n\nReasons for score:\nOverall I vote for rejection since the weakness outweighs the strength. The lack of theoretical analysis of the algorithm makes the paper incomplete. \n\nTypo:\nPage 5: two periods after word “poorly”.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "00OGPI-NfFl", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents parallels between algorithmic fairness and domain generalization literatures. The authors explore a learning setup where the goal is to learn some representation $\\Phi(x)$ that is \"independent\" of some environmental variable $e$. The authors explore cases where $e$ is known or not and come up with some algorithms that draw connections between recent work on domain generalization, specifically invariant risk minimization (Arjovsky et al 2019) and fairness. The authors conclude the paper with three different examples addressing domain generalization and fairness. While the supporting experimental results are not very strong, the connections observed are interesting. I have several points for clarification that I detail below.\n\n**Exposition.** While the introduction of the paper is written nicely and the ideas are communicated nicely, the mathematical exposition and presentation in this paper is not self-contained. For example, it is close to impossible for a reader to follow this paper without having read (Arjovsky et al 2019) and (Liu et al. 2018). I suggest that the authors expand on the mathematical exposition, defining all terms, and explaining different things came from, especially since the paper is supposed to have a broad audience across two communities.\n\n**Convergence of Algorithm.** The proposed algorithm comes with no guarantees and I suspect it will not converge in a variety of situations, especially in cases where $C^{\\text{IRM}}$ is nonconvex in (EIIL). Why would you need to run the inner optimization and outer optimizations to convergence each time? Have you tried a GDA version of the algorithm?\n\n**Cost of the Algorithm.** The proposed algorithm is costly because IRM has to be solved multiple times before it converges. Can you please comment on the computational cost of the proposed algorithm as compared to ERM and other baselines in each experimental setup?\n\n**Limitations of generalization-first fairness.** This section is nicely written and much appreciated.\n\n**Choice of $\\Phi_{spurious}$.** The algorithm is sensitive to initialization choice of $\\phi_{spurious}$  as the authors also find with their Color MNIST experiments. In particular, there is a huge performance gap in Fig 1.b. for $\\theta_y \\in (0, 0.15)$ that needs to be addressed. On the other hand, the algorithm seems to work well in the severely overfitting regime where ERM can be thought of having learnt $\\Phi_{spurious},$ as also discussed by the authors in the second paragraph of Section 3.2. However, the real world is not so black and white and hence this poses a severe limitation. Can you please explain?\n\n**Connection with (Liu et al. 2018).** In the third paragraph of **Fairness** section in Page 4, the authors claim a connection between IRMv1 (btw, IRMv1 is misspelled as IMRv1 there) and (Liu et al. 2018). This connection is not obvious to me. Can the authors make it rigorous?\n \n**Continuous $e$.** Can you please comment on how this setup may generalize to continuous $e$? At least can you please comment on the scaling of the algorithm with the cardinality of the set of environments?\n\n**Theorem 1.** Unfortunately, Theorem 1 and entire Section 3.2 is only applicable to a severely unusual and overfitting case (similar to the Color MNIST example) where there is perfect correlation between the environment variable and the label. The fact that the algorithm works well in this situation is not surprising. The real world, however, is not black and white and the limitations of the proposed framework in real-world situations (similar to $\\theta_y \\in (0, 0.15)$ in Fig 1.b.) remain to be understood.\n\n**Confounded Adult Dataset.** Please make it clear that this is constructed by the authors. I only understood that when started looking for the details in Appendix.\n\n**Typos.** (1) In Eq. (2), $w\\circ\\Phi$ should be replaced with $w.\\Phi$. (2). second paragraph of **Fairness** paragraph on Page 4, the word attribute is missing after \"sensitive\".\n\nOverall, while the subject area of the paper is exciting, unfortunately, the execution (both empirical and theoretical) is weak. I tend to remain to vote for rejection with encouragement for a more thorough empirical and theoretical investigation of the problem.\n\n---post rebuttal---\n\nAfter reading the authors' response, the other reviews, and the revision to the paper, I find that my comments are not sufficiently addressed. The author did not even acknowledge the existence of the prior work, REPAIR, in the revised paper. The imprecise mathematical expressions are still in the paper despite feedback from multiple reviewers. From a practical point of view, the developed algorithm is not scalable as it requires to (almost) solve the inner maximization at each iteration (based on the rebuttal), and it only works in the significantly overfitting regime (the authors are yet to show its performance in a more interesting regime). From a theoretical point of view, the applicability of the theory is also extremely limited to the perfectly overfitting regime, which does not capture the real world. In addition, I agree with AnonReviewer4 that the proofs are inscrutable.  I regret to say that despite the fact that the subject area of the paper is exciting, I am adjusting my score to 4 post rebuttal.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper analyzes connections between algorithmic fairness and domain generalization literatures. The reviewers found the paper interesting but they also raised some important concerns about it.

The applicability of the method presented in the paper is not clear nor well-discussed in the paper.

The papers and the revised version do not not cite important related work.

The mathematical exposition in the paper is a bit hard to read. Even after revision, the reviewers find part of the paper(Appendix F) very hard to read.

Overall, the paper in the current version is below the high acceptance bar of ICLR.
Decision: Reject