Multi-Expert Distributionally Robust Optimization for Out-of-Distribution Generalization
Authors: Jinyong Jeong
Hyungu Kahng
Seoung Bum Kim
Venue: NeurIPS 2025 poster ()
Keywords: Out-of-Distribution Generalization
Distributional Shift
Subpopulation Shift
Domain Generalization
Distributionally Robust Optimization
Primary area: deep_learning
TL;DR: We introduce Multi-Expert DRO (MEDRO), a novel distributionally robust optimization framework with multiple experts explicitly modeling cross-environment risks to improve out-of-distribution generalization against complex distributional shifts.

Abstract: Distribution shifts between training and test data undermine the reliability of deep neural networks, challenging real-world applications across domains and subpopulations. While distributionally robust optimization (DRO) methods like GroupDRO aim to improve robustness by optimizing worst-case performance over predefined groups, their use of a single global classifier can be restrictive when facing substantial inter-environment variability. We propose Multi-Expert Distributionally Robust Optimization (MEDRO), a novel extension of GroupDRO designed to address such complex shifts. MEDRO employs a shared feature extractor with $m$ environment-specific expert classifier heads, and introduces a min-max objective over all $m^{2}$ expert-environment pairings, explicitly modeling cross-environment risks. This expanded uncertainty set captures fine-grained distributional variations that a single classifier might overlook. Empirical evaluations on a range of standard distribution shift benchmarks demonstrate that MEDRO often achieves robust predictive performance compared to existing methods. Furthermore, MEDRO offers practical inference strategies, such as ensembling or gating mechanisms, for typical scenarios where environment labels are unavailable at test time. Our findings suggest MEDRO as a promising step toward resilient and generalizable machine learning under real-world distribution shifts.

Multi-Expert Distributionally Robust Optimization for Out-of-Distribution Generalization
Jinyong Jeong' Hyungu Kahng²* Seoung Bum Kim
Department of Industrial and Management Engineering, Korea University, Seoul a Department of Convergence Business, Korea University, Sejong [jy-jeong, hgkahng, abkim))kkorea... ar.
Abstract
Distribution shifts between training and test data undermine the reliability of deep neural networks, challenging real-world applications across domains and subpopu- lations. While distributionally robust optimization (DRO) methods like GroupDRO aim to improve robustness by optimizing worst-case performance over predefined groups, their use of single global classifier can be restrictive when facing sub- stantial inter-environment variability. We propose Multi- Distributionally Robust Optimization (MEDRO), novel extension of GroupDRO designed address such complex shifts. MEDRO employs shared feature extractor with m environment-speeific expert classifier heads, and introduces a min-max objective over all m² enviroment pairings, explicitly modeling cross-environment risks. This expanded uncertainty set captures fine- fine-grained distributional variations that single classifier might overlook. Empirical evaluations distribution shift benchmarks demonstrate that MEDRO often achieves robust predictive performance compared to existing methods. Furthermore, MEDRO offers practical inference strategies, such as ensembling or gating mechanisms, for typical scenarios where environment labels unavailable test time. Our findings suggest MEDRO as a promising step toward resilient and generalizable machine learning under real-world distribution shifts.
range of standard
1 Introduction
Deep neural networks have achieved remarkable success under the assumption that training and test data are drawn from the same distribution. In real-world applications, however, this assumption often breaks down, and even minor deviations- known distribution shifts- can significantly impair performance 1 2. Such shifts are common across domains like medical diagnostics (e.g., changing demographics imaging protocols) and natural language processing (e.g., text from emerging sources), posing serious challenges for model reliability B
Two common types of distribution shift are subpopulation shift and domain shift (5] Subpopulation shift arises when specific subgroups such minority demographics exhibit different feature or label distributions from the majority population (6] 77. Domain shift, by contrast, involves broader changes data- gentrrtnnn processes, such as differences in visual style or linguistic domain [ We follow Koh et al. (5) in defining subpopulation shifts as variations within domain, and domain shifts as those occurring between distinct domains. Both types can degrade performance and require robust
generalization approaches 5 6 9
leading approach distributionally robust optimization (DRO), which minimizes the worst-case risk over a set of possible distributions 10
6 In GroupDRO, a prominent DRO variant, the model is
"Corresponingg authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

trained to minimize the maximum loss across predefined environments, promoting robustness under subpopulation shifts. However, GroupDRO relies on single global classifier and may struggle when optimal decision boundaries vary substantially between environments [[]. Moreover, its uncertainty set-defined over convex mixtures of environments-maa overlook nuanced interactions between them.
To address these limitations, we propose Mulri-Experr DRO (MEDRO), which extends GroupDRO by assigning a specialized expert head to each environment while sharing a common feature extractor. This design enlarges the uncertainty set from an (m 1) dimmennsill simplex an (m² 1)- dimensional one, accounting for all enverteenvio combinations. Each expert minimizes risk on its designated environment while also learning generalize across mismatched inputs from other environments.
We evaluate MEDRO across variety of distribution shift settings. On subpopulation shift benchmarks 66 12), MEDRO significantly improves worst-group accuracy. It also generalizes effectively to domain and mixed-shift benchmarks [5), consistently outperforming single-head DRO baselines.
Since environment labels are unavailable at test time, we introduce two inference strategies: simple ensemble that averages expert outputs, and gating network that adaptively weighs them based on the input. These strategies enable effective deployment of MEDRO without requiring environment annotations at inference time.
Overall, MEDRO extends the DRO framework by explicitly modeling expert-envirromennt interac- tions, offering a unified and principled approach to both subpopulation and domain-level shifts.
The remainder of the paper is organized follows. Section reviews out-of-distribution generaliza- tion and DRO, Section 3 provides the requisite background and introduces the proposed MEDRO framework. Section reports experimental findings, and Section concludes with potential future research avenues.
Summary of contributions:
Multi-expert DRO formulation: We propose principled extension of DRO that assigns a specialized expert head to each environment and optimizes over all expert-enviromment pairs. This formulation expands the DRO uncertainty set and captures cross-environment variations beyond the capacity of single-head models.
Theoretical analysis: We show that MEDRO extends GroupDRO by explicitly model- ing expert-environment interactions beyond group-wise risks. Our formulation recovers GroupDRO special case and enjoys convergence guarantees under standard assumptions.
Unified approach to subpopulation and domain shift: By modeling all ex- pert-environment interactions, our approach unifies subpopulation robustness and domain generalization within a single framework, supported by theoretical analysis and empirical results for both types of shifts.
2 Related work
2.1 Out-of-distribution generalization
Duttof-diittibuutio (OOD) generalization aims to ensure robust model performance when test distribu- tions deviate from those seen during training. A variety of methods address this challenge. One line of work focuses on invariant feature learning, which encourages representations that remain predictive across environments [13 14) often through distribution alignment adversarial training [15] [6]. Another direction builds on causal inference, assuming structural knowledge such as causal graphs or conditional independencies to identify stable predictors [17]. and includes approaches that model
domain shift as selection bias [8]
Complementary perspectives include meta-learning, which optimizes models for cross-domaln generalization by simulating distribution shifts during episodie training [19] and data augmentation, which enriches the training distribution via transformations or perturbations 20 21.. Recent work
also explores generating synthetic domains 22 or aligning optimization dynamics 23 further improve generalization.

---

Despite their diversity, existing methods often rely on fixed assumptions about invariance or training distribution coverage, without explicitly addressing worst-case scenarios. As result, performance may degrade under severe unanticipated shifts. This motivates the use of more principled ap- proaches such as distributionally robust optimization (DRO), which explicitly targets worst-case robustness.
2.2
Distributionally robust optimization and generalization of robust models
DRO addresses distribution shift by optimizing worst-case risk
uncertainty [O 24 25
Classical DRO formulations define an uncertainty set Q around the empirical training distribution typically using eivergence-bbsed radius (e.g., an f-divergence Wasserstein ball) [24 25 26
27 Alternatively, formulations based on maximum mean discrepancy define an uncertainty set and establish connections between DRO and regularization in kernel methods (28]. While these approaches offer robustness to localized perturbations in (x, y). they are often less effective for
structured shifts, such as those involving subpopulations or domains.
To address this, early work explored DRO objectives under structured scenarios like label or data- source shift [29] 30]. Building on this direction, GroupDRO [6] defines the uncertainty set predefined groups, enabling reweighting within observed groups while disallowing shifts beyond their convex hull. By adversarially adjusting group weights, GroupDRO mitigates the risk of underrepresented groups being overlooked, thereby improving worst-group performance on minority groups. Recent work extends GroupDRO to affine combinations of training risks, encouraging robustness beyond the convex hull 9
However, GroupDRO and its extensions typically rely single classifier represent all groups. While shared models ensure coverage, they may overlook critical inter-group or domain-specific variation (5). This motivates DRO formulations that can explicitly model interactions between experts and environments a direction we pursue in this work.
Methodology
3.1 Problem setup
We consider supervised learning problem in which training data drawn from m distinct envi- ronments (also referred to as groups or domains), indexed by = .,., m. Each environment characterized by a probability distribution Pe over the input-output space X' x V. We denote sample from environment by (x,y) ~ Pe. The model parameters are denoted by 0 E O, where θ the parameter space, and a loss function L: y x y + R. quantifies prediction error.
Empirical risk minimization (ERM) is a common strategy that minimizes the average loss across observed environments (31). However, ERM often struggles when the test distribution deviates from what was seen during training, leading to poor generalization [5) 12)
3.2 From ERM to DRO
To tackle distributional shifts, distributionally robust optimization (DRO) focuses minimizing the worst-case expected loss over an uncertainty set Q. Formally, DRO solves:
min max E(x.x)-, [<(o)))))
(1)
#5e oEQ
The design of Q governs the types of distribution shifts the model becomes robust against. For instance, might comprise all distributions within a specified distance (e.g., Wasserstein ball) around an empirical measure 24 25
or consist of mixtures of known group distributions 6
3.3 GroupDRO framework
GroupDRO 6 well-known form of DRO for multiple environments, defines Q as the set of convex combinations of ..,....77 Pmn:
Q =
Po
A A Am).
(2)
c=
3

---

R.
61
Figure I: Conceptual visualization of uncertainty sets for m = 3 environments. Environments P, are represented by distinct locations in the (z.y)-plans the z-axis denotes risk. (a) GroupDRO: The risk surface (shaded triangle) for single classifier, defined over the convex hull of the = 3 base environment distributions. GroupDRO optimizes for the worst-case risk on this surface. (b) MEDRO (Individual Experts): Illustrates the three distinct risk surfaces (triangular patches), one for each of MEDRO's
experts (w; o Φ). Each surface shows how an individual expert i performs across all environments (i.e., visualizing its Ri, profile). (c) MEDRO (Expanded Uncertainty Set): The convex hull of all m² = 9 cross-environmed component is maximized over this significantly more comprehensive set, explicitly considering all expert-environment pairings. The size of this polytope adapts naturally to the degree of environment heterogeneity, preventing excessive pessimism when environments are similar while capturing complex variations when environments substantially differ.
risks (R4, points). MEDRO's 's worst-case
where = (AI,..., Am) lies in the probability simplex A. = { (X E Rm environment-wiss risk model on environment e is:
E
de = 1}. The
20
(0) = (x.))>>, (((f
(3)
Hence, GroupDRO solves:
min max 8EB XEAp
Ae (0).
This objective prioritizes the highest-loss environment, driving the model perform well even under worst-case shifts. In practice, at any training iteration t. GroupDRO uses an iterative procedure where R. (0c) is estimated for each environment e; ti) the weights A are updated (c.g.. by exponentiated gradient) to emphasize environments with higher risks; iii) gradient step updates 6g to reduce the weighted sum of losses. Under mild assumptions (e.g., convexity, bounded gradients, losses, and Lipschitz continuity), convergence occurs at rate O(1VVT) 32
3.4 Motivation for extensions
GroupDRO trains a single classifier to be robust against worst-case mixtures predefined environ- ment distributions. While effective, this single-classifie approach can be restrictive when optimal decision strategies inherently diverge across environments. Furthermore, its uncertainty set, based on evaluating this single strategy across mixtures, may explicitly isolate or optimize against all distinct classes of vulnerabilities. This motivates exploring extensions that can address these aspects through a more expressive uncertainty framework (see Appendix A.I for more detailed discussion).
3.5 Proposed method: Multi-Expert Distributionally Robust Optimization (MEDRO)
To this end, we propose MEDRO which extends GroupDRO by enlarging the uncertainty set Q to handle more potential distribution shifts. Consider neural net classifier X' - typically decomposed into shared feature extractor ø X' - Z and a linear (or shallow) head w Z - y. In MEDRO, maintain a shared feature extractor ó but introduce m expert heads, Wi,... Wms each tailored to one environment. Let 0 = {ő. ,i, wm J. For a pair (i, where expert i is used on data from environment, j. define the cross-environment risk
Ri, (0) (xxx))>, [e((ure o (x))
(5)

---

Rather than tracking just m risks {Re). MEDRO considers m² such cross- cross-enviromment risks (Ri,) serving as candidates for worst-case evaluation. This broader set of risks, corresponding to each potential mismatch between expert i and environment j. forms the hasis for our expanded uncertainty set (see Figure T for a conceptual illustration). This design recognizes that test-time inputs from environment may be inadvertently labeled or processed as if they came from i, especially when domain labels are uncertain.
We assign weights A to each pair (i,j), forming a matrix A E R that lies in the (m² - 1)- dimensional probability simplex Am² = {A E Rgo
= 1]. =1]. We further
>n ا
Σ
introduce a specialization term, > TM im1 (2), to ensure expert i remains proficient in environment i. Our complete objective
m
m
m X R. (0)
min
Σ (0) +
max AEA,.. 1=
(6)
where 7 balances the importance of cross-environment mismatches. We emphasize that naively enlarging the DRO uncertainty set does not guarantee improved performance and may lead to overly pessimistic models or loss of task-specific performance 6 3
MEDRO's design incorporates two key safeguards against these failure modes. First, the worst-case optimization over the expanded uncertainty set remains adaptive and bounded: the m² cross-environment risks are grounded in the training data and reflect plausible distribution shifts rather than arbitrary worst-case scenarios. When environments exhibit substantial variation, the expanded risk polytope (Figure [Tc captures this heterogeneity, enabling targeted robustness. Conversely, when environments are more homogeneous, the effective uncertainty set contracts naturally, preventing excessive pessimism. Second, the specialization term Σ, (f) serves as a critical regularizer that anchors each expert to its native environment, explicitly preserving domain-speciffe knowledge and preventing the model collapse that could result from overly aggressive robustness optimization. This principled balance between adaptive worst-case protection and specialization anchoring distinguishes MEDRO from naive uncertainty set expansion.
In essence, MEDRO retains the min-max flavor of GroupDRO across m² risk terms, emphasizing both native and mismatched expert-environmen pairs. Under conditions where A R 0 for i & j. the objective of MEDRO simplifies GroupDRO by focusing on the diagonal terms { Conversely,
if cross- environmii risk is significant (Aa.j > 0), the expanded uncertainty set can yield enhanced robustness against distribution shifts that singleenvironment weighting might fail to capture. Full
proofs are provided in Appendix B showing how this broader formulation encompasses GroupDRO while capturing cross-environment risks.
To solve this objective in 6 we adapt GroupDRO's online approach, At iteration f, we (1) estimate (0,) with mini-batches from environment j. (2) update Ac-1 the (m² 10--1)-hiie 1)-dimensional simplex via exponentiated gradient, and (3) update 0.+1 to minimize the weighted sum of crosseenviromment risks plus the specialization term. Pseudocode is provided in Algorithm of Appendix A.3 This procedure also converges at a rate of O(IVVT), albeit with a factor of log(m²) from the higher- dimensional simplex. We provide the full convergence analysis in Appendix
Connection to domain generalization Although MEDRO is primarily motivated by subpopulation shifts, our theoretical analysis shows that, under the assumption of approximately equal cross- environment risks, the shared feature extractor 6² approximately satisfies label-conditional invariance across environments (i.e., PY (X)) = P3 | xx))) for all i,j). This follows from MEDRO's ss expanded uncertainty set over all m²
environment pairs, which promotes the removal of environment-specific variations in the learned representation (Appendix D) We empirically
evaluate this effect on standard domain-genen somain-generalization benchmarks in Section 4.4
3.6 Inference with unknown environments
As environment membership is not available at test time in our experimental settings, we cannot simply route each sample to its matching expert. Two general strategies are considered for combining the outputs of MEDRO's expert heads: (1) simple ensemble, which averages predictions; and (2) learned gating network, which produces sample-adaptive weights for the experts. Both approaches leverage MEDRO's 's multiple heads without requiring explicit environment labels at inference.
5

---

Approach 1 (simple ensemble) Let each expert i produce logit vector (wi xx) E R for m. Then, the ensemble logit is:
2
wi o (x).
m
Finally, the predicted label is j(x) no additional training.
arg max; &k. This approach is straightforward and requires
Approach 2 (gating network) Alternatively, gating function 2 + R can be employed.
This approach is a form Mixture of Experts (MoE) model [34 35 36], where the gating function learns to assign weights different experts based on the input. Specifically, the gating function takes the shared features ((x) as input and produces logits (g o (x). These logits are then passed through a softmax function obtain environment--pecifie gating weights a(x) = softmax ((g o ())x)
Am, where Am = {a E Ro Σi= ag = 1}. Each expert i still produces its logit vector (w) (x) E _ The final combined logit vector is then computed as weighted sum:
2 as (x) o (x).
from which the final predicted label ((x) arg max; & is derived. The choice between these inference strategies can depend on factors such the availability of a suitable validation set for training an effective gating mechanism; in its absence, the simple ensemble provides robust default (see Appendix |A.2 for details). our experiments, we default the ensemble approach unless specified otherwise.
4 Experiments
4.1 Experimental design and datasets
Our experimental evaluation uses datasets that span diverse data modalities and distribution shift scenarios to thoroughly evaluate the effectiveness of MEDRO. These datasets are summarized in Table 5 of Appendix
We selected these datasets for the following key reasons:
1. Comprehensive coverage of shift types: Our selection includes both subpopulation shift datasets (Waterbirds (37], CelebA [38]. CivilComments [39], MultiNLI [40], MetaShift [41], NICO+ [42]. CheXpert [43, Living17 (4) and domain generalization datasets (Came-
Iyon17 17 455. iWildCam (46)), well hybrid settings (ao [47]) that exhibit both
types of shifts simultaneously.
2. Controlled validation: CelebA and Waterbirds serve as our foundational evaluation envi- ronments, following the experimental setup of GroupDRO 6 enabling direct comparison with this established baseline.
3. Standardized benchmarks:
By leveraging datasets from established benchmarks such as SubpopBench [2] and WILDS [5), we ensure that our evaluation follows rigorous protocols that facilitate fair comparison with state-of-the-ar methods.
4. Varied technical challenges: These datasets present different technical challenges, from handling high-dimensional image data to processing natural language, and from simple binary classification to multi-class classification with hundreds of categories.
This diversity allows us to evaluate whether MEDRO can provide consistent improvements across different types of distribution shifts, data modalities, and application areas, demonstrating its potential as general- general-purposs purpose solution for robust learning.
4.2 Method validation on controlled settings
We followed the experimental protocol established in 6 using strong E regularization to prevent and optimizer configurations. Additional training and hyperparameter details are in Appendix E.]
overfitting to majority groups. All methods used identical ResNet-50 architectures

---

MC360
13"
10"
13"
Figure 2: Effect of balancing factor y on CelebA and Waterbirds (log scale). Left: average accu- racy; Right: worst-group accuracy.
Figure 3: Worst-case training risk of MEDRO (blue) and GroupDRO (gray), over training itera- tions. Left: CelebA; Right: Waterbirds.
Table 1: Average and gorst accuracy (%) on CelebA and Waterbirds. Mean and standard deviation are calculated across five independent runs. The best worst-group accuracy is boldfaced.
CelebA
Waterbirds
Method
Avg.
Worst
Method
Avg.
Worst
ERM
95.9 (+001)
39.2 (+2.1)
ERM
95.5 (=1.5)
23.5 (+3.8)
GroupDRO
93.7 (+0.3)
85.7 (+2.2)
GroupDRO
96.1 (=1.3)
84.1 (+2.4)
MEDRO (ours)
92.2 (+0.8)
89.4 (#1.2)
MEDRO (ours)
94.2 (=1.8)
86.3 (=1.9)
Balance factor sensitivity analysis We analyzed how the balancing coefficient y affects model performance in our controlled settings, varying 7 from 10-3 10", As shown in Figure average accuracy inereased with larger values and saturated near 1, while worst-group accuracy peaked y = 0.5 and declined for larger values. These results suggest that intermediate y values offer the best trade-off between average and worst-case performance for these experimental conditions.
Comparing worst-case training risk Figure 3 Billustrates the worst-case training risk, defined as the empirical loss of the most challenging expert-enviromment pairing at each training iteration for MEDRO, compared to GroupDRO's worst-group risk. MEDRO's ' explicit consideration of all m² cross-enviromment mismatches (i.e., evaluating expert on environment j) means its objective may identify higher worst-case risk in early training stages, particularly if certain specialized experts initially perform very poorly on non-native environments. This initial emphasis on the most severe of these m² potential failure modes directly shapes the learning process. It compels the model (both the shared feature extractor & and the expert heads w/) to prioritize the mitigation of these specific, high-risk vulnerabilities from the outset. As training progresses, MEDRO actively minimizes these challenging mismatch risks. This process cultivates expert heads that are not only proficient in their native domains (due the specialization term in MEDRO's objective) but also more resilient when applied to mismatched data, ultimately leading to improved robustness on the most challenging subpopulations where such vulnerabilities are critical.
Results controlled settings Under these controlled conditions (Table T.. MEDRO consistently improved worst-group accuracy over GroupDRO (+3.7% on CelebA, +2.2% on Waterbirds) while maintaining comparable average accuracy (e.g., 92.2% vs. 93.7% for GroupDRO on CelebA). This demonstrates MEDRO's ability to substantially close the performance gap challenging groups with only a minor trade-off in overall average accuracy. These findings suggest MEDRO's 's explicit consideration of cross-environment risks effectively mitigates spurious correlations and enhances robustness for underrepresented groups
4.3 Large-scale evaluation on SubpopBench
Following the controlled evaluations, we now investigate the robustness of MEDRO under more diverse and challenging subpopulation shifts. To this end, we evaluate SubpopBench [12), comprehensive benchmark designed to systematically assess robustness under subpopulation shift. across various application areas.
7

---

Table 2: Worst-group accuracy (%) eight SubpopBench datasets, reported mean + standard deviation over three runs. Best results per column are bolded.
Method
Waterhinds
CelebA
CivilC.
MultiIII
MetaShift
NICO+
CheXpert
Living17
Overal
69.1 14.
62.6 215
63.7
111 668
=05
82.6
37.6
37.6
120
50.2
138
28.2
21.5
57.6
Mixup
78.2 10.
57.8
208
66.1
113 68.5
-06
81.0
108
42.7
114
37.4
135
29,8
118
57.7
GroupDRO
78.6 +1.0
89.0
#0.7
70.6
+12
76,0
-07
85.6
20.4
37.8
+18
74.5
+02
27.2
+15
67.4
IRM
74.5 115
63.0 =25
63.2
#08
63.6=1a
83.0
201
40.0
200
34,4
+17
28.2
56.2
CVARDRO
75.5 222
64.1 228
68.7
213
63.0-1
84.6
2000
36.7
227
57.9
104
28.3
207
59.9
+03
1002
69,1
=81
204
40.0
+49
28.8
+1.1
L.F
75.2 #0.
53.0 =43
51.0
161
63
73.1
11.6
30.4
11
13.7
198
+1.1
48.3
LISA
88.7 1006
86.5 112
73.7
103
73.3
10
84,1
104
42.7
122
75.6
106
29.8 1009
69.3
MMD
83.9 1:14
24.4 220
54.5
69.1
1.5
85.9
10,7
40.7
40.
50.2
138
26.6
1:88
54.4
ReSample
77.7 #11
87.4 +08
73.3
405
72.3
-08
85.6
#0.4
40.0
+0,0
75.3
+0.5
30.7
#:1
67.8
ReWeight
86.9 =0.7
89.7
72.5
-04
85.6
41,9
28.2
+1.
68.7
WeighW
78.6 1:011
82.4 20.5
71.7
104
69.5
0.7
84,6
20.7
40.0
10.0
70.0
223
+1.5
65.6
CBLoss
89,4
73.3
402
00
204
17.8
+0.
28.2 +1.5
Focal
71.6 1:08
59.1 =200
62.0
110
007
81.5
2000
36,7
227
42.1
140
28.0
+1.2
56.3
LDAM
71.0 1:1.3
59.6 224
37.4
181
69.6
1.6
83.6
104
42.0
10.
36.4
100
24.7
108
53.0
BSofmma
74.1 +:0.9
83,3
712
+04
66.9
-04
+0,7
40.4
+0.3
75.4
405
27.5 +088
65.2
DFR
91.0 +:03
90.4 10.1
69.6
402
68.5
-02
85.4
204
23.7
20.7
71.7
#02
29.0
#022
66.2
CRT
79.7
87.2
71.1
70.7
43.3
74.6
33,9
68.1
RoWeightCRT
78.4 1:0.1
87.2 203
71.0
10.1
69.0
02
85.6
20.
23.3
114
76.0
10.1
33.7
10.1
65.5
MEDRO
90.4 204
734
405
15.7
88
85.9
204
19.1
225
75.0
+04
32,6
207
69.5
MEDRO (w/ gating)
84.1 110
91.1
202
74.1
104
75.7
007
87.2
2007
39.1
#25
75.0
104
33.6
106
70.0
Experimental setup We evaluated MEDRO on eight representative SubpopBench tasks spanning diverse modalities. Following the official benchmark protocol, we used worst-group group accuracy as the primary metric. Full experimental details and baseline descriptions are provided in Appendix E.2
Results on SubpopBench Sableerepore the worst-group accuracy for all methods across the eight SubpopBench datasets. Consistent with previous findings [12], many existing robustness methods exhibit strong performance on specific datasets but not generalize their effectiveness universally across the varied subpopulation shift scenarios present in the benchmark. In this challenging evalu- ation, MEDRO (ours) achieved the highest overall worst-group accuracy of 69.5% when averaged across all cight tasks. This leading performance underscores MEDRO's s ability to effectively address a diverse range of subpopulation shifts. We attribute this strong generalization to its expanded risk structure, which explicitly considers m cross-environment risks, allowing it to better identify and mitigate vulnerabilitiis beyond conventional worst-case weighting over groups Furthermore, when augmented with lightweight gating mechanism for inference (MEDRO w/ gating, described in Section 3.6). the overall worst-group accuracy improved further to 70.0%, demonstrating the potential for adaptive expert combination test time.
Isolating the effect of multi-head architecture Since MEDRO employs multiple expert heads, natural concern is whether its improvements arise from the expanded uncertainty formulation or from ensemble effects. To isolate these factors, evaluated GroupDRO with multi-head architecture controlled baseline that maintains the same architectural structure but trains each head independently on the standard GRoupDRO objective, differing only in random initialization. Both methods use identical ensemble inference at test time. As detailed in Appendix [F.2.6] across eight SubpopBench datasets, GroupDRO with multi-head architecture achieves 67.8% 77. overall worst-group accuracy, a +0.4% improvement over single-head GroupDRO (67 (6). MEDRO achieves 69.5%, a +1.7% improvement over the multi-head variant. This pattern indicates that MEDRO's s gains stem primarily from modeling cross-cnvironment risks through its expanded m² uncertainty set rather than from architectural choices.
4.4 Evaluation under domain shifts
While MEDRO was initially designed primarily with subpopulation shifts in mind, we now investigate its effectiveness in broader domain generalization scenarios, where shifts between training and test distributions can be more substantial. To this end, we evaluated MEDRO on three representative tasks from the WILDS benchmark [5: Camelyon17 (histopathology image classification), iWildCam (wildlife camera trap image classification), and Poverty (satellite image regression for poverty prediction). These datasets are characterized by more fundamental and structural differences between
8

---

Table 3: Performance on WILDS benchmarks, reported as mean E standard deviation. (a) Came- lyon17: OOD validation and test accuracy (%) from 10 independent runs. (b) iWildCam: OOD validation and test macro Fl scores (%) from independent runs. The best result in each column bolded. A dash indicates missing results that were unavailable in the original benchmark study [18]
(a) Camelyon17
(b) iWildCam
Method
Validation
Test
Validation
Test
ERM (scratch)
84.9 (#3.1)
70.8 (#7.2)
ERM (ImageNet)
91.3 (=0.2)
59..7) (#2.1)
37.4 (#1.3)
31.0 (#1.3)
CORAL
86.2 (#1.4)
37.0 (#1.2)
32.8 (=0.1)
IRM
86.2 (+1.4)
64.2
20.2 (#7.6)
15.1 (#4.9)
GroupDRO
85.5 (+2.4)
(47.3)
26.3 (#0.2)
23,9 (+2.1)
DANN
31.9 (+1.
VREx
82.3 (1:.3)
71.5 (#8.3)
LISA
(+1.3)
77.1 (+6.5)
Fish
(+1.2)
79.5 (+6.0)
(+005)
24.2 (+0.9)
SWAD
(+1.5)
83.9
(+02)
29.1 (+0.1)
L2A-OT
(43.44
77.5 (+6.7)
22.8 (+2.9)
18.1 (#3.2)
HeckmanDG
90.6 (#2.4)
87.3 (#2.4)
34.5 (+0.9)
31.8 (+0.3)
MEDRO (ours)
92.6 (#005)
87.8 (#1.9)
34.1 (+007)
31.5 (=.0.)
training and test environments compared to the group-level variations typically seen in subpopulation shift problems. Theoretical motivations for domain generalization are provided Appendix D
Experimental setup Following the official WILDS protocol, used the designated out-of- distribution (OOD) validation sets for model selection and reported performance on the OOD test sets using dataset-specific metrics (accuracy for Camelyon17, 7, macro F1 for iWildCam, and Pearson correlation for PovertyMap). All baseline results were taken from prior work that adheres to the WILDS codebase and evaluation guidelines [1]] Additional experimental details provided in Appendix E.3
Results on WILDS domain generalization tasks is presented in Table Table On Camelyon17 (Table MEDRO achieves the best OOD validation (92.6%) and test (87.8% performance, surpassing all listed baselines, including GroupDRO and other strong domain generalization methods. For iWildCam (Table 3p), MEDRO (31.5% test FI) again substantially outperforms GroupDRO and remains highly competitive. In the regression task PovertyMap (Table 4 using a linear regressor head, MEDRO demonstrates strong results. Its average Pearson correlation (0.80 test) is competitive with uop-prforrmig methods and notably surpasses GroupDRO. For worst-group correlation, MEDRO (0.49 test) shows significant improvement over GroupDRO and is second only to HeckmanDG. Across these diverse WILDS benchmarks, MEDRO consistently delivers substantial improvements over GroupDRO. These results indicate its expanded uncertainty set, which considers specific expert-environment mismatches, is beneficial for tackling complex domain shifts and enhances out-of-distribution performance, complementing existing domain generalization strategies.
The performance of MEDRO and baseline methods on these challenging
5 Conclusion
In this work, we investigated the limitations of single-classifier GroupDRO handling heterogeneous environments and proposed MEDRO, which significantly broadens the uncertainty set to include cross-enviromment mismatches. By assigning individual heads to each environment but sharing a common feature extractor, MEDRO retains the core adversarial weighting principle of GroupDRO while offering greater flexibility to specialize across range of subpopulations or domains. Our empirical results on both subpopulation-shift and domain-shift benchmarks confirm that this approach substantially improves test-time robustness, especially in settings with strong spurious correlations or large domain discrepancies.

---

Table Performance comparison on the PovertyMap dataset from WILDS benchmark, reported as the Pearson correlation coefficient (mean + std over five runs) on both the OOD validation and test sets. Experiments use the original 5-fold dataset splits provided in WILDS. The best result in each column is bolded. We do not report worst-group performance for 'Fish', because it has not been reported in [23]
Average Corr.
Worst-Group Corr.
Method
Validation
Test
Validation
Test
ERM
0.80 (#0.04)
0.78 (+0.033
0.51 (#0.006
0.45 (#0.06)
CORAL
0.80 (#0.04)
0.77 (=0.05)
0.52 (#0.066
0.44 (#0.06)
IRM
0.81 (+0.03)
0.77 (#00005)
0.53 (#:0066
0.43 (=0.07)
GroupDRO
0.78 (=0.05)
0.75 (#0.077
0.46 (+0.07)
0.39 (#0.06)
DANN
0.77 (#0.04)
0.69 (#0.04)
0.44 (+0.11)
0.33 (=0.10)
Fish
0.81 (=0.01)
0.81 (#0.01))
SWAD
0.78 (+:0.03)
0.77 (+00.04)
0.48 (#00.09)
0.45 (+0.111)
HeckmanDG
0.81 (+00.3)
0.81 (+:0.03)
0.53 (+00006
0.51 (+0.04)
MEDRO (ours)
0.80 (+0.04)
0.80 (+-0.03)
0.50 (+00006
0.49 (+0.03)
In addition, we showed that MEDRO naturally accommodates situations where environment labels are unavailable at inference by employing either simple ensemble of experts or a learned gating mechanism. These test-time strategies enable practical deployment without the need to know the environment membership of new samples. Moving forward. we see multiple directions for advancing this methodology: examining sunersupereieed or dynamically evolving environments, and exploring theoretical questions regarding the minimal conditions under which expert heads significantly enhance robustness. Overall, our findings affirm that aavironment-specing parameterization compelling way to address complex, real-worke world distribution shifts, offering a principled balance between worst- case robustness and overall accuracy.
Acknowledgements
This research was supported by Brain Korea 21 FOUR, the Ministry of Science and ICT (MSIT) in Korea under the ITRC support program supervised by the Institute for Information Communication Korea University grant, and the National Research Foundation of Korea grant funded by the Korea government (RS 000-2022-00144490).
Technology Planning and Evaluation (IITP-2020--01172))
References
[1] Joaquin Salioonero-Canded Masashi Sugiyama, Anton Schwaighofer, and Neil Lawrence. Dataset shift in machine learning. Mit Press, 2022.
[2] Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR 2011, pages 1521-1528. IEEE, 201
[3] Luke 27(1):106-11 2020.
Exploring large-scale public medical image datasets. Academic radiology.
[4] Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. Allennip: deep semantic natural language processing platform. arXiv preprint arXiv: 07640, 2018.
[5] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: benchmark of in-the-wild distribution shifts. In International conference on machine learning. pages 5637-5664 PMLR, 2021.
[6] Shiori Sagawa* Pang Wei Koh*, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2020. URL.
10

---

[7] Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. Fairness without demographics in repeated loss minimization. In International Conference on Machine Learning, pages 1929-1938. PMLR, 2018.
[8] Ishaan Gulrajani and David Lopez-Par Paz. In search of lost domain generalization. In International Conference on Learning Representations, 2021. URL httpp:://ppeeeeeeee..eeeeeeeeeee www.wwrwee//rmm
id=IQdXeXDoWtI
[9] David Krueger, Ethan Caballero, Joern- Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk ex- trapolation (rex). In International conference on machine learning, pages 5815-5826. PMLR, 2021.
[10] Aharon Ben-Tal, Dick Den Hertog, Anja De Waegenaere, Bertrand Melenberg, and Gijs Rennen. Robust solutions of optimization problems affected by uncertain probabilities. Management Science, 59(2):341-357, 2013.
[11] Huaxiu Yao, Xinyu Yang, Xinyi Pan, Shengchao Liu, Pang Wei Koh, and Chelsea Finn. Improv- ing domain generalization with domain relations. In The Twelfth International Conference Learning Representations, 2024.
net pest/forum?? cacrummid-DDe ACH
[12] Yuzhe Yang, Haoran Zhang, Dina Katabi, and Marzyeh Ghassemi. Change is hard: A closer look subpopulation shift. In International Conference on Machine Learning, pages 39584-39622. PMLR, 2023.
[13] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk mini- mization. arXiv preprint arXiv: 1907.02893, 2019.
[14] Elan Rosenfeld, Pradeep Kumar Ravikumar, and Andrej Risteski. The risks of invariant risk minimization. In International Conference on Learning Representations, 2021. URL
https://openrevier. net htt://ormm?? 7id- Pttp://rnmB-BBBIBIII---------
[15] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Domain-adversarti training of neural networks. Journal of machine learning research, 17(59):1-35, 2016.
[16] Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In Computer vision-ECCY 2016 workshops: Amsterdam, the Netherlands, October 8-10 and 15-16, 2016, proceedings, part III 14, pages 443-450 Springer, 2016.
[17] Jonas Peters, Peter Bühlmaann and Nicolai Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society Series B: Statistical Methodology, 78(5):947-0112, 2016.
[18] Hyungu Kahng, Hyungrok Do, and Judy Zhong, Domain generalization via heckman-ty selection models. In The Eleventh International Conference on Learning Representations, 2023. URL httphttts///oprreiiiee
[19] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Learning generalize: Meta- learning for domain generalization. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018
[20] Alexander Robey, George J Pappas, and Hamed Hassani. Model-based domain generalization. Advances in Neural Information Processing Systems, 34:20210-20222 2021.
[21] Huaxiu Yao, Yu Wang, Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn. Im- proving out-of-distribution robustness via selective augmentation. In International Conference on Machine Learning, pages 25407-25437. PMLR, 2022.
[22] Kaiyang Zhou, Yongxin Yang. Timothy Hospedales, and Tao Xiang. Learning to generate novel domains for domain generalization. In Computer vision-ECCV 2020: 16th European conference, glasgow, UK, August 23-28, 2020, proceedings, part XVI 16, pages 561-578. Springer, 2020.

---

[23] Yuge Shi, Jeffrey Seely, Philip Torr, Siddharth N, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. In International Conference on Learning Representations, 2022. URL htttps://openrreieex http://wwwnnettrrtrrr//////////a
[24] Aman Sinha, Hongseok Namkoong, and John Duchi. Certifiable distributional robustness with principled adversarial training. In International Conference on Learning Representations, 2018. URL htttps://opprreeiecee http:///pppppeeeeeeeeeee/rrrrr/a .att//torumm.dd-llkkP222....
[25] Daniel Levy. Yair Carmon, John Duchi, and Aaron Sidford. Large-scale methods for distributionally robust optimization. Advances in Neural Information Processing Systems, 33: 8847-8860, 8860, 2020.
[26] Aharon Ben-Tal, Arkadi Nemirovski, and Laurent El Ghaoui Robust optimization. 2009.
[27] John C Duchi, Peter W Glynn, and Hongseok Namkoong. Statistics of robust optimization: generalized empirical likelihood approach. Mathematics of Operations Research, 46(3): 946-969, 2021.
[28] Matthew Staib and Stefanie Jegelka. Distributionally robust optimization and generalization in kernel methods. Advances Neural Information Processing Systems, 32, 2019.
[29] Weihua Hu, Gang Niu, Issei Sato, and Masashi Sugiyama. Does distributionally robust super- vised learning give robust classifiers? In International Conference on Machine Learning, pages 2029-2037. PMLR, 2018
[30] Yonatan Oren, Shiori Sagawa, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust language modeling. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-UCNLLP), pages 4227-4237, 2019.
[31] Vladimir Vapnik. Principles of risk minimization for learning theory. Advances in neural information processing systems, 4, 1991.
[32] Alexander Rakhlin, Karthik Sridharan, and Ambuj Tewari. Online learning: Stochastic, con- strained, and smoothed adversaries. Advances in neural information processing systems, 24, 2011.
[33] Peyman Mohajerin Esfahani and Daniel Kuhn. Data-driven distributionally robust optimization using the wasserstein metrie: Performance guarantees and tractable reformulations. Mathemati- cal Programming. ( 15-166, 2018.
[34] Robert A Jacobs, Michael Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
[35] Michael Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181-214, 1994.
[36] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V Le, Geoffrey E Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of. experts layer. In International Conference on Learning Representations (ICLR), 2017.
[37] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
[38] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730-3738, 2015.
[39] Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. Companion proceedings of the 2019 world wide web conference, pages 491-400 2019.
12

---

[40] Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, 2018.
[41] Weixin Liang and James Zou. Metashift: A dataset of datasets for evaluating contextual distri- bution shifts and training conflicts. In International Conference on Learning Representations, 2022. URL
[42] Xingxuan Zhang, Yue He, Renzhe Xu, Han Yu, Zheyan Shen, and Peng Cui. Nico++: Towards better benchmarking for domain generalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16036-16047, 2023.
[43] Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 590-597, 2019.
[44] Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. (BREEDS): Benchmarks for subpopulation shift. In International Conference on Learning Representations, 2021. URL
[45] Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, al. From detection of individual metastases to classification of lymph node status at the patient level: the camelyon1 17 challenge. IEEE transactions on medical imaging, 38(2)5500-560, 2018.
[46] Sara Beery. Elijah Cole, and Arvi Gjoka. The iwildcam 2020 competition dataset. arXiv preprint arXivv2004.10344 2020.
[47] Christopher Yeh, Anthony Perez, Anne Driscoll, George Azzari, Zhongyi Tang, David Lobell, Stefano Ermon, and Marshall Burke. Using publicly available satellite imagery and deep learning to understand economic well-being in africa. Nature communications, 11(1):2583, 2020.
[48] Geoffrey E Hinton. How neural networks learn from experience. Scientific American, 267(3): 144 151, 1992.
[49] Nitish Srivastava and Russ R Salakhutdinov. Discriminative transfer learning with tree-basee rree-based priors. Advances in neural information processing xystems, 26, 2013.
[50] Arkadi Nemirovski, Anatoli Juditsky, Guanghui Lan, and Alexander Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on optimization, 19(4): 1574-1609, 2009.
[51] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[52] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pages 4171 4171-4186, 2019.
[53] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL httpps:/ wtwp:/ewwweieeeeeteeeeeeee net/forum?
id Bkg6RRiqq
[54] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
[55] Evan Liu, Behzad Haghgoo, Annie Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pages 6781-6792. PMLR, 2021.
13

---

[56] Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re- re-training is suffi- cient for robustness to spurious correlations. In The Eleventh International Conference on Learn- ing Representations, 2023. URI. httpps://openrevie htt.://.oooo////--22b----------
[57] Bingyi Kang. Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In International Conference Learning Representations, 2020. URL / soaerummmd=rrggeeg defffoommmdd-rggtttie
[58] Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: De-biasing classifier from biased classifier. Advances in Neural Information Processing Systems, 33:20673-20684, 2020.
[59] John C Duchi and Hongseok Namkoong. Learning models with uniform performance via distributionally robust optimization. The Annals of Statistics, 49(3):1378- 2021.
[60] Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex Kot. Domain generalization with adversarial feature learning. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 5400-5409, 2018.
[61] Yin Cui, Menglin Jia, Tsung Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based effective number of samples. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 2268-9277, 2019.
[62] Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
[63] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with abhel-ditributionnaaaaa margin loss. Advances neural information processing systems, 32, 2019.
[64] Nathalie Japkowicz. The class imbalance problem: Significance and strategies. In Proc. of the Int'l Conf. on artificial intelligence, volume 56, pages 111-117, 2000.
[65] Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al. Balanced meta-softmax for long- tailed visual recognition. Advances in neural information processing systems, 33:4175-4186. 2020.
[66] Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. Advances Neural Information Processing Systems, 34:22405 -4212805-222488, 2021.
14

---

A Supplementary Details for Multi-Expert DRO (MEDRO)
A.1 Rationale for the expanded uncertainty set of MEDRO
Distributional robustness hinges on what set of risks the learner treats as plausible test-time failures.
Jm GroupDRO considers the m risks [ of a single classifier across the m training environments and safeguards against their worst-case mixture. This is effective when a common decision rule can serve all environments, but becomes brittle whenever the Bayes-optimal strategies diverge.
MEDRO remedies this by (i) allocating an expert W, o Φ to each environment P. and (ii) enlarging the uncertainty set to the m² cross-environment risks:
Ri ()) == (x.x))> ((((u ()),) 1 1 m.
These risks quantify specialization (the diagonal Ri,)) and mismatch (the off-diagonal R.,j) in single view. The max term in Eq. 66 therefore searches the convex hull of all m² points-visualle the shaded polytope in Figure Ic innteedd of the simple in Figure Ta Because the former strictly contains the latter, MEDRO does not underestimate worst-case error and reduces to GroupDRO when Xs. for + j.
toy illustration helps ground this expansion. In the Waterbirds dataset, an expert trained on land backgrounds (i.e., environment éland) may rely on background cues that are spurious for water birds. If at test time, a water-background image were mis-routed to the land expert, the loss could spike despite GroupDRO's guarantee. MEDRO penalizes exactly this scenario through Randd waters compelling the shared encoder & either to discard the cue or to make the experts mutually resilient.
Finally, the additional specialization term E: (0) keeps each expert near its environment's s Bayes risk, preventing the trivial solution where all heads collapse into an identical, over-regularized classifier. The balance hyperparameter mediates this trade-off and is studied empirically in Section 4.2 (Figure 2}.
In summary, MEDRO's uncertainty set is deliberately broader, capturing both per-environment optimality and cross-expert vulnerabilities, while retaining the computational structure of GroupDRO.
A.2 Gating network for inference: training details
The gating network g can, in principle, be trained end-to-end with the main MEDRO objective, a common strategy in MoE literature [35 36), trained separately known environment labels. In our experiments, we adopted the latter approach, training the gating network dedicated validation set. This staged training approach, while potentially suboptimal compared to joint optimization, is often employed for practical reasons such as simplifying the training process when expert models are pre-trained 48 49 For the gate learn effectively, this validation set should ideally be representative of the various environments. For instance, in subpopulation shift problems, having roughly balanced representation of subgroups within this validation data may be beneficial, although this constitutes a practical consideration regarding data availability. While sufficiently large and balanced validation is ideal, relatively small number of labeled examples per environment might still suffice to train useful gate. We opted not to explore end-to-end training of the gating network concurrently with the MEDRO objective due the anticipated complexities in the optimization dynamics arising from such joint training.
held-out validation set with
15

---

A.3 MEDRO Training Algorithm
Algorithm 1 Multi-Expert Distributionally Robust Optimization (MEDRO) Training Procedure
Require: Training data from environments (De Peje Pejm
2:
Require: Balance factor >
Require: Learning rate for model parameters T7
4: Require: Learning rate for risk weights 7A
5: Require: Number of training iterations (batch steps) T
6: Initialize: Model parameters Op {0o, w01,::...;mmm..
7: Initialize: Risk weights Ao = {(Ao)..) (egg, uniform: (Ao)us = 1/m² for all i,
8: for .,... do
9: # Step 1: Estimate all cross-enviromment risks
for ..... do
11:
for j = 1,...,77 m do Sample mini-batch B; ~ P, from environment j.
13:
Estimate R., [ (xx)een ((wii, o (x)(x).
14:
end for
15:
end for
16:
# Step 2: Update risk weights A
17:
Let R₂ [Ri, (0.) (0.) [ej-
18:
Update A4+1 from A₁ to emphasize higher risks in RR. # e.g. using exponentiated gradient
19:
for I..... do
20:
for j = 1,...,,m do
21:
(1.+) +
(o)))
22:
end for
23:
end for
24:
Normalize A (At+1) + (144))ss/ Enn,(Netzaaa
25:
# Step 3: Update model parameters O
26:
Define the loss for parameter update using A++1:
27:
LMEDRO A++1) = Em
k=1 (Be) + Ex= Dj= (o.)
28:
Compute gradient St- = V TNMDROO (z,, A4t1).
Update parameters: 0.+1 +- 6g Ne9t- # This involves updating det1 and all Wh.41
30:
end for
31:
Return: Trained parameters 0r.
B Containment analysis of GroupDRO in MEDRO
In this appendix, we provide an argument that our proposed MEDRO formulation either exactly or approximately contains the standard GroupDRO objective. We show how GroupDRO's worst-case weighting over m environments can be embedded our (m² 1)-dimensional simplex, ensuring that MEDRO subsumes GroupDRO as special (or near-special) case, even when the diagonal risk RR,5 in MEDRO does not perfectly match the single-classifier risk R; from GroupDRO.
B.1 Revisiting GroupDRO and MEDRO
GroupDRO Let there be environments The GroupDRO objective
Jm each associated with a distribution Pe.
min max Bg X
X. R.
where A, is the (m 1) dimeensiool simplex, and (fG))) is the expected loss under environment with parameters fg-
MEDRO In our approach, we no longer rely on single set parameters 0g. Instead, we introduce shared parameter d, and environment-specific heads fWi..... Wm, one per environment. We
16

---

combine them as = fØr ...) This entire set of parameters defines a multi- head model. The MEDRO objective is
min
(0) 7 AEAm
Aay Ri,
(8)
i=1
where Am²
{A
E
Rmm 20
m Σ j== As. =1 =1 is the probability simplex of dimension (m² - 1).
Parameter space relationship An important connection between GroupDRO and MEDRO cxists when we constrain all classifier heads in MEDRO to be identical (wi = w2 this restricted case, MEDRO model with parameters 0 f0r w.... w) becomes functionally equivalent to a GroupDRO model with parameters 6g = {o,w). This equivalence allows us to compare (() with (&g)) in following analysis.
Wm = w). In
B.2 Embedding GroupDRO's weights into (m² - 1) dimensions
A key step in showing that MEDRO replicate GroupDRO's worst-case solution is embed any weighting À {Ai...., A Am into Am by placing all mass on the diagonal entries:
A if Aa. = (o
otherwise.
Since Eis Ai. - X indeed A lies in Am? Moreover, for any function Fij.
ApP3..
This diagonal embedding is crucial it enables MEDRO's maxa optimization to directly capture GroupDRO's max objective when focusing on diagonal elements.
B.3
Exact containment if (0) R2(BG)
If each diagonal risk in MEDRO equals the single-classifier risk from GroupDRO, we obtain exact
containment.
Exact case Consider the setting where MEDRO parameter 0 = (.,.,...,,) have identical classifier heads, corresponding to GroupDRO parameters 0G = (0, w] Suppose that R,,( = (RG0)) for each i in this setting. Then,
T
m
max AEA,,.
AL. (0)
Z
max Σ xEAm 1=1
(
max DEAm
Σ (,,,) j=1
(10)
where the last equality follows from the assumption that (0) (o,).). Minimizing over shows that MEDRO's min-max objective most the GroupDRO optimum, hence MEDRO includes GroupDRO as subproblem.
B.4 Approximate containment if (0) R2(O0) s
While the exact containment provides a clean theoretical guarantee when (0) R1())), in practice there may be small discrepancy even when using identical classifier heads. We analyze how MEDRO approximates GroupDRO when these risks are not exactly equal.
Approximate case Suppose that (P) (ec)) <; for i = 1,..., when using equivalent parameter configurations. Let X" be the optimal worst-case weighting for GroupDRO. We embed X
17

---

into A via diagonal entries as in 9 Then
MM (i
Σ (0)
4.
j=1
m
2 Σ (2(R((06) - Eg]
1=1
m
(06) Σ XJE
j=1
(06) Σ Xj
max &N
j=1
(0G) max
because Σラ= 23 1. Hence MEDRO approximates GroupDRO within O(max, Ey), providing
precise bound on the approximation error.
B.5 Empirical illustration: GrouDDRO-MEDRO continuum
The theoretical analysis above shows that MEDRO contains GroupDRO as special case when expert heads are constrained to be identical. To illustrate this relationship empirically, we conducted preliminary experiment on Waterbirds.
We added a regularization term MEDRO that penalizes expert dissimilarity: C m En ||اا ا|
where Φ is the average parameter vector across all m expert heads. As increases, this penalty drives experts toward similar parameters. Small values allow expert specialization (MEDRO-like) while large C. values enforce similarity (GroupDRO-like).
87.0
Worst-grous accuracy (%)
MEDRO (86.3%)
86.5
86.0
85.5
85.0
84.5
84.0
GroupDRO (84.1%)
83.5
83.0
10
10
10
10
10a
10a
10a
Regularization strength C
Figure 4: Worst-group test accuracy on Waterbirds as function of constraint strength C. Error bars represent standard deviation across five runs.
As shown in Figure performance exhibits plwer-law decay from 86.3% (C 0.001) to 83.9% (C 1000). At high constraint levels, performance converges to GroupDRO baseline (84. I % from Table I.. consistent with the theoretical prediction. The 2.4 percentage point gap between the two extremes suggests that expert specialization provides measurable benefits on this dataset, though evaluation across multiple datasets would be needed to characterize this effect more broadly.
18

---

C Convergence analysis of MEDRO
C.1 Problem formulation and assumptions
As established Section MEDRO operates in an expanded uncertainty set compared to GroupDRO. Considering m² cross-environment risks rather than environment risks. Recall that MEDRO involves optimizing over shared parameter & and environment-specife experts [wi,.... Wm ). with the objective including both specialization term and worst-case cross-environment robustness term.
Saddle-point reformulation Let == {0,, .1............. Wm) collectively denote all parameters of the model. By introducing randomness & (e.g., stochastic mini-batches) in the data sampling, we can write the expected objective in min-max (saddle-pon poiddt) form:
min max
IEg [,, A, E)
(11)
0 AEA,,,
where T(0,,,g)) = R2,(0,5) >
i=
m
m 1==1
As.j Ri.
Assumptions
1. Convexity / Concavity: (0, {) is convex in 0 and A + - Evs (i, is concave over Am².
2. Compactness: The parameter f compact, |0]] S Bg, and A is simplex, whose geometry yields a log(m²) term in mirror descent.
3. Lipschitz gradients: R (0,E)
()ll S By implies || S Bv, and ||4 S BB on the
dual side, where 4 = R (O,, (,))) Isijsm See Eq. 12 for formal definition of gr.
Under these assumptions, we show an (1//VT) convergence rate under an online mirror descent procedure.
C.2 Online mirror descent algorithm
Gradient definitions
At iteration t, we observe a stochastic mini-batch & We define:
St Ve
Σ (0,
(12)
where [ denotes the (ith element of the weight matrix A, at iteration L. The primal gradient is w.r.t. 0, and the dual update uses lg (a matrix of scaled risks).
Primal update (mirror descent) We update 0 by mirror descent step:
0.+1 = arg min (gt, 0) + (0,))))
(13)
0
where 2 0 is the primal learning rate, and D is a Bregman divergence measuring the distance between 9 and 0g.
Dual update (mirror ascent) Simultaneously, we update the dual variable A e A,, by a mirror- ascent step:
A++1 = arg max (nA (A,AA))) ΛE
(14)
where 7/Λ 2 0 is the dual learning rate and D, is a Bregman divergence on the A simplex. This extends GroupDRO's dual update to our expanded uncertainty set.
The dual updates operate over an (m² - 1)-dimensional simplex rather than the (m - -iddimension , T yields sequences (0,, Ar). simplex in GroupDRO. Repeating this procedure for 1,,,....
C.3 Regret analysis
Our convergence analysis follows the approach (6) which builds upon the stochastic approximation framework of [50]. Let 0" and A be saddle point of the min-max objective.
19

---

Regret definitions
Define the cumulative regrets for primal and dual variables:
ket. O:
07), R Σ -4,A--Ai
(15)
where (4,,4)
Ei=1 [Ak, represents the element-wise product sum between matrices.
Primal regret bound and || S Bv. then
From mirror descent theory for saddle-point optimization 50 if |0] S Bg
E
ge. 02
0*)
(16)
B
1
T
18.
(17)
1
2
t=1
For simplicity, assume E||lg.17] $ B Then
E E
S
B
17
T
(18)
778
2
Dual regret bound For mirror ascent on the simplex A Am? 114elloo I BB, the standard regret bound gives
E[A = E
Σ (1e, A* A1)
(19)
log(m²)
(20)
7A
2
The log(m²) term arises from the maximum divergence between any two points in the (m² - 1)- dimensional simplex.
Total regret bound
Summing the two regrets:
[[
E[ +
RAJ
(21)
B3
log(m²)
ToT
(22)
1/0
7/A
2
Optimal learning rate
The optimal learning rates that minimize this bound are:
2B3
2log(m²)
1/
" V
(23)
TB
T-2B3
Substituting these optimal learning rates:
B|Rr] < /2TB3RR + / /21 =0
(24)
Dividing by T shows the average regret is O(I/VT).
C.4 Convergence rate
Given the bound on the average regret:
1
(+)
E3
R
+
RA
O
(25)
20

---

we can define the average iterates 0r = T Eí= 6n and = A,. By Jensen's 's inequality applied the convex concave objective, the expected suboptimality at these average iterates is bounded by the average regret:
E| T
B[I(Or, T(0", Ar)]
O
(26)
VT
This establishes that MEDRO achieves the same convergence rate of (1/vVT) standard Group- DRO, despite expanded parameter and uncertainty spaces. The log(m)) term the dual regret bound introduces constant factor of 2 compared to the log(m) term in GroupDRO.
D Theoretical connection between MEDRO objective, expert optimality, and feature invariance
This subsection lays the theoretical groundwork for understanding how the Multi-Expert Distribu- tionally Robust Optimization (MEDRO) objective facilitates domain generalization. We begin by defining key terms and then explore the relationship between the objective function, the optimality of individual environment experts, and the emergence of Nomain-invariane feature representations under specific conditions.
D.1 Preliminaries and definitions
We consider a domain generalization setting with m distinct source domains, each with a data distribution P. over the input-output space X' x y, for E {1, 21,2.... ........ m). Our model, parameterized by = {6, w Wm J. consists of shared feature extractor X' + Z and m environment- specific "expert" heads wi
y for m.).
The cross-enviromment risk for expert head evaluated on data from environment j is defined as:
Ri (0) B(x,))~,, [[((w; o (x), y)]
(27)
where ((xy)) is given loss function. The MEDRO objective function be minimized is:
L(0) = Σ Rr (f) + +7 max Ri iJE(L....mm)
(28)
where > 0 is a hyperparameter balancing native-environment specialization with worst-case cross-enviromment robustness.
Let Ri denote the Bayes risk for environment
R min Bxx [e(f(x). y)] f-x-y
(29)
This minimum is achieved by the Bayes optimal predictor fü. for environment k. Let 0* {o* wi..... with } be set parameters that globally minimizes L(0). We define the deviation from native Bayes risk for expert under 6" (²*) = Re, Ri. Note that 5e(0") 2 0.
We consider
denoted Bayes (dBayes W1, Bayess Wm, Boyes ). where each expert achieves its native Bayes risk. That is, (0,xeOnayes) = Rt for all E {l..... m). The existence of such a Bayes within the model's hypothesis space is assumed under conditions of sufficient model capacity. Let ((0) = max,,; (0). We can then define MBayes M(0ea))s) and M ((*).
hypothetical parameter configuration,
D.2 Bound on deviation from native Bayes optimality
The MEDRO objective, while promoting robustness, also includes specialization term Σm (0) that encourages each expert wj. & to perform well on its native environment Pi. The following proposition quantifies how far the native risks Re. (0*) might deviate from their theoretical minima R at the MEDRO optimum 0".
21

---

Proposition 1 (bound on deviation from native Bayes optimality) Let 0 be a global mininizer of the MEDRO objective L(0) as defined in Eq. (28). with 7 > Under the assumption that
configuration 0 Bayes exists such that Ri (Byes) = Rig for all k, the sum of deviations from native Bayes risks at f is bounded as:
Σ (k*( S (Myase M*)
(30)
Furthermore, since M 2 0 (assuming non-negative risks), simpler bound is:
Σ (x() S Y
(31)
Proof. By the optimality of 8", we have L((*)= I B(ABay)s). Substituting the definitions:
m
m
Ra,(00) + +"M 2
RR R Boyee) +
Using R4,x( = R7 + (0*) and Rr (0 (BBayes = Rt:
Σ (R + 5x (0")) + <
R + Mayes
The Σ ng terms cancel:
m Σ
6o((0) + S 5yMBayee
Rearranging yields the first bound in Eq. 30 Since M MByes M S. MByes (as M > 0), the second bound in Eq. (31) also holds.
Discussion Proposition 1 demonstrates that the total deviation from native Bayes optimality,
Σ Eu (0*), is controlled by and the extent of robustness improvement, M Bayes - M The deviation Σ (0*) will be small if (i) is chosen to be sufficiently small; (ii) the term (M MM) is small. This latter case occurs if M Bayez itself is small (i.e., native Bayes optimal predictors are inher- ently robust), or if MByes is not significantly larger than M (i.e., there's limited scope for MEDRO to improve worst-case robustness beyond what Bayes already offers, possibly "robustness-harr problem settings where further reduction in M incurs substantial costs native performance).
D.3 Assumption on expert optimality
The specialization term Σx= R (0) in the MEDRO objective directly encourages each expert w% é to minimize risk on its corresponding environment Pa. Proposition I provides a quantitative framework for understanding when the resulting native risks Rr (0*) will be close their theoretical minima RT For the subsequent analysis concerning the properties of the learned representation ø', we make the following assumption:
Assumption 1 (expert optimality) For each environment i E (I....,m). the expert head w combined with the shared feature extractor o² at the MEDRO optimum 0* achieves, closely approximates, the Bayes-optimal prediction on environment i:
w o arg
min B(x.))pP [((f(x), v)]
(32)
f-X-+y
This is equivalent to assuming that each deviation 6.(0*) is small or negligible.
This assumption is grounded in the structure of the MEDRO objective and the insights from Proposition It posits that under ideal conditions (e.g., sufficient model capacity, successful convergence to global optimum 8*) and an appropriate choice of y, the balance struck by the MEDRO objective is such that the conditions for bound in Proposition to small (e.g., small y, or problem structure where M Moges M is small for reasons not presupposing advanced feature invariance) are effectively met. This ensures that experts remain highly proficient in their native domains.
Justification
22

---

D.4 Domain-invariant conditionals under equalized risks
Building upon the assumption of expert optimality, we now explore the properties of the learned rep- resentation 6 under a particularly strong and idealized scenario of cross coos-ennvroomen generalization. We consider situation where not only does each expert W achieve near- Beye- optimality on its native domain P; (per Assumption 1), but also its performance level on any other domain Pj matches this optimal native performance. This implies that R,,; Ri (i, 2 R2 for all j.
While the MEDRO objective aims to minimize the single worst-case risk m R.,,5. (0).. it does not explicitly enforce that all Ri.j are reduced precisely to Rt; However, if the minimization of maxa.6 à Ra, b were so effective that all such risks were driven down to this fundamental limit, the following proposition elucidates a crucial property of σ²" Proposition 2 thus explores the structure of 6" in such an idealized state of uniform expert generalization.
Proposition 2 (domain-invariant conditionals) Under Assumption (Expert Optimality), if an MEDRO solution 0 = =6"", wi....,, W achieves equal risk across all source domain pairs, i.e. for all i m.).
Riz((") =R,)
(33)
then the conditional distribution of labels given the learned representation e' (x) is invariant across these domains:
P: 1 (x)) = PY (x))) Vi.j E (1,..,,m). for o² (x) in the support.
(34)
Proof. By Assumption 1, , o o" (denoted (6" (xx))x) for brevity) achieves (near) Bayes-optimal risk on environment i. Thus, (0*) = RØ, , the minimal possible risk on environment given the representation o² oxx)) The condition in Eq. (33), (f*) = Rt,x(f",, implies that h, (x)) also achieves this same (minimal) risk R3 when evaluated data from environment j. This means (x)) is effectively Bayes-optimal for predicting Y from "²xx under Pj as well, and achieves the same risk value RÇ
Letz (x). The Baycs-optimal predictor for environment & using representation z is fz( = arg my (Y)==)((1=))(((// Y)]. Our premise is that w (z) serves as fi (achieving risk Re)) and also as (z) (achieving the same risk R,). For the same predictor (-) to be Bayes- optimal for two different conditional distributions P. P. (Y = z) and PY = z) and yield the same Bayes risk value, these conditional distributions must be identical. If they were different, say P. (z) + (Y 1z) for some in the support, then f((z) would generally differ from (z). or they were the same function, the achieved Bayes risks would generally differ, contradicting the premise that w/ (z) achieves risk R, for both. Thus, Po(Y'o*(x (x) = =) = PY (oY)0 (x)(x) z) for all in the common support of the representations across these domains.
Remark (approximate domain invariance) for all i.j. (0r) (00) E, then under mild smoothness assumptions on the loss, the conditional distributions PY 0 (x)) and PY o² (z)) differ by at most O(Vs) in total variation.
Implication Proposition 2 suggests that if MEDRO can find solution where each expert is not only optimal in native domain but also performs identically well (at its optimal native-domain risk level) across all other source domains, then the learned feature extractor d² must capture representation that neutralizes domain-specific variations relevant to XXY|X), revealing a core, Nomain-invarianl conditional relationship P(Y jo* (X)). This is highly desirable property for out-of-distribution generalization.
E Experimental details and hyperparameter search
Our codes are available http://iittbbcco/yyyoonnkk/////
E.1
Controlled experiments on CelebA and Waterbirds
We followed the experimental protocol introduced in the GroupDRO paper 6 to evaluate MEDRO under controlled binary classification tasks. These experiments were conducted on the CelebA and
23

---

Table Summary benchmark datasets used in our experiments.
Dataset
Modality
Samples
Classes
Dommins/Atriiitees
Task
Shift Type
Waterbirds
Natural image
11,788
2
(background)
Bird classification
Subpop.
CelobA
Natural image
202,599
2
(gender)
Hair color prediction
Subpop.
CiviCComments
Text
448,0000
2
(demugraphhcss)
Toxicity detection
Subpop.
MuliNLI
Text
392,702
3
(negation)
Natural language inference
Suhpop.
MetaShift
Natural image
3,499
(background)
Car/Dog classification
Subpop
NICO++
Natural image
88,866
60
6 (background)
Multi-category classification
Subpop.
CheXpert
Biomedical image
222,792
frace x gender)
Disease diagnosis
Subpop.
Living17
Natural image
45,900
17
NIA (Attr. gencralization)
Multi-category classification
Subpop.
Map
Satellite image
19,569
Regression
23x2 (county/arraa)
Asset wealth prediction
Hybrid
Camelyon17
Biomedical image
455,954
2
(hospitals)
Tumor detection
Domain
iWildCam
Natural image
203,029
182
323 (locations)
Animal clussification
Domain
Waterbirds datasets, which exhibit clear subpopulation shifts due to strong correlations between target labels and spurious attributes. We compared MEDRO against ERM and GroupDRO under the same backbone architecture and optimization settings, using known group labels for both training and validation.
E.1.1 CelebA
We used the binary attribute Blond_Hair as the prediction target and Male as the confounding variable. This results in four subgroups based combinations of hair color and gender. The training set is highly imbalanced, with the smallest group (blond-hr) haired males) accounting for only a small fraction of samples.
Training subgroup sizes:
(0, 0): 71,629
(not blond, female)
(0, 1): 66,874
(not blond. male)
* (1, 0): 22,880
(blond, female)
. (1, 1): 1,387 (blond, male)
Models were trained for 50 epochs using a pretrained ResNet-50 backbone (51) We used SGD with momentum 0.9, batch size 128, learning rate 10-5, and strong & regularization (weight decay 0.1). No data augmentation was used.
We evaluated all methods using the final epoch model (i.c., without validation-basee model selection), GroupDRO typically converges near the end of training and our setup reused its original fixed typerparameters. 10% validation split was retained for consistency with GroupDRO, but it was not used for early stopping model selection.
both GroupDRO and MEDRO, we followed the same robust optimization setup by setting the step size for the adversarial group weighting update (also called DRO step size) to 0.01, as used in the original GroupDRO implementation.
The balance factor was selected as 0.5 based on sensitivity analysis conducted on both CelebA and Waterbirds under strong lg regularization. We reported results averaged over five runs with different random seeds.
E.1.2 Waterbirds
We followed the same setup for the Waterbirds dataset (also known as CUB), where the binary target label caterbirddcoompettee indicates whether the bird is a waterbird (1) or landbird (0), and the confounder forest2water2 denotes the background environment (water forest). This setup produces four subgroups with strong spurious correlations.
Training subgroup sizes:
(0, 0): 3,498
(landbird on land)
(0, 1): 184
(landbird on water)
(1, 0): 56
(waterbird on land)
24

---

(1, 1): 1,057 (waterbird on water)
Models were trained for 300 epochs using the same ResNet-50 backbone. used SGD (momentum 0.9), batch size 128, learning rate 10-5, and weight decay 1.0. No data augmentation used.
We again evaluated using the final epoch model, without validation- model selection. The 10% validation split was retained for comparability but unused in selection.
As in CelebA, same value = 0.5 was used for MEDRO, as determined from sensitivity analysis. No additional tuning was conducted for Waterbirds.
the DRO step size to 0.01, consistent with the GroupDRO implementation. The
E.2
SubpopBench settings
E.2.1 Benchmark overview
SubpopBench covers wide range of real-world datasets and subpopulation structures. We focused on eight representative tasks- Wattebrrd,,,, CelebA, CivilCCommenss, MultiNL.I, MetaShift, NICO++ CheXpert, and L chhceen to span different modalities and types of subpopulation shift. For all datasets, we followed the SubpopBench-prowe provided data splits, group definitions, and preprocessing procedures without modification.
E.2.2 Evaluation metric
Consistent with the official protocol, we adopted worst-group accuracy as the primary evaluation metric. the standard criterion used the benchmark and enables direct comparison across methods under shared notion of robustness. Although it does not capture all performance trade-offs (e.g., precision-recall balance), worst-group accuracy remains the most widely accepted measure of subgroup-level reliability, particularly in scenarios where minimizing failure on the most vulnerable group is critical.
E.2.3 Model selection
We adopted the oracle selection setting, where group attributes available during both training and validation. This is considered the most ideal scenario the benchmark specification, allowing test worst-group accuracy identify optimal algorithm performance. While feasible in real-world deployment, this setup provides a standardized estimate of each method's potential under full group supervision. It enables fair comparison by isolating algorithmic differences from model selection challenges.
E.2.4 Hyperparameter search
For MEDRO, we followed the official protocol, tuning across 16 randomized hyperparameter configu- rations. Each configuration was drawn from predefined search space in Table 6 Best configurations were selected based on validation worst-group accuracy, using group labels as specified in the protocol. For final evaluation, MEDRO retrained using the best hyperparameters and evaluated with three different random seeds.
To ensure consistency across tasks, we fixed the balance factor to in all experiments. This choice simplifies tuning and reflects a realistie scenario for evaluating MEDRO's generalization under uniform configuration.
In addition tuning the model learning rate ne (as part of the general search space), we tuned the risk weight step size nA which controls the group weight update during training. The range 10 Uniform( 3, 3-11 was used, consistent with the GroupDRO setting.
E.2.5 Implementation details
We followed the SubpopBench protocol for backbone, preprocessing. and optimization. For image datasets, we used ResNet-50 pretrained on ImageNet- K [51]: for text datasets, we used BERT-base beert-bas--uncased) [52] Image inputs were resized and center-cropped to 224x224, then normalized using ImageNet statistics. Image models were trained with SGD (momentum 0.9), and text models
25

---

Table 6: Search space for general hyperparameters used for MEDRO our SubpopBench experiments. All values were sampled per run via random search.
Backbone
Parameter
Search Range
Learning rate
10 [QUniform[44, -
Weight decay
10 [nUnfoor[[-G (-
ResNet-50
Batch size
2
Dropout
0.0 (fixed)
Learning rate
10 [quuifor[-55555
Weight decay
10 Uniioomm-6,---
BERT-base
Batch size
2nitom[(,5555
Dropout
Choice 0.1,
Training steps followed the SubpopBench standard: for Waterbirds and MetaShift, 20k for CheXpert, 30k for CelebA, CivilComments, MultiNI.I, and NICO++, and 60k for Living17.
with AdamW 53
E.2.6 Isolating ensemble effects: Multi-head architecture analysis
To determine whether MEDRO's performance improvements stem from its expanded uncertainty formulation from ensemble effects of using multiple expert heads, we conducted controlled comparison with GroupDRO applied to multi-hen head architecture. We implemented this variant with the following characteristics:
Same multi-head structure (m heads shared feature extractor)
Each head independently performs GroupDRO optimization
Total loss average of GroupDRO losses across heads
Same ensemble inference at test time
Heads differ only in random initialization and are not environment-speciffe experts. We evaluated this variant on eight SubpopBench datasets using the same hyperparameter search protocol as other baselines.
Table presents the comparison. GroupDRO with multi-head architecture achieves 67.5 overall worst-group accuracy, +0.4% difference from single-head GroupDRO (67.4%)) MEDRO achieves 69.5%, a +1.7% difference from the multi-head variant. These results indicate that performance difference between MEDRO and single-head GroupDRO (+2.1 1%) exceeds the difference between multi-head and single-head GroupDRO (+0.4%). This suggests that MEDRO's gains arise primarily from its algorithmic formulation rather than from ensemble effects.
Table Worst- Norst-group accuracy (%) comparison across SubpopBench datasets. Mean and standard deviation over three runs.
Method
Waterbirds
CalebA
CivilC.
MultiNLI
MetaShift
NICO+
CheXpon
Living Overall
GrapDRO GroupDRO (wl multi-head)
78.6 +10 89.0 207
70.6 212
76.0 207
85.6 204
37.8 #18
74.5 202
67.4 =04
27.2 215
90.2 103
71.5
74,8
11.3
85.4
424
73.5
28.8
67.8 32.6
MEDRO
83.8 113 90.4 204
73.4 205
75.7 40x
85.9 204
39.1 #35
75.0
207
69.5
E.2.7 Baseline method descriptions
We compared MEDRO against 19 baseline methods as implemented and defined in SubpopBench 12 These span a broad spectrum of approaches for mitigating subpopulation shift, including robust optimization, data augmentation, loss reweighting, and two-stage learning:
ERM 31: Standard empirical risk minimization without robustness interventions.
26

---

Minimizes worst-group risk by dynamically upweighting poorly perform- ing groups during training.
GroupDRO [6]
IRM [3] Encourages predictors remain invariant across multiple environments by enforcing equal optimality of predictors across domains.
Mixup 54
Regularizes training by interpolating random pairs of inputs and labels to generate synthetic examples.
JTT [55] Two-stage method that identifies high-loss examples via ERM and retrains model by upsampling them once.
DFR [56]. CRT [57], Re WeWeihtCRRT: Two-stage methods that first learn representations
via ERM and then retrain the classifier head on a balanced or reweighted dataset.
LfF [58] Simultaneously trains biased model and main model, reweighting samples in the latter based on difficulty estimated by the former.
LISA [2I]: Uses selective mixup across domains and classes to promote invariant prediction while reducing reliance spurious features.
CVaRDRO [59]:
Minimizes conditional value-at-risk over per-group losses to target high-
loss groups.
MMD 60] Minimizes the maximum mean discrepancy between group feature distributions to align representations.
Loss-level modifications designed to address class imbalance: CBLoss and Focal adjust per-sample loss weights, while LDAM adds class- dependent margins to logits improve separation.
CBLoss [6]] Focal 62] LDAM 63
ReSample, Weight,, SgrtReWeight [64]: Sampling- or weighting-based methods that
rebalance subgroup proportions during training.
Adjusts softmax normalization to account for class imbalance by using empirical class frequencies.
BSoftmax
65
E.3
WILDS settings
We evaluated MEDRO on three domain generalization tasks from the WILDS benchmark: Came- lyon17, iWildCam, PovertyMap. All experiments followed the official WILDS codebase and evalua- tion protocol without modification. The balance factor 7 was fixed to for all tasks, consistent with SubpopBench configuration.
Table summarizes the training configurations used for MEDRO on WILDS datasets. For Came- Iyon17, 7, we enabled ImageNet pretraining. following prior work [18 that demonstrated improved
performance with this setting.
Following the original GroupDRO implementation, we used default risk weight step size yA = 0.01 for MEDRO. For iWildCam, we found that a smaller step size of Чд = 0.0001 improved training stability and therefore adopted this value. We also enabled data augmentation specifically for iWildCam, using the corresponding option the WILDS codebase, as was found to enhance generalization in this setting.
F.3.1 Baseline method descriptions
compared MEDRO against 11 baseline methods commonly used in domain generalization, as evaluated in the WILDS benchmark. These methods span diverse strategies such invariant representation learning, adversarial alignment, risk variance minimization, data augmentation, model averaging, and two-stage selection correction:
ERM 31: Standard empirical risk minimization over pooled domains, without explicit robustness domain shift.
Minimizes the worst-case domain risk by upweighting domains with higher current loss during training.
GroupDRO (6:
IRM 13: Learns representations such that shared optimal classifier performs well across all training domains.
27

---

Table 8: Training configurations used for MEDRO on the WILDS datasets. Learning rate and weight decay were selected from the official WILDS search grids. For Map, the learning rate decays by a factor of 0.96 per epoch. Other settings follow the benchmark defaults unless otherwise noted.
Parameter
Camelyon17
iWildCam
Pov
Backbone
DenseNet- 121
ResNet-50
ResNet-18-ML
ImageNet Pretrained
True
True
True
Data Augmentation
None
RandAugment
None
Optimizer
SGD
Adam
Adam
Learning Rate
10
3 x 10-5
10-3
Weight Decay
10-2
0
0
Batch Size
32
16
64
Epochs
10
12
200
CORAL [16]: Aligns feature distributions across domains by minimizing discrepancies in second-order statisties (mean and covariance).
DANN [5] Uses adversarial training learn domain-invarian features by confusing domain classifier.
VREx 9
Minimizes variance in per-domain training risks to enforce uniform perfor- mance across domains.
LISA 2I: Applies selective mixup augmentation between domains and classes to weaken spurious correlations and promote invariant prediction.
Fish 23
Encourages gradient alignment across domains learn representations that generalize consistently.
SWAD 66
Averages model weights during training to find flatter minima that generalize better to unseen domains.
L2A-OT [22]:
Synthesizes pseudo-nove domains via optimal transport-based based data augmen- tation expand domain diversity.
HeckmanDG [18]
Models domain selection bias via a two-stage Heckman-style estimator to correct for distributional mismatch during training.
28

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately reflect the paper's contributions. The proposed multi-exper DRO (MEDRO) framework and its motivation, which involve addressing the limitations of GroupDRO under distribution shifts, are clearly stated. These claims are supported by theoretical analysis (Section 3) and empirical evaluation (Section 4). The paper also substantiates claims regarding improved robustness, expanded uncertainty set through eppert-environment modeling, and practical inference strategies for test-time deployment.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: While the main paper does not include a dedicated limitations section, Ap- pendix D contains theoretical analysis linking MEDRO domain generalization. This connection relies on strong assumptions, including expert optimality and uniformity of cross-environment risk, which may not hold in practical settings. These assumptions are acknowledged in the appendix, and we recognize that the theoretical claims under these assumptions may have limited applicability in more complex real-world scenarios.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-ter system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they seale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
29

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
The paper includes theoretical results with clearly stated assumptions and proofs. Appendix formally shows that GroupDRO is exactly or approximately contained within MEDRO, providing embedding arguments and approximation bounds. Appendix C presents a convergence analysis of the MEDRO optimization procedure under standard assumptions (convexity, Lipschitz gradients, compactness), establishing an ((1/vT) re- gret bound. Appendix D provides a theoretical connection between MEDRO's ' objective and domain-conditional invariance, proving that under idealized conditions the learned representation becomes label-conditionally invariant across domains. Key assumptions and objectives are defined in Section 3, and intuitive proof sketches are provided in the main text to aid understanding.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: All experiments follow established and publicly documented protocols. Con-
trolled setting evaluations replicate the procedure from the original GroupDRO paper [6]
(Section 4.2), SubpopBench evaluations follow the official benchmark protocol [2] (Sec-
tion 4.3), and WILDS evaluations use the standardized WILDS protocol (5] (Section 4.4). Section 4 provides high-level summaries of model configurations, metrics, and evaluation settings, while full implementation details-covering datasets, architectures, optimization hyperparameters, and training procedures-ae are provided Appendix E. These disclosures ensure that the main results and claims can be independently reproduced based on publicly available benchmarks, even without access to the original codebase.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
30

---

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: All datasets used in our experiments are publicly available. For subpopulation shift experiments, we datasets from SubpopBench (e.g.. CelebA, CivilComments), and for domain shift experiments, we use datasets from WILDS (e.g., Camelyon17, iWildCam). Data access and preparation follow the official benchmark protocols, and links or scripts will be included in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
CoodeSumeeeeeeeeeeeeee ssi aasionPPlicyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
31

---

6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All experiments follow the official protocols of the GroupDRO paper, Sub- popBench, and WILDS benchmarks. Relevant implementation and configuration details provided in Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report and standard deviation for all main experiments. For controlled setting experiments (CelebA and Waterbirds), results are averaged over five independent runs with different random seeds. For SubpopBench and WILDS benchmarks, we follow the number of repetitions defined by the respective protocols (c.g.. 3 runs for SubpopBench and iWildCam, 10 for Camelyon 17, and for PovertyMap) Error bars represent the standard deviation across these runs and are reported as mean + std in the corresponding tables. The number of runs and method of calculation are clearly stated in the captions and experimental sections.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Answer: [No]
32

---

The paper does not include detailed information compute resources such as GPU type or runtime. However, all experiments were executed using standard resources compatible with the official benchmark protocols (e.g., SubpopBench and WILDS), which are designed to be runnable on single modern GPU such as an NVIDIA RTX 3090 or 4090.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the wwwpllic/Ethisshideeel.eee
NeurlPS Code of Ethics httpp://neuut
Answer: [Yes]
Justification: The research uses only publicly available datasets (c.g., WILDS, Subpop- Bench), involves no human subjects or private data, and follows the experimental protocols of prior work. No ethical concerns related to privacy, safety, faimess were identified. We have reviewed the NeurIPS Code of Ethics and confirm full compliance.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
33

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks. mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual- dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All benchmarks have been properly credited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. com datasete has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We plan to provide the full codebase a public GitHub repository.
Guidelines:
The answer NA means that the paper does not release new assets.
34

---

Researchers should communicate the details of the dataset/codeemmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing o-nor research with human sub- jects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification:
The core method development in this research does not involve LLMss.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
35

---

Please refer to our LLM policy httpss//Deerriiscccc//oorrrrrres for what should or should be described.
36


Reviews:
[{"review_id": "lPL7uA0WLN", "reviewer": "Reviewer_2XX6", "summary": "This paper proposes an extension of GroupDRO that addresses limitations of single-classifier approaches in handling distribution shifts. Instead of using one global classifier, this paper employs multiple expert heads with a shared feature extractor. The key innovation is expanding the uncertainty set from m environments to m² expert-environment pairs, explicitly modeling cross-environment risks. The method is evaluated on both subpopulation shift and domain generalization benchmarks.", "questions": "1. How does MEDRO perform when m is large (e.g., m=10 or 20)? The m² scaling could become problematic.\n\n2. Beyond γ, how sensitive is the method to other hyperparameters? The balance between specialization and cross-environment robustness seems delicate.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Pros: \n\n1. The paper provides solid theoretical analysis showing how the proposed method generalizes GroupDRO by expanding the uncertainty set. The connection to domain generalization through label-conditional invariance is well-motivated, though relies on strong assumptions.\n\n2. The experiments span diverse settings - from controlled environments (CelebA, Waterbirds) to large-scale benchmarks (SubpopBench, WILDS). The consistent improvements across different types of distribution shifts demonstrate the method's generalizability.\n\n3. The paper addresses the real-world challenge of unknown environment labels at test time through ensemble and gating approaches, making the method more practically viable.\n\n4. The paper is well-written with good intuitive explanations and helpful visualizations (especially Figure 1) that clarify the conceptual differences between approaches.\n\nCons:\n\n1. While the execution is solid, the fundamental idea of using multiple expert heads isn't particularly novel. The main contribution is essentially replacing GroupDRO's single classifier with multiple heads and expanding the uncertainty set accordingly.\n\n2. The connection to domain generalization relies on assumptions like expert optimality and uniform cross-environment risks that may not hold in practice. The gap between theory and real-world applicability could be larger than suggested.\n\n3. Training m² expert-environment pairs likely increases computational cost significantly, but this isn't discussed. For large m, this could be prohibitive.\n\n4.  The paper doesn't provide clear guidance on when the expanded uncertainty set is most beneficial versus when GroupDRO might suffice.", "was_revised": true, "final_justification": "This paper proposed a new and principled min-max objective for DRO. My major concerns have been adequately addressed in the rebuttal.", "rebuttal": "### `Response to Cons 1`:\n> While the execution is solid, the fundamental idea of using multiple expert heads isn't particularly novel. The main contribution is essentially replacing GroupDRO's single classifier with multiple heads and expanding the uncertainty set accordingly.\n\nWe thank the reviewer for recognizing our solid execution. We agree that multi-expert architectures are an established concept. Our core novelty, however, is not the architecture itself, but the formulation of a new, principled min-max objective for distributionally robust optimization (DRO) that generalizes the GroupDRO framework.\n\nThe main contribution is the introduction of an expanded uncertainty set that explicitly models all $m^2$ cross-environment risks. MEDRO's objective optimizes over the convex hull of these $m^2$ risks, defined on an $(m^2-1)$-dimensional probability simplex. This is a fundamental extension of the GroupDRO uncertainty set, which is defined over the convex hull of $m$ environment risks for a single classifier. As we prove in Appendix B, our formulation is a direct generalization that recovers GroupDRO as a special case, highlighting a principled advancement of the DRO framework itself.\n\n### `Response to Cons 2`:\n> The connection to domain generalization relies on assumptions like expert optimality and uniform cross-environment risks that may not hold in practice. The gap between theory and real-world applicability could be larger than suggested.\n\nWe thank the reviewer for this insightful comment. We agree that the assumptions in our theoretical analysis (Appendix D) are strong and describe an idealized endpoint. We will clarify this in the paper.\n\nHowever, the MEDRO objective is designed to push the model towards a state where these conditions are *approximated*. The $\\max\\_{i,j} \\mathcal{R}\\_{i,j}$ term drives uniformity by repeatedly identifying the single worst-performing expert-environment pair and updating the model to specifically reduce that risk. This process constantly suppresses the highest values in the risk matrix, encouraging a more uniform risk distribution.\n\nSimultaneously, the $\\sum\\_k \\mathcal{R}\\_{k,k}$ term enforces expert optimality. While these two objectives are in tension, they are not mutually exclusive. Due to the high-dimensional and over-parameterized nature of neural networks, there exists a large space of solutions that can achieve low native-environment risk. The $\\max\\_{i,j} \\mathcal{R}\\_{i,j}$ term acts as a regularizer, guiding the optimization to select a solution from this space that is not only specialized but also robust, by learning a shared representation $\\phi$ that discards harmful, environment-specific features. The hyperparameter $\\gamma$ explicitly controls this trade-off.\n\nUltimately, our strong empirical results across SubpopBench and WILDS demonstrate that a favorable equilibrium, where experts are both highly proficient and robust, is achievable in practice.\n\n### `Response to Cons 3`:\n> Training $m^2$ expert-environment pairs likely increases computational cost significantly, but this isn't discussed. For large $m$, this could be prohibitive.\n\nWe thank the reviewer for raising this important point. We will add a discussion on computational cost to the appendix.\n\nFortunately, the additional cost of MEDRO is highly manageable and can be implemented efficiently. The most computationally intensive component, the shared feature extractor $\\phi$, is computed only once per sample, identical to GroupDRO. The overhead from the $m$ expert heads is minimal because their predictions can be computed in a single, parallelized operation. Instead of $m$ separate forward passes, the expert heads can be implemented as a single linear layer whose weights are the stacked weights of all experts. This allows all $m$ expert predictions for a batch to be computed with one efficient vector-matrix multiplication.\n\nThe following PyTorch snippet illustrates this:\n\n```python\nimport torch\nimport torch.nn as nn\n\nclass MultiExpertClassifier(nn.Module):\n    def __init__(self, feature_dim, num_experts, num_classes):\n        super().__init__()\n        self.num_experts = num_experts\n        self.num_classes = num_classes\n        # A single linear layer for all m experts\n        self.experts = nn.Linear(feature_dim, num_experts * num_classes)\n    \n    def forward(self, features):\n        # features shape: (batch_size, feature_dim)\n        # Efficiently compute all expert logits with one matrix multiplication\n        stacked_logits = self.experts(features)\n        # Reshape to get per-expert logits\n        # Output shape: (batch_size, num_experts, num_classes)\n        return stacked_logits.view(-1, self.num_experts, self.num_classes)\n```\nOur strong empirical results on iWildCam ($m=323$) and PovertyMap ($m \\ge 23$) further demonstrate that MEDRO is practical even for problems with a large number of environments.\n\n### `Response to Cons 4`:\n> The paper doesn't provide clear guidance on when the expanded uncertainty set is most beneficial versus when GroupDRO might suffice.\n\nWe thank the reviewer for this constructive suggestion. We will add this guidance to the paper, supported by both theoretical and empirical evidence.\n\n*Theoretically*, MEDRO is designed for scenarios with significant inter-environment heterogeneity, where optimal decision rules diverge across environments. As noted in Appendix A.1, GroupDRO's single-classifier approach is effective when a common decision rule can serve all environments, but it can be restrictive otherwise. MEDRO's formulation directly addresses this limitation:\n\n- Its multi-expert architecture ($\\omega_1,...,\\omega_m$) explicitly models the possibility of $m$ different optimal decision rules.\n- Its expanded uncertainty set over all $m^2$ cross-risks ($\\mathcal{R}_{i,j}$) specifically penalizes failures arising from this heterogeneity, i.e., when an expert $\\omega_i$ is applied to data from a mismatched environment $j$.\n\n*Empirically*, our results confirm that MEDRO's advantage is most pronounced in such high-heterogeneity settings:\n\n- On iWildCam (Table 3b), a domain generalization task with large distribution shifts across 323 camera trap locations, MEDRO achieves a test macro F1 score of 31.5%, substantially outperforming GroupDRO's 23.9%.\n- On PovertyMap (Table 4), a regression task with hybrid shifts across different countries, MEDRO improves the worst-group Pearson correlation to 0.49, a significant improvement over GroupDRO's 0.39.\n- On Waterbirds (Tables 1 & 2), where strong spurious correlations create a need for different decision logic, MEDRO consistently outperforms GroupDRO in worst-group accuracy.\n\nFurthermore, our Multi-Head GroupDRO experiment (see response to Reviewer J412's Weakness 1) provides additional evidence: when we use multiple heads without cross-environment modeling, the improvement over single-head GroupDRO is minimal. This confirms that MEDRO's gains come specifically from handling inter-environment heterogeneity, not from the multi-head architecture alone.\n\nConversely, GroupDRO may suffice in settings with milder, more homogeneous shifts. However, our results suggest that even in these cases, MEDRO often provides additional robustness.\n\n### `Response to Question 1`: \n> How does MEDRO perform when m is large (e.g., $m=$10 or 20)? The $m^2$ scaling could become problematic.\n\nThe concern about performance for large $m$ is closely tied to computational cost. As we discussed in our response to Weakness 3, our efficient implementation makes the method practical even at scale.\n\nThe empirical results confirm this. On **iWildCam** ($m=323$), MEDRO achieves a 31.5% test macro F1, and on **PovertyMap** ($m \\ge 23$), it achieves a 0.49 worst-group correlation. For a comprehensive view of environment counts across all our experiments, please refer to Table 5 in Appendix E.\n\nThis strong performance demonstrates that the method is not only computationally feasible but also highly effective in the large-$m$ regime.\n\n### `Response to Question 2`:\n> Beyond $\\gamma$, how sensitive is the method to other hyperparameters? The balance between specialization and cross-environment robustness seems delicate.\n\nWe thank the reviewer for this question. While the balance between specialization and cross-environment robustness may seem delicate, our empirical results suggest otherwise. Importantly, for all SubpopBench and WILDS experiments, we fixed $\\gamma=1$ without dataset-specific tuning (See Appendix E.2.4 and Appendix E.3), yet achieved strong performance across diverse tasks. This demonstrates that MEDRO is robust to this key hyperparameter—a single value works well across different domains, data modalities, and types of distribution shift. For other hyperparameters, we followed the same tuning protocols as all baseline methods—using identical search spaces and procedures specified by each benchmark. This ensures fair comparison where all methods benefit equally from general hyperparameter optimization (e.g., learning rate, weight decay). The fact that MEDRO achieves competitive performance with a fixed $\\gamma$ shows the method is practical and not overly sensitive. Moreover, this suggests potential for further improvements through task-specific tuning of $\\gamma$ in deployment scenarios."}, {"review_id": "CbwTm7CjX2", "reviewer": "Reviewer_Vdg5", "summary": "The authors propose Multi-Expert DRO, which generalizes the Group DRO through multiple prediction heads. Experimental results demonstrate the effectiveness of the proposed method. And some standard theoretical anlyses are given in Appendix.", "questions": "Although the results are impressive—indeed, surprisingly so—I have the following questions and concerns:\n\n* **Source of Performance Gains**: Could the authors more clearly illustrate the source of the performance improvement, either through an intuitive explanation or a more detailed ablation study? I am very familiar with this research area, and I would not have expected such a substantial performance gain from the proposed method without additional justification.\n\n* **Role of the Gating Mechanism**: Based on Table 2, it appears that the gating mechanism contributes minimally to the overall performance. This suggests that the primary gains may stem from the averaging over multiple heads. I encourage the authors to conduct further analysis here—for instance, by varying the number of heads and examining how performance is affected.\n\n* **Impact of Hyperparameter Search Space**: Tables 6 and 7 suggest that the key distinction between results on the Subpopulation benchmark and other domain generalization (DG) benchmarks lies in the hyperparameter search space. When considered alongside the results in Table 2 (where MEDRO performs best) and Tables 3 & 4 (where MEDRO is not the best), this raises the question of whether hyperparameter tuning plays an outsized role in the reported gains. I strongly recommend that the authors report the optimal hyperparameters used for both MEDRO and all baseline methods, to ensure transparency and demonstrate that fair tuning was applied across all comparisons.", "limitations": "There lacks three challenging datasets from sub-population benchmark.", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n* **Methodology**: While the multi-head architecture itself is not novel, this paper may be the first to integrate it with Group DRO. This combination can be considered a meaningful technical contribution.\n\n* **Experimental Results**: The results presented in the main paper are strong, showcasing the effectiveness of MEDRO on the selected datasets.\n\n**Weaknesses:**\n\n* **Dataset Selection**: The authors chose only a subset of datasets from the Subpopulation benchmark, referring to them as \"representative.\" However, the original Subpopulation benchmark paper identifies ImageNetBG, Living17, and NICO++ as the most challenging datasets. Why were these omitted from the evaluation? I strongly encourage the authors to include results on these three datasets. Additionally, the paper should clearly report the optimal hyperparameters used (please also see my related comments below).\n\n* **Domain Generalization (DG) Performance**: In the appendix, the authors show that the learned representations can be domain-invariant. However, this raises a question: why does MEDRO fail to outperform the baselines on standard DG benchmarks? A deeper analysis or explanation here would be valuable.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback. We will address each point below.\n\n---\n\n### `Response to Weakness 1: Dataset Selection`\n> \"... ImageNetBG, Living17, and NICO++ as the most challenging datasets. ... I strongly encourage the authors to include results on these three datasets. ... report the optimal hyperparameters ....\"\n\nWe thank the reviewer for this constructive feedback and for pushing us to evaluate on the most challenging SubpopBench datasets. We agree that a more comprehensive evaluation strengthens the paper.\n\n- **Regarding dataset selection**: We have now run experiments on NICO++ and Living17, as suggested. We were unable to include ImageNetBG as the official download link is unavailable, and we received no response from the benchmark's maintainers.\n- **Summary of new results**: On these new, challenging datasets, MEDRO remains highly competitive. On Living17, \"MEDRO (w/ gating)\" achieves a worst-group accuracy of __33.6%__ , placing it among the top-performing methods and substantially outperforming the GroupDRO baseline (27.2%). On NICO++, MEDRO's performance is on par with several strong baselines. **The full table will be presented in the updated manuscript.**\n- **Overall Performance**: Most importantly, the inclusion of these datasets confirms the robustness of our approach. As shown in the updated table, MEDRO (w/ gating) achieves the highest overall worst-group accuracy of __70.0%__ across all eight benchmarks, underscoring its strong and consistent performance.\n- **Hyperparameters**: We will add a table to the paper detailing the optimal hyperparameters used for all datasets, including NICO++ and Living17, in the final version.\n\n| Method | NICO++ | Living17 | Overall |\n| :--- | :---: | :---: | :---: |\n| ERM | 37.6 (±2.0) | 28.2 (±1.5) | 57.6 |\n| Mixup | 42.7 (±1.4) | 29.8 (±1.8) | 57.7 |\n| GroupDRO | 37.8 (±1.8) | 27.2 (±1.5) | 67.4 |\n| IRM | 40.0 (±0.0) | 28.2 (±1.5) | 56.2 |\n| CVaRDRO | 36.7 (±2.7) | 28.3 (±0.7) | 59.9 |\n| JTT | 40.0 (±0.0) | 28.8 (±1.1) | 61.1 |\n| LfF | 30.4 (±1.3) | 26.2 (±1.1) | 48.3 |\n| LISA | 42.7 (±2.2) | 29.8 (±0.9) | 69.3 |\n| MMD | 40.7 (±0.5) | 26.6 (±1.8) | 54.4 |\n| ReSample | 40.0 (±0.0) | 30.7 (±2.1) | 67.8 |\n| ReWeight | 41.9 (±1.6) | 28.2 (±1.5) | 68.7 |\n| SqrtReWeight | 40.0 (±0.0) | 28.2 (±1.5) | 65.6 |\n| CBLoss | 37.8 (±1.8) | 28.2 (±1.5) | 68.4 |\n| Focal | 36.7 (±2.7) | 28.0 (±1.2) | 56.3 |\n| LDAM | 42.0 (±0.9) | 24.7 (±0.8) | 53.0 |\n| BSoftmax | 40.4 (±0.3) | 27.5 (±0.8) | 65.2 |\n| DFR | 23.7 (±0.7) | 29.0 (±0.2) | 66.2 |\n| CRT | 43.3 (±2.7) | 33.9 (±0.1) | 68.1 |\n| ReWeightCRT | 23.3 (±1.4) | 33.7 (±0.1) | 65.5 |\n| **MEDRO** | 39.1 (±2.5) | 32.6 (±0.7) | 69.5 |\n| **MEDRO (w/ gating)** | 39.1 (±2.5) | 33.6 (±0.6) | 70.0 |\n\n---\n\n### `Response to Weakness 2: DG Performance`\n> \"In the appendix, the authors show that the learned representations can be domain-invariant. However, this raises a question: why does MEDRO fail to outperform the baselines on standard DG benchmarks? A deeper analysis or explanation here would be valuable.\"\n\nWe thank the reviewer for this important question. The theory in Appendix D presents an idealized condition to motivate MEDRO's use for domain generalization (DG), not to guarantee universal state-of-the-art (SOTA) performance. Our main goal was to show a significant advantage over our primary baseline, GroupDRO, which our results consistently confirm.\n\nWhen compared to other specialized DG methods, MEDRO is highly competitive and directly counters the premise that it fails to outperform baselines:\n- On Camelyon17, MEDRO achieves the highest test accuracy (87.8%) of all listed methods.\n- On PovertyMap and iWildCam, it is highly competitive with other SOTA methods.\n- Critically, on all three WILDS datasets, MEDRO offers substantial improvements over the GroupDRO baseline, with a +7.6% F1 gain on iWildCam and a +0.10 worst-group correlation gain on PovertyMap.\n\nWhile no single algorithm excels on all complex DG benchmarks, MEDRO's competitive performance validates its design as a principled and effective approach for domain generalization.\n\n---\n\n### `Response to Question 1: Source of Performance Gains`\n> \"Could the authors more clearly illustrate the source of the performance improvement, either through an intuitive explanation or a more detailed ablation study? ....\"\n\nWe thank the reviewer for this insightful question. The performance gain stems from MEDRO's balanced objective, which jointly optimizes for worst-case robustness and expert specialization.\n\nThe first component of our objective, the expanded $m^2$ risk set, allows MEDRO to identify harder, more specific failure modes than GroupDRO (see Figure 3). However, optimizing this worst-case objective alone is risky, as it could lead to an overly pessimistic model.\n\nThis is why the second component, the **specialization term** ($\\sum_{i=1}^{m} \\mathcal{R}_{i,i}$), is not a minor detail but a **co-equal pillar of our framework**. It acts as a crucial regularizer that anchors each expert to its native domain, preserving specialized knowledge and preventing the kind of model collapse that would otherwise occur. This explicit balance between finding the worst-case risk and grounding the experts in their own environments is the true source of MEDRO's robust performance.\n\nOur ablations confirm this two-part mechanism.\n1.  The performance gap between MEDRO and GroupDRO (Tables 1-4) proves the value of the expanded $m^2$ risk objective.\n2.  To prove the critical role of the specialization term, we ran a new ablation on MetaShift & Waterbirds where this term was removed. The results are stark:\n\n(Table for MetaShift)\n| Method | Avg. | Worst |\n| :--- | :---: | :---: |\n| ERM | 91.3 (±0.3) | 82.6 (±0.4) |\n| GroupDRO | 91.0 (±0.1) | 85.6 (±0.4) |\n| **MEDRO (w/o expert)** | 91.0 (±0.1) | 83.1 (±0.1) |\n| **MEDRO (Full)** | 91.3 (±0.2) | 85.9 (±0.4) |\n\n\n(Table for Waterbirds)\n| Method | Avg. | Worst |\n| :--- | :---: | :---: |\n| ERM | 84.1 (±1.7) | 69.1 (±4.7) |\n| GroupDRO | 88.8 (±1.8) | 78.6 (±1.0) |\n| **MEDRO (w/o expert)** | 90.7 (±0.5) | 80.9 (±1.8) |\n| **MEDRO (Full)** | 90.5 (±0.6) | 83.8 (±1.3) |\n\nAs the tables show, without the specialization term, MEDRO’s performance collapses, becoming **2.8% worse on Metashift, 2.9% worse on Waterbirds than the GroupDRO baseline**. This empirically confirms that the gains from the expanded uncertainty set are only realized when counter-balanced by the specialization term, validating our balanced design.\n\n---\n\n### `Response to Question 2: Role of the Gating Mechanism`\n> \"Based on Table 2, it appears that the gating mechanism contributes minimally to the overall performance. This suggests that the primary gains may stem from the averaging over multiple heads. I encourage the authors to conduct further analysis here---for instance, by varying the number of heads and examining how performance is affected.\"\n\nWe thank the reviewer for this insightful question regarding the role of the inference mechanism. The reviewer's observation is correct: the performance gain from the gating network over a simple ensemble is incremental, as shown on SubpopBench where it improves the overall worst-group accuracy from 80.7% to 81.2%.\n\nWe wish to clarify that our core contribution is the MEDRO *training* framework itself, which produces the robust experts through cross-environment risk modeling. The inference strategies are presented as practical, downstream solutions for the common scenario where test-time environment labels are unavailable. We believe the strong performance of the simple ensemble is, in fact, a testament to the quality of the experts learned by MEDRO; they are so mutually resilient that their simple average is already a very powerful baseline.\n\nRegarding the excellent suggestion to analyze performance by varying the number of heads, we would like to clarify a key aspect of our framework. The number of expert heads is *not* a free hyperparameter but is determined by the number of predefined environments, $m$, in a given problem (e.g., $m=4$ for CelebA). Our framework's one-to-one expert-environment mapping is central to its design for modeling all $m^2$ cross-risks. Therefore, varying the number of heads is infeasible within this formulation.\n\nTo address whether gains come from multi-head averaging or our cross-environment objective, we conducted an ablation with Multi-Head GroupDRO (see our response to Reviewer J412's Weakness 1). This experiment shows minimal gains from multi-head architecture alone (+0.3% on SubpopBench), while MEDRO achieves +1.6%, confirming that our performance stems from the expanded uncertainty set, not merely ensemble effects. We will clarify these points in the final paper. Thank you for this important question.\n\n---\n\n### `Response to Question 3: Impact of Hyperparameter Search Space`\n> \"Tables 6 and 7 suggest that the key distinction between results on the Subpopulation benchmark and other domain generalization (DG) benchmarks lies in the hyperparameter search space. ... I strongly recommend that the authors report the optimal hyperparameters used for both MEDRO and all baseline methods, ....\"\n\nWe thank the reviewer for this important question on hyperparameter tuning. We agree that transparency is crucial and clarify that our evaluation for all benchmarks (SubpopBench and WILDS) strictly followed their respective, published official protocols to ensure fair comparison. MEDRO was subject to the exact same standardized tuning procedures as the baselines, as detailed in our Appendix E.\n\nRegarding the request to report hyperparameters, we will add a table to the appendix with the optimal hyperparameters for MEDRO on all datasets. As is standard practice, the baseline results are taken from the original benchmark papers. We therefore respectfully refer the reviewer to those publications for the baselines' specific tuning details. We are confident our adherence to these rigorous public protocols ensures a fair evaluation."}, {"review_id": "ALhRFAkezL", "reviewer": "Reviewer_WNwh", "summary": "The authors deal with the GroupDRO problem, which uses distributionally robust optimisation over the convex hull of a set of distributions, each corresponding to one of $m$ environments, samples of each compose the dataset. Here, learning amounts to minimising the worst case loss over all environments. The authors extend this work by suggesting a mixture-of-experts formulation, where, keeping a common representation $\\phi$, they learn an expert $ \\omega_i\\circ \\phi $ for each environment. They then optimise for a weigted average of both the ERM loss (assuming a MoE model) as well as the worst case of all experts individually, over all combinations of environments. During prediction (where the environments are not known) the full MoE is used. As a MoE for prediction they either use a gating network, or simply an average of all the experts. They offer good experimental coverage of their method.", "questions": "1. How would you expect the performance to change (test accuracy) if the experts were increasingly more constrained to be similar?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "# Strengths\n* This is a good idea, which is well motivated, and the exposition is quite neat and clear.\n* A decent amount of theory is provided, albeit hidden in the supplementary.\n* The experimental design is thorough, CIs have been provided, and established benchmarks have been used.\n* The method performs well in appropriate datasets (CelebA which is large scale and Waterbirds which is a medium-scaled one).\n* A good intuition is provided, good coverage of Related works, and good mathematical formalisation.\n* MEDRO is a strict generalisation of GroupDRO.\n\n\n\n# Weaknesses\n* The general idea is relatively simple, but seems well-executed.\n* The theory is too hidden in the supplementary materia. I would suggest that key claims are mentioned in the main body of the paper, with references to the proofs in the appendix.\n* The method provided is not uniformly superior to other SOTA methods, but does perform on par, and seems to broaden the available toolset of GroupDRO.\n* Some directions in the related work to more general methods seem potentially useful. For instance, risk extrapolation [1], CKDRO [2], IPM-DROs [3], etc.\n\n\n# Notes:\nI would like to mention that generally increasing the area over which DRO (and, hence, GroupDRO, MEDRO) is performed, does not necessarilly lead to improvements. This can mean loss of specificity or overly pessimistic regressors/classifiers.\\\nFrom this perspective, and to compare the effects of your model to its closest cousin, the GroupDRO, it could perhaps be interesting to perform an analysis on whether/when the extended set of distributions used would be of benefit.\n\nIn lieu of a theoretical analysis, one could demonstrate the curve of accuracy as one smoothly moves from the GroupDRO setting to that of MEDRO.\n\nFor instance, consider a paradigm where the weights of each head $\\omega_i$ were constrained to lie within a ball of constantly increading radius, starting from 0 (GroupDRO) to the actual unconstrained one.\n\n# Suggestions\n* I would suggest the use of parentheses in eq 5 around the composition: $ (\\omega_i\\circ \\phi)(x) $, to align with standard practices.\n\n* Also, consider opting for the official sources instead of preprint versions, when applicable. For instance:\n   * Improving Domain Generalization with Domain Relations\\\n      $ \\to $ ICLR24\n   * Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization\\\n     $ \\to $ Distributionally Robust Neural Networks ICLR20\n   * The Risks of Invariant Risk Minimization\\\n     $ \\to $ ICLR21\n\n# References\n[1] Y. Chen, J. Kim, and J. Anderson, “Distributionally Robust Decision Making Leveraging Conditional Distributions,” in 2022 IEEE 61st Conference on Decision and Control (CDC), Dec. 2022, pp. 5652–5659. doi: 10.1109/CDC51059.2022.9992782.\n\n[2] D. Krueger et al., “Out-of-Distribution Generalization via Risk Extrapolation (REx),” in Proceedings of the 38th International Conference on Machine Learning, PMLR, Jul. 2021, pp. 5815–5826. Accessed: Jul. 02, 2025. [Online]. Available: https://proceedings.mlr.press/v139/krueger21a.html\n\n[3] M. Staib and S. Jegelka, “Distributionally Robust Optimization and Generalization in Kernel Methods,” in Advances in Neural Information Processing Systems, Curran Associates, Inc., 2019. Accessed: Jul. 02, 2025. [Online]. Available: https://proceedings.neurips.cc/paper/2019/hash/1770ae9e1b6bc9f5fd2841f141557ffb-Abstract.html", "was_revised": true, "final_justification": "This is an incremental work, that is, however, worthwhile: it is well written, while it offers interesting and supportive experimental results.\n\nI maintain my original belief that it should be accepted.\n\nAdditionally, the authors addressed certain minor concerns during the rebutal.", "rebuttal": "We thank the reviewer for their insightful and positive feedback. We appreciate the thoughtful assessment and will address the suggestions and questions point by point below.\n\n---\n\n### `Response to Weakness 1:`\n> \"The general idea is relatively simple, but seems well-executed.\"\n\nThank you for recognizing the quality of our execution. We view the simplicity of our core idea as a strength—it suggests we've identified a natural solution to a fundamental problem. The fact that this straightforward insight (cross-environment risk modeling) leads to both theoretical guarantees (domain invariance) and consistent empirical improvements across diverse benchmarks demonstrates its effectiveness. We appreciate your positive assessment of how we developed and validated this approach.\n\n---\n\n### `Response to Weakness 2:`\n> \"The theory is too hidden in the supplementary material. I would suggest that key claims are mentioned in the main body of the paper, with references to the proofs in the appendix.\"\n\nThank you for pointing this out. While we described our theoretical results narratively, we agree they should be more prominent. We will revise to highlight key claims (e.g., as propositions or remarks) within the existing narrative flow, ensuring theoretical contributions are clear while maintaining the paper's readability. Detailed proofs will remain in the appendices.\n\n---\n\n### `Response to Weakness 3:`\n> \"The method provided is not uniformly superior to other SOTA methods, but does perform on par, and seems to broaden the available toolset of GroupDRO.\"\n\nWe appreciate your accurate assessment of our empirical results. You are correct that MEDRO is not uniformly superior across all settings. \n\nOur contribution is not claiming universal dominance, but rather providing a principled extension of GroupDRO that:\n* Consistently improves over GroupDRO itself (e.g., +3.7% on CelebA, +2.2% on Waterbirds worst-group accuracy)\n* Achieves overall best performance on SubpopBench (80.7% average worst-group accuracy across 6 datasets)\n* Remains highly competitive on domain shift tasks while being designed primarily for subpopulation robustness\n\nAs you noted, MEDRO broadens the DRO toolset by explicitly modeling cross-environment risks: a perspective that complements existing approaches. The fact that this principled framework achieves competitive performance across diverse settings while providing theoretical insights (domain invariance connection) validates its value as a new tool in the robustness toolkit rather than a universal replacement.\n\n---\n\n### `Response to Weakness 4:`\n> \"Some directions in the related work to more general methods seem potentially useful. For instance, risk extrapolation [1], CKDRO [2], IPM-DROs [3], etc.\"\n\nThank you for these valuable suggestions for strengthening our related work section. We will incorporate discussions of:\n* **risk extrapolation, REx** [1]: this is indeed a highly relevant work. REx minimizes risk variance across environments to encourage invariance, while MEDRO takes a different approach by explicitly modeling cross-environment risks $\\mathcal{R}_{i,j}$. Both methods aim to learn robust representations beyond standard GroupDRO, but through complementary mechanisms.\n* **CKDRO** [2]: This conditional approach is indeed relevant as it also goes beyond standard DRO. We will discuss how MEDRO's expert-based conditioning differs from their distribution-conditional framework.\n* **IPM-DROs** [3]: These kernel-based methods provide another perspective on robustness. We will highlight how MEDRO's discrete environment structure contrasts with their continuous uncertainty sets.\n\nThese additions will better position MEDRO within the broader DRO landscape, showing how our cross-environment risk modeling complements these other generalizations of distributionally robust optimization. Thank you for helping us provide a more comprehensive literature review.\n\n---\n\n### `Response to Notes and Question 1`\n__On overly pessimistic uncertainty set:__\n> \"I would like to mention that generally increasing the area over which DRO (and, hence, GroupDRO, MEDRO) is performed, does not necessarilly lead to improvements. This can mean loss of specificity or overly pessimistic regressors/classifiers.\"\n\n We thank you for raising this crucial point. You are correct that naively enlarging the uncertainty set in DRO does not guarantee improvement and can lead to a loss of specificity or overly pessimistic models. We designed MEDRO's objective specifically to counteract these known failure modes.\n\nThe source of MEDRO's performance gain is its ability to optimize over a more expressive risk landscape by considering all $m^2$ expert-environment pairings. However, the objective includes two crucial safeguards to prevent the exact issues you mentioned:\n\n1.  **Adaptive and Bounded Pessimism:** The pessimism is principled because the uncertainty set is **bounded by the training data**; it guards against plausible failures, not arbitrary ones. Furthermore, as visualized by the risk polytope (the \"cheesecake\") in **Figure 1(c)**, the objective is **adaptive**. When environments are very different, the polytope is large, reflecting a truly high worst-case risk. But if environments were more similar, the polytope would naturally shrink, causing MEDRO’s objective to relax and behave more like standard GroupDRO. This prevents the model from being overly pessimistic on simpler problems.\n\n2.  **Preventing Loss of Specificity:** The **specialization term** ($\\sum_{i=1}^{m} \\mathcal{R}_{i,i}$) is a co-equal pillar of our framework. It acts as a powerful regularizer that anchors each expert to its native domain, explicitly preserving specialized knowledge and preventing a loss of specificity, which would otherwise lead to model collapse.\n\nIt is this explicit balance between an adaptive worst-case search and the strong anchor of the specialization term that allows MEDRO to improve robustness while avoiding these common pitfalls.\n\n__On smoothly moving from GroupDRO to MEDRO:__\n> \"From this perspective, and to compare the effects of your model to its closest cousin, the GroupDRO, it could perhaps be interesting to perform an analysis on whether/when the extended set of distributions used would be of benefit. In lieu of a theoretical analysis, one could demonstrate the curve of accuracy as one smoothly moves from the GroupDRO setting to that of MEDRO. For instance, consider a paradigm where the weights of each head $\\omega_i$ were constrained to lie within a ball of constantly increading radius, starting from 0 (GroupDRO) to the actual unconstrained one.\"\n\nWe thank the reviewer for the excellent suggestion and the insightful related question. We believe that it cuts to the heart of the trade-off between a single global classifier and our multi-expert approach. We agree that an analysis smoothly moving from the GroupDRO setting to MEDRO would be very valuable.\n\nOur understanding is that this experiment can be implemented by adding a penalty term, $C \\cdot \\sum \\|\\omega_i - \\bar{\\omega}\\|_2$, to our objective. By varying the hyperparameter $C$ from a large value down to zero, we could effectively trace the path from a GroupDRO-like model (where all experts are forced to be similar) to the full MEDRO model.\n\nThis directly addresses your Question 1: \n> \"How would you expect the performance to change (test accuracy) if the experts were increasingly more constrained to be similar?\"\n\nWe hypothesize that on **heterogeneous datasets** where environments have meaningfully different properties, the **worst-group test accuracy would smoothly decrease** as the constraint to be similar increases (i.e., as $C$ gets larger). The curve would start at the higher performance level of the full MEDRO model and decay towards the lower performance of the GroupDRO baseline. This is because constraining the experts prevents them from learning the specialized decision boundaries needed to handle the challenging minority groups that differ most across environments. On simpler, more homogeneous datasets, we would expect this curve to be much flatter.\n\nWhile this is a valuable (though computationally intensive) analysis that we were unable to complete within the tight rebuttal period, we will prioritize running a preliminary version of this experiment during the author discussion period and give our best efforts to report back with the results. Thank you for providing this clear direction for future investigation.\n\n---\n\n### `Response to Suggestions`\n> \"I would suggest the use of parentheses in eq 5 around the composition: $(\\omega \\circ \\phi)(x)$, to align with standard practices. Also, consider opting for the official sources instead of preprint versions, when applicable. For instance: ...\"\n\nThank you for these constructive suggestions. We agree that adding parentheses around the function composition will improve clarity and align with standard mathematical notation. We will update Eq. (5) and all similar expressions throughout the paper accordingly.\n\nRegarding the citations, we appreciate you identifying the official publication venues. We will update our references to cite the published versions. We will also review our entire bibliography to ensure all papers are cited using their final published versions rather than preprints where applicable."}, {"review_id": "dMx5tqzzyN", "reviewer": "Reviewer_J412", "summary": "In this paper, the authors propose a distributionally robust optimization (DRO) model, MEDRO, extending the well-known GroupDRO by incorporating a multi-experts learning architecture. The idea is that by introducing the extra expert-environment interactions, more spurious relations can be identified and eliminated via optimization. The authors justify the methodology via thorough theoretical studies. The effectiveness of MEDRO over prior works is then confirmed via comprehensive experiments.", "questions": "1. About the guideline of choosing ensemble or gating function MEDRO in line 197-199, does the validation data follow train or test data distribution?\n2. Is MEDRO (ours) in Table 1 based on simple ensemble or gating function? Also, do the authors consider reporting performance of both ensemble and gating function for all the experiments?\n3. Why are the ERM’s performance on CelebA and Waterbirds significantly different between Table 2 and 3?\n4. About Figure 2, maybe aligning the y-axes of the two plots, so that the impacts on average and worst-case performance can be compared easily?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n1. The authors build a solid theoretical foundation supporting the arguments and the proposed model.\n2. The experiments are comprehensive in terms of both distribution shifts and out-of-distribution generalization methods.\n3. The writing is impressive. The authors do a great job presenting the background knowledge and their research under the page limit.\n\n**Weaknesses**\n1. Whether the performance gain comes from ensemble may be unclear. MEDRO natively possesses the ensemble mechanism but most of the other models do not.Therefore, for example, can we expect the performance gap becomes smaller if we implement GroupDRO with ensemble?\n2. Although the proposed algorithm is reasonable, it is not particularly surprising.\n3. Some details of experiments are unclear and may impact the justification of model performance. ( See Question 1 and 2)", "was_revised": true, "final_justification": "The authors resolve the concern that the perfomance comes from ensemble instead of the desired cross-environment optimization. The new experiments suggest the advantage from the core algorithm is solid. Also, the authors help clarify the experiment settings, justifying the evaluation results. The overall quality of this work is good, while the novelty of the methodology is slightly limited. I therefore recommend borderline accept.", "rebuttal": "### `Weakness 1:`\n> \"Whether the performance gain comes from ensemble may be unclear. MEDRO natively possesses the ensemble mechanism but most of the other models do not. Therefore, for example, can we expect the performance gap becomes smaller if we implement GroupDRO with ensemble?\"\n\nThank you for raising this important point about disentangling the ensemble effect from MEDRO's core contribution. To address this concern, we implemented Multi-Head GroupDRO with the same architecture as MEDRO:\n- Same multi-head structure ($m$ expert heads + shared feature extractor)\n- Each head independently performs GroupDRO optimization\n- Total loss = average of $m$ GroupDRO losses\n- Same ensemble inference at test time\n\nNote that Multi-head GroupDRO's heads are not environment-specific experts, but merely differ in random initialization---each head minimizes the same robust objective independently.\n\nResults demonstrate clear separation between ensemble effects and MEDRO's core contribution.\n\nKey findings:\n1. Multi-Head GroupDRO shows minimal improvement over single-head GroupDRO (+0.2% on CelebA, +0.8% on Waterbirds), confirming limited ensemble benefits in this setting.\n2. MEDRO significantly outperforms Multi-Head GroupDRO (+3.5% on CelebA, +1.4% on Waterbirds), demonstrating that our gains primarily come from modeling cross-environment risks.\n3. SubpopBench results further validate this: across 6 datasets, Multi-Head GroupDRO achieves 79.4% (only +0.3% over single-head), while MEDRO achieves 80.7% (+1.3% over multi-head).\n4. This experiment clearly shows that MEDRO's advantage stems from its expanded uncertainty set created by modeling cross-environment risks, not merely the ensemble mechanism.\n\nThe performance gap remains significant even after controlling for ensemble effects, validating that our method's contribution goes beyond architectural choices to a fundamentally different optimization objective.\n| | CelebA | | Waterbirds | |\n|--------|--------|-------|-----------|-------|\n| **Method** | **Avg.** | **Worst** | **Avg.** | **Worst** |\n| GroupDRO | 93.7 (±0.3) | 85.7 (±2.2) | 96.1 (±1.3) | 84.1 (±2.4) |\n| GroupDRO (w/ multi-head) | 93.3 (±0.4) | 85.9 (±1.4) | 93.7 (±1.2) | 84.9 (±1.6) |\n| MEDRO | 92.2 (±0.8) | 89.4 (±1.2) | 94.2 (±1.8) | 86.3 (±1.9) |\n\n| Method | Waterbirds | CelebA | CivilC. | MultiNLI | MetaShift | CheXpert | Overall |\n|--------|------------|--------|---------|----------|-----------|----------|---------|\n| GroupDRO | 78.6 (±1.0) | 89.0 (±0.7) | 70.6 (±1.2) | **76.0** (±0.7) | 85.6 (±0.4) | 74.5 (±0.2) | 79.1 |\n| GroupDRO (w/ multi-head) | 81.1 (±1.3) | 90.2 (±0.3) | 71.5 (±1.4) | 74.8 (±1.3) | 85.4 (±1.0) | 73.5 (±0.1) | 79.4 |\n| **MEDRO** | **83.8** (±1.3) | **90.4** (±0.4) | **73.4** (±0.5) | 75.7 (±0.8) | **85.9** (±0.4) | **75.0** (±0.4) | **80.7** |\n\n### `Weakness 2:`\n>\"Although the proposed algorithm is reasonable, it is not particularly surprising.\"\n\nThank you for this thoughtful comment. While we understand why MEDRO might appear as a straightforward extension of GroupDRO, we respectfully argue that it addresses fundamental limitations through a principled new approach.\n\n**Cross-environment risk modeling**\n\nThe key difference lies in what we optimize. GroupDRO minimizes the worst-case risk over m environments using a single classifier. MEDRO introduces cross-environment risks R_{i,j}—the risk when expert i processes data from environment j—creating m² distinct risks to optimize. This is not merely quantitative expansion. Each R_{i,j} captures a specific failure mode. For instance, in Waterbirds, R_{land, water} measures how an expert trained on land background fails when processing water backgrounds. By explicitly minimizing all such mismatches, MEDRO pressures the shared feature extractor φ to learn background-invariant representations.\n\n**Overcoming the weighted ERM limitation**\n\nCritically, GroupDRO as weighted ERM has fundamental limitations in learning invariant features (IRM, Proposition 2). It can only reweight samples, not change the representation itself.\n\nMEDRO's architecture—multiple **expert** heads with a shared feature extractor—combined with our objective that balances specialization (Σᵢ R_{i,i}) and cross-environment robustness (max_{i,j} R_{i,j}) encourages invariant representation learning. Our analysis (Appendix D) shows that when cross-environment risks are equalized (R_{i,j} ≈ R_{i,i}), the learned features tend toward satisfying P_i(Y|φ*(x)) ≈ P_j(Y|φ*(x)). This theoretical connection emerges naturally from our objective.\n\nImportantly, optimizing m² risks does not significantly slow convergence: we maintain O(1/√T) rate with only a logarithmic factor difference (Appendix C).\n\nIn summary, MEDRO's contribution lies in recognizing that different expert-environment interactions represent distinct failure modes worth modeling explicitly, and showing how this leads to both theoretical insights and empirical improvements.\n\n### `Weakness 3:`\n>\"Some details of experiments are unclear and may impact the justification of model performance. (See Question 1 and 2)\"\n\nWe appreciate you pointing out these important experimental details. As addressed in our responses to Questions 1 and 2:\n\n- **Validation distributions**: We have clarified that validation sets follow different distribution patterns depending on the task type (ID for domain generalization, same groups but different proportions for subpopulation shift).\n\n- **Inference strategy**: We have provided complete results for both strategies. We reported selectively based on context: ensemble for fair comparison with single-stage methods (Table 1), both when two-stage methods exist (Table 2), and ensemble for domain shift tasks where gating offers limited OOD benefits (Table 3-4).\n\nThese clarifications demonstrate that our experimental choices were principled and consistent. We will revise the manuscript to make these methodological details explicit upfront, ensuring full transparency.\n\n### `Question 1:`\n>\"About the guideline of choosing ensemble or gating function MEDRO in line 197-199, does the validation data follow train or test data distribution?\"\n\nThank you for this clarification question. The validation distributions are:\n\n- **Domain Generalization**: ID validation set (follows train distribution, while test is out-of distribution)\n\n- **Subpopulation shift**: All datasets (train/valid/test) contain the same groups, but the group proportions typically differ across splits\n\nWe will incorporate these clarifications into the revised manuscript to ensure readers clearly understand our experimental setup and choices.\n\n### `Question 2:`\n>\"Is MEDRO (ours) in Table 1 based on simple ensemble or gating function? Also, do the authors consider reporting performance of both ensemble and gating function for all the experiments?\"\n\nThank you for this important clarification question. The MEDRO results in Table 1 are based on the simple ensemble approach. \n\nWe actually evaluated both inference strategies across all experiments during our research. However, we selectively reported results based on the experimental context:\n\n- **Table 1 (GroupDRO benchmark)**: We reported only simple ensemble to ensure fair comparison with 1-stage baselines (ERM, GroupDRO)\n- **Table 2 (SubpopBench)**: We included both strategies since the benchmark already contains 2-stage methods like DFR, CRT, and JTT\n- **Table 3 & 4 (WILDS)**: We focused on simple ensemble as gating networks trained on ID validation data provide limited benefits for OOD test performance\n\nPer your request, here are the complete results for both inference strategies across all experiments:\n| | CelebA | | Waterbirds | |\n|--------|--------|-------|-----------|-------|\n| **Method** | **Avg.** | **Worst** | **Avg.** | **Worst** |\n| MEDRO | 92.2 (±0.8) | 89.4 (±1.2) | 94.2 (±1.8) | 86.3 (±1.9) |\n| MEDRO (w/ gating) | 92.3 (±0.5) | 90.1 (±0.8) | 92.6 (±1.7) | 88.7 (±2.2) |\n\n| | Camelyon17 | | iWildCam | |\n|--------|--------|-------|-----------|-------|\n| **Method** | **Avg.** | **Worst** | **Avg.** | **Worst** |\n| MEDRO | 92.6 (±0.5) | 87.8 (±1.9) | 34.1 (±0.7) | 31.5 (±0.8) |\n| MEDRO (w/ gating) | 92.7 (±0.4) | 87.8 (±2.0) | 36.4 (±1.1) | 31.6 (±0.8) |\n\n**PovertyMap**\n| Method | Avg. Val | Avg. Test | Worst Val | Worst Test |\n|--------|-------------|--------------|-----------------|------------------|\n| MEDRO | 0.80 (±0.04) | 0.80 (±0.03) | 0.50 (±0.06) | 0.49 (±0.03) |\n| MEDRO (w/ gating)| 0.81 (±0.04) | 0.80 (±0.04) | 0.53 (±0.06) | 0.49 (±0.05) |\n\nAs shown, gating provides modest improvements for subpopulation shift tasks but limited benefits for domain generalization. We are grateful for this suggestion and will include a comprehensive table in the appendix showing both inference strategies across all experiments.\n\n### `Question 3:`\n>\"Why are the ERM's performance on CelebA and Waterbirds significantly different between Table 2 and 3?\"\n\nWe believe the reviewer is referring to the difference between Table 1 and 2 (not Table 2 and 3). The performance difference in ERM stems from different experimental protocols: \n\n- **Table 1** uses fixed hyperparameters from the GroupDRO paper\n- **Table 2** uses SubpopBench's hyperparameter search protocol, which searches over predefined ranges and selects the best configuration\n\nAs shown in Table 6 in Appendix E, the hyperparameter search in SubpopBench allows ERM to find more optimal configurations for each dataset, explaining the performance improvement.\n\n### `Question 4:`\n>\"About Figure 2, maybe aligning the y-axes of the two plots, so that the impacts on average and worst-case performance can be compared easily?\"\n\nWe thank the reviewer for this valuable suggestion. Following your recommendation, we aligned the y-axes of both plots (40-100%), which indeed makes the comparison between average and worst-group performance much clearer. We will include this improved figure in the revised manuscript."}]

Author rebuttal:

Meta-review: The authors present a robust optimization framework for handling distribution shift across environments. They extend GroupDRO by expanding the uncertainty set to all expert-environment pairs with connections to domain generalization through label-conditional invariance. 

Strengths: The work has both a novel theoretical component that yields a min-max objective over all expert-environment pairings and extensive empirical results on diverse settings, including small, controlled environments to large, real world, diverse benchmarks. Reviewers found the paper well written and clear.

Weaknesses: There were concerns that the method itself was not very novel, and the increased computational overhead of optimizing over all possible environment pairings. The empirical results also show that the method is on par with baselines rather than being significantly better. Finally, there are missing experiments to determine what the role of various components are, and to gain a better understanding of hyperparameters.

Reasons for decision: Overall, I found the strengths of this work to outweigh the weaknesses, especially given the authors' promise to investigate more challenging benchmarks for the camera ready.

Rebuttal summary: Reviewers generally appreciated the intuition and theory behind the work, and find MEDRO to be an additional useful tool for handling distribution shift. There were some concerns about the method, missing ablations, and additional environments for evaluation, which were mostly addressed in the rebuttal phase. Only one reviewer Vdg5 had more serious reservations and voted to reject the paper, but did not respond sufficiently to the rebuttal response and I believe the authors addressed those concerns satisfactorily.
Decision: Accept (poster)