Structure-informed Risk Minimization for Robust Ensemble Learning
Authors: Fengchun Qiao
Yanlin Chen
Xi Peng
Venue: ICML 2025 poster (2025)
Keywords: Out-of-distribution Generalization
Primary area: deep_learning->robustness
TL;DR: We leverage distributional graph to learn robust ensemble weights for out-of-distribution generalization.

Abstract: Ensemble learning is a powerful approach for improving generalization under distribution shifts, but its effectiveness heavily depends on how individual models are combined. Existing methods often optimize ensemble weights based on validation data, which may not represent unseen test distributions, leading to suboptimal performance in out-of-distribution (OoD) settings. Inspired by Distributionally Robust Optimization (DRO), we propose Structure-informed Risk Minimization (SRM), a principled framework that learns robust ensemble weights without access to test data. Unlike standard DRO, which defines uncertainty sets based on divergence metrics alone, SRM incorporates structural information of training distributions, ensuring that the uncertainty set aligns with plausible real-world shifts. This approach mitigates the over-pessimism of traditional worst-case optimization while maintaining robustness. We introduce a computationally efficient optimization algorithm with theoretical guarantees and demonstrate that SRM achieves superior OoD generalization compared to existing ensemble combination strategies across diverse benchmarks. Code is available at: https://github.com/deep-real/SRM.

Structure-informed
Risk Minimization for Robust Ensemble Learning
Fengchun Qiao
Yanlin Chen
Xi Peng'
Abstract
et al., 2022; Ramé al., 2023), leading more reliable predictions under distribution shifts.
Ensemble learning is a powerful approach for im- proving generalization under distribution shifts, but its effectiveness heavily depends on how in- dividual models are combined. Existing methods often optimize ensemble weights based on vali- dation data, which may not represent unseen test distributions, leading to suboptimal performance in diittii-ittiittiionn (OoD) settings. Inspired by Distributionally Robust Optimization (DRO), we propose Structure-infomme Risk Minimiza- tion (SRM), a principled framework that learns robust ensemble weights without access to test data. Unlike standard DRO, which defines un- certainty sets based divergence metrics alone, SRM incorporates structural information of train- ing distributions, ensuring that the uncertainty set aligns with plausible real-world shifts. This approach mitigates the over-pessimism of tradi- tional worst-case optimization while maintaining robustness. We introduce computationally ef- ficient optimization algorithm with theoretical guarantees and demonstrate that SRM achieves superior OoD generalization compared to exist- ing ensemble combination strategies across di- verse henchmarks. Code available at: https: //github. com/ deep-reall SSRM.
The success of ensemble learning hinges on two key steps: (1) training diverse individual models and (2) optimally combining them into a final predictor. While significant advances have been made ensemble training to produce diverse and complementary models (Benoit et al., 2024; Pagliardini et al., 2023; Rubinstein et al., 2024), poor com- bination strategies can severely degrade performance (Rame et al., 2022). This underscores the importance of determin- ing optimal model combinations. Existing methods fall into four main categories: selection/pruning (He et al., 2024), voting (Tsoumakas et al., 2008), weighted voting (Wu et al., 2023). and stacking (Chatzimparmpas approaches typically optimize ensemble weights based on validation data, either from training split holdout set. However, due to distribution shifts, such learned combi- nations may underperform on unseen test distributions. A recent work (Qiao & Peng, 2024) attempted to mitigate this issue by explicitly promoting prediction diversity on test data. However, test data are typically unavailable in OoD generalization. This raises a fundamental question: Can we learn ensemble weights that effectively generalize unseen test distributions?
al., 2020). These
Distributionally Robust Optimization (DRO) (Shalev- Shwartz & Wexler, 2016), originally designed for training individual models under distribution shifts, offers a promis- ing approach learning robust ensemble weights. DRO optimizes the worst-case risk over an uncertainty set of po- tential test distributions, typically defined as divergence ball around the training distribution using distance metries like f-divergence (Namkoong & Duchi, 2016), Wasserstein distance (Shafieezadeh Abadeh et al., 2018) or Maximum Mean Discrepancy (Staib & Jegelka, 2019). While DRO enhances robustness, it often leads to overly pessimistic solu- tions (Hu al., 2018; Frogner et al., 2021: Dai et al., 2023). degrading average performance by considering implausible distributions that rarely occur in practice. This issue arises because DRO treats distributions in the uncertainty set in- dependently, ignoring the intrinsic structure governing their real-world relationships.
1. Introduction
Ensemble learning has emerged as promising approach for nit-o--distribution (OoD) generalization (Lee et al., 2023; Pagliardini et al., 2023), demonstrating superior robustness and adaptability compared to individual models when de- ployed in environments that differ from their training con- ditions. By combining multiple diverse models, ensemble methods can capture different aspects of the data distribution and reduce the impact of individual model biases (Wortsman
'DeepREAL Lab, Department of Computer and Information Sciences, University Delaware, DE, USA. Correspondence to: Xi Peng uud...eeegue....uu...... ude..
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
To overcome this limitation, we propose Structure-inforen informed Risk Minimization (SRM), principled framework that

---

Structure-informed Risk Minimization for Robust Ensemble Learning
leverages distributional graphs optimize ensemble weights for OoD generalization. Our key insight is to incor- porate structural relationships between distributions into the uncertainty set, capturing how distributions naturally vary and influence each other. By focusing on realistic distribu- tion shifts rather than arbitrary worst-case scenarios, SRM retains the robustness benefits of DRO while mitigating its excessive conservatism.
However, ERM assumes that the training and test distribu- tions are identical, which rarely holds in practice. Conse- quently, ERM-weighted ensembles may reinforce spurious correlations that do not generalize (Arjovsky et al., 2019).
2.3. Worst-case Risk Minimization
Distributionally Robust Optimization (DRO) (Shalev- Shwartz & Wexler, 2016) offers an alternative by minimiz- ing the worvt-case risk over an uncertainty set (Pimnn)::
Our main contributions are: (1) A principled framework for learning ensemble weights under distribution shifts, bridg- ing critical gap in ensemble learning for OoD general- ization. (2) A novel method for constructing uncertainty sets that integrate distributional relationships, providing a more realistic approximation of potential test distributions. (3) A computationally efficient optimization algorithm that balances worst-case robustness with average performance, supported by theoretical guarantees.
min max Rp(w). W (E
(4)
The uncertainty set is often defined as divergence ball centered at Prain:
u(Puran)) = (P: D(P, Pirain) S
(5)
where controls the set size, and D(-) denotes a discrep- ancy measure between probability distributions, such as f-divergence (Namkoong & Duchi, 2016), Wasserstein dis- tance (Shafieezadeh Abadeh al., 2018; Qiao & Peng, 2021), and other statistical distance measures.
2. Background
2.1. Problem Formulation
Let (X,Y) Y) be random variables where instances x E X C Rd and labels y E y follow an unknown joint distribution P(X,Y) Y). We observe data under different environmental conditions where samples are drawn from a set of distribu- tions Eall such that (ar, y)- Pa(X, Y). Given m pre-trained models F = {1. .... } trained on Euain ?. Eau, where each model fi : + )/ maps inputs to predictions, our ob- jective is to construct an optimal ensemble that generalizes to unseen test distributions Eiesa = Ean't EErain-
While DRO enhances robustness, it often produces overly conservative solutions (Frogner et al., 2021; Sagawa et al., 2019) by treating all distributions within distance as cqually likely, regardless of their practical relevance. This limitation arises because DRO considers each potential test distribution independently, ignoring the underlying structure that governs real-world distributional shifts.
3. Structure-informed Risk Minimization
We achieve this by learning ensemble weights w A m (the probability simplex) to form the predictor:
We propose Structure-infore Risk Minimization (SRM), principled framework that leverages distributional graphs to optimize ensemble weights for -ut-o--distribution gen- eralization. The key innovation lies in incorporating the inherent relationships between distributions into uncertainty set construction, capturing how distributions naturally vary and influence each other. Using this structural information, SRM guides the uncertainty set to focus on distributions likely to occur under real-world conditions, rather than arbi- trary shifts. This maintains the robustness benefits of DRO while mitigating its tendency toward excessive pessimism. SRM remains computationally tractable through carefully formulated optimization problem that balances worst-case robustness with average performance.
Sw Σ wr)r(1).
(1)
For a distribution Per the expected risk is defined as: Rr. (w) w)) (x)) [( where l is the loss
function. During deployment, the ensemble encounters test distributions from Eust that are distinct from Euain- The goal is to minimize the worst-case risk across Exes:
min sup Rr. (w)-
TE eEE.
The key challenge lies in determining the optimal ensem- ble weights w using only training distributions Exain while ensuring generalization to unseen test distributions Exest
3.1. Structure-informed Uncertainty Sets
2.2. Average Risk Minimization
Let g (V,A) be weighted graph, where the set of vertices V corresponds to the n training distributions in Epain = {P...., P.). The edges between distributions are represented by the adjacency matrix A E R=*", where each entry quantifies the similarity between distributions.
A natural approach is Empirical Risk Minimization (ERM) (Vapnik& & Vapnik, 1998), which assigns ensemble weights minimize the average risk on P
min R (w) = min ((xr,)) Pum ( (x)) w)]. (3)
2

---

Structure-infommed Risk Minimization for Robust Ensemble Learning
Specifically, the edge weight between distributions Pj and P3 is defined Aij D(P:, Pj), where (--) is distributional distance metric (e.g.. Wasserstein distance).
3.2. Optimization Framework
Given the uncertainty set Q, we formulate the ensemble weight optimization constrained minimax problem:
To incorporate this structure into ensemble learning, we use graph centrality measures (e.g.. closeness centrality) to refine the construction of the uncertainty set robust optimization. Central distributions serve as "anchors" that better represent the structure of the training domain, guiding the ensemble weighting process. By considering central- ity, we ensure that the uncertainty set focuses on plausible test distributions that are well-supported by training data, rather than arbitrarily distant shifts. This helps mitigate over-pessimism in worst-case optimization while improving generalization to unseen distributions.
min max
Σ
Re (w) s.t. D(qll)) S T,
(8)
WEA mh QEa
where Rp, (w) E(x.,)~,,, (te.p)=,[[(([((( (Z)) v)] denotes the ex- pected risk on distribution P.. To address computational challenges in Eq. we use the Karush- Kathh-KKun----kke (KKT)
conditions (Boyd, 2004) derive a Lagrangian reformula- tion, where the constraint replaced with penalty term:
Σ le R... (w) XD(g))))
min Max WEA th AEA
Definition 3.1 (Centrality Prior). Given a graph g = (V, A), we define the prior p E A as:
where >> controls the trade-off between risk minimiza- tion and structural consistency. We solve this optimization problem using an alternating gradient algorithm: (1) Updat- ing ensemble weights w via:
dPP)) Pe Ei=1 (P.))
e n..
(6)
w++ PAm PPmm(w' Thw V C(w',' ,')).
(10)
Here e Pe R maps each distribution to non-negative value measuring its centrality in G.
(2) Updating mixture weights q via:
q ++ = =PAn (q' + V Vaa(w' d')).
(11)
The uncertainty set is then defined as a mixture of training distributions constrained by the prior p:
Here L(w, q) = Σn= le R. (w) - AD T/ (7a) are step sizes, and Pam (PAn) denote projection (Duchi et al., 2008) onto the probability simplex.
By assigning higher weights to distributions with both high empirical risk and high centrality, our approach bridges the gap between overly conservative worst-case approaches and the fragility of average risk minimization under distribution shifts.
=
Σ Pe A", STJ. ST
(7)
where D.)) denotes the distance between two distributions (e.g., by distance or KL divergence), and 7 controls the al- lowable deviation from the structural prior. This ensures the uncertainty set focuses on plausible mixtures aligned with the structural relationships among training distributions.
Implementation Details. (1) Distribution distance D(-)- In contrast to traditional graph learning methods (Jin et al., 2020; Dong et al., 2019) which primarily focus point- wise similarities, our approach models correlation on a distribution-wise level. While the Wasserstein distance is a natural choice for measuring distributional similari- ties, its standard computation has cubic complexity, pos- ing practical challenges in large-scale applications. To ad- dress this, we adopt computationally efficient Gaussian- based approximation of the 2-Wasserstein distance. For two Gaussian distributions P: NN Σj) and P3 N(H,, Σg),, the 2- 2-Wasserstein distance is: (P,,,, PJ) || - 11,12 tr(,, + Σ 2(>: ////,,,,/ Σ, /////////////// When the covariance matrices Σ, and Σ, commute, this simplifies to: D2, Pz) || 11 + || / This approximation maintains the key geometric properties of the Wasserstein distance while reducing computational com- plexity from cubic to quadratic time.
Alternative Priors. We consider two alternatives our centrality-based prior: (1) Uniform prior, where Pe n for all distributions, ignoring the graph structure. (2) Graph Laplacian-batt based prior, which uses the constraint q LL < т. where L = D A is the graph Lapla- cian, with D = diag( (A1) being the degree matrix. Here, Ag exp(-D22 (P,, P2) measures the similarity be- tween distributions Pt and Py. Although the Laplacian constraint encourages smooth variations over the graph, it focuses on local smoothness rather than global influence. In contrast, our centrality-bas based prior directly measures global proximity, better aligning with our goal of identifying dis- tributions informative about unseen test conditions. The empirical results in Sec. 4 validate this choice.
3

---

Structure-informed Risk Minimization for Robust Ensemble Learning
Algorithm I Struucture-iformeed Risk Minimization (SRM)
ting oo), SRM interpolates between these extremes, leveraging the distributional graph achieve balance he- tween average performance and worst-case robustness.
Input: Data of Enain- Step sizes l/w and 7/
Output: Learned ensemble weights w
I/ Construct graph 9 and compute prior p
for j,j E {1, (l,...,TE n do
4. Experiments
D(P,, < || - myl12 + + 12
Ai c D(P:. P2)
We evaluate SRM on two common OoD generalization benchmarks, DomainBed (Gulrajani & Lopez-Paz, 2020) and WILDS (Koh et al., 2021). Following the standard prac- tice, we use held-out validation set from training distribu- tions on DomainBed benchmark and validation distributions on WILDS benchmark for model selection. We provide im- plementation details and additional results in the Appendix. We provide the source code in the supplementary material.
end
c(Po) + شー= d(Pa: P,)1- 1/ Closeness centrality Pe + dP.)/ 2j=10 oP;) // Prior distribution
// Optimize weights
Initialize w - 1 1. q + 1
m while not converged do
Calculate L(w, q) via Eq. 9
Update ensemble weights w via Eq. 10
Baselines. We compare SRM with the following methods: (I) Uniform Ensemble; (2) Greedy Selection; (3) Empiri- cal Risk Minimization (ERM) (Vapnik & Vapnik, 1998);
Update mixture weights q++1 via Eq.
end
(4) Uniform Prior; (5) Laplacian Prior; (6) Group Distribu- tionally Robust Optimization (DRO) (Sagawa et al., 2019). These methods can be grouped into two categories: (1) Non- optimization-based where the ensemble weight is obtained without the need for optimization (Uniform Ensemble and Greedy Selection); (2) Optimization-basee based, where the en- semble weight is learned through an optimization process (ERM, Uniform Prior, Laplacian Prior and DRO).
(2) Centrality prior p. We employ closeness centrality to compute the prior as it effectively captures distribution's 's global influence across the entire graph, rather than just considering immediate neighbors. Specifically, for each distribution Pe. its closeness centrality is computed as: where d(Pe. v) denotes the shortest path distance in the G. The prior is then normal- ized as Pe (Eq. 6). This formulation aligns well with our goal of identifying distributions that are most representative of the overall graph, as distributions with high closeness centrality are likely to share characteristics with a broader range of unseen test distributions, making them particularly informative for constructing robust uncertainty sets.
c(P2)= = -Ev d(Pe.
4.1. DomainBed Benchmark
Datasets. We conduct experiments five datasets: Ter- ralncognita (Beery et al., 201 8), VLCS (Fang et al., 2013), Officellome (Venkateswara et al., 2017), PACS (Li et al., 2017), and DomainNer (Peng et al., 2019). PACS con- sists of images from four different distributions: art, car- toons, photos, and sketches. It contains a total of 9,991 im- ages with dimensions of (3, 224, 224) pixels and classes. VLCS contains photographic images from four distributions: Caltech101, LabelMe, SUN09, and VOC2007. There are 10,729 total images with dimensions of (3, 224, 224) pix- els across 5 classes OfficeHome is made up of images from four distributions: art, clipart, product images, and real-world photos. There are 15,588 images in this dataset with dimensions of (3, 224, 224) pixels and 65 classes. Ter- raIncognita consists of photos of wild animals captured by camera traps at four different locations. The dataset con- tains 24,788 images of size (3, 224. 224) pixels from 10 different classes. DomainNet is a large-scale dataset with images from six distributions: clipart, infographics, paint- ings, quickdraw sketches, real-world photos, and sketches. In total, there are 586,575 images of dimension (3, 224, 224) pixels across 345 classes. For each dataset, we hold one distribution out for test and train on the remaining ones, and report the average accuracies over all test distributions. Fol- lowing (Gulrajani & Lopez-Paz, 2020), all the experimental results are averaged over 3 trials.
Discussion. (1) Connection between T (Eq. 7) and p (Eq. 5). While DRO considers the uncertainty set defined as a divergence ball with radius of p. our method oper- ates on mixtures of training distributions. These formu- lations are closely connected: when the mixture weights & satisfy D(allp)= S T, the resulting mixture distribution P. E.=1 Pe lies within a divergence ball centered at the reference distribution P₂- Σ" Pa. This provides computationally tractable way to approximate divergence ball constraints while leveraging the structure induced by the distributional graph via centrality prior p.
(2) Connection to existing methods. SRM provides a unified framework that encompasses existing approaches as special cases through the constraint D(qllp) S T. (i) When and Pa- 1. the constraint forces q = p. reducing to stan-
n.
dard ERM that equally weights all training distributions. (ii) With 7 = 0 and centrality prior p. we recover weighted risk minimization that prioritizes influential distributions. (iii)
When T = co, the constraint becomes inactive, allowing q to be arbitrary, which recovers Group DRO (Sagawa et al., 2019) that optimizes for the worst-case distribution. By set-
4

---

Structure-infommed Risk Minimization for Robust Ensemble Learning
Non-optinization-
Optimization-bat
Dataset
Uniform
Greedy
ERM
Uniform Prior
Laplacian
DRO
SRM
PACS
87.8
87.1
86.9
87.0
87.0
87.1
87.1
VLCS
79.5
79.6
79.6
79.8
78.7
79.8
79.9
OfficeHome
70.8
70.8
70.7
70.7
70.8
70,5
71.0
DomainNet
43.3
44.6
45.0
45.0
45.0
45.0
45.1
Teralncognita
48.5
48.1
48.8
48.8
48.8
49.0
49.5
Average
66.0
66.0
66.2
66.2
66.1
66.2
66.5
Table Average accuracies (%) over test distributions across datasets on DomainBed benchmark. We bold the best results.
Non-optimization-ban
Optimization-based
Test Distributions
Uniform
Greedy
ERM
Uniform Prior
Laplacian
DRO
SRM
Before 2004
53.7
53.3
54.4
54.1
54.4
53.2
54.4
2009-2011
62.3
62.3
62.3
62.4
62.3
62.3
62.5
After 2016
37.6
37.5
37.7
37.7
37.8
37.0
37.8
Average
51.2
51.0
51.4
51.4
51.5
50.8
51.6
Table Worst-region accuracy (%) ) the FMOW-WILDS dataset. We evaluate the algorithms under three different train-test split
schemes. SRM consistently outperforms other baselines in both Distribution Interpolation and Distribution Extrapolation settings.
Results. We report the results on DomainBed in Tab. 1. SRM achieves state-of-the-art performance on four datasets (VLCS, OfficeHome, DomainNet, and Terralncognita), out- performing both non-optimization based and optimization- based baselines. Notably, while other optimization based methods deliver inconsistent results (e.g., DRO outperforms ERM and Uniform Prior PACS and TerraIncognita, but inferior to them on OfficeHome), SRM exhibits consistent improvements over prior optimization- based methods, with particularly significant gains on Terralncognita (49.54% vs DRO's 48.965. 96%). This indicates SRM achieves improved generalization across diverse distributions by taking distri- butional relations into account.
and Distribution Extrapolation cases proposed by (Wang et al., 2020), providing more thorough comparison of base- lines than single train-test split scheme.
Results. Tab. 2 presents the worst-region accuracy for dif- ferent methods. We observe that SRM consistently outper- forms existing approaches across all three train-test settings. Notably, SRM improves worst-region accuracy compared to DRO, demonstrating its ability to mitigate over over-pessimism while maintaining robustness. In the Test Before 2004 set- ting, SRM achieves 54.44%, outperforming DRO (53.21 %)) and standard ERM-based methods. In the Test Middle (2009-201 setting, SRM attains 62.46%, slightly outper- forming DRO and achieving the best generalization. In the Test After 2016 setting, SRM improves worst-region accu- racy to 37.81% % surpassing DRO (36.98%), highlighting its advantage in handling temporal distribution shifts.
4.2. WILDS Benchmark
Dataset. We evaluate SRM on W-WLD (Koh al., 2021) dataset, which comprises satellite images collected from different geographical regions across five continents at different time. We study temporal distribution shift, where distribution represents the year the image was taken. Apart from the original train-test split scheme (Test After 2016), where training distributions consist years 2002 to 2013, test distributions consists of years 2016 and 2017, and years 2013 2016 are reserved for validation. we further propose two train-test split schemes which cover more diverse distri- bution shift scenarios: (1) Test Before 2004, where years 2007 to 2018 are for training, 2002 to 2004 are for testing, 2004 to 2007 are for validation; (2) Test Middle, where years 2002 to 2008 and years 2012 to 2018 are for training,
Graph Visualization. Fig. visualizes the learned distribu- tion graphs under different train-test split settings. Unlike DRO, which focuses on distributions with the worst test ac- curacy (even if they are far from the training distributions), SRM assigns higher weights to influential distributions with high centrality in the training graph. This structure-awan weighting strategy improves worst-region accuracy while avoiding excessive pessimism.
Robustness to Severe Distribution Shift. To test SRM's robustness severe distribution shift, simulate different level of distribution shift by adding corruptions to test data with different severity (Hendryeks & Dietterich, 2019). We apply two types of corruptions: Blur and Digital. The results under the setting of Test After 2016 are reported in Fig. 2. As
2009-2011 are for testing, 2008 and 2011 are for validation.
These three settings cover both Distribution Interpolation
5

---

Structure-informed Risk Minimization for Robust Ensemble Learning
0 (2002)
(2007)
(2
15 (2017)
Distribution ID (Year):
Centrality
Worst-case Distribution
Worst-case Distribution
Worst-case Distribution
Test
Val
Train
Train
Val
Test Val
Train
Train
Val
Test
0 2
15
0
5
8 9
15
10
13 14 15
(a) Test Before 2004
(b) Test Midile
(c) TestAAfer 2016
Figure 1. Visualization distributional graph G under different train-test split schemes on FMoW-WILDS dataset. Distributions from year 2002 to year 2017 are labeled by 0 to 15, respectively. The thickness of edges indicates the similarity between distributions. While DRO focus on worst-case distributions (far from other distributions), SRM assigns higher mixture weights (q) to influential distributions.
Aec. (N
DRO
Arc (5)
Ours DRO
Test Distribution
min
Individual Learners
max
mean
std
Uniform
Ensemble
SRM
Clipart
52.74
59.51
56,79
2.39
58.75
63.02
Infograph
18.19
20.12
18.98
0.61
21,30
21.77
Puinting
44.99
48.36
46.57
1.09
51.07
51.92
Quickdraw
10.98
13.51
12.25
0.75
14.11
14.96
Real-world
57.31
61.30
59.67
1.09
62,20
64.51
Comuption Severty
Computiion Severity
Sketch
46.76
51.12
49.11
1.21
$2.31
54.66
(a) Blur
(b) Digital
Table 3. Accuracy (%) of individual learners and ensemble meth- ods on DomainNet dataset.
Figure Worst-region accuracy (%) on FMOW-WILDS dataset under different severity of corruptions test data. Left: results of Blur corruption. Right: results of Digital corruption. SRM consistently outperforms DRO under different types and different severity of corruptions.
gains.
Regularization Strength X. To assess the role of X, which controls the trade-off between risk minimization and struc- tural consistency, we vary E [0.0, 2.0] and report the worst-region accuracy in Fig. 3. When À = 0, SRM reduces to Group DRO, which focuses purely on worst-case risk minimization without leveraging the structure of training distributions. As À increases, the worst-region accuracy improves, indicating that incorporating topological informa- tion helps refine the uncertainty set. However, overly large values of À (e.g., À >> 1.5) degrade performance, suggesting that excessive reliance structural consistency may overly restrict the uncertainty set. Overall, moderate À value (around 1.0) provides the best trade-off between robustness and average-case performance.
observed, under different types and severity of corruptions, SRM consistently outperforms the DRO baseline, demon- strating the superior robustness to severe distribution shift of SRM. However, the performance gap between SRM and DRO shrinking as the severity increases. This may at- tribute to SRM tries construct realistic uncertainty set while DRO focus on the worst-case distribution.
4.3. Ablation Study
We conduct the following ablation study under the Test After 2016 setting to investigate how each component affects the effectiveness of SRM.
Comparison with Individual Learners. To further isolate the benefit of structure-informed weighting, we compare SRM with both individual models and uniform ensemble baseline. Tab. reports results on the DomainNet dataset across six test domains. SRM significantly outperforms both the average individual performance and uniform ensemble. For example, on the "Clipart" domain, accuracy improves from 56.79% (mean of individual models) to 58.75% (uni- form ensemble) and then to 63.02% with SRM, showing that leveraging structural relationships provides substantial
Number of Models in the Ensemble Pool. We analyze the effect of varying the number of models in the ensem- ble pool from to 10. Fig. presents the worst-region accuracy across different train-test splits. More models gen- erally improve performance, as larger ensemble provides greater flexibility. However, diminishing returns are ob- served beyond 7-8 models, indicating that excessive models may introduce redundancy.
Distance Metric for Distribution Graph g. We evaluate
6

---

Structure-informed Risk Minimization for Robust Ensemble Learning
Vorst-eegion Act. 1960
Worsto
(a) Test Before 2004
(h) Test Middle
(c) Test After 2016
Figure Ablation study of the regularization strength À (Eq. 9) on FMoW-WILDS dataset. Note that when Group DRO. The worst-region accuracy first increases then decreases as À gets larger in Test Middle and Test After 2016. In Test Before 2004, the performance does not degrade because the worst-case distribution is from other distributions (Fig. ((a),, thereby leveraging structural prior consistently brings performance gain.
SRM reduces
Worst-region Acc. 195)
Worce tegion Arc.
corsttrogion
10c
(a) Test Before 2004
(b) Test Middle
(c) Test After 2016
Figure 4. Ablation study of number of models (m) in the ensemble pool on FMoW-W WWWDD datasct. More models generally improves performance, while in some cases (e.g., Test Before 2004) excessive models roduce efficiency due model redundancy.
Distance
Worst-region Ace. (%)
Running Time (s)
on shortest paths rather than overall structure. Despite these, the performance gap between best and worst centrality is not obvious (0.07%), suggesting SRM is not sensitive the choice of centrality.
Wasserstein 38.1
28.1
Diffusion EMD
38.0
2.6
EMD
38.1
516.4
Graph Sparsity. In our experiments, we use complete graph by default to calculate centrality. However, we also in- vestigate how the graph sparsity (i.e. the percentage of edges retained the distribution graph with respect to complete graph) influences performance by pruning edges between distant distributions. Fig. shows that moderate sparsity (40-70 % edges) yields the best results. Excessive sparsity (below 30 % edges) degrades accuracy most structural information is missing. These results suggest that balancing connectivity is crucial, as overly dense or sparse graphs lead to suboptimal uncertainty sets.
Table Ablation study of distributional distance metrics for distri- bution graph G. Our choice of 2- Wasserstein distance strikes the hest balance between performance and time complexity.
three different metrics for computing pairwise distributional distances the training graph G: 2- Wasserstein Distance, Diffusion EMD (Diffusion Earth Mover's Distance (Tong al., 2021)), and EMD (Standard Earth Mover's Distance). Tab. 4 reports worst-region accuracy for each method. Con- sidering both performance and computational complexity, we choose 2- Wassserstei Distance for constructing the dis- tribution graph.
5. Related Work
Ensemble learning has emerged as powerful paradigm for enhancing model robustness and improving out-of- distribution performance (Pagliardini et al., 2023; Lee et al., 2023). The fundamental principle underlying ensemble effectiveness lies in the strategic combination of diverse pre- dictors, where diversity serves as crucial factor reducing ensemble prediction error (Ueda & Nakano, 1996). This di- versity can manifest in various forms, including architectural
Comparison of Different Centrality Metrics. To assess the role of graph centrality in defining the uncertainty set, we compare different centrality measures in Fig. 5. Closeness centrality achieves the highest worst-region accuracy, rein- forcing our choice in the main method. Laplacian centrality and Katz centrality perform slightly worse, suggesting that they are less effective in capturing global influence. Be- tweenness centrality underperforms, likely due to its focus

---

Structure-informed Risk Minimization for Robust Ensemble Learning
37.02
do not guarantee diverse predictive behaviors. Consequently, such methods may inadvertently generate redundant models that exhibit similar decision boundaries, potentially com- promising ensemble performance when combined through uniform weighting schemes.
Worst-region Acc. (%)
The challenge of effective model selection within ensem- ble frameworks represents another critical consideration. prevalent practice in the field is greedy selection (Wortsman et al., 2022), where individual models are sequentially in- corporated into the ensemble based their contribution to validation accuracy improvements. This validation- centric approach, while computationally efficient and theoretically motivated, suffers from fundamental limitation: validation- based selection criteria may not adequately generalize to shifted test distributions that differ significantly from the validation set. This generalization gap becomes particularly pronounced in scenarios involving substantial domain shifts or distributional changes.
12:72
Betweenness
Faiz
Lapiacian
Closeness
Harmanic
Eigenvector
Figure 5. Ablation study of graph centrality. Performance gap between different choices of centrality is negligible.
37.22
Worst-region (96)
37.82
31.0
Distributionally Robust Optimization (DRO) (Shalev- Shwartz Wexler, 2016) provides a principled mathemati- cal framework for addressing distribution shifts by optimiz- ing worst-case performance over a carefully constructed set of potential test distributions. Traditional DRO formulations define uncertainty sets using various divergence metrics, in- cluding f f-divergence (Namkoong & Duchi, 2016), Wasser- stein distance (Shafieezadeh Abadeh et al., 2018; Qiao et al., 2020), and other statistical distance measures. These uncer- tainty sets characterize the space plausible distribution shifts that the model might encounter during deployment.
37.22
37.72
37.62
100
Graph Sparsity
Figure 6. Ablation study of graph sparsity. Note "30" denotes 70% edges are pruned from the complete graph.
Despite the theoretical elegance and empirical success of DRO in training individual robust models, the framework suffers from well-documented over-pessimistic issue. This over over-pessimism manifests as excessive conservatism the worst-case optimization objective, often leading to subop- timal performance on the actual test distribution. Several methods (Liu et al., 2022; Qiao & Peng, 2023; Ma al., 2024; Huang & Ding, 2025) have been proposed to mitigate this over wver-pessimistic behavior. However, these existing so- lutions predominantly focus on training individual models rather than learning robust ensemble combinations, leaving a significant gap in the ensemble learning literature.
differences, training procedures, or learned representations that capture complementary aspects of the underlying data distribution.
Contemporary approaches to ensemble diversity optimiza- tion can be broadly categorized into two main strategies. The first category encompasses methods that directly opti- mize prediction diversity on target data (Pagliardini et al., 2023; Lee et al., 2023). These approaches explicitly encour- age disagreement among ensemble members on specific data points, thereby promoting complementary decision bound- aries. However, such methods inherently require access to test data during the optimization process, which presents a significant limitation in practical deployment scenarios where test distributions remain unknown at training time. The second category promotes diversity through the adop- tion of varied learning procedures (Arpit et al.. 2022; Rame al., 2022; Wortsman et al., 2022). These methods gener- ate ensemble diversity by systematically varying training configurations, including hyperparameter settings, optimiza- tion algorithms, data augmentation strategies, initialization schemes, and architectural choices. While this approach of- fers the advantage of not requiring test data access, it faces the fundamental challenge that diverse training procedures
6. Conclusion
We proposed innfouer-- Risk Minimization (SRM), principled framework for OoD generalization in ensem- ble learning. The innovation lies in utilizing distributional graphs to construct uncertainty sets that focus on plausi- ble distribution shifts. By incorporating the relationships between training distributions, SRM achieves a balance be- tween robustness unseen shifts and strong average-case performance. Our theoretical analysis establishes guaran-
8

---

Structure-informed Risk Minimization for Robust Ensemble Learning
tees for both convergence and generalization, while exten- sive experiments across diverse benchmarks demonstrate SRM's effectiveness in real-world scenarios.
2019. URL https://ieeexplore ieee org/ document/880066
Duchi, J., Shalev-Shwarzz. S., Singer, Y., and Chandra, T.
Efficient projections onto the 1-ball for learning in high dimensions. In Proceedings of the 25th international conference Machine learning, pp. 272-279, 2008.
Acknowledgments
This work is supported by the National Science Foundation under grant numbers CAREER 2340074, SLES 2416937, III CORE 2412675, National Institutes of Health under grant number R21 R21CA330,,, Department of Defense under grant number AFOSR FA9550-23-1-44 13-1-4944
Fang, C., Xu, Y., and Rockmore, D. N. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1657- 1664, 2013.
Impact Statement
Frogner, C.,, Claici, S., Chien, E., and Solomon, In- corporating unlabeled data into distributionally robust learning. Journal of Machine Learning Research, 22(56):
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
1-46, 2021.
Gulrajani, I. and Lopez-Paz, D. In search of lost domain generalization. International Conference on Learning Representations, 2020.
References
Arjovsky, M., Bottou, L., Gulrajani, I. and Lopez- Paz, D. Invariant risk minimization. arXiv preprint arXiv: 1907:00907.02933, 2019.
He, K., Zhang, X., Ren, S., and Sun, J. Deep Residual Learning for Image Recognition. In JEEE Conference on Computer Vision and Puttern Recognition, pp. 770-778, 2016.
Arpit, D., Wang, H., Zhou, Y., and Xiong, C. Ensemble of averages: Improving model selection and boosting per- formance in domain generalization. Advances in Neural Information Processing Systems, 35:8265- 3588265-8277,, 2022.
He, Y--.., Wu, Y.-C., Qian, C., and Zhou, Z.-H. Mar- gin distribution and structural diversity guided ensemble pruning. Machine Learning, pp. 1-23, 2024.
Beery, S., Van Hom, G., and Perona, P. Recognition in terra incognita. In Proceedings of the European conference on computer vision, pp. 473,473, 2018.
Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and pertur- bations. Proceedings of the International Conference on Learning Representations, 2019.
Benoit, H. L., Jiang, L., Atanov, A., Kar, O. F., Rigotti, M., and Zamir, A. Unraveling the key components of OOD generalization via diversification. In The Twelfth International Conference on Learning Representations, 2024. URL http::::////ee/eeviieeeee wwtpriiwwweeeeeeeeeeeeeeeeeeeeee n forum? id-Lvf 77Garu..
Hu, W., Niu, G., Sato, I., and Sugiyama, M. Does distribu- tionally robust supervised learning give robust classifiers? In International Conference on Machine Learning, pp. 2029-200 2239-003.. PMLR, 2018.
Huang, J. and Ding, H. An effective manifold based opti- mization method for distributionally robust classification. In The Thirleenth International Conference on Learning Represemations, 2025.
Boyd, S. Convex optimization. Cambridge UP, 2004.
Chatzimparmpas, A., Martins, R. M., Kucher, K., and Ker- ren, A. Stackgenvis: Alignment of data, algorithms, and models for stacking ensemble learning using performance metrics. IEEE Transactions on Visualization and Com- puter Graphics, 1547 2020.
Jin, W., Ma, Y., Liu, X., Tang. X.,, Wang, S., and Tang, J. Graph structure learning for robust graph neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp- 66-74, 2020.
Dai, R., Zhang, Y., Fang, Z., Han, B., and Tian, X. Moder- ately distributional exploration for domain generalization. In International Conference on Machine Learning, pp. 6786 6786-6817. PMLR, 2023.
Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., et al. Wilds: A benchmark of in-the- wild distribution shifts. In International Conference on Machine Learning, pp. 5637-5664. PMIR, 2021.
Dong, X., Thanou, D., Rabbat, M., and Frossard, P. Learn- ing graphs from data: A signal representation perspec- tive. IEEE Signal Processing Magazine, 36(3):4466) 63,,

---

Structure-informed Risk Minimization for Robust Ensemble Learning
Lee, Y., Yao, H., and Finn, C. Diversify and disambiguate: Out-of-distribution robustness via disagreement. In Inter- national Conference on Learning Representations, 2023.
Rubinstein, A., Scimeca, L., Teney, D., and Oh, S. J. Scal- able ensemble diversification for ood generalization and detection. arXiv preprint arXiv:2409. 16797, 2024.
Li, D. al. Deeper, Broader and Artier Domain General- ization. In 2017.
Sagawa, S., Koh, P. W., Hashimoto, T. B., and Liang, P. Distributionally robust neural networks. In International Conference on Learning Representations, 2019.
Liu, J., Wu, J., Li, B., and Cui, Distributionally robust optimization with data geometry. Advances in neural information processing systems, 35.33689-33701, 2022.
Shafieezadhh Abadeh, S., Nguyen, V. A., Kuhn, D., and Mohajerin Esfahani, P. M. Wasserstein distributionally robust kalman filtering. Advances Neural Information Processing Systems, 31, 2018.
Ma, M., Li, T., and Peng, X. Beyond the federation: Topology-award federated learning for generalization to unseen clients. In International Conference on Machine Learning, 2024.
Shalev-Shwartz, and Wexler, Y. Minimizing the maximal loss: How and why. In International Conference on Machine Learning, pp. 793-801. PMLR, 2016.
Namkoong, H. and Duchi, J. C. Stochastic gradient methods for distributionally robust optimization with f- divergences. Advances in neural information processing systems, 29, 2016.
Staib, M. and Jegelka, S. Distributionally robust optimiza- tion and generalization in kernel methods. Advances in Neural Information Processing Systems, 32, 2019.
Tong, A. Y., Huguet, G., Natik, A., MacDonald, K., Kuchroo, M., Coifman, R., Wolf, G., and Krishnaswamy, S. Diffusion earth mover's distance and distribution em- beddings. In International Conference on Machine Learn- ing, pp. 10336-10346. PMLR, 2021.
Pagliardini, M., Jaggi, M., Fleuret, F., and Karimireddy, S. P. Agree to disagree: Diversity through disagreement for better transferability, International Conference on Learning Representations, 2023.
Tsoumakas, G., Partalas, I., and Vlahavas, I. A taxonomy and short review of ensemble selection. In Workshop on Supervised and Unsupervised Ensemble Methods and Their Applications, pp. 1-6, 2008
Peng, X., Bai, Q.. Xia, X., Huang. Z., Saenko, K., and Wang, B. Moment matching for multi-soure domain adaptation. In Proceedings of the IEEE/CVF international conference computer vision, pp. 1406-1415, 1415, 2019.
Ueda, N. and Nakano, R. Generalization error of ensemble estimators. In Proceedings of International Conference on Neural Networks, volume 1, pp. 90-95. IEEE, 1996.
Qiao, F. and Peng, X. Uncertainty-guide model gener- alization unseen domains. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
Vapnik, V. and Vapnik, V. Statistical learning theory, 1998.
Qiao, and Peng. X. Topology-aware robust optimization generalization. In International Conference on Leurning Representations, 2023.
Venkateswara, H., Eusebio, J., Chakraborty, S., and Pan- chanathan, Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pp. 5018-5027, 118--502,221111 2017.
for
Qiao, F. and Peng, X. Ensemble pruning for out-of- distribution generalization. In Forty-first International Conference on Machine Learning, 2024.
Wang. H., He, H., and Katabi, D. Continuously indexed domain adaptation. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Ma- chine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 9898-9901 PMIR, 13-18 Jul 2020. URL https: http://ooceeennn.ccoo mat.:/ssrm.pprr../.mmmmm v v119 2199/ang20h.. html.
Qiao, F., Zhao, L., and Peng, X. Learning learn single domain generalization. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
Rame, A., Kirchmeyer, M., Rahier, T., Rakotomamonjy. A., Gallinari, P., and Cord, M. Diverse weight averaging for out-of-distribution generalization. Advances in Neural Information Processing Systems, 35:10821-10836, 2022.
Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R. Gontijo-Lppss R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., et al. Model soups: averaging weights of multiple fine-tuned mod- els improves accuracy without increasing inference time. International Conference on Machine Learning, pp. 23965-23999 PMLR, 2022.
Ramé, A., Ahuja, K., Zhang, J., Cord, M., Bottou, L., and Lopez-Paz, D. Model ratatouille: Recycling diverse mod- els for distribuuttn generalization. In International Conference on Machine Learning, 2023.
10

---

Structure-informed Risk Minimization for Robust Ensemble Learning
Wu, S., Li, and Ding, W. A geometric framework for multiclass ensemble classifiers. Machine Learning, 112 (12):4929-4958, 2023.

---

Structure-informed Risk Minimization for Robust Ensemble Learning
A. Experiments
A.I. Implementation Details
We use DiWA (Rame et al., 2022) to train the models in the ensemble pool. Each model in the ensemble pool is a ResNet- 50 (He et al., 2016) model trained with ERM (Vapnik Vapnik, 1998) using different hyper-paramen settings. The number of models (n) used in the experiments 10. A random model in the ensemble pool is chosen to construct the distribution graph. For optimizing and q.
set Thw 0.1 and 72 0.1, and for WILDS, we set The 3e and 7h = 0.1. À is selected from [0.0, 2.0] for each dataset. We use in-distribution validation set to optimize w and and the number of steps is 100 and 50 for DomainBed and WILDS, respectively.
SGD optimizer. For the experiments
DomainBed,
A.2. Additional Results on DomainBed
We provide detailed experiment results for each test environment DomainBed benchmark.
Non-optimizaation-aased
Optimization-based
Env.
Uniform
Greedy
ERM
Uniform Prior
Laplacian
DRO
SRM
PACS
Art
88.2
88.5
88.0
88.3
88.0
88.0
88.5
Cartoon
82.1
80.0
80.3
80.2
80.3
80.4
80.4
Photo
98.5
98.3
98.0
98.0
98.0
98.1
98.1
Sketch
82.6
81.6
81.5
81.3
81.5
81.8
81.8
VL.CS
Caltech101
98.2
98.8
99.1
99.2
98.5
99.1
99.3
LabelMe
64.5
64.3
65.4
65.3
65.1
65.3
65.4
SUN09
75.3
75.6
73.9
74.3
72.4
74.4
75.6
VOC2007
80.1
79.8
80.3
80.5
78.7
80.3
80.6
OfficeHome
Art
67.6
67.3
66.9
66.8
66.7
66.7
67.3
Clipart
56.9
56.4
56.3
56.3
56.1
57.0
Product
78.1
78.3
79.4
79.2
79.5
79.1
79.6
Real-world
80.6
80.6
80.4
80.4
80.6
80.2
80.6
DomainNet
Clipart
58.8
62.0
63.0
62.9
62.9
62.9
63.0
Infograph
21.3
21.6
21.7
21.7
21.7
21.7
21.8
Painting
51.1
51.7
51.6
51.6
51.5
51.6
51.9
Quickdraw
14.1
15.0
14.9
14.9
14.9
15.0
Real-world
62.2
63.3
64.4
64.4
64.4
64.4
64.5
Sketch
52.3
54.5
54.4
54.6
54.4
54.7
Terralncognita
L,100
55.3
58.9
55.1
55.8
55.8
56.2
58.9
L,38
40.8
39.6
40.9
40.6
40.7
40.9
41.5
L43
60.4
60.6
60.8
60.7
60.6
60.9
61.1
L46
37.3
33.4
38.4
38.1
38.2
37.9
38.9
Table 5. Average accuracy (%) over all test distributions for PACS, VLCS, OfficeHome, DomainNet, and TerraIncognita datasets.
12


Reviews:
[{"review_id": "OrGkuHOoIr", "reviewer": "Reviewer_Qxor", "summary": "This work proposes structure-informed risk minimization (SRM), which can be seen as a modification to the Group DRO algorithm, and applies it to robust ensemble learning.\nMore specifically, SRM optimizes the ensemble weight of multiple fixed pre-trained models to reduce the ensemble’s risk in the worst mixture of training domains that is not too far from the “center” of training domains.\nThe “center” is also a mixture of training domains whose mixing weights are determined by the distance from one domain to other domains.\nIn this way, SRM places more optimization pressure on domains that are closer to all the other domains, unlike Group DRO which assigns weights only based on the risk of a domain.\nEmpirical results show that SRM outperforms several baselines on various OOD generalization benchmarks.\n\n## Update after rebuttal\nAfter carefully reading through the authors' response and other reviewers' comment, I decide to maintain my position (weak reject) for this paper.\nI agree with Reviewer mDcw that the paper's main idea (i.e., centrality metric) is novel and the theoretical results are solid.\nMoreover, the paper is well written and easy to follow.\nMy concern, however, lies in the theoretical and empirical significance of the paper.\n\nThe paper focuses on a relatively specific (one might say narrow) problem, namely robust ensemble learning, that aims to learn good mixing weights (with a linear layer) for the predictions of different pre-trained models.\nWhile focusing on this particular problem is fine, the proposed method, SRM, and the theoretical results are only loosely related to the problem: the \"ensemble\" part of the problem seems to be largely irrelevant.\nIn this respect, I feel that SRM only addresses a small issue of (robust) ensemble learning.\n\nAnother issue of the paper is the gap between the stated goal and the actual realization of SRM.\nMore specifically, the goal is to balance robustness and *average* performance, but the proposed method optimizes for the worst-domain performance in the vicinity of the *central* domain instead of simply the *average* domain.\nNo clear reason is given for why the more complicated approach is opted for.\nUnder the current setting, I fail to see any meaningful difference between SRM and the simpler approach using the average domain.\n\nEmpirically, SRM only marginally improves the baselines, validating my concerns above. I think the paper would greatly benefit from a better setting or scenario to demonstrate the distinct properties and effectiveness of SRM. I sincerely hope that the authors not to be discouraged if this paper is rejected because the idea of domain centrality is really interesting and I believe it will probably shine under a slightly different light.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful feedback and acknowledging the novelty of our work. We appreciate the thoughtful comments and address them point-by-point below.\n\n**1. Why Ensemble Learning? Can SRM Improve the OOD Performance of Individual Models?**\n\nWe agree that SRM is a general framework that extends beyond ensembles. Recent studies have shown that no single model can perform well across all OOD scenarios, and ensemble learning has emerged as a promising paradigm that leverages the complementary strengths of diverse models. With the growing availability of pretrained models in repositories like HuggingFace, it becomes increasingly practical and impactful to study how to effectively ensemble existing models, rather than focusing solely on training new ones.\n\nThat said, SRM can indeed be applied to train individual models. To demonstrate this, we adapted SRM for single-model training and evaluated it on the PACS benchmark. Our method achieves a +0.6% average gain over CORAL, the current SOTA, showing its applicability beyond ensembles:\n\n| Method         | Art              | Cartoon           | Photo             | Sketch            | Average   |\n|----------------|------------------|--------------------|--------------------|--------------------|-----------|\n| ERM            | **88.1 (0.1)**    | 77.9 (1.3)         | 97.8 (0)           | 79.1 (0.9)         | 85.7      |\n| Group DRO      | 86.4 (0.3)        | 79.9 (0.8)         | **98.0 (0.3)**     | 72.1 (0.7)     | 84.1      |\n| CORAL (SOTA)   | 87.7 (0.6)        | 79.2 (1.1)         | 97.6 (0)           | **79.4 (0.7)**     | 86.0      |\n| SRM            | 87.6 (0.5)        | **82.0 (0.6)**     | **98.0 (0.1)**     | 78.8 (1.3)     | **86.6**  |\n\n**2. Diverse Model Pools**\n\nFollowing the reviewer’s suggestion, we evaluated SRM with a more diverse ensemble pool by incorporating models trained with ERM, Mixup, and CORAL. The table below shows results on TerraIncognita:\n\n| Algorithm    | L100 | L38  | L43  | L46  | Avg  |\n|--------------|------|------|------|------|------|\n| Uniform      | 50.4 | 42.6 | **62.0** | 38.7 | 48.4 |\n| ERM          | 51.3 | 43.7 | 61.4 | 39.5 | 49.0 |\n| Group DRO    | 50.9 | 43.2 | 61.8 | 39.2 | 48.8 |\n| SRM          | **52.1** | **44.3** | 61.5 | **40.0** | **49.5** |\n\nThe strong result of Uniform on L43 may stem from complementary patterns captured by the diverse models. Still, SRM consistently provides better overall performance. We will include this in the revised paper. All code used is provided in the supplementary materials.\n\n**3. Related Work on Balancing Average and Worst-case Risks**\n\nWe thank the reviewer for pointing out relevant related work. SRM shares the goal of balancing robustness and generalization with Dual Risk Minimization (DRM) [1] and Quantile Risk Minimization (QRM) [2], but differs in both formulation and implementation:\n\n   - DRM fine-tunes models using concept descriptions, while QRM offers probabilistic guarantees by minimizing quantile risk.\n\n   -  SRM, in contrast, introduces a structural prior derived from a distributional graph and centrality measures, offering a complementary perspective grounded in geometric relationships among training distributions.\n\nWe will expand the related work section to explicitly discuss these connections in the revised manuscript.\n\n**4. Central Domain vs. Average Domain**\n\nThe central domain differs from the average domain (uniform mixture) in that it emphasizes global influence within the distributional graph, not just equal weighting. For instance, in PACS (training: Art, Photo, Sketch; test: Cartoon), our graph identified Art as the most central domain. We hypothesize this occurs because \"Art\" inherently combines photographic elements with artistic styles, making it structurally influential.\n\nWhen all distributions have equal centrality, our prior reduces to a uniform prior, demonstrating that average-case optimization is a special case of SRM. This connection is discussed in lines 142–144, and empirically validated in Table 1, where SRM consistently outperforms the uniform prior across all DomainBed datasets.\n\n**5. Clarification on 2-Wasserstein Distance**\n\nThe 2-Wasserstein distance used for constructing the distribution graph is computed over features extracted by the ERM-trained model with the highest validation accuracy. We will clarify this implementation detail in the revised version.\n\nWe sincerely thank the reviewer again for the constructive feedback. Please let us know if you have any further questions or suggestions. We would be glad to elaborate.\n\nReferences:\n\n[1] Li et al. \"Dual Risk Minimization: Towards Next-Level Robustness in Fine-tuning Zero-Shot Models.\" NeurIPS 2024.\n\n [2] Eastwood et al. \"Probable domain generalization via quantile risk minimization.\" NeurIPS 2022.", "claims_and_evidence": "The authors claim that the proposed method achieves superior OoD generalization compared to existing ensemble combination strategies across diverse benchmarks. However, as shown in Tables 1 and 2, the empirical improvement over ERM is marginal (66.23 $\\to$ 66.54 on DomainBed, and 51.45 $\\to$ 51.57 on FMoW-WILDS). Moreover, if SRM does, as the authors claim, provide a more realistic approximation of potential test distributions, then it should be better demonstrated under more general settings than ensemble learning. For example, can SRM improve the OOD performance of individual models by updating their parameters?", "methods_and_evaluation_criteria": "Both the methods and the evaluation criteria make sense for the problem. The only thing that I would like to point out is that the proposed structure-informed risk minimization (SRM) is only loosely connected with ensemble learning. SRM feels like a more general principle for OOD generalization where ensemble learning is just a very narrow use case. It is unclear what consideration in SRM is specifically for ensemble learning for it to be effective.", "theoretical_claims": "Proofs are not carefully checked, but the theoretical claims look sound to me.", "experimental_designs_or_analyses": "The experiments considered a range of baselines (both non-optimization and optimization-based ones) on common benchmarks of OOD generalization (DomainBed and WILDS). The model pool for the ensemble consists of ResNet-50 models trained with ERM under different hyperparameter settings. This is fine, but I would suggest the authors also consider some more diverse pools of models, e.g., models trained with different algorithms and/or on different datasets.", "relation_to_broader_scientific_literature": "I think SRM might be useful for OOD generalization in some more general settings. A lot of previous work on OOD generalization focuses on worst-case risk minimization, e.g., IRM, Group DRO, and VREx, which tend to overemphasize the importance of the worst-case domain. SRM, on the other hand, additionally considers the centrality of a domain when weighing its importance.", "essential_references_not_discussed": "The discussion on related work is a bit lacking. A closely related work, dual risk minimization [1], also proposes to combine average risk minimization with worst-case risk minimization to mitigate the latter's over-conservativeness but does not assume multiple training distributions. Another related work is quantile risk minimization [2], which focuses on high probability domain generalization and is not mentioned in the paper either. The motivation of these papers is quite similar to the reviewed paper, although the contexts are slightly different.\n\n[1] Li et al. \"Dual Risk Minimization: Towards Next-Level Robustness in Fine-tuning Zero-Shot Models.\" NeurIPS 2024.  \n[2] Eastwood et al. \"Probable domain generalization via quantile risk minimization.\" NeurIPS 2022.", "other_strengths_and_weaknesses": "- The centrality of a domain is an interesting and novel concept (at least to me) in the context of OOD generalization. It plays a central role in the proposed method, SRM, but I don't think it's sufficiently discussed. In particular, how does the central domain relate to the average domain? The authors seem to suggest that these two concepts are roughly equivalent, stating that SRM balances worst-case robustness with average performance, but do they really? Suppose there are two different training domains, one of which is much more likely than the other (e.g., camels are more likely to appear in deserts than on grasslands). In this case, the average domain is heavily tilted towards the more likely one, while the central domain is more like a uniform mixture of the two domains. If they are indeed different, why should one value the central domain more than the average domain?\n- It is mentioned that a computationally efficient Gaussian-based approximation is used to estimate the 2-Wasserstein distance between data distributions, but many important details of this process are not provided. For example, was the distance computed over the input features or features extracted by some pre-trained model? If it was the latter case, which models were used?", "other_comments_or_suggestions": "The paper is well-written and easy to follow. I didn't find any typos or inconsistencies.", "questions_for_authors": "Please see \"Other Strengths And Weaknesses\".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vKKhjbADbD", "reviewer": "Reviewer_PHpH", "summary": "This paper proposes a framework for learning robust ensemble weights without requiring access to test data. It aims to mitigate the over-pessimism of Distributionally Robust Optimization (DRO) by focusing the uncertainty set on more plausible structures. The idea is solid, and the proposed algorithm is computationally feasible. However, the experimental results do not provide enough evidence to convincingly support the claimed superiority of the method.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback. We address each concern below.\n\n**1. Comparison with [1-3] and Other OOD Generalization Methods**\n\nWhile SRM and the methods in [1–3] all aim to reduce the pessimism often observed in DRO, they approach the problem from different directions:\n\n- [1–3] primarily focus on data-point-level robustness, targeting noisy samples or outliers.\n\n- SRM, by contrast, operates at the distribution level by identifying influential training distributions via centrality in a distribution graph.\n\nThis makes SRM particularly suitable for domain generalization scenarios where relationships among distributions are more informative than individual noisy points. We also note that GCDRO [2] builds a data graph (not a distribution graph) and is mainly designed for regression tasks, which further differentiates it from our setting. We will add a discussion of these works and clarify the distinction in the revised version.\n\nRegarding baselines in DomainBed, our goal is not to introduce a new training algorithm for OOD generalization. Instead, SRM is a model-agnostic framework designed to learn ensemble weights from pretrained models, which can be drawn from any existing training algorithm or model zoo.\n\n**2. Statistical Significance of Results**\n\nWe appreciate your concern about the similarity in test accuracies across baselines. This is primarily due to our use of DiWA, which requires all models in the ensemble pool to be initialized identically so they can be merged for efficient inference. This likely reduces diversity among models.\n\nTo address this, and following Reviewer Qxor’s suggestion, we constructed a more diverse model pool using different training algorithms (ERM, Mixup, CORAL). The table below shows results on TerraIncognita:\n\n| Algorithm    | L100 | L38  | L43  | L46  | Avg  |\n|--------------|------|------|------|------|------|\n| Uniform      | 50.4 | 42.6 | **62.0** | 38.7 | 48.4 |\n| ERM          | 51.3 | 43.7 | 61.4 | 39.5 | 49.0 |\n| Group DRO    | 50.9 | 43.2 | 61.8 | 39.2 | 48.8 |\n| SRM          | **52.1** | **44.3** | 61.5 | **40.0** | **49.5** |\n\nThe strong result of Uniform on L43 may stem from complementary patterns captured by the diverse models. Still, SRM consistently provides better overall performance. We will include this in the revised paper. All code used is provided in the supplementary materials.\n\n**3. Why SRM Performs Well in Distribution Extrapolation Settings**\n\nWe appreciate the reviewer’s interest in our method’s performance under distribution extrapolation. In the FMoW-WILDS benchmark, each year from 2002 to 2017 is treated as a distinct distribution. The distribution extrapolation setting evaluates generalization to test years that lie outside the range of training years:\n-  Test Before 2004: training on 2007–2018, validation on 2004–2007, testing on 2002–2004\n- Test After 2016: training on 2002–2013, validation on 2014–2016, testing on 2016–2017\n\nIn “Test Before 2004”, the test years are temporally closest to 2007, and Group DRO focuses its optimization almost entirely on this single year (the worst-case distribution under its framework, shown in Figure 1). However, this strategy ignores other potentially useful training distributions, which can limit generalization. This behavior is also supported by theory: the solution to Group DRO's linear program tends to lie at a vertex of the feasible region.\n\nOur method achieves better performance by balancing the worst-case distribution with structurally influential distributions, as identified through centrality in the distribution graph. Rather than collapsing onto a single training domain, SRM spreads attention across those distributions that are both risky and influential, yielding improved generalization even in extrapolation scenarios.\n\nThis aligns with the insights from Dual Risk Minimization and Quantile Risk Minimization (references from Reviewer Qxor), which also highlight that strong OOD generalization emerges from a trade-off between average-case and worst-case risk, rather than optimizing either in isolation.\n\n**4. Experiments on Tabular Datasets and Comparison with Boosting Methods**\n\nWe appreciate the suggestion to compare SRM with XGBoost, LightGBM, and CatBoost. However, there is an important distinction:\n- Boosting methods build ensembles by training weak learners sequentially, whereas\n- SRM assumes a pool of already trained models and focuses on learning the optimal weighting scheme for combining them under distribution shift.\n\nThat said, we agree that applying SRM to tabular datasets is worthwhile. We plan to include experiments on the eICU dataset, which includes electronic health records (EHRs) from 208 hospitals in the U.S. Due to time constraints, we may not include results in the rebuttal but will provide them in the final paper.\n\nThank you again for your valuable feedback. Please let us know if you have further questions. We would be happy to clarify or expand on any point.", "claims_and_evidence": "1. Reasonable assumption and practical setting: The paper tackles the problem of learning robust ensemble weights that generalize well to unseen test distributions. The method performs effectively under a relatively mild assumption that the test performance can be approximated by a mixture of training domains. \n2. Good solution to the problem: This paper overcome the over-pessimism of DRO by not weighting too much on distant distribution and focusing more on centralized training distributions.", "methods_and_evaluation_criteria": "- Robust across different centrality metrics and distance metrics: Ablation study illustrates the framework is conceptually valid, and can be replaced with different centrality metrics and distance metrics while maintaining good performance.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "1. Lack of comparison with closely related work: It would strengthen the paper to compare the proposed method with other related approaches aimed at addressing over-pessimism in DRO, such as [1,2,3], as well as other OOD generalization methods (see baselines in DomainBed).\n2. Statistical significance of results: In table 1, I notice that all of your baselines have **very very similar test accuracy** on many of the datasets, which seems implausible. Also, for some lines in table 1 and table 2, the improvement achieved by your method over other baselines is not statistically significant. \n3. Performance in extrapolation settings: In table 2, you claim your method outperforms other baselines in both distribution interpolation and distribution extrapolation settings. As your method learns an optimal mixture of training distributions constrained by prior p, can you explain why it achieves good performance in distribution extrapolation settings?\n4. Also, I would like to see some results on tabular datasets compared with some well-known ensemble methods like XGBoost, Light GBM, CatBoost, etc.\n\n[1] DORO: Distributional and Outlier Robust Optimization. Runtian Zhai, Chen Dan, Zico Kolter, Pradeep Ravikumar\n\n[2] Geometry-Calibrated DRO: Combating Over-Pessimism with Free Energy Implications. Jiashuo Liu, Jiayun Wu, Tianyu Wang, Hao Zou, Bo Li, Peng Cui\n\n[3] Boosted CVaR Classification. Runtian Zhai, Chen Dan, Arun Sai Suggala, Zico Kolter, Pradeep Ravikumar", "relation_to_broader_scientific_literature": "This paper presents a framework for learning robust ensemble weights without requiring access to test data. It aims to mitigate the over-pessimism of Distributionally Robust Optimization (DRO) by focusing the uncertainty set on more plausible structures. The idea is solid, and the proposed algorithm is computationally feasible.", "essential_references_not_discussed": "[1] DORO: Distributional and Outlier Robust Optimization. Runtian Zhai, Chen Dan, Zico Kolter, Pradeep Ravikumar\n\n[2] Geometry-Calibrated DRO: Combating Over-Pessimism with Free Energy Implications. Jiashuo Liu, Jiayun Wu, Tianyu Wang, Hao Zou, Bo Li, Peng Cui\n\n[3] Boosted CVaR Classification. Runtian Zhai, Chen Dan, Arun Sai Suggala, Zico Kolter, Pradeep Ravikumar", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "46QNfql4dR", "reviewer": "Reviewer_PexF", "summary": "This paper presents SRM, a method to improve how ensemble models handle unseen data changes by leveraging the relationships between training data distributions. SRM builds a network of these distributions, measuring their similarities with a simplified distance metric. It prioritizes \"central\" distributions that are most representative of the overall data structure. Using an efficient optimization process, SRM balances worst-case robustness and average performance. Tests on standard benchmarks show SRM outperforms existing methods like DRO and ERM in adapting to new environments. The approach is backed by theory explaining why focusing on structural relationships improves reliability and convergence speed.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for acknowledging the novelty of our work. Below, we address the main points raised:\n\n1. Assumption of Our Paper and Extreme OOD Scenarios\n\nIn Equation (14), we assume that the test distribution lies within a bounded divergence from a mixture of training distributions. This assumption is aligned with a well-established foundation in learning under distribution shift, as formalized by Ben-David et al. [1]. Specifically, Theorem 2 in [1] shows that the risk on the test distribution is bounded by the empirical risk on the source domain(s) plus a divergence term (e.g., H-divergence) between the source and target distributions. When this divergence becomes too large, no method can guarantee meaningful generalization bounds.\n\nOur method is designed for natural distribution shifts and evaluated on two challenging and realistic benchmarks: DomainBed and WILDS, both of which are widely adopted in the OOD generalization community. We acknowledge that adversarial robustness is an important but orthogonal research direction and is outside the scope of this work.\n\n2. Comparison with Individual Base Learners\n\nTo further isolate the benefit of structure-informed weighting, we compare SRM with both individual models and a uniform ensemble baseline. The table below reports results on the DomainNet dataset across six test domains. SRM significantly outperforms both the average individual performance and uniform ensemble. For example, on the “Clipart” domain, accuracy improves from 56.79% (mean of individual models) to 58.75% (uniform ensemble) and then to 63.02% with SRM, showing that leveraging structural relationships provides substantial gains.\n\n| test domain | min   | max   | mean  | std  | Uniform | Ours  |\n| -------- | ----- | ----- | ----- | ---- | ------- | ----- |\n| Clipart         | 52.74 | 59.51 | 56.79 | 2.39 | 58.75   | 63.02 |\n| Infograph       | 18.19 | 20.12 | 18.98 | 0.61 | 21.30   | 21.77 |\n| Painting        | 44.99 | 48.36 | 46.57 | 1.09 | 51.07   | 51.92 |\n| Quickdraw        | 10.98 | 13.51 | 12.25 | 0.75 | 14.11   | 14.96 |\n| Real-world        | 57.31 | 61.30 | 59.67 | 1.09 | 62.20   | 64.51 |\n| Sketch        | 46.76 | 51.12 | 49.11 | 1.21 | 52.31   | 54.66 |\n\nThese results confirm that our method’s improvement is not simply due to ensemble averaging, but stems from meaningful structural modeling.\n\n3. Runtime Validation Against Exact Methods\n\nWe provide a runtime comparison of different distance metrics used for constructing the distributional graph, using the PACS dataset (3 domains, 1000 samples, 2048-dimensional features). As shown below, our Gaussian-approximated 2-Wasserstein distance achieves the best trade-off between accuracy and computational efficiency.\n\n| Distance      | Running Time | Worst-region Acc. (%) | Time Complexity     |\n| ------------- | ------------ |------------------------|---------------------|\n| 2-Wasserstein | 28.05s       | 38.10                  | O(nd² + d³)         |\n| Diffusion EMD | 2.56s        | 37.99                  | O(nd)               |\n| EMD           | 516.41s      | 38.06                  | O(n³d²)             |\n\nWhile Diffusion EMD is faster, it sacrifices accuracy. Exact EMD is computationally infeasible for large-scale datasets. Our method balances these trade-offs effectively, enabling practical deployment on real-world benchmarks.\n\nWe hope this addresses your concerns. Please let us know if you have any further questions or suggestions. We would be happy to elaborate.\n\nReference\n\n[1] Ben-David, Shai, et al. \"A theory of learning from different domains.\" Machine Learning 79.1-2 (2010): 151–175.", "claims_and_evidence": "The paper’s claims are partially supported by evidence. Key strengths include empirical validation on benchmarks, where SRM shows modest but consistent improvements over DRO and ERM, and ablation studies confirming the value of structural priors like closeness centrality. However, claims about computational efficiency lack direct timing comparisons, and theoretical assumptions remain untested for extreme distribution shifts. While the core idea of leveraging distributional graphs is novel and empirically validated, gaps in validating efficiency, base learner diversity, and extreme OOD performance weaken the overall support for the method’s broad applicability.", "methods_and_evaluation_criteria": "The proposed methods in SRM are reasonably designed to address the challenges of robust ensemble learning under distribution shifts. By leveraging structural relationships and centrality priors, SRM mitigates the over-conservatism of traditional DRO and better models real-world distribution shifts. The Gaussian Wasserstein approximation is practical for image data, reducing computational complexity, while the alternating gradient optimization ensures valid constraints. The evaluation on benchmarks like DomainBed and WILDS covers key OOD scenarios effectively, but the reliance on Gaussian assumptions limits its applicability to non-Gaussian data. The absence of extreme OOD tests and runtime efficiency comparisons leaves room for broader validation to strengthen the method's claims. Overall, the methods and evaluation criteria are sensible within the scope of the paper but require expanded testing for wider applicability.", "theoretical_claims": "The paper’s theory is mathematically solid and explains how structural priors ( distribution graphs) improve robustness. It works well for gradual shifts but probably has limits in extreme real-world OOD cases (totally new domains or adversarial attacks). The key issue is its reliance on assuming test data can be approximated by training mixtures-a condition rarely met in unpredictable scenarios. Without validation for these extremes, the theory’s practical guarantees remain unproven.", "experimental_designs_or_analyses": "The experimental design effectively validates SRM’s performance on standard benchmarks like DomainBed and WILDS, demonstrating strengths in methodological rigor ( ablation studies on centrality metrics) and benchmark relevance. However, it exhibits critical limitations: 1) no comparison with individual base learners precludes distinguishing whether gains stem from structural optimization or ensemble diversity; 2) omission of extreme OOD scenarios (novel domains, adversarial attacks) undermines claims about robustness boundaries; and 3) efficiency claims lack runtime validation against exact methods (Sinkhorn for EMD). While the design supports controlled efficacy, addressing these gaps is essential to establishing SRM’s scalability and real-world applicability.", "relation_to_broader_scientific_literature": "SRM’s contributions sit at the intersection of robust optimization, ensemble learning, and distributional geometry. By integrating structural priors into distributional robustness, it advances the state-of-the-art in OOD generalization. However, its practical validity hinges on assumptions (test distributions as training mixtures) that diverge from extreme OOD scenarios documented in literature (cross-modal shifts, adversarial attacks). Future work could bridge this gap by incorporating meta-learning or non-parametric distribution modeling.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The paper presents a creative fusion of graph theory, distributional robustness, and ensemble learning, offering a novel framework (SRM) that addresses key limitations of traditional methods like DRO and deep ensembles. Its practical relevance to real-world OOD scenarios ( climate change, healthcare) is underscored by strong empirical results on benchmarks like DomainBed and WILDS, and efficiency gains from Gaussian-Wasserstein approximations enhance scalability. However, the work is constrained by overreliance on untested assumptions ( test distributions as training mixtures), gaps in extreme OOD validation ( adversarial shifts), and ambiguity in theoretical contributions ( geometric assumptions on distribution manifolds). While the integration of structural priors and robust optimization is conceptually compelling, the method’s scalability to complex, nonlinear distribution shifts remains uncertain without broader experimental validation and clearer justifications for its theoretical premises.", "other_comments_or_suggestions": "1. Notation Consistency: Ensure consistent notation for key variables (e.g., clarify whether \\lambda refers to regularization strength or another parameter across sections). Define all symbols in equations (e.g., d for dimensionality) when first introduced.\n\n2. Experimental Details: Specify the number of training/validation splits used for hyperparameter tuning (e.g., how many \\lambda values were tested). Report computational runtime for SRM vs. baselines (even if approximate) to contextualize efficiency claims.", "questions_for_authors": "Q1. Theoretical Claims: Relies on assumptions with weak generalizability (test distributions as training mixtures) and geometric/geometric assumptions (e.g., manifold structure) that lack validation for non-Gaussian/non-manifold data.\n\nQ2. Experimental Design: Omits critical validations: comparisons with single base learners, extreme OOD scenarios (e.g., adversarial shifts), and runtime efficiency benchmarks.\n\nQ3. Methods: Over-reliant on Gaussian approximations (limiting non-Gaussian scalability) and ignores the impact of graph pruning on robustness.\n\nQ4. Clarity: Lack of clear and unambiguous definitions of all symbols.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZHaNI3D3kG", "reviewer": "Reviewer_mDcw", "summary": "This paper introduces a novel framework to learn ensemble weights to improved out-of-distribution (OOD) robustness. The key idea is to incorporate structure relationships between training distributions to build a realistic uncertainty set. The authors proposed a computationally efficient optimization algorithm with theoretical guarantees. Empirically, the proposed method consistently outperforms existing ensemble methods across diverse benchmarks including DomainBed and WILDS, demonstrating the superior OOD generalization capability.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their thoughtful and positive feedback. We appreciate your recognition of our paper as \"very well-written and easy to follow\" and that our approach is \"theoretically sound while remaining computationally tractable.\" Your comments about the \"compelling unified framework\" and the clarity of our visualizations are particularly encouraging. Below, we address your specific questions:\n\n**1. Proof on Convergence Analysis**.\n\nOur convergence analysis leverages Assumption 4.3 (Lipschitz continuity and smoothness) to track gradient norm decreases across iterations. The key insight is that the regularization term $\\lambda D(q\\|\\|p)$ creates a more strongly convex objective function, accelerating convergence proportionally to $\\lambda$. With step sizes $\\eta_w = \\eta_q = 1/\\sqrt{T}$, we apply standard convex optimization techniques to bound the expected squared gradient norm after $T$ iterations by $C/(1+\\lambda)\\sqrt{T}$, where $C$ depends on $\\beta$ and $L^2$. This explains SRM's faster convergence compared to standard DRO ($\\lambda = 0$), demonstrating how our structure-informed approach improves both robustness and computational efficiency. The detailed proof will be included in the revised manuscript.\n\n**2. Laplacian-based Prior Implementation**\n\nFor the Laplacian-based prior, we construct the graph Laplacian matrix $L=D-A$ where $D$ is the degree matrix (diagonal with $D_{ii} = \\sum_j A_{ij}$) and $A$ is our adjacency matrix. The optimization constraint becomes $q^T L q \\leq \\tau$, penalizing weight differences between connected distributions. We solve this via a similar Lagrangian approach as in our main method. While this enforces local smoothness, our centrality-based prior better captures global influence within the distribution manifold, explaining its superior performance in identifying representative distributions.\n\n**3. What if there is only one training distribution or distribution ID is unknown?**\n\nRecent methods [1-2] have been proposed to infer distribution IDs from data. Our approach is orthogonal to these methods and can be built on top of them to handle scenarios where distribution IDs are unknown.\n\n**4. How does SRM perform when the number of training distributions is very small (2-3)?**\n\nSRM requires at least three training distributions to build a meaningful graph. In our PACS experiment (using \"Art,\" \"Sketch,\" and \"Photo\" as training with \"Cartoon\" as test), our graph analysis identified \"Art\" as the most central domain. We hypothesize this occurs because \"Art\" inherently combines photographic elements with artistic styles, making it informative for generalizing to \"Cartoon.\" Even with just three distributions, SRM successfully identifies influential distributions through structural analysis.\n\nAll noted typos will be corrected in the revised manuscript. Please let us know if you have any further questions or suggestions. We would be happy to elaborate.\n\nReferences:\n\n[1] Creager et al. \"Environment inference for invariant learning.\" ICML 2021.\n\n[2] Liu et al. \"Just train twice: Improving group robustness without training group information.\" ICML 2021.", "claims_and_evidence": "The claim that the proposed method (SRM) balances worst-case robustness with average performance is supported by theoretical guarantees in Section 4 and empirical results showing consistent improvements over both ERM and DRO approaches.\n\nThe effectiveness of structure-informed uncertainty sets is validated by ablation studies investigating different graph construction methods, centrality measures, and regularization strengths.\n\nPerformance improvements over existing methods are demonstrated consistently across multiple datasets and evaluation settings.", "methods_and_evaluation_criteria": "The idea of modeling relationships between training distributions to build a realistic uncertainty set is intuitive and provides a principled way to address the overly-pessimistic issue of DRO.\n\nEvaluation on common benchmarks follows standard practices in the field.\n\nThe experiments of various train/test split scenarios on temporal distribution shifts and corruption tests further strengthens the evaluation by considering diverse distribution shifts.", "theoretical_claims": "The proofs in the appendix appear sound. The density-centrality relationship proof correctly decomposes distances and establishes bounds based on the distribution space properties. The generalization bound proof properly utilizes the mixture distribution approximation and applies concentration inequalities. However, the proof on convergence analysis seems missing, but the main conclusion seems correct.", "experimental_designs_or_analyses": "Experiments follow standard practices for OOD evaluation where multiple environments are used for training and average performance across test environments is reported.\n\nThe comparison against multiple baselines is fair and comprehensive.\n\nAblation studies thoroughly examine the impact of key components.\n\nPersonally, I really like the experiment design of various train-test split scenarios on temporal distribution shifts which clearly shows the overly-conservative issue of Group DRO.", "relation_to_broader_scientific_literature": "The discussion on ensemble learning for OOD generalization is comprehensive and get to the point. When test data is unavailable, how to combine models for improved OOD robustness remains a seldom investigated problem. The common methods are uniform (equally aggregate all models) and greedy (select models based on accuracy on an in-distribution validation set). However, neither of them is a promising solution under distributional shifts. DRO is a promising technique to improve robustness but suffers from overly-pessimistic issues. The novelty of this work lies in bridging the two areas by applying DRO to ensemble weight optimization while incorporating structural information to mitigate over-pessimism.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Pros:\nUtilizing the structural information to build a realistic uncertainty set is novel and well-motivated. And this paper presents a compelling unified framework that encompasses existing approaches ERM and DRO as special cases through the constraint in Equation 7.\n\nThe paper is very well-written and easy to follow. The discussion on alternative methods and connection to prior works shows the authors’ in-depth understanding.\n\nThe visualizations of distributional graphs clearly illustrate how SRM assigns higher weights to influential distributions compared to DRO's focus on worst-case distributions. And the experiment on multiple train/test splits under temporal distribution shifts is a clever and insightful design. \n\nThe approach is theoretically sound while remaining computationally tractable.\n\nCons:\nThe effectiveness of SRM might be limited in scenarios with few training distributions, as the graph structure would be less informative.\nAlthough the discussion of alternative priors (Section 3.1) is excellent, the details of Laplacian-based prior is missing. I understand Laplacian based method is widely used in semi-supervised learning and few-shot learning tasks. It would be more beneficial if the author can provide more implementation details.", "other_comments_or_suggestions": "Typos:\n\n1. In Line 122, it should be G=(V, A). \n2. In Definition 3.1, it should be c: P_e -> R^+.", "questions_for_authors": "SRM requires multiple training distributions. What if there is only one training distribution or distribution ID is unknown?\n\nHow does SRM perform when the number of training distributions is very small (2-3)? Does the graph structure still provide meaningful guidance in such scenarios?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "n/a"}}]

Author rebuttal:

Meta-review: The paper proposes new approaches for Structure-informed Risk Minimization (SRM), a novel modification of the Group Distributionally Robust Optimization (Group DRO) framework, applied to robust ensemble learning. SRM improves upon Group DRO by incorporating structural information about the training domains, optimizing ensemble weights of fixed pre-trained models to minimize risk under adversarial domain mixtures centered around a data-driven “central” domain. The method is theoretically well-grounded, general in formulation, and consistently delivers performance gains across challenging datasets. Its clear design and conceptual contribution offer potential for future work in distributional robustness and out-of-distribution (OOD) generalization.

However, most of the reviewers expressed concerns about the scope and practical relevance of the method. While the paper is positioned within the context of robust ensemble learning, the actual contributions of SRM appear only tangentially connected to the ensemble aspect, focusing more broadly on risk minimization strategies. Furthermore, the gap between the paper’s stated objectives—balancing robustness and average performance—and the proposed implementation, which optimizes worst-case performance near a central domain, lacks clear justification. In addition, the added complexity over a simpler average-domain baseline is not convincingly motivated. To strengthen the work, a more compelling application scenario would be valuable in highlighting SRM’s unique strengths and necessity.
Decision: Accept (poster)