Title: Mitigating Spurious Correlation via Distributionally Robust Learning with Hierarchical Ambiguity Sets
Venue: NeurIPS 2025
Accepted: no
Keywords: Spurious Correlation

Subpopulation Shift

Group Distributionally Robust Optimization

Wasserstein Distributionally Robust Optimization
Abstract: Conventional supervised learning methods are often vulnerable to spurious correlations, particularly under distribution shifts in test data. To address this issue, several approaches, most notably Group DRO, have been developed. While these methods are highly robust to subpopulation or group shifts, they remain vulnerable to intra-group distributional shifts, which frequently occur in minority groups with limited samples. We propose a hierarchical extension of Group DRO that addresses both inter-group and intra-group uncertainties, providing robustness to distribution shifts at multiple levels. We also introduce new benchmark settings that simulate realistic minority group distribution shifts—an important yet previously underexplored challenge in spurious correlation research. Our method demonstrates strong robustness under these conditions—where existing robust learning methods consistently fail—while also achieving superior performance on standard benchmarks. These results highlight the importance of broadening the ambiguity set to better capture both inter-group and intra-group distributional uncertainties.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors argue that current approaches to tackling spurious correlations and distribution shifts assume \"clean\" groups and do not account for intra-group variations. To tackle this problem, they propose a hierarchical version of GroupDRO, which models both the mixing proportions between groups via $\\beta$ weights belonging to the $\\Delta_{m-1}$ simplex, where $m$ is the number of groups and the intra-group variability via a second constraint on the ambiguity set, modeled via Wasserstein distances and a cost function based on the distance between representations in the latent space.\n\nTo make the problem tractable, they optimize a surrogate objective that corresponds to an upper bound on the actual loss, arguing that the hierarchical ambiguity set can be modeled (or over-approximated) using adversarial perturbations in the latent space. In practice, for a single sample $x_i$ they:\n\n1. Update the adversarial perturbation $z_i$ via a single gradient ascent step, bounding it to a sphere defined by $\\epsilon_g$ via projections as needed.\n2. Update the weights $\\beta$\n3. Update the model parameters via SGD.\n\nThe hyper-parameter $\\epsilon_g$ is picked by:\n\n1. Encoding the group training samples.\n2. Projecting the embeddings to one dimension via t-SNE.\n3. Splitting the samples into 5 quantiles and then training models using either the bottom 4 or top 4 quantiles and the remaining quantile as a validation set. The authors argue that this simulates realistic distribution shifts for minority groups.\n4. They then average the two runs and pick the $\\epsilon_g$ parameter that maximizes the minority-group accuracy on average.\n\nTheir method is then evaluated on standard benchmarks with group annotations, namely CMNIST, Waterbirds, and CelebA, and on newly introduced variations of them that model intra-group shifts:\n\n1. Shifted CMNIST, where all images in the minority group (label 1, red color) are rotated by 90 degrees at test time.\n2. Shifted Waterbirds, where the training set minority group (waterbird, land background) only contains waterfowls, and the test set only seabirds.\n3. Shifted CelebA, where the minority group (blonde, male) only has images without glasses during training and with glasses at test time.\n\nOn shifted benchmarks, their method consistently achieves the best performance, improving by up to ~15% over GroupDRO and more than other baselines that utilize labels.\n\nOn standard benchmarks, they achieve performances that are either slightly better than or equal to GroupDRO, both for the group with the worst accuracy and for the average accuracy. Similarly, they outperform every other baseline that either uses group annotations or generates its own, with the exception of PDE on CelebA. Still, they largely surpass it in a shifted setup.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. The paper is well-written and easy to follow.\n2. The paper tackles an important problem, namely the performance of machine learning models in the presence of spurious correlations and distribution shifts. Furthermore, the paper goes further than traditional group annotations, by considering the problem of distribution shifts within groups. These shifts are modeled by altering existing and well-known benchmark datasets.\n3. The method is reasonably simple and intuitive, and leads to performance improvements on both traditional benchmarks with \"clean\" groups and on the newly introduced benchmarks. On the latter, the proposed method improves by up to 15% over GroupDRO, and more over other baselines that make use of group annotations.\n\n### Weaknesses\n\n1. While the authors clearly explain their experimental setup, models, and hyper-parameter ranges used in their experiment, the exact configuration(s) for each dataset are missing. Reporting those details would facilitate reproducibility and make it easier and faster. Similarly, the authors mention that they use Torchvision's ResNet-50 model, but do they mean the version with IMAGENET1K_V1 or IMAGENET1K_V2 weights? Is this the exact same model used by other methods whose performance was reported from other papers, e.g. [59] and [19]? I am not super confident on it being the case, as e.g. [59] is from ICML2021 (Summer 2021), while the IMAGENET1K_V2 weights (the current default) were published in November 2021 according to the blog post linked on the model page.\n2. The evaluation is limited to datasets with a small number of label/group combinations. It would be interesting to see how well this method scales as the number of label/group combinations grows, e.g. on iWildCam2020-WILDS.\n\nI am willing to increase my score in case the doubts regarding the ResNet-50 backbone can be clarified.\n\nQUESTIONS:\n1. It would be interesting to know the training runtime of the newly proposed method compared to GroupDRO and ERM.\n\nLIMITATIONS:\nThe authors properly addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

comments: SUMMARY:
This paper proposes a method to address spurious correlations, building on top of ideas from Group DRO by adding a hierarchical ambiguity set that allows for changing the intra-group mixture in case each group's conditional distributional is not captured. This results in a min-max objective that is is approximated by an inner gradient-ascent step on a latent feature and a mirror-descent update on group weights. The authors test on various vision benchmarks with synthetic spurious correlations.

STRENGTHS AND WEAKNESSES:
Strengths:
- The method leads to better performance when there are severe intra-group shifts and minority subgroup alteration. 
- The framework is a natural extension of prior DRO variants.

Weaknesses:
- The only significant performance gain occurs with the author-devised shifted splits. It would be good to show that this occurs in more complex benchmarks or real-world datasets. There are no experiments on more difficult benchmarks, and the method does not improve performance on the standard benchmarks. Can you demonstrate benefits on more realistic benchmarks and improve the thoroughness of the evaluation?  
- The method adds quite a bit of extra computational overhead along with extra hyperparameters, e.g epsilon is chosen in a bit of an adhoc manner. It seems like a lot of extra complexity that might not be that convincing to use in practice and might not generalize to more realistic situations. Can you quantify the time and memory increase relative to Group DRO?

QUESTIONS:
See weaknesses.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a hierarchical extension of Group DRO that addresses both inter-group and intra-group uncertainties. It also introduces new benchmark settings that simulate realistic minority-group distribution shifts. Experiments show that the proposed method exhibits robustness under conditions where existing robust learning methods fail, while also achieving good performance on standard benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The paper examines an important yet often overlooked question of both inter-group and intra-group distribution shifts. Current methods mostly focus on the inter-group distribution shift. The paper extends Group DRO to a hierarchical ambiguity set that captures both inter-group and intra-group uncertainties.\n- The paper is well-written and easy to follow, with theoretical support clearly stated, datasets well explained, and experiments well illustrated.\n- Extensive experiments are conducted to validate the method.\n- The paper also establishes challenging evaluation scenarios to introduce minority-group distribution shifts that prior work has overlooked, with careful designs.\n\n**Weaknesses**\n\n- While the modified datasets are valuable, the intra-group shifts are somewhat specific and limited (e.g., rotation for CMNIST, species change for Waterbirds, glasses for CelebA). It would be beneficial to discuss the generalizability of the proposed method to other types of intra-group shifts that might be more complex, or to settings where such clear “attributes” for shifting are not readily available.\n\nQUESTIONS:\n- Can the proposed method be applied to other Group DRO variants?\n- What is the computational cost of the method?\n- The variance of the baseline methods on the shifted datasets is somewhat high (e.g., ±8.2 points on LISA). What causes this variability, and could it undermine the reported comparisons? Why does the proposed method not experience this high variance?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper proposes a hierarchical extension to Group Distributionally Robust Optimization that addresses both inter-group and intra-group distribution shifts. The method improves robustness, especially for minority groups, by considering both group-level and within-group uncertainties. Experiments on datasets like CMNIST, Waterbirds, and CelebA show that the method outperforms existing techniques in handling distribution shifts.

STRENGTHS AND WEAKNESSES:
Strength: This paper introduces a new method that improves robustness for both inter-group and intra-group distribution shifts. It is supported by both theoretical foundations and strong experimental validation. 

Weakness: The hierarchical approach may come with increased computational costs. A discussion of computational efficiency and the method’s performance on larger datasets would strengthen the practical applicability of the approach.

QUESTIONS:
1. Could the authors provide a more detailed definition and explanation of "spurious correlation"?
2. What is the computational cost of the proposed method?
3. Are there any statistical guarantees associated with the method, especially regarding its performance in terms of mitigating spurious correlations?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles subpopulation shift by focusing on the more demanding setting of intra-group distribution shifts. It extends GroupDRO with a hierarchical Wasserstein ambiguity set, enlarging the uncertainty region allowed within each group. To evaluate the idea, the authors craft image benchmarks that introduce controlled distribution shifts inside a group (through rotations or selective attribute changes) between training and testing. The proposed method then delivers strong worst-group accuracy across various benchmarks, with especially large gains when inter-group shifts are severe.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n- The introduction of the intra-group distribution shift is meaningful.\n- The proposed method is intuitive and effective. \n- The paper was well-written and easy to follow.\n\nWeakness\n- My biggest concern with this paper is its limited novelty compared to prior structured DRO papers. Hierarchical or multi-level Wasserstein ambiguity sets have appeared before in robust representation learning [1] and in Group-DRO style RL formulations [2]. The present work’s novelty is therefore incremental, where its main distinction is the subpopulation-wise robust learning in vision datasets. Algorithmically, it remains close to vanilla GroupDRO.\n- Standard benchmarks usually include CivilComments-WILDS and MultiNLI; although I don’t expect results on their specially shifted text variants, the paper should still report performance on the original (unshifted) versions of these datasets.\n- The selection of $\\epsilon$ is very heuristic and costly.\n\n[1] Hierarchically Robust Representation Learning. CVPR 2020\n\n[2] Group Distributionally Robust Reinforcement Learning with Hierarchical Latent Variables. AISTATS 2023\n\nQUESTIONS:\n- In the shifted-image benchmarks, does the validation split contain the same perturbations used at test time (e.g., rotated images), those at training time?\n- Could you clarify how you define “inter-group” versus “intra-group” in the preliminaries?\n- How much additional wall-clock training time does your method incur compared with standard GroupDRO?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}