Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper extends the Mixup method by utilizing a dynamic uncertainty score as importance weight for Mixup samples. The uncertainty score (for a given x_i) is obtained by tracking the average prediction error of x_i through out the training trajectory, as an indication of how "hard" it is to learn this example.

The method is evaluated on 4 datasets with subpopulation shift. The proposed method outperforms others on "worst case" accuracy, but under-performs the baseline (ERM) on average accuracy.
strengths_and_weaknesses: The overall writing is clear and easy to follow. However, I feel the originality and significance of the proposed method is thin. It is a straightforward extension of Mixup by combining importance weight and uncertainty score. The paper didn't explore or discuss other choices of this "uncertainty score" as importance weight. There's a whole literature on influence score (how influential is an example to the learning process), which can also play a similar role here.

The evaluation results are ok but not impressive. I'm not convinced that a better "worst" accuracy but degraded avg accuracy is a significant contribution, given all the extra compute required by UMix. Perhaps other reviewers could convince me how this is important and relevant to the research community.

questions: The uncertainty score u_i is updated every epoch, right? i.e. as the training epoch (T) progresses, you will get updated value of u_i(T). If so, please reflect that in Algorithm 1.

In Algorithm 2, you don't have to "save" all the predictions for every example at every epoch. Keeping a running average of $\kappa(y_i, f_{\theta_t}(x_i))$ over epoch t for each $x_i$ would be more memory efficient, right?
limitations: NA
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a new mixup method (UMIX) based on uncertainty estimation and importance weighting for subpopulation shift. The authors also prove theoretically that UMIX can achieve better generalization error bounds over IW methods without mixup. 
strengths_and_weaknesses: **Strengths**

Originality: Though the ideas of uncertainty estimation, importance weighting and mixup are not new, the combination of these ideas are novel and intriguing.

Quality: The authors proved empirically and theoretically that their method is better than other baselines. The results are rather solid.

Clarity: The paper is well-written and easy to follow.

Significance: The paper provides a new method for augmenting mixup via uncertainty.

**Weaknesses**

I have not identified major weaknesses of this paper. However, I do have some concerns and listed them in the “Questions” part below.
questions: 
1. While the idea proposed by the paper is straightforward, there are some missing details that are essential to justify the method’s effectiveness. For example, UMIX can be seen as a new data augmentation method/loss function, but how is the training procedure like? Is it similar as ERM, or is it adopted from some other algorithm. Adding this discussion would help to verify whether the comparisons are fair or not.

2. The choices of parameters are missing and including them (such as T_{s} and T in Algorithm 2) would be nice.

3. In Figure1, why is there hardly any samples with uncertainty from 0.3 to 0.8? Is it an effect of KDE, or is it due to the attributes of the datasets?

Minor questions:

1. Line59: contirbution $\rightarrow$ contributions
limitations: The authors have thouroughly discussed the limitations and social impacts of their work in the appendix.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors tackle the problem of subpopulation shift without subpopulation labels. They propose a method called uncertainty-aware mixup (UMIX), which augments regular mixup with importance weights computed based on predictive uncertainty. The authors benchmark their method against a large set of group aware and group unaware baselines on standard datasets, finding that their method shows competitive worst-case accuracy. Finally, the authors provide theoretical insight into why their method outperforms traditional importance weighting-based methods. 
strengths_and_weaknesses: Strengths:
- The paper is generally well-written and easy to follow.
- The authors compare against a wide range of methods on the standard datasets, and report their results with standard deviations (in the appendix).
- The method performs competitively relative to the baselines.

Weaknesses:
1. Though the method is novel, is it only an incremental change from what is proposed in JTT and LISA.
2. The authors should provide more empirical justification for the uncertainty measure that they use. Though it is inspired by SWAG, it does seem quite different from SWAG. How does the measure of uncertainty compare with SWAG, and why can SWAG not be used directly in this method (by sampling from the posterior over weights)?
3. It seems that the uncertainty measure takes both epistemic and aleatoric uncertainty into account. Would it be more ideal to weight based on epistemic uncertainty only, as some groups may inherently have more error than others?
4. It would be interesting to explore how the uncertainty measure (Fig. 1) changes over time during the process of running Algorithm 2. Does this empirically support the intuition from Lines 195-198?  
5. For Camelyon-17, I would suggest showing performance on the source domain test set along with the target domain results.
6. It would be ideal if the authors could test on a couple more domain generalization datasets, as this seems relatively unexplored (only Camelyon-17), and many of the baseline methods are targeted towards the domain generalization setting. 
7. The authors should state the range of hyperparameters tuned, as well as the total tuning budget, in Appendix B3.

questions: Please comment on points 2-6 from above. 
limitations: The authors adequately address the limitations in the appendix.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a importance reweighting scheme for mixup which promotes a fair training balance between subgroups in the data. The does not require group
labels during training.
strengths_and_weaknesses: # Strengths

- The paper is well placed among recent works in both mixup, and importance reweighting methods.
- The method performs well without group information during training which is impressive
- The theoretical analysis provides a nice
- Overall I think this is a solid work, but I some clarification on the issues raised in the following sections.

# Weaknesses

- L190 says that you assume a reasonable importance weight is linearly
positively correlated to the corresponding uncertainty. What is a 'reasonable'
importance weight, and on what basis can this assumption be made? I realize
this is shown in Figure 1, but there should be some provided justification or
reference to the fact that this will be empirically tested later.

- How does UMIX perform without the group labels in the validation set? If the
method is assumed to be oblivious to the group information, then it may not be
reasonable to assume that they are always in the validation set.

# Minor

- L113: BNN's is proposed --> BNN's have been proposed.
- L115: Performing ensembles --> ensembling them.
- L122: upweight mixed samples thus can encourage --> upweight mixed samples, and thus can encourage.
- L158: weights of UMIX are posed --> What does this mean?
- L178: $p(theta; \mathcal{D})$ --> shouldn't this be $p(\theta | \mathcal{D})$?
- L180: with $T$ Monte Carlo Sampling --> with $T$ Monte Carlo samples
- L193: in practice we could set $c = 1$ --> Does this mean that you do set $c=1$ in practice? It should be clearly stated if that is the case.
- L295: proportion is --> proportion being
- L296: in specific --> specifically

questions: - Section 5 states "Thus our bound can be tighter when
315 the intrinsic dimension of data is small (i.e., rank(Σ) ≪ d)." Is this really useful? Can you point to some intuition or concrete examples of when this would usefully lower the bound? When can we reasonably expect that to not be full rank? On the same note, is this referring to the dimension in the input space, the output space before the loss function, or something else entirely?

- Are you actually MC sampling the saved SWAG models by building the Gaussian
posterior throughout the training process in order to get the importance
weights? OR are you just using the snapshots to get the model disagreement?

- Section 3.1 says that the $N$ training examples are i.i.d. and sampled from
$P$, but then immediately says that the subpopulation $g$ follows the
distribution $P_g$. Shouldn't the assumption then be that the dataset is not
i.i.d. and only conditionally i.i.d. given $P_g$?
limitations: The authors have addressed both in the appendix.
review_title: null

Meta-review: metareview: The reviewers unanimously agreed here that incorporating uncertainty scores as importance weights for mixup, and empirically the authors' method seems to lead to substantial quantitative performance improvements. I think the heuristic use of the model's parameter history to estimate uncertainty is reasonable. However, while I am recommending acceptance, the SAC and I feel that there are a few concerns that arose in discussion we'd urge the authors to address in the camera ready version. In particular:

1. The authors should clarify whether the approximation in (6) actually converges in any meaningful sense to the posterior expectation in (5). My initial impression is that the answer to this is probably no. While the discussion on lines 195-202 reasonably motivates the use of equation 6, I think motivating this approach through a posterior expectation in equation 5 may slightly oversell the rigor of equation 6, at least as currently described. The authors should consider address whether the approximation is good by running an experiment and comparing their approximation to equation (5) to a monte carlo approximation on a toy scale model where this is feasible, which would more carefully isolate whether (6) is an approximation to (5) or a heuristic.

2. Some of the advantages of the authors' approach are a bit overstated. For example, using SWAG with optimizers other than SGD is fairly common in practice. While obviously this doesn't diminish the authors' results, I think it's worth fixing to ensure technical correctness here in the camera ready.



confidence: Less certain
Decision: Accept