Title: Undersampling is a Minimax Optimal Robustness Intervention in Nonparametric Classification
Authors: Niladri Shekhar Chatterji
Saminul Haque
Tatsunori Hashimoto
Venue: NeurIPS 2022
Keywords: theory of distribution shift
nonparametric classification
undersampling

Abstract: While a broad range of techniques have been proposed to tackle distribution shift, the simple baseline of training on an \emph{undersampled} dataset often achieves close to state-of-the-art-accuracy across several popular benchmarks. This is rather surprising, since undersampling algorithms discard excess majority group data. To understand this phenomenon, we ask if learning is fundamentally constrained by a lack of minority group samples. We prove that this is indeed the case in the setting of nonparametric binary classification. Our results show that in the worst case, an algorithm cannot outperform undersampling unless there is a high degree of overlap between the train and test distributions (which is unlikely to be the case in real-world datasets), or if the algorithm leverages additional structure about the distribution shift. In particular, in the case of label shift we show that there is always an undersampling algorithm that is minimax optimal. While in the case of group-covariate shift we show that there is an undersampling algorithm that is minimax optimal when the overlap between the group distributions is small. We also perform an experimental case study on a label shift dataset and find that in line with our theory the test accuracy of robust neural network classifiers is constrained by the number of minority samples.

TL;DR: We show that under distribution shift in the worst case the undersampling algorithm is optimal, and that the performance of any algorithm is fundamentally constrained by the lack of minority group samples.

Review discussion:
Strengths:
* The authors ask an important question about min-max optimality of undersampling.
* The paper is nicely written, nicely exposed, and the results appear to be novel to me.
* The upper bounds for undersampling are intuitive, and the analysis is nicely done.
Weaknesses:
* The main weakness of the paper lies in the constant $c$ in Theorem 4.1. A closer look at the proof on page 19, line 514, shows that the constant $c = e^{-\frac{n_{maj}}{3 n_{min}}},$ which *does* depend on both $n_{min}$ and $n_{maj}$ as opposed to the general story of the paper.
In fact, $c \to 0,$ as $n_{maj} \to \infty$ which contradicts parts of the story of the paper. The only way to keep $c$ to be a constant as $n_{maj} \to \infty$ is to let $n_{min} \to \infty$ with $\frac{n_{maj}}{n_{min}} \to \eta$ for constant $\rho$ in which case the result loses its interestingness because there is no distinguishable difference between $\frac{c}{n_{min}^{1/3}} = \frac{c_2}{n_{maj}^{1/3}}$ for some other constant $c_2$ and the theorem doesn't tell us anything non-trivial about class imbalance.
In summary, the lower bound is vacuous unless $\frac{n_{maj}}{n_{min}} \to \eta,$ in which case the lower bound is trivial.
* A similar weakness applies to Theorem 4.2 for any $\tau \in (0,1)$, where $c$ vanishes as $n_{maj} \to \infty$. This can be seen based on the equation on line 637 which vanishes as $n_{maj} \to \infty$.
Hence, similarly the lower bound is either vacuous or trivial in this setting as well.
* Since the metric of interest is min-max excess risk, I wonder why the authors didn't consider a min-max baseline (instead of ERM) in this setting;
Also smoothened versions of such min-max loss for better generalization, e.g., tilted loss (Li et al 2021):
Li, T., Beirami, A., Sanjabi, M. and Smith, V., 2021. Tilted empirical risk minimization. ICLR.
I also wonder if these baselines would be subject to the same empirical observations of Figure 2.
* Can you please elaborate on the main weaknesses listed above with respect to the lower bounds in Theorems 4.1 and 4.2? There might be a way to fix this by updating the construction that is considered and working through the details but I was not able to immediately see a way out.
* The proof of Theorems 4.1 and 4.2 are dense. It would be best to give an outline of how the proof goes first.
For example, I was confused when $K$ was introduced for the first time, as it was not clear how this was going to be used.
The main limitation of this paper is that Theorems 4.1 and 4.2 in their current form are either vacuous or trivial (depending on how $n_{maj}$ is related to $n_{min}$), and hence they do not support the narrative of the paper. It is not clear to me how this might be fixed.
No
We thank the reviewer for their careful and thorough review! We would like to quickly address the technical flaws regarding the constant $c$ identified by the reviewer before addressing the other questions raised by both this reviewer and others. We have uploaded a new revision of the supplementary material correcting a typo and providing additional details in the proof to address the concerns raised by the reviewer.
> The main weakness of the paper lies in the constant  in Theorem 4.1. A closer look at the proof on page 19, line 514, shows that the constant $c = e^{-\frac{n_{maj}}{3n_{min}}}$ which _does_ depend on both $n_{maj}$ and $n_{min}$ as opposed to the general story of the paper.
In fact, $c \to 0$ as $n_{maj}\to \infty$ which contradicts parts of the story of the paper. The only way to keep $c$ to be a constant as $n_{maj} \to \infty$ is to let $n_{maj} \to \infty$ with $\frac{n_{maj}}{n_{min}} \to \eta$ for constant $\rho$ in which case the result loses its interestingness because there is no distinguishable difference between $\frac{c}{n_{min}^{1/3}}=\frac{c_2}{n_{maj}^{1/3}}$ for some other constant $c_2$ and the theorem doesn't tell us anything non-trivial about class imbalance.
In summary, the lower bound is vacuous unless $\frac{n_{maj}}{n_{min}}\to \eta$ in which case the lower bound is trivial.
There was an unfortunate typo in the proof on page 19 on line 514 which led to a mistake.
However, we note that correcting this typo ensures that our theorem statement is correct and is as stated.
We invoke Lemma B.2 from above where the upper bound is $\frac{1}{18K} - \frac{1}{144K}\exp\left(-\frac{n_{min}}{3K^3}\right)$ that involves $n_{min}$ rather than $n_{maj}$ which we copied into the line 514 incorrectly. With this _correct bound_, our choice of $K = \lceil n_{min}^{1/3}\rceil$  leads to the constant $\frac{\exp(-\frac{n_{min}}{3\lceil n_{min}^{1/3}\rceil^3})}{288} \ge c$ where $c$ is a universal constant *independent* of $n_{maj}$ and $n_{min}$.
We would invite the reviewer to check page 19 on line 516 in the latest version of our supplementary material where we have corrected this typo and hence the proof.
> A similar weakness applies to Theorem 4.2 for any $\tau \in (0,1)$, where $c$ vanishes as $n_{maj}\to \infty$. This can be seen based on the equation on line 637 which vanishes as $n_{maj} \to \infty$.
Hence, similarly the lower bound is either vacuous or trivial in this setting as well.
Likewise, we do not believe that this weakness exists in Theorem 4.2 and that $c$ is indeed an absolute constant independent of $n_{maj}$ and $n_{min}$. For our minimax lower bound, the construction of the distributions and hence $K$ can depend on both $n_{maj}$ and $n_{min}$, therefore when we maximize the RHS in line 637 with respect to $K$, which leads to the choice $K = \lceil(n_{min}(2-\tau)+n_{maj} \tau)^{1/3}\rceil$, the resulting constant $c$ is again *independent* of $n_{maj}$ and $n_{min}$. We have added additional steps in the proof on lines 639-640 on page 26 to help clarify this. We hope that this addresses your concern here.
We hope that this clarifies the technical questions of correctness raised in your review. We would be very happy to engage and clarify if you have any further questions!
Finally, we shall be happy to add a proof sketch before the start of the proofs to provide a roadmap to the reader as you suggested.
We would like to reach out to ask if our comment and edits in the new version of the supplementary material has addressed your concerns. We would be very happy to further clarify or answer any further questions.
Dear authors:
Thank you!
While I haven’t been able to reverify whether the updated proof supports the narrative yet, I think my question on why a min-max baseline has not been considered given the chosen metric is a min-max objective is left unaddressed unless I’ve missed something.
Thanks,
Reviewer JtaE
Thank you for your reply! We had indeed forgotten to address the question about the minimax baseline in our initial reply.  We would like to add a quick clarification. In DRO and tilted loss type methods, the minimization player aims to identify a single, robust predictor with the goal of using this predictor. In our minimax analysis, the goal is different. We are interested in the question of how well _any_ algorithm can do on this problem, so our minimization player operates over all learning algorithms (incl algorithms such as DRO and tilted loss) and we show that no single algorithm can perform better than undersampling.
Our analysis suggests that all learning algorithms including robust learning procedures are limited by minority group samples, and this prediction is borne out in Idrissi et al. (2022), who tried the group-DRO algorithm and found that undersampling algorithms were competitive with them  (see Table 1). We therefore expect similar trends to hold in our experiments as well.
That being said, we are currently in the process of adding experiments with the tilted loss and group-DRO algorithms and we will be happy to add them in as they are completed!
Dear Authors:
Thanks for the revisions, and for your patience as I went through them. I think your revisions have made it much easier to navigate the proofs. I can now see that the proof of Theorem 4.1 was suffering from a typo which is now fixed. I can also see that Theorem 4.2 does not have any issues.
Here are some remaining minor suggestions:
* Please make the constants in Theorems 4.1 and 4.2 explicit. I think these constants of reasonable magnitude (compared to many theoretical results), and it is probably best to highlight this fact.
* Line 457, *We continue to define* is repeated.
As I articulated in the summary of the strengths in the original review, I think this is a solid and significant piece of work, and I believe that it should be highlighted in the conference.
As such, given that there are no concerns on correctness, I adjusted the evaluation score to strong accept (8).
Thanks,\
Reviewer JtaE
We would like to thank the reviewer again for their thorough review and multiple suggestions!
We would like happy to make the constants explicit in Theorems 4.1 and 4.2, and shall also correct the typos identified.
Also, as promised earlier we are in the process of drafting a proof sketch and adding in the minimax baselines.
Strength
- The paper presents lower bounds for the lower bounds for the minimax excess risk for label shift and group-covariate shifts.
- Their proposed "Undersampled Binning Estimator" are optimal (or optimal in certain scenarios) up to a constant.
- The explanations and intuition provided of the setting and results.
Weakness
- There are a few symbols which are not defined and some components in the experiments section which are not clear.
Questions / Comments / Suggestions
1. Is the 1-Lipschitz assumption common? It would be useful for a discussion on if this assumption appears in practice or if its is a common technical assumption.
2. Short definitions / descriptions of $\mathrm{TV}$ and VS loss would be useful for completeness
3. There are short notes about the generalization of Theorem 4.1 and 4.2 for higher dimensions. Does the "1/3" to "1/3d" also hold for corresponding Theorem 5.1 and 5.2?
Minor / Typos
- There seems to be a few typos / errors in the Appendix, the set of equation below Line 483: (1) on the first line the summation seems to be misplaced; and (2) the second last line seems to be incorrect / should be removed. This doesn't invalidate the proof.
- In the Appendix, equations below Line 514: "$n_{maj}$" -> "$ n_{min}$"
Assumptions / limitations of analysis is clear.
No
We would like to thank the reviewer for our careful reading of our paper! We appreciate the feedback and will incorporate the many constructive suggestions made in the review and correct the typos identified.
> Is the 1-Lipschitz assumption common? It would be useful for a discussion on if this assumption appears in practice or if its is a common technical assumption.
It is indeed a common technical assumption made in the literature of non-parametric classification/regression. It is also possible to extend our theoretical guarantees to the case where the distributions are L-Lipschitz. We will add a comment about this when we introduce the assumption.
> There are short notes about the generalization of Theorem 4.1 and 4.2 for higher dimensions. Does the "1/3" to "1/3d" also hold for corresponding Theorem 5.1 and 5.2?
Yes, it is possible to generalize Theorem 5.1 and 5.2 to hold in higher dimensions where the exponents become “1/3d”. We shall comment about this.
Thanks you for the response.
I am happy with the response provided.
I believe the important point to address are those concerning the comments on correctness as brought up by reviewer JtaE (the first point being the second typo I spotted).
I do not intend on changing my numeric scores before further discussion with reviewers.
The paper targets an important problem and the paper does a very good job pointing out both why undersampling does wonders in many cases and also why further work toward improve OOD should assume more structure. Good work!
I think many existing methods do infact use more structure than is allowed by the theory in the paper. Can the authors comment on what kinds of structure in the existing OOD literature helps do better than the proved lower-bound?
See questions.
No
We would like to thank the reviewer for their careful review and positive comments!
> I think many existing methods do infact use more structure than is allowed by the theory in the paper. Can the authors comment on what kinds of structure in the existing OOD literature helps do better than the proved lower-bound?
Our theory predicts that any problem structure that enables transfer from the majority group to the minority group would help. Some examples of this are as follows:
- overlap between the class conditional distributions ($p(x \mid y = 1)$ and $p(x\mid y=-1)$) in the label shift setting or between $p(y \mid x)$ in the group covariate shift setting;
- additional unlabeled data as in domain adaptation setting;
- parametric assumptions as is commonly made in the linear causal models.
As you suggested we would be happy to add these examples in the paper.

Meta-review:
role: AC
data: metareview: This paper provides bounds and some empirical results on specific distribution shift scenarios, where there is a majority and minority group (group identity is known to the learner) and while at training time data from the two groups is unbalanced at the test distribution is assume to be a balanced mixture. The paper considers two specific scenarios, one where a type of covariate shift and one where a type of label shift is induced.

This submission considers the non-parametric setting, a one-dimensional feature space (examples are assumed to be from [0,1] x {-1, +1}) and Lipschitz continuity for the conditional label probability function. It then analyzes error rates for the above two scenarios and also provides some empirical confirmation that "undersampling", namely subsampling from the majority group so that the two groups are balanced at training time, is minimax optimal.

Given the large literature on learning bounds for domain adaptation (for parametric, but also for non-parametric learning) this submission appears suprisingly unaware of these existing studies and bounds. This literature should be acknowledged and compared to  in details before publication (if the results in here are not in fact just specific cases of known results). I can therefore not support acceptance, despite the reviewers positive recommendations.

Steve Hanneke, Samory Kpotufe:
On the Value of Target Data in Transfer Learning. NeurIPS 2019: 9867-9877

Samory Kpotufe, Guillaume Martinet:
Marginal Singularity, and the Benefits of Labels in Covariate-Shift. COLT 2018: 1882-1886

Christopher Berlind, Ruth Urner:
Active Nearest Neighbors in Changing Environments. ICML 2015: 1870-1879

Shai Ben-David, Ruth Urner:
Domain adaptation-can quantity compensate for quality? Ann. Math. Artif. Intell. 70(3): 185-202 (2014)

Shai Ben-David, Ruth Urner:
On the Hardness of Domain Adaptation and the Utility of Unlabeled Target Samples. ALT 2012: 139-153

Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, Jennifer Wortman Vaughan:
A theory of learning from different domains. Mach. Learn. 79(1-2): 151-175 (2010)

Shai Ben-David, Tyler Lu, Teresa Luu, Dávid Pál:
Impossibility Theorems for Domain Adaptation. AISTATS 2010: 129-136

Shai Ben-David, John Blitzer, Koby Crammer, Fernando Pereira:
Analysis of Representations for Domain Adaptation. NIPS 2006: 137-144


role: PC

Decision: Reject