Title: Concentration and excess risk bounds for imbalanced classification with synthetic oversampling
Venue: NeurIPS 2025
Accepted: yes
Keywords: Synthetic oversampling; Imbalanced classification; Concentration inequality; Excess risk bound; AM-risk
Abstract: Synthetic oversampling of minority examples using SMOTE and its variants is a leading strategy for addressing imbalanced classification problems. Despite the success of this approach in practice, its theoretical foundations remain underexplored. We develop a theoretical framework to analyze the behavior of SMOTE and related methods when classifiers are trained on synthetic data. We first derive a uniform concentration bound on the discrepancy between the empirical risk over synthetic minority samples and the population risk on the true minority distribution. We then provide a nonparametric excess risk guarantee for kernel-based classifiers trained using such synthetic data. These results lead to practical guidelines for better parameter tuning of both SMOTE and the downstream learning algorithm. Numerical experiments are provided to illustrate and support the theoretical findings.


Peer reviews:
comments: SUMMARY:
This paper presents a rigorous theoretical framework for analyzing SMOTE and KDE-based oversampling techniques in imbalanced classification. The work makes significant theoretical contributions while maintaining practical relevance.

STRENGTHS AND WEAKNESSES:
**Strengths:**

1. Theoretical contributions. Fills important theoretical gaps in understanding SMOTE and its variants and provides mathematical foundation for widely-used oversampling techniques.
2. Extends classical non-parametric risk analysis to imbalanced settings.
3. Comprehensive numerical experiments support theoretical findings.

**Weaknesses:**
1. Enhanced readability. For readers with limited mathematical background, the paper can improve: more intuitive explanations of key concepts, intuitive interpretations of theorems, and visualizations of the theoretical relationships.
2. Expanded experimental analysis. Evaluation on a broader range of datasets, especially extremely imbalanced cases and analysis of scalability with increasing dimensionality.
4. Lack conclusion. The paper ends directly after the experimental part without conclusion.

QUESTIONS:
How does performance scale with increasing dimensionality?

LIMITATIONS:
The author has established important theoretical results in the paper and should discuss how to guide the selection of hyperparameters in practical applications.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a theoretical framework for analyzing SMOTE and its variants, widely used for imbalanced classification. It establishes an exponential inequality bounding the gap between empirical and population risk on synthetic data, and proves that kernel-based classifiers trained on such data can achieve minimax optimal rates. The results offer practical insights for tuning SMOTE and downstream models, supported by numerical experiments.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper establishes an exponential concentration inequality quantifying the discrepancy between empirical and population risk on synthetic minority samples. It also proves minimax optimality of kernel-based classifiers trained with SMOTE-generated data, an important contribution that formalizes a widely used heuristic.\n\n2. The theoretical insights directly inform practical guidelines for tuning SMOTE parameters and the downstream classifier, enhancing the usability of the findings.\n\n3. Establishing minimax rates when learning from synthetic examples is a fresh angle in theoretical ML and adds depth to the understanding of oversampling strategies.\n\nWeaknesses: \n\n1. The theoretical results rely on smoothness assumptions and kernel-based classifiers, which may not capture the behavior of neural networks commonly used in practice.\n\n2. The framework appears focused on binary classification. An extension to multi-class imbalanced settings would broaden applicability and address a more general class of real-world problems.\n\nQUESTIONS:\n1. How do the theoretical guarantees extend to or break down when using modern non-kernel classifiers, such as deep neural networks, which are commonly used in imbalanced classification tasks?\n\n2. Can the authors comment on the robustness of their bounds in high-dimensional settings, where synthetic samples generated by SMOTE may suffer from over-smoothing or introduce unrealistic interpolations?\n\n3. How does the proposed framework handle or adapt to more advanced oversampling strategies (e.g., BorderlineSMOTE, ADASYN), and can similar concentration and excess risk bounds be established for those methods?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a theoretical analysis of synthetic oversampling methods—specifically SMOTE and KDE-based techniques—for binary classification under severe class imbalance. The authors analyze the statistical properties of oversampling from both a concentration and excess risk perspective:\n\t1.\tConcentration bounds (Theorems 1 & 2, pages 5–6) are derived to relate the empirical risk computed on synthetic minority samples to the true population risk. These are obtained for both SMOTE (via nearest neighbors) and KDE (via random samples from an estimated density).\n\t2.\tExcess AM-risk bound (Theorem 4, page 11) is established for plug-in classifiers trained on synthetic data generated via KDE. The bound decomposes the error into bias and variance components and suggests optimal bandwidths for both sample generation and classifier construction.\n\t3.\tPractical guidance is derived from the theory—e.g., justifying small k in SMOTE and the importance of tuning bandwidths in KDE—especially under extreme imbalance n_1/n_0 \\to 0.\n\t4.\tExperiments on both synthetic and real datasets (Section 5) validate these theoretical findings and examine the effect of different oversampling hyperparameters on performance across various classifiers.\n\nThe work unifies existing data-level and model-level approaches to class imbalance, offers new non-asymptotic theoretical tools, and motivates principled hyperparameter selection in oversampling pipelines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\t•\tNovel theoretical analysis: The concentration bounds (Theorems 1 & 2) and excess risk result (Theorem 4) represent significant theoretical contributions that go beyond standard margin-based generalization theory. The analysis draws from nonparametric statistics and covers both SMOTE-like (nearest-neighbor-based) and kernel-based oversampling.\n\t•\tPractical motivation and implications: The work sheds light on popular heuristics such as SMOTE with k=5, showing when such defaults are suboptimal. The theoretical results suggest parameter tuning strategies, particularly in severely imbalanced regimes.\n\t•\tUnifying perspective: The framing of synthetic oversampling as a reweighting scheme that alters the data distribution while maintaining the form of the classifier allows the authors to bridge the gap between model-level and data-level solutions.\n\t•\tEmpirical alignment: The experiments (Section 5, Figures 1–3) are consistent with theoretical insights. For example, small k in SMOTE and tuned bandwidths in KDE improve performance in high-imbalance settings, especially with nonparametric classifiers like KNN.\n\n⸻\n\nWeaknesses\n\nNo excess risk bound for SMOTE:\nWhile Theorem 4 provides an excess AM-risk bound for KDE-based oversampling, no analogous bound is developed for SMOTE. This creates an asymmetry in the paper’s theoretical treatment, despite positioning SMOTE and KDE as parallel methods. The authors acknowledge this in Remark 3 (line 257, page 12), stating that the analysis is difficult, but the lack of even a heuristic or qualitative bound limits the completeness of the comparison.\n\n Optimality claim is narrow in scope:\nThe authors argue in Remark 3 (lines 244–247) that the KDE bound is minimax-optimal in the regime n_1/n_0 \\to 0, but no general lower bound or tightness analysis is provided outside this limit. In more typical imbalance regimes, it is unclear whether the bound remains informative. A discussion of whether the bias–variance tradeoff changes with milder imbalance would help contextualize the theoretical result.\n\n Experiments do not test theory-guided hyperparameter choices:\nAlthough Theorem 4 and the surrounding discussion motivate specific bandwidth choices h, s for KDE sample generation and classification, the experiments (Section 5) rely on heuristic rules (e.g., Scott’s rule, line 326) or cross-validation. There is no test of whether the theory-derived bandwidths improve performance, nor whether cross-validated choices align with theoretical prescriptions.\n\nAssumption 5 may be overly strong:\nAssumption 5 (line 201, page 10) requires that\n\\inf_{x \\in \\mathcal{X}} P(B(x,r)) \\geq C_1 r,\nwhich is unrealistic in high-dimensional Euclidean spaces where measure scales as r^d, not r. Unless the support of P(X|Y=1) lies on a one-dimensional manifold, this assumption is too strong. Clarifying whether this is a typo or whether the support is indeed lower-dimensional would improve the paper’s theoretical transparency.\n\nNo uniform concentration: The concentration inequalities (Theorems 1 & 2) are stated for a fixed Lipschitz function G. As noted in line 165, these do not extend to uniform convergence over function classes. This means the results do not directly guarantee generalization for learned classifiers trained on synthetic data.\n\nQUESTIONS:\n1.\tExtension of Theorem 4 to SMOTE:\nYou mention (line 257) that excess risk analysis for SMOTE is more difficult. Is this due to the discontinuous support of the interpolated samples? Could an approximation (e.g., under smoothness of f^*) yield a rough bound?\n\t2.\tBandwidth mismatch:\nTheorem 4 (line 237) suggests that the optimal KDE bandwidth for sample generation may differ from the optimal classifier bandwidth. Do your experiments support this? Can you comment on how sensitive the AM-risk is to this mismatch?\n\t3.\tLogistic regression stability:\nIn line 335 (page 14), you mention that logistic regression is relatively unaffected by oversampling. Given that Theorems 1–2 still apply in principle, is this due to insensitivity to data augmentation, or model robustness to density perturbation?\n\t4.\tApplicability of concentration bounds to learned classifiers:\nWould it be possible to extend Theorems 1–2 to uniform convergence over a class of Lipschitz classifiers (e.g., KNN or kernel classifiers)? If not, could empirical Rademacher complexity or covering number arguments be layered on top?\n\nSuggestions for Improvement\n\t•\tAdd a brief discussion on how Assumption 5 aligns with the geometry of P(X|Y=1). If the data lives in \\mathbb{R}^d, the measure condition should reflect dimensionality.\n\t•\tConsider including a KDE vs. SMOTE comparison in excess risk terms, even if only heuristic or simulated.\n\t•\tInclude a toy experiment showing the effect of varying h and s in accordance with Theorem 4, to highlight the practical relevance of the theory.\n\t•\tClarify whether your theoretical guidance can improve hyperparameter tuning in practice compared to cross-validation.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops a theoretical framework to analyze the behavior of SMOTE and related synthetic oversampling methods in imbalanced classification. The authors establish an exponential inequality to characterize the gap between empirical risk on synthetic samples and the true population risk on the minority class. Furthermore, they demonstrate that a kernel-based classification rule trained on synthetic data can achieve the minimax rate of convergence. The paper claims these theoretical findings lead to practical guidelines for better parameter tuning of both oversampling techniques and downstream learning algorithms, with numerical experiments provided to support the theoretical findings.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- The paper tackles the significant and practical problem of imbalanced classification, which is prevalent across many real-world machine learning applications.\n- The paper develops a theoretical framework for understanding SMOTE and related oversampling methods, establishing exponential inequalities for the gap between empirical and true population risk on synthetic data, and demonstrating that a kernel-based classifier trained on synthetic data can achieve minimax optimal rates of convergence.\n\n## Weaknesses\n- The abstract claims the theoretical findings \"lead to practical guidelines for better parameter tuning of both SMOTE and the downstream learning algorithm\". However, the introduction does not elaborate on what these specific guidelines are, making this core claim feel unsubstantiated.\n- Assumption 1 imposes a strong condition on the minority class distribution requiring a minimum probability within any Euclidean ball and boundedness of its support. This assumption, particularly the minimum probability condition, is quite stringent and is unlikely to hold for many real-world minority class distributions. Relying on such a strong assumption significantly limits the generalizability and practical applicability of the subsequent theoretical proofs and results.\n- Section 3.2 provides detailed excess risk bounds for a specific kernel smoothing plug-in classifier, forming a cornerstone of the paper's theoretical contribution and leading to optimal bandwidth choices for KDE oversampling. However, in Section 4, the authors do not use the theoretically analyzed kernel smoothing classifier. Instead, they switch to a KNN classifier, justifying this only with \"it is a slight variation of kernel smoothing plug-in classifiers\" and \"for comparison purposes.\" While KNN and kernel methods share conceptual links, they are distinct algorithms with different statistical properties and convergence behaviors. Theoretical guarantees derived for one algorithm cannot be directly and unprovenly generalized to another.\n- The paper's title and abstract both claim to analyze SMOTE and related methods. While concentration inequalities are provided for both SMOTE and KDE , the more critical excess risk bound analysis only presents results based on KDE oversampling, conspicuously lacking a corresponding analysis for SMOTE. This incompleteness implies that the theoretical framework does not fully cover SMOTE, despite its prominent mention in the paper's stated scope.\n- The experimental results show that once cross-validation is used to select the classifier's parameters, the influence of the oversampling parameter becomes \"minimal\" . This finding substantially undermines the value of the paper's core claim to provide \"practical guidelines for better parameter tuning of both SMOTE and the downstream learning algorithm.\"\n\nQUESTIONS:\n- SMOTE is a classic algorithm with extensive research over the years, which has identified many shortcomings, such as its inability to fundamentally increase information, its failure to address issues of insufficient representation or noise in the original minority class, its sensitivity to noise and outliers, its tendency to cause boundary overlap, the curse of dimensionality, its inability to handle sub-cluster structures, and its limited performance in extreme imbalance ratios. Does the theoretical framework proposed in this paper address these known problems, or does it provide any specific insights or directions for resolving them?\n- Does Assumption 1 implicitly require that minority class samples are drawn independently and identically distributed from the overall distribution, or does the framework account for other sampling scenarios?\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-05"}