Title: Robust Minimax Boosting with Performance Guarantees
Venue: NeurIPS 2025
Accepted: yes
Keywords: boosting

robust methods

label noise

performance guarantees

supervised classification
Abstract: Boosting methods often achieve excellent classification accuracy, but can experience notable performance degradation in the presence of label noise. Existing robust methods for boosting provide theoretical robustness guarantees for certain types of label noise, and can 
exhibit only moderate performance degradation. However, previous theoretical results do not account for realistic types of noise and finite training sizes, and existing robust methods can provide unsatisfactory accuracies, even without noise. This paper presents methods for robust minimax boosting (RMBoost) that minimize worst-case error probabilities and are robust to general types of label noise. In addition, we provide finite-sample performance guarantees for RMBoost with respect to the error obtained without noise and with respect to the best possible error (Bayes risk). The experimental results corroborate that RMBoost is not only resilient to label noise but can also provide strong classification accuracy.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces RMBoost, a boosting method that directly minimizes the worst-case error probability over an uncertainty set of distributions. This provides a principled approach to achieving robustness against label noise, moving beyond the use of specific surrogate loss functions. The key contributions are: (1) A novel minimax formulation for robust boosting that is shown to be equivalent to a tractable convex optimization problem. (2) Finite-sample performance guarantees that, unlike much prior work, holds for general types of label noise and explicitly characterizes the impact of noise properties.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nPrincipled and Novel Formulation: The core contribution—framing robust boosting as a distributionally robust optimization problem (Eq. 4-5)—is elegant and powerful. It sidesteps the often heuristic choice of robust potential functions (e.g., unhinged, sigmoid) and instead tackles the problem from first principles by directly minimizing the worst-case 0-1 loss. Showing this is equivalent to a tractable linear program (Theorem 1) makes the approach practical.\n\nStrong and General Theoretical Guarantees: The paper provides significant theoretical results. The finite-sample generalization bound (Theorem 2) is a key strength. It holds for general label noise, going beyond the common but restrictive assumptions of symmetric or uniform noise. The bound's explicit dependence on the noise probability (Pnoise) and variance (Varp∗{ρy(x)}) offers a much clearer picture of how different noise characteristics affect performance. Furthermore, the proof of Bayes consistency (Theorem 3) establishes the method's soundness in the limit of expressive base-learners.\n\nEfficient and Scalable Algorithm: The proposed learning algorithm (Algorithm 1), based on column generation, is a standard and well-suited technique for this type of problem (similar to LPBoost). This demonstrates that the theoretically-principled formulation does not come at the cost of practicality. The analysis of the algorithm's convergence and optimality (Theorem 4) is also a valuable contribution.\nWeaknesses\n\nThe weaknesses of this paper are minor and do not detract from the overall quality of the work.\n\nMotivation for Hyperparameter Choice (λ): The performance of the method depends on the radius of the uncertainty set, λ. The authors propose a default value of λ=1/\\sqrt(n), which is well-motivated by standard concentration inequalities. While the sensitivity analysis in Appendix H (Figure 6) is excellent and shows that the method is not overly sensitive to this choice, the main text could benefit from a slightly deeper discussion. For instance, briefly mentioning that this parameter could be tuned via cross-validation and explaining the trade-off it controls (a larger λ implies more robustness but potentially more bias) would be helpful for the reader.\n\nClarity on the Adversarial Noise Model: The adversarial noise model is a good way to test a worst-case scenario. However, its definition relies on a \"reference rule found with the LogitBoost method using clean labels\" (Appendix H, line 611). This implies an oracle with access to the true labels is needed to construct the noisy dataset for the simulation. This is perfectly acceptable for an experimental setup, but it should be stated with perfect clarity in the main text that this is a simulated worst-case attack designed to probe the algorithm's limits, rather than a naturally occurring noise process, to prevent any misunderstanding.\n\nQUESTIONS:\n1. The minimax risk R computed by the algorithm is shown to be a good estimate of the final classification error (Table 1). Could this value be used in a practical way, for example, for model selection? For instance, could one use R to help tune the hyperparameter λ without a separate validation set?\n\n2. The margin constraints in the optimization problem (6), prevent any single data point from having an arbitrarily large positive or negative margin, which intuitively explains the robustness. How does this compare to other regularization techniques in boosting, such as explicitly penalizing the variance of the margins as in [23, 24]?\n\nLIMITATIONS:\nnothing to add", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper formulates a min-max optimization method over uncertainty sets (max), finding a sequence of base learners minimizing the loss. This seems analogous to DRO, though the authors only give brief mention to this. The authors present several analytical results, and a convincing set of empirical results over many baseline models and datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. Overall, the authors present a rigorous set of analytical results on a novel model. (Presumably) the proofs seem to be of high quality and require a good deal of analysis in the appendix lemmas.\n\n2. The method seems to evaluate quite well. The authors primary results demonstrate consistent quantitative results (Table 1), and some simple qualitative and runtime results (in Appendix). So, the method does seem to have strong, practical value. \n\nWeaknesses:\n\n1. Overall, this was an extremely difficult paper to interpret. While I am familiar with both ERM and DRO works, I found the presentation to be very difficult to translate to those works. The notation is very terse, and the authors don't elaborate on base-rules, or the uncertainty set in (5). While I understand this refers to a \"uncertainty set,\" I don't actually understand the intuition of (5): 140-144 define the terms but don't give a high level statement outside of notation. \n\nSince this is a primary element in the minmax formulation (defined after the problem formulation in (4)), I have to reduce my confidence to 1, and adjust the quality and clarity. I further could not interpret the Thm 1 or its proof (including its lemmas) \n\n2. Similar to 1, even the algorithm description is impenetrable to me. This is primarily a wall of symbols with insufficient description. One could scarcely implement this method from this algorithm box.  The authors only cite one subproblem (16). Further use of this might mitigate the presentation complexity. Overall, from Algorithm 1 I couldn't describe the method in any more technical terms than Eq (4). \n\n3. The authors could improve their related work to better taxonomize the gap they are addressing. The authors mix some of the \"background\" notation, which takes away from the presentation of the gap. This really muddles the primary contributions (which are presented in L44, but are less specific in the prior work)\n\nQUESTIONS:\n1. Could you please reformulate both Eq 5 and 7 s.t. I can understand the intuition of these problems? \n\n2. Could you also present Algorithm 1 in more abstraction, so that someone might understand it outside of the dense notation?\n\nLIMITATIONS:\nThe authors don't address this issue. The authors state \"The limitations are discussed over the paper. In particular the theorems state\n697 the hypothesis required.\" but this misses the above intent and makes finding relevant text 'throughout' very difficult. Presumably, the authors state about the scope of their problem: if the assumptions or hypotheses don't hold then their method might fail (i.e. every method's most general limitation).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel boosting framework designed to improve robustness against general types of label noise, called Robust Minimax Boosting (RMBoost). Existing boosting methods provide theoretical robustness guarantees for symmetric and uniform/non-uniform label noise, but they only offer bounds in expectation, not for finite samples. Moreover, the accuracy of these methods often drops significantly. RMBoost, instead, aims to be robust while also maintaining high accuracy. It directly minimizes the worst-case error probabilities over an uncertainty set defined by the training samples and base rules—that is, the base models of the boosting algorithm, where empirical averages can deviate from expected values within a threshold. The paper proposes an alternative formalization of the problem, whose solution is a classifier formed by a linear combination of the base rules. Moreover, the resulting classifier is proven to be optimal for minimizing the worst-case error, and the paper provides and proves a bound on its maximum error in the presence of label noise. The paper also proposes an efficient column generation algorithm for learning RMBoost rules. The experimental evaluation considers four state-of-the-art training techniques, four robust training techniques, and eight relatively small tabular datasets suited for training boosted models, evaluating both symmetric and uniform label noise as well as adversarial label noise. The experiments demonstrate RMBoost’s higher resilience to label noise and its comparable performance to standard boosting algorithms in the absence of noise.\n\nSTRENGTHS AND WEAKNESSES:\nThanks to the authors for submitting this interesting paper. I appreciate the effort to provide theoretically well-grounded algorithms for training machine learning classifiers—particularly boosted models, that are both accurate and robust to noise (random or adversarial), as I believe such properties are highly desirable in practical applications. I think this is a good paper that proposes a solid theoretical contribution, although I believe the experimental evaluation could be improved a bit. I detail below the strengths and weaknesses of the paper and hope that the authors can address my questions during the rebuttal phase.\n\n\n## Strengths\n\nRegarding novelty, this paper is certainly original, as it concerns the problem of finding boosted models that are robust to general label noise (not just specific types of noise), characterizing their robustness as a function of the noise, and ensuring they remain accurate. The theoretical contributions are well-grounded (all proofs are provided in the Appendix) and appreciable, since they demonstrate the existence of a boosted model that minimizes the worst-case error and for which it is possible to theoretically characterize both robustness to general noise and accuracy. Finally, the experimental evaluation is adequate, although it could be improved; it considers several baselines and datasets and shows that the proposed model is more robust and sufficiently accurate compared with previous works.\n\n## Weaknesses\n\nFirst, the experimental evaluation considers both uniform-symmetric and adversarial label noise. However, in Section 2.2, the paper mentions [A], which provides robustness results going beyond symmetric and uniform cases, including cases with symmetric non-uniform noise. This baseline is not considered in the experimental evaluation, even though integrating it would be very interesting since RMBoost is designed to handle general label noise. I suggest that the authors include this baseline or specifically explain why it was not considered.\n\nSecond, I suggest considering more realistic and larger datasets in the experimental evaluation, such as Covtype, HIGGS, and SUSY from the UCI Machine Learning Repository, which have also been previously used to evaluate tree ensembles (e.g., in [B]). This addition would make the experimental evaluation more convincing. If it is not possible to include these datasets, I kindly ask the authors to explain why.\n\n## Minor\n- line 107, page 3 -> should it be $\\phi$ and not $\\phi'$?\n- line 156, page 4 -> \"margins' distribution\" -> \"distribution of the margins\"\n- line 210, page 5 -> \"base-rules' values\" -> \"values of the base rules\"\n\n[A] Ghosh et. al., Making risk minimization tolerant to label noise, Neurocomputing, 2015.\n\n[B] Chen et. al., Robustness Verification of Tree-based Models, in NeurIPS 2019.\n\nQUESTIONS:\n- Why was this baseline [A] not included in the experimental evaluation? Can you add it?\n- Could you explain why larger and more realistic datasets such as Covtype, HIGGS, or SUSY were not included and if you plan to evaluate you approach on them?\n\n**Update after rebuttal**\nThe authors have clarified the omission of baseline [A] in the experimental evaluation, explaining that they attempted to include this baseline but its performance did not reach the same level as the one of the other baselines in the paper. Regarding the second question, the authors provided results on a limited portion of the suggested datasets, that support the effectiveness of their approach. They have also clearly stated the limitation of their method in handling large tabular datasets. I find this more than acceptable, given that the limitation is explicitly acknowledged in the paper.\n\nLIMITATIONS:\nThe paper should include a limitations section clearly stating the proposal's limitations, such as the additional computational time required compared to other training solutions like AdaBoost and the absence of evaluation on large datasets.\n\n**Update after rebuttal**\nThe authors will add this section in the final version of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel method called Robust Minimax Boosting (RMBoost) that aims to enhance the robustness of boosting algorithms against label noise. The authors introduce a minimax approach that directly minimizes worst-case error probabilities and provides finite-sample performance guarantees. The paper also includes efficient algorithms for RMBoost learning and demonstrates through experiments that RMBoost not only withstands label noise but also achieves strong classification accuracy in practice.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**  \n(1) The introduction of RMBoost, which focuses on minimizing worst-case error probabilities, is a novel advancement in the field of robust boosting methods.\n\n(2) The paper provides comprehensive theoretical guarantees, including finite-sample performance and Bayes consistency, which are well-supported by solid proofs.\n\n(3) Experiments are conducted to show that RMBoost outperforms existing methods in the presence of noisy labels and maintains high classification accuracy without noise.\n\n(4) The proposed learning algorithms are efficient and scalable, which can be applied to handle large datasets.\n\n**Weaknesses**   \n(1) While the experiments are thorough, additional testing on more diverse and real-world datasets (e.g., UCI ML datasets) could further validate the robustness and applicability of RMBoost.\n\n(2) The paper assumes certain types of label noise, and the robustness to other, less common types of noise does not seem to be fully explored. Other papers such as [1][2] from the reference list below, also discussed methods for handling other types of label noise, e.g., one-sided label noise. It would be great if the authors can discuss their differences and perhaps include them in the experiments if applicable. \n\n**Reference**:   \n[1] Liu, et al. \"An analysis of boosted linear classifiers on noisy data with applications to multiple-instance learning.\" 2017 IEEE International Conference on Data Mining (ICDM). IEEE, 2017.   \n[2] Luan, et al. \"Multi-Instance Learning with One Side Label Noise.\" ACM Transactions on Knowledge Discovery from Data 18.5 (2024): 1-24.\n\nQUESTIONS:\nsee my comments above.\n\nLIMITATIONS:\nno", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}