Title: Revisiting Agnostic Boosting
Venue: NeurIPS 2025
Accepted: yes
Keywords: Weak-to-Strong Learning

Agnostic Learning

Sample Complexity

Margin-based Analysis

Boosting
Abstract: Boosting is a key method in statistical learning, allowing for converting weak learners into strong ones.
While well studied in the realizable case, the statistical properties of weak-to-strong learning remain less understood in the agnostic setting, where there are no assumptions on the distribution of the labels.
In this work, we propose a new agnostic boosting algorithm with substantially improved sample complexity compared to prior works under very general assumptions.
Our approach is based on a reduction to the realizable case, followed by a margin-based filtering of high-quality hypotheses.
Furthermore, we show a nearly-matching lower bound, settling the sample complexity of agnostic boosting up to logarithmic factors.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper revisits the classic problem of agnostic boosting. Leveraging recent techniques by Hopkins et al. [2024] , the authors propose a new agnostic boosting algorithm which, under some general assumptions, improves on the sample complexity of existing algorithms. Finally, the authors complement their upper bound with a near matching lower bound.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n- This paper is extremely well-written and organized. The authors do a great job at explaining their algorithm and providing the right balance between intuition and technical details. \n- The results seem novel and provide a clear contribution to the literature on agnostic boosting. To the best of my knowledge, this is the first time that the \"all possible labelings\" approach of Hopkins et al. [2024] has been used for agnostic boosting. \n\nWeaknesses: I don't have much to say beyond a few presentational choices. \n\n- The Theorem statements in the introduction are a bit dense. I think it would be better to replace these with informal Theorem statements that capture only the point that is trying to be made, perhaps by dropping some of the quantifiers. \n- The majority of the main text is dedicated to proving the upper bound in Theorem 1.3. However, in the introduction, Theorem 1.2 is presented before Theorem 1.3. I think it would be better to swap the order of these two Theorems and place more emphasis on Theorem 1.3. Or, I think it would be beneficial to include more intuition about Theorem 1.2 in the main text. \n- As written, it is a bit difficult to contextualize the results of this paper with existing results in agnostic boosting. The authors attempt to do this in Section 1.1. I think it might be nice to include a table that compares your upper/lower bounds with existing results when the assumptions and weak learning definitions line up.\n\nQUESTIONS:\nIn line 270, the authors justify the need for the filtering step by arguing that the size of $B_1$ is exponential in the sample size. However, instead of taking *all* possible labels of the training sample, it seems to me that one can just loop over all possible behaviors of the class $F$ on $x_1, ..., x_m$. If $F$ has finite VC, $B_1$ now has size polynomial in $m$ (compared to exponential). It would be great if the authors could comment on this and what sort of upper bounds this approach would yield.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-28"}

comments: SUMMARY:
This paper presents a new agnostic boosting method with comprehensive theoretical analysis. The new algorithm can convert any non-trivial agnostic weak learner into a strong learner. The core contributions are (1) the proposed algorithm achieves a significantly improved sample complexity upper bound compared to prior work under very general assumptions. (2) the paper establishes a nearly matching lower bound on the sample complexity for agnostic boosting, effectively settling the statistical complexity of the problem up to logarithmic factors.

STRENGTHS AND WEAKNESSES:
Strengths:
1. Establishes nearly tight upper and lower bounds on the sample complexity of agnostic weak-to-strong learning, which advances the fundamental understanding of boosting in the challenging agnostic setting
2. The proposed algorithm and its analysis operate under remarkably mild and general assumptions

Weakness:
1. The proposed algorithm is computationally inefficient and could be hard to implement in practice.
2. There is a lack of experiments to demonstrate the practice usefulness of this method.

QUESTIONS:
1. Do you have any experiments, including synthetic ones, to show the proposed modified Adaboost in Algorithm 1 works in practice?

2. Can you further discuss how this work relates to / extends Hopkins et al. (2024) (in line 195)?

3. As one limitation is the algorithm's computational inefficiency steming from iterating over all possible re-labelings. Are there any foreseeable approaches or heuristics to make this algorithm (or a similar one) computationally feasible while substantially preserving its strong statistical guarantees?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel agnostic boosting algorithm that significantly improves sample complexity under general assumptions. The proposed method reduces the problem to the realizable case and employs a margin-based filtering step to select high-quality hypotheses. Additionally, the paper establishes a lower bound on the sample complexity of agnostic boosting, matching the upper bound up to logarithmic factors.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:  \n(1) The paper presents a new agnostic boosting algorithm that offers substantial improvements in sample complexity compared to previous works, which is a novel contribution to the community.\n\n(2) The authors provide thorough and solid theoretical analysis, including a lower bound on the sample complexity that aligns closely with the upper bound.\n\n(3) The approach is based on very general assumptions, making it broadly applicable across different scenarios, which is indeed an advantage of the paper.\n\n(4) The paper includes detailed proofs and theoretical justifications for the proposed algorithm and its properties, which facilitates people's understanding of the proposed method.\n\n**Weaknesses**:  \n(1) While the algorithm is statistically near-optimal, I am not sure about its computationally efficiency, which may limit its practical applicability.\n\n(2) I understand this paper is mainly focused on the theory. Although not always required, it might be a bit beneficial if the paper also provides empirical results or experiments to validate the theoretical findings and demonstrate the algorithm's performance in practice. For example, some other boosting-related theoretical papers also have some (though not too many) experiments included in their papers (see references 1-3 from the list below).\n\n**Reference**:   \n[1] Ghai, et al. \"Sample-Efficient Agnostic Boosting.\" arXiv preprint arXiv:2410.23632 (2024).   \n[2] Liu, et al. \"An analysis of boosted linear classifiers on noisy data with applications to multiple-instance learning.\" 2017 IEEE International Conference on Data Mining (ICDM). IEEE, 2017.   \n[3] Bootkrajang, et al. \"Boosting in the presence of label noise.\" arXiv preprint arXiv:1309.6818 (2013).\n\nQUESTIONS:\nsee my comments above.\n\nLIMITATIONS:\nno", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new boosting algorithm in the agnostic setting where labels are arbitrary, with no realizability assumption. Their method reduces agnostic boosting to the realizable case through a margin‑based filtering step that selects high‑quality hypotheses, achieving near-optimal sample complexity (up to logarithmic factors). This paper focuses only on statistical and information-theoretic\naspects of the problem, without constructing a computationally efficient algorithm.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- Boosting in the agnostic setting is a problem known to be challenging. The proposed boosting algorithm, using the technique from a recent work to reduce the agnostic problem to the realizable one, is a fresh approach, offering tighter sample complexity guarantees in the agnostic setting.\n- The authors adopt a highly general notion of agnostic weak learners that encompasses many variants from prior literature, providing a unifying framework and fair comparison.\n- This paper also proves a lower bound on the problem, demonstrating that the proposed method achieves the optimal sample complexity bound up to logarithmic factors.\n- The paper is clearly written with enough information about the background and existing results. The mathematical notations are a bit dense, but the authors did a good job explaining the intuitions.\n\n**Weakness**\n- The proposed algorithm is not computationally efficient due to the exhaustive relabeling step and enumeration over exponentially many hypotheses. While the authors acknowledge this, some discussion on possible heuristics or approximations would be helpful.\n- While the bounds are tight asymptotically, the constants hidden in the big-O notation may be large due to the fat-shattering terms. There is no discussion on this.\n\nQUESTIONS:\n1. The proposed algorithm is acknowledged to be computationally inefficient. To what extent can the current theoretical results be extended to efficient algorithms without significant degradation in sample complexity?\n2. The lower bound (Theorem 1.2) includes an extra logarithmic factor. Is this believed to be an artifact of the proof, or is there any evidence that it reflects an actual complexity barrier?\n3. I suggest that the authors put a table summarizing the existing works and the proposed method. The table can include sample complexity, whether it is computationally efficient, and any other notable things, so that the readers can more easily understand the current status of this problem.\n4. What is $D_t^{m_0}$ in Algorithm 1? It is not clearly defined.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}