Multiclass Boosting and the Cost of Weak Learning

Abstract: Nataly Brukhim, Elad Hazan, Shay Moran, Indraneel Mukherjee, Robert E. Schapire

Reviews and author responses:

 Thanks for the great questions. 
To answer the first question, our lower bound result shows that there exists a hypotheses class H, and a weak-learner satisfying the WL condition, such that for *any* possible boosting algorithm B, when used in this setup
it will have to suffer a cost that is polynomial in k (the number of labels). Since gradient boosting algorithms are ultimately instances of B, the lower bound holds.

We refer the reviewer to Section 5.1 in which the general boosting schemes are defined, and Theorem 7 for the formal statement of the lower bound.
 
Regarding experiments, to validate the upper bound, one could use a simple base learner such as decision stumps, implement the boosting algorithm given in Algorithm 1, and repeat this experiment for larger values of k. Regarding the lower bound, since the proof uses a probabilistic argument and is thus not constructive, it cannot be easily verified empirically. 


The paper presents an analysis of multiclass boosting, showing that the boosting problem itself is simpler to learn (in terms of the sample complexity wrt the number of classes), than the problem of producing acceptable weak learners which can be boosted.  It would be helpful to contrast the boosting algorithm in algorithm 1 against the other algorithms described in reference 13, especially Adaboost.MM, and to unify the notation a little (as this paper refers to sigma-gains, which are the cost matrices in ref 13).

The text after Theorem 6 is confusing. If any ERM learner can learn a gamma-realizable hypothesis with O(log(k)) sample complexity, then it would be helpful to clearly explain what restrictions force the weak learner to have a sample complexity of O(k^2).

Update: thank you for the responses, I think some of the clarifications for the other reviewers and I should be included in the final text. The theoretical work is hard to place in the literature of boosting as there is little analysis of what the results imply for extant multiclass boosting algorithms, or the relative complexity of large multiclass problems. It's also unclear at what level the number of classes starts to be the dominant complexity issue for boosting weak learners. Much of the text says k > 2, but some sections require k >> 2, and I'm unclear as to the boundary between the two, or appropriate inferences to draw from the theory when applied to smaller problems.

 I thank the authors for taking the time to answer my questions/feedback. I would appreciate it if the authors could:

1. expand on the applicability of their results to gradient boosting (any boosting scheme). Why do they apply to any boosting scheme?

2. describe how they would design a set of experiments to validate their theory (I understand that the paper is focused on theoretical contributions which I appreciate and I am not asking the authors to add experimental results)

 We thank the reviewer for reading the manuscript and for their feedback and suggestions. We will clarify the comments pointed out by the reviewer in the final version.

- Informally, “easy-to-learn” means learnable by a fast algorithm, e.g., decision stumps. We formally express it via the Natarajan dimension, an extension of the VC-dimension to the non-binary setting. See Section 4 for the formal definition and discussion.

- The resources under study are in fact mentioned in the abstract: we mention the sample complexity and the accuracy parameter (epsilon) of the weak learner, as well as the trade-off of those with number of oracle calls. (2nd paragraph of the abstract).

- The lower bound, the WL condition and overall setup would apply to any boosting algorithm -- gradient boosting or otherwise. We will further clarify it in the paper.

- The challenges with using tailored loss functions require the non-trivial assumption that a tailored weak learner with guarantees w.r.t this loss also exists, which is less natural.  Note that our assumption is simple and handy: it allows to take any *PAC* learnable class and use it to *PAC* learn a bigger class.

- The log dependence of the algorithm on k is shown in Theorem 3, with T ~= log(k). 

- Regarding experiments, it is indeed possible to design with decision trees, however, the focus of this work is theoretical investigation of multi-class boosting and so experiments were beyond the scope of this work.

- Regarding noisy datasets, in this work we assume realizability, and a noisy/agnostic setting is an interesting case to consider in future work. 


 We thank the reviewer for reading the manuscript and for their feedback and suggestions. We will clarify the comments pointed out by the reviewer in the final version.

The reviewer correctly notes that the large gap between the sample complexity of any ERM learner and weak learner used via boosting, for learning *the same* distribution - is indeed very surprising. As for the intuitive explanation for why that is the case, we tried to address that in Section 5, and in particular the proof overview of our lower bound. 

To answer that perhaps more concisely - the idea is that the guarantees of the weak learner are such that, unless directed to the correct label, it has no way of finding it, given no other information. We formalize this concept in the “Finder-Chooser” framework described in lines 363. 

The analogy is that a “chooser” is the random choice of some correct label (i.e, there exists some H that satisfies this), and the “finder” is the booster which is only allowed to submit simple queries to the weak learner. When no other knowledge of the class is available to the booster,
except the responses of the weak learner, it is forced to exahustively search for the right label. That is, submit many such queries to the weak learner, proportional to k the number of labels. 

Regarding the reviewer’s concern for the relevance of these results, we note that there is a lot of literature on similar cases that address the issue of large dependency on k. We briefly addressed ​​the practical interest in the introduction, but can cite several more works on similar applications.

“​​Much of the text says k > 2, but some sections require k >> 2” - we note that it is not required, and all our results apply for all k > 2, or for k > 2/gamma (Thm.7). 


 We thank the reviewer for reading the manuscript and for their feedback and suggestions. We will correct typos and clarify the comments pointed out by the reviewer in the final version. 

1. Algorithm 1 builds on classical multi-class boosting algorithms, e.g., Adaboost.MR [17] and of Mukherjee and Schapire (2011) [13] , but is an improvement in the sense that Alg. 1 only assumes the standard classification loss learner, and not other intricate losses. Alg. 1 makes use of the reduction we give between the loss given by Mukherjee and Schapire (2011) [13] and the standard 0-1 classification loss, as described in Lemma 2, and is used in the proof of Theorem 3. Moreover, albeit being simpler, Alg.1 still achieves optimal results from time and booster sample complexity perspectives. 

2. Indeed, the boosting algorithm has a black-box access to an algorithm which satisfies the WL assumptions, but otherwise can be arbitrary (as is standard in literature). Thanks, we will emphasize this in the final version. 

3. Boosting is not considered by Daniely et al. (2015) [9] and their lower bounds are pertinent to this setting as they are to any multiclass learning task, with no dependency on k. In contrast, our lower bound is *polynomial* in k. Of course, to get this stronger lower bound one has to exploit (as we do) the fact that the boosting algorithm only has a black-box access to the weak learner.

Regarding general principles for designing multiclass weak learners. If one has access to a learner which can learn the base class 
 w.r.t the $\sigma$-gain function, or in other words - it satisfies the WL condition given by Mukherjee and Schapire (2011) [13], then it will not incur the cost of a polynomial dependency on k. 

The non-trivial assumption that such a tailored weak learner, with guarantees with respect to this tailored loss, also exists - is less natural. In this work we considered the simpler setting which assumes a *PAC* learnable class and use it to *PAC* learn a bigger class. 






The paper “Multiclass Boosting and the Cost of Weak Learning”  studies boosting in the multiclass regime, especially by focusing on the how number of classes k affects the resources required for boosting.

The contribution that I found most interesting is the demonstrated gap between the sample complexity dependence on the number of classes needed for boosting versus weak learning. (log(k) versus k^2). 

The other contribution of this paper is a modified boosting algorithm for the framework above.  I think the paper's contributions are interesting and worthwhile.  They are not groundbreaking contributions to the theory of multiclass boosting, but they elucidate some important aspects.  I would recommend acceptance subject to the following comments being addressed.

main comments

1. How is your Algorithm 1 distinguished from the multiclass boosting algorithm of Mukherjee and Schapire (2011)?  (As in, what does section 3 add on its own?)  Or is section 3 meant as a tool and inspiration to the tradeoff later shown?
2. It seems to me that Theorem 7 / Corollary 8 depend on a (justified) assumption where the weak learning oracle is used as a black box.  Otherwise, one could presumably use the upper bounds of [9] to find a linear combination of weak learners without the k^2 dependence.  I think this should be pointed out to avoid confusion. (Or please correct me if I am mistaken.)
3. Are lower bounds from Daniely et al. (2015) on multiclass learnability pertinent to your results? (You cite only their upper bounds.)

minor

l. 74 “which they proved to be necessary and sufficient” (who is “they”? You should state the authors’ names.)

Do these results point to any general principles for designing multiclass weak learners? (c.f. Reyzin (AAAI 2014) for such a discussion for the binary case.)
 no concerns

This paper analyzes theoretically the resources required by multiclass boosting to guarantee convergence. Namely, the authors propose an AdaBoost algorithm variant tailored to solve multiclass tasks. The algorithm uses the standard classification loss function, as opposed to state-of-the-art multiclass AdaBoost algorithms that use tailored loss functions. The convergence of the proposed algorithm is guaranteed if the boosting sample complexity has a logarithmic dependency on the number of classes k in the training dataset and if the number of samples used by the weak learner at each boosting iteration has a k^2 complexity. The paper also shows that the weak learner must be best in class when the number of classes is very large, making each call to the weak learner oracle very costly / challenging.  In terms of originality, the paper proposes a novel theoretical analysis of multiclass boosting by using a multiclass variant of the AdaBoost algorithm. In contrast to state-of-the-art algorithms, the proposed method uses the standard classification loss function and does not convert the learning of the multiclass problem into multiple binary problems. 

In terms of significance, the paper provides a set of interesting theoretical findings regarding the resources required by multiclass boosting to guarantee convergence especially in the case of a large number of classes. It would be good to understand how these findings could be applied (if possible) to other boosting schemes, such as gradient boosting. In other words, how general are the theoretical findings for all the multiclass boosting schemes? Also, how can these findings be used in practice. 

In terms of clarity and quality, the paper could be improved as suggested in the next paragraphs:

- What is the definition of an "easy-to-learn" base class, and what would be an example of such a class, a tree stump? Are decision trees of max_depth N > 1 an easy-to-learn base class?
- It would be good to mention the boosting resources under study also in the abstract.
- A very popular boosting framework nowadays is gradient boosting. A discussion would be appreciated if the authors could explain how the findings could be applied to gradient boosting. If they do not apply, would the challenges of multiclass boosting described in the paper apply to gradient boosting as well? 
- One of the main contribution of the paper is that the proposed algorithm and theoretical analysis use the standard classification loss as opposed to tailored loss functions. Could the authors explain why using a tailored loss function as proposed in [13] is a disadvantage compared to their proposed algorithm? What are the challenges with using tailored loss functions?
- It would be good if the authors would explain the statement in lines 197-198 about the logarithmic dependence of the algorithm on k.
- Would it be possible for the authors to design a set of experiments in which their proposed method is applied to a real dataset, where the base class is the decision tree? As base learners need to be "easy-to-learn", probably tree stumps could be used as a base class. Could such experiments validate the theoretical findings and show that if k is large and the sample complexity / number of calls to the weak learner oracle is not of the complexity suggested by the theoretical analysis, then the model does not converge even for a large number of iterations?
- AdaBoost is known to be sensitive to noisy datasets. How is this taken into account by the theoretical analysis proposed in this paper? The authors do not discuss the potential negative societal impact of their work, but that looks reasonable to me. I do not see any negative impact of their proposed work.

In terms of limitations, the authors describe the cost using the standard classification loss on the lower bound of the excess error.


Meta-review: This paper studies the problem of multiclass boosting and proves a few different results. 1) proving that the number of samples required by a 
weak learner is exponentially more than the number of samples needed by the booster; 2) proving that the weak learner's accuracy parameter must be smaller than an inverse polynomial; and 3) giving a trade-off between number of oracle calls and the
resources required of the weak learner.  This is a nice set of interesting and even surprising set of results about multiclass boosting and would add to the conference.

Decision: accept