Title: Boosting with Tempered Exponential Measures
Venue: NeurIPS 2023
Accepted: yes
Keywords: Boosting

optimization

exponential families
Abstract: One of the most popular ML algorithms, AdaBoost, can be
derived from the dual of a relative entropy
minimization problem subject to the fact that the positive weights
on the examples sum to one. Essentially, harder examples receive higher probabilities. We generalize this setup to the recently introduced *tempered
exponential measure*s (TEMs) where normalization is enforced on a specific power of the measure and not the measure itself.
TEMs are indexed by a parameter $t$ and generalize exponential families ($t=1$). Our algorithm, $t$-AdaBoost, recovers AdaBoost as a special case ($t=1$). We show that $t$-AdaBoost retains AdaBoost's celebrated exponential convergence rate when $t\in [0,1)$ while allowing a slight improvement of the rate's hidden constant compared to $t=1$. $t$-AdaBoost partially computes on a generalization of classical arithmetic over the reals and brings notable properties like guaranteed bounded leveraging coefficients for $t\in [0,1)$. From the loss that $t$-AdaBoost minimizes (a generalization of the exponential loss), we show how to derive a new family of *tempered* losses for the induction of domain-partitioning classifiers like decision trees. Crucially, strict properness is ensured for all while their boosting rates span the full known spectrum. Experiments using $t$-AdaBoost+trees display that significant leverage can be achieved by tuning $t$.

Boosting with Tempered Exponential Measures

Abstract

One of the most popular ML algorithms, ADABOOST, can be derived from the dual of a relative entropy minimization problem subject to the fact that the positive weights on the examples sum to one. Essentially, harder examples receive higher probabilities. We generalize this setup to the recently introduced tempered exponential measure s (TEMs) where normalization is enforced on a specific power of the measure and not the measure itself. TEMs are indexed by a parameter t and generalize exponential families ( t GLYPH<16> 1 ). Our algorithm, t -ADABOOST, recovers ADABOOST as a special case ( t GLYPH<16> 1 ). We show that t -ADABOOST retains ADABOOST's celebrated exponential convergence rate on margins when t P r 0 , 1 q while allowing a slight improvement of the rate's hidden constant compared to t GLYPH<16> 1 . t -ADABOOST partially computes on a generalization of classical arithmetic over the reals and brings notable properties like guaranteed bounded leveraging coefficients for t P r 0 , 1 q . From the loss that t -ADABOOST minimizes (a generalization of the exponential loss), we show how to derive a new family of tempered losses for the induction of domain-partitioning classifiers like decision trees. Crucially, strict properness is ensured for all while their boosting rates span the full known spectrum of boosting rates. Experiments using t -ADABOOST+trees display that significant leverage can be achieved by tuning t .

1 Introduction

ADABOOST is one of the most popular ML algorithms [8, 30]. It efficiently aggregates weak hypotheses into a highly accurate linear combination [10]. The common motivations of boosting algorithms focus on choosing good linear weights (the leveraging coefficients) for combining the weak hypotheses. A dual view of boosting highlights the dual parameters, which are the weights on the examples. These weights define a distribution, and ADABOOST can be viewed as minimizing a relative entropy to the last distribution subject to a linear constraint introduced by the current hypothesis [12]. For this reason (more in § 2), ADABOOST's weights define an exponential family.

In this paper , we go beyond weighing the examples with a discrete exponential family distribution, relaxing the constraint that the total mass be unit but instead requiring it for the measure's 1 {p 2 GLYPH<1> t q 'th power, where t is a temperature parameter. Such measures, called tempered exponential measures (TEMs), have been recently introduced [4]. Here we apply the discrete version of these TEMs for deriving a novel boosting algorithm called t -ADABOOST. Again the measures are solutions to a relative entropy minimization problem, but the relative entropy is built from Tsallis entropy and 'tempered' by a parameter t . As t GLYPH<209> 1 TEMs become standard exponential family distributions and our new algorithm merges into ADABOOST. As much as ADABOOST minimizes the exponential loss, t -ADABOOST minimizes a generalization of this loss we denote as the tempered exponential loss .

TEMs were introduced in the context of clustering, where they were shown to improve the robustness to outliers of clustering's population minimizers [4]. They have also been shown to bring low-level sparsity features to optimal transport [3]. Boosting is a high-precision machinery: ADABOOST is known to achieve near-optimal boosting rates under the weak learning assumption [1], but it has

long been known that numerical issues can derail it, in particular, because of the unbounded weight update rule [14]. So the question of what the TEM setting can bring for boosting is of primordial importance. As we show, t -ADABOOST can suffer no rate setback as boosting's exponential rate of convergence on margins can be preserved for all t P r 0 , 1 q . Several interesting features emerge: the weight update becomes bounded, margin optimization can be tuned with t to focus on examples with very low margin and besides linear separators, the algorithm can also learn progressively clipped models 1 . Finally, the weight update makes appear a new regime whereby weights can "switch off and on": an example's weight can become zero if too well classified by the current linear separator, and later on revert to non-zero if badly classified by a next iterate. t -ADABOOST makes use of a generalization of classical arithmetic over the reals introduced decades ago [18].

Boosting algorithms for linear models like ADABOOST bring more than just learning good linear separators: it is known that (ada)boosting linear models can be used to emulate the training of decision trees (DT) [16], which are models known to lead to some of the best of-the-shelf classifiers when linearly combined [9]. Unsurprisingly, the algorithm obtained emulates the classical top-down induction of a tree found in major packages like CART [6] and C4.5 [23]. The loss equivalently minimized, which is, e.g. , Matusita's loss for ADABOOST [30, Section 4.1], is a lot more consequential. Contrary to losses for real-valued classification, losses to train DTs rely on the estimates of the posterior learned by the model; they are usually called losses for Class Probability Estimation (CPE [25]). The CPE loss is crucial to elicit because (i) it is possible to check whether it is "good" from the standpoint of properness (Bayes rule is optimal for the loss [28]), and (ii) it conditions boosting rates, only a handful of them being known, for the most popular CPE losses [11, 22, 31].

In this paper , we show that this emulation scheme on t -ADABOOST provides a new family of CPE losses with remarkable constancy with respect to properness: losses are strictly proper (Bayes rule is the sole optimum) for any t P pGLYPH<1>8 , 2 q and proper for t GLYPH<16> GLYPH<1>8 . Furthermore, over the range t P rGLYPH<1>8 , 1 s , the range of boosting rates spans the full spectrum of known boosting rates [11].

We provide experiments displaying the boosting ability of t -ADABOOST over a range of t encompassing potentially more than the set of values covered by our theory, and highlight the potential of using t as a parameter for efficient tuning the loss [25, Section 8]. Due to a lack of space, proofs are relegated to the appendix (APP.). A primer on TEMs is also given in APP., Section I.

2 Related work

Boosting refers to the ability of an algorithm to combine the outputs of moderately accurate, "weak" hypotheses into a highly accurate, "strong" ensemble. Originally, boosting was introduced in the context of Valiant's PAC learning model as a way to circumvent the then-existing amount of related negative results [10, 34]. After the first formal proof that boosting is indeed achievable [29], ADABOOST became the first practical and proof-checked boosting algorithm [8, 30]. Boosting was thus born in a machine learning context, but later on, it also emerged in statistics as a way to learn from class residuals computed using the gradient of the loss [9, 21], resulting this time in a flurry of computationally efficient algorithms, still called boosting algorithms, but for which the connection with the original weak/strong learning framework is in general not known.

Our paper draws its boosting connections with ADABOOST's formal lineage. ADABOOST has spurred a long line of work alongside different directions, including statistical consistency [5], noise handling [15, 16], low-resource optimization [22], etc . The starting point of our work is a fascinating result in convex optimization establishing a duality between the algorithm and its memory of past iteration's performances, a probability distribution of so-called weights over examples [12]. From this standpoint, ADABOOST solves the dual of the optimization of a Bregman divergence (constructed from the negative Shannon entropy as the generator) between weights subject to zero correlation with the last weak classifier's performance. As a consequence, weights define an exponential family. Indeed, whenever a relative entropy is minimized subject to linear constraints, then the solution is a member of an exponential family of distributions (see e.g. [2, Section 2.8.1] for an axiomatization of exponential families). ADABOOST's distribution on the examples is a member of a discrete exponential family where the training examples are the finite support of the distribution, sufficient statistics are defined from the weak learners, and the leveraging coefficients are the natural parameters. In summary, there

is an intimate relationship between boosting à-la-ADABOOST, exponential families, and Bregman divergences [7, 12, 20] and our work "elevates" these methods above exponential families.

3 Definitions

We define the t -logarithm and t -exponential [17, Chapter 7],

where the case t GLYPH<16> 1 is supposed to be the extension by continuity to the log and exp functions, respectively. To preserve the concavity of log t and the convexity of exp t , we need t ¥ 0 . In the general case, we also note the asymmetry of the composition: while exp t log t p z q GLYPH<16> z, @ t P R , we have log t exp t p z q GLYPH<16> z for t GLYPH<16> 1 ( @ z P R ), but

Comparisons between vectors and real-valued functions written on vectors are assumed componentwise. We assume t GLYPH<24> 2 and define notation t GLYPH<6> . GLYPH<16> 1 {p 2 GLYPH<1> t q . We now define the key set in which we model our weights (boldfaces denote vector notation).

The letters q will be used to denote TEMs in ˜ ∆ m while p denote the co-density q 1 t GLYPH<6> or any element of the probability simplex. We define the general tempered relative entropy as

where r m s . GLYPH<16> t 1 , ..., m u . The tempered relative entropy is a Bregman divergence with convex generator ϕ t p z q . GLYPH<16> z log t z GLYPH<1> log t GLYPH<1> 1 p z q (for t P R ) and ϕ t p z q 1 GLYPH<16> log t p x q . As t GLYPH<209> 1 , D t p q , q 1 q becomes the relative entropy with generator ϕ 1 p x q GLYPH<16> x log p x q GLYPH<1> x .

4 Tempered boosting as tempered entropy projection

We start with a fixed sample S GLYPH<16> tp x i , y i q : i P r m su where observations x i lie in some domain X and labels y i are GLYPH<8> 1 . ADABOOST maintains a distribution p over the sample. At the current iteration, this distribution is updated based on a current weak hypothesis h P R X using an exponential update:

In [12] this update is motivated as minimizing a relative entropy subject to the constraint that p 1 is a distribution summing to 1 and p 1J u GLYPH<16> 0 . Following this blueprint, we create a boosting algorithm maintaining a discrete TEM over the sample which is motivated as a constrained minimization of the tempered relative entropy, with a normalization constraint on the co-simplex of R m :

We now show that the solution q 1 is a tempered generalization of ADABOOST's exponential update. Theorem 1. For all t P R zt 2 u , all solutions to (3) have the form

where Z t ensures co-simplex normalization of the co-density. Furthermore, the unknown µ satisfies

Algorithm 1 t -ADABOOST p t, S , J q

Input: t P r 0 , 1 s , training sample S , # iterations J ;

Output: classifiers H J , H p 1 { 1 GLYPH<1> t q J

(see (9));

Step 1 : initialize tempered weights: q 1 GLYPH<16> p 1 { m t GLYPH<6> q GLYPH<4> 1 pP ˜ ∆ m q ;

Step 2 : for j GLYPH<16> 1 , 2 , ..., J

Step 2.1 : get weak classifier

Step 2.2 : choose weight update coefficient µ

Step 2.3 :

h j -weak_learner p q j , S q ;

j P R ;

@ i P r m s , for u ji . GLYPH<16> y i h j p x i q , update the tempered weights as

or equivalently is a solution to the nonlinear equation

Finally, if either (i) t P R ¡ 0 zt 2 u or (ii) t GLYPH<16> 0 and q is not collinear to u , then Z t p µ q is strictly convex: the solution to (3) is thus unique, and can be found from expression (4) by finding the unique minimizer of (5) or (equivalently) the unique solution to (6) .

(Proof in APP., Section II.1) The t -product b t , which satisfies exp t p a GLYPH<0> b q GLYPH<16> exp t p a q b t exp t p b q , was introduced in [18]. Collinearity never happens in our ML setting because u contains the edges of a weak classifier: q ¡ 0 and collinearity would imply that GLYPH<8> the weak classifier performs perfect classification, and thus defeats the purpose of training an ensemble. @ t P R zt 2 u , we have the simplified expression for the normalization coefficient of the TEM and the co-density p 1 of q 1 :

5 Tempered boosting for linear classifiers and clipped linear classifiers

Models A model (or classifier) is an element of R X . For any model H , its empirical risk over S is F 0 { 1 p H, S q . GLYPH<16> p 1 { m q GLYPH<4> GLYPH<176> i J y i GLYPH<24> sign p H p x i qq K where J . K , Iverson's bracket [13], is the Boolean value of the inner predicate. We learn linear separators and clipped linear separators. Let p v j q j ¥ 1 be the terms of a series and δ ¥ 0 . The clipped sum of the series is:

and we define the base case J GLYPH<16> 1 by replacing the inner clipped sum with 0. Note that clipped summation is non-commutative, and so is different from clipping in rGLYPH<1> δ, δ s the whole sum itself 2 . Given a set of so-called weak hypotheses h j P R X and leveraging coefficients α j P R (for j P r J s ), the corresponding linear separators and clipped linear separators are

Tempered boosting and its general convergence Our algorithm, t -ADABOOST, is presented in Algorithm 1, using presentation conventions from [30]. Before analyzing its convergence, several properties are to be noted for t -ADABOOST: first, it keeps the appealing property, introduced by ADABOOST, that examples receiving the wrong class by the current weak classifier are reweighted

higher (if µ j ¡ 0 ). Second, the leveraging coefficients for weak classifiers in the final classifier ( α j s) are not the same as the ones used to update the weights ( µ j s), unless t GLYPH<16> 1 . Third and last, because of the definition of exp t (1), if t GLYPH<160> 1 , tempered weights can switch off and on, i.e. , become 0 if an example is "too well classified" and then revert back to being ¡ 0 if the example becomes wrongly classified by the current weak classifier (if µ j ¡ 0 ). To take into account those zeroing weights, we denote r m s : j . GLYPH<16> t i : q ji GLYPH<16> 0 u and m : j . GLYPH<16> Card pr m s : j q ( @ j P r J s and Card denotes the cardinal). Let R j . GLYPH<16> max i Rr m s : j | y i h j p x i q|{ q 1 GLYPH<1> t ji and q : j . GLYPH<16> max i Pr m s : j | y i h j p x i q| 1 {p 1 GLYPH<1> t q { R 1 {p 1 GLYPH<1> t q j . It is worth noting that q : j is homogeneous to a tempered weight.

Theorem 2. At iteration j , define the weight function q 1 ji . GLYPH<16> q ji if i R r m s : j and q : j otherwise; set

In algorithm t -ADABOOST , consider the choices (with the convention -0 k GLYPH<16> 1 v k . GLYPH<16> 1 )

where M q p a, b q . GLYPH<16> pp a q GLYPH<0> b q q{ 2 q 1 { q is the q -power mean. Then for any H P t H J , H p 1 { 1 GLYPH<1> t q J u , its empirical risk is upperbounded as:

(Proof in APP., Section II.2) We jointly comment t -ADABOOST and Theorem 2 in two parts.

Case t GLYPH<209> 1 GLYPH<1> : t -ADABOOST converges to ADABOOST and Theorem 2 to its convergence analysis: t -ADABOOST converges to ADABOOST as presented in [30, Figure 1]: the tempered simplex becomes the probability simplex, b t converges to regular multiplication, weight update (8) becomes ADABOOST's, α j GLYPH<209> µ j in (11) and finally the expression of µ j converges to ADABOOST's leveraging coefficient in [30] ( lim t GLYPH<209> 1 M 1 GLYPH<1> t p a, b q GLYPH<16> ? ab ). Even guarantee (12) converges to ADABOOST's popular guarantee of [30, Corollary 1] ( lim t GLYPH<209> 1 K t p z q GLYPH<16> ? 1 GLYPH<1> z 2 , m : j GLYPH<16> 0 ). Also, in this case, we learn only the unclipped classifier since lim t GLYPH<209> 1 GLYPH<1> H p 1 { 1 GLYPH<1> t q J GLYPH<16> H J .

Case t GLYPH<160> 1 : Let us first comment on the convergence rate. The proof of Theorem 2 shows that K t p z q / exp pGLYPH<1> z 2 {p 2 t GLYPH<6> qq . Suppose there is no weight switching, so m : j GLYPH<16> 0 , @ j (see Section 7) and, as in the boosting model, suppose there exists γ ¡ 0 such that | ρ j | ¥ γ, @ j . Then t -ADABOOST is guaranteed to attain empirical risk below some ε ¡ 0 after a number of iterations equal to J GLYPH<16> p 2 t GLYPH<6> { γ 2 q GLYPH<4> log p 1 { ε q . t GLYPH<6> being an increasing function of t P r 0 , 1 s , we see that t -ADABOOST is able to slightly improve upon ADABOOST's celebrated rate [32]. However, t GLYPH<6> GLYPH<16> 1 { 2 for t GLYPH<16> 0 so the improvement is just on the hidden constant. This analysis is suited for small values of | ρ j | and does not reveal an interesting phenomenon for better weak hypotheses. Figure 1 compares K t p z q curves ( K 1 p z q . GLYPH<16> lim t GLYPH<209> 1 K t p z q GLYPH<16> ? 1 GLYPH<1> z 2 for ADABOOST, see [30, Corollary 1]), showing the case t GLYPH<160> 1 can be substantially better, especially when weak hypotheses are not "too weak". If m : ¡ 0 , switching weights can impede our convergence analysis , though exponential convergence is j always possible if m : j q : j 2 GLYPH<1> t is small enough; also, when it is not, we may in fact have converged to a good model (see APP., Remark 1). A good criterion to train weak hypotheses is then the optimization of the edge ρ j , thus using q 1 j normalized in the simplex. Other key features of t -ADABOOST are as follows. First, the weight update and leveraging coefficients of weak classifiers are bounded because | µ j | GLYPH<160> 1 {p R j p 1 GLYPH<1> t qq (APP., Lemma H) (this is not the case for t GLYPH<209> 1 GLYPH<1> ). This guarantees that new weights are bounded before normalization (unlike for t GLYPH<209> 1 GLYPH<1> ). Second, we remark that µ j GLYPH<24> α j if t GLYPH<24> 1 . Factor m 1 GLYPH<1> t GLYPH<6> is added for convergence analysis purposes; we can discard it to train the unclipped classifier: it does not change its empirical risk. This is, however, different for factor -j GLYPH<1> 1 k GLYPH<16> 1 Z k : from (12), we conclude that this is an indication of how well the past ensemble performs.

❑

0

0.2

0.4

0.6

0.8

1

-1

-0.75

-0.5

-0.25

0

0.25

0.5

0.75

1

t=1

(AdaBoost)

t=0

t

z

0

0.05

0.1

0.96

1

✭

/a0

✁

As it gets better and better, it progressively dampens the leverage of the next weak classifiers, a phenomenon that does not occur in boosting, where an excellent weak hypothesis on the current weights can have a leveraging coefficient so large that it wipes out the classification of the past ones. This can be useful to control numerical instabilities.

Extension to margins A key property of boosting algorithms like ADABOOST is to be able to boost not just the empirical risk but more generally margins [19, 30], where a margin integrates both the accuracy of label prediction but also the confidence in prediction (say | H | ). We generalize the margin notion of [19] to the tempered arithmetic and let ν t pp x , y q , H q . GLYPH<16> tanh t p yH p x q{ 2 q denote the margin of H on example p x , y q , where tanh t p z q . GLYPH<16> p 1 GLYPH<1> exp t pGLYPH<1> 2 z qq{p 1 GLYPH<0> exp t pGLYPH<1> 2 z qqpP rGLYPH<1> 1 , 1 sq is the tempered hyperbolic tangent. The objective of minimizing the empirical risk is generalized to minimizing the margin risk, F t,θ p H, S q . GLYPH<16> p 1 { m q GLYPH<4> GLYPH<176> i J ν t pp x i , y i q , H q / θ K , where θ P pGLYPH<1> 1 , 1 q . Guarantees on the empirical risk are guarantees on the margin risk for θ GLYPH<16> 0 only. In just a few steps, we can generalize Theorem 2 to all θ P pGLYPH<1> 1 , 1 q . For space reason, we state the core part of the generalization, from which extending it to a generalization of Theorem 2 is simple.

Theorem 3. For any θ P pGLYPH<1> 1 , 1 q and t P r 0 , 1 s , the guarantee of algorithm t -ADABOOST in Theorem 2 extends to the margin risk, with notations from Theorem 2, via:

(Proof in APP., Section II.3) At a high level, t -ADABOOST brings similar margin maximization properties as ADABOOST. Digging a bit in (13) reveals an interesting phenomenon for t GLYPH<24> 1 on how margins are optimized compared to t GLYPH<16> 1 . Pick θ GLYPH<160> 0 , so we focus on those examples for which the classifier H has high confidence in its wrong classification. In this case, factor pp 1 GLYPH<0> θ q{p 1 GLYPH<1> θ qq 2 GLYPH<1> t is increasing as a function of t P r 0 , 1 s (and this pattern is reversed for θ ¡ 0 ). In words, the smaller we pick t P r 0 , 1 s and the better is the bound in (13), suggesting increased "focus" of t -ADABOOST on increasing the margins of examples with low negative margin ( e.g. the most difficult ones) compared to the case t GLYPH<16> 1 .

The tempered exponential loss In the same way as ADABOOST introduced the now famous exponential loss, (12) recommends to minimize the normalization coefficient, following (7),

We cannot easily unravel the normalization coefficient to make appear an equivalent generalization of the exponential loss, unless we make several assumptions, one being max i | h j p x i q| is small enough for any j P r J s . In this case, we end up with an equivalent criterion to minimize which looks like

where we have absorbed in H the factor m 1 GLYPH<1> t GLYPH<6> appearing in the exp t (scaling H by a positive value does not change its empirical risk). This defines a generalization of the exponential loss which we call the tempered exponential loss . Notice that one can choose to minimize F t p H, S q disregarding any constraint on | H | .

6 A broad family of boosting-compliant proper losses for decision trees

Losses for class probability estimation When it comes to tabular data, it has long been known that some of the best models to linearly combine with boosting are decision trees (DT, [9]). Decision trees, like other domain-partitioning classifiers, are not trained by minimizing a surrogate loss defined over real-valued predictions, but defined over class probability estimation (CPE, [26]), those estimators being posterior estimation computed at the leaves. Let us introduce a few definitions for those. A

J K J K Functions glyph[lscript] 1 , glyph[lscript] GLYPH<1> 1 are called partial losses. The pointwise conditional risk of local guess u P r 0 , 1 s with respect to a ground truth v P r 0 , 1 s is:

A loss is proper iff for any ground truth v P r 0 , 1 s , L p v, v q GLYPH<16> inf u L p u, v q , and strictly proper iff u GLYPH<16> v is the sole minimizer [26]. The (pointwise) Bayes risk is L p v q . GLYPH<16> inf u L p u, v q . The log/cross-entropy-loss, square-loss, Matusita loss are examples of CPE losses. One then trains a DT minimizing the expectation of this loss over leaves' posteriors, E λ r L p p λ qs , p λ being the local proportion of positive examples at leaf λ - or equivalently, the local posterior.

Deriving CPE losses from (ada)boosting Recently, it was shown how to derive in a general way a CPE loss to train a DT from the minimization of a surrogate loss with a boosting algorithm [16]. In our case, the surrogate would be ˜ Z tj (14) and the boosting algorithm, t -ADABOOST. The principle is simple and fits in four steps: (i) show that a DT can equivalently perform simple linear classifications, (ii) use a weak learner that designs splits and the boosting algorithm to fit the leveraging coefficient and compute those in closed form, (iii) simplify the expression of the loss using those, (iv) show that the expression simplified is, in fact, a CPE loss. To get (i), we remark that a DT contains a tree (graph). One can associate to each node a real value. To classify an observation, we sum all reals from the root to a leaf and decide on the class based on the sign of the prediction, just like for any real-valued predictor. Suppose we are at a leaf. What kind of weak hypotheses can create splits "in disguise"? Those can be of the form

J K where the observation variable x k is assumed real valued for simplicity and the test J x k ¥ a j K splits the leaf's domain in two non-empty subsets. This creates half of the split. h j p x q . GLYPH<16> J x k GLYPH<160> a j K GLYPH<4> GLYPH<1> b j creates the other half of the split. Remarkably h j satisfies the weak learning assumption iff h j does [16]. So we get the split design part of (ii). We compute the leveraging coefficients at the new leaves from the surrogate's minimization / boosting algorithm, end up with new real predictions at the new leaves (instead of the original b j , GLYPH<1> b j ), push those predictions in the surrogate loss for (iii), simplify it and, quite remarkably end up with a loss of the form E λ r L p p λ qs , where L turns out to be the pointwise Bayes risk L of a proper loss [16] (notation from [26]).

In the case of [16], it is, in fact, granted that we end up with such a "nice" CPE loss because of the choice of the surrogates at the start. In our case, however, nothing grants this a priori if we start from the tempered exponential loss ˜ Z tj (14) so it is legitimate to wonder whether such a chain of derivations (summarized) can happen to reverse engineer an interesting CPE loss:

When such a complete derivation happens until the partial losses glyph[lscript] 1 ; glyph[lscript] GLYPH<1> 1 and their properties, we shall write that minimizing ˜ Z tj elicits the corresponding loss and partial losses.

Theorem 4. Minimizing ˜ Z tj elicits the CPE loss we define as the tempered loss , with partial losses:

The tempered loss is symmetric, differentiable, strictly proper for t P pGLYPH<1>8 , 2 q and proper for t GLYPH<16> GLYPH<1>8 .

perr (not clipped)

perr (clipped)

min weight

max weight

abalone

,

t

¥

0

.

0

,

η

GLYPH<16>

0

.

0

qsar

,

t

¥

0

.

6

,

η

GLYPH<16>

0

.

1

adult

,

t

¥

0

.

6

,

η

GLYPH<16>

0

.

1

0.212

0.211

0.21

Adaboost(t=1.0)

0.8_0.212

0.211

0.214

0.213

0.215

Adaboost (t=1.0)

0.8

(6Ae)

0.00025

0.0002

Adaboost (t=1.0)

0.0012

0.0016

0.0014

Adaboost (t=1.0)

0.8

0.209

0.209

0.6

0.00015

0.001

0.6

0.208

0.4

0.0001

0.0008

0.4

0.207

0.207

0.0006

0.206

0.2

0.206

0.2

5x10-5

0.2

0.0004

0.2

0.205

0.205

2468101214161820

0

0.204

2468101214161820

0

0

2468101214161820

0

0.0002

24 6 8 10 1214 16 18 20

0

#trees

#trees

#trees

#trees

0.185

0.18

0.19

Adaboost (t=1.0)

1.1

0.185

0.18

Adaboost (t=1.0)

0.0007

0.0008

Adaboost (t=1.0)

1

1.1

0.0045

0.004

Adaboost (t=1.0)

.1

0.175

0.17

0.175

0.17

60

1co-density

0.0005

0.0035

0.9

0.165

0.003

0.165

0.8

0.0004

0.8

0.16

0.0025

0.155

0.15

0.7

0.155

0.16

0.7

0.0003

0.0002

0.002

0.7

0.145

0.6

0.15

0.6

0.0001

0.6

0.0015

0.6

2

468101214161820

2

2468101214161820

2468101214161820

2468 101214 16 18 20

#trees

#trees

#trees

#trees

0.178

Adaboost (t=1.0)

1.1

0.18

Adaboost (t=1.0)

1.1

0.000028

Adaboost (t=1.0)

1.10.00018

Adaboost (t=1.0)

1.1

0.174

0.176

0.178

0.176

1

0.000024

0.000026

0.00016

0.168

0.17

0.174

0.172

0.9

0.000020

0.000018

O.00016

0.9

0.166

0.168

0.8

80.000014

O.000012

O.00008

0.8

0.164

0.162

0.7

0.164

0.162

0.7

0.000010

0.000008

0.00006

0.7

0.16

0.6

0.16

0.6

0.000006

0.6 0.00004

0.6

2468101214 16 18 20

2468101214161820

246 8 10121416 18 20

2468101214161820

#trees

#trees

#trees

#trees

Differentiability means the partial losses are differentiable, and symmetry follows from the relationship between partial losses [20] (the proof, in APP., Section II.4, derives the infinite case, glyph[lscript] pGLYPH<1>8q 1 p u q GLYPH<16> 2 GLYPH<4> u / 1 { 2 ) Let us explicit the Bayes risk of the tempered loss and a key property.

J K Lemma 1. The Bayes risk of the tempered loss is ( M q defined in Theorem 2):

and it satisfies @ u P r 0 , 1 s , @ z P r 2 GLYPH<4> min t u, 1 GLYPH<1> u u , 1 s , D t P rGLYPH<1>8 , 2 s such that L p t q p u q GLYPH<16> z .

Lemma 1, whose proof is trivial, allows to show a key boosting result: t GLYPH<16> 1 retrieves Matusita's loss, for which a near-optimal boosting rate is known [11] while t GLYPH<16> GLYPH<1>8 retrieves the empirical risk, which yields the worst possible guarantee [11]. In between, we have, for example, CART's Gini criterion for t GLYPH<16> 0 , which yields an intermediate boosting guarantee. Continuity with respect to t of the Bayes risks in between the empirical risk and Matusita's loss means the boosting ranges of the tempered loss cover the full known spectrum of boosting rates for t P rGLYPH<1>8 , 1 s . We know of no (differentiable and) proper CPE loss with such coverage. Note that (i) this is a non-constructive result as we do not associate a specific t for a specific rate, and (ii) the state-of-the-art boosting rates for DT induction do not seem to cover the case t P p 1 , 2 q , thus left as an open question.

7 Experiments

We have performed experiments on a testbed of 10 UCI domains, whose details are given in APP. (Section A3). Experiments were carried out using a 10-fold stratified cross-validation procedure.

To compare t -ADABOOST with ADABOOST, we ran t -ADABOOST with a first range of values of t P t 0 . 0 , 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 , 0 . 9 u . This is in the range of values covered by our convergence result for linear separators in Theorem 2. Our results on decision tree induction cover a much wider range, in particular for t P p 1 , 2 q . To assess whether this can be an interesting range to study, we added t GLYPH<16> 1 . 1 to the set of tested t values. When t ¡ 1 , some extra care is to be put into computations because the weight update becomes unbounded, in a way that is worse than ADABOOST. Indeed, as can be seen from (8), if µ j y i h j p x i q / GLYPH<1> 1 {p t GLYPH<1> 1 q (the example is badly classified by the current weak hypothesis, assuming wlog µ j ¡ 0 ), the weight becomes infinity before renormalization. In our experiments, picking a value of t close to 2 clearly shows this problem, so to be able to still explore whether t ¡ 1 can be useful, we picked a value close to 1 , namely t GLYPH<16> 1 . 1 , and checked that in our experiments this produced no such numerical issue. We also considered training clipped and not clipped models.

All boosting models were trained for a number of J GLYPH<16> 20 decision trees (The appendix provides experiments on training bigger sets). Each decision tree is induced using the tempered loss with the corresponding value of t (see Theorem 4) following the classical top-down template, which consists in growing the current heaviest leaf in the tree and picking the best split for the leaf chosen. We implemented t -ADABOOST exactly as in Section 5, including computing leveraging coefficients as suggested. Thus, we do not scale models. More details are provided in the appendix. In our experiments, we also included experiments on a phenomenon highlighted more than a decade ago [15] and fine-tuned more recently [16], the fact that a convex booster's model is the weakest link when it has to deal with noise in training data. This is an important task because while the tempered exponential loss is convex, it does not fit into the blueprint loss of [15, Definition 1] because it is not C 1 if t GLYPH<24> 1 . One might thus wonder how t -ADABOOST behaves when training data is affected by noise. Letting η denote the proportion of noisy data in the training sample, we tried η P t 0 . 0 , 0 . 1 u (The appendix provides experiments on more noise levels). We follow the noise model of [15] and thus independently flip the true label of each example with probability η .

For each run, we recorded the average test error and the average maximum and minimum co-density weight. Table 1 presents a subset of the results obtained on three domains. Table 2 presents a more synthetic view in terms of statistical significance of the results for t GLYPH<24> 1 vs. t GLYPH<16> 1 (ADABOOST). The table reports only results for t ¥ 0 . 6 for synthesis purposes. Values t GLYPH<160> 0 . 6 performed on average slightly worse than the others but on some domains, as the example of abalone suggests in Table 2 (the plots include all values of t tested in r 0 , 1 . 1 s ), we clearly got above-par results for such small values of t , both in terms of final test error but also fast early convergence to low test error. This comment can be generalized to all values of t .

The weights reveal interesting patterns as well. First, perhaps surprisingly, we never encountered the case where weights switch off, regardless of the value of t . The average minimum weight curves of Table 1 generalize to all our tests (see the appendix). This does not rule out the fact that boosting for a much longer number of iterations might lead to weights switching off/on, but the fact that this does not happen at least early during boosting probably comes from the fact that the leveraging coefficients for weights ( µ . ) are bounded. Furthermore, their maximal absolute value is all the smaller as t decreases to 0 . Second, there is a pattern that also repeats on the maximum weights, not on all domains but on a large majority of them and can be seen in abalone and adult in Table 1: the maximum weight of ADABOOST tends to increase much more rapidly compared to t -ADABOOST with t GLYPH<160> 1 . In the latter case, we almost systematically observe that the maximum weight tends to be upperbounded, which is not the case for ADABOOST (the growth of the maximal weight looks almost linear). Having bounded weights could be of help to handle numerical issues of (ada)boosting [14].

Our experiments certainly confirm the boosting nature of t -ADABOOST if we compare its convergence to that of ADABOOST: more often than not, it is in fact comparable to that of ADABOOST. While this applies broadly for t ¥ 0 . 6 , we observed examples where much smaller values (even t GLYPH<16> 0 . 0 ) could yield such fast convergence. Importantly, this applies to clipped models as well and it is important to notice because it means attaining a low "boosted" error does not come at the price of learning models with large range. This is an interesting property: for t GLYPH<16> 0 . 0 , we would be guaranteed that the computation of the clipped prediction is always in rGLYPH<1> 1 , 1 s . Generalizing our comment on small values of t above, we observed that an efficient tuning algorithm for t could be able to get very substantial leverage over ADABOOST. Table 2 was crafted for a standard limit p -val of 0.1 and "blurs" the best results that can be obtained. On several domains ( winered , abalone , eeg , creditcard , adult ), applicable p -values for which we would conclude that some t GLYPH<24> 1 performs better than t GLYPH<16> 1 drop in

between 7 E GLYPH<1> 4 and 0 . 05 . Unsurprisingly, ADABOOST also manages to beat significantly alternative values of t in several cases. Our experiments with training noise ( η GLYPH<16> 0 . 1 ) go in the same direction. Looking at Table 1, one could eventually be tempted to conclude that t slightly smaller than 1.0 may be a better choice than adaboosting ( t GLYPH<16> 1 ), as suggested by our results for t GLYPH<16> 0 . 9 , but we do not think this produces a general "rule-of-thumb". There is also no apparent "noise-dependent" pattern that would obviously separate the cases t GLYPH<160> 1 from t GLYPH<16> 1 even when the tempered exponential loss does not fit to [15]'s theory. Finally, looking at the results for t ¡ 1 also yields the same basic conclusions, which suggests that boosting can be attainable outside the range covered by our theory (in particular Theorem 2).

All this brings us to the experimental conclusion that the question does not reside on opposing the case t GLYPH<24> 1 to the case t GLYPH<16> 1 . Rather, our experiments suggest - pretty much like our theory does - that the actual question resides in how to efficiently learn t on a domain-dependent basis. Our experiments indeed demonstrate that substantial gains could be obtained, to handle overfitting or noise.

8 Discussion, limitations and conclusion

ADABOOST is one of the original and simplest Boosting algorithms. In this paper we generalized ADABOOST to maintaining a tempered measure over the examples by minimizing a tempered relative entropy. We kept the setup as simple as possible and therefore focused on generalizing ADABOOST. However more advanced boosting algorithms have been designed based on relative entropy minimization subject to linear constraints. There are versions that constrain the edges of all past hypotheses to be zero [36]. Also, when the maximum margin of the game is larger than zero, then ADABOOST cycles over non-optimal solutions [27]. Later Boosting algorithms provably optimize the margin of the solution by adjusting the constraint value on the dual edge away from zero (see e.g. [24]). Finally, the ELRP-Boost algorithm optimizes a trade off between relative entropy and the edge [35]. We conjecture that all of these orthogonal direction have generalizations to the tempered case as well and are worth exploring.

These are theoretical directions that, if successful, would contribute to bring more tools to the design of rigorous boosting algorithms. This is important because boosting suffers several impediments, not all of which we have mentioned: for example, to get statistical consistency for ADABOOST, it is known that early stopping is mandatory [5]. More generally, non-Lipschitz losses like the exponential loss seem to be harder to handle compared to Lipschitz losses [33] (but they yield in general better convergence rates). The validity of the weak learning assumption of boosting can also be discussed, in particular regarding the negative result of [15] which advocates, beyond just better (ada)boosting, for boosting for more classes of models / architectures [16]. Alongside this direction, we feel that our experiments on noise handling give a preliminary account of the fact that there is no "one t fits all" case, but a much more in depth analysis is required to elicit / tune a "good" t . This is a crucial issue for noise handling [16], but as we explain in Section 7, this could bring benefits in much wider contexts as well.

Acknowledgments

The authors thank the reviewers for numerous comments that helped improving the paper's content.

References

N. Alon, A. Gonen, E. Hazan, and S. Moran. Boosting simple learners. In STOC'21 , 2021.

S.-I. Amari. Information Geometry and Its Applications . Springer-Verlag, Berlin, 2016.

E. Amid, F. Nielsen, R. Nock, and M.-K. Warmuth. Optimal transport with tempered exponential measures. CoRR , abs/2309.04015, 2023.

E. Amid, R. Nock, and M.-K. Warmuth. Clustering above exponential families with tempered exponential measures. In 26 th AISTATS , 2023.

P. Bartlett and M. Traskin. Adaboost is consistent. In NIPS*19 , 2006.

L. Breiman, J. H. Freidman, R. A. Olshen, and C. J. Stone. Classification and regression trees . Wadsworth, 1984.

M. Collins, R. Schapire, and Y. Singer. Logistic regression, adaboost and Bregman distances. In Proc. of the 13 th International Conference on Computational Learning Theory , pages 158-169, 2000.

Y. Freund and R. E. Schapire. A Decision-Theoretic generalization of on-line learning and an application to Boosting. J. Comp. Syst. Sc. , 55:119-139, 1997.

J. Friedman, T. Hastie, and R. Tibshirani. Additive Logistic Regression : a Statistical View of Boosting. Ann. of Stat. , 28:337-374, 2000.

M.J. Kearns. Thoughts on hypothesis boosting, 1988. ML class project.

M.J. Kearns and Y. Mansour. On the boosting ability of top-down decision tree learning algorithms. In Proc. of the 28 th ACM STOC , pages 459-468, 1996.

J. Kivinen and M.-K. Warmuth. Boosting as entropy projection. In COLT'99 , pages 134-144, 1999.

D.-E. Knuth. Two notes on notation. The American Mathematical Monthly , 99(5):403-422, 1992.

R. Kohavi. Improving accuracy by voting classification algorithms: Boosting, bagging, and variants. In Workshop on Computation-Intensive Machine Learning Techniques , 1998.

P.-M. Long and R.-A. Servedio. Random classification noise defeats all convex potential boosters. MLJ , 78(3):287-304, 2010.

Y. Mansour, R. Nock, and R.-C. Williamson. Random classification noise does not defeat all convex potential boosters irrespective of model choice. In 40 th ICML , 2023.

J. Naudts. Generalized thermostatistics . Springer, 2011.

L. Nivanen, A. Le Méhauté, and Q.-A. Wang. Generalized algebra within a nonextensive statistics. Reports on Mathematical Physics , 52:437-444, 2003.

R. Nock and F. Nielsen. A R eal Generalization of discrete AdaBoost. Artificial Intelligence , 171:25-41, 2007.

R. Nock and F. Nielsen. On the efficient minimization of classification-calibrated surrogates. In NIPS*21 , pages 1201-1208, 2008.

R. Nock and F. Nielsen. The phylogenetic tree of Boosting has a bushy carriage but a single trunk. PNAS , 117:8692-8693, 2020.

R. Nock and R.-C. Williamson. Lossless or quantized boosting with integer arithmetic. In 36 th ICML , pages 4829-4838, 2019.

J. R. Quinlan. C4.5 : programs for machine learning . Morgan Kaufmann, 1993.

G. Rätsch and M.-K. Warmuth. Efficient margin maximizing with boosting. JMLR , pages 2131-2152, december 2005.

M.-D. Reid and R.-C. Williamson. Composite binary losses. JMLR , 11:2387-2422, 2010.

M.-D. Reid and R.-C. Williamson. Information, divergence and risk for binary experiments. JMLR , 12:731-817, 2011.

C. Rudin, I. Daubechies, and R.-E. Schapire. Dynamics of adaboost: cyclic behavior and convergence of margins. JMLR , pages 1557-1595, December 2004.

L.-J. Savage. Elicitation of personal probabilities and expectations. J. of the Am. Stat. Assoc. , pages 783-801, 1971.

R. E. Schapire. The strength of weak learnability. MLJ , pages 197-227, 1990.

R. E. Schapire and Y. Singer. Improved boosting algorithms using confidence-rated predictions. MLJ , 37:297-336, 1999.

T. Sypherd, R. Nock, and L. Sankar. Being properly improper. In 39 th ICML , 2022.

M. Telgarsky. A primal-dual convergence analysis of boosting. JMLR , 13:561-606, 2012.

M. Telgarsky. Boosting with the logistic loss is consistent. In 26 th COLT , pages 911-965, 2013.

L. G. Valiant. A theory of the learnable. Communications of the ACM , 27:1134-1142, 1984.

M.-K. Warmuth, K.-A. Glocer, and S.-V.-N. Vishwanathan. Entropy regularized LPBoost. In Algorithmic Learning Theory , pages 256-271. Springer Berlin Heidelberg, 2008.

M.-K. Warmuth, J. Liao, and G. Rätsch. Totally corrective boosting algorithms that maximize the margin. In icml '06: proceedings of the 23rd international conference on machine learning , pages 1001-1008, 2006.

Appendix

Abstract

This is the Appendix to Paper "Boosting with Tempered Exponential Measures". To differentiate with the numberings in the main file, the numbering of Theorems, Lemmata, Definitions is letter-based (A, B, ...).

Table of contents

I A short primer on Tempered Exponential Measures

We describe here the minimal amount of material necessary to understand how our approach to boosting connects to these measures. We refer to [4] for more details. With a slight abuse of notation, we define the perspective transforms p log t q GLYPH<6> p z q . GLYPH<16> t GLYPH<6> GLYPH<4> log t GLYPH<6> p z { t GLYPH<6> q and p exp t q GLYPH<6> p z q . GLYPH<16> t GLYPH<6> GLYPH<4> exp t GLYPH<6> p z { t GLYPH<6> q . Recall that t GLYPH<6> . GLYPH<16> 1 {p 2 GLYPH<1> t q .

Definition A. [4] A tempered exponential measure ( TEM ) family is a set of unnormalized densities in which each element admits the following canonical expression:

where θ is the element's natural parameter, ϕ p x q is the sufficient statistics and

is the (convex) cumulant, ξ being a base measure (implicit).

Except for t GLYPH<16> 1 (which reduces a TEM family to a classical exponential family), the total mass of a TEM is not 1 (but it has an elegant closed form expression [4]). However, the exponentiated q 1 { t GLYPH<6> t | θ does sum to 1. In the discrete case, this justifies extending the classical simplex to what we denote as the co-simplex.

The connection between t -ADABOOST's update and TEM's is immediate from the equation's update ((4) in MF). We can show that ˜ ∆ m can also be represented as TEMs.

Lemma A. ˜ ∆ m is a (discrete) family of tempered exponential measures.

Proof. We proceed as in [2, Section 2.2.2] for exponential families: let q P ˜ ∆ m , which we write

J π K , the Iverson bracket [13], takes value 1 if Boolean predicate π is true (and 0 otherwise). We create m GLYPH<1> 1 natural parameters and the cumulant,

and end up with (22) also matching the atom mass function

which clearly defines a tempered exponential measure over r m s . This ends the proof of Lemma A.

II Supplementary material on proofs

II.1 Proof of Theorem 1

To improve readability, we remove dependency in t in normalization coefficient Z . We use notations from [4, proof of Theorem 3.2] and denote the Lagrangian

which yields B L {B ˜ q i GLYPH<16> log t ˜ q i GLYPH<1> log t q i GLYPH<0> λ ˜ q 1 GLYPH<1> t i GLYPH<1> ν i GLYPH<0> µu i ( λ absorbs factor 2 GLYPH<1> t ), and, rearranging (absorbing factor 1 GLYPH<1> t in ν i ),

We see that λ GLYPH<24> GLYPH<1> 1 {p 1 GLYPH<1> t q otherwise the Lagrangian drops its dependence in the unknown. In fact, the solution necessarily has 1 GLYPH<0> p 1 GLYPH<1> t q λ ¡ 0 . To see this, we distinguish two cases: (i) if some u k GLYPH<16> 0 , then since log t q k ¥ GLYPH<1> 1 {p 1 GLYPH<1> t q there would be no solution to (24) if 1 GLYPH<0> p 1 GLYPH<1> t q λ GLYPH<160> 0 because of the KKT conditions ν i ¥ 0 , @ i P r m s ; (ii) otherwise, if all u k GLYPH<24> 0 , @ k P r m s , then there must be two coordinates of different signs otherwise there is no solution to our problem (3) (main file, we must have indeed ˜ q ¥ 0 because of the co-simplex constraint). Thus, there exists at least one coordinate k P r m s for which GLYPH<1>p 1 GLYPH<1> t q µu k ¡ 0 and since log t q k ¥ GLYPH<1> 1 {p 1 GLYPH<1> t q (definition of log t ) and ν k ¥ 0 (KKT conditions), the RHS of (24) for i GLYPH<16> k is ¡ 0 , preventing 1 GLYPH<0> p 1 GLYPH<1> t q λ GLYPH<160> 0 in the LHS.

We thus have 1 GLYPH<0>p 1 GLYPH<1> t q λ ¡ 0 . The KKT conditions p ν i ¥ 0 , ν i ˜ q i GLYPH<16> 0 , @ i P r m sq yield the following: 1 GLYPH<0> p 1 GLYPH<1> t qp log t q i GLYPH<1> µu i q ¡ 0 imply ν i GLYPH<16> 0 and 1 GLYPH<0> p 1 GLYPH<1> t qp log t q i GLYPH<1> µu i q / 0 imply ˜ q 1 GLYPH<1> t i GLYPH<16> 0 so we get the necessary form for the optimum:

where λ or Z t . GLYPH<16> exp t λ ensures normalisation for the co-density. Note that we have a simplified expression for the co-density:

with Z co t . GLYPH<16> Z 1 { t GLYPH<6> t GLYPH<16> GLYPH<176> i p ji b t GLYPH<6> exp t GLYPH<6> pGLYPH<1> µu i { t GLYPH<6> q . For the analytic form in (25), we can simplify the Lagrangian to a dual form that depends on µ solely:

The proof of (5) (main file) is based on a key Lemma.

Lemma B. For any ˜ q having form (25) such that ˜ q J u GLYPH<16> 0 , D p µ q GLYPH<16> GLYPH<1> log t Z t p µ q .

Proof. For any ˜ q having form (25), denote

We first compute (still using λ . GLYPH<16> log t Z t p µ q for short):

Remark that in the last identity, we have put back summations over the complete set r m s of indices. We note that B GLYPH<16> 0 because ˜ q J u GLYPH<16> 0 . We then remark that without replacing the expression of ˜ q , we have in general for any ˜ q P ˜ ∆ m :

and we can check that for any ˜ q , q P ˜ ∆ m , E GLYPH<16> ∆ p ˜ q } q q . We then develop ∆ p ˜ q } q q with a partial replacement of ˜ q by its expression:

Rearranging gives that for any ˜ q , q P ˜ ∆ m such that (i) ˜ q has the form (25) for some µ P R and (ii) ˜ q J u GLYPH<16> 0 ,

as claimed. This ends the proof of Lemma B.

We thus get from the definition of the dual that µ GLYPH<16> arg max GLYPH<1> log t Z t p µ q GLYPH<16> arg min Z t p µ q . We have the explicit form for Z t :

where r m s GLYPH<6> is defined in (28). We remark that the last expression is differentiable in µ , and get

so

and we get that any critical point of Z t p µ q satisfies ˜ q p µ q J u GLYPH<16> 0 . A sufficient condition to have just one critical point, being the minimum sought is the strict convexity of Z t p µ q . The next Lemma provides the proof that it is for all t ¡ 0 .

Proof. After simplifications, we have

where we have let Q i . GLYPH<16> exp t p log t q i GLYPH<1> µu i q ¥ 0 . Since a 2 GLYPH<0> b 2 ¥ 2 ab , we note that for any i GLYPH<24> k ,

so we split (33) in two terms and get

where we have used (34) in (35). Since Z t p µ q ¡ 0 , we get the statement of Lemma C after reorganising (36).

Lemma C shows the strict convexity of Z t p µ q for any t ¡ 0 . The case t GLYPH<16> 0 follows by direct differentiation: we get after simplification

Cauchy-Schwartz inequality allows to conclude that Z 2 t p µ q ¥ 0 and is in fact ¡ 0 unless ˜ q is collinear to u . This completes the proof of Theorem 1.

II.2 Proof of Theorem 2

The proof involves several arguments, organized into several subsections. Some are more general than what is strictly needed for the proof of the Theorem, on purpose.

II.II.2.1 Clipped summations

For any δ ¥ 0 , we define clipped summations of the sequence of ordered elements v 1 , v 2 , ..., v J : if J ¡ 1 ,

and the base case ( J GLYPH<16> 1 ) is obtained by replacing the inner sum by 0. We also define the doubly clipped summation:

with the same convention for the base case. We prove a series of simple but useful properties of the clipped summation.

Lemma D. The following properties hold true for clipped summation:

(doubly) clipped summations are noncommutative;

(doubly) clipped summations are ordinary summation in the limit: for any J ¥ 1 and any sequence v 1 , v 2 , ..., v J ,

clipped summations sandwich ordinary summation and the doubly clipped summation: for any δ ¥ 0 , any J ¥ 1 and any sequence v 1 , v 2 , ..., v J ,

Proof. Noncommutativity follows from simple counterexamples: for example, for v . GLYPH<16> GLYPH<1> 1 and w . GLYPH<16> 2 , if we fix v 1 . GLYPH<16> v, v 2 . GLYPH<16> w , then p 0 q , 2 j GLYPH<16> 1 v j GLYPH<16> 1 while p 0 q , 2 j GLYPH<16> 1 v 3 GLYPH<1> j GLYPH<16> GLYPH<1> 1 . Property [2.] is trivial. The set of leftmost inequalities of property [3.] can be shown by induction, noting the base case is trivial and otherwise, using the induction hypothesis in the leftmost inequality,

and similarly

A similar argument holds for the set of rightmost inequalities: for example, the induction's general case holds

for the leftmost inequality. This ends the proof of Lemma D.

II.II.2.2 Unravelling weights

Then @ J ¥ 1 , weights unravel as:

.

Proof. We start for the case t GLYPH<160> 1 . We proceed by induction, noting first that the normalization constraint for the initial weights imposes q 1 i GLYPH<16> 1 { m 1 {p 2 GLYPH<1> t q GLYPH<16> 1 { m t GLYPH<6> and so (using p 1 GLYPH<1> t q t GLYPH<6> GLYPH<16> 1 GLYPH<1> t GLYPH<6> )

completing the base case. Using the induction hypothesis, we unravel at iteration J GLYPH<0> 1 :

which simplifies into (using p 1 GLYPH<1> t q t GLYPH<6> GLYPH<16> 1 GLYPH<1> t GLYPH<6> )

with

(we used twice the definition of clipped summation), which completes the proof of Lemma E for t GLYPH<160> 1 .

We now treat the case t ¡ 1 . The base induction is equivalent, while unraveling gives, instead of (39):

and, this time,

which completes the proof of Lemma E.

II.II.2.3 Introducing classifiers

Ordinary linear separators Suppose we have a classifier

where µ j P R , @ j P r J s . We remark that J z GLYPH<24> r K / exp 2 GLYPH<1> t t pGLYPH<1> zr q for any t / 2 and z, r P R , and z GLYPH<222>GLYPH<209> exp 2 GLYPH<1> t t pGLYPH<1> z q is decreasing for any t / 2 , so using [3.] in Lemma D, we get for our training

sample S . GLYPH<16> tp x i , y i q , i P r m su and any t GLYPH<160> 1 (from Lemma E),

where

Using Lemma E with those definitions, we get

because q J P ˜ ∆ m . We thus have proven the following Lemma.

Lemma F. For any t GLYPH<160> 1 and any linear separator

where Z tk is the normalization coefficient of q in (25) with u ji . GLYPH<16> y i h j p x i q ,

Clipped linear separators Suppose we have a classifier ( t GLYPH<160> 1 )

We can now replace (44) by

The first identity has used the fact that y i P tGLYPH<1> 1 , 1 u , so it can be folded in the doubly clipped summation without changing its value, and the second inequality used [3.] in Lemma D. This directly leads us to the following Lemma.

Lemma G. For any t GLYPH<160> 1 and any clipped linear separator

where Z tk is the normalization coefficient of q in (25) with u ji . GLYPH<16> y i h j p x i q ,

II.II.2.4 Geometric convergence of the empirical risk

To get the right-hand side of (47) and (49) as small as possible, we can independently compute each µ j so as to minimize

We proceed in two steps, first computing a convenient upperbound for (50), and then finding the µ that minimizes this upperbound.

Step 1 : We distinguish two cases depending on weight q ji . Let r m s GLYPH<0> j . GLYPH<16> t i : q ji ¡ 0 u and r m s : j . GLYPH<16> t i : q ji GLYPH<16> 0 u :

Case 1 i P r m s GLYPH<0> j . Let r ji . GLYPH<16> u ji { q 1 GLYPH<1> t ji and suppose R j ¡ 0 is a real that satisfies

For any convex function f defined on rGLYPH<1> 1 , 1 s , we have f p z q / pp 1 GLYPH<0> z q{ 2 q GLYPH<4> f p 1 q GLYPH<0> pp 1 GLYPH<1> z q{ 2 q GLYPH<4> f pGLYPH<1> 1 q , @ z P rGLYPH<1> 1 , 1 s (the straight line is the chord crossing f at z GLYPH<16> GLYPH<1> 1 , 1 ). Because

Case 2 i P r m s : j . Let q : j ¡ 0 be a real that satisfies

Using the same technique as in case 1, we find for any i P r m s : j

Folding both cases into one and letting

we get after summation, using m : j . GLYPH<16> Card pr m s : j q and

that

Step 2 : we have our upperbound for (50). We now compute the minimizer µ GLYPH<6> of (55). If this minimizer satisfies

then it can be found by ordinary differentiation, as the solution to

which is equivalently

where we recall a a t b . GLYPH<16> p a GLYPH<1> b q{p 1 GLYPH<0> p 1 GLYPH<1> t q b q . Solving it yields

where M q p a, b q . GLYPH<16> pp a q GLYPH<0> b q q{ 2 q 1 { q is the power mean with exponent q . We now check (56). Lemma H. For any t P R , let

Then | µ j | / 1 {p R j | 1 GLYPH<1> t |q .

Proof. Equivalently, we must show

which is equivalent to showing

Define function f p z, t q . GLYPH<16> p 1 GLYPH<1> z 1 GLYPH<1> t q{p 1 GLYPH<0> z 1 GLYPH<1> t q over R ¥ 0 GLYPH<2> R : it is easy to check that for t / 1 , f p z, t q P rGLYPH<1> 1 , 1 s , and the symmetry f p z, t q GLYPH<16> GLYPH<1> f p z, 2 GLYPH<1> t q also allows to conclude that for t ¥ 1 , f p z, t q P rGLYPH<1> 1 , 1 s . This ends the proof of Lemma H.

For the expression of µ j in (57), we get from (55) the upperbound on Z 2 GLYPH<1> t tj p µ j q :

We conclude that for both sets of classifiers defined in Lemmata F and G, with the choice of µ j in (57), we get

To complete the proof of Theorem 2, we just need to elicit the best R j (51) and q : j (52); looking at their constraints suggests

This completes the proof of Theorem 2. We complete the proof by two Lemmata of additional useful results in the context of Algorithm t -ADABOOST, and finally an important remark on the interpretation of Theorem 2.

Lemma I. The following holds true: (i) ρ j P rGLYPH<1> 1 , 1 s ; (ii) if, among indexes not in r m s : j , there exists at least one index with u ji ¡ 0 and one index with u ji GLYPH<160> 0 , then for any µ GLYPH<24> 0 , Z 2 GLYPH<1> t tj p µ q ¡ 0 in (50) (in words, the new weigh vector q j GLYPH<0> 1 cannot be the null vector before normalization).

Proof. To show (i) for ρ / 1 , we write (using u . GLYPH<16> y h p x q , @ i P r m s for short),

showing ρ j / 1 . Showing ρ j ¥ GLYPH<1> 1 proceeds in the same way. Property (ii) is trivial.

Lemma J.

Proof. We remark that for t P r 0 , 1 q , z ¥ 0 , K 1 t p z q is concave and K 2 t p 0 q GLYPH<16> GLYPH<1>p 2 GLYPH<1> t q , so K 1 t p z q / GLYPH<1>p 2 GLYPH<1> t q z, @ z ¥ 0 , from which it follows by integration

and since 1 GLYPH<1> z / exp pGLYPH<1> z q , we get the statement of the Lemma.

Remark 1. The interpretation of Theorem 2 for t GLYPH<160> 1 are simplified to the case where there is no weight switching, i.e. m : j GLYPH<16> 0 , @ j . While we have never observed weight switching in our experiments - perhaps owing to the fact that we did never boost for a very long number of iterations or just because our weak classifiers, decision trees, were in fact not so weak -, it is interesting, from a theoretical standpoint, to comment on convergence when this happens. Let Q j . GLYPH<16> 1 GLYPH<0> m : j p q : j q 2 GLYPH<1> t and ˜ ρ j . GLYPH<16> Q j ρ j (Notations from Theorem 2). We note that ˜ ρ j GLYPH<19> β GLYPH<4> E p j r y i h j p x i qs , where p j lives on the simplex and | yh | / 1 , β / 1 . Using Lemma J and (12) (main file), to keep geometric convergence, it is roughly sufficient that Q j log Q j / p ˜ ρ j q 2 {p 2 t GLYPH<6> q . Since q : j is homogeneous to a tempered weight, one would expect in general m : j p q : j q 2 GLYPH<1> t ! 1 , so using the Taylor approximation Q j log Q j GLYPH<19> GLYPH<1> 1 GLYPH<0> Q j , one gets the refined sufficient condition for geometric convergence

What does that imply? We have two cases:

If this holds, then we have geometric convergence;

if it does not hold, then for a "large" number of training examples, we must have q ji GLYPH<16> 0 which, because of the formula for q (8) implies that all these examples receive the right class with a sufficiently large margin. Breaking geometric convergence in this case is not an issue: we already have a good ensemble.

II.3 Proof of Theorem 3

Starting from the proof of Theorem 2, we indicate the additional steps to get to the proof of Theorem 3. The key is to remark that our margin formulation has the following logical convenience:

We then remark that since z ¥ 0 / exp 2 GLYPH<1> t t p z q , we get

We then just have to branch to (44), replacing the J sign p H J p x i qq GLYPH<24> y i K s by J ν t pp x i , y i q , H q / θ K , which yields in lieu of (46) the sought inequality:

II.4 Proof of Theorem 4

The proof proceeds in three parts. Part (A) makes a brief recall on encoding linear classifiers with decision trees. Part (B) solves (6) in MF, i.e. finds boosting's leveraging coefficients as solution of:

we then simplify the loss obtained and elicit the conditional Bayes risk of the tempered loss, i.e. (20) in MF. Part (C) elicits the partial losses and shows properness and related properties.

Part (A): encoding linear models with a tree architecture We use the reduction trick of (author?) [16] to design a decision tree ( DT ) boosting procedure, find out the (concave) loss equivalently minimized, just like in classical top-down DT induction algorithms [6]. The trick is simple: a DT can be thought of as a set of constant linear classifiers. The prediction is the sum of predictions put at all nodes. Boosting fits those predictions at the nodes and percolating those to leaves gets a standard DT with real predictions at the leaves. Figure 2 provides a detailed description of the procedure. Let λ denote a leaf node of the current tree H , with H λ P R the function it implements for leaf λ . If parent p λ q denotes its parent node (assuming wlog it is not the root node), we have

Part (B): eliciting the Bayes risk of the tempered loss With our simple classifiers at hand, the tempered exponential loss Z 2 GLYPH<1> t tj in (14) (MF) can be simplified to loss

<latexi sh

1_b

64="oc/

P8Fk

D

UJ

3rG

+

HB

0

>A

COX

V

NT

E

5

w

I

q

R

Q9

S

g

pu mvZ

fY

W

j

y

M

L

7

K

d

z

n

2

<latexi sh

1_b

64="Z0mDfj rcJRw9EW

8

+5K

No

>A

T

H

V

2

k

S

F

X

Q

B

L

d

n

p

ug

y

I

q

7

/

3

vY

O

z

P

C

U

G

M

<latexi sh

1_b

64="d

YUo2

NE

S

O

WnAc

KH

9

V

>

C

X

B

T

w

0

pg

L

F

I

k

j

q

R

uJM7

r

v

/

+

y

5

8zf

m

3

G

Z

Q

P

D

<latexi sh

1_b

64="HqLOWfV7Nd

0T

cn5

By

m

I

>A

+

j

w2E

3

r

J

D

Q

o

Y

F

9

C

K

R

P

/

S

k

Z

z

M

p

8u

U

g

v

X

G

<latexi sh

1_b

64="

o

L8Ygy

7pU

K

+fF5vMm

>A

T

H

cjV

N

9w2E

kX

n

Q

B

d

0/ZSI

G

u

q

P

W

r

R

O

3

z

D

<latexi sh

1_b

64="y+

E

K

L

uo

P

A

F

V

0wQ

>

CNn

c

T

B

zS

JMU

D

r

2

q

IG

5

Z

3

j

Xdv

g

f/

9

k

m

8

HY

W

p7

R

O

<latexi sh

1_b

64="FpJLO9

y

/KjDR

0n8

2E

>A

T+H

c

V

w

d

I

mQ

M

B

zUS

N

Y

k

q

7

o

r

Z

g

3X

5

v

f

C

G

P

W

u

<latexi sh

1_b

64="kou/

KpQY

Bw jP

R

Oc

E

>A

T+H

V

2

N30mrq

J

7

D

f

Fn

5

9

C

y

d

S

Z

L

z

M

8

I

g

X

GU

v

W

<latexi sh

1_b

64="

G5qYv7P

p

rCmMI28

/

F

U

>A

Nn

c

VB

T9

E

3

0gZK

ykHD

WjS

R

X

o

Lw

J

+

df

u

Q

O

z

<latexi sh

1_b

64="oc/

P8Fk

D

UJ

3rG

+

HB

0

>A

COX

V

NT

E

5

w

I

q

R

Q9

S

g

pu mvZ

fY

W

j

y

M

L

7

K

d

z

n

2

<latexi sh

1_b

64="Z0mDfj rcJRw9EW

8

+5K

No

>A

T

H

V

2

k

S

F

X

Q

B

L

d

n

p

ug

y

I

q

7

/

3

vY

O

z

P

C

U

G

M

<latexi sh

1_b

64="FpJLO9

y

/KjDR

0n8

2E

>A

T+H

c

V

w

d

I

mQ

M

B

zUS

N

Y

k

q

7

o

r

Z

g

3X

5

v

f

C

G

P

W

u

<latexi sh

1_b

64="HqLOWfV7Nd

0T

cn5

By

m

I

>A

+

j

w2E

3

r

J

D

Q

o

Y

F

9

C

K

R

P

/

S

k

Z

z

M

p

8u

U

g

v

X

G

<latexi sh

1_b

64="3ONST

/8Lv5WI

pUw

Koq

7Qk

>A

H

cjV

9

2EF

X

Y

P

Z

drJu

0z

R

m

g

B

M

Df

+

G

C

n

y

<latexi sh

1_b

64="3ONST

/8Lv5WI

pUw

Koq

7Qk

>A

H

cjV

9

2EF

X

Y

P

Z

drJu

0z

R

m

g

B

M

Df

+

G

C

n

y

<latexi sh

1_b

64="d

YUo2

NE

S

O

WnAc

KH

9

V

>

C

X

B

T

w

0

pg

L

F

I

k

j

q

R

uJM7

r

v

/

+

y

5

8zf

m

3

G

Z

Q

P

D

<latexi sh

1_b

64="y+

E

K

L

uo

P

A

F

V

0wQ

>

CNn

c

T

B

zS

JMU

D

r

2

q

IG

5

Z

3

j

Xdv

g

f/

9

k

m

8

HY

W

p7

R

O

<latexi sh

1_b

64="HqLOWfV7Nd

0T

cn5

By

m

I

>A

+

j

w2E

3

r

J

D

Q

o

Y

F

9

C

K

R

P

/

S

k

Z

z

M

p

8u

U

g

v

X

G

<latexi sh

1_b

64="

o

L8Ygy

7pU

K

+fF5vMm

>A

T

H

cjV

N

9w2E

kX

n

Q

B

d

0/ZSI

G

u

q

P

W

r

R

O

3

z

D

<latexi sh

1_b

64="FpJLO9

y

/KjDR

0n8

2E

>A

T+H

c

V

w

d

I

mQ

M

B

zUS

N

Y

k

q

7

o

r

Z

g

3X

5

v

f

C

G

P

W

u

<latexi sh

1_b

64="3ONST

/8Lv5WI

pUw

Koq

7Qk

>A

H

cjV

9

2EF

X

Y

P

Z

drJu

0z

R

m

g

B

M

Df

+

G

C

n

y

<latexi sh

1_b

64="3ONST

/8Lv5WI

pUw

Koq

7Qk

>A

H

cjV

9

2EF

X

Y

P

Z

drJu

0z

R

m

g

B

M

Df

+

G

C

n

y

<latexi sh

1_b

64="kou/

KpQY

Bw jP

R

Oc

E

>A

T+H

V

2

N30mrq

J

7

D

f

Fn

5

9

C

y

d

S

Z

L

z

M

8

I

g

X

GU

v

W

<latexi sh

1_b

64="FpJLO9

y

/KjDR

0n8

2E

>A

T+H

c

V

w

d

I

mQ

M

B

zUS

N

Y

k

q

7

o

r

Z

g

3X

5

v

f

C

G

P

W

u

<latexi sh

1_b

64="

G5qYv7P

p

rCmMI28

/

F

U

>A

Nn

c

VB

T9

E

3

0gZK

ykHD

WjS

R

X

o

Lw

J

+

df

u

Q

O

z

<latexi sh

1_b

64="3ONST

/8Lv5WI

pUw

Koq

7Qk

>A

H

cjV

9

2EF

X

Y

P

Z

drJu

0z

R

m

g

B

M

Df

+

G

C

n

y

µ

1

b

1

µ

3

b

3

µ

1

b

1

µ

1

b

1

µ

2

b

2

µ

2

b

2

µ

4

b

4

µ

5

b

5

µ

5

b

5

µ

4

b

4

µ

3

b

3

µ

2

b

2

µ

1

b

1

+

+

+

+

+

x

2

a

2

x

2

< a

2

x

3

a

3

x

3

< a

3

x

2

a

2

x

2

< a

2

x

3

a

3

x

3

< a

3

where λ p x q is the leaf reached by observation x and λ p H q its set of leaf nodes of H , and H λ sums all relevant values in (60). Also, m GLYPH<0> λ , m GLYPH<1> λ denote the cardinal of positive and negative examples at λ and p λ . GLYPH<16> m GLYPH<0> λ {p m GLYPH<0> λ GLYPH<0> m GLYPH<1> λ q the local proportion of positive examples at λ , and finally r λ . GLYPH<16> p m GLYPH<0> λ GLYPH<0> m GLYPH<1> λ q{ m the total proportion of examples reaching λ .

Theorem A. If we compute µ λ the solution of (59) , we end up with the prediction H λ :

and the loss of the decision tree equals:

Proof. To compute µ λ , (6) is reduced to the examples reaching λ , that is, it simplifies to m GLYPH<0> λ exp t GLYPH<0> log t q 1 i GLYPH<1> H parent p λ q GLYPH<1> R λ µ λ h λ GLYPH<8> GLYPH<16> m GLYPH<1> λ exp t GLYPH<0> log t q 1 i GLYPH<0> H parent p λ q GLYPH<0> R λ µ λ h λ GLYPH<8> (66) that we solve for µ λ . Equivalently,

or, using exp t p u q{ exp t p v q GLYPH<16> exp t p u a t v q ,

after reorganizing:

which yields the prediction at λ :

We plug H λ back in the loss for all leaves and get, using q 1 i GLYPH<16> 1 { m 1 {p 2 GLYPH<1> t q :

We simplify. First, and then

and we can simplify the loss,

as claimed. This ends the proof of Theorem A.

Part (C): partial losses and their properties The proof relies on the following Theorem. We recall that a loss is symmetric iff its partial losses satisfy glyph[lscript] 1 p u q GLYPH<16> glyph[lscript] GLYPH<1> 1 p 1 GLYPH<1> u q , @ u P r 0 , 1 s [20] and differentiable iff its partial losses are differentiable.

Theorem B. Suppose t GLYPH<160> 2 . A set of partial losses having the conditional Bayes risk L p t q in (65) are

The tempered loss is then symmetric and differentiable. It is strictly proper for any t P pGLYPH<1>8 , 2 q and proper for t GLYPH<16> GLYPH<1>8 .

Proof. Symmetry and differentiability are straightforward. To check strict properness, we analyze the cases for t GLYPH<24> 1 (otherwise, it is Matusita's loss, and thus strictly proper), we compute the solution u to

To this end, let N p u q . GLYPH<16> v p 1 GLYPH<1> u q 2 GLYPH<1> t GLYPH<0> p 1 GLYPH<1> v q u 2 GLYPH<1> t and the q -sum

We also let D p u q . GLYPH<16> S 2 GLYPH<1> t 1 GLYPH<1> t p u, 1 GLYPH<1> u q . Noting L p t q p u, v q GLYPH<16> 2 2 GLYPH<1> t 1 GLYPH<1> t GLYPH<4> N p u q{ D p u q and D p u q GLYPH<24> 0 , @ u P r 0 , 1 s , the set of solutions of (83) are the set of solutions to N 1 p u q D p u q GLYPH<16> N p u q D 1 p u q , which boils down, after simplification, to

developing and simplifying yields a first simplified expression p 1 GLYPH<1> 2 v qp u p 1 GLYPH<1> u qq 1 GLYPH<1> t GLYPH<16> vu GLYPH<1> 1 p 1 GLYPH<1> u q 2 GLYPH<1> t GLYPH<1>p 1 GLYPH<1> v qp 1 GLYPH<1> u q GLYPH<1> t u 2 GLYPH<1> t , which, after reorganising to isolate expressions depending on v , yields

Assuming v P p 0 , 1 q , we multiply by p u p 1 GLYPH<1> u qq 1 GLYPH<1> t (we shall check u P p 0 , 1 q ) and simplify, which yields u p 1 GLYPH<1> u q GLYPH<0> u 2 GLYPH<16> v pp 1 GLYPH<1> u q 2 GLYPH<0> u 2 GLYPH<0> 2 u p 1 GLYPH<1> u qq , and indeed yields

and we check from (85) that if v GLYPH<16> 0 (resp. v GLYPH<16> 1 ), then necessarily u GLYPH<16> 0 (resp. u GLYPH<16> 1 ). To complete the proof, using the previous derivations, we can then simplify

which shows that if 2 GLYPH<1> t ¡ 0 but t GLYPH<24> GLYPH<1>8 , u GLYPH<16> v is a strict minimum of the pointwise conditional risk, completing the proof for strict properness. Strict properness is sufficient to show by a simple computation that L p t q is (65). For t GLYPH<16> GLYPH<1>8 , we pass to the limit and use the fact that we can also write

t GLYPH<209> GLYPH<1>8 is equivalent to t GLYPH<6> GLYPH<209> 0 GLYPH<0> . If u GLYPH<160> 1 { 2 , u {p 1 GLYPH<1> u q GLYPH<160> 1 and so we see that

because M 1 is the arithmetic mean. When u ¡ 1 { 2 , u {p 1 GLYPH<1> u q ¡ 1 and so this time

Hence, which is (twice) the partial loss of the 0/1 loss [26].

This ends the proof of Theorem 4.

Domain

Source

m

d

sonar

UCI

208

60

https://archive.ics.uci.edu/ml/datasets/wine+quality

winered

UCI

1 599

12

https://archive.ics.uci.edu/ml/datasets/wine+quality

abalone

UCI

4 177

9

https://archive.ics.uci.edu/ml/datasets/abalone

qsar

UCI

1 055

41

https://archive.ics.uci.edu/ml/datasets/QSAR+biodegradation

winewhite

UCI

4 898

12

https://archive.ics.uci.edu/ml/datasets/wine+quality

hillnonoise

UCI

1 212

101

http://archive.ics.uci.edu/ml/datasets/hill-valley

UCI

1 212

101

http://archive.ics.uci.edu/ml/datasets/hill-valley

eeg

UCI

14 980

15

https://archive.ics.uci.edu/ml/datasets/EEG+Eye+State

creditcard

GLYPH<6>

UCI

14 599

24

https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients

adult

UCI

32 561

15

https://archive.ics.uci.edu/ml/datasets/adult

III Supplementary material on experiments

III.1 Domains

Table A3 presents the 10 domains we used for our experiments.

III.2 Implementation details and full set of experiments on linear combinations of decision trees

Summary This Section depicts the full set of experiments summarized in Table 2 (MF), from Table A4 to Table A15. Tables are ordered in increasing size of the domain (Table A3). In all cases, up to J GLYPH<16> 20 trees have been trained, of size 15 (total number of nodes, except the two biggest domains, for which the size is 5). For all datasets, except creditcard and adult , we have tested t in the complete range, t P t 0 . 0 , 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 , 0 . 9 , 1 . 0 , 1 . 1 u (the MF only reports results for t ¥ 0 . 6 ), and in all cases, models both clipped and not clipped. For each dataset, we have set a 10-fold stratified cross-validation experiment, and report the averages for readability (Table 2 in MF gives the results of a Student paired t -test on error averages for comparison, limit p -val = 0.1). We also provide two examples of training error averages for domains hillnoise and hillnonoise (Tables A10 and A12).

Implementation details of t -ADABOOST First, regarding file format, we only input a .csv file to t -ADABOOST. We do not specify a file with feature types as in ARFF files. t -ADABOOST recognizes the type of each feature from its column content and distinguishes two main types of features: numerical and categorical. The distinction is important to design the splits during decision tree induction: for numerical values, splits are midpoints between two successive observed values. For categorical, splits are partitions of the feature values in two non-empty subsets. Our implementation of t -ADABOOST (programmed in Java) makes it possible to choose t not just in the range of values for which we have shown that boosting-compliant convergence is possible ( t P r 0 , 1 s ), but also t ¡ 1 . Because we thus implement ADABOOST ( t GLYPH<16> 1 ) but also for t ¡ 1 , weights can fairly easily become infinite, we have implemented a safe-check during training, counting the number of times the weights become infinite or zero (note that in this latter case, this really is a problem just for ADABOOST because in theory this should never happen unless the weak classifiers achieve perfect (or perfectly wrong) classification), but also making sure leveraging coefficients for classifiers do not become infinite for ADABOOST, a situation that can happen because of numerical approximations in encoding. In our experiments, we have observed that none of these problematic cases did occur (notice that this could not be the case if we were to boost for a large number of iterations). We have implemented algorithm t -ADABOOST exactly as specified in MF. The weak learner is implemented to train a decision tree in which the stopping criterion is the size of the tree reaching a user-fixed number of nodes. There is thus no pruning. Also, the top-down induction algorithm proceeds by iteratively picking the heaviest leaf in the tree and then choosing the split that minimizes the expected Bayes risk of the tempered loss, computing using the same t values as for t -ADABOOST, and with the constraint to not get pure leaves (otherwise, the real prediction at the leaves, which relies on the

perr (not clipped)

perr (clipped)

min weight

max weight

η

GLYPH<16>

0

.

0

η

GLYPH<16>

0

.

1

0.08

Adaboost (t=1.0)

0.1

Adaboost (t=1.0)

0.0045

Adaboost (t=1.0)

0.04

Adaboost (t=1.0)

0.07

0.06

0.8

0.07

0.09

0.08

0.8

0.0035

0.004

0.8

0.035

0.8

0.05

0.06

0.003

0.03

0.04

0.6

0.05

0.6

0.0025

90

0.025

0.6

0.04

0.002

0.03

0.03

0.4

0.0015

0.02

0.4

0.02

0.2

0.02

0.2

0.001

0.2

0.015

0.2

0.01

0.01

0.0005

0

0

0

0

0

0.01

2468101214161820

2468101214161820

2468101214161820

2468101214161820

#trees

#trees

#trees

#trees

0.13

daboost(t=1.0)

0.13

Adaboost (t=1.0)

0.0045

Adaboost (t=1.0)

0.035

Adaboost (t=1.0)

0.125

0.004

0.12

0.8

truerisk(avg)

0.115

0.12

0.11

0.8

0.0035

0.003

0.8g

0.03

0.8

600

0.1

0.6

0.4

0.105

0.095

0.1

0.6

0.4

u

0.0015

0.0025

0.002

90

0.015

0.02

0.6

0.4

0.08

0.2

0.09

0.2

0.001

0.2

0.01

0.2

0.085

0.0005

0.07

2468101214161820

0

0.08

2468101214161820

0

0

2468101214161820

0

0.005

2468101214161820

0

#trees

#trees

#trees

#trees

link of the loss, would be infinite for ADABOOST). In our implementation of decision-tree induction, when the number of possible splits exceeds a fixed number S (currently, 2 000), we pick the best split in a subset of S splits picked at random.

Results First, one may notice in several plots that the average test error increases with the number of trees. This turns out to be a sign of overfitting, as exemplified for domains hillnonoise and hillnoise , for which we provide the training curves. If we align the training curves at T GLYPH<16> 1 (the value is different because the splitting criterion for training the tree is different), we notice that the experimental convergence on training is similar for all values of t (Tables A10 and A12). The other key experimental result, already visible from Table 2 (MF), is that pretty much all tested values of t are necessary to get the best results. One could be tempted to conclude that t slightly smaller than 1 . 0 seems to be a good fit from Table 2 (MF), but the curves show that this is more a consequence of the Table being computed for J GLYPH<16> 20 trees. The case of eeg illustrates this phenomenon best: while small t -values are clearly the best when there is no noise, the picture is completely reversed when there is training noise. Notice that this ordering is almost reversed on creditcard and adult : when there is noise, small values of t tend to give better results. Hence, in addition to getting (i) a pruning mechanism that works for all instances of the tempered loss and (ii) a way to guess the right number of models in the ensemble, a good problem to investigate is in fact appropriately tuning t in a domain-dependent way. Looking at all plots reveals that substantial gains could be obtained with an accurate procedure (over the strategy that would be to always pick a fixed t , e.g. t GLYPH<16> 1 ).

III.3 Supplementary experiments: learning with more trees / against more noise

We have performed some additional experiments on several domains, on which we have trained bigger ensembles and / or considered more noise levels than in the previous experiments, with the objective to see if / when overfitting happens and how performances degrade with noise as a function of t . Table A16 summarizes the results obtained. A few comments we can make based on those experiments are:

overfitting can indeed happen ( sonar for η GLYPH<16> 0 . 4 ) but affects differently the algorithm depending on t and whether clipped models are learned instead of regular linear models; results also display that tuning t can also have the purpose of handling overfitting;

perr (not clipped)

perr (clipped)

min weight

max weight

η

GLYPH<16>

0

.

0

η

GLYPH<16>

0

.

1

Adaboost (t=1.0)

0.238

Adaboost (t=1.0)

0.0006

Adaboost (t=1.0)

0.004

Adaboost (t=1.0)

0.238

0.236

1

0.236

0.234

0.0005

0.0035

0.232

0.234

0.8

0.232

0.8

0.0004

0.8

0.003

0.8

0.23

0.228

0.23

0.6

0.0003

0.6

0.0025

0.6

0.226

0.224

0.226

0.4

0.0002

xew

0.0015

0.002

0.4

0.222

0.22

0.2

0.222

0.22

0.2

0.0001

0.2

0.001

0.2

0.218

0

0.218

0

0

0

0.0005

0

2468101214161820

2468101214161820

2468101214161820

2468101214161820

#trees

#trees

#trees

#trees

0.24

Adaboost (t=1.0)

0.24

Adaboost (t=1.0)

0.0006

Adaboost (t=1.0)

0.0028

Adaboost (t=1.0)

0.0026

()

0.235

0.23

0.8

truerisk(avg)

0.235

0.23

0.8

0.0004

0.0005

0.8

(avg)

0.0022

0.0024

0.8

0.225

90

0.225

0.6

0.0003

0.0018

0.002

0.6

0.22

0.4

0.22

0.4

uu

0.0002

xew

0.0016

0.0014

0.4

0.215

0.2

0.215

0.2

0.0001

0.2

0.0012

0.2

0.001

0.21

2468101214161820

0

0.21

2468101214161820

0

0

2468101214161820

0

0.0008

2468101214161820

0

#trees

#trees

#trees

#trees

perr (not clipped)

perr (clipped)

min weight

max weight

η

GLYPH<16>

0

.

0

η

GLYPH<16>

0

.

1

0.212

Adaboost (t=1.0)

0.214

0.215

Adaboost (t=1.0)

0.00025

Adaboost (t=1.0)

0.0016

Adaboost (t=1.0)

0.211

0.213

0.0002

0.0014

0.21

0.8

0.212

0.8

0.0012

0.8

0.21

0.6

0.00015

0.001

0.6

0.208

0.4

0.0001

0.0008

0.4

0.207

0.207

0.0006

0.206

0.2

0.206

0.205

0.2

5x10-5

0.2

0.0004

0.2

0.205

2468101214161820

0

0.204

2468101214161820

0

0

2468101214161820

0

0.0002

2468101214161820

0

#trees

#trees

#trees

#trees

0.215

0.214

Adaboost (t=1.0)

0.215

0.214

Adaboost (t=1.0)

0.00024

0.00022

Adaboost (t=1.0)

0.00085

0.0008

Adaboost(t=1.0)

0.213

0.212

0.211

0.21

0.8

0.212

0.213

0.21

0.8

.00020

0.00018

0.00055

0.00075

0.00065

0.0007

0.0006

0.8

0.6

0.209

0.0005

0.4

0.208

0.207

xew

0.00045

0.207

0.2

0.206

0.2

0.2

0.0004

0.2

0.206

0.205

0.00008

0.00035

0.205

0

0.204

0

0.00006

0

0.0003

0

2468101214161820

2468101214161820

2468101214161820

246810 12141618 20

#trees

#trees

#trees

#trees

clamped models can be very useful to handle overfitting ( sonar for η GLYPH<16> 0 . 4 , qsar for η ¥ 0 . 2 ); this provides another justification to learn clamped models;

the overall diversity of curves as a function of t supports the idea that good strategies could in fact tune t at training time and change its value with iterations.

perr (not clipped)

perr (clipped)

min weight

max weight

η

GLYPH<16>

0

.

0

η

GLYPH<16>

0

.

1

0.2

Adaboost (t=1.0)

0.19

Adaboost(t=1.0)

0.0009

Adaboost (t=1.0)

0.01

Adaboost (t=1.0)

0.19

0.18

0.0008

0.009

0.18

0.8

0.6

0.16

0.8

0.6

0.0007

0.0006

0.0005

0.8

0.006

0.8

0.6

0.16

0.0004

0.005

0.15

0.4

0.15

0.4

0.0003

xew

0.004

0.4

0.14

0.2

0.14

0.2

0.0002

0.0001

eee

0.2

0.003

0.002

0.2

0.13

2468101214161820

0.13

2468101214161820

0

0

2468101214161820

0

0.001

2468101214161820

0

#trees

#trees

#trees

#trees

0.19

Adaboost (t=1.0)

0.185

Adaboost (t=1.0)

6000'0

Adaboost (t=1.0)

0.0045

Adaboost (t=1.0)

0.185

0.18

0.0008

0.004

0.175

0.17

0.18

0.8

0.6

truerisk(avg)

0.175

0.165

0.17

0.8

0.6

0.0007

0.0005

0.0006

80

0.6

0.0035

0.003

0.8

0.6

0.165

0.155

0.16

0.15

0.2

0.4

0.155

0.16

0.15

0.2

0.4

18

uw

0.0004

0.0003

0.0002

0.2

0.0025

0.002

0.4

0.2

0.145

2468101214161820

0

0.145

2468101214161820

0

0.0001

24681012141618 20

0

0.0015

2468101214161820

0

#trees

#trees

#trees

#trees

perr (not clipped)

perr (clipped)

min weight

max weight

η

GLYPH<16>

0

.

0

η

GLYPH<16>

0

.

1

0.236

Adaboost (t=1.0)

0.236

Adaboost (t=1.0)

0.0002

Adaboost (t=1.0)

0.0018

Adaboost (t=1.0)

0.00018

0.0016

0.234

0.234

0.00016

0.8

0.8

（6Ae)

0.00014

0.8

0.23

0.6

0.23

0.6

0.00012

0.0001

0.0012

0.001

0.6

0.228

0.228

8x10-5

6x10-5

xew

0.0008

0.0006

0.4

0.226

0.2

0.226

0.2

4x10-5

2x10-5

0.2

0.0004

0.2

0.224

0

0.224

0

0

0

0.0002

0

2468101214161820

2468101214161820

2468101214161820

2468101214161820

#trees

#trees

#trees

#trees

0.238

Adaboost (t=1.0)

0.236

Adaboost (t=1.0)

0.00020

Adaboost (t=1.0)

0.0007

Adaboost(t=1.0)

0.236

0.234

0.00018

0.00065

0.234

0.8

0.232

0.8

0.0016

80

0.00055

0.0006

0.8

0.232

0.6

0.23

0.6

0.6

0.0005

0.6

0.228

0.23

0.4

0.228

0.226

0.4

18

O.00010

xeu

0.00045

0.0004

0.4

0.226

0.2

0.224

0.2

E

0.00008

0.2

0.00035

0.0003

0.2

0.224

0

0.222

0

0.00006

0

0.00025

0

2468101214161820

2468101214161820

2468101214161820

2468101214161820

#trees

#trees

#trees

#trees

perr (not clipped)

perr (clipped)

min weight

max weight

η

GLYPH<16>

0

.

0

η

GLYPH<16>

0

.

1

0.39

Adaboost（t=1.0)

0.385

Adaboost (t=1.0)

0.0009

Adaboost (t=1.0)

0.0016

Adaboost (t=1.0)

0.385

0.38

0.38

0.00085

0.0008

0.0015

0.375

0.365

0.36

0.37

0.6

0.8

0.375

0.365

0.37

0.48

0.8

0.6

-density

0.00075

0.00055

0.00065

0.0007

0.0006

0.0013

0.0012

0.0014

0.8

0.4

0.6

0.355

0.36

0.0005

xew

0.0011

0.345

0.35

0.2

0.355

0.2

0.00045

0.0004

0.2

0.001

0.2

0.34

0

0.35

0

0.00035

0

0.0009

0

2468101214161820

2468101214161820

2468101214161820

2468101214161820

#trees

#trees

#trees

#trees

0.42

Adaboost(t-1.0)

0.425

Adaboost(t=1.0)

0.0009

Adaboost (t=1.0)

0.0016

Adaboost (t=1.0)

0.415

0.42

0.00085

0.0015

0.0008

0.405

0.41

0.6

0.8

0.405

0.415

0.41

0.8

0.6

0.00075

0.00065

0.0007

0.0013

0.0014

0.6

0.8

0.395

0.4

0.395

0.4

0.4

u

0.00055

0.0006

xew

0.0012

0.0011

0.4

0.39

0.2

0.39

0.2

0.00045

0.0005

0.2

0.001

0.2

0.385

0

0.385

0

0.0004

0

0.0009

0

2468101214161820

2468101214161820

2468101214161820

2468101214161820

#trees

#trees

#trees

#trees

training err (not clipped)

training err (clipped)

η

GLYPH<16>

0

.

0

η

GLYPH<16>

0

.

1

0.375

Adaboost (t=1.0)

0.372

Adaboost (t=1.0)

0.37

0.37

0.368

0.8

0.364

0.366

0.8

0.36

0.6

0.362

0.6

0.36

0.355

0.4

0.358

0.4

0.356

0.35

0.2

0.354

0.2

0.352

0.345

0

0.35

2468101214161820

2468101214161820

#trees

#trees

0.41

Adaboost(t=1.0)

0.41

Adaboost(t=1.0)

0.405

0.405

0.395

0.4

0.8

emp.risk(avg)

0.395

0.4

0.8

0.39

0.6

0.39

0.6

0.385

0.4

0.385

0.4

0.38

0.38

0.2

0.2

0.375

0.375

0.37

0

0.37

0

2468101214161820

2468101214161820

#trees

#trees

perr (not clipped)

perr (clipped)

min weight

max weight

η

GLYPH<16>

0

.

0

η

GLYPH<16>

0

.

1

0.375

Adaboost (t=1.0)

0.365

Adaboost(t=1.0)

0.0009

Adaboost (t=1.0)

0.00135

Adaboost(t=1.0)

0.37

0.36

0.00085

0.0013

risk

0.355

0.365

0.36

0.35

0.8

0.6

0.345

0.355

0.35

0.8

0.6

0.00075

0.0008

0.0007

0.00125

0.00115

0.0012

0.8

0.6

0.345

0.335

0.34

0.2

0.335

0.34

0.2

0.4

0.00065

0.00055

0.0006

0.2

xe

0.00105

0.0011

0.001

0.4

0.2

0.33

0.33

0.0005

0.00095

0.325

0

0.325

0

0.00045

0

0.0009

2468101214161820

2468101214161820

2468101214161820

2468101214161820

#trees

#trees

#trees

#trees

0

Adaboost (t=1.0)

0.405

Adaboost(t=1.0)

0.0009

Adaboost (t=1.0)

0.0015

Adaboost (t=1.0)

0.395

0.4

0.00085

0.0014

0.385

0.39

0.6

0.8

0.385

0.395

0.39

0.8

0.6

0.00075

0.0008

0.0007

0.0012

0.8

0.6

0.38

0.00065

0.375

0.37

0.2

0.375

0.37

0.38

0.2

0.4

0.00055

0.0005

0.0006

0.2

0.0011

0.001

0.4

0.2

0.365

0

0.365

0

0.00045

0

0.0009

0

24681012141618 20

24681012141618 20

2468101214161820

2468101214161820

#trees

#trees

#trees

#trees

training err (not clipped)

training err (clipped)

η

GLYPH<16>

0

.

0

η

GLYPH<16>

0

.

1

Adaboost (t=1.0)

0.35

Adaboost(t=1.0)

0.355

0.345

0.345

0.35

0.8

emp.risk(avg)

0.335

0.34

0.8

0.34

0.6

0.33

0.6

0.335

0.4

0.325

0.4

0.325

0.33

0.2

0.315

0.32

0.2

0.32

0

0.31

2468101214161820

2468101214161820

#trees

#trees

0.376

Adaboost (t=1.0)

0.385

Adaboost (t=1.0)

0.374

0.372

0.38

0.37

0.8

0.8

0.6

0.37

0.6

0.4

0.365

0.4

0.36

0.2

0.36

0.2

0.358

0.356

0

0.355

0

2468101214161820

2468101214161820

#trees

#trees

perr (not clipped)

perr (clipped)

min weight

max weight

η

GLYPH<16>

0

.

0

η

GLYPH<16>

0

.

1

0.29

Adaboost (t=1.0)

0.286

Adaboost (t=1.0)

0.00007

Adaboost (t=1.0)

0.00024

Adaboost (t=1.0)

0.285

0.28

0.8

0.284

0.278

0.282

0.28

0.8

o.00005

0.00006

0.8

0.00020

0.00022

0.8

0.275

0.6

seeeese

0.6

0.27

0.4 80.00014

0.4

0.265

0.2

0.268

0.27

0.2

0.00002

0.2

0.00010

0.2

0.26

0

0.266

0

0.00001

0

0.00008

0

2468101214161820

2468101214161820

2468101214161820

2468101214161820

#trees

#trees

#trees

#trees

0.285

Adaboost (t=1.0)

0.285

Adaboost(t=1.0)

0.00007

Adaboost (t=1.0)

0.00020

Adaboost (t=1.0)

0.00007

0.28

0.28

0.00006

0.00018

0.8

0.8

0.000060

0.00016

0.8

0.27

90

0.27

0.6

0.00005

O0.00014

0.6

0.265

0.265

0.4

0.26

0.2

0.26

0.2

F0.00003

0.2

0.00010

0.2

0.00003

0.255

24681012141618 20

0

0.255

2468101214161820

0

0.00002

2468101214161820

0

0.00008

2468101214161820

0

#trees

#trees

#trees

#trees

1

perr (not clipped)

perr (clipped)

min weight

max weight

η

GLYPH<16>

0

.

0

η

GLYPH<16>

0

.

1

0.22

Adaboost (t=1.0)

1.1

0.216

Adaboost (t=1.0)

1.1

0.00006

Adaboost (t=1.0)

1.1

0.0004

Adaboost (t=1.0)

0.218

0.214

0.00006

0.00035

0.214

0.216

60

0.212

0.21

0.00005

0.0003

0.9

0.212

0.00025

0.208

0.206

0.21

0.7

0.208

0.206

0.204

0.7

80.00004

0.00003

380

0.7

0.00015

0.0002

0.8

0.7

0.204

0.6

0.202

0.6

0.00002

0.6

0.0001

0.6

2468101214161820

2468101214161820

2468101214161820

2468101214161820

#trees

#trees

#trees

#trees

0.218

Adaboost (t=1.0)

.1

0.216

Adaboost(t=1.0)

1.10.00006

Adaboost (t=1.0)

1.1

0.00024

Adaboost(t=1.0)

0.216

0.214

0.00006

0.00022

0.214

0.212

@0.00005

(6Ae)

0.0002

0.212

0.9

0.21

0.9

0.00018

0.208

0.21

0.206

0.208

0.8

0.00004

0.00016

0.8

0.206

0.7

0.204

0.7

0.00004

0.00014

0.7

0.204

0.6

0.202

0.6

0.00003

0.6

0.00012

0.6

2468101214161820

2468101214161820

2468101214161820

2468101214161820

#trees

#trees

#trees

#trees

perr (not clipped)

perr (clipped)

min weight

max weight

η

GLYPH<16>

0

.

0

η

GLYPH<16>

0

.

1

0.18

Adaboost (t=1.0)

0.178

Adaboost (t=1.0)

.1

3x10

Adaboost (t=1.0)

1.10.00050

Adaboost (t=1.0)

0.178

0.176

2.5x10-5

0.00045

0.174

0.176

0.9

(avg)

0.172

0.174

0.9

nco-density(

(6Ae)

2x10-5

60

p.00040

0.00035

0.00030

0.9

0.172

risk

0.17

1.5x10-5

0.168

0.166

0.17

0.7

0.166

0.164

0.8

0.7

1x10-5

5x10-6

0.7E0.00015

0.8

0.00020

0.00025

0.00010

0.8

0.7

0.164

0.6

0.162

0.6

0

0.60.00005

0.6

2468101214161820

2468101214161820

2468101214161820

2468101214161820

#trees

#trees

#trees

#trees

0.172

0.176

0.174

0.168

0.178

0.17

Adaboost (t=1.0)

0.9

0.172

0.174

0.178

0.176

0.17

0.18

Adaboost(t=1.0)

0.9

0.000016

0.000022

0.000020

0.000018

0.000024

0.000026

0.000028

Adaboost (t=1.0)

1.10.00018

1

0.00016

Adaboost (t=1.0)

0.9

0.166

0.8 ± 0.168

2

0.166

0.8

80.000014

.000012

0.00008

0.8

0.164

0.162

0.7

0.164

0.162

0.7

0.000010

0.000008

0.00006

0.7

0.16

2468101214161820

0.6

0.16

24681012141618 20

0.6

0.000006

2468101214161820

0.6 0.00004

2468101214161820

0.6

#trees

#trees

#trees

#trees

η

GLYPH<16>

0

0.1

0.2

0.3

0.4

winered

, clamped

sonar

, unclamped

sonar

, clamped

qsar

, unclamped

qsar

, clamped

0.265

0.255

0.245

0.26

0.25

Adaboost (t=1.0)

0.9 0.246

0.8 0.242

1.1 0.254

0.248

0.252

0.244

0.25

0.24

Adaboost (t=1.0)

60

1.1 0.272

0.8

0.264

0.258

0.268

0.266

0.262

0.256

0.26

0.27

Adaboost (t=1.0)

0.8 0.275

0.90.28

1.1

1

0.285

0.29

Adaboost (t=1.0)

1.1

0.9

0.8

0.355

0.345

0.36

0.35

0.34

Adaboost (t=1.0)

0.9

0.8

1.1

0.24

0.7 0.238

0.7 0.254

0.7

0.27

0.7 0.335

0.7

0.236

0.252

0.235

20 40 60 80 100 120 140

0.6 0.234

10 20 30 40 50 60 70 80 90100

0.60.25

10 20 30 40 50 60 70 80 90100

0.6 0.265

10 20 30 40 50 60 70 80 90100

0.60.33

10 20 30 40 50 60 70 80 90100

0.6

# trees

#trees

#trees

#trees

#trees

0.14

Adaboost (t=1.0)

1.1

0.2

Adaboost (t=1.0)

1.1 0.22

Adaboost (t=1.0)

1.1 0.28

Adaboost (t=1.0)

1.1 0.45

Adaboost (t=1.0)

0.12

0.19

0.21

0.27

0.44

0.1

10.08

0.9 0.17

0.18

6'0

0.2

0.9.0.25

0.26

0.90.42

0.43

60

0.06

0.04

0.02

0.8 0.16

0.7

0.15

0.14

0.7

8'0

0.17

0.19

0.18

AAAD

0.8 0.24

0.7

0.23

0.22

0.8

0.7

0.41

0.39

0.38

0.4

0.8

0.7

10 20 30 40 50 60 70 80 90100

0.6 0.13

10 20 30 40 50 60 70 80 90100

0.6 0.16

10 20 30 40 50 60 70 80 90100

0.6 0.21

10 20 30 40 50 60 70 80 90100

0.6 0.37

10 20 30 40 50 60 70 80 90100

0.6

#trees

#trees

#trees

#trees

# trees

0.16

Adaboost (t=1.0)

1.1 0.2

Adaboost (t=1.0)

1.1 0.205

Adaboost (t=1.0)

1.1 0.27

Adaboost (t=1.0)

1.1 0.45

Adaboost (t=1.0)

1.1

0

210.60

0.8 0.15

0.18

0.19

0.16

0.8

0.9

0.195

0.185

0.175

0.19

0.18

0.2

0.9

0.8

: 0.25

0.26

0.24

0.23

0.9. 0.42

0.44

0.41

0.43

0.9

0.8

0.04

0.02

0.7

0.14

0.13

0.165

0.7

0.39

0.4

0.7

Q

10 20 30 40 50 60 70 80 90100

0.6 0.12

10 20 30 40 50 60 70 80 90100

0.60.16

10 20 30 40 50 60 70 80 90100

0.6 0.21

10 20 30 40 50 60 70 80 90100

0.6 0.38

10 20 30 40 50 60 70 80 90100

0.6

#trees

#trees

#trees

#trees

#trees

0.17

0.18

Adaboost (t=1.0)

1.1

0.195

0.19

0.2

Adaboost (t=1.0)

1.1 0.205

0.2

Adaboost (t=1.0)

0.23

Adaboost (t=1.0)

0.265

0.2

Adaboost (t=1.0)

0.16

0.14

0.13

0.15

0.90.18

0.7 0.155

0.8 0.165

0.185

0.175

0.17

0.15

0.16

0.8

60

0.7

0.195

0.185

0.18

0.175

0.19

0.17

0.9

0.8

0.7

0.19

0.22

0.21

0.2

0.9

0.7

0.8

0.245

0.255

0.26

0.25

0.9

0.8

0.7

0.12

10 20 30 40 50 60 70 80 90100

0.6 0.145

10 20 30 40 50 60 70 80 90100

0.6 0.165

10 20 30 40 50 60 70 80 90100

0.6 0.18

10 20 30 40 50 60 70 80 90100

0.60.24

10 20 30 40 50 60 70 80 90100

0.6

#trees

# trees

#trees

#trees

# trees

0.175

0.135

0.145

0.165

0.155

0.16

0.18

0.17

0.15

0.14

Adaboost (t=1.0)

60

0.8

0.7 0.15

1.1

: 0.18

0.19

0.16

0.17

0.2

M

Adaboost (t=1.0)

0.7 0.165

60

80

1.1

: 0.185

0.195

0.175

0.19

0.18

0.17

0.2

Adaboost (t=1.0)

0.7 0.18

0.9

0.8

1.1 0.23

0.21

0.22

0.19

0.2

Adaboost (t=1.0)

0.9'0.255

0.8 0.245

0.7

0.27

0.265

0.24

0.25

0.26

Adaboost (t=1.0)

0.9

0.8

0.7

1.1

0.13

0.16

0.235

0.125

10 20 30 40 50 60 70 80 90100

0.6 0.14

10 20 30 40 50 60 70 80 90100

0.6 0.155

10 20 30 40 50 60 70 80 90100

0.6 0.17

10 20 30 40 50 60 70 80 90100

0.60.23

10 20 30 40 50 60 70 80 90100

0.6

# trees

#trees

#trees

#trees

#trees

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a generalization of the popular ADABOOST algorithm based on the use of the t-logorithm/exponential. Their method is derived by replacing the standard relative entropy with the _tempered_ relative entropy (introduced in eq. 2), and solving a constrained optimization problem (eq. 3). This results in a solution (eq.4) which is a tempered generalization of ADABOOST's exponential update. Their approach recovers the standard ADABOOST updates at t->1, and maintains the exponential convergence rate of ADABOOST for values of 't' between 0 and 1. A new family of tempered losses is derived from the loss that t-ADABOOST minimizes, and experimental results using t-ADABOOST+trees show significant improvements can be gained by tuning 't'.\n\nSTRENGTHS:\n- This paper has both strong theoretical and experimental results, and show compelling improvements over baselines\n- ADABOOST is a very popular and performant ensemble technique, and so this work has the potential to have significant applications for ML practitioners, and could easily be implemented in existing libraries (sklearn etc )\n- The paper is well written and easy to follow, and their algorithm is accompanied with a comprehensive theoretical analysis\n\nWEAKNESSES:\n- The major limitation is that performance is highly sensitive to the choice of t, and currently it looks like the only way to choose t is to perform expensive hyperparameter sweeps. Further, their Table 1 (should that be \"Figure 1\"?) seems to imply there is no rhyme or reason to the datasets which perform better with large/medium/small t. Unless practitioners have a way of learning/tuning t it's unlikely this approach will be adopted widely in practice\n- AFAICT, there are no ablation experiments testing the effect of the new losses, separate from the effect of the new exponential update. Which loss(es) were used in section 7? How do we know they work? How should a practitioner choose their loss? \n\n\n## Four dimensions \n\n**Originality:** \n- Are the tasks or methods new? \n  - Yes\n- Is the work a novel combination of well-known techniques?\n  - Yes, the work builds off developments in ADABOOST showing connections to exponential families and bregman divergences, and uses this to generalize ADABOOST by replacing exp/log with t-exp/t-log\n- Is it clear how this work differs from previous contributions? \n  - Yes, section 2 discusses this well.\n- Is related work adequately cited?\n  - Yes. \n   \n**Quality:** \n- Is the submission technically sound?\n  - Yes, the technical contributions seem sound.\n- Are claims well supported (e.g., by theoretical analysis or experimental results)? \n  - Theoretically yes, experimentally yes, with the exception of missing experiments evaluation the new losses \n- Are the methods used appropriate? \n  - Yes \n- Is this a complete piece of work or work in progress? \n  - Yes, modulo missing loss experiments \n- Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?\n  - yes\n\n**Clarity:** \n- Is the submission clearly written? \n  - Yes\n- Is it well organized? \n  - Yes\n- Does it adequately inform the reader? \n  - Yes\n**Significance:**\n- Are the results important? \n  - If the authors provide a mechanism of choosing t, yes the results would be important.\n- Are others (researchers or practitioners) likely to use the ideas or build on them? \n  - Yes \n- Does the submission address a difficult task in a better way than previous work? \n  - Yes \n- Does it advance the state of the art in a demonstrable way?\n  - Yes \n-  Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?\n   - Yes\n\nQUESTIONS:\n- In https://arxiv.org/abs/2107.00745 the authors show t is a highly sensitive hyperparameter, and best results are obtained using tiny perturbations away from 1. This is due to numerical problems associated with using a log-t density instead of the more numerically-stable log-density. Did the authors see similar effects? Might their grid search of [0,0.2, 0.4, 0.6, 0.8, 0.9] be too coarse? Could significant gains be achieved with a finer grid search? \n- If I'm a practitioner who is willing to do a grid search to find the best t, why wouldn't I spend that effort just tuning the hyperparameters of standard ADABOOST instead? If I take a hyperparameter-tuned ADABOOST, _then_ tune t, do I get significant gains compared to tuning t with untuned ADABOOST?  \n- With the introduction of the new losses in section 6, does that introduce a second t I need to tune, or should I use the same one? \n- Does the second column of table 1 indicate clamping isn't effective? \n- Do all/most values of t pick out the same \"difficult\" examples, or are the weights totally different for diff values of t? \n- Is there any relationship between number of examples N and number of features D, and the best t? i.e do \"tall+skinny\" datasets get different optimal t's than 'short and fat' datasets, and might this offer tuning suggestions to practitioners? What about the depth of the weak classifier? \n- ADABOOST can also be used for regression - would their approach work in this setting? A few comments about regression would be useful even if not feasible under the proposed approach.\n\nLIMITATIONS:\nThere does not appear to be a \"Limitations and Broader Impacts\" statement in this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a generalization of the classic adaboost algorithm to apply to a family of certain exponential losses, called TEMs (tempered  exponential measures). They demonstrate the validity of their approach both theoretically  and empirically.\n\nSTRENGTHS:\nThis technique allows their method to overcome numerical issues that typically arise when classic adaboost is employed. \nExperiments show that tuning t can lead to significant improvements, compared to adaboost. \nMoreover, the algorithm is simple and gives an interesting and practical generalization of adaboost.\n\nWEAKNESSES:\nMy main issue here is the notational choices and somewhat unclear technical presentation. \n\nNotation issues : \n- Intuition regarding the used notation (in particular the terms in Eq. 4) would have been helpful, and also for the definitions of log_t and exp_t. Especially as these are key parts of the main algorithm which is very simple otherwise. \n- Equation 5 is unclear - how is \\mu defined? and the notation Z_t(\\mu)  is confusing. \n- What is Card? (Line 140) is that short for cardinality? \n- Eq 14 - I’m assuming these are indicators returning +1/-1  ? \n\n\nBecause of the notation issues above, Theorem 2 is harder to parse. However, it does resemble in form to the standard bounds on adaboost - maybe it would be nice to emphasize that comparison more explicitly.\nAlso, consider giving more explanation of the meaning of theorem 2. \nMaybe the paragraphs below it try to do that, but it was a little dense and technical. \n\nI am also confused about Figure 1 - it is essentially plotting the convergence rate, the arrows indicate the case of t=0 and t=1, but is that something easily seen by the connection of the x-axis of rho to the value of t? Is it possible to also give a plot with an axis for values of t ? \n\nTable 1 does not say what each line indicates (it did say on the appendix though) but still not sure what is the red line?\n\nQUESTIONS:\nQuestions in the above comment\n\nLIMITATIONS:\nnot applicable", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a variant of AdaBoost algorithm based on a generalized exponential function parametrized by the temperature. The generalized exp function also induces a new criterion for splitting nodes of decision trees. The paper shows a training error bound of the AdaBoost variant. The experimental results show that the generalized AdaBoost often perform better than AdaBoost.\n\nSTRENGTHS:\nThe strength of the contribution is the potential advantages of the proposed exp function. The exp function can derive new Bregman divergences and be applied to online learning or Boosting. In a practical sense, the new exp function and updates derived based on it seem more robust in numerical computation. \n\nAnother strength is a theoretical guarantee of its convergence rate of training error, which is the same as AdaBoost's, when given weak hypotheses with edge gamma. This iteration bound is optimal in the worst case (when the final hypothesis is a majority vote).\n\nWEAKNESSES:\nA crucial weakness of the paper is the lack of explanation about the generalization ability of proposed algorithms. Previous boosting algorithms, including AdaBoost, are motivated by their margin maximization properties. In fact, standard generalization bounds of weighted-voting classifiers depend on the margin over the sample.  Furthermore, there are boosting algorithms explicitly designed to optimize the soft margin optimization problem (e.g., SoftBoost or entropy regularized LPBoost). So far, the paper only considers the convergence rate of the training error. \n\nSo far, theoretical results do not improve previous ones, say, the convergence rate. So, the merit or theoretical advantages of the proposed function and the resulting boosting algorithm are unclear yet. \n\n\nI read the rebuttal comments and the new analysis seems to resolve my concern.\n\nQUESTIONS:\n-Does the temperature control the smoothness of the function or resulting distributions over the sample? If so, please discuss the relationship between the previous boosting algorithms that keep distributions smooth (e.g., SmoothBoost by Servedio or AdaFlat by Gavinsly).\n\n\n\n\n-\n\nLIMITATIONS:\nAs raised above, the paper does not show any theoretical advantages of the proposed function. That is a huge drawback of the paper and thus the paper seems pre-matured.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a generalization of Ada Boost called “t-Ada Boost”. Boosting algorithms aggregate multiple weak classifiers into a strong classifier. Ada Boost is a well-established boosting algorithm.\nt-Ada Boost generalizes Ada Boost by introducing an additional parameter $t$, which is a “tempering” parameter. When $t=1$, the algorithm becomes identical to Ada Boost. Thus, the main focus of the paper is to analyze whether or not any benefit can be gained by selecting a different value of $t$, besides $t=1$. \nThe paper provides a thorough theoretical and experimental analysis of t-AdaBoost, for various values of $t$. The theoretical analysis shows that different values of $t$ are viable to study, and bounds the convergence rate. The experimental analysis shows that, while $t=1$ works best for some data sets, and some values of $t$ seem equivalent to AdaBoost on other data sets, there do exist some data sets for which various values of $t$ perform better than Ada Boost. So, the overall analysis suggests that $t$ would be a good additional parameter to introduce on top of AdaBoost. For some data sets, tuning $t$ could lead to improvements on top of AdaBoost, so it’s worth consideration.\n\nSTRENGTHS:\nI think the paper is well-written, and conducts a thorough analysis of the proposed technique. It is built upon well-respected approaches.\n\nWEAKNESSES:\nThe proposed method can have numerical instabilities, resulting from exponentiation, but the authors give reasonable consideration of this in their theory and experiments. This weakness is not very significant.\n\nAlso, see my experiment question below.\n\nQUESTIONS:\n-\tOriginally, I went looking for the full plots from the experiments, and did not see them in the supplement. I then realized that they occurred after the references. Perhaps add an additional entry in the “Table of Contents” would be appropriate, or else make sure the experiment plots are before the references.\n-\tPlease include the code, or a link to the repository, along with the final version.\n-\tFor some data sets, the plots see a high variability from changing the decision trees (see, for example, sonar, bottom). I know you mentioned that some of this might be due to overfitting. However, I also see that, in some cases, the error repeatedly rises and falls as the number of decision trees increases. This makes me wonder whether the variability here is something intrinsic. For example, if you rerun your experiments (perhaps with a very slight variation), do you get the same results? If not, then I think it is important to execute multiple runs and give a standard deviations, or other variance measure.\n\nLIMITATIONS:\nI don't think this paper has any negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a generalization of the ADABOOST algorithm using tempered exponential measures. To do this, they begin by introducing $log_t$ and $exp_t$ and the generalization of entropy. A first theorem is proposed to explain how to find the solution for the minimization of this entropy and an adaptation of the ADABOOST algorithm, called t-ADABOOST, is presented. Section 5 is dedicated to explaining the convergence of the t-ADABOOST algorithm and a theoretical study of its behavior. In section 6, the authors take the time to introduce tempered loss and the associated decision trees. The authors conclude with experiments and a discussion.\n\nSTRENGTHS:\nI think the paper is quite original in that the authors generalize ADABOOST using the functions $\\log_t$ and $\\exp_t$. In this way, they obtain an algorithm generalizing the former and extending the perspectives.\nI was interested in the reflections on $\\log_t$ and $\\exp_t$ and how the introduction of these functions forced the authors to rethink the rest of the procedure.\nThe authors propose theorems that are consistent with our expectations as readers of these new notions, and the order seems logical.\nFinally, the authors have chosen a complete path (new notions + theory + experiments + discussions), which makes the whole coherent in my opinion.\n\nWEAKNESSES:\nFor me, a big weakness of the paper is the introduction of numerous notations that I didn't always find interesting (for example, I didn't understand the use of clamped sums) and which makes reading difficult. Certain notations, such as $t^*$, complicate reading in my opinion (because when $t$ and $t^*$ behave in opposite ways, you have to do some mental gymnastics to follow the reasoning). Finally, some notations are used differently: for example, $\\textbf{q}$ is sometimes a vector (Definition 3.1) and sometimes a matrix (Algorithm 1), or $Z_t$ is sometimes a function in $\\mu$ and sometimes not.\nSimilarly, the graphs are not intuitive for me. For example, why did the authors choose a color scale when there are only 7 values $t$ presented?\nA last small weakness of this paper for me is the fact that the authors present \"just\" an extension of the ADABOOST algorithm but I do not have the impression to know if it will help or not to improve things (it is a point of discussion). I would have liked them to highlight at least one case where the algorithm actually worked around a weakness in ADABOOST.\n\nQUESTIONS:\nI think the paper would be easier to read if the number of notations were reduced (for example, $t^*$ is not compulsory in my opinion, as it's often $1/t^*$ that's used). I think it should be made clear whether $Z_t$ is a function or not. In the same vein, I think some equations lack rigor. For example, $\\log_t$ and $\\exp_t$ are introduced at the same time, even though their spaces of definition are different. On line 106, $\\mu$ is not introduced either.\nI didn't understand where clamped sum was used. Given the limited space available, I think there's a point in putting a paragraph on it. Could the authors explain?\nWhat does the $\\|\\|\\cdot\\|\\|_{1/t^*}$ norm represent? Is it just the Hölderian norm with $\\alpha=1/t^*$? Or is it a new norm based on the new dissimilarity measure introduced?\nI thought I read a few typos:\n* Page 2, line 90: [11, 6, 17] is not in increasing order.\n* Page 7, line 246: Table A1 or Section III1 instead of Section A1?\n* Page 8, Table 1: isn't it a figure?\n\nLIMITATIONS:\nI didn't see any limitations that weren't addressed by the authors. I appreciated their honesty.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an estension of ADABoost algorithm for binary classification, called t-ADABoost, by modifying the weight optimisation under simplex constraint formulation of the original algorithm. The generalised formulation involves optimisation of modified Bregman divergence between new and old weights under a so called co-simplex constraint (i.e. a sum constraint on the power of the weights). The paper establishes that these new optimisation problems amount to a one parameter optimisation problem in a way similar to ADABoost, then that the resulting construction of a strong learner will, under a generalised assumption, will have an empirical error which decreases exponentially quick.\n\nThe paper complements their results by experiments comparing the behavior of the t-ADABoost algorithm for decision tree using different t values. The results show that depending on the data and presence of noise, the relationship between t value and test error can change.\n\nSTRENGTHS:\nThe paper is well introduced and describes an original extension of ADABoost algorithm, by incorporating the notion of tempered exponential measures which was found to improve robustness in clustering.\n\nThe authors give sufficient evidence that the extended algorithm is able to decrease the empirical risk as at least the same rate as the original ADABoost algorithm  (Theorem 2 and discussion following) and that the use of an oracle t-ADABoost algorithm can reduce the test error compared to the initial ADABoost algorithm (Table 1). This, as the authors note, imply that while work remains to be done in order to learn the optimal t for each new dataset, this generalisation has its use cases.\n\nWEAKNESSES:\nPart of section 5 and Algorithm 1 are confusing. Notably, in the description of Algorithm 1, the beginning of Step 2.2 starts with what seems to be a typo (I believe $\\mu_{ji} = y_i h_j(x_i)$ should read $ u_{ji} = y_i h_j(x_i)$, then neither the coefficients $\\nu_j$ and $\\alpha_j$ are defined until Theorem 2. The wording \"choose leveraging coefficient\" seems to imply that the choice can be arbitrary, while, considering that Algorithm 1 is described just after Theorem 1., one could infer that the coefficient $\\nu_j$ is the minimizer of equation (5). \nIn equation 11, the quantity $m^{1-t^*}$ is not introduced anywhere.  The next mention is in line 180, where it is stated that it can be discarded in the unclamped case, but that it does play a role in equation 12, where the quantity does not appear (while $1 + m^\\dag {q_j^\\dag}^{2-t}}$ appears and could play the \"dampening part\" mentionned later on).\n\nParts of section 6 could be clarified. There is some confusion between the notations $L$ and $\\underline{L}$ (line 200, $\\mathbb{E}_\\lambda[\\underline{L}(p_\\lambda)$, but $\\mathbb{E}_\\lambda[L(p_\\lambda)]$ line 219, which in itself does not make sense as $L$ takes 2 arguments. Moreover in equation (16), from equation 200 implies that the DT is trained from the Bayes risk, and as such, reverse engineering should start by computing the Bayes risk rather than CPE loss).\n\nQUESTIONS:\nThe algorithms presented here works for binary classification, as is the case for the initial ADABoost algorithm. ADABoost has been extended to multiclass settings. Could such an extension also work for t-ADABoost?\n\nThe interpretation of theorem 2 relies (line 161) on the assumption that $\\lvert \\rho_j\\rvert \\geq \\gamma$. If $m_j^\\dag > 0$, the value of $\\rho_j$ depends on $t$ and therefore the assumption above could be false for certain $t$. Is there any insight on how this impacts the exponential decay? Moreover, even in the case where $m_j^\\dag = 0$, considering that $\\rho_j = \\frac{1}{R_j}\\sum_{i\\in[m]} q_{ji}, y_i h_j(x_i)$  and that $q_{ji}$ no longer need sum to $1$ but rather should belong to a co-simplex, the assumption that $\\lvert\\rho\\rvert >\\gamma$ does not have the same implication depending on the value on $t$. Could you comment on whether this is a stronger requirement in the case where $t<1$.\n\nAre there any overfitting issues related to t-ADABoost, notably when $t \\ll 1$ or when $t > 1$? Could these be investigated by allowing the number of trees grown to be larger than 20 in the experiments?\n\nIn section 6, it is mentionned that reference 10 note that Matusita's loss implies a near optimal boosting rate while the empriical risk gives the worst possible guarantee. Could you specify where these two results are stated in the reference?\n\nLIMITATIONS:\nThe current form of the algorithm seems to be usable only the case of binary classifiation, while the original ADABoost has been extended to multiclass settings. It is unclear whether this restriction can be lifted efficiently.\n\nThe authors clearly mention that the current work should be complemented by further insights on the selection of $t$, which is coherent with their experiment results (where the optimal $t$ value depends on the dataset and noise).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-31"}