SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Extension to margins A key property of boosting algorithms like ADABOOST is to be able to boost not just the empirical risk but more generally margins [19, 30], where a margin integrates both the accuracy of label prediction but also the confidence in prediction (say  $|H|$ ). We generalize the margin notion of [19] to the tempered arithmetic and let  $\nu_{t}((\boldsymbol{x},y),H)\doteq \tanh_{t}(yH(\boldsymbol{x}) / 2)$  denote the margin of  $H$  on example  $(\boldsymbol{x},y)$ , where  $\tanh_t(z)\doteq (1 - \exp_t(-2z)) / (1 + \exp_t(-2z))(\in [-1,1])$  is the tempered hyperbolic tangent. The objective of minimizing the empirical risk is generalized to minimizing the margin risk,  $F_{t,\theta}(H,\mathcal{S})\doteq (1 / m)\cdot \sum_i[[\nu_t((\boldsymbol{x}_i,y_i),H)\leqslant \theta]]$ , where  $\theta \in (-1,1)$ . Guarantees on the empirical risk are guarantees on the margin risk for  $\theta = 0$  only. In just a few steps, we can generalize Theorem 2 to all  $\theta \in (-1,1)$ . For space reason, we state the core part of the generalization, from which extending it to a generalization of Theorem 2 is simple.
Theorem 3. For any  $\theta \in (-1,1)$  and  $t\in [0,1]$ , the guarantee of algorithm  $t$ -ADABOOST in Theorem 2 extends to the margin risk, with notations from Theorem 2, via:
$$
F _ {t, \theta} (H, \mathcal {S}) \leqslant \left(\frac {1 + \theta}{1 - \theta}\right) ^ {2 - t} \prod_ {j = 1} ^ {J} Z _ {t j} ^ {2 - t}. \tag {13}
$$
(Proof in APP., Section II.3) At a high level,  $t$ -ADABOOST brings similar margin maximization properties as ADABOOST. Digging a bit in (13) reveals an interesting phenomenon for  $t \neq 1$  on how margins are optimized compared to  $t = 1$ . Pick  $\theta < 0$ , so we focus on those examples for which the classifier  $H$  has high confidence in its wrong classification. In this case, factor  $((1 + \theta) / (1 - \theta))^{2 - t}$  is increasing as a function of  $t \in [0,1]$  (and this pattern is reversed for  $\theta > 0$ ). In words, the smaller we pick  $t \in [0,1]$  and the better is the bound in (13), suggesting increased "focus" of  $t$ -ADABOOST on increasing the margins of examples with low negative margin (e.g. the most difficult ones) compared to the case  $t = 1$ .
The tempered exponential loss In the same way as ADABOOST introduced the now famous exponential loss, (12) recommends to minimize the normalization coefficient, following (7),
$$
Z _ {t j} ^ {2 - t} (\mu) = \left\| \exp_ {t} \left(\log_ {t} \boldsymbol {q} _ {j} - \mu \cdot \boldsymbol {u} _ {j}\right) \right\| _ {1 / t ^ {*}} ^ {1 / t ^ {*}} \quad \left(\text {w i t h} u _ {j i} \doteq y _ {i} h _ {j} \left(\boldsymbol {x} _ {i}\right)\right). \tag {14}
$$
We cannot easily unravel the normalization coefficient to make appear an equivalent generalization of the exponential loss, unless we make several assumptions, one being  $\max_{i} |h_{j}(\pmb{x}_{i})|$  is small enough for any  $j \in [J]$ . In this case, we end up with an equivalent criterion to minimize which looks like
$$
F _ {t} (H, \mathcal {S}) = \frac {1}{m} \cdot \sum_ {i} \exp_ {t} ^ {2 - t} (- y _ {i} H (\boldsymbol {x} _ {i})) \tag {15}
$$
where we have absorbed in  $H$  the factor  $m^{1 - t^*}$  appearing in the  $\exp_t$  (scaling  $H$  by a positive value does not change its empirical risk). This defines a generalization of the exponential loss which we call the tempered exponential loss. Notice that one can choose to minimize  $F_t(H, \mathcal{S})$  disregarding any constraint on  $|H|$ .
6 A broad family of boosting-compliant proper losses for decision trees
Losses for class probability estimation When it comes to tabular data, it has long been known that some of the best models to linearly combine with boosting are decision trees (DT, [9]). Decision trees, like other domain-partitioning classifiers, are not trained by minimizing a surrogate loss defined over real-valued predictions, but defined over class probability estimation (CPE, [26]), those estimators being posterior estimation computed at the leaves. Let us introduce a few definitions for those. A CPE loss  $\ell : \{-1, 1\} \times [0, 1] \to \mathbb{R}$  is
$$
\ell (y, u) \quad \doteq \quad [   [ y = 1 ]   ] \cdot \ell_ {1} (u) + [   [ y = - 1 ]   ] \cdot \ell_ {- 1} (u). \tag {16}
$$
Functions  $\ell_1, \ell_{-1}$  are called partial losses. The pointwise conditional risk of local guess  $u \in [0,1]$  with respect to a ground truth  $v \in [0,1]$  is:
$$
L (u, v) \quad \doteq \quad v \cdot \ell_ {1} (u) + (1 - v) \cdot \ell_ {- 1} (u). \tag {17}
$$
A loss is proper iff for any ground truth  $v \in [0,1]$ ,  $L(v,v) = \inf_u L(u,v)$ , and strictly proper iff  $u = v$  is the sole minimizer [26]. The (pointwise) Bayes risk is  $\underline{L}(v) \doteq \inf_u L(u,v)$ . The log/cross-entropy-loss, square-loss, Matusita loss are examples of CPE losses. One then trains a DT minimizing the expectation of this loss over leaves' posteriors,  $\mathbb{E}_{\lambda}[\underline{L}(p_{\lambda})]$ ,  $p_{\lambda}$  being the local proportion of positive examples at leaf  $\lambda$  or equivalently, the local posterior.
Deriving CPE losses from (ada)boosting Recently, it was shown how to derive in a general way a CPE loss to train a DT from the minimization of a surrogate loss with a boosting algorithm [16]. In our case, the surrogate would be  $\tilde{Z}_{tj}$  (14) and the boosting algorithm,  $t$ -ADABOOST. The principle is simple and fits in four steps: (i) show that a DT can equivalently perform simple linear classifications, (ii) use a weak learner that designs splits and the boosting algorithm to fit the leveraging coefficient and compute those in closed form, (iii) simplify the expression of the loss using those, (iv) show that the expression simplified is, in fact, a CPE loss. To get (i), we remark that a DT contains a tree (graph). One can associate to each node a real value. To classify an observation, we sum all reals from the root to a leaf and decide on the class based on the sign of the prediction, just like for any real-valued predictor. Suppose we are at a leaf. What kind of weak hypotheses can create splits "in disguise"? Those can be of the form
$$
h _ {j} (\boldsymbol {x}) \quad \doteq \quad \llbracket x _ {k} \geqslant a _ {j} \rrbracket \cdot b _ {j}, \quad a _ {j}, b _ {j} \in \mathbb {R},
$$
where the observation variable  $x_{k}$  is assumed real valued for simplicity and the test  $[[x_{k}\geqslant a_{j}]]$  splits the leaf's domain in two non-empty subsets. This creates half of the split.  $\overline{h}_j(\pmb {x})\doteq [[x_k < a_j]]\cdot -b_j$  creates the other half of the split. Remarkably  $h_j$  satisfies the weak learning assumption iff  $\overline{h}_j$  does [16]. So we get the split design part of (ii). We compute the leveraging coefficients at the new leaves from the surrogate's minimization / boosting algorithm, end up with new real predictions at the new leaves (instead of the original  $b_{j}, - b_{j}$ ), push those predictions in the surrogate loss for (iii), simplify it and, quite remarkably end up with a loss of the form  $\mathbb{E}_{\lambda}[\mathrm{L}(p_{\lambda})]$ , where L turns out to be the pointwise Bayes risk  $\underline{L}$  of a proper loss [16] (notation from [26]).
In the case of [16], it is, in fact, granted that we end up with such a "nice" CPE loss because of the choice of the surrogates at the start. In our case, however, nothing grants this a priori if we start from the tempered exponential loss  $\tilde{Z}_{tj}$  (14) so it is legitimate to wonder whether such a chain of derivations (summarized) can happen to reverse engineer an interesting CPE loss:
$$
\tilde {Z} _ {t j} \stackrel {?} {\rightarrow} L \stackrel {?} {\rightarrow} \underline {{L}} ^ {(t)} \stackrel {?} {\rightarrow} \ell_ {1} ^ {(t)}; \ell_ {- 1} ^ {(t)} \quad (\text {p r o p e r ? s t r i c t l y p r o p e r ? f o r w h i c h t s ? , . . .}) \tag {18}
$$
When such a complete derivation happens until the partial losses  $\ell_1; \ell_{-1}$  and their properties, we shall write that minimizing  $\tilde{Z}_{tj}$  elicits the corresponding loss and partial losses.
Theorem 4. Minimizing  $\tilde{Z}_{t,j}$  elicits the CPE loss we define as the tempered loss, with partial losses:
$$
\ell_ {1} ^ {(t)} (u) \doteq \left(\frac {1 - u}{M _ {1 - t} (u , 1 - u)}\right) ^ {2 - t}, \quad \ell_ {- 1} ^ {(t)} (u) \doteq \ell_ {1} ^ {(t)} (1 - u), \quad (t \in [ - \infty , 2 ]). \tag {19}
$$
The tempered loss is symmetric, differentiable, strictly proper for  $t \in (-\infty, 2)$  and proper for  $t = -\infty$ .
Table 1: Experiments on  $t$ -ADABOOST comparing with ADABOOST ( $t = 1$ , bullets) on three domains (rows), displaying from left to right the estimated true error of non-clipped and clipped models, and the min and max codensity weights. These domains were chosen to give an example of three different situations: small values of  $t$  perform well (abalone), the best performance is achieved by the largest  $t$  (e.g. ADABOOST, qsar), and the worst performance is achieved by the largest  $t$  (adult). Topmost row is without noise ( $\eta = 0$ ) while the others are with  $10\%$  training noise;  $t$  scale displayed with varying color and width (colormap indicated on each plot). Averages shown for readability: see Table 2 for exhaustive statistical tests.
Differentiability means the partial losses are differentiable, and symmetry follows from the relationship between partial losses [20] (the proof, in APP., Section II.4, derives the infinite case,  $\ell_1^{(-\infty)}(u) = 2 \cdot [[u \leqslant 1/2]]$ ) Let us explicit the Bayes risk of the tempered loss and a key property.
Lemma 1. The Bayes risk of the tempered loss is  $(M_q$  defined in Theorem 2):
$$
\underline {{L}} ^ {(t)} (u) = \frac {2 u (1 - u)}{M _ {1 - t} (u , 1 - u)}, \tag {20}
$$
and it satisfies  $\forall u\in [0,1],\forall z\in [2\cdot \min \{u,1 - u\} ,1],\exists t\in [-\infty ,2]$  such that  $\underline{L}^{(t)}(u) = z$
Lemma 1, whose proof is trivial, allows to show a key boosting result:  $t = 1$  retrieves Matusita's loss, for which a near-optimal boosting rate is known [11] while  $t = -\infty$  retrieves the empirical risk, which yields the worst possible guarantee [11]. In between, we have, for example, CART's Gini criterion for  $t = 0$ , which yields an intermediate boosting guarantee. Continuity with respect to  $t$  of the Bayes risks in between the empirical risk and Matusita's loss means the boosting ranges of the tempered loss cover the full known spectrum of boosting rates for  $t \in [-\infty, 1]$ . We know of no (differentiable and) proper CPE loss with such coverage. Note that (i) this is a non-constructive result as we do not associate a specific  $t$  for a specific rate, and (ii) the state-of-the-art boosting rates for DT induction do not seem to cover the case  $t \in (1, 2)$ , thus left as an open question.
We have performed experiments on a testbed of 10 UCI domains, whose details are given in APP. (Section A3). Experiments were carried out using a 10-fold stratified cross-validation procedure.
To compare  $t$ -ADABOOST with ADABOOST, we ran  $t$ -ADABOOST with a first range of values of  $t \in \{0.0, 0.2, 0.4, 0.6, 0.8, 0.9\}$ . This is in the range of values covered by our convergence result for linear separators in Theorem 2. Our results on decision tree induction cover a much wider range, in particular for  $t \in (1, 2)$ . To assess whether this can be an interesting range to study, we added  $t = 1.1$  to the set of tested  $t$  values. When  $t > 1$ , some extra care is to be put into computations because the weight update becomes unbounded, in a way that is worse than ADABOOST. Indeed, as can be seen from (8), if  $\mu_j y_i h_j(\boldsymbol{x}_i) \leqslant -1 / (t - 1)$  (the example is badly classified by the current weak hypothesis, assuming wlog  $\mu_j > 0$ ), the weight becomes infinity before renormalization. In our experiments, picking a value of  $t$  close to 2 clearly shows this problem, so to be able to still explore whether  $t > 1$  can be useful, we picked a value close to 1, namely  $t = 1.1$ , and checked that in our experiments this produced no such numerical issue. We also considered training clipped and not clipped models.
All boosting models were trained for a number of  $J = 20$  decision trees (The appendix provides experiments on training bigger sets). Each decision tree is induced using the tempered loss with the corresponding value of  $t$  (see Theorem 4) following the classical top-down template, which consists in growing the current heaviest leaf in the tree and picking the best split for the leaf chosen. We implemented  $t$ -ADABOOST exactly as in Section 5, including computing leveraging coefficients as suggested. Thus, we do not scale models. More details are provided in the appendix. In our experiments, we also included experiments on a phenomenon highlighted more than a decade ago [15] and fine-tuned more recently [16], the fact that a convex booster's model is the weakest link when it has to deal with noise in training data. This is an important task because while the tempered exponential loss is convex, it does not fit into the blueprint loss of [15, Definition 1] because it is not  $C^1$  if  $t \neq 1$ . One might thus wonder how  $t$ -ADABOOST behaves when training data is affected by noise. Letting  $\eta$  denote the proportion of noisy data in the training sample, we tried  $\eta \in \{0.0, 0.1\}$  (The appendix provides experiments on more noise levels). We follow the noise model of [15] and thus independently flip the true label of each example with probability  $\eta$ .
For each run, we recorded the average test error and the average maximum and minimum co-density weight. Table 1 presents a subset of the results obtained on three domains. Table 2 presents a more synthetic view in terms of statistical significance of the results for  $t \neq 1$  vs.  $t = 1$  (ADABOOST). The table reports only results for  $t \geqslant 0.6$  for synthesis purposes. Values  $t < 0.6$  performed on average slightly worse than the others but on some domains, as the example of abalone suggests in Table 2 (the plots include all values of  $t$  tested in [0, 1.1]), we clearly got above-par results for such small values of  $t$ , both in terms of final test error but also fast early convergence to low test error. This comment can be generalized to all values of  $t$ .
The weights reveal interesting patterns as well. First, perhaps surprisingly, we never encountered the case where weights switch off, regardless of the value of  $t$ . The average minimum weight curves of Table 1 generalize to all our tests (see the appendix). This does not rule out the fact that boosting for a much longer number of iterations might lead to weights switching off/on, but the fact that this does not happen at least early during boosting probably comes from the fact that the leveraging coefficients for weights  $(\mu_{\cdot})$  are bounded. Furthermore, their maximal absolute value is all the smaller as  $t$  decreases to 0. Second, there is a pattern that also repeats on the maximum weights, not on all domains but on a large majority of them and can be seen in abalone and adult in Table 1: the maximum weight of ADABOOST tends to increase much more rapidly compared to  $t$ -ADABOOST with  $t < 1$ . In the latter case, we almost systematically observe that the maximum weight tends to be upperbounded, which is not the case for ADABOOST (the growth of the maximal weight looks almost linear). Having bounded weights could be of help to handle numerical issues of (ada)boosting [14].
Our experiments certainly confirm the boosting nature of  $t$ -ADABOOST if we compare its convergence to that of ADABOOST: more often than not, it is in fact comparable to that of ADABOOST. While this applies broadly for  $t \geqslant 0.6$ , we observed examples where much smaller values (even  $t = 0.0$ ) could yield such fast convergence. Importantly, this applies to clipped models as well and it is important to notice because it means attaining a low "boosted" error does not come at the price of learning models with large range. This is an interesting property: for  $t = 0.0$ , we would be guaranteed that the computation of the clipped prediction is always in  $[-1, 1]$ . Generalizing our comment on small values of  $t$  above, we observed that an efficient tuning algorithm for  $t$  could be able to get very substantial leverage over ADABOOST. Table 2 was crafted for a standard limit  $p$ -val of 0.1 and "blurs" the best results that can be obtained. On several domains (winered, abalone, eeg, creditcard, adult), applicable  $p$ -values for which we would conclude that some  $t \neq 1$  performs better than  $t = 1$  drop in
ηt[clipped]	0.0	0.1
0.6	0.8	0.9	1.1	0.6	0.8	0.9	1.1
0	1	0	1	0	1	0	1	0	1	0	1	0	1
#better	2	3	1	2	1	3	1	1	1	2	2	1
#equivalent	5	5	6	6	7	7	6	7	4	8	8	7	8	9
#worse	3	2	3	2	2	4	3	5	1	1	1	2
Table 2: Outcomes of student paired  $t$ -tests over 10 UCI domains, with training noise  $\eta \in \{0.0, 0.1\}$ , for  $t \in \{0.6, 0.8, 0.9, 1.0, 1.1\}$  and with / without clipped models. For each triple  $(\eta, t, [[\text{clipped}]])$ , we give the number of domains for which the corresponding setting of  $t$ -ADABOOST is statistically better than ADABOOST(#better), the number for which it is statistically worse (#worse) and the number for which we cannot reject the assumption of identical performances. Threshold  $p$ -val = 0.1.
between  $7E - 4$  and 0.05. Unsurprisingly, ADABOOST also manages to beat significantly alternative values of  $t$  in several cases. Our experiments with training noise  $(\eta = 0.1)$  go in the same direction. Looking at Table 1, one could eventually be tempted to conclude that  $t$  slightly smaller than 1.0 may be a better choice than adaboosting  $(t = 1)$ , as suggested by our results for  $t = 0.9$ , but we do not think this produces a general "rule-of-thumb". There is also no apparent "noise-dependent" pattern that would obviously separate the cases  $t < 1$  from  $t = 1$  even when the tempered exponential loss does not fit to [15]'s theory. Finally, looking at the results for  $t > 1$  also yields the same basic conclusions, which suggests that boosting can be attainable outside the range covered by our theory (in particular Theorem 2).
All this brings us to the experimental conclusion that the question does not reside on opposing the case  $t \neq 1$  to the case  $t = 1$ . Rather, our experiments suggest – pretty much like our theory does – that the actual question resides in how to efficiently learn  $t$  on a domain-dependent basis. Our experiments indeed demonstrate that substantial gains could be obtained, to handle overfitting or noise.
8 Discussion, limitations and conclusion
ADABOOST is one of the original and simplest Boosting algorithms. In this paper we generalized ADABOOST to maintaining a tempered measure over the examples by minimizing a tempered relative entropy. We kept the setup as simple as possible and therefore focused on generalizing ADABOOST. However more advanced boosting algorithms have been designed based on relative entropy minimization subject to linear constraints. There are versions that constrain the edges of all past hypotheses to be zero [36]. Also, when the maximum margin of the game is larger than zero, then ADABOOST cycles over non-optimal solutions [27]. Later Boosting algorithms provably optimize the margin of the solution by adjusting the constraint value on the dual edge away from zero (see e.g. [24]). Finally, the ELRP-Boost algorithm optimizes a trade off between relative entropy and the edge [35]. We conjecture that all of these orthogonal direction have generalizations to the tempered case as well and are worth exploring.
These are theoretical directions that, if successful, would contribute to bring more tools to the design of rigorous boosting algorithms. This is important because boosting suffers several impediments, not all of which we have mentioned: for example, to get statistical consistency for ADABOOST, it is known that early stopping is mandatory [5]. More generally, non-Lipschitz losses like the exponential loss seem to be harder to handle compared to Lipschitz losses [33] (but they yield in general better convergence rates). The validity of the weak learning assumption of boosting can also be discussed, in particular regarding the negative result of [15] which advocates, beyond just better (ada)boosting, for boosting for more classes of models / architectures [16]. Alongside this direction, we feel that our experiments on noise handling give a preliminary account of the fact that there is no "one  $t$  fits all" case, but a much more in depth analysis is required to elicit / tune a "good"  $t$ . This is a crucial issue for noise handling [16], but as we explain in Section 7, this could bring benefits in much wider contexts as well.
The authors thank the reviewers for numerous comments that helped improving the paper's content.
[1] N. Alon, A. Gonen, E. Hazan, and S. Moran. Boosting simple learners. In STOC'21, 2021.
[2] S.-I. Amari. Information Geometry and Its Applications. Springer-Verlag, Berlin, 2016.
[3] E. Amid, F. Nielsen, R. Nock, and M.-K. Warmuth. Optimal transport with tempered exponential measures. CoRR, abs/2309.04015, 2023.
[4] E. Amid, R. Nock, and M.-K. Warmuth. Clustering above exponential families with tempered exponential measures. In  $26^{th}$  AISTATS, 2023.
[5] P. Bartlett and M. Traskin. Adaboost is consistent. In NIPS*19, 2006.
[6] L. Breiman, J. H. Freidman, R. A. Olshen, and C. J. Stone. Classification and regression trees. Wadsworth, 1984.
[7] M. Collins, R. Schapire, and Y. Singer. Logistic regression, adaboost and Bregman distances. In Proc. of the  $13^{th}$  International Conference on Computational Learning Theory, pages 158-169, 2000.
[8] Y. Freund and R. E. Schapire. A Decision-Theoretic generalization of on-line learning and an application to Boosting. J. Comp. Syst. Sc., 55:119-139, 1997.
[9] J. Friedman, T. Hastie, and R. Tibshirani. Additive Logistic Regression: a Statistical View of Boosting. Ann. of Stat., 28:337-374, 2000.
[10] M.J. Kearns. Thoughts on hypothesis boosting, 1988. ML class project.
[11] M.J. Kearns and Y. Mansour. On the boosting ability of top-down decision tree learning algorithms. In Proc. of the  $28^{th}$  ACM STOC, pages 459-468, 1996.
[12] J. Kivinen and M.-K. Warmuth. Boosting as entropy projection. In  $COLT'99$ , pages 134-144, 1999.
[13] D.-E. Knuth. Two notes on notation. The American Mathematical Monthly, 99(5):403-422, 1992.
[14] R. Kohavi. Improving accuracy by voting classification algorithms: Boosting, bagging, and variants. In Workshop on Computation-Intensive Machine Learning Techniques, 1998.
[15] P.-M. Long and R.-A. Servedio. Random classification noise defeats all convex potential boosters. MLJ, 78(3):287-304, 2010.
[16] Y. Mansour, R. Nock, and R.-C. Williamson. Random classification noise does not defeat all convex potential boosters irrespective of model choice. In  $40^{th}$  ICML, 2023.
[17] J. Naudts. Generalized thermostatistics. Springer, 2011.
[18] L. Nivanen, A. Le Méhauté, and Q.-A. Wang. Generalized algebra within a nonextensive statistics. Reports on Mathematical Physics, 52:437-444, 2003.
[19] R. Nock and F. Nielsen. A Real Generalization of discrete AdaBoost. Artificial Intelligence, 171:25-41, 2007.
[20] R. Nock and F. Nielsen. On the efficient minimization of classification-calibrated surrogates. In NIPS*21, pages 1201-1208, 2008.
[21] R. Nock and F. Nielsen. The phylogenetic tree of Boosting has a bushy carriage but a single trunk. PNAS, 117:8692-8693, 2020.
[22] R. Nock and R.-C. Williamson. Lossless or quantized boosting with integer arithmetic. In  $36^{th}$  ICML, pages 4829-4838, 2019.
[23] J. R. Quinlan. C4.5: programs for machine learning. Morgan Kaufmann, 1993.
[24] G. Ratsch and M.-K. Warmuth. Efficient margin maximizing with boosting. JMLR, pages 2131-2152, December 2005.
[25] M.-D. Reid and R.-C. Williamson. Composite binary losses. JMLR, 11:2387-2422, 2010.