SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Richard Nock
richardnock@google.com
Ehsan Amid
eamid@google.com
Manfred K. Warmuth
manfred@google.com
One of the most popular ML algorithms, ADABOOST, can be derived from the dual of a relative entropy minimization problem subject to the fact that the positive weights on the examples sum to one. Essentially, harder examples receive higher probabilities. We generalize this setup to the recently introduced tempered exponential measures (TEMs) where normalization is enforced on a specific power of the measure and not the measure itself. TEMs are indexed by a parameter  $t$  and generalize exponential families  $(t = 1)$ . Our algorithm,  $t$ -ADABOOST, recovers ADABOOST as a special case  $(t = 1)$ . We show that  $t$ -ADABOOST retains ADABOOST's celebrated exponential convergence rate on margins when  $t \in [0,1)$  while allowing a slight improvement of the rate's hidden constant compared to  $t = 1$ .  $t$ -ADABOOST partially computes on a generalization of classical arithmetic over the reals and brings notable properties like guaranteed bounded leveraging coefficients for  $t \in [0,1)$ . From the loss that  $t$ -ADABOOST minimizes (a generalization of the exponential loss), we show how to derive a new family of tempered losses for the induction of domain-partitioning classifiers like decision trees. Crucially, strict properness is ensured for all while their boosting rates span the full known spectrum of boosting rates. Experiments using  $t$ -ADABOOST+trees display that significant leverage can be achieved by tuning  $t$ .
ADABOOST is one of the most popular ML algorithms [8, 30]. It efficiently aggregates weak hypotheses into a highly accurate linear combination [10]. The common motivations of boosting algorithms focus on choosing good linear weights (the leveraging coefficients) for combining the weak hypotheses. A dual view of boosting highlights the dual parameters, which are the weights on the examples. These weights define a distribution, and ADABOOST can be viewed as minimizing a relative entropy to the last distribution subject to a linear constraint introduced by the current hypothesis [12]. For this reason (more in § 2), ADABOOST's weights define an exponential family.
In this paper, we go beyond weighing the examples with a discrete exponential family distribution, relaxing the constraint that the total mass be unit but instead requiring it for the measure's  $1 / (2 - t)$ ’th power, where  $t$  is a temperature parameter. Such measures, called tempered exponential measures (TEMs), have been recently introduced [4]. Here we apply the discrete version of these TEMs for deriving a novel boosting algorithm called  $t$ -ADABOOST. Again the measures are solutions to a relative entropy minimization problem, but the relative entropy is built from Tsallis entropy and "tempered" by a parameter  $t$ . As  $t \to 1$  TEMs become standard exponential family distributions and our new algorithm merges into ADABOOST. As much as ADABOOST minimizes the exponential loss,  $t$ -ADABOOST minimizes a generalization of this loss we denote as the tempered exponential loss.
TEMs were introduced in the context of clustering, where they were shown to improve the robustness to outliers of clustering's population minimizers [4]. They have also been shown to bring low-level sparsity features to optimal transport [3]. Boosting is a high-precision machinery: ADABOOST is known to achieve near-optimal boosting rates under the weak learning assumption [1], but it has
long been known that numerical issues can derail it, in particular, because of the unbounded weight update rule [14]. So the question of what the TEM setting can bring for boosting is of primordial importance. As we show,  $t$ -ADABOOST can suffer no rate setback as boosting's exponential rate of convergence on margins can be preserved for all  $t \in [0,1)$ . Several interesting features emerge: the weight update becomes bounded, margin optimization can be tuned with  $t$  to focus on examples with very low margin and besides linear separators, the algorithm can also learn progressively clipped models<sup>1</sup>. Finally, the weight update makes appear a new regime whereby weights can "switch off and on": an example's weight can become zero if too well classified by the current linear separator, and later on revert to non-zero if badly classified by a next iterate.  $t$ -ADABOOST makes use of a generalization of classical arithmetic over the reals introduced decades ago [18].
Boosting algorithms for linear models like ADABOOST bring more than just learning good linear separators: it is known that (ada)boosting linear models can be used to emulate the training of decision trees (DT) [16], which are models known to lead to some of the best of-the-shelf classifiers when linearly combined [9]. Unsurprisingly, the algorithm obtained emulates the classical top-down induction of a tree found in major packages like CART [6] and C4.5 [23]. The loss equivalently minimized, which is, e.g., Matusita's loss for ADABOOST [30, Section 4.1], is a lot more consequential. Contrary to losses for real-valued classification, losses to train DTs rely on the estimates of the posterior learned by the model; they are usually called losses for Class Probability Estimation (CPE [25]). The CPE loss is crucial to elicit because (i) it is possible to check whether it is "good" from the standpoint of properness (Bayes rule is optimal for the loss [28]), and (ii) it conditions boosting rates, only a handful of them being known, for the most popular CPE losses [11, 22, 31].
In this paper, we show that this emulation scheme on  $t$ -ADABOOST provides a new family of CPE losses with remarkable constancy with respect to properness: losses are strictly proper (Bayes rule is the sole optimum) for any  $t \in (-\infty, 2)$  and proper for  $t = -\infty$ . Furthermore, over the range  $t \in [-\infty, 1]$ , the range of boosting rates spans the full spectrum of known boosting rates [11].
We provide experiments displaying the boosting ability of  $t$ -ADABOOST over a range of  $t$  encompassing potentially more than the set of values covered by our theory, and highlight the potential of using  $t$  as a parameter for efficient tuning the loss [25, Section 8]. Due to a lack of space, proofs are relegated to the appendix (APP.). A primer on TEMs is also given in APP., Section I.
Boosting refers to the ability of an algorithm to combine the outputs of moderately accurate, "weak" hypotheses into a highly accurate, "strong" ensemble. Originally, boosting was introduced in the context of Valiant's PAC learning model as a way to circumvent the then-existing amount of related negative results [10, 34]. After the first formal proof that boosting is indeed achievable [29], ADABOOST became the first practical and proof-checked boosting algorithm [8, 30]. Boosting was thus born in a machine learning context, but later on, it also emerged in statistics as a way to learn from class residuals computed using the gradient of the loss [9, 21], resulting this time in a flurry of computationally efficient algorithms, still called boosting algorithms, but for which the connection with the original weak/strong learning framework is in general not known.
Our paper draws its boosting connections with ADABOOST's formal lineage. ADABOOST has spurred a long line of work alongside different directions, including statistical consistency [5], noise handling [15, 16], low-resource optimization [22], etc. The starting point of our work is a fascinating result in convex optimization establishing a duality between the algorithm and its memory of past iteration's performances, a probability distribution of so-called weights over examples [12]. From this standpoint, ADABOOST solves the dual of the optimization of a Bregman divergence (constructed from the negative Shannon entropy as the generator) between weights subject to zero correlation with the last weak classifier's performance. As a consequence, weights define an exponential family. Indeed, whenever a relative entropy is minimized subject to linear constraints, then the solution is a member of an exponential family of distributions (see e.g. [2, Section 2.8.1] for an axiomatization of exponential families). ADABOOST's distribution on the examples is a member of a discrete exponential family where the training examples are the finite support of the distribution, sufficient statistics are defined from the weak learners, and the leveraging coefficients are the natural parameters. In summary, there
Traditionally, clipping a sum is done after it has been fully computed. In our case, it is clipped after each new summand is added.
is an intimate relationship between boosting à-la-ADABOOST, exponential families, and Bregman divergences [7, 12, 20] and our work "elevates" these methods above exponential families.
3 Definitions
We define the  $t$ -logarithm and  $t$ -exponential [17, Chapter 7],
$$
\log_ {t} (z) \doteq \frac {1}{1 - t} \cdot (z ^ {1 - t} - 1), \exp_ {t} (z) \doteq [ 1 + (1 - t) z ] _ {+} ^ {1 / (1 - t)} ([ z ] _ {+} \doteq \max \{0, z \}), (1)
$$
where the case  $t = 1$  is supposed to be the extension by continuity to the log and exp functions, respectively. To preserve the concavity of  $\log_t$  and the convexity of  $\exp_t$ , we need  $t \geqslant 0$ . In the general case, we also note the asymmetry of the composition: while  $\exp_t \log_t(z) = z, \forall t \in \mathbb{R}$ , we have  $\log_t \exp_t(z) = z$  for  $t = 1 (\forall z \in \mathbb{R})$ , but
$$
\log_ {t} \exp_ {t} (z) = \max \left\{- \frac {1}{1 - t}, z \right\} (t <   1) \mathrm {a n d} \log_ {t} \exp_ {t} (z) = \min \left\{\frac {1}{t - 1}, z \right\} (t > 1).
$$
Comparisons between vectors and real-valued functions written on vectors are assumed componentwise. We assume  $t \neq 2$  and define notation  $t^* \doteq 1 / (2 - t)$ . We now define the key set in which we model our weights (boldfaces denote vector notation).
Definition 3.1. The co-simplex of  $\mathbb{R}^m$ ,  $\tilde{\Delta}_m$  is defined as  $\tilde{\Delta}_m \doteq \{q \in \mathbb{R}^m : q \geqslant 0 \wedge 1^\top q^{1/t^*} = 1\}$ .
The letters  $\mathbf{q}$  will be used to denote TEMs in  $\tilde{\Delta}_m$  while  $\pmb{p}$  denote the co-density  $q^{\frac{1}{t^{*}}}$  or any element of the probability simplex. We define the general tempered relative entropy as
$$
D _ {t} \left(\boldsymbol {q} ^ {\prime} \| \boldsymbol {q}\right) \doteq \sum_ {i \in [ m ]} q _ {i} ^ {\prime} \cdot \left(\log_ {t} q _ {i} ^ {\prime} - \log_ {t} q _ {i}\right) - \log_ {t - 1} q _ {i} ^ {\prime} + \log_ {t - 1} q _ {i}, \tag {2}
$$
where  $[m] \doteq \{1, \dots, m\}$ . The tempered relative entropy is a Bregman divergence with convex generator  $\varphi_t(z) \doteq z \log_t z - \log_{t-1}(z)$  (for  $t \in \mathbb{R}$ ) and  $\varphi_t(z)' = \log_t(x)$ . As  $t \to 1$ ,  $D_t(\boldsymbol{q}, \boldsymbol{q}')$  becomes the relative entropy with generator  $\varphi_1(x) = x \log(x) - x$ .
4 Tempered boosting as tempered entropy projection
We start with a fixed sample  $\mathcal{S} = \{(x_i, y_i) : i \in [m]\}$  where observations  $x_i$  lie in some domain  $\mathcal{X}$  and labels  $y_i$  are  $\pm 1$ . ADABOOST maintains a distribution  $\pmb{p}$  over the sample. At the current iteration, this distribution is updated based on a current weak hypothesis  $h \in \mathbb{R}^{\mathcal{X}}$  using an exponential update:
$$
p _ {i} ^ {\prime} = \frac {p _ {i} \cdot \exp (- \mu u _ {i})}{\sum_ {k} p _ {k} \cdot \exp (- \mu u _ {k})}, \text {w h e r e} u _ {i} \doteq y _ {i} h (\boldsymbol {x} _ {i}), \mu \in \mathbb {R}.
$$
In [12] this update is motivated as minimizing a relative entropy subject to the constraint that  $\pmb{p}^{\prime}$  is a distribution summing to 1 and  $\pmb{p}^{\prime \top}\pmb{u} = 0$ . Following this blueprint, we create a boosting algorithm maintaining a discrete TEM over the sample which is motivated as a constrained minimization of the tempered relative entropy, with a normalization constraint on the co-simplex of  $\mathbb{R}^m$ :
$$
\boldsymbol {q} ^ {\prime} \quad = \quad \arg \quad \min  _ {\substack {\widetilde {\boldsymbol {q}} \in \tilde {\Delta} _ {m} \\ \widetilde {\boldsymbol {q}} ^ {\top} \boldsymbol {u} = 0}} D _ {t} (\widetilde {\boldsymbol {q}} \| \boldsymbol {q}), \quad \text {with} \boldsymbol {u} \in \mathbb {R} ^ {m}. \tag{3}
$$
We now show that the solution  $\pmb{q}^{\prime}$  is a tempered generalization of ADABOOST's exponential update.
Theorem 1. For all  $t \in \mathbb{R} \setminus \{2\}$ , all solutions to (3) have the form
$$
q _ {i} ^ {\prime} = \frac {\exp_ {t} \left(\log_ {t} q _ {i} - \mu u _ {i}\right)}{Z _ {t}} \quad \left(= \frac {q _ {i} \otimes_ {t} \exp_ {t} (- \mu u _ {i})}{Z _ {t}}, w i t h a \otimes_ {t} b \doteq \left[ a ^ {1 - t} + b ^ {1 - t} - 1 \right] _ {+} ^ {\frac {1}{1 - t}}\right), \tag {4}
$$
where  $Z_{t}$  ensures co-simplex normalization of the co-density. Furthermore, the unknown  $\mu$  satisfies
$$
\mu \in \arg \max  - \log_ {t} \left(Z _ {t} (\mu)\right) \quad (= \arg \min  Z _ {t} (\mu)), \tag {5}
$$
Algorithm 1  $t$ -ADABOOST(t, S, J)
Input:  $t \in [0,1]$ , training sample  $S$ , #iterations  $J$ ;
Output: classifiers  $H_{J}, H_{J}^{(1/1 - t)}$  (see (9));
Step 1: initialize tempered weights:  $\pmb{q}_1 = (1 / m^{t^*}) \cdot \mathbf{1}$  ( $\in \tilde{\Delta}_m$ );
Step 2: for  $j = 1, 2, \dots, J$
Step 2.1: get weak classifier  $h_j \gets \text{weak\_learner}(\mathbf{q}_j, \mathcal{S})$
Step 2.2: choose weight update coefficient  $\mu_j \in \mathbb{R}$ ;
Step  $2.3:\forall i\in [m]$  , for  $u_{ji}\doteq y_ih_j(\pmb {x}_i)$  , update the tempered weights as
$$
q _ {(j + 1) i} = \frac {q _ {j i} \otimes_ {t} \exp_ {t} (- \mu_ {j} u _ {j i})}{Z _ {t j}}, \quad \text {w h e r e} Z _ {t j} = \left\| \boldsymbol {q} _ {j} \otimes_ {t} \exp_ {t} (- \mu_ {j} \boldsymbol {u} _ {j}) \right\| _ {1 / t ^ {*}}. \tag {8}
$$
Step 2.4: choose leveraging coefficient  $\alpha_{j}\in \mathbb{R}$
or equivalently is a solution to the nonlinear equation
$$
\boldsymbol {q} ^ {\prime} (\mu) ^ {\top} \boldsymbol {u} = 0. \tag {6}
$$
Finally, if either (i)  $t \in \mathbb{R}_{>0} \backslash \{2\}$  or (ii)  $t = 0$  and  $\pmb{q}$  is not collinear to  $\pmb{u}$ , then  $Z_{t}(\mu)$  is strictly convex: the solution to (3) is thus unique, and can be found from expression (4) by finding the unique minimizer of (5) or (equivalently) the unique solution to (6).
(Proof in APP., Section II.1) The  $t$ -product  $\otimes_t$ , which satisfies  $\exp_t(a + b) = \exp_t(a)\otimes_t\exp_t(b)$ , was introduced in [18]. Collinearity never happens in our ML setting because  $\pmb{u}$  contains the edges of a weak classifier:  $\pmb{q} > 0$  and collinearity would imply that  $\pm$  the weak classifier performs perfect classification, and thus defeats the purpose of training an ensemble.  $\forall t\in \mathbb{R}\backslash \{2\}$ , we have the simplified expression for the normalization coefficient of the TEM and the co-density  $\pmb{p}^{\prime}$  of  $\pmb{q}^{\prime}$ :
$$
Z _ {t} = \left\| \exp_ {t} \left(\log_ {t} \boldsymbol {q} - \mu \cdot \boldsymbol {u}\right) \right\| _ {1 / t ^ {*}}; p _ {i} ^ {\prime} = \frac {p _ {i} \otimes_ {t *} \exp_ {t *} \left(- \frac {\mu u _ {i}}{t ^ {*}}\right)}{Z _ {t} ^ {\prime}} \left(\text {w i t h} Z _ {t} ^ {\prime} \doteq Z _ {t} ^ {1 / t ^ {*}}\right). \tag {7}
$$
5 Tempered boosting for linear classifiers and clipped linear classifiers
Models A model (or classifier) is an element of  $\mathbb{R}^{\mathcal{X}}$ . For any model  $H$ , its empirical risk over  $\mathcal{S}$  is  $F_{0/1}(H, \mathcal{S}) \doteq (1/m) \cdot \sum_{i} [y_i \neq \mathrm{sign}(H(\boldsymbol{x}_i))]$  where  $[.]$ , Iverson's bracket [13], is the Boolean value of the inner predicate. We learn linear separators and clipped linear separators. Let  $(v_j)_{j \geqslant 1}$  be the terms of a series and  $\delta \geqslant 0$ . The clipped sum of the series is:
$$
\sum_ {(- \delta) j \in [ J ]} ^ {(\delta)} v _ {j} \doteq \min  \left\{\delta , \max  \left\{- \delta , v _ {J} + \sum_ {j \in [ J - 1 ]} ^ {(\delta)} v _ {j} \right\} \right\} (\in [ - \delta , \delta ]), \text {f o r} J > 1,
$$
and we define the base case  $J = 1$  by replacing the inner clipped sum with 0. Note that clipped summation is non-commutative, and so is different from clipping in  $[-\delta, \delta]$  the whole sum itself<sup>2</sup>. Given a set of so-called weak hypotheses  $h_j \in \mathbb{R}^{\mathcal{X}}$  and leveraging coefficients  $\alpha_j \in \mathbb{R}$  (for  $j \in [J]$ ), the corresponding linear separators and clipped linear separators are
$$
H _ {J} (\boldsymbol {x}) \doteq \sum_ {j \in [ J ]} \alpha_ {j} h _ {j} (\boldsymbol {x}); H _ {J} ^ {(\delta)} (\boldsymbol {x}) \doteq \sum_ {\substack {(- \delta) \\ j \in [ J ]}} ^ {(\delta)} \alpha_ {j} h _ {j} (\boldsymbol {x}). \tag{9}
$$
Tempered boosting and its general convergence Our algorithm,  $t$ -ADABOOST, is presented in Algorithm 1, using presentation conventions from [30]. Before analyzing its convergence, several properties are to be noted for  $t$ -ADABOOST: first, it keeps the appealing property, introduced by ADABOOST, that examples receiving the wrong class by the current weak classifier are reweighted
Fix for example  $a = -1, b = 3, \delta = 2$ . For  $v_{1} = a, v_{2} = b$ , the clipped sum is  $2 = -1 + 3$ , but for  $v_{1} = b, v_{2} = a$ , the clipped sum becomes  $1 = 2 - 1$ .
higher (if  $\mu_j > 0$ ). Second, the leveraging coefficients for weak classifiers in the final classifier  $(\alpha_j s)$  are not the same as the ones used to update the weights  $(\mu_j s)$ , unless  $t = 1$ . Third and last, because of the definition of  $\exp_t(1)$ , if  $t < 1$ , tempered weights can switch off and on, i.e., become 0 if an example is "too well classified" and then revert back to being  $> 0$  if the example becomes wrongly classified by the current weak classifier (if  $\mu_j > 0$ ). To take into account those zeroing weights, we denote  $[m]_j^\dagger \doteq \{i : q_{ji} = 0\}$  and  $m_j^\dagger \doteq \mathrm{Card}([m]_j^\dagger)$  ( $\forall j \in [J]$  and Card denotes the cardinal). Let  $R_j \doteq \max_{i \notin [m]_j^\dagger} |y_i h_j(\boldsymbol{x}_i)| / q_{ji}^{1 - t}$  and  $q_j^\dagger \doteq \max_{i \in [m]_j^\dagger} |y_i h_j(\boldsymbol{x}_i)|^{1/(1 - t)} / R_j^{1/(1 - t)}$ . It is worth noting that  $q_j^\dagger$  is homogeneous to a tempered weight.
Theorem 2. At iteration  $j$ , define the weight function  $q_{ji}' \doteq q_{ji}$  if  $i \notin [m]_j^\dagger$  and  $q_j^\dagger$  otherwise; set
$$
\rho_ {j} \quad \doteq \quad \frac {1}{\left(1 + m _ {j} ^ {\dagger} q _ {j} ^ {\dagger 2 - t}\right) R _ {j}} \cdot \sum_ {i \in [ m ]} q _ {j i} ^ {\prime} y _ {i} h _ {j} \left(\boldsymbol {x} _ {i}\right) \quad (\in [ - 1, 1 ]). \tag {10}
$$
In algorithm  $t$ -ADABOOST, consider the choices (with the convention  $\prod_{k=1}^{0} v_k \doteq 1$ )
$$
\mu_ {j} \doteq - \frac {1}{R _ {j}} \cdot \log_ {t} \left(\frac {1 - \rho_ {j}}{M _ {1 - t} \left(1 - \rho_ {j} , 1 + \rho_ {j}\right)}\right), \quad \alpha_ {j} \doteq m ^ {1 - t ^ {*}} \cdot \left(\prod_ {k = 1} ^ {j - 1} Z _ {k}\right) ^ {1 - t} \cdot \mu_ {j}, \tag {11}
$$
where  $M_q(a, b) \doteq ((a^q + b^q) / 2)^{1/q}$  is the  $q$ -power mean. Then for any  $H \in \{H_J, H_J^{(1/1 - t)}\}$ , its empirical risk is upperbounded as:
$$
F _ {0 / 1} (H, \mathcal {S}) \leqslant \prod_ {j = 1} ^ {J} Z _ {t j} ^ {2 - t} \leqslant \prod_ {j = 1} ^ {J} \left(1 + m _ {j} ^ {\dagger} q _ {j} ^ {\dagger 2 - t}\right) \cdot K _ {t} (\rho_ {j}) \quad \left(K _ {t} (z) \doteq \frac {1 - z ^ {2}}{M _ {1 - t} (1 - z , 1 + z)}\right). \tag {12}
$$
(Proof in APP., Section II.2) We jointly comment  $t$ -ADABOOST and Theorem 2 in two parts.
Case  $t \to 1^{-}$ :  $t$ -ADABOOST converges to ADABOOST and Theorem 2 to its convergence analysis:  $t$ -ADABOOST converges to ADABOOST as presented in [30, Figure 1]: the tempered simplex becomes the probability simplex,  $\otimes_{t}$  converges to regular multiplication, weight update (8) becomes ADABOOST's,  $\alpha_{j} \to \mu_{j}$  in (11) and finally the expression of  $\mu_{j}$  converges to ADABOOST's leveraging coefficient in [30] ( $\lim_{t \to 1} M_{1 - t}(a, b) = \sqrt{ab}$ ). Even guarantee (12) converges to ADABOOST's popular guarantee of [30, Corollary 1] ( $\lim_{t \to 1} K_{t}(z) = \sqrt{1 - z^{2}}, m_{j}^{\dagger} = 0$ ). Also, in this case, we learn only the unclipped classifier since  $\lim_{t \to 1^{-}} H_{J}^{(1/1 - t)} = H_{J}$ .
Case  $t < 1$ : Let us first comment on the convergence rate. The proof of Theorem 2 shows that  $K_{t}(z) \leqslant \exp(-z^{2} / (2t^{*}))$ . Suppose there is no weight switching, so  $m_{j}^{\dagger} = 0, \forall j$  (see Section 7) and, as in the boosting model, suppose there exists  $\gamma > 0$  such that  $|\rho_{j}| \geqslant \gamma, \forall j$ . Then  $t$ -ADABOOST is guaranteed to attain empirical risk below some  $\varepsilon > 0$  after a number of iterations equal to  $J = (2t^{*} / \gamma^{2}) \cdot \log(1 / \varepsilon)$ .  $t^{*}$  being an increasing function of  $t \in [0,1]$ , we see that  $t$ -ADABOOST is able to slightly improve upon ADABOOST's celebrated rate [32]. However,  $t^{*} = 1 / 2$  for  $t = 0$  so the improvement is just on the hidden constant. This analysis is suited for small values of  $|\rho_{j}|$  and does not reveal an interesting phenomenon for better weak hypotheses. Figure 1 compares  $K_{t}(z)$  curves  $(K_{1}(z) \doteq \lim_{t \to 1} K_{t}(z) = \sqrt{1 - z^{2}}$  for ADABOOST, see [30, Corollary 1]), showing the case  $t < 1$  can be substantially better, especially when weak hypotheses are not "too weak". If  $m_{j}^{\dagger} > 0$ , switching weights can impede our convergence analysis, though exponential convergence is always possible if  $m_{j}^{\dagger} q_{j}^{\dagger 2 - t}$  is small enough; also, when it is not, we may in fact have converged to a good model (see APP., Remark 1). A good criterion to train weak hypotheses is then the optimization of the edge  $\rho_{j}$ , thus using  $q_{j}'$  normalized in the simplex. Other key features of  $t$ -ADABOOST are as follows. First, the weight update and leveraging coefficients of weak classifiers are bounded because  $|\mu_{j}| < 1 / (R_{j}(1 - t))$  (APP., Lemma H) (this is not the case for  $t \to 1^{-}$ ). This guarantees that new weights are bounded before normalization (unlike for  $t \to 1^{-}$ ). Second, we remark that  $\mu_{j} \neq \alpha_{j}$  if  $t \neq 1$ . Factor  $m^{1 - t^{*}}$  is added for convergence analysis purposes; we can discard it to train the unclipped classifier: it does not change its empirical risk. This is, however, different for factor  $\prod_{k=1}^{j-1} Z_{k}$ : from (12), we conclude that this is an indication of how well the past ensemble performs.
Figure 1: Plot of  $K_{t}(z)$  in (12),  $t \in [0, 1]$  (the smaller, the better for convergence).
As it gets better and better, it progressively dampens the leverage of the next weak classifiers, a phenomenon that does not occur in boosting, where an excellent weak hypothesis on the current weights can have a leveraging coefficient so large that it wipes out the classification of the past ones. This can be useful to control numerical instabilities.