SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Part (A): encoding linear models with a tree architecture We use the reduction trick of (author?) [16] to design a decision tree (DT) boosting procedure, find out the (concave) loss equivalently minimized, just like in classical top-down DT induction algorithms [6]. The trick is simple: a DT can be thought of as a set of constant linear classifiers. The prediction is the sum of predictions put at all nodes. Boosting fits those predictions at the nodes and percolating those to leaves gets a standard DT with real predictions at the leaves. Figure 2 provides a detailed description of the procedure. Let  $\lambda$  denote a leaf node of the current tree  $H$ , with  $H_{\lambda} \in \mathbb{R}$  the function it implements for leaf  $\lambda$ . If parent  $(\lambda)$  denotes its parent node (assuming wlog it is not the root node), we have
$$
H _ {\lambda} \stackrel {{\cdot}} {{=}} H _ {\text {p a r e n t} (\lambda)} + \mu_ {\lambda} h _ {\lambda}, \tag {60}
$$
Part (B): eliciting the Bayes risk of the tempered loss With our simple classifiers at hand, the tempered exponential loss  $Z_{tj}^{2 - t}$  in (14) (MF) can be simplified to loss
$$
\begin{array}{l} L (H) \doteq \sum_ {i} \exp_ {t} ^ {2 - t} \left(\log_ {t} q _ {1 i} - y _ {i} H _ {\lambda (\boldsymbol {x} _ {i})}\right) \\ = \sum_ {\lambda \in \Lambda (H)} m _ {\lambda} ^ {+} \exp_ {t} ^ {2 - t} \left(\log_ {t} q _ {1 i} - H _ {\lambda}\right) + m _ {\lambda} ^ {-} \exp_ {t} ^ {2 - t} \left(\log_ {t} q _ {1 i} + H _ {\lambda}\right), \tag {61} \\ \end{array}
$$
Figure 2: The weak learner provides weak hypotheses of the form  $\llbracket x_k \geqslant a_j \rrbracket \cdot b_j$ . From the boosting standpoint, this weak hypothesis is "as good as" the weak hypothesis  $\overline{h}_j(x) \doteq \llbracket x_k < a_j \rrbracket \cdot -b_j$ . The predicates of both are used to craft a split, e.g. for the root (in our depiction,  $b_3 = -b_2$ ) and then solving (59) provides the leveraging coefficients  $\mu$ . We then repeat this for as many splits as necessary. At the end, we can "percolate" nodes reals towards the leaves below and get an equivalent classifier that resembles a decision tree (right). See [16] for further details.
where  $\lambda (\pmb {x})$  is the leaf reached by observation  $\pmb{x}$  and  $\lambda (H)$  its set of leaf nodes of  $H$ , and  $H_{\lambda}$  sums all relevant values in (60). Also,  $m_{\lambda}^{+},m_{\lambda}^{-}$  denote the cardinal of positive and negative examples at  $\lambda$  and  $p_{\lambda}\doteq m_{\lambda}^{+} / (m_{\lambda}^{+} + m_{\lambda}^{-})$  the local proportion of positive examples at  $\lambda$ , and finally  $r_\lambda \doteq (m_\lambda^+ +m_\lambda^-) / m$  the total proportion of examples reaching  $\lambda$ .
Theorem A. If we compute  $\mu_{\lambda}$  the solution of (59), we end up with the prediction  $H_{\lambda}$ :
$$
\begin{array}{l} H _ {\lambda} = \frac {q _ {1 i} ^ {1 - t}}{1 - t} \cdot \frac {\left(\frac {m _ {\lambda} ^ {+}}{m _ {\lambda} ^ {-}}\right) ^ {1 - t} - 1}{\left(\frac {m _ {\lambda} ^ {+}}{m _ {\lambda} ^ {-}}\right) ^ {1 - t} + 1} (62) \\ = \frac {q _ {1 i} ^ {1 - t}}{1 - t} \cdot \frac {p _ {\lambda} ^ {1 - t} - (1 - p _ {\lambda}) ^ {1 - t}}{p _ {\lambda} ^ {1 - t} + (1 - p _ {\lambda}) ^ {1 - t}}, (63) \\ \end{array}
$$
and the loss of the decision tree equals:
$$
\begin{array}{l} L (H) = \sum_ {\lambda \in \Lambda (H)} r _ {\lambda} \cdot \frac {2 p _ {\lambda} \left(1 - p _ {\lambda}\right)}{M _ {1 - t} \left(p _ {\lambda} , 1 - p _ {\lambda}\right)}, (64) \\ = \mathbb {E} _ {\lambda} \left[ \underline {{L}} ^ {(t)} \left(p _ {\lambda}\right) \right]. (65) \\ \end{array}
$$
Proof. To compute  $\mu_{\lambda}$ , (6) is reduced to the examples reaching  $\lambda$ , that is, it simplifies to
$$
m _ {\lambda} ^ {+} \exp_ {t} \left(\log_ {t} q _ {1 i} - H _ {\text {p a r e n t} (\lambda)} - R _ {\lambda} \mu_ {\lambda} h _ {\lambda}\right) = m _ {\lambda} ^ {-} \exp_ {t} \left(\log_ {t} q _ {1 i} + H _ {\text {p a r e n t} (\lambda)} + R _ {\lambda} \mu_ {\lambda} h _ {\lambda} 6\right)
$$
that we solve for  $\mu_{\lambda}$ . Equivalently,
$$
\frac {\exp_ {t} \left(\log_ {t} q _ {1 i} + H _ {\mathrm {p a r e n t} (\lambda)} + R _ {\lambda} \mu_ {\lambda} h _ {\lambda}\right)}{\exp_ {t} \left(\log_ {t} q _ {1 i} - H _ {\mathrm {p a r e n t} (\lambda)} - R _ {\lambda} \mu_ {\lambda} h _ {\lambda}\right)} = \frac {m _ {\lambda} ^ {+}}{m _ {\lambda} ^ {-}},
$$
or, using  $\exp_t(u) / \exp_t(v) = \exp_t(u\ominus_t v)$
$$
\frac {2 H _ {\mathrm {p a r e n t} (\lambda)} + 2 R _ {\lambda} \mu_ {\lambda} h _ {\lambda}}{1 + (1 - t) (\log_ {t} q _ {1 i} - H _ {\mathrm {p a r e n t} (\lambda)} - R _ {\lambda} \mu_ {\lambda} h _ {\lambda})} = \log_ {t} \left(\frac {m _ {\lambda} ^ {+}}{m _ {\lambda} ^ {-}}\right),
$$
after reorganizing:
$$
R _ {\lambda} \mu_ {\lambda} h _ {\lambda} = \frac {(1 + (1 - t) (\log_ {t} q _ {1 i} - H _ {\mathrm {p a r e n t} (\lambda)})) \cdot \log_ {t} \left(\frac {m _ {\lambda} ^ {+}}{m _ {\lambda} ^ {-}}\right) - 2 H _ {\mathrm {p a r e n t} (\lambda)}}{2 + (1 - t) \log_ {t} \left(\frac {m _ {\lambda} ^ {+}}{m _ {\lambda} ^ {-}}\right)},
$$
which yields the prediction at  $\lambda$ :
$$
\begin{array}{l} H _ {\lambda} = H _ {\text {p a r e n t} (\lambda)} + \frac {\left(1 + (1 - t) \left(\log_ {t} q _ {1 i} - H _ {\text {p a r e n t} (\lambda)}\right)\right) \cdot \log_ {t} \left(\frac {m _ {\lambda} ^ {+}}{m _ {\lambda} ^ {-}}\right) - 2 H _ {\text {p a r e n t} (\lambda)}}{2 + (1 - t) \log_ {t} \left(\frac {m _ {\lambda} ^ {+}}{m _ {\lambda} ^ {-}}\right)} (67) \\ = \frac {\left(1 + (1 - t) \log_ {t} q _ {1 i}\right) \cdot \log_ {t} \left(\frac {m _ {\lambda} ^ {+}}{m _ {\lambda} ^ {-}}\right)}{2 + (1 - t) \log_ {t} \left(\frac {m _ {\lambda} ^ {+}}{m _ {\lambda} ^ {-}}\right)} (68) \\ = q _ {1 i} ^ {1 - t} \cdot \frac {\log_ {t} \left(\frac {m _ {\lambda} ^ {+}}{m _ {\lambda} ^ {-}}\right)}{2 + (1 - t) \log_ {t} \left(\frac {m _ {\lambda} ^ {+}}{m _ {\lambda} ^ {-}}\right)} (69) \\ = \frac {q _ {1 i} ^ {1 - t}}{1 - t} \cdot \frac {\left(\frac {m _ {\lambda} ^ {+}}{m _ {\lambda} ^ {-}}\right) ^ {1 - t} - 1}{\left(\frac {m _ {\lambda} ^ {+}}{m _ {\lambda} ^ {-}}\right) ^ {1 - t} + 1} (70) \\ = \frac {q _ {1 i} ^ {1 - t}}{1 - t} \cdot \frac {p _ {\lambda} ^ {1 - t} - (1 - p _ {\lambda}) ^ {1 - t}}{p _ {\lambda} ^ {1 - t} + (1 - p _ {\lambda}) ^ {1 - t}}. (71) \\ \end{array}
$$
We plug  $H_{\lambda}$  back in the loss for all leaves and get, using  $q_{1i} = 1 / m^{1 / (2 - t)}$ :
$$
L (H) = \sum_ {\lambda \in \Lambda (H)} \left\{ \begin{array}{l} m _ {\lambda} ^ {+} \exp_ {t} ^ {2 - t} \left(\log_ {t} q _ {1 i} - \frac {q _ {1 i} ^ {1 - t}}{1 - t} \cdot \frac {p _ {\lambda} ^ {1 - t} - (1 - p _ {\lambda}) ^ {1 - t}}{p _ {\lambda} ^ {1 - t} + (1 - p _ {\lambda}) ^ {1 - t}}\right) \\ + m _ {\lambda} ^ {-} \exp_ {t} ^ {2 - t} \left(\log_ {t} q _ {1 i} + \frac {q _ {1 i} ^ {1 - t}}{1 - t} \cdot \frac {p _ {\lambda} ^ {1 - t} - (1 - p _ {\lambda}) ^ {1 - t}}{p _ {\lambda} ^ {1 - t} + (1 - p _ {\lambda}) ^ {1 - t}}\right) \end{array} . \right. \tag {72}
$$
We simplify. First,
$$
\begin{array}{l} m _ {\lambda} ^ {+} \exp_ {t} ^ {2 - t} \left(\log_ {t} q _ {1 i} - \frac {q _ {1 i} ^ {1 - t}}{1 - t} \cdot \frac {p _ {\lambda} ^ {1 - t} - (1 - p _ {\lambda}) ^ {1 - t}}{p _ {\lambda} ^ {1 - t} + (1 - p _ {\lambda}) ^ {1 - t}}\right) \\ = m _ {\lambda} ^ {+} \left[ q _ {1 i} ^ {1 - t} \cdot \left(1 - \frac {p _ {\lambda} ^ {1 - t} - (1 - p _ {\lambda}) ^ {1 - t}}{p _ {\lambda} ^ {1 - t} + (1 - p _ {\lambda}) ^ {1 - t}}\right) \right] _ {+} ^ {\frac {2 - t}{1 - t}} \\ = \frac {m _ {\lambda} ^ {+}}{m} \cdot \left[ \frac {2 (1 - p _ {\lambda}) ^ {1 - t}}{p _ {\lambda} ^ {1 - t} + (1 - p _ {\lambda}) ^ {1 - t}} \right] _ {+} ^ {\frac {2 - t}{1 - t}} (73) \\ = \frac {m _ {\lambda} ^ {+}}{m} \cdot \left(\frac {1 - p _ {\lambda}}{M _ {1 - t} \left(p _ {\lambda} , 1 - p _ {\lambda}\right)}\right) ^ {2 - t}, (74) \\ \end{array}
$$
$$
\begin{array}{l} m _ {\lambda} ^ {-} \exp_ {t} ^ {2 - t} \left(\log_ {t} q _ {1 i} + \frac {q _ {1 i} ^ {1 - t}}{1 - t} \cdot \frac {p _ {\lambda} ^ {1 - t} - (1 - p _ {\lambda}) ^ {1 - t}}{p _ {\lambda} ^ {1 - t} + (1 - p _ {\lambda}) ^ {1 - t}}\right) \\ = m _ {\lambda} ^ {-} \left[ q _ {1 i} ^ {1 - t} \cdot \left(1 + \frac {p _ {\lambda} ^ {1 - t} - (1 - p _ {\lambda}) ^ {1 - t}}{p _ {\lambda} ^ {1 - t} + (1 - p _ {\lambda}) ^ {1 - t}}\right) \right] _ {+} ^ {\frac {2 - t}{1 - t}} \\ = \frac {m _ {\lambda} ^ {-}}{m} \cdot \left[ \frac {2 p _ {\lambda} ^ {1 - t}}{p _ {\lambda} ^ {1 - t} + (1 - p _ {\lambda}) ^ {1 - t}} \right] _ {+} ^ {\frac {2 - t}{1 - t}} (75) \\ = \frac {m _ {\lambda} ^ {-}}{m} \cdot \left(\frac {p _ {\lambda}}{M _ {1 - t} \left(p _ {\lambda} , 1 - p _ {\lambda}\right)}\right) ^ {2 - t}, (76) \\ \end{array}
$$
and we can simplify the loss,
$$
\begin{array}{l} L (H) = \sum_ {\lambda \in \Lambda (H)} r _ {\lambda} p _ {\lambda} \left(\frac {1 - p _ {\lambda}}{M _ {1 - t} \left(p _ {\lambda} , 1 - p _ {\lambda}\right)}\right) ^ {2 - t} + r _ {\lambda} \left(1 - p _ {\lambda}\right) \left(\frac {p _ {\lambda}}{M _ {1 - t} \left(p _ {\lambda} , 1 - p _ {\lambda}\right)}\right) ^ {2 - t} (77) \\ = \sum_ {\lambda \in \Lambda (H)} r _ {\lambda} \cdot \frac {p _ {\lambda} \left(1 - p _ {\lambda}\right) ^ {2 - t} + \left(1 - p _ {\lambda}\right) p _ {\lambda} ^ {2 - t}}{M _ {1 - t} ^ {2 - t} \left(p _ {\lambda} , 1 - p _ {\lambda}\right)} (78) \\ = \sum_ {\lambda \in \Lambda (H)} r _ {\lambda} \cdot \frac {p _ {\lambda} \left(1 - p _ {\lambda}\right) \cdot \left(p _ {\lambda} ^ {1 - t} + \left(1 - p _ {\lambda}\right) ^ {1 - t}\right)}{M _ {1 - t} ^ {2 - t} \left(p _ {\lambda} , 1 - p _ {\lambda}\right)} (79) \\ = \sum_ {\lambda \in \Lambda (H)} r _ {\lambda} \cdot \frac {2 p _ {\lambda} (1 - p _ {\lambda}) \cdot M _ {1 - t} ^ {1 - t} \left(p _ {\lambda} , 1 - p _ {\lambda}\right)}{M _ {1 - t} ^ {2 - t} \left(p _ {\lambda} , 1 - p _ {\lambda}\right)} (80) \\ = \sum_ {\lambda \in \Lambda (H)} r _ {\lambda} \cdot \frac {2 p _ {\lambda} (1 - p _ {\lambda})}{M _ {1 - t} \left(p _ {\lambda} , 1 - p _ {\lambda}\right)}, (81) \\ \end{array}
$$
as claimed. This ends the proof of Theorem A.
Part (C): partial losses and their properties The proof relies on the following Theorem. We recall that a loss is symmetric iff its partial losses satisfy  $\ell_1(u) = \ell_{-1}(1 - u), \forall u \in [0,1]$  [20] and differentiable iff its partial losses are differentiable.
Theorem B. Suppose  $t < 2$ . A set of partial losses having the conditional Bayes risk  $\underline{L}^{(t)}$  in (65) are
$$
\ell_ {1} ^ {(t)} (u) \doteq \left(\frac {1 - u}{M _ {1 - t} (u , 1 - u)}\right) ^ {2 - t}, \quad \ell_ {- 1} ^ {(t)} (u) \doteq \ell_ {1} ^ {(t)} (1 - u). \tag {82}
$$
The tempered loss is then symmetric and differentiable. It is strictly proper for any  $t \in (-\infty, 2)$  and proper for  $t = -\infty$ .
Proof. Symmetry and differentiability are straightforward. To check strict properness, we analyze the cases for  $t \neq 1$  (otherwise, it is Matusita's loss, and thus strictly proper), we compute the solution  $u$  to
$$
\frac {\partial}{\partial u} L (u, v) = 0. \tag {83}
$$
To this end, let  $N(u) \doteq v(1 - u)^{2 - t} + (1 - v)u^{2 - t}$  and the  $q$ -sum
$$
S _ {q} (a, b) \quad \doteq \quad \left(a ^ {q} + b ^ {q}\right) ^ {1 / q} = 2 ^ {1 / q} \cdot M _ {q} (a, b). \tag {84}
$$
We also let  $D(u) \doteq S_{1 - t}^{2 - t}(u, 1 - u)$ . Noting  $L^{(t)}(u, v) = 2^{\frac{2 - t}{1 - t}} \cdot N(u) / D(u)$  and  $D(u) \neq 0, \forall u \in [0, 1]$ , the set of solutions of (83) are the set of solutions to  $N'(u)D(u) = N(u)D'(u)$ , which boils down, after simplification, to
$$
\begin{array}{l} ((1 - v) u ^ {1 - t} - v (1 - u) ^ {1 - t}) S _ {1 - t} ^ {2 - t} (u, 1 - u) \\ = \left(v (1 - u) ^ {2 - t} + (1 - v) u ^ {2 - t}\right) \left(u ^ {- t} - (1 - u) ^ {- t}\right) S _ {1 - t} (u, 1 - u), \\ \end{array}
$$
developing and simplifying yields a first simplified expression  $(1 - 2v)(u(1 - u))^{1 - t} = vu^{-1}(1 - u)^{2 - t} - (1 - v)(1 - u)^{-t}u^{2 - t}$ , which, after reorganising to isolate expressions depending on  $v$ , yields
$$
(u (1 - u)) ^ {1 - t} + (1 - u) ^ {- t} u ^ {2 - t} = v \cdot (u ^ {- t} (1 - u) ^ {2 - t} + (1 - u) ^ {- t} u ^ {2 - t} + 2 (u (1 - u)) ^ {1 - t} \tag {5}
$$
Assuming  $v \in (0,1)$ , we multiply by  $(u(1 - u))^{1 - t}$  (we shall check  $u \in (0,1)$ ) and simplify, which yields  $u(1 - u) + u^2 = v((1 - u)^2 + u^2 + 2u(1 - u))$ , and indeed yields
$$
u = v, \tag {86}
$$
and we check from (85) that if  $v = 0$  (resp.  $v = 1$ ), then necessarily  $u = 0$  (resp.  $u = 1$ ). To complete the proof, using the previous derivations, we can then simplify
$$
\frac {\partial}{\partial u} L (u, v) = (2 - t) \cdot 2 ^ {\frac {2 - t}{1 - t}} \cdot \frac {u - v}{(u (1 - u)) ^ {t} \cdot S _ {1 - t} ^ {3 - 2 t} (u , 1 - u)}, \tag {87}
$$
which shows that if  $2 - t > 0$  but  $t \neq -\infty$ ,  $u = v$  is a strict minimum of the pointwise conditional risk, completing the proof for strict properness. Strict properness is sufficient to show by a simple computation that  $\underline{L}^{(t)}$  is (65). For  $t = -\infty$ , we pass to the limit and use the fact that we can also write
$$
\ell_ {1} ^ {(t)} (u) = \frac {1}{M _ {1 - t ^ {*}} \left(1 , \left(\frac {u}{1 - u}\right) ^ {\frac {1}{t ^ {*}}}\right)} \quad (\text {w e r e c a l l} t ^ {*} \doteq 1 / (2 - t)) \tag {88}
$$
$t\to -\infty$  is equivalent to  $t^*\to 0^+$  . If  $u <   1 / 2,u / (1 - u) <   1$  and so we see that
$$
\lim  _ {t ^ {*} \rightarrow 0 ^ {+}} M _ {1 - t ^ {*}} \left(1, \left(\frac {u}{1 - u}\right) ^ {\frac {1}{t ^ {*}}}\right) = \frac {1}{2},
$$
because  $M_{1}$  is the arithmetic mean. When  $u > 1/2$ ,  $u / (1 - u) > 1$  and so this time
$$
\lim  _ {t ^ {*} \rightarrow 0 ^ {+}} M _ {1 - t ^ {*}} \left(1, \left(\frac {u}{1 - u}\right) ^ {\frac {1}{t ^ {*}}}\right) = + \infty .
$$
$$
\ell_ {1} ^ {(- \infty)} (u) = 2 \cdot [   [ u \leqslant 1 / 2 ]   ], \tag {89}
$$
which is (twice) the partial loss of the  $0 / 1$  loss [26].
This ends the proof of Theorem 4.
Domain	Source	m	d
sonar	UCI	208	60	https://archive.ics.uci.edu/ml/datasets/wine+quality
winered	UCI	1599	12	https://archive.ics.uci.edu/ml/datasets/wine+quality
abalone	UCI	4177	9	https://archive.ics.uci.edu/ml/datasets/abalone
qsar	UCI	1055	41	https://archive.ics.uci.edu/ml/datasets/QSAR+biodegradation
winwhite	UCI	4898	12	https://archive.ics.uci.edu/ml/datasets/wine+quality
hillnonoise	UCI	1212	101	http://archive.ics.uci.edu/ml/datasets/hill-valley
hillnoise	UCI	1212	101	http://archive.ics.uci.edu/ml/datasets/hill-valley
eeg	UCI	14980	15	https://archive.ics.uci.edu/ml/datasets/EEG+Eye+State
creditcard*	UCI	14599	24	https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients
adult	UCI	32561	15	https://archive.ics.uci.edu/ml/datasets/adult
Table A3: Public domains considered in our experiments ( $m =$  total number of examples,  $d =$  total number of example's features, including class), ordered in increasing  $m \times d$  (see text). (*) first  $m$  rows in the domain.
III Supplementary material on experiments
III.1 Domains
Table A3 presents the 10 domains we used for our experiments.
III.2 Implementation details and full set of experiments on linear combinations of decision trees
Summary This Section depicts the full set of experiments summarized in Table 2 (MF), from Table A4 to Table A15. Tables are ordered in increasing size of the domain (Table A3). In all cases, up to  $J = 20$  trees have been trained, of size 15 (total number of nodes, except the two biggest domains, for which the size is 5). For all datasets, except creditcard and adult, we have tested  $t$  in the complete range,  $t \in \{0.0, 0.2, 0.4, 0.6, 0.8, 0.9, 1.0, 1.1\}$  (the MF only reports results for  $t \geqslant 0.6$ ), and in all cases, models both clipped and not clipped. For each dataset, we have set a 10-fold stratified cross-validation experiment, and report the averages for readability (Table 2 in MF gives the results of a Student paired  $t$ -test on error averages for comparison, limit  $p$ -val  $= 0.1$ ). We also provide two examples of training error averages for domains hillnoise and hillnonoise (Tables A10 and A12).
Implementation details of  $t$ -ADABOOST First, regarding file format, we only input a .csv file to  $t$ -ADABOOST. We do not specify a file with feature types as in ARFF files.  $t$ -ADABOOST recognizes the type of each feature from its column content and distinguishes two main types of features: numerical and categorical. The distinction is important to design the splits during decision tree induction: for numerical values, splits are midpoints between two successive observed values. For categorical, splits are partitions of the feature values in two non-empty subsets. Our implementation of  $t$ -ADABOOST (programmed in Java) makes it possible to choose  $t$  not just in the range of values for which we have shown that boosting-compliant convergence is possible ( $t \in [0,1]$ ), but also  $t > 1$ . Because we thus implement ADABOOST ( $t = 1$ ) but also for  $t > 1$ , weights can fairly easily become infinite, we have implemented a safe-check during training, counting the number of times the weights become infinite or zero (note that in this latter case, this really is a problem just for ADABOOST because in theory this should never happen unless the weak classifiers achieve perfect (or perfectly wrong) classification), but also making sure leveraging coefficients for classifiers do not become infinite for ADABOOST, a situation that can happen because of numerical approximations in encoding. In our experiments, we have observed that none of these problematic cases did occur (notice that this could not be the case if we were to boost for a large number of iterations). We have implemented algorithm  $t$ -ADABOOST exactly as specified in MF. The weak learner is implemented to train a decision tree in which the stopping criterion is the size of the tree reaching a user-fixed number of nodes. There is thus no pruning. Also, the top-down induction algorithm proceeds by iteratively picking the heaviest leaf in the tree and then choosing the split that minimizes the expected Bayes risk of the tempered loss, computing using the same  $t$  values as for  $t$ -ADABOOST, and with the constraint to not get pure leaves (otherwise, the real prediction at the leaves, which relies on the
Table A4: Experiments on  $t$ -ADABOOST comparing with ADABOOST ( $t = 1$ , bullets) on domain sonar, when trained without noise ( $\eta = 0.0$ , top row) and with noise ( $\eta = 0.1$ , bottom row). Columns are, from left to right, the estimated true error of non-clipped and clipped models, and the min and max codensity weights. The set of  $t$  values used is displayed in each plot with a colormap (right), and varying thickness of curves for an additional ease of reading (the thicker the curve, the larger  $t$ ). ADABOOST's reference results are displayed with bullets. Averages shown for readability.
link of the loss, would be infinite for ADABOOST). In our implementation of decision-tree induction, when the number of possible splits exceeds a fixed number  $S$  (currently, 2000), we pick the best split in a subset of  $S$  splits picked at random.
Results First, one may notice in several plots that the average test error increases with the number of trees. This turns out to be a sign of overfitting, as exemplified for domains hillnonoise and hillnoise, for which we provide the training curves. If we align the training curves at  $T = 1$  (the value is different because the splitting criterion for training the tree is different), we notice that the experimental convergence on training is similar for all values of  $t$  (Tables A10 and A12). The other key experimental result, already visible from Table 2 (MF), is that pretty much all tested values of  $t$  are necessary to get the best results. One could be tempted to conclude that  $t$  slightly smaller than 1.0 seems to be a good fit from Table 2 (MF), but the curves show that this is more a consequence of the Table being computed for  $J = 20$  trees. The case of eeg illustrates this phenomenon best: while small  $t$ -values are clearly the best when there is no noise, the picture is completely reversed when there is training noise. Notice that this ordering is almost reversed on creditcard and adult: when there is noise, small values of  $t$  tend to give better results. Hence, in addition to getting (i) a pruning mechanism that works for all instances of the tempered loss and (ii) a way to guess the right number of models in the ensemble, a good problem to investigate is in fact appropriately tuning  $t$  in a domain-dependent way. Looking at all plots reveals that substantial gains could be obtained with an accurate procedure (over the strategy that would be to always pick a fixed  $t$ , e.g.  $t = 1$ ).
III.3 Supplementary experiments: learning with more trees / against more noise
We have performed some additional experiments on several domains, on which we have trained bigger ensembles and / or considered more noise levels than in the previous experiments, with the objective to see if / when overfitting happens and how performances degrade with noise as a function of  $t$ . Table A16 summarizes the results obtained. A few comments we can make based on those experiments are:
- overfitting can indeed happen (sonar for  $\eta = 0.4$ ) but affects differently the algorithm depending on  $t$  and whether clipped models are learned instead of regular linear models; results also display that tuning  $t$  can also have the purpose of handling overfitting;
Table A5: Experiments on  $t$ -ADABOOST comparing with ADABOOST ( $t = 1$ , bullets) on domain winered. Conventions follow Table A4.
Table A6: Experiments on  $t$ -ADABOOST comparing with ADABOOST ( $t = 1$ , bullets) on domain abalone. Conventions follow Table A4.
- clamped models can be very useful to handle overfitting (sonar for  $\eta = 0.4$ , qsr for  $\eta \geqslant 0.2$ ); this provides another justification to learn clamped models;
- the overall diversity of curves as a function of  $t$  supports the idea that good strategies could in fact tune  $t$  at training time and change its value with iterations.
Table A7: Experiments on  $t$ -ADABOOST comparing with ADABOOST ( $t = 1$ , bullets) on domain qsr. Conventions follow Table A4.
Table A8: Experiments on  $t$ -ADABOOST comparing with ADABOOST ( $t = 1$ , bullets) on domain winwhite. Conventions follow Table A4.
Table A9: Experiments on  $t$ -ADABOOST comparing with ADABOOST ( $t = 1$ , bullets) on domain hillnonoise. Conventions follow Table A4.
Table A10: Experiments on  $t$ -ADABOOST comparing with ADABOOST ( $t = 1$ , bullets) on domain hillnonoise: training errors displayed for all algorithms using conventions from Table A4. See text for details.
Table A11: Experiments on  $t$ -ADABOOST comparing with ADABOOST ( $t = 1$ , bullets) on domain hillnoise. Conventions follow Table A4.
Table A12: Experiments on  $t$ -ADABOOST comparing with ADABOOST ( $t = 1$ , bullets) on domain hillnoise: training errors displayed for all algorithms using conventions from Table A4. See text for details.
Table A13: Experiments on  $t$ -ADABOOST comparing with ADABOOST ( $t = 1$ , bullets) on domain eeg. Conventions follow Table A4.
Table A14: Experiments on  $t$ -ADABOOST comparing with ADABOOST ( $t = 1$ , bullets) on domain creditcard. Conventions follow Table A4.
Table A15: Experiments on  $t$ -ADABOOST comparing with ADABOOST ( $t = 1$ , bullets) on domain adult. Conventions follow Table A4.
Table A16: Additional experiments with a larger number of trees and various levels of noise, on winered  $(J = 150$  trees,  $T = 2$  splits per tree) and sonar and qsr  $(J = 100, T = 2)$ . Note that the range of noise levels tested is broader than for the other experiments.