SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} q _ {(J + 1) i} \\ = \frac {\exp_ {t} \left(\log_ {t} q _ {J i} - \mu_ {J} u _ {J i}\right)}{Z _ {J}} \\ = \frac {\exp_ {t} \left(\log_ {t} \left(\frac {1}{m ^ {t *} \prod_ {j = 1} ^ {J - 1} Z _ {t j}} \cdot \exp_ {t} \left(- \sum_ {j = 1} ^ {(1 / 1 - t)} v _ {j} u _ {j i}\right)\right) - \mu_ {J} u _ {J i}\right)}{Z _ {J}} \\ = \frac {\exp_ {t} \left(\log_ {t} \exp_ {t} \left(- ^ {(1 / 1 - t)} \sum_ {j = 1} ^ {J - 1} v _ {j} u _ {j i}\right) \ominus_ {t} \log_ {t} \left(m ^ {t ^ {*}} \prod_ {j = 1} ^ {J - 1} Z _ {t j}\right) - \mu_ {J} u _ {J i}\right)}{Z _ {J}} \\ = \frac {1}{Z _ {J}} \cdot \exp_ {t} \left(\frac {\max  \left\{- \frac {1}{1 - t} , - ^ {(1 / 1 - t)} \sum_ {j = 1} ^ {J - 1} v _ {j} u _ {j i} \right\} - \log_ {t} \left(m ^ {t ^ {*}} \prod_ {j = 1} ^ {J - 1} Z _ {t j}\right)}{1 + (1 - t) \log_ {t} \left(m ^ {t ^ {*}} \prod_ {j = 1} ^ {J - 1} Z _ {t j}\right)} - \mu_ {J} u _ {J i}\right) \\ = \frac {1}{Z _ {J}} \cdot \left[ 1 + \frac {(1 - t) \cdot \max  \left\{- \frac {1}{1 - t} , - \sum_ {j = 1} ^ {(1 / 1 - t)} v _ {j} u _ {j i} \right\} - (1 - t) \log_ {t} \left(m ^ {t *} \prod_ {j = 1} ^ {J - 1} Z _ {t j}\right)}{1 + (1 - t) \log_ {t} \left(m ^ {t *} \prod_ {j = 1} ^ {J - 1} Z _ {t j}\right)} \right] _ {+} ^ {\frac {1}{1 - t}} \\ = \frac {1}{Z _ {J}} \cdot \left[ \begin{array}{c} 1 + \frac {(1 - t) \cdot \max  \left\{- \frac {1}{1 - t} , - \sum_ {j = 1} ^ {(1 / 1 - t)} v _ {j} u _ {j i} \right\} - \left(\left(m ^ {t *} \prod_ {j = 1} ^ {J - 1} Z _ {t j}\right) ^ {1 - t} - 1\right)}{\left(m ^ {t *} \prod_ {j = 1} ^ {J - 1} Z _ {t j}\right) ^ {1 - t}} \\ - (1 - t) \mu_ {J} u _ {J i} \end{array} \right] _ {+} ^ {\frac {1}{1 - t}}, \\ \end{array}
$$
which simplifies into (using  $(1 - t)t^{*} = 1 - t^{*})$
$$
\begin{array}{l} q _ {(J + 1) i} \\ = \frac {1}{m ^ {t ^ {*}} \prod_ {j = 1} ^ {J} Z _ {t j}} \cdot \left[ 1 + (1 - t) \cdot \left(\max  \left\{- \frac {1}{1 - t}, - \sum_ {j = 1} ^ {(1 / 1 - t)} v _ {j} u _ {j i} \right\} - v _ {J} u _ {J i}\right) \right] _ {+} ^ {\frac {1}{1 - t}} \tag {39} \\ = \frac {1}{m ^ {t ^ {*}} \prod_ {j = 1} ^ {J} Z _ {t j}} \cdot \exp_ {t} (- S _ {J}), \\ \end{array}
$$
$$
\begin{array}{l} S _ {J} \quad \doteq \quad \min  \left\{- \max  \left\{- \frac {1}{1 - t}, - \sum_ {j = 1} ^ {(1 / 1 - t) J - 1} v _ {j} u _ {j i} \right\} + v _ {J} u _ {J i}, \frac {1}{1 - t} \right\} \\ = \min  \left\{v _ {J} u _ {J i} + \min  \left\{\frac {1}{1 - t}, \sum_ {j = 1} ^ {(1 / 1 - t) J - 1} v _ {j} u _ {j i} \right\}, \frac {1}{1 - t} \right\} \\ = \min  \left\{v _ {J} u _ {J i} + \sum_ {j = 1} ^ {(1 / 1 - t) J - 1} v _ {j} u _ {j i}, \frac {1}{1 - t} \right\} \\ \stackrel {{\quad}} {{=}} \quad^ {(1 / 1 - t)} \sum_ {j = 1} ^ {J} v _ {j} u _ {j i} \\ \end{array}
$$
(we used twice the definition of clipped summation), which completes the proof of Lemma E for  $t < 1$ .
We now treat the case  $t > 1$ . The base induction is equivalent, while unraveling gives, instead of (39):
$$
\begin{array}{l} q _ {(J + 1) i} \\ = \frac {1}{m ^ {t *} \prod_ {j = 1} ^ {J} Z _ {t j}} \cdot \left[ 1 + (1 - t) \cdot \left(\min  \left\{- \frac {1}{1 - t}, - \sum_ {j = 1} ^ {J - 1} v _ {j} u _ {j i} \right\} - v _ {J} u _ {J i}\right) \right] _ {+} ^ {\frac {1}{1 - t}} \\ { = } { \frac { 1 } { m ^ { t ^ { * } } \prod _ { j = 1 } ^ { J } Z _ { t j } } \cdot \exp _ { t } \left( - S _ { J } \right) , } \\ \end{array}
$$
and, this time,
$$
\begin{array}{l} S _ {J} \quad \doteq \quad \max  \left\{- \min  \left\{- \frac {1}{1 - t}, - \sum_ {j = 1} ^ {J - 1} v _ {j} u _ {j i} \right\} + v _ {J} u _ {J i}, - \frac {1}{t - 1} \right\} (40) \\ = \max  \left\{v _ {J} u _ {J i} + \max  \left\{- \frac {1}{t - 1}, \sum_ {j = 1} ^ {J - 1} v _ {j} u _ {j i} \right\}, - \frac {1}{t - 1} \right\} (41) \\ = \max  \left\{v _ {J} u _ {J i} + \sum_ {- (1 / t - 1)} ^ {J - 1} v _ {j} u _ {j i}, - \frac {1}{t - 1} \right\} (42) \\ \dot {=} \quad_ {\left(- 1 / t - 1\right)} \sum_ {j = 1} ^ {J} v _ {j} u _ {j i}, (43) \\ \end{array}
$$
which completes the proof of Lemma E.
II.II.2.3 Introducing classifiers
Ordinary linear separators Suppose we have a classifier
$$
H _ {J} (\boldsymbol {x}) \quad \doteq \quad \sum_ {j = 1} ^ {J} \beta_ {j} ^ {1 - t} \mu_ {j} \cdot h _ {j} (\boldsymbol {x}), \quad \beta_ {j} \doteq m ^ {t ^ {*}} \prod_ {k = 1} ^ {j - 1} Z _ {t k},
$$
where  $\mu_j \in \mathbb{R}, \forall j \in [J]$ . We remark that  $\llbracket z \neq r\rrbracket \leqslant \exp_t^{2 - t}(-zr)$  for any  $t \leqslant 2$  and  $z, r \in \mathbb{R}$ , and  $z \mapsto \exp_t^{2 - t}(-z)$  is decreasing for any  $t \leqslant 2$ , so using [3.] in Lemma D, we get for our training
sample  $\mathcal{S} \doteq \left\{(\pmb{x}_i, y_i), i \in [m]\right\}$  and any  $t < 1$  (from Lemma E),
$$
\begin{array}{l} \frac {1}{m} \cdot \sum_ {i \in [ m ]} \llbracket \operatorname {s i g n} \left(H _ {J} \left(\boldsymbol {x} _ {i}\right)\right) \neq y _ {i} \rrbracket \\ \leqslant \sum_ {i \in [ m ]} \frac {\exp_ {t} ^ {2 - t} \left(- \sum_ {j = 1} ^ {J} m ^ {1 - t ^ {*}} \left(\prod_ {k = 1} ^ {j - 1} Z _ {t k}\right) ^ {1 - t} \mu_ {j} \cdot y _ {i} h _ {j} (\boldsymbol {x} _ {i})\right)}{m} \\ \leqslant \sum_ {i \in [ m ]} \frac {\exp_ {t} ^ {2 - t} \left(- ^ {(1 / 1 - t)} \sum_ {j = 1} ^ {J} m ^ {1 - t ^ {*}} \left(\prod_ {k = 1} ^ {j - 1} Z _ {t k}\right) ^ {1 - t} \mu_ {j} \cdot y _ {i} h _ {j} \left(\boldsymbol {x} _ {i}\right)\right)}{m} \tag {44} \\ = \sum_ {i \in [ m ]} \frac {\exp_ {t} ^ {2 - t} \left(- ^ {(1 / 1 - t)} \sum_ {j = 1} ^ {J} v _ {j} u _ {j i}\right)}{m}, \\ \end{array}
$$
$$
v _ {j} \doteq m ^ {1 - t ^ {*}} \left(\prod_ {k = 1} ^ {j - 1} Z _ {t k}\right) ^ {1 - t} \mu_ {j}; u _ {j i} \doteq y _ {i} h _ {j} \left(\boldsymbol {x} _ {i}\right). \tag {45}
$$
Using Lemma E with those definitions, we get
$$
\begin{array}{l} \frac {1}{m} \cdot \sum_ {i \in [ m ]} \llbracket \operatorname {s i g n} (H _ {J} (\boldsymbol {x} _ {i})) \neq y _ {i} \rrbracket \leqslant \sum_ {i \in [ m ]} \frac {\left(q _ {(J + 1) i} m ^ {t ^ {*}} \prod_ {j = 1} ^ {J} Z _ {t j}\right) ^ {2 - t}}{m} \\ = \prod_ {j = 1} ^ {J} Z _ {t j} ^ {2 - t} \cdot \sum_ {i \in [ m ]} q _ {(J + 1) i} ^ {2 - t} \\ = \prod_ {j = 1} ^ {J} Z _ {t j} ^ {2 - t}, \tag {46} \\ \end{array}
$$
because  $q_{J}\in \tilde{\Delta}_{m}$  . We thus have proven the following Lemma.
Lemma F. For any  $t < 1$  and any linear separator
$$
H _ {J} (\boldsymbol {x}) \doteq \sum_ {j = 1} ^ {J} \beta_ {j} ^ {1 - t} \mu_ {j} \cdot h _ {j} (\boldsymbol {x}), \quad \left(\beta_ {j} \doteq m ^ {t ^ {*}} \prod_ {k = 1} ^ {j - 1} Z _ {t k}, \mu_ {j} \in \mathbb {R}, h _ {j} \in \mathbb {R} ^ {\mathcal {X}}, \forall j \in [ J ]\right),
$$
where  $Z_{tk}$  is the normalization coefficient of  $\pmb{q}$  in (25) with  $u_{ji} \doteq y_i h_j(\pmb{x}_i)$ ,
$$
\frac {1}{m} \cdot \sum_ {i \in [ m ]} \llbracket \operatorname {s i g n} \left(H _ {J} \left(\boldsymbol {x} _ {i}\right)\right) \neq y _ {i} \rrbracket \leqslant \prod_ {j = 1} ^ {J} Z _ {t j} ^ {2 - t}. \tag {47}
$$
Clipped linear separators Suppose we have a classifier  $(t < 1)$
$$
H_{J}^{(1 / 1 - t)}(\boldsymbol {x})\quad \doteq \quad \sum_{(-1 / 1 - t)}^{(1 / 1 - t)}\beta_{j}^{1 - t}\mu_{j}\cdot h_{j}(\boldsymbol {x}),\quad \beta_{j}\doteq m^{t^{*}}\prod_{k = 1}^{j - 1}Z_{tk}.
$$
We can now replace (44) by
$$
\begin{array}{l} \frac {1}{m} \cdot \sum_ {i \in [ m ]} \llbracket \operatorname {s i g n} \left(H _ {J} ^ {(1 / 1 - t)} \left(\boldsymbol {x} _ {i}\right)\right) \neq y _ {i} \rrbracket \\ \leqslant \sum_ {i \in [ m ]} \frac {\exp_ {t} ^ {2 - t} \left(- y _ {i} \cdot \begin{array}{c} (1 / 1 - t) \\ (- 1 / 1 - t) \end{array} \sum_ {j = 1} ^ {J} m ^ {1 - t ^ {*}} \left(\prod_ {k = 1} ^ {j - 1} Z _ {t k}\right) ^ {1 - t} \mu_ {j} \cdot h _ {j} (\boldsymbol {x} _ {i})\right)}{m} \\ = \sum_ {i \in [ m ]} \frac {\exp_ {t} ^ {2 - t} \left(- _ {(- 1 / 1 - t)} ^ {(1 / 1 - t)} \sum_ {j = 1} ^ {J} m ^ {1 - t ^ {*}} \left(\prod_ {k = 1} ^ {j - 1} Z _ {t k}\right) ^ {1 - t} \mu_ {j} \cdot y _ {i} h _ {j} (\boldsymbol {x} _ {i})\right)}{m} \\ \leqslant \sum_ {i \in [ m ]} \frac {\exp_ {t} ^ {2 - t} \left(- ^ {(1 / 1 - t)} \sum_ {j = 1} ^ {J} m ^ {1 - t ^ {*}} \left(\prod_ {k = 1} ^ {j - 1} Z _ {t k}\right) ^ {1 - t} \mu_ {j} \cdot y _ {i} h _ {j} (\boldsymbol {x} _ {i})\right)}{m} \tag {48} \\ = \sum_ {i \in [ m ]} \frac {\exp_ {t} ^ {2 - t} \left(- ^ {(1 / 1 - t)} \sum_ {j = 1} ^ {J} v _ {j} u _ {j i}\right)}{m}. \\ \end{array}
$$
The first identity has used the fact that  $y_{i} \in \{-1, 1\}$ , so it can be folded in the doubly clipped summation without changing its value, and the second inequality used [3.] in Lemma D. This directly leads us to the following Lemma.
Lemma G. For any  $t < 1$  and any clipped linear separator
$$
H_{J}^{(1 / 1 - t)}(\boldsymbol {x})\quad \doteq \quad \sum_{(-1 / 1 - t)}^{J}\beta_{j}^{1 - t}\mu_{j}\cdot h_{j}(\boldsymbol {x}),\quad \left(\beta_{j}\doteq m^{t^{*}}\prod_{k = 1}^{j - 1}Z_{tk},\mu_{j}\in \mathbb{R},h_{j}\in \mathbb{R}^{\mathcal{X}},\forall j\in [J]\right),
$$
$$
\frac {1}{m} \cdot \sum_ {i \in [ m ]} \llbracket \operatorname {s i g n} \left(H _ {J} ^ {(1 / 1 - t)} \left(\boldsymbol {x} _ {i}\right)\right) \neq y _ {i} \rrbracket \leqslant \prod_ {j = 1} ^ {J} Z _ {t j} ^ {2 - t}. \tag {49}
$$
II.II.2.4 Geometric convergence of the empirical risk
To get the right-hand side of (47) and (49) as small as possible, we can independently compute each  $\mu_{j}$  so as to minimize
$$
Z _ {t j} ^ {2 - t} (\mu) \quad \doteq \quad \sum_ {i \in [ m ]} \exp_ {t} ^ {2 - t} \left(\log_ {t} q _ {j i} - \mu u _ {j i}\right). \tag {50}
$$
We proceed in two steps, first computing a convenient upperbound for (50), and then finding the  $\mu$  that minimizes this upperbound.
Step 1: We distinguish two cases depending on weight  $q_{ji}$ . Let  $[m]_j^+ \doteq \{i : q_{ji} > 0\}$  and  $[m]_j^\dagger \doteq \{i : q_{ji} = 0\}$ :
Case 1  $i \in [m]_j^+$ . Let  $r_{ji} \doteq u_{ji} / q_{ji}^{1 - t}$  and suppose  $R_j > 0$  is a real that satisfies
$$
\left| r _ {j i} \right| \leqslant R _ {j}, \forall i \in [ m ] _ {j} ^ {+}. \tag {51}
$$
For any convex function  $f$  defined on  $[-1, 1]$ , we have  $f(z) \leqslant ((1 + z)/2) \cdot f(1) + ((1 - z)/2) \cdot f(-1), \forall z \in [-1, 1]$  (the straight line is the chord crossing  $f$  at  $z = -1, 1$ ). Because
$$
\begin{array}{l} z \mapsto [ 1 - z ] _ {+} ^ {\frac {2 - t}{1 - t}} \text {i s c o n v e x f o r} t \leqslant 2, \text {f o r a n y} i \in [ m ] _ {j} ^ {+} \\ \exp_ {t} ^ {2 - t} \left(\log_ {t} q _ {j i} - \mu u _ {j i}\right) \\ = \left[ q _ {j i} ^ {1 - t} - (1 - t) \mu u _ {j i} \right] _ {+} ^ {\frac {2 - t}{1 - t}} \\ = q _ {j i} ^ {2 - t} \cdot \left[ 1 - (1 - t) \mu R _ {j} \cdot \frac {r _ {j i}}{R _ {j}} \right] _ {+} ^ {\frac {2 - t}{1 - t}} \\ \leqslant q _ {j i} ^ {2 - t} \cdot \frac {R _ {j} + r _ {j i}}{2 R _ {j}} [ 1 - (1 - t) \mu R _ {j} ] _ {+} ^ {\frac {2 - t}{1 - t}} + q _ {j i} ^ {2 - t} \cdot \frac {R _ {j} - r _ {j i}}{2 R _ {j}} [ 1 + (1 - t) \mu R _ {j} ] _ {+} ^ {\frac {2 - t}{1 - t}} \\ = \frac {q _ {j i} ^ {2 - t} R _ {j} + q _ {j i} u _ {j i}}{2 R _ {j}} \left[ 1 - (1 - t) \mu R _ {j} \right] _ {+} ^ {\frac {2 - t}{1 - t}} + \frac {q _ {j i} ^ {2 - t} R _ {j} - q _ {j i} u _ {j i}}{2 R _ {j}} \left[ 1 + (1 - t) \mu R _ {j} \right] _ {+} ^ {\frac {2 - t}{1 - t}}. \\ \end{array}
$$
Case 2  $i \in [m]_j^\dagger$ . Let  $q_j^\dagger > 0$  be a real that satisfies
$$
\frac {\left| u _ {j i} \right|}{q _ {j} ^ {\dagger 1 - t}} <   R _ {j}, \forall i \in [ m ] _ {j} ^ {\dagger}. \tag {52}
$$
Using the same technique as in case 1, we find for any  $i \in [m]_j^\dagger$
$$
\begin{array}{l} \exp_ {t} ^ {2 - t} \left(\log_ {t} q _ {j i} - \mu u _ {j i}\right) \\ = \exp_ {t} ^ {2 - t} \left(- \frac {1}{1 - t} - \mu u _ {j i}\right) \\ = \left[ - (1 - t) \mu u _ {j i} \right] _ {+} ^ {\frac {2 - t}{1 - t}} \\ \leqslant \left[ q _ {j} ^ {\dagger 1 - t} - (1 - t) \mu u _ {j i} \right] _ {+} ^ {\frac {2 - t}{1 - t}} \\ \leqslant \frac {q _ {j} ^ {\dagger 2 ^ {- t}} R _ {j} + q _ {j} ^ {\dagger} u _ {j i}}{2 R _ {j}} \left[ 1 - (1 - t) \mu R _ {j} \right] _ {+} ^ {\frac {2 - t}{1 - t}} + \frac {q _ {j} ^ {\dagger 2 ^ {- t}} R _ {j} - q _ {j} ^ {\dagger} u _ {j i}}{2 R _ {j}} \left[ 1 + (1 - t) \mu R _ {j} \right] _ {+} ^ {\frac {2 - t}{1 - t}}. \\ \end{array}
$$
Folding both cases into one and letting
$$
q _ {j i} ^ {\prime} \doteq \left\{ \begin{array}{l l} q _ {j i} & \text {i f} \quad i \in [ m ] _ {j} ^ {+} \\ q _ {j} ^ {\dagger} & \text {i f} \quad i \in [ m ] _ {j} ^ {\dagger} \end{array} , \right. \tag {53}
$$
we get after summation, using  $m_j^\dagger \doteq \operatorname{Card}([m]_j^\dagger)$  and
$$
\rho_ {j} \quad \doteq \quad \frac {1}{\left(1 + m _ {j} ^ {\dagger} q _ {j} ^ {\dagger 2 - t}\right) R _ {j}} \cdot \sum_ {i \in [ m ]} q _ {j i} ^ {\prime} u _ {j i} \quad (\in [ - 1, 1 ]), \tag {54}
$$
$$
\begin{array}{l} Z _ {t j} ^ {2 - t} (\mu) \\ \leqslant \frac {\left(1 + m _ {j} ^ {\dagger} q _ {j} ^ {\dagger 2 - t}\right) R _ {j}}{2 R _ {j}} \cdot \left(\left(1 + \rho_ {j}\right) \left[ 1 - (1 - t) \mu R _ {j} \right] _ {+} ^ {\frac {2 - t}{1 - t}} + \left(1 - \rho_ {j}\right) \left[ 1 + (1 - t) \mu R _ {j} \right] _ {+} ^ {\frac {2 - t}{1 - t}}\right) \\ = \frac {1 + m _ {j} ^ {\dagger} q _ {j} ^ {\dagger 2 - t}}{2} \cdot \left((1 + \rho_ {j}) \cdot \exp_ {t} ^ {2 - t} (- \mu R _ {j}) + (1 - \rho_ {j}) \cdot \exp_ {t} ^ {2 - t} (\mu R _ {j})\right). \tag {55} \\ \end{array}
$$
Step 2: we have our upperbound for (50). We now compute the minimizer  $\mu^{*}$  of (55). If this minimizer satisfies
$$
\left| \mu^ {*} \right| <   \frac {1}{R _ {j} \left| 1 - t \right|}, \tag {56}
$$
then it can be found by ordinary differentiation, as the solution to
$$
(1 - \rho_ {j}) \cdot \exp_ {t} (\mu^ {*} R _ {j}) - (1 + \rho_ {j}) \cdot \exp_ {t} (- \mu^ {*} R _ {j}) = 0,
$$
which is equivalently
$$
\begin{array}{l} \frac {\exp_ {t} \left(- \mu^ {*} R _ {j}\right)}{\exp_ {t} \left(\mu^ {*} R _ {j}\right)} = \exp_ {t} \left(- \mu^ {*} R _ {j} \ominus_ {t} \mu^ {*} R _ {j}\right) \\ = \frac {1 - \rho_ {j}}{1 + \rho_ {j}}, \\ \end{array}
$$
where we recall  $a\ominus_{t}b\doteq (a - b) / (1 + (1 - t)b)$ . Solving it yields
$$
\begin{array}{l} \mu^ {*} = \frac {1}{R _ {j}} \cdot - \frac {1}{1 - t} \cdot \left(\frac {\left(1 - \rho_ {j}\right) ^ {1 - t} - \left(1 + \rho_ {j}\right) ^ {1 - t}}{\left(1 - \rho_ {j}\right) ^ {1 - t} + \left(1 + \rho_ {j}\right) ^ {1 - t}}\right) \\ = \frac {1}{R _ {j}} \cdot - \frac {1}{1 - t} \cdot \left(\frac {2 (1 - \rho_ {j}) ^ {1 - t}}{(1 - \rho_ {j}) ^ {1 - t} + (1 + \rho_ {j}) ^ {1 - t}} - 1\right) \\ = - \frac {1}{R _ {j}} \cdot \log_ {t} \left(\frac {1 - \rho_ {j}}{M _ {1 - t} (1 - \rho_ {j} , 1 + \rho_ {j})}\right), \\ \end{array}
$$
where  $M_q(a, b) \doteq ((a^q + b^q) / 2)^{1/q}$  is the power mean with exponent  $q$ . We now check (56).
Lemma H. For any  $t\in \mathbb{R}$  , let
$$
\mu_ {j} \stackrel {{\cdot}} {{=}} - \frac {1}{R _ {j}} \cdot \log_ {t} \left(\frac {1 - \rho_ {j}}{M _ {1 - t} \left(1 - \rho_ {j} , 1 + \rho_ {j}\right)}\right). \tag {57}
$$
Then  $|\mu_j| \leqslant 1 / (R_j |1 - t|)$ .
Proof. Equivalently, we must show
$$
\left| \log_ {t} \left(\frac {1 - z}{M _ {1 - t} (1 - z , 1 + z)}\right) \right| \leqslant \frac {1}{| 1 - t |}, \forall z \in [ - 1, 1 ],
$$
which is equivalent to showing
$$
\left| \frac {2 (1 - z) ^ {1 - t}}{(1 - z) ^ {1 - t} + (1 + z) ^ {1 - t}} - 1 \right| \left(= \left| \frac {1 - \left(\frac {1 + z}{1 - z}\right) ^ {1 - t}}{1 + \left(\frac {1 + z}{1 - z}\right) ^ {1 - t}} \right|\right) \leqslant 1, \forall z \in [ - 1, 1 ].
$$
Define function  $f(z,t) \doteq (1 - z^{1 - t}) / (1 + z^{1 - t})$  over  $\mathbb{R}_{\geqslant 0} \times \mathbb{R}$ : it is easy to check that for  $t \leqslant 1$ ,  $f(z,t) \in [-1,1]$ , and the symmetry  $f(z,t) = -f(z,2 - t)$  also allows to conclude that for  $t \geqslant 1$ ,  $f(z,t) \in [-1,1]$ . This ends the proof of Lemma H.
For the expression of  $\mu_{j}$  in (57), we get from (55) the upperbound on  $Z_{t_j}^{2 - t}(\mu_j)$ :
$$
\begin{array}{l} Z _ {t j} ^ {2 - t} (\mu_ {j}) \leqslant \frac {1 + m _ {j} ^ {\dagger} q _ {j} ^ {\dagger 2 - t}}{2} \cdot \left((1 + \rho_ {j}) \cdot \exp_ {t} ^ {2 - t} (- \mu_ {j} R _ {j}) + (1 - \rho_ {j}) \cdot \exp_ {t} ^ {2 - t} (\mu_ {j} R _ {j})\right) \\ = \frac {1 + m _ {j} ^ {\dagger} q _ {j} ^ {\dagger 2 - t}}{2} \cdot \left(\frac {(1 + \rho_ {j}) (1 - \rho_ {j}) ^ {2 - t}}{M _ {1 - t} ^ {2 - t} (1 - \rho_ {j} , 1 + \rho_ {j})} + \frac {(1 - \rho_ {j}) (1 + \rho_ {j}) ^ {2 - t}}{M _ {1 - t} ^ {2 - t} (1 - \rho_ {j} , 1 + \rho_ {j})}\right) \\ = \left(1 + m _ {j} ^ {\dagger} q _ {j} ^ {\dagger 2 - t}\right) \cdot \frac {(1 - \rho_ {j} ^ {2}) M _ {1 - t} ^ {1 - t} (1 - \rho_ {j} , 1 + \rho_ {j})}{M _ {1 - t} ^ {2 - t} (1 - \rho_ {j} , 1 + \rho_ {j})} \\ = \left(1 + m _ {j} ^ {\dagger} q _ {j} ^ {\dagger 2 - t}\right) \cdot \frac {1 - \rho_ {j} ^ {2}}{M _ {1 - t} (1 - \rho_ {j} , 1 + \rho_ {j})}. \\ \end{array}
$$
We conclude that for both sets of classifiers defined in Lemmata F and G, with the choice of  $\mu_{j}$  in (57), we get
$$
\frac {1}{m} \cdot \sum_ {i \in [ m ]} \llbracket \mathrm {s i g n} (H (\pmb {x} _ {i})) \neq y _ {i} \rrbracket \leqslant \prod_ {j = 1} ^ {J} \left(1 + m _ {j} ^ {\dagger} q _ {j} ^ {\dagger^ {2 - t}}\right) \cdot \frac {1 - \rho_ {j} ^ {2}}{M _ {1 - t} (1 - \rho_ {j} , 1 + \rho_ {j})}, \forall H \in \{H _ {J}, H _ {J} ^ {(1 / 1 - t)} \}.
$$
To complete the proof of Theorem 2, we just need to elicit the best  $R_{j}$  (51) and  $q_{j}^{\dagger}$  (52); looking at their constraints suggests
$$
{R _ {j}} \doteq {\max _ {i \notin [ m ] _ {j} ^ {\dagger}} \frac {| y _ {i} h _ {j} (\boldsymbol {x} _ {i}) |}{q _ {j i} ^ {1 - t}},}
$$
$$
q _ {j} ^ {\dagger} \stackrel {{\dagger}} {{=}} \frac {\operatorname* {m a x} _ {i \in [ m ] _ {j} ^ {\dagger}} | y _ {i} h _ {j} (\boldsymbol {x} _ {i}) | ^ {1 / (1 - t)}}{R _ {j} ^ {1 / (1 - t)}}.
$$
This completes the proof of Theorem 2. We complete the proof by two Lemmata of additional useful results in the context of Algorithm  $t$ -ADABOOST, and finally an important remark on the interpretation of Theorem 2.
Lemma I. The following holds true: (i)  $\rho_{j} \in [-1, 1]$ ; (ii) if, among indexes not in  $[m]_j^\dagger$ , there exists at least one index with  $u_{ji} > 0$  and one index with  $u_{ji} < 0$ , then for any  $\mu \neq 0$ ,  $Z_{tj}^{2 - t}(\mu) > 0$  in (50) (in words, the new weigh vector  $\mathbf{q}_{j + 1}$  cannot be the null vector before normalization).
Proof. To show (i) for  $\rho_{j} \leqslant 1$ , we write (using  $u_{ji} \doteq y_i h_j(\pmb{x}_i)$ ,  $\forall i \in [m]$  for short),
$$
\begin{array}{l} {(1 + m _ {j} ^ {\dagger} q _ {j} ^ {\dagger 2 - t}) R _ {j} \cdot \rho_ {j}} = {\sum_ {i \in [ m ]} q _ {j i} ^ {\prime} u _ {j i}} \\ \leqslant \sum_ {i \in [ m ]} q _ {j i} ^ {\prime} | u _ {j i} | \\ = \sum_ {i \in [ m ] _ {j} ^ {+}} q _ {j i} ^ {2 - t} \cdot \frac {| u _ {j i} |}{q _ {j i} ^ {1 - t}} + q _ {j} ^ {\dagger 2 - t} \cdot \frac {\sum_ {i \in [ m ] _ {j} ^ {\dagger}} | u _ {j i} |}{q _ {j} ^ {\dagger 1 - t}} \\ \leqslant R_{j}\cdot \underbrace{\sum_{i\in[m]_{j}^{+}}q_{ji}^{2 - t}}_{= 1} + q_{j}^{\dagger 2 - t}\cdot \frac{R_{j}\sum_{i\in[m]_{j}^{\dagger}}|u_{ji}|}{\max_{i\in[m]_{j}^{\dagger}}|u_{ji}|} \\ \leqslant R _ {j} + q _ {j} ^ {\dagger 2 - t} m _ {j} ^ {\dagger} R _ {j} = (1 + m _ {j} ^ {\dagger} q _ {j} ^ {\dagger 2 - t}) R _ {j}, \\ \end{array}
$$
showing  $\rho_{j} \leqslant 1$ . Showing  $\rho_{j} \geqslant -1$  proceeds in the same way. Property (ii) is trivial.
Lemma J.
$$
K _ {t} (z) \leqslant \exp \left(- \left(1 - \frac {t}{2}\right) \cdot z ^ {2}\right).
$$
Proof. We remark that for  $t \in [0,1)$ ,  $z \geqslant 0$ ,  $K_{t}^{\prime}(z)$  is concave and  $K_{t}^{\prime \prime}(0) = -(2 - t)$ , so  $K_{t}^{\prime}(z) \leqslant -(2 - t)z, \forall z \geqslant 0$ , from which it follows by integration
$$
K _ {t} (z) \leqslant 1 - \left(1 - \frac {t}{2}\right) \cdot z ^ {2},
$$
and since  $1 - z \leqslant \exp(-z)$ , we get the statement of the Lemma.
Remark 1. The interpretation of Theorem 2 for  $t < 1$  are simplified to the case where there is no weight switching, i.e.  $m_j^\dagger = 0, \forall j$ . While we have never observed weight switching in our experiments – perhaps owing to the fact that we did never boost for a very long number of iterations or just because our weak classifiers, decision trees, were in fact not so weak –, it is interesting, from a theoretical standpoint, to comment on convergence when this happens. Let  $Q_j \doteq 1 + m_j^\dagger (q_j^\dagger)^{2 - t}$  and  $\tilde{\rho}_j \doteq Q_j \rho_j$  (Notations from Theorem 2). We note that  $\tilde{\rho}_j \approx \beta \cdot \mathbb{E}_{\pmb{p}_j} [y_i h_j(\pmb{x}_i)]$ , where  $\pmb{p}_j$  lives on the simplex and  $|yh| \leqslant 1, \beta \leqslant 1$ . Using Lemma J and (12) (main file), to keep geometric convergence, it is roughly sufficient that  $Q_j \log Q_j \leqslant (\tilde{\rho}_j)^2 /(2t^*)$ . Since  $q_j^\dagger$  is homogeneous to a tempered weight, one would expect in general  $m_j^\dagger (q_j^\dagger)^{2 - t} \ll 1$ , so using the Taylor approximation  $Q_j \log Q_j \approx -1 + Q_j$ , one gets the refined sufficient condition for geometric convergence
$$
m _ {j} ^ {\dagger} \left(q _ {j} ^ {\dagger}\right) ^ {2 - t} \leqslant \left(\tilde {\rho} _ {j}\right) ^ {2} / \left(2 t ^ {*}\right) = O \left(\left(\tilde {\rho} _ {j}\right) ^ {2}\right).
$$
What does that imply? We have two cases:
- If this holds, then we have geometric convergence;
- if it does not hold, then for a "large" number of training examples, we must have  $q_{ji} = 0$  which, because of the formula for  $\mathbf{q}$  (8) implies that all these examples receive the right class with a sufficiently large margin. Breaking geometric convergence in this case is not an issue: we already have a good ensemble.
II.3 Proof of Theorem 3
Starting from the proof of Theorem 2, we indicate the additional steps to get to the proof of Theorem 3. The key is to remark that our margin formulation has the following logical convenience:
$$
\begin{array}{l} \llbracket \nu_ {t} ((\boldsymbol {x} _ {i}, y _ {i}), H) \leqslant \theta \rrbracket = \llbracket - y H (\boldsymbol {x}) + \log_ {t} \left(\frac {1 + \theta}{1 - \theta}\right) - (1 - t) y H (\boldsymbol {x}) \log_ {t} \left(\frac {1 + \theta}{1 - \theta}\right) \geqslant 0 \rrbracket \\ = \llbracket (- y H (\boldsymbol {x})) \oplus_ {t} \log_ {t} \left(\frac {1 + \theta}{1 - \theta}\right) \geqslant 0 \rrbracket . \\ \end{array}
$$
We then remark that since  $\llbracket z\geqslant 0\rrbracket \leqslant \exp_t^{2 - t}(z)$ , we get
$$
\begin{array}{l} \llbracket \nu_ {t} ((\boldsymbol {x} _ {i}, y _ {i}), H) \leqslant \theta \rrbracket \leqslant \exp_ {t} ^ {2 - t} \left((- y H (\boldsymbol {x})) \oplus_ {t} \log_ {t} \left(\frac {1 + \theta}{1 - \theta}\right)\right) \\ = \exp_ {t} ^ {2 - t} \left(\log_ {t} \left(\frac {1 + \theta}{1 - \theta}\right)\right) \cdot \exp_ {t} ^ {2 - t} (- y H (\boldsymbol {x})) \\ = \left(\frac {1 + \theta}{1 - \theta}\right) ^ {2 - t} \cdot \exp_ {t} ^ {2 - t} (- y H (\boldsymbol {x})). \\ \end{array}
$$
We then just have to branch to (44), replacing the  $\llbracket \mathrm{sign}(H_J(\pmb{x}_i))\neq y_i\rrbracket$  by  $\llbracket \nu_t((\pmb{x}_i,y_i),H)\leqslant \theta \rrbracket$  which yields in lieu of (46) the sought inequality:
$$
F _ {t, \theta} (H, \mathbb {S}) \leqslant \left(\frac {1 + \theta}{1 - \theta}\right) ^ {2 - t} \prod_ {j = 1} ^ {J} Z _ {t j} ^ {2 - t}. \tag {58}
$$
II.4 Proof of Theorem 4
The proof proceeds in three parts. Part (A) makes a brief recall on encoding linear classifiers with decision trees. Part (B) solves (6) in MF, i.e. finds boosting's leveraging coefficients as solution of:
$$
\boldsymbol {q} (\mu) ^ {\top} \boldsymbol {u} = 0. \tag {59}
$$
we then simplify the loss obtained and elicit the conditional Bayes risk of the tempered loss, i.e. (20) in MF. Part (C) elicits the partial losses and shows properness and related properties.