SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[26] M.-D. Reid and R.-C. Williamson. Information, divergence and risk for binary experiments. JMLR, 12:731-817, 2011.
[27] C. Rudin, I. Daubechies, and R.-E. Schapire. Dynamics of adaboost: cyclic behavior and convergence of margins. JMLR, pages 1557-1595, December 2004.
[28] L.-J. Savage. Elicitation of personal probabilities and expectations. J. of the Am. Stat. Assoc., pages 783-801, 1971.
[29] R. E. Schapire. The strength of weak learnability. MLJ, pages 197-227, 1990.
[30] R. E. Schapire and Y. Singer. Improved boosting algorithms using confidence-rated predictions. MLJ, 37:297-336, 1999.
[31] T. Sypherd, R. Nock, and L. Sankar. Being properly improper. In  $39^{th}$  ICML, 2022.
[32] M. Telgarsky. A primal-dual convergence analysis of boosting. JMLR, 13:561-606, 2012.
[33] M. Telgarsky. Boosting with the logistic loss is consistent. In  $26^{th}$  COLT, pages 911-965, 2013.
[34] L. G. Valiant. A theory of the learnable. Communications of the ACM, 27:1134-1142, 1984.
[35] M.-K. Warmuth, K.-A. Glocker, and S.-V.-N. Vishwanathan. Entropy regularized LPBoost. In Algorithmic Learning Theory, pages 256-271. Springer Berlin Heidelberg, 2008.
[36] M.-K. Warmuth, J. Liao, and G. Ratsch. Totally corrective boosting algorithms that maximize the margin. In icml '06: proceedings of the 23rd international conference on machine learning, pages 1001-1008, 2006.
This is the Appendix to Paper "Boosting with Tempered Exponential Measures". To differentiate with the numberings in the main file, the numbering of Theorems, Lemmata, Definitions is letter-based (A, B, ...).
A short primer on Tempered Exponential Measures Pg 14
Supplementary material on proofs Pg 15
$\hookrightarrow$  Proof of Theorem 1  $\mathrm{Pg}15$
$\hookrightarrow$  Proof of Theorem 2  $\mathrm{Pg}18$
$\hookrightarrow$  Proof of Theorem 3  $\mathrm{Pg}27$
$\hookrightarrow$  Proof of Theorem 4  $\mathrm{Pg}27$
Supplementary material on experiments Pg 32
I A short primer on Tempered Exponential Measures
We describe here the minimal amount of material necessary to understand how our approach to boosting connects to these measures. We refer to [4] for more details. With a slight abuse of notation, we define the perspective transforms  $(\log_t)^* (z)\doteq t^*\cdot \log_{t^*}(z / t^*)$  and  $(\exp_t)^* (z)\doteq t^*\cdot \exp_{t^*}(z / t^*)$ . Recall that  $t^{*}\doteq 1 / (2 - t)$ .
Definition A. [4] A tempered exponential measure (TEM) family is a set of unnormalized densities in which each element admits the following canonical expression:
$$
q _ {t | \boldsymbol {\theta}} (\boldsymbol {x}) \doteq \frac {\exp_ {t} \left(\boldsymbol {\theta} ^ {\top} \boldsymbol {\varphi} (\boldsymbol {x})\right)}{\exp_ {t} \left(G _ {t} (\boldsymbol {\theta})\right)} = \exp_ {t} \left(\boldsymbol {\theta} ^ {\top} \boldsymbol {\varphi} (\boldsymbol {x}) \ominus_ {t} G _ {t} (\boldsymbol {\theta})\right) \quad \left(a \ominus_ {t} b \doteq \frac {a - b}{1 + (1 - t) b}\right), \tag {21}
$$
where  $\pmb{\theta}$  is the element's natural parameter,  $\varphi (\pmb {x})$  is the sufficient statistics and
$$
G _ {t} (\boldsymbol {\theta}) = \left(\log_ {t}\right) ^ {*} \int \left(\exp_ {t}\right) ^ {*} \left(\boldsymbol {\theta} ^ {\top} \boldsymbol {\varphi} (\boldsymbol {x})\right) \mathrm {d} \xi
$$
is the (convex) cumulant,  $\xi$  being a base measure (implicit).
Except for  $t = 1$  (which reduces a TEM family to a classical exponential family), the total mass of a TEM is not 1 (but it has an elegant closed form expression [4]). However, the exponentiated  $q_{t|\theta}^{1 / t^*}$  does sum to 1. In the discrete case, this justifies extending the classical simplex to what we denote as the co-simplex.
Definition B. The co-simplex of  $\mathbb{R}^m$ ,  $\tilde{\Delta}_m$  is defined as  $\tilde{\Delta}_m \doteq \{ \pmb{q} \in \mathbb{R}^m : \pmb{q} \geqslant 0 \wedge \mathbf{1}^\top \pmb{q}^{1 / t^*} = 1 \}$ .
The connection between  $t$ -ADABOOST's update and TEM's is immediate from the equation's update ((4) in MF). We can show that  $\tilde{\Delta}_m$  can also be represented as TEMs.
Lemma A.  $\tilde{\Delta}_m$  is a (discrete) family of tempered exponential measures.
Proof. We proceed as in [2, Section 2.2.2] for exponential families: let  $\pmb{q} \in \tilde{\Delta}_m$ , which we write
$$
q (n) \quad = \sum_ {i \in [ m ]} q _ {i} \cdot [   [ i = n ]   ], n \in [ m ]. \tag {22}
$$
$\llbracket \pi \rrbracket$ , the Iverson bracket [13], takes value 1 if Boolean predicate  $\pi$  is true (and 0 otherwise). We create  $m - 1$  natural parameters and the cumulant,
$$
\theta_ {i} \doteq \log_ {t} \frac {q _ {i}}{q _ {m}}, i \in [ m - 1 ]; G _ {t} (\pmb {\theta}) \doteq \log_ {t} \frac {1}{q _ {m}},
$$
and end up with (22) also matching the atom mass function
$$
{ q ( n ) } { = } { \frac { \exp _ { t } \left( \sum _ { i \in [ m - 1 ] } \theta _ { i } \cdot [   [ i = n ]   ] \right) } { \exp _ { t } G _ { t } ( \pmb { \theta } ) } , }
$$
which clearly defines a tempered exponential measure over  $[m]$ . This ends the proof of Lemma A.
II Supplementary material on proofs
II.1 Proof of Theorem 1
To improve readability, we remove dependency in  $t$  in normalization coefficient  $Z$ . We use notations from [4, proof of Theorem 3.2] and denote the Lagrangian
$$
\mathcal {L} = \Delta (\tilde {\boldsymbol {q}} \| \boldsymbol {q}) + \lambda \left(\sum_ {i} \tilde {q} _ {i} ^ {1 / t ^ {*}} - 1\right) - \sum_ {i} \nu_ {i} \tilde {q} _ {i} + \mu \sum_ {i} \tilde {q} _ {i} u _ {i}, \tag {23}
$$
which yields  $\partial \mathcal{L} / \partial \tilde{q}_i = \log_t\tilde{q}_i - \log_tq_i + \lambda \tilde{q}_i^{1 - t} - \nu_i + \mu u_i$  ( $\lambda$  absorbs factor  $2 - t$ ), and, rearranging (absorbing factor  $1 - t$  in  $\nu_{i}$ )
$$
(1 + (1 - t) \lambda) \tilde {q} _ {i} ^ {1 - t} = \nu_ {i} + 1 + (1 - t) \left(\log_ {t} q _ {i} - \mu u _ {i}\right), \forall i \in [ m ]. \tag {24}
$$
We see that  $\lambda \neq -1 / (1 - t)$  otherwise the Lagrangian drops its dependence in the unknown. In fact, the solution necessarily has  $1 + (1 - t)\lambda > 0$ . To see this, we distinguish two cases: (i) if some  $u_{k} = 0$ , then since  $\log_t q_k \geqslant -1 / (1 - t)$  there would be no solution to (24) if  $1 + (1 - t)\lambda < 0$  because of the KKT conditions  $\nu_i \geqslant 0, \forall i \in [m]$ ; (ii) otherwise, if all  $u_{k} \neq 0, \forall k \in [m]$ , then there must be two coordinates of different signs otherwise there is no solution to our problem (3) (main file, we must have indeed  $\tilde{\pmb{q}} \geqslant 0$  because of the co-simplex constraint). Thus, there exists at least one coordinate  $k \in [m]$  for which  $-(1 - t)\mu u_k > 0$  and since  $\log_t q_k \geqslant -1 / (1 - t)$  (definition of  $\log_t$ ) and  $\nu_k \geqslant 0$  (KKT conditions), the RHS of (24) for  $i = k$  is  $> 0$ , preventing  $1 + (1 - t)\lambda < 0$  in the LHS.
We thus have  $1 + (1 - t)\lambda > 0$ . The KKT conditions  $(\nu_{i} \geqslant 0, \nu_{i}\tilde{q}_{i} = 0, \forall i \in [m])$  yield the following:  $1 + (1 - t)(\log_{t}q_{i} - \mu u_{i}) > 0$  imply  $\nu_{i} = 0$  and  $1 + (1 - t)(\log_{t}q_{i} - \mu u_{i}) \leqslant 0$  imply  $\tilde{q}_{i}^{1 - t} = 0$  so we get the necessary form for the optimum:
$$
\begin{array}{l} \tilde {q} _ {i} = \frac {\exp_ {t} \left(\log_ {t} q _ {i} - \mu u _ {i}\right)}{\exp_ {t} \lambda} \\ = \frac {q _ {i} \otimes_ {t} \exp_ {t} (- \mu u _ {i})}{Z _ {t}}, \tag {25} \\ \end{array}
$$
where  $\lambda$  or  $Z_{t} \doteq \exp_{t}\lambda$  ensures normalisation for the co-density. Note that we have a simplified expression for the co-density:
$$
p _ {i} = \frac {p _ {j i} \otimes_ {t *} \exp_ {t *} (- \mu u _ {i} / t ^ {*})}{Z _ {t} ^ {\mathrm {c o}}}, \tag {26}
$$
with  $Z_{t}^{\mathrm{co}} \doteq Z_{t}^{1/t^{*}} = \sum_{i} p_{ji} \otimes_{t^{*}} \exp_{t^{*}}(-\mu u_{i}/t^{*})$ . For the analytic form in (25), we can simplify the Lagrangian to a dual form that depends on  $\mu$  solely:
$$
\mathcal {D} (\mu) = \Delta (\tilde {\boldsymbol {q}} (\mu) \| \boldsymbol {q}) + \mu \sum_ {i} \tilde {q} _ {i} (\mu) u _ {i}. \tag {27}
$$
The proof of (5) (main file) is based on a key Lemma.
Lemma B. For any  $\tilde{\pmb{q}}$  having form (25) such that  $\tilde{\pmb{q}}^{\top}\pmb {u} = 0$ ,  $\mathcal{D}(\mu) = -\log_tZ_t(\mu)$ .
Proof. For any  $\tilde{\pmb{q}}$  having form (25), denote
$$
[ m ] _ {*} \quad \doteq \quad \{i: \tilde {q} _ {i} \neq 0 \}. \tag {28}
$$
We first compute (still using  $\lambda \doteq \log_t Z_t(\mu)$  for short):
$$
\begin{array}{l} A \quad \doteq \quad \sum_ {i} \tilde {q} _ {i} \cdot \log_ {t} \tilde {q} _ {i} \\ = \sum_ {i \in [ m ] _ {*}} \tilde {q} _ {i} \cdot \log_ {t} \left(\frac {\exp_ {t} \left(\log_ {t} q _ {i} - \mu u _ {i}\right)}{\exp_ {t} \lambda}\right) \\ = \sum_ {i \in [ m ] _ {*}} \tilde {q} _ {i} \cdot \left(\frac {1}{1 - t} \cdot \left[ \frac {1 + (1 - t) (\log_ {t} q _ {i} - \mu u _ {i})}{1 + (1 - t) \lambda} - 1 \right]\right) \\ = \frac {1}{1 - t} \cdot \sum_ {i \in [ m ] _ {*}} \tilde {q} _ {i} \cdot \left(\frac {q _ {i} ^ {1 - t} - (1 - t) \mu u _ {i}}{1 + (1 - t) \lambda}\right) - \frac {1}{1 - t} \cdot \sum_ {i \in [ m ] _ {*}} \tilde {q} _ {i} \\ = - \frac {\mu}{1 + (1 - t) \lambda} \cdot \sum_ {i \in [ m ] _ {*}} \tilde {q} _ {i} u _ {i} + \frac {1}{(1 - t) (1 + (1 - t) \lambda)} \cdot \sum_ {i \in [ m ] _ {*}} \tilde {q} _ {i} q _ {i} ^ {1 - t} - \frac {1}{1 - t} \cdot \sum_ {i \in [ m ] _ {*}} \tilde {q} _ {i} \\ = \underbrace {- \frac {\mu}{1 + (1 - t) \lambda} \cdot \tilde {\boldsymbol {q}} ^ {\top} \boldsymbol {u}} _ {\doteq B} + \underbrace {\frac {1}{(1 - t) (1 + (1 - t) \lambda)} \cdot \sum_ {i \in [ m ]} \tilde {q} _ {i} q _ {i} ^ {1 - t}} _ {\doteq C} - \underbrace {\frac {1}{1 - t} \cdot \sum_ {i \in [ m ]} \tilde {q} _ {i}} _ {\doteq D}. \tag {29} \\ \end{array}
$$
Remark that in the last identity, we have put back summations over the complete set  $[m]$  of indices. We note that  $B = 0$  because  $\tilde{\pmb{q}}^{\top}\pmb {u} = 0$ . We then remark that without replacing the expression of  $\tilde{\pmb{q}}$ , we have in general for any  $\tilde{\pmb{q}}\in \tilde{\Delta}_m$ :
$$
\begin{array}{l} E \quad \doteq \quad \sum_ {i \in [ m ]} \tilde {q} _ {i} \cdot \left(\log_ {t} \tilde {q} _ {i} - \log_ {t} q _ {i}\right) \\ = \sum_ {i \in [ m ]} \tilde {q} _ {i} \cdot \left(\frac {1}{1 - t} \cdot \left(\tilde {q} _ {i} ^ {1 - t} - 1\right) - \frac {1}{1 - t} \cdot \left(q _ {i} ^ {1 - t} - 1\right)\right) \\ = \frac {1}{1 - t} \cdot \sum_ {i \in [ m ]} \tilde {q} _ {i} ^ {2 - t} - \frac {1}{1 - t} \cdot \sum_ {i \in [ m ]} \tilde {q} _ {i} q _ {i} ^ {1 - t} \\ = \frac {1}{1 - t} \cdot \left(1 - \sum_ {i \in [ m ]} \tilde {q} _ {i} q _ {i} ^ {1 - t}\right), \\ \end{array}
$$
and we can check that for any  $\tilde{\pmb{q}},\pmb {q}\in \tilde{\Delta}_m,E = \Delta (\tilde{\pmb{q}}\| \pmb {q})$  . We then develop  $\Delta (\tilde{\pmb{q}}\| \pmb {q})$  with a partial replacement of  $\tilde{\pmb{q}}$  by its expression:
$$
\begin{array}{l} \Delta (\tilde {\boldsymbol {q}} \| \boldsymbol {q}) = A - \sum_ {i} \tilde {q} _ {i} \log_ {t} q _ {i} \\ = A - \frac {1}{1 - t} \cdot \sum_ {i} \tilde {q} _ {i} q _ {i} ^ {1 - t} + \frac {1}{1 - t} \cdot \sum_ {i} \tilde {q} _ {i} \\ = C - \frac {1}{1 - t} \cdot \sum_ {i} \tilde {q} _ {i} q _ {i} ^ {1 - t} \\ = \frac {1}{1 - t} \cdot \left(\frac {1}{1 + (1 - t) \lambda} - 1\right) \cdot \sum_ {i} \tilde {q} _ {i} q _ {i} ^ {1 - t} \\ = - \frac {\lambda}{1 + (1 - t) \lambda} \cdot \sum_ {i} \tilde {q} _ {i} q _ {i} ^ {1 - t} \\ = - \frac {\lambda}{1 + (1 - t) \lambda} \cdot (1 - (1 - t) \cdot \Delta (\tilde {\boldsymbol {q}} \| \boldsymbol {q})) \\ \end{array}
$$
Rearranging gives that for any  $\tilde{\pmb{q}},\pmb {q}\in \tilde{\Delta}_m$  such that (i)  $\tilde{\pmb{q}}$  has the form (25) for some  $\mu \in \mathbb{R}$  and (ii)  $\tilde{\pmb{q}}^{\top}\pmb {u} = 0$
$$
\begin{array}{l} \Delta (\tilde {\boldsymbol {q}} \| \boldsymbol {q}) = - \lambda \\ = - \log_ {t} (Z _ {t}), \\ \end{array}
$$
as claimed. This ends the proof of Lemma B.
We thus get from the definition of the dual that  $\mu = \arg \max -\log_tZ_t(\mu) = \arg \min Z_t(\mu)$ . We have the explicit form for  $Z_{t}$ :
$$
\begin{array}{l} Z _ {t} (\mu) = \left(\sum_ {i} \exp_ {t} ^ {2 - t} \left(\log_ {t} q _ {i} - \mu u _ {i}\right)\right) ^ {\frac {1}{2 - t}} \\ = \left(\sum_ {i \in [ m ] _ {*}} \exp_ {t} ^ {2 - t} \left(\log_ {t} q _ {i} - \mu u _ {i}\right)\right) ^ {\frac {1}{2 - t}}, \\ \end{array}
$$
where  $[m]_{*}$  is defined in (28). We remark that the last expression is differentiable in  $\mu$ , and get
$$
\begin{array}{l} Z _ {t} ^ {\prime} (\mu) = \frac {1}{2 - t} \cdot \left(\sum_ {i \in [ m ] _ {*}} \exp_ {t} ^ {2 - t} \left(\log_ {t} q _ {i} - \mu u _ {i}\right)\right) ^ {- \frac {1 - t}{2 - t}} \\ \cdot (2 - t) \sum_ {i \in [ m ] _ {*}} \exp_ {t} ^ {1 - t} (\log_ {t} q _ {i} - \mu u _ {i}) \cdot \exp_ {t} ^ {t} (\log_ {t} q _ {i} - \mu u _ {i}) \cdot - u _ {i} \\ = - Z _ {t} ^ {t - 1} \cdot \sum_ {i \in [ m ] _ {*}} \exp_ {t} \left(\log_ {t} q _ {i} - \mu u _ {i}\right) \cdot u _ {i} \\ { = } { - Z _ { t } ^ { t } \cdot \sum _ { i \in [ m ] _ { * } } \tilde { q } _ { i } u _ { i } } \\ = - Z _ {t} ^ {t} \cdot \sum_ {i \in [ m ]} \tilde {q} _ {i} u _ {i} \\ = - Z _ {t} ^ {t} \cdot \tilde {\boldsymbol {q}} ^ {\top} \boldsymbol {u}, \tag {30} \\ \end{array}
$$
$$
\begin{array}{l} \frac {\partial - \log_ {t} (Z _ {t})}{\partial \mu} = - Z _ {t} ^ {- t} Z _ {t} ^ {\prime} \\ = \tilde {\boldsymbol {q}} (\mu) ^ {\top} \boldsymbol {u}, \\ \end{array}
$$
and we get that any critical point of  $Z_{t}(\mu)$  satisfies  $\tilde{\pmb{q}} (\mu)^{\top}\pmb {u} = 0$ . A sufficient condition to have just one critical point, being the minimum sought is the strict convexity of  $Z_{t}(\mu)$ . The next Lemma provides the proof that it is for all  $t > 0$ .
Lemma C.  $Z_{t}^{\prime \prime}(\mu)\geqslant t\cdot Z_{t}(\mu)^{2t - 1}(\tilde{\pmb{q}} (\mu)^{\top}\pmb {u})^{2}.$
Proof. After simplifications, we have
$$
\begin{array}{l} Z _ {t} ^ {3 - 2 t} \cdot Z _ {t} ^ {\prime \prime} = (t - 1) \cdot \left(\sum_ {i \in [ m ]} \exp_ {t} \left(\log_ {t} q _ {i} - \mu u _ {i}\right) \cdot u _ {i}\right) ^ {2} (31) \\ + \left(\sum_ {i \in [ m ]} \exp_ {t} ^ {2 - t} \left(\log_ {t} q _ {i} - \mu u _ {i}\right)\right) \cdot \left(\sum_ {i \in [ m ]} \exp_ {t} ^ {t} \left(\log_ {t} q _ {i} - \mu u _ {i}\right) \cdot u _ {i} ^ {2}\right) (32) \\ = (t - 1) \cdot \sum_ {i, k \in [ m ]} Q _ {i} Q _ {k} u _ {i} u _ {k} + \sum_ {i, k \in [ m ]} Q _ {i} ^ {2 - t} Q _ {k} ^ {t} u _ {k} ^ {2}, (33) \\ \end{array}
$$
where we have let  $Q_{i} \doteq \exp_{t}\left(\log_{t} q_{i} - \mu u_{i}\right) \geqslant 0$ . Since  $a^2 + b^2 \geqslant 2ab$ , we note that for any  $i \neq k$ ,
$$
\begin{array}{l} Q _ {i} ^ {2 - t} Q _ {k} ^ {t} u _ {k} ^ {2} + Q _ {k} ^ {2 - t} Q _ {i} ^ {t} u _ {i} ^ {2} \geqslant 2 \sqrt {Q _ {i} ^ {2 - t} Q _ {k} ^ {t} Q _ {k} ^ {2 - t} Q _ {i} ^ {t}} u _ {i} u _ {k} \\ = 2 Q _ {i} Q _ {k} u _ {i} u _ {k}, \tag {34} \\ \end{array}
$$
so we split (33) in two terms and get
$$
\begin{array}{l} Z _ {t} ^ {3 - 2 t} \cdot Z _ {t} ^ {\prime \prime} = (t - 1) \cdot \sum_ {i \in [ m ]} Q _ {i} ^ {2} u _ {i} ^ {2} + \sum_ {i \in [ m ]} Q _ {i} ^ {2 - t} Q _ {i} ^ {t} u _ {i} ^ {2} \\ + \sum_ {i, k \in [ m ], i <   k} 2 (t - 1) Q _ {i} Q _ {k} u _ {i} u _ {k} + \sum_ {i, k \in [ m ], i <   k} Q _ {i} ^ {2 - t} Q _ {k} ^ {t} u _ {k} ^ {2} + Q _ {k} ^ {2 - t} Q _ {i} ^ {t} u _ {i} ^ {2} \\ = t \cdot \sum_ {i \in [ m ]} Q _ {i} ^ {2} u _ {i} ^ {2} \\ + \sum_ {i, k \in [ m ], i <   k} 2 (t - 1) Q _ {i} Q _ {k} u _ {i} u _ {k} + \sum_ {i, k \in [ m ], i <   k} Q _ {i} ^ {2 - t} Q _ {k} ^ {t} u _ {k} ^ {2} + Q _ {k} ^ {2 - t} Q _ {i} ^ {t} u _ {i} ^ {2} \\ \geqslant t \cdot \sum_ {i \in [ m ]} Q _ {i} ^ {2} u _ {i} ^ {2} + 2 t \cdot \sum_ {i, k \in [ m ], i <   k} Q _ {i} Q _ {k} u _ {i} u _ {k} (35) \\ = t \cdot \left(\sum_ {i \in [ m ]} \exp_ {t} \left(\log_ {t} q _ {i} - \mu u _ {i}\right) \cdot u _ {i}\right) ^ {2} \\ = t Z _ {t} ^ {2} \cdot \left(\tilde {\boldsymbol {q}} ^ {\top} \boldsymbol {u}\right) ^ {2}, (36) \\ \end{array}
$$
where we have used (34) in (35). Since  $Z_{t}(\mu) > 0$ , we get the statement of Lemma C after reorganising (36).
Lemma C shows the strict convexity of  $Z_{t}(\mu)$  for any  $t > 0$ . The case  $t = 0$  follows by direct differentiation: we get after simplification
$$
{Z _ {t} ^ {\prime \prime} (\mu)} = {\frac {\left(\sum_ {i \in [ m ]} u _ {i} ^ {2}\right) \cdot \left(\sum_ {i \in [ m ]} (q _ {i} - \mu u _ {i}) ^ {2}\right) - \left(\sum_ {i \in [ m ]} (q _ {i} - \mu u _ {i}) u _ {i}\right) ^ {2}}{\left(\sum_ {i \in [ m ]} (q _ {i} - \mu u _ {i}) ^ {2}\right) ^ {\frac {3}{2}}}.}
$$
Cauchy-Schwartz inequality allows to conclude that  $Z_{t}^{\prime \prime}(\mu) \geqslant 0$  and is in fact  $> 0$  unless  $\tilde{\pmb{q}}$  is collinear to  $\pmb{u}$ . This completes the proof of Theorem 1.
II.2 Proof of Theorem 2
The proof involves several arguments, organized into several subsections. Some are more general than what is strictly needed for the proof of the Theorem, on purpose.
II.II.2.1 Clipped summations
For any  $\delta \geqslant 0$ , we define clipped summations of the sequence of ordered elements  $v_{1}, v_{2}, \ldots, v_{J}$ : if  $J > 1$ ,
$$
{ } ^ { ( \delta ) } \sum _ { j = 1 } ^ { J } v _ { j } \doteq \operatorname* { m i n } \left\{ v _ { J } + \sum _ { j = 1 } ^ { ( \delta ) J - 1 } v _ { j } , \delta \right\} , \quad \sum _ { ( - \delta ) } ^ { J } v _ { j } \doteq \operatorname* { m a x } \left\{ v _ { J } + \sum _ { j = 1 } ^ { J - 1 } v _ { j } , - \delta \right\} , \tag {37}
$$
and the base case  $(J = 1)$  is obtained by replacing the inner sum by 0. We also define the doubly clipped summation:
$$
\sum_ {(- \delta)} ^ {(\delta)} v _ {j} \doteq \max  \left\{\min  \left\{v _ {J} + \sum_ {(- \delta)} ^ {(\delta)} v _ {j}, \delta \right\}, - \delta \right\},
$$
with the same convention for the base case. We prove a series of simple but useful properties of the clipped summation.
Lemma D. The following properties hold true for clipped summation:
1. (doubly) clipped summations are noncommutative;
2. (doubly) clipped summations are ordinary summation in the limit: for any  $J \geqslant 1$  and any sequence  $v_{1}, v_{2}, \ldots, v_{J}$ ,
$$
\lim  _ {\delta \rightarrow + \infty} \sum_ {j = 1} ^ {J} v _ {j} = \lim  _ {\delta \rightarrow + \infty} \sum_ {(- \delta) j = 1} ^ {J} v _ {j} = \lim  _ {\delta \rightarrow + \infty} \sum_ {(- \delta) j = 1} ^ {(J)} v _ {j} = \sum_ {j = 1} ^ {J} v _ {j}
$$
3. clipped summations sandwich ordinary summation and the doubly clipped summation: for any  $\delta \geqslant 0$ , any  $J \geqslant 1$  and any sequence  $v_{1}, v_{2}, \ldots, v_{J}$
$$
\sum_ {j = 1} ^ {J} v _ {j} \leqslant \sum_ {j = 1} ^ {J} v _ {j} \leqslant \sum_ {(- \delta) j = 1} ^ {J} v _ {j}; \quad \sum_ {j = 1} ^ {J} v _ {j} \leqslant \sum_ {(- \delta) j = 1} ^ {(\delta)} v _ {j} \leqslant \sum_ {(- \delta) j = 1} ^ {J} v _ {j}
$$
Proof. Noncommutativity follows from simple counterexamples: for example, for  $v = -1$  and  $w \doteq 2$ , if we fix  $v_{1} \doteq v, v_{2} \doteq w$ , then  $\sum_{j=1}^{2} v_{j} = 1$  while  $\sum_{j=1}^{2} v_{3-j} = -1$ . Property [2.] is trivial. The set of leftmost inequalities of property [3.] can be shown by induction, noting the base case is trivial and otherwise, using the induction hypothesis in the leftmost inequality,
$$
\sum_ {j = 1} ^ {(\delta) \sum_ {j = 1} ^ {J + 1}} v _ {j} \doteq \min \left\{v _ {J + 1} + \sum_ {j = 1} ^ {(\delta) \sum_ {j = 1} ^ {J}} v _ {j}, \delta \right\} \leqslant \min \left\{v _ {J + 1} + \sum_ {j = 1} ^ {J} v _ {j}, \delta \right\} \leqslant v _ {J + 1} + \sum_ {j = 1} ^ {J} v _ {j} = \sum_ {j = 1} ^ {J + 1} v _ {j},
$$
$$
\begin{array}{l} \sum_ {(- \delta) j = 1} ^ {J + 1} v _ {j} \quad \doteq \quad \max  \left\{v _ {J + 1} + \sum_ {(- \delta) j = 1} ^ {J} v _ {j}, - \delta \right\} \\ \geqslant \max  \left\{v _ {J + 1} + \sum_ {j = 1} ^ {J} v _ {j}, - \delta \right\} \geqslant v _ {J + 1} + \sum_ {j = 1} ^ {J} v _ {j} = \sum_ {j = 1} ^ {J + 1} v _ {j}. \\ \end{array}
$$
A similar argument holds for the set of rightmost inequalities: for example, the induction's general case holds
$$
\begin{array}{l} { } ^ { ( \delta ) } \sum _ { j = 1 } ^ { J + 1 } v _ { j } \quad \doteq \quad \min  \left\{ v _ { J + 1 } + { } ^ { ( \delta ) } \sum _ { j = 1 } ^ { J } v _ { j } , \delta \right\} \\ \leqslant \min  \left\{v _ {J + 1} + \sum_ {(- \delta)   j = 1} ^ {(\delta)} v _ {j}, \delta \right\} \\ \leqslant \max  \left\{\min  \left\{v _ {J + 1} + \begin{array}{c} (\delta) \\ (- \delta) \end{array} \sum_ {j = 1} ^ {J} v _ {j}, \delta \right\}, - \delta \right\} = \begin{array}{c} (\delta) \\ (- \delta) \end{array} \sum_ {j = 1} ^ {J} v _ {j}. \\ \end{array}
$$
for the leftmost inequality. This ends the proof of Lemma D.
II. II.2.2 Unravelling weights
Lemma E. Define
$$
v _ {j} \quad \doteq \quad m ^ {1 - t ^ {*}} \cdot \left(\prod_ {k = 1} ^ {j - 1} Z _ {t k}\right) ^ {1 - t} \cdot \mu_ {j} \quad \left(\text {c o n v e n t i o n}: \prod_ {k = 1} ^ {0} u _ {k} \doteq 1\right). \tag {38}
$$
Then  $\forall J\geqslant 1$  , weights unravel as:
$$
q _ {(J + 1) i} = \left\{ \begin{array}{l l} \frac {1}{m ^ {t *} \prod_ {j = 1} ^ {J} Z _ {t j}} \cdot \exp_ {t} \left(- ^ {(1 / 1 - t)} \sum_ {j = 1} ^ {J} v _ {j} u _ {j i}\right) & i f \quad t <   1 \\ \frac {1}{m ^ {t *} \prod_ {j = 1} ^ {J} Z _ {t j}} \cdot \exp_ {t} \left(- _ {(- 1 / 1 - t)} \sum_ {j = 1} ^ {J} v _ {j} u _ {j i}\right) & i f \quad t > 1 \end{array} \right..
$$
Proof. We start for the case  $t < 1$ . We proceed by induction, noting first that the normalization constraint for the initial weights imposes  $q_{1i} = 1 / m^{1 / (2 - t)} = 1 / m^{t^*}$  and so (using  $(1 - t)t^{*} = 1 - t^{*}$ )
$$
\begin{array}{l} {q _ {2 i}} = {\frac {\exp_ {t} (\log_ {t} q _ {1 i} - \mu_ {1} u _ {1 i})}{Z _ {1}}} \\ = \frac {1}{Z _ {1}} \cdot \left[ 1 + (1 - t) \cdot \left(\frac {1}{1 - t} \cdot \left(\frac {1}{m ^ {\frac {1 - t}{2 - t}}} - 1\right) - \mu_ {1} u _ {1 i}\right) \right] _ {+} ^ {\frac {1}{1 - t}} \\ = \frac {1}{Z _ {1}} \cdot \left[ \frac {1}{m ^ {1 - t ^ {*}}} - (1 - t) \mu_ {1} u _ {1 i} \right] _ {+} ^ {\frac {1}{1 - t}} \\ = \frac {1}{m ^ {t *} Z _ {1}} \cdot \left[ 1 - (1 - t) m ^ {1 - t *} \mu_ {1} u _ {1 i} \right] _ {+} ^ {\frac {1}{1 - t}} \\ = \frac {1}{m ^ {t ^ {*}} Z _ {1}} \cdot \exp_ {t} \left(- \sum_ {j = 1} ^ {(1 / 1 - t)} v _ {j} u _ {j i}\right), \\ \end{array}
$$
completing the base case. Using the induction hypothesis, we unravel at iteration  $J + 1$ :