SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 2 TUNER(theoretically tractable version
1: Input:  $\beta \in [0,1]$ ,  $W_0$ : initial "wealth".  
2:  $v_0 \gets 0 \in \mathbb{R}$ ,  $r_0 \gets 0 \in \mathbb{R}$ ,  $\mathbf{q}_t \gets 0 \in \mathbb{R}$ ,  $m_0 \gets 0 \in \mathbb{R}$ ,  $s_1 \gets 0 \in \mathbb{R}$   
3: for  $t = 1 \dots T$  do  
4: Output  $s_t$ .  
5: Receive  $h_t$ .  
6:  $m_t \gets \max(\beta m_{t-1}, h_t)$   
7:  $\hat{h}_t = \text{clip}(h_t, -m_{t-1}, m_{t-1})$  // Red text are steps that we omit in practice (see Algorithm 1)  
8:  $\mathbf{q}_t \gets \beta \mathbf{q}_{t-1} + \hat{h}_t$   
9:  $r_t \gets \beta r_{t-1} - s_{t-1} \hat{h}_t$   
10:  $W_t \gets W_0 + r_t$  // Blue text is changed in practice (see Algorithm 1)  
11:  $v_t \gets \beta^2 v_{t-1} + \hat{h}_t^2$   
12:  $s_{t+1} \gets \frac{W_t}{\sqrt{4m_t^2 + v_t} + \epsilon} \cdot \text{clip}(\mathbf{q}_t / \sqrt{4m_t^2 + v_t}, 0,1)$   
13: end for
First, we conjecture that the clip operation using  $\mathbf{q}_t / \sqrt{v_t}$  may even be unnecessary in theory: we observed no change from removing this operation in practice, and observe that the update has an intuitive interpretation via the Taylor expansion discussed in Section 3.
Second, the clip operation on  $h_t$  using  $m_{t-1}$  is essentially designed to prevent the wealth  $W_t$  from becoming negative or zero using the gradient truncation technique employed by [44, 45, 12]. While less consistent with known theory, we found it simpler to ensure the wealth  $W_t$  does become negative simply by clipping  $r_t$  directly (we did not clip  $W_t$  to be nonnegative as  $W_t = 0$  would cause the algorithm to output  $s_t = 0$  for all future iterations). We found these changes simplified the algorithm while having no noticeable effect on the performance. Although these deviations technically do not come with guarantees, the accomplish similar intuitive goals and so we expected (and observed) that they simplified the implementation while not damaging performance.
D.2 Eliminating  $W_{0}$  in favor of  $s_{init}$
While TUNER makes use of the "initial wealth" value  $W_0$ , MECHANIC instead adopts a varying value for  $W_0$  proportional to  $s_{init} \cdot m_t$ . This makes the first  $s$  value proposed by MECHANIC equal to  $s_{init}$ , which is more intuitive to set than  $W_0$ . The exponential growth in  $s$  allows us to set  $s_{init}$  to a very small value of  $10^{-8}$ . It also makes the values for  $s$  "scale-free" in the sense that rescaling the updates  $\mathbf{u}_t$  by any constant will have no effect on the resulting  $s_t$ .
D.3 Regret Bound
Theorem 2. With  $\beta = 1$ , Algorithm 2 guarantees for all  $\mathring{s} \geq 0$ :
$$
\sum_ {t = 1} ^ {T} h _ {t} \left(s _ {t} - \mathring {s}\right) \leq W _ {0} + \left(\mathring {s} + \max  _ {t} s _ {t}\right) \cdot m _ {T} + O \left(\mathring {s} \cdot \log \left(T \mathring {s} m _ {T} / m _ {1} W _ {0}\right) \sqrt {\sum_ {t = 1} ^ {T} h _ {t} ^ {2}}\right).
$$
Proof. First, we employ an argument developed by [44]:
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} h _ {t} \left(s _ {t} - \mathring {s}\right) \leq \sum_ {t = 1} ^ {T} \hat {h} _ {t} \left(s _ {t} - \mathring {s}\right) + \sum_ {t = 1} ^ {T} \left| \hat {h} _ {t} - h _ {t} \right| \left(\left| s _ {t} \right| + \left| \mathring {s} \right|\right) \\ \leq \sum_ {t = 1} ^ {T} \hat {h} _ {t} \left(s _ {t} - \mathring {s}\right) + \left(\max  _ {t} | s _ {t} | + | \mathring {s} |\right) \sum_ {t = 1} ^ {T} \left| \hat {h} _ {t} - h _ {t} \right| \\ = \sum_ {t = 1} ^ {T} \hat {h} _ {t} \left(s _ {t} - \mathring {s}\right) + m _ {T} \left(\max  _ {t} | s _ {t} | + | \mathring {s} |\right). \\ \end{array}
$$
2Removing the clip in theory may requiring some additional non-worst-case assumption.
So, in the following, it suffices to bound  $\sum_{t=1}^{T} \hat{h}_t(s_t - \bar{s})$ . This is helpful because we will be able to use the bound  $|\hat{h}_t| \leq m_{t-1}$ , and  $m_{t-1}$  is known before  $\hat{h}_t$  is revealed.
As is typical in the analysis of parameter-free algorithms, the proof proceeds by lower-bounding the wealth. Define a function  $a(x)$  piecewise by:
$$
a (x) = \left\{ \begin{array}{l l} 0 & x \leq 0 \\ x ^ {2} / 2 & x \in [ 0, 1 ] \\ x - 1 / 2 & x \geq 1 \end{array} \right.
$$
Notice that  $a(x)$  is differentiable, monotonically increasing and 1-Lipschitz. We are going to roughly show that  $W_{t} \geq \Omega (\exp (a(-\sum_{k = 1}^{t}\hat{h}_{k} / \sqrt{v_{t}})))$ , after which the regret bound will follow from the wealth-regret duality [10].
The key technical inequality in the proof is the following: for any  $A, B, m$  with  $B \geq 4m^2$ , and any  $|x| \leq m$ , we have:
$$
a \left(\frac {- A}{\sqrt {B}}\right) - \frac {x}{\sqrt {B}} \operatorname {c l i p} \left(\frac {- A}{\sqrt {B}}, 0, 1\right) \geq a \left(\frac {- A - x}{\sqrt {B + x ^ {2}}}\right) - \frac {x ^ {2}}{B}. \tag {6}
$$
Once (6) is established, we proceed as follows: defining  $c_{t} = \frac{\mathrm{clip}(\sum_{k=1}^{t-1} \hat{h}_{k} / \sqrt{4m_{t-1}^{2} + v_{t-1}}, 0, 1)}{\sqrt{v_{t}}}$ , we have:
$$
\begin{array}{l} \log (W _ {t}) = \log (W _ {t - 1}) + \log (1 - \hat {h} _ {t} c _ {t}) \\ \geq \log (W _ {t - 1}) - \hat {h} _ {t} c _ {t} - \hat {h} _ {t} ^ {2} c _ {t} ^ {2}, \\ \end{array}
$$
where we have used  $c_t \leq 1/2$  and the identity  $\log(1 - x) \geq -x - x^2$  for  $x \leq 1/2$  (which applies since  $\hat{h}_t \leq m_{t-1}$  by definition). Now, set  $A = \sum_{k=1}^{t-1} \hat{h}_k$  and  $B = 4m_{t-1}^2 + v_{t-1}$  and  $x = \hat{h}_t$  in (6), we see that:
$$
\begin{array}{l} \log (W _ {t}) - \log (W _ {t - 1}) \geq - \frac {x}{\sqrt {B}} \mathrm {c l i p} \left(\frac {- A}{\sqrt {B}}, 0, 1\right) - \frac {\hat {h} _ {t} ^ {2}}{4 m _ {t - 1} ^ {2} + \mathbf {v} _ {t - 1}} \\ \geq a \left(\frac {- A - x}{\sqrt {B + x ^ {2}}}\right) - a \left(\frac {- A}{\sqrt {B}}\right) - \frac {x ^ {2}}{B} - \frac {\hat {h} _ {t} ^ {2}}{4 m _ {t - 1} ^ {2} + \mathbf {v} _ {t - 1}} \\ \geq a \left(\frac {- \sum_ {k = 1} ^ {t} \hat {h} _ {k}}{\sqrt {4 m _ {t - 1} ^ {2} + v _ {t}}}\right) - a \left(\frac {- \sum_ {k = 1} ^ {t - 1} \hat {h} _ {k}}{\sqrt {4 m _ {t - 1} ^ {2} + v _ {t - 1}}}\right) - \frac {2 \hat {h} _ {t} ^ {2}}{4 m _ {t - 1} ^ {2} + v _ {t - 1}} \\ \geq a \left(\frac {- \sum_ {k = 1} ^ {t} \hat {h} _ {k}}{\sqrt {4 m _ {t} ^ {2} + v _ {t}}}\right) - a \left(\frac {- \sum_ {k = 1} ^ {t - 1} \hat {h} _ {k}}{\sqrt {4 m _ {t - 1} ^ {2} + v _ {t - 1}}}\right) - \frac {2 \hat {h} _ {t} ^ {2}}{4 m _ {t - 1} ^ {2} + v _ {t - 1}}. \\ \end{array}
$$
Thus by telescoping the sum, we have:
$$
\log (W _ {T}) \geq \log (W _ {0}) + a \left(\frac {- \sum_ {k = 1} ^ {T} \hat {h} _ {k}}{\sqrt {v _ {T}}}\right) - \sum_ {t = 1} ^ {T} \frac {2 \hat {h} _ {t} ^ {2}}{4 m _ {t - 1} ^ {2} + v _ {t - 1}}.
$$
Now, observe that  $\frac{2\hat{h}_t^2}{4m_{t - 1}^2 + v_{t - 1}}\leq \frac{2\hat{h}_t^2}{v_t}\leq 2(\log (v_t) - \log (v_{t - 1}))$ , so we have  $\sum_{t = 1}^{T}\frac{\hat{h}_t^2}{v_{t - 1}}\leq 2\log (Tm_T / m_1)$  so that overall:
$$
W _ {T} \geq \frac {W _ {0} m _ {1}}{T ^ {2} m _ {T}} \exp \left[ a \left(\frac {- \sum_ {k = 1} ^ {T} \hat {h} _ {k}}{\sqrt {v _ {T}}}\right) \right].
$$
Thus, if we define  $p(H) = \frac{W_0m_1}{T^2m_T} \exp \left[a\left(\frac{H}{\sqrt{v_T}}\right)\right]$ , we have  $W_{T} \geq p(-\sum_{k=1}^{T} \hat{h}_{k})$ . Now, we employ the reward-regret duality:
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \hat {h} _ {t} \left(s _ {t} - \mathring {s}\right) = s _ {\text {i n i t}} \cdot m + \mathring {s} \sum_ {t = 1} ^ {T} \left(- \hat {h} _ {t}\right) - W _ {T} \\ \leq W _ {0} + \sup  _ {G} \stackrel {\circ} {s} \cdot G - p (G) \\ = W _ {0} + p ^ {\star} (\hat {s}) \\ \leq W _ {0} + O \left(s \log \left(s T / s _ {\text {i n i t}}\right) \sqrt {v _ {T}}\right). \\ \end{array}
$$
Where  $p^{\star}$  is the Fenchel conjugate of  $p$  and the evaluation of the conjugate follows from direct calculation (see, e.g. [10, 28, 46]).
Thus, to prove the theorem we need only show (6). This is established via casework in a manner similar to [46].
Case 1.  $\frac{-A}{\sqrt{B}} \leq 0$ : In this case, the statement is equivalent to:  $\frac{x^2}{B} \geq a\left(\frac{-A - x}{\sqrt{B + x^2}}\right)$ . Note that since  $\frac{-A}{\sqrt{B}} \leq 0$ , we have  $A \geq 0$ . Therefore:
$$
\begin{array}{l} \frac {- A - x}{\sqrt {B + x ^ {2}}} = \frac {- A}{\sqrt {B + x ^ {2}}} - \frac {x}{\sqrt {B + x ^ {2}}} \\ \leq - \frac {x}{\sqrt {B + x ^ {2}}}. \\ \end{array}
$$
Further, we clearly have  $-\frac{x}{\sqrt{B + x^2}} \leq 1$  so that:
$$
a \left(\frac {- A - x}{\sqrt {B + x ^ {2}}}\right) \leq a \left(- \frac {x}{\sqrt {B + x ^ {2}}}\right) = \frac {x ^ {2}}{2 (B + x ^ {2})} \leq \frac {x ^ {2}}{B}.
$$
So, in the following we assume  $\frac{-A}{\sqrt{B}} \geq 0$ .
Case 2.  $\frac{-A - x}{\sqrt{B + x^2}} \leq 0$ : In this case, it suffices to show  $\frac{-x}{\sqrt{B}} \mathrm{clip}\left(\frac{-A}{\sqrt{B}}, 0, 1\right) \geq -\frac{x^2}{B}$ . The case assumption implies  $m^2 \geq x \geq -A \geq 0$ . Therefore, since  $B \geq 4m^2$ ,  $\mathrm{clip}\left(\frac{-A}{\sqrt{B}}, 0, 1\right) = \frac{-A}{\sqrt{B}}$  so that  $\frac{-x}{\sqrt{B}} \mathrm{clip}\left(\frac{-A}{\sqrt{B}}, 0, 1\right) = \frac{xA}{B} \geq \frac{-x^2}{B}$  as desired.
So, in the following we now further assume  $\frac{-A - x}{\sqrt{B + x^2}} \geq 0$ .
Case 3.  $\frac{-A}{\sqrt{B}} \in [0,1]$ : We have  $a\left(\frac{-A}{\sqrt{B}}\right) = \frac{A^2}{2B}$ , and also since  $a(z) \leq \frac{1}{2} z^2$  for all  $z$ ,  $a\left(\frac{-A - x}{\sqrt{B + x^2}}\right) \leq \frac{(A + x)^2}{2(B + x^2)}$ . Thus, it suffices to show that  $\frac{A^2}{2B} + \frac{xA}{B} \geq \frac{(A + x)^2}{2(B + x^2)} - \frac{x^2}{B}$ , but this is equivalent to  $\frac{(A + x)^2}{2B} \geq \frac{(A + x)^2}{2(B + x^2)} - \frac{x^2}{2B}$ , which clearly holds.
Case 4:  $\frac{-A}{\sqrt{B}} \geq 1$  and  $\frac{-A - x}{\sqrt{B + x^2}} \geq 1$ : In this case it suffices to show  $\frac{-A}{\sqrt{B}} - \frac{x}{\sqrt{B}} \geq \frac{-A - x}{\sqrt{B + x^2}} - \frac{x^2}{B}$ . To see this, we have:
$$
\begin{array}{l} \frac {- A - x}{\sqrt {B + x ^ {2}}} = \frac {- A}{\sqrt {B + x ^ {2}}} - \frac {x}{\sqrt {B + x ^ {2}}} \\ \leq \frac {- A}{\sqrt {B}} - \frac {x}{\sqrt {B}} + x \left(\frac {1}{\sqrt {B}} - \frac {1}{\sqrt {B + x ^ {2}}}\right) \\ \leq \frac {- A}{\sqrt {B}} - \frac {x}{\sqrt {B}} + \frac {2 x ^ {3}}{3 B ^ {3 / 2}} \\ \leq \frac {- A}{\sqrt {B}} - \frac {x}{\sqrt {B}} + \frac {x ^ {2}}{B}, \\ \end{array}
$$
where in the second-to-last line we have used the fact that  $h \mapsto \frac{1}{\sqrt{B + h}}$  is a convex in  $h$ , and in the last line we have used  $\sqrt{B} \geq m \geq x$ .
Case 5:  $\frac{-A}{\sqrt{B}} \geq 1$  and  $\frac{-A - x}{\sqrt{B + x^2}} \in [0,1)$ : In this case we need to show  $\frac{-A}{\sqrt{B}} - \frac{1}{2} \geq \frac{(A + x)^2}{2(B + x^2)} - \frac{x^2}{B} + \frac{x}{\sqrt{B}}$ . To see this, we first observe that since  $\frac{-A - x}{\sqrt{B + x^2}} \in [0,1)$ , we have
$$
A ^ {2} + 2 A x + x ^ {2} \leq B + x ^ {2}
$$
$$
A ^ {2} + 2 A x \leq B.
$$
Thus, by quadratic formula,  $A \geq -x - \sqrt{x^2 + B}$ , so that we have  $A \in [-x - \sqrt{x^2 + B}, -\sqrt{B}]$ .
Next, our target identity can be rearranged into an equivalent form as follows:
$$
\frac {- A}{\sqrt {B}} - \frac {1}{2} \geq \frac {(A + x) ^ {2}}{2 (B + x ^ {2})} - \frac {x ^ {2}}{B} + \frac {x}{\sqrt {B}}
$$
$$
0 \geq \frac {(A + x) ^ {2}}{2 (B + x ^ {2})} + \frac {A}{\sqrt {B}} + \frac {1}{2} - \frac {x ^ {2}}{B} + \frac {x}{\sqrt {B}},
$$
so that it suffices to show the second line above. Notice that the RHS of this expression is convex in  $A$  and so is maximized at the boundary of the range  $[-x - \sqrt{x^2 + B}, -\sqrt{B}]$ . When  $A = -\sqrt{B}$  we have:
$$
\frac {(A + x) ^ {2}}{2 (B + x ^ {2})} + \frac {A}{\sqrt {B}} + \frac {1}{2} - \frac {x ^ {2}}{B} + \frac {x}{\sqrt {B}} \leq \frac {(A + x) ^ {2}}{2 B} + \frac {A}{\sqrt {B}} + \frac {1}{2} - \frac {x ^ {2}}{B} + \frac {x}{\sqrt {B}} = - \frac {x ^ {2}}{2 B} \leq 0.
$$
Alternatively, when  $A = -x - \sqrt{x^2 + B}$ , we have
$$
\begin{array}{l} \frac {(A + x) ^ {2}}{2 (B + x ^ {2})} + \frac {A}{\sqrt {B}} + \frac {1}{2} - \frac {x ^ {2}}{B} + \frac {x}{\sqrt {B}} = 1 - \frac {\sqrt {x ^ {2} + B}}{\sqrt {B}} - \frac {x ^ {2}}{B} \\ \leq 0. \\ \end{array}
$$
This establishes the claimed inequality (6) and completes the proof.