SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Aaron Defazio
Meta, FAIR
defazio@meta.com
Harsh Mehta
harshm@google.com
We introduce a technique for tuning the learning rate scale factor of any base optimization algorithm and schedule automatically, which we call MECHANIC. Our method provides a practical realization of recent theoretical reductions for accomplishing a similar goal in online convex optimization. We rigorously evaluate MECHANIC on a range of large scale deep learning tasks with varying batch sizes, schedules, and base optimization algorithms. These experiments demonstrate that depending on the problem, MECHANIC either comes very close to, matches or even improves upon manual tuning of learning rates.
Modern deep learning is driven by first-order stochastic optimization algorithms. These are algorithms that are designed to solve the classical stochastic optimization problem:
$$
\min  F (\mathbf {x}) = \min  _ {\mathbf {z}} \mathbb {E} [ f (\mathbf {x}, \mathbf {z}) ]
$$
where  $\mathbf{z}$  is a minibatch of examples,  $\mathbf{x} \in \mathbb{R}^d$  is the model parameters, and  $f$  is the loss incurred by using weights  $\mathbf{x}$  on the minibatch  $\mathbf{z}$ . A first-order algorithm follows the protocol:
1. Output a  $t$ th iterate  $\mathbf{x}_t$ .
2. Sample a random minibatch  $\mathbf{z}_t$ .
3. Compute  $\mathbf{g}_t = \nabla f(\mathbf{x}_t,\mathbf{z}_t)$  (the gradient is taken with respect to  $\mathbf{x}_t$  only).
4. Possibly update some internal algorithm state based upon  $\mathbf{g}_t$  in preparation for computing the next iterate  $\mathbf{x}_{t + 1}$ .
The prototypical such optimization algorithm is stochastic gradient descent (SGD), which exemplifies the attractive features of this approach: it is computationally cheap (running in  $O(d)$  time per update), and with proper tuning obtains minimax optimal convergence guarantees [1, 2]. Modern practice makes use of a wide range of variants of SGD, such SGD with momentum, AdaGrad [3], Adam [4], AdamW [5] or Lion [6]. Interest in such improvements to SGD is driven by the increasing computational demands of training large neural networks: better optimization means cheaper training, which translates to significant savings in terms of time, cost, and environmental impact.
Most modern algorithms for training neural networks are equipped with a scalar "scale factor" or "learning rate" hyperparameter  $s \in \mathbb{R}$ . Roughly speaking, these algorithms produce iterates of the form  $\mathbf{x}_{t+1} = \mathbf{x}_t + s \cdot \mathbf{u}_t$  where  $\mathbf{u}_t$  is some update vector produced as a function of the observed gradients  $\mathbf{g}_1, \ldots, \mathbf{g}_t$  (we will use bold font for vectors in  $\mathbb{R}^d$  like  $\mathbf{u}$  and normal font for all other quantities like  $s$ ). As an example, the classical SGD algorithm sets  $\mathbf{u}_t = -\eta_t \mathbf{g}_t$  for some sequence of scalars  $\{\eta_t\}$  typically called the schedule. The formula for the SGD update is:
$$
\mathbf {x} _ {t + 1} = \mathbf {x} _ {1} - s \cdot \sum_ {i = 1} ^ {t} \eta_ {i} \mathbf {g} _ {i}. \tag {1}
$$
The process of selecting the optimal  $s$  is called "tuning", and is a key resource sink in machine learning. The typical approach is simply to try many possibilities to find the empirically optimal  $s$ , which requires multiple expensive training runs. This paper introduces a technique for choosing  $s$  automatically on-the-fly in order to avoid this expense.
Our procedure, which we call MECHANIC, is a generic wrapper around any base optimization algorithm (BASE) that produces a new optimizer which does not require tuning of the scalar  $s$ . The base optimization algorithm is allowed to make any kind of update (for example, it may use any kind of schedule, preconditioner or weight decay). If  $\mathbf{x}_t^{\mathrm{BASE}} \in \mathbb{R}^d$  is the  $t$ th iterate of BASE, then the wrapper will produce a scalar  $s_t \in \mathbb{R}$  and set the  $t$ th iterate of the wrapped algorithm to be  $\mathbf{x}_t = \mathbf{x}_1^{\mathrm{BASE}} + s_t (\mathbf{x}_t^{\mathrm{BASE}} - \mathbf{x}_1^{\mathrm{BASE}})$ . As an example, suppose that BASE is the classical SGD algorithm with update equation (1). Then, given  $s_t$ , we would set  $\mathbf{x}_t = \mathbf{x}_1^{\mathrm{BASE}} - s_t \sum_{i=1}^{t-1} \eta_i \mathbf{g}_i$ . Disregarding for now the fact that the gradients  $\mathbf{g}_i$  actually depend on the iterates  $\mathbf{x}_i^1$ , we see that  $\mathbf{x}_t$  is what the  $t$ th iterate of SGD would have been if the schedule were scaled by  $s_t$ .
Removing tuning of learning rate scalars is already a well-studied problem. One of the main attractions of early work in "adaptive" optimization such as AdaGrad and Adam [3, 7, 4] is that these algorithms require less tuning than ordinary SGD. Over the last decade, a number of works have aimed to tackle this problem from both an empirical and theoretical perspective [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]. An intuitive approach might take the route of "hypergradient descent": that is, differentiating the update step of the optimization algorithm itself (e.g. [8, 9]). Strangely, it appears to be difficult to prove that such schemes behave well: theory-based approaches often adopt rather different strategies. Instead, we start from known theory and propose a few important modifications to produce a simple and effective practical implementation. We then rigorously evaluate our algorithm on a variety of datasets. We emphasize that our primary contribution is not new theoretical development, but instead the translation between theory and practice, which involves fusing several known analytical techniques as well as subtle departures from theory.
Previous works that investigate deep learning performance of "learning-rate free" optimization inspired by theory (e.g. [20, 21, 16, 15]) have already demonstrated impressive results. However, these works typically build "hand-crafted" algorithms that often blend theoretical analysis with specific empirically successful algorithms such as Adam. In contrast, our wrapper works well with any base algorithm and so can seamlessly integrate new empirical advances in optimization: one does not need intimate familiarity with the analysis of our approach to apply it to a new algorithm.
2 Background: Online Convex Optimization
We develop our formalism via online convex optimization (OCO) [22, 23, 24]. OCO is a popular framework for design and analysis of stochastic optimization algorithms. In brief, for each of  $T$  rounds (corresponding to  $T$  iterations of optimization), the OCO algorithm must first output a  $t$ th iterate  $\mathbf{x}_t$ , after which the algorithm is presented with a  $t$ th loss function  $\ell_t$ . Typically, one envisions the case  $\ell_t(\mathbf{x}) = \ell (\mathbf{x},\mathbf{z}_t)$  for some fixed loss  $\ell$  and new data point  $\mathbf{z}_t$ . The goal of an algorithm ALG is to minimize the regret  $R^{\mathrm{ALG}}(\mathring{\mathbf{x}})$ :
$$
R ^ {\mathrm {A L G}} (\mathring {\mathbf {x}}) \triangleq \sum_ {t = 1} ^ {T} \ell_ {t} (\mathbf {x} _ {t}) - \ell_ {t} (\mathring {\mathbf {x}}).
$$
Many references focus primarily on the case  $\mathbf{\dot{x}} = \mathrm{argmin}\sum_{t = 1}^{T}\ell_{t}(\mathbf{x})$  in order to consider the single scalar value  $\sup_{\hat{\mathbf{x}}}R_T(\tilde{\mathbf{x}})$  [25, 26], but we will employ the formulation of regret as a function above instead as it is strictly more general. When  $\ell_t$  is convex, then with  $\mathbf{g}_t\triangleq \nabla \ell_t(\mathbf{x}_t)$  (or, more generally when  $\mathbf{g}_t$  is a subgradient of  $\ell_t$  at  $\mathbf{x}_t$ ), we have:
$$
R ^ {\mathrm {A L G}} (\mathring {\mathbf {x}}) \leq \sum_ {t = 1} ^ {T} \left\langle \mathbf {g} _ {t}, \mathbf {x} _ {t} - \mathring {\mathbf {x}} \right\rangle \triangleq R _ {\text {l i n e a r}} ^ {\mathrm {A L G}} (\mathring {\mathbf {x}}).
$$
As a result, the vast majority of OCO algorithms provide analysis that bounds only the linearized regret  $R_{\mathrm{linear}}^{\mathrm{ALG}}(\hat{\mathbf{x}})$ . Such algorithms do not need to observe the entire function  $\ell_t$ : instead, they only make use of the gradients  $\mathbf{g}_t$ . That is, the  $t$ th output of ALG (i.e.  $\mathbf{x}_t$ ) is purely a function of the previous sequence of gradients  $\mathbf{g}_1, \ldots, \mathbf{g}_{t-1}$  so that ALG is a first-order algorithm.
This seems like a significant issue to disregard, but we will provide mathematical justification shortly.
2.1 Learning the Scale in OCO
Just like stochastic optimization algorithms, most OCO algorithms also require a scale factor  $s$ . In fact, many stochastic optimization algorithms (such as SGD and AdaGrad) are also OCO algorithms. Setting  $\eta_t = \eta$  for all  $t$ , SGD ensures the regret bound:
$$
R ^ {\mathrm {S G D}} (\mathring {\mathbf {x}}) \leq R _ {\text {l i n e a r}} ^ {\mathrm {S G D}} (\mathring {\mathbf {x}}) \leq O \left(\frac {\| \mathring {\mathbf {x}} - \mathbf {x} _ {1} \| ^ {2}}{s \eta} + s \eta \sum_ {t = 1} ^ {T} \| \mathbf {g} _ {t} \| ^ {2}\right). \tag {2}
$$
From this equation, one can deduce in hindsight that for any given  $\mathbf{\dot{x}}$ , the optimal value for  $s$  is  $\frac{\|\hat{\mathbf{x}} - \mathbf{x}_1\|}{\eta\sqrt{\sum_{t = 1}^{T}\|\mathbf{g}_t\|^2}}$ , which would provide the bound:
$$
R _ {\text {l i n e a r}} ^ {\mathrm {S G D W I T H T U N E D} s} (\mathring {\mathbf {x}}) \leq O \left(\| \mathring {\mathbf {x}} - \mathbf {x} _ {1} \| \sqrt {\sum_ {t = 1} ^ {T} \| \mathbf {g} _ {t} \| ^ {2}}\right). \tag {3}
$$
This result is minimax optimal [27], but requires knowledge of the unknown optimal  $s$ . Very recently, [14, 16, 15] have produced algorithms that estimate the value of  $\| \mathbf{x}_1^{\mathrm{BASE}} - \mathbf{\dot{x}}\|$  on-the-fly and use this estimate to quickly identify the optimal scaling value  $s$ . These algorithms achieve impressive practical performance, but they require an understanding of the closed-form solution for the optimal  $s$  value above. Our goal is to learn the correct scaling regardless of the base algorithm.
To this end, we will leverage a scheme recently developed by [28] that allows one to automatically tune the scale of a base OCO algorithm using another "meta" OCO algorithm. We reproduce their result below (with notation altered to suit our application) along with the short proof:
Theorem 1 ([28]). Suppose BASE and TUNER are both OCO algorithms. Let  $\{\mathbf{x}_t^{\mathrm{BASE}}\} \subset \mathbb{R}^d$  indicate the iterates of BASE in response to an arbitrary sequence of gradients  $\{\mathbf{g}_t\}$ , and let  $\{s_t\} \subset \mathbb{R}$  indicate the iterates of TUNER in response to the sequence of scalars  $\{h_t = \langle \mathbf{g}_t, \mathbf{x}_t^{\mathrm{BASE}} - \mathbf{x}_1 \rangle\}$ . Define a new online algorithm MECHANIC via:
$$
\mathbf {x} _ {t} ^ {\text {M E C H A N I C}} = \mathbf {x} _ {1} ^ {\text {B A S E}} + s _ {t} \cdot \left(\mathbf {x} _ {t} ^ {\text {B A S E}} - \mathbf {x} _ {1} ^ {\text {B A S E}}\right).
$$
Then  $\mathbf{x}_t^{\mathrm{MECHANIC}}$  ensures regret:
$$
R _ {l i n e a r} ^ {\text {M E C H A N I C}} (\mathring {\mathbf {x}}) \leq \inf  _ {\mathring {s}} R _ {l i n e a r} ^ {\text {T U N E R}} (\mathring {s}) + \mathring {s} R _ {l i n e a r} ^ {\text {B A S E}} \left(\left(\mathring {\mathbf {x}} - \mathbf {x} _ {1} ^ {\text {B A S E}}\right) / \mathring {s}\right).
$$
Proof. By definition, for any  $\mathring{s}$ , we have:
$$
\begin{array}{l} R _ {\text {l i n e a r}} ^ {\text {M E C H A N I C}} (\mathring {\mathbf {x}}) = \sum_ {t = 1} ^ {T} \left\langle \mathbf {g} _ {t}, \mathbf {x} _ {1} ^ {\text {B A S E}} + s _ {t} \cdot \left(\mathbf {x} _ {t} ^ {\text {B A S E}} - \mathbf {x} _ {1} ^ {\text {B A S E}}\right) - \mathring {\mathbf {x}} \right\rangle \\ = \sum_ {t = 1} ^ {T} \left\langle \mathbf {g} _ {t}, \mathbf {x} _ {t} ^ {\text {B A S E}} - \mathbf {x} _ {1} ^ {\text {B A S E}} \right\rangle \left(s _ {t} - \mathring {s}\right) + \mathring {s} \sum_ {t = 1} ^ {T} \left\langle \mathbf {g} _ {t}, \mathbf {x} _ {t} ^ {\text {B A S E}} - \mathbf {x} _ {1} ^ {\text {B A S E}} - \left(\mathring {\mathbf {x}} - \mathbf {x} _ {1} ^ {\text {B A S E}}\right) / \mathring {s} \right\rangle \\ = R _ {\text {l i n e a r}} ^ {\text {T U N E R}} (\mathring {s}) + \mathring {s} R _ {\text {l i n e a r}} ^ {\text {B A S E}} \left(\mathbf {x} _ {1} ^ {\text {B A S E}} + \left(\mathring {\mathbf {x}} - \mathbf {x} _ {1} ^ {\text {B A S E}}\right) / \mathring {s}\right). \\ \end{array}
$$
With this result, the job of finding the optimal  $s$  can usually be completely relegated to TUNER. Although the value  $\mathring{s}$  appears in both terms of the sum  $R_{\mathrm{linear}}^{\mathrm{TUNER}}(\mathring{s}) + \mathring{s} R_{\mathrm{linear}}^{\mathrm{BASE}}(\mathbf{x}_1^{\mathrm{BASE}} + (\mathbf{\dot{x}} - \mathbf{x}_1^{\mathrm{BASE}}) / \mathring{s})$ , it turns out that for essentially all plausible BASE algorithms, there is a particular value  $\mathring{s}$  that causes  $\mathring{s} R_{\mathrm{linear}}^{\mathrm{BASE}}(\mathbf{x}_1^{\mathrm{BASE}} + (\mathbf{\dot{x}} - \mathbf{x}_1^{\mathrm{BASE}}) / \mathring{s})$  to obtain the optimal regret bound (3). Thus, by setting  $\mathring{s}$  to be this value, which is unknown a priori, we need only ensure that  $R_{\mathrm{linear}}^{\mathrm{TUNER}}(\mathring{s})$  is small enough to not significantly affect the overall regret bound. Note that this setting of  $\mathring{s}$  is done entirely in the analysis. For example, if BASE is actually SGD with a learning rate  $\eta$  and  $s = 1$  as in (2), we have
$$
R ^ {\text {M E C H A N I C}} (\mathbf {\hat {x}}) \leq R _ {\text {l i n e a r}} ^ {\text {M E C H A N I C}} (\mathbf {\hat {x}}) \leq \inf  _ {\mathfrak {s}} R _ {\text {l i n e a r}} ^ {\text {T U N E R}} (\mathfrak {s}) + O \left(\frac {\| \mathbf {\hat {x}} - \mathbf {x} _ {1} \| ^ {2}}{\mathfrak {s} \eta} + \mathfrak {s} \eta \sum_ {t = 1} ^ {T} \| \mathbf {g} _ {t} \| ^ {2}\right),
$$
$$
\begin{array}{l} \text {s e t t i n g} \quad \mathring {s} = \frac {\| \dot {\mathbf {x}} - \mathbf {x} _ {1} \|}{\eta \sqrt {\sum_ {t = 1} ^ {T} \| \mathbf {g} _ {t} \| ^ {2}}}: \\ \leq R _ {\mathrm {l i n e a r}} ^ {\mathrm {T U N E R}} \left(\frac {\| \mathbf {\hat {x}} - \mathbf {x} _ {1} \|}{\eta \sqrt {\sum_ {t = 1} ^ {T} \| \mathbf {g} _ {t} \| ^ {2}}}\right) + O \left(\| \mathbf {\hat {x}} - \mathbf {x} _ {1} \| \sqrt {\sum_ {t = 1} ^ {T} \| \mathbf {g} _ {t} \| ^ {2}}\right). \\ \end{array}
$$
Thus, if TUNER obtains low regret, then we will obtain the same regret bound as if we had optimally tuned the scaling factor for SGD. Intuitively, the gradient  $h_t$  provided to TUNER approximates the gradient over the entire course of the base optimizer rather than just at the most recent iterate. That is, for SGD,  $h_t \approx \frac{df(\mathbf{x}_t, \mathbf{z}_t)}{ds}$  where  $\mathbf{x}_t = \mathbf{x}_1 - s\sum_{k=1}^{t-1}\eta_k\mathbf{g}_k$ .
2.2 Parameter-Free Online Optimization
The problem with the above result is that we seem to have simply pushed the problem off to TUNER: what if TUNER itself requires us to set a scale factor? Solving this problem has been the focus of a substantial effort in the online optimization community [29, 30, 10, 11, 28, 12]. The most advanced such algorithms are able to ensure for all  $\mathfrak{S}$  simultaneously:
$$
R _ {\text {l i n e a r}} (\mathring {s}) = \sum_ {t = 1} ^ {T} h _ {t} \left(s _ {t} - \mathring {s}\right) \leq \tilde {O} \left(\left| \mathring {s} \right| \sqrt {\sum_ {t = 1} ^ {T} h _ {t} ^ {2}}\right). \tag {4}
$$
Thus, if we set  $h_t = \langle \mathbf{g}_t, \mathbf{x}_t^{\mathrm{BASE}} - \mathbf{x}_1^{\mathrm{BASE}} \rangle$ , we obtain:
$$
R _ {\text {l i n e a r}} ^ {\text {T U N E R}} (\mathring {s}) \leq \tilde {O} \left(\left| \mathring {s} \right| \sqrt {\sum_ {t = 1} ^ {T} \langle \mathbf {g} _ {t} , \mathbf {x} _ {t} ^ {\text {B A S E}} - \mathbf {x} _ {1} ^ {\text {B A S E}} \rangle^ {2}}\right).
$$
In a theoretical development of this technique, it is necessary to prevent the terms  $\langle \mathbf{g}_t,\mathbf{x}_t^{\mathrm{BASE}} - \mathbf{x}_1^{\mathrm{BASE}}\rangle^2$  from becoming too large (as otherwise  $R_{\mathrm{linear}}^{\mathrm{TUNER}}$  is too large). Typically, this is accomplished by constraining the base algorithm to satisfy  $\| \mathbf{x}_t^{\mathrm{BASE}} - \mathbf{x}_1^{\mathrm{BASE}}\| \leq \rho$  for some user-specified arbitrary  $\rho$ . Enforcing such a constraint means that the regret bound (2) would only apply to  $\| \hat{\mathbf{x}}\| \leq \rho$ , but ensures that  $\langle \mathbf{g}_t,\mathbf{x}_t^{\mathrm{BASE}} - \mathbf{x}_1^{\mathrm{BASE}}\rangle^2\leq \rho^2\| \mathbf{g}_t\| ^2$ . Thus, by setting  $\dot{s} = \| \dot{\mathbf{x}} -\mathbf{x}_1^{\mathrm{BASE}}\| /\rho$ , the combined algorithm obtains the optimal regret bound of  $O(\| \hat{\mathbf{x}} -\mathbf{x}_1^{\mathrm{BASE}}\| \sqrt{\sum_{t = 1}^{T}\|\mathbf{g}_t\|^2})$  (amazingly, the value of  $\rho$  is irrelevant!). In practice however, we do not attempt to explicitly enforce any such constraints and simply rely on the intuition that any non-diverging algorithm is unlikely to produce excessively large iterates.
At no point in this process do we need access to the internal state of the base algorithm BASE. This means that improvements to BASE will automatically be reflected in improvements to the overall algorithm. In this paper, we investigate the performance of MECHANIC on deep learning tasks. We consider a variety of settings for the base algorithm BASE (i.e. AdamW, Lion, SGD, with various batch sizes and learning rate schedules of various shapes), and employ a parameter-free algorithm as the TUNER to automatically find the best scale factor for the base algorithm.
3 The MECHANIC algorithm
Our MECHANIC algorithm is specified in Algorithm 1. The algorithm is built by applying Theorem 1 to a parameter-free TUNER algorithm presented in Algorithm 2, which is described along with theoretical analysis in Appendix D. However, when building MECHANIC, we modify the "pure" theoretically tractable Algorithm 2 to simplify the implementation while still capturing the essential intuition and maintaining the same performance. In the remainder of this section we will provide some intuition behind the TUNER update as used in MECHANIC as well as describing some potentially unfamiliar subtleties relating to our use of exponentially weighted moving averages.
MECHANIC takes as input a base algorithm that generates update vectors  $\mathbf{u}_t$  as described in the previous sections. We then set  $\pmb{\Delta}_{t + 1} = \sum_{k = 1}^{t}\mathbf{u}_k = \mathbf{x}_{t + 1}^{\mathrm{BASE}} - \mathbf{x}_1^{\mathrm{BASE}}$ . The majority of the algorithm contains our TUNER method, which is a variant of the analytically tractable Algorithm 2, with a
Algorithm 1 MECHANIC
1: Input: Base algorithm BASE,  $\beta \in [0,1]^n$  (default  $n = 6$ ,  $\beta = (0.9,0.99,0.999,0.9999,0.99999,0.999999)$ ,  $\lambda \in \mathbb{R}$  (default  $\lambda = 0.01$ ).  $s_{init} \in \mathbb{R}$ : first non-zero  $s$  value (default  $s_{init} = 10^{-8}$ ).  $\epsilon = 10^{-8}$ : small value for numerical stability.  
2:  $v_0 \gets 0 \in \mathbb{R}^n$ ,  $r_0 \gets 0 \in \mathbb{R}^n$ ,  $m_0 \gets 0 \in \mathbb{R}^n$ ,  $\mathbf{x}_{ref} \gets \mathbf{x}_1^{\mathrm{BASE}}$ .  
3:  $\Delta_1 \gets 0 \in \mathbb{R}^d$   
4:  $s_1 \gets 0 \in \mathbb{R}^n$ . // We will use  $s_{t,i}$  to indicate the  $i$ th coordinate of  $s_t$ .  
5: for  $t = 1\dots T$  do  
6:  $\mathbf{g}_t \gets \nabla f(\mathbf{x}_t, \mathbf{z}_t)$ . //  $\mathbf{x}_t$  is the  $t$ th set of model parameters and  $\mathbf{z}_t$  is the  $t$ th minibatch.  
7: Send  $\mathbf{g}_t$  to BASE, receive update  $\mathbf{u}_k$ . // On its own, BASE would update  $\mathbf{x}_{t+1} \gets \mathbf{x}_t + \mathbf{u}_k$ .  
8: [Optional] Set  $\Delta_t = \frac{\mathbf{x}_t - \mathbf{x}_{ref}}{\left(\sum_{i=1}^n s_{t,n}\right) + \epsilon}$  to save memory instead of storing  $\Delta_t$  from last round.  
9:  $\Delta_{t+1} \gets \Delta_t + \mathbf{u}_t$ .  
10:  $h_t \gets \left\langle \Delta_t, \mathbf{g}_t + \frac{\lambda\left(\sum_{i=1}^n s_{t,n}\right) \| \mathbf{g}_t \|_{\mathbf{x}_t}}{\|\mathbf{x}_t\|} \right\rangle$  // Note use of  $\Delta_t$  rather than  $\Delta_{t+1}$ .  
11:  $m_t \gets \max(\beta m_{t-1}, h_t)$  (multiplications by  $\beta$  and maximum are taken coordinate-wise)  
12:  $v_t \gets \beta^2 v_{t-1} + h_t^2$   
13:  $r_t \gets \beta r_{t-1} - s_{t-1} h_t$   
14:  $r_t \gets \max(0, r_t)$  // This step is used instead of more complicated procedures in Algorithm 2  
15:  $W_t \gets s_{init} \cdot m_t + r_t$   
16:  $s_{t+1} \gets \frac{n}{\sqrt{v_t} + \epsilon}$   
17:  $\mathbf{x}_{t+1} \gets \mathbf{x}_1^{\mathrm{BASE}} + (\sum_{i=1}^n s_{t+1,i}) \cdot \Delta_{t+1}$   
18: end for
few modifications. Note that the indexing on  $\pmb{\Delta}$  is very important and may be counterintuitive: the definition of  $h_t$  does not include  $\Delta_{t + 1}$ , but rather  $\Delta_t$ .  $h_t$  is the "gradient" that is supplied to TUNER, as described by Theorem 1.
To gain some intuition behind the update, let us consider the case that  $n = 1$  and  $\beta = 1.0$  (that is, without employing any exponentially-weighted moving averages). We keep track of the quantity  $W_{t} = s_{init} \cdot m_{t} - \sum_{k=1}^{t} h_{k} s_{k}$ , which is usually called the "wealth" of the algorithm (the quantity  $r_{t} = -\sum_{k=1}^{t} h_{k} s_{k}$  is sometimes called the "reward").  $s_{init}$  specifies the starting value for  $s_{t}$  and should be an under-estimate of the true optimal scaling. We then set  $s_{t+1} = \frac{W_{t}}{\sqrt{v_{t}}}$  (neglecting the  $\epsilon$  included for numerical stability). To understand this update strategy, we can re-write the update as:
$$
s _ {t + 1} = s _ {t} \cdot \frac {\sqrt {v _ {t - 1}}}{\sqrt {v _ {t}}} - \frac {s _ {t} h _ {t}}{\sqrt {v _ {t}}} \approx \left(1 - \frac {h _ {t} ^ {2}}{2 v _ {t}}\right) s _ {t} - \frac {s _ {t} h _ {t}}{\sqrt {v _ {t}}}.
$$
Thus, the update looks like a combination of an AdaGrad-esque gradient descent step with learning rate scaled by  $s_t$  and a kind of "adaptive decay" (multiplication by  $1 - \frac{h_t^2}{2v_t}$ ). The adaptive decay is very important for stabilizing the algorithm: without it the values for  $s_t$  are prone to unchecked exponential growth due to scaling by  $s_t$  in  $\frac{s_t h_t}{\sqrt{v_t}}$ . Intuitively, this decay is the minimum amount required to prevent instabilities.
In Appendix D, we provide a formal Theorem bounding the regret of a variant of the procedure described above. Roughly speaking, for  $\beta = 1$  this result suggests:
$$
\sum_ {t = 1} ^ {T} h _ {t} \left(s _ {t} - \mathring {s}\right) \leq O \left(\left(\mathring {s} + \max  _ {t} s _ {t}\right) \cdot m _ {T} + \mathring {s} \cdot \log \left(T \mathring {s} / s _ {\text {i n i t}}\right) \sqrt {\sum_ {t = 1} ^ {T} h _ {t} ^ {2}}\right). \tag {5}
$$
In fact, the dependence of  $O(\log (T))$  in equation (5) can be improved to  $O(\sqrt{\log(T)})$  via more complicated algorithms (e.g. [28, 12, 31, 32]). However, we favor the simpler update and pleasing resemblance to familiar algorithms like AdaGrad via the Taylor expansion analysis above. Of note, the dependence on  $s_{init}$  is very mild: this suggests that we should be able to set  $s_{init}$  to a very small
value without damaging performance. In practice, we choose  $s_{init} = 10^{-8}$ , which we expect to dramatically underestimate the optimal value in all scenarios.
We hypothesize that the simplified TUNER we use in MECHANIC in fact possesses a rigorous theoretical analysis (although perhaps only with respect to simpler non-fully-worst-case adversaries), but demonstrating such a bound appears to involve difficult technical hurdles. In particular, our implementation is designed to be "scale-free": rescaling the values of  $\mathbf{g}_t$  by any constant scalar will have no effect on  $s_t$ . This property was first achieved only recently in theoretical analysis of parameter-free algorithms [12], and as-yet requires significantly more involved algorithms [12, 33].
3.1 The use of  $\beta$