SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Frederik Kunstner Victor S. Portella Mark Schmidt Nick Harvey
{kunstner,victorsp,schmidtm,nickhar}@cs.ubc.ca
University of British Columbia Canada CIFAR AI Chair (Amii)†
The backtracking line-search is an effective technique to automatically tune the step-size in smooth optimization. It guarantees similar performance to using the theoretically optimal step-size. Many approaches have been developed to instead tune per-coordinate step-sizes, also known as diagonal preconditioners, but none of the existing methods are provably competitive with the optimal per-coordinate step-sizes. We propose multidimensional backtracking, an extension of the backtracking line-search to find good diagonal preconditioners for smooth convex problems. Our key insight is that the gradient with respect to the step-sizes, also known as hypergradients, yields separating hyperplanes that let us search for good preconditioners using cutting-plane methods. As black-box cutting-plane approaches like the ellipsoid method are computationally prohibitive, we develop an efficient algorithm tailored to our setting. Multidimensional backtracking is provably competitive with the best diagonal preconditioner and requires no manual tuning.
When training machine learning models, tuning the hyperparameters of the optimizer is often a major challenge. For example, finding a reasonable step-size hyperparameter for gradient descent typically involves trial-and-error or a costly grid search. In smooth optimization, a common approach to set the step-size without user input is a backtracking line-search: start with a large step-size, and decrease it when it is too big to make sufficient progress. For ill-conditioned problems, however, there are limits to the improvement achievable by tuning the step-size. Per-coordinate step-sizes—also known as diagonal preconditioners—can drastically improve performance. Many approaches have been developed to automatically tune per-coordinate step-sizes. Those are often described as "adaptive" methods, but the meaning of this term varies widely, from describing heuristics that set per-coordinate step-sizes, to ensuring performance guarantees as if a particular property of the problem were known in advance. Yet, even on the simplest case of a smooth and strongly convex deterministic problem where a good fixed diagonal preconditioner exists (i.e., one that reduces the condition number), none of the existing adaptive methods are guaranteed to find per-coordinate step-sizes that improve the convergence rate. We discuss approaches to adaptive methods in the next section.
Contribution. We propose multidimensional backtracking, an extension of the standard backtracking line-search to higher dimension, to automatically find good per-coordinate step-sizes. Our method recovers the convergence rate of gradient descent with the optimal preconditioner for the problem, up to a  $\sqrt{2d}$  factor where  $d$  is the number of coordinates. This is a direct generalization of the line-search guarantee, with a penalty depending on dimension due to the extra degrees of freedom, as expected.
1.1 Adaptive step-sizes and preconditioning methods
Adaptive and parameter-free methods in online learning are an example where adaptive methods have a well-defined meaning. AdaGrad (McMahan and Streeter, 2010; Duchi et al., 2011) and Coin Betting (Orabona and Pál, 2016; Orabona and Tommasi, 2017) can adapt to problem-specific constants without user input and have strong guarantees, even in the adversarial setting. However,
this resilience to adversaries is a double-edged sword; to satisfy this definition of adaptivity, AdaGrad uses monotonically decreasing step-sizes. While AdaGrad still converges at the desired asymptotic rate on smooth, Lipschitz functions (Ward et al., 2019; Li and Orabona, 2019), its performance can be worse than plain gradient descent. This motivated investigations of workarounds to avoid the monotonically decreasing updates, including augmenting the update with an increasing step-size schedule (Agarwal et al., 2020), a line-search (Vaswani et al., 2020), or modifying the update to the preconditioner (Defazio et al., 2022). Methods commonly used in deep learning, such as RMSProp and Adam (Hinton et al., 2012; Kingma and Ba, 2015), are often motivated as adaptive by analogy to AdaGrad, but without decreasing step-sizes (e.g., Defossez et al., 2022, §4.3). This change is crucial for their practical performance, but nullifies their online-learning adaptivity guarantees.
Adaptive gain and hypergradient heuristics. Many heuristics that tune the hyperparameters of the optimization procedure use the gradient with respect to the hyperparameters, or hypergradients (Maclaurin et al., 2015). Methods have been proposed to tune the step-size (Masse and Ollivier, 2015), a preconditioner (Moskovitz et al., 2019), any hyperparameter (Baydin et al., 2018), or to maintain a model of the objective (Bae et al., 2022). "Stacking" such optimizers recursively has been shown to reduce the dependency on user-specified hyperparameters in practice (Chandra et al., 2022). This idea pre-dates the hypergradient nomenclature; Kesten (1958) presents a method to update the step-size based on the sign of successive gradients, and Saridis (1970) presents a control perspective for per-coordinate step-sizes, which can be cast as a hypergradient update to a diagonal preconditioner. This approach has led to adaptive gain methods such as Delta-Delta and variants (Barto and Sutton, 1981; Jacobs, 1988; Silva and Almeida, 1990; Sutton, 1992a,b), and further developed using the sign of the hypergradient (Riedmiller and Braun, 1993), full-matrix updates (Almeida et al., 1999), a larger history (Plagianakos et al., 2001), updates in log-space (Schraudolph, 1999; Schraudolph et al., 2005), heuristics to adjust the outer step-size (Mahmood et al., 2012), or multiplicative weight updates (Amid et al., 2022). While showing promising practical performance in some settings, existing methods are often motivated from intuition rather than a formal definition of adaptivity, giving no guarantee that the tuned method will converge faster, if at all. Indeed, hypergradient methods are often unstable, and may require as much manual tuning as the original optimizer they are intended to tune.
Second-order methods. A classical approach to preconditioning is to use second-order information, as in Newton's method or its regularized variants (e.g., Nesterov and Polyak, 2006). To avoid the load of computing and inverting the Hessian, quasi-Newton methods (Dennis and Moré, 1977) such as L-BFGS (Liu and Nocedal, 1989) fit an approximate Hessian using the secant equation. Variants using diagonal approximations have also been proposed, framed as Quasi-Cauchy, diagonal BFGS, or diagonal Barzilai-Borwein methods (Zhu et al., 1999; Andrei, 2019; Park et al., 2020), while other methods use the diagonal of the Hessian (LeCun et al., 2012; Yao et al., 2021). Some second-order and quasi-Newton methods converge super linearly (although not the diagonal or limited memory variants used in practice), but those guarantees only hold locally when close to the minimum. To work when far from a solution, those methods require "globalization" modifications, such as regularization or a line-search. Unfortunately, analyses of second-order methods do not capture the global benefit of preconditioning and instead lead to worse rates than gradient descent, as in the results of Byrd et al. (2016, Cor. 3.3), Bollapragada et al. (2018, Thm. 3.1), Meng et al. (2020, Thm. 1), Yao et al. (2021, Apx.), Berahas et al. (2022, Thm. 5.2), or Jahani et al. (2022, Thm. 4.9).
Line-searches. Adaptivity in smooth optimization is most closely related to line-searches. The standard guarantee for gradient descent on an  $L$ -smooth function requires a step-size of  $1 / L$ , but  $L$  is typically unknown. The backtracking line-search based on the Armijo condition (Armijo, 1966) approximately recovers this convergence guarantee by starting with a large step-size, and backtracking; halving the step-size whenever it does not yield sufficient improvement. However, line-searches are often overlooked in the discussion of adaptive methods, as they do not provide a way to set more than a scalar step-size. While line-searches can be shown to work in the stochastic overparameterized setting and have been applied to train neural networks (Vaswani et al., 2019), improvements beyond backtracking have been limited. Additional conditions (Wolfe, 1969), non-monotone relaxations (Grippo et al., 1986), or solving the line-search to higher precision (More and Thuente, 1994) can improve the performance in practice, but even an exact line-search cannot improve the convergence rate beyond what is achievable with a fixed step-size (Klerk et al., 2017).
<sup>1</sup>The hypergradient with respect to a diagonal preconditioner  $\mathbf{P} = \mathrm{Diag}(\mathbf{p})$  is, by the chain rule, the element-wise product  $(\odot)$  of subsequent gradients,  $-\nabla_{\mathbf{p}}f(\mathbf{x} - \mathrm{Diag}(\mathbf{p})\nabla f(\mathbf{x})) = \nabla f(\mathbf{x})\odot \nabla f(\mathbf{x} - \mathrm{Diag}(\mathbf{p})\nabla f(\mathbf{x}))$ .
Figure 1: Multidimensional backtracking can find the optimal diagonal preconditioner. Example on a linear regression where the optimal preconditioner can be computed. Left: Performance of Gradient Descent (GD), optimally preconditioned GD  $(\mathbf{P}_{*}\mathrm{GD})$  with a line-search (+LS), and Multidimensional Backtracking (MB) with the strategies in Section 5. The ellipsoid variant can outperform the globally optimal preconditioner by selecting preconditioners that leads to more local progress. Right: Optimal per-coordinate step-sizes  $(\bullet)$  and the ones found by MB (box) across iterations.
1.2 Summary of main result: adaptivity to the optimal preconditioner
Our approach is inspired by the work discussed above, but addresses the following key limitation: none of the existing methods attain better global convergence rates than a backtracking line-search. Moreover, this holds even on smooth convex problems for which a good preconditioner exists.
We generalize the backtracking line-search to handle per-coordinate step-sizes and find a good preconditioner. As in quasi-Newton methods, we build a preconditioner based on first-order information. However, instead of trying to approximate the Hessian using past gradients, our method searches for a preconditioner that minimizes the objective function at the next step. Our convergence result depends on the best rate achievable by an optimal diagonal preconditioner, similarly to how methods in online learning are competitive against the best preconditioner in hindsight. However, our notion of optimality is tailored to smooth strongly-convex problems and does not require decreasing step-sizes as in AdaGrad. Our update to the preconditioner can be interpreted as a hypergradient method, but instead of a heuristic update, we develop a cutting-plane method that uses hypergradients to guarantee a good diagonal preconditioner. Our main theoretical contribution is summarized below.
Theorem 1.1 (Informal). On a smooth, strongly-convex function  $f$  in  $d$  dimensions, steps accepted by multidimensional backtracking guarantee the following progress
$$
f \left(\mathbf {x} _ {t + 1}\right) - f \left(\mathbf {x} _ {*}\right) \leq \left(1 - \frac {1}{\sqrt {2 d}} \frac {1}{\kappa_ {*}}\right) \left(f \left(\mathbf {x} _ {t}\right) - f \left(\mathbf {x} _ {*}\right)\right),
$$
where  $\kappa_{*}$  is the condition number achieved by the optimal preconditioner defined in Section 2. The number of backtracking steps is at most linear in  $d$  and logarithmic in problem-specific constants.
Multidimensional backtracking finds per-coordinate step-sizes that lead to a provable improvement over gradient descent on badly conditioned problems that can be improved by diagonal preconditioning, i.e., if the condition number of  $f$  is at least  $\sqrt{2d} \cdot \kappa_*$ . Moreover, this guarantee is worst-case, and multidimensional backtracking can outperform the globally optimal preconditioner by finding a better local preconditioner, as illustrated on an ill-conditioned linear regression problem in Figure 1.
To find a competitive diagonal preconditioner, we view backtracking line-search as a cutting-plane method and generalize it to higher dimensions in Section 3. In Section 4 we show how to use hypergradients to find separating hyperplanes in the space of preconditioners, and in Section 5 we develop an efficient cutting-plane methods tailored to the problem. In Section 6, we illustrate the method through preliminary experiments and show it has consistent performance across problems.
Notation. We use standard font weight  $d$ ,  $n$ ,  $\alpha$  for scalars, bold  $\mathbf{x}$ ,  $\mathbf{y}$  for vectors, and capital bold  $\mathbf{P}$ ,  $\mathbf{A}$  for matrices. We use  $\mathbf{p}[i]$  for the  $i$ -th entry of  $\mathbf{p}$ , use  $\odot$  for element-wise multiplication, and define  $\mathbf{p}^2 \coloneqq \mathbf{p} \odot \mathbf{p}$ . We use  $\mathbf{P} = \mathrm{Diag}(\mathbf{p})$  to denote the diagonal matrix with diagonal  $\mathbf{p}$ , and  $\mathbf{p} = \mathrm{diag}(\mathbf{P})$  to denote the vector of diagonal entries of  $\mathbf{P}$ . We say  $\mathbf{A}$  is larger than  $\mathbf{B}$ ,  $\mathbf{A} \succeq \mathbf{B}$ , if  $\mathbf{A} - \mathbf{B}$  is positive semidefinite. If  $\mathbf{A} = \mathrm{Diag}(\mathbf{a})$ ,  $\mathbf{B} = \mathrm{Diag}(\mathbf{b})$ , the ordering  $\mathbf{A} \succeq \mathbf{B}$  is equivalent to  $\mathbf{a}[i] \geq \mathbf{b}[i]$  for all  $i$ , which we write  $\mathbf{a} \geq \mathbf{b}$ . We use  $\mathbf{I}$  for the identity matrix and  $\mathbf{1}$  for the all-ones vector.
2 Optimal preconditioning and sufficient progress
Consider a twice-differentiable function  $f\colon \mathbb{R}^d\to \mathbb{R}$  that is  $L$  -smooth and  $\mu$  -strongly convex, i.e.,
$$
\mu \frac {1}{2} \| \mathbf {x} - \mathbf {y} \| ^ {2} \leq f (\mathbf {y}) - f (\mathbf {x}) - \langle \nabla f (\mathbf {x}), \mathbf {y} - \mathbf {x} \rangle \leq L \frac {1}{2} \| \mathbf {y} - \mathbf {x} \| ^ {2}, \quad \mathrm {f o r a l l} \mathbf {x}, \mathbf {y},
$$
or  $\mu \mathbf{I} \preceq \nabla^2 f(\mathbf{x}) \preceq L\mathbf{I}$  for all  $\mathbf{x}$ . We measure the quality of a preconditioner  $\mathbf{P}$  by how well  $\mathbf{P}^{-1}$  approximates the Hessian  $\nabla^2 f(\mathbf{x})$ . We define an optimal diagonal preconditioner for  $f$  as
$$
\mathbf {P} _ {*} \in \arg \min  _ {\mathbf {P} \succeq 0, \text {d i a g o n a l}} \kappa \quad \text {s u c h t h a t} \quad \frac {1}{\kappa} \mathbf {P} ^ {- 1} \preceq \nabla^ {2} f (\mathbf {x}) \preceq \mathbf {P} ^ {- 1} \text {f o r a l l} \mathbf {x}, \tag {1}
$$
and denote by  $\kappa_{*}$  the optimal  $\kappa$  above. A related and known measure for the convergence rate of preconditioned methods is  $\kappa (\mathbf{P}^{1 / 2}\nabla^{2}f(\mathbf{x})\mathbf{P}^{1 / 2})$  (Bertsekas, 1999, §1.3.2). Moreover, (1) reduces to the definition of optimal preconditioning for linear systems (Jambulapati et al., 2020; Qu et al., 2022) when  $f$  is quadratic. Alternatively, the optimal preconditioner can be viewed as the matrix  $\mathbf{P}_{*}$  such that  $f$  is 1-smooth and maximally strongly-convex in the norm  $\| \mathbf{x}\|_{\mathbf{P}_*^{-1}}^2 = \langle \mathbf{x},\mathbf{P}_*^{-1}\mathbf{x}\rangle$
$$
\frac {1}{\kappa_ {*}} \frac {1}{2} \| \mathbf {x} - \mathbf {y} \| _ {\mathbf {P} _ {*} ^ {- 1}} ^ {2} \leq f (\mathbf {y}) - f (\mathbf {x}) - \langle \nabla f (\mathbf {x}), \mathbf {y} - \mathbf {x} \rangle \leq \frac {1}{2} \| \mathbf {y} - \mathbf {x} \| _ {\mathbf {P} _ {*} ^ {- 1}} ^ {2}, \quad \text {f o r a l l} \mathbf {x}, \mathbf {y}. \tag {2}
$$
Similar definitions of smoothness and strong-convexity relative to a matrix are common in coordinate descent methods (e.g., Qu et al., 2016; Safaryan et al., 2021), where the matrices are assumed to be known a priori. If we knew  $\mathbf{P}_{*}$ , preconditioned gradient descent using  $\mathbf{P}_{*}$  would converge at the rate
$$
f (x - \mathbf {P} _ {*} \nabla f (\mathbf {x})) - f (\mathbf {x} _ {*}) \leq \left(1 - \frac {1}{\kappa_ {*}}\right) (f (\mathbf {x}) - f (\mathbf {x} _ {*})),
$$
where  $\mathbf{x}_{*}$  minimizes  $f$ . We do not know  $\mathbf{P}_{*}$  and will be searching for a good approximation.
For the standard backtracking line-search on  $L$ -smooth functions, the goal is to find a step-size that works as well as  $1 / L$  without knowledge of  $L$ . To do so, we can start with a large step-size  $\alpha \gg 1 / L$  and check the Armijo condition: the step-size  $\alpha$  makes progress as if  $f$  were  $1 / \alpha$ -smooth, that is,
$$
f (\mathbf {x} - \alpha \nabla f (\mathbf {x})) \leq f (\mathbf {x}) - \alpha \frac {1}{2} \| \nabla f (\mathbf {x}) \| ^ {2}. \tag {3}
$$
If the condition is satisfied, we take the step  $\mathbf{x} - \alpha \nabla f(\mathbf{x})$ . By the descent lemma, (Bertsekas, 1999, A.24), the condition is satisfied if  $\alpha \leq 1 / L$ . So if the condition fails, we know  $\alpha$  is too large and can decrease  $\alpha$ . For diagonal preconditioners, the Armijo condition checks whether the preconditioner makes sufficient progress in the norm induced by  $\mathbf{P}^{-1}$ , as if  $f$  were 1-smooth in Equation (2), that is,
$$
f (\mathbf {x} - \mathbf {P} \nabla f (\mathbf {x})) \leq f (\mathbf {x}) - \frac {1}{2} \| \nabla f (\mathbf {x}) \| _ {\mathbf {P}} ^ {2}. \tag {4}
$$
As with a scalar step-size, sufficient progress holds for any matrix  $\mathbf{P}$  that satisfies  $\nabla^2 f(\mathbf{x})\preceq \mathbf{P}^{-1}$ .
3 Multidimensional Backtracking
The typical presentation of the backtracking line-search maintains a step-size and decreases it when the Armijo condition fails (e.g., Nocedal and Wright, 1999, Alg 3.1). We instead take the following non-standard view, which generalizes more naturally to high dimension; as maintaining a set containing the optimal step-size, and using bisection to narrow down the size of the set. Starting with an interval  $S = [0, \alpha_{\max}]$  containing  $1 / L$ , we pick a candidate step-size  $\alpha$  by "backtracking" by  $\gamma < 1$  from the largest step-size in  $S$ , taking  $\alpha = \gamma \alpha_{\max}$  to balance two properties;
1. Large progress: If the candidate step-size satisfies the Armijo condition and the step is accepted, the value of  $f$  decreases proportionally to  $\alpha$  as in (3). To maximize the progress,  $\gamma$  should be large.
2. Volume shrinkage: If the candidate step-size fails the Armijo condition, we learn that  $\alpha > 1 / L$  and can cut the interval to  $S' = [0, \gamma \alpha_{\max}]$ . To ensure the interval shrinks fast,  $\gamma$  should be small.
Taking  $\gamma = 1/2$  balances both properties;  $\alpha$  is at least  $1/2$  as large as any step-size in  $S$ , and we can halve the interval if the Armijo condition fails. We do not use  $\alpha_{\max}$  as a candidate since, although the largest in  $S$ , it would give no information to update the interval in case it failed the Armijo condition.
For multidimensional backtracking, we can check whether a candidate preconditioner yields sufficient progress with Equation (4) instead of the Armijo condition, and replace the intervals by sets of diagonal preconditioners. The high-level pseudocode is given in Figure 2, where each iteration either leads to an improvement in function value or shrinks the sets of potential step-sizes/preconditioners.
2While we use strong-convexity and twice-differentiability of  $f$  to define the optimal preconditioner, those assumptions can be relaxed to only rely on the PL inequality (Polyak, 1963; Lojasiewicz, 1963) (see Appendix B).
Backtracking line-search
Input: starting point  $\mathbf{x}_0$ , backtracking coefficient  $\gamma$ , set  $\mathcal{S}_0 = [0, \alpha_0^{\max}]$  containing the optimal step-size  $1/L$ .
Iterate for  $t$  in  $0,1,\dots,T$
Pick step-size  $\alpha_{t} = \gamma \alpha_{t}^{\max}$
If  $(\mathbf{x}_t,\bar{\alpha}_t)$  satisfy the Armijo condition (3)
Accept  $\mathbf{x}_{t + 1} = \mathbf{x}_t - \alpha_t\nabla f(\mathbf{x}_t)$
Keep max step-size  $\alpha_{t + 1}^{\mathrm{max}} = \alpha_{t}^{\mathrm{max}}$
Otherwise,
Don't move,  $\mathbf{x}_{t + 1} = \mathbf{x}_t$
Cut max step-size  $\alpha_{t + 1}^{\mathrm{max}} = \gamma \alpha_{t}^{\mathrm{max}}$
Output:  $\mathbf{x}_T$
Multidimensional Backtracking
Input: starting point  $\mathbf{x}_0$ , backtracking coefficient  $\gamma$ , set  $\mathcal{S}_0$  of preconditioners containing the optimal  $\mathbf{P}_{*}$ .
Pick step-sizes  $\mathbf{P}_t = \mathrm{CANDIDATE}(\mathcal{S}_t,\gamma ,\mathbf{x}_t)$
If  $(\mathbf{x}_t,\bar{\mathbf{P}}_t)$  satisfy the Armijo condition (4)
Accept  $\mathbf{x}_{t + 1} = \mathbf{x}_t - \mathbf{P}_t\nabla f(\mathbf{x}_t)$
Keep set  $\mathcal{S}_{t + 1} = \mathcal{S}_t$
Cut set  $\mathcal{S}_{t + 1} = \mathrm{CUT}(\mathcal{S}_t,\mathbf{x}_t,\mathbf{P}_t)$
Figure 2: Pseudocode for the backtracking line-search and multidimensional backtracking. We view backtracking as maintaining a set of step-sizes, testing one at each iteration that either make progress on  $f$  or reduce the size of the set. Steps marked by  $(\dagger)$ , are the subject of Sections 3-5.
To complete the algorithm, we need to define the steps marked as  $(\dagger)$  to select preconditioners that lead to large progress when the step is accepted, while significantly reducing the search space when the preconditioner does not yield sufficient progress. For computational efficiency, we want methods that take  $O(d)$  time and memory like plain gradient descent.
3.1 Guaranteed progress competitive with the optimal preconditioner
We start by formalizing the progress guarantee. If  $\mathbf{P}_t$  satisfies the Armijo condition (4) at  $\mathbf{x}_t$ , the function value decreases by at least  $\| \nabla f(\mathbf{x}_t)\|_{\mathbf{P}_t}^2$ . If we can guarantee that  $\| \nabla f(\mathbf{x}_t)\|_{\mathbf{P}_t}^2 \geq \gamma \| \nabla f(\mathbf{x}_t)\|_{\mathbf{P}_*}^2$  for some  $\gamma > 0$ , we can recover the convergence rate of gradient descent preconditioned with  $\mathbf{P}_*$  up to a factor of  $\gamma$ . However, we do not know  $\mathbf{P}_*$ , but know a set  $S_t$  that contains preconditioners we have not yet ruled out, including  $\mathbf{P}_*$ . To guarantee that  $\mathbf{P}_t$  is competitive with  $\mathbf{P}_*$ , we can enforce that  $\mathbf{P}_t$  is competitive with all the preconditioners in  $S_t$ , as captured by the following definition.
Definition 3.1 ( $\gamma$ -competitive candidate preconditioners). A matrix  $\mathbf{P}_t \in S_t$  is  $\gamma$ -competitive in  $S_t$ , for a gradient  $\nabla f(\mathbf{x}_t)$ , if  $\| \nabla f(\mathbf{x}_t) \|_{\mathbf{P}_t}^2 \geq \gamma \| \nabla f(\mathbf{x}_t) \|_{\mathbf{Q}}^2$  for any  $\mathbf{Q} \in S_t$ .
If  $\mathbf{P}_t$  is  $\gamma$ -competitive, then it is competitive with  $\mathbf{P}_*$  as  $\max_{\mathbf{Q} \in \mathcal{S}_t} \| \nabla f(\mathbf{x}_t) \|_{\mathbf{Q}}^2 \geq \| \nabla f(\mathbf{x}_t) \|_{\mathbf{P}_*}^2$ . However, this is a strong requirement. To illustrate what competitive ratios are attainable, we show in Appendix B that even the optimal preconditioner  $\mathbf{P}_*$  might only be  $1/d$ -competitive, as other preconditioners can lead to more local progress depending on  $\nabla f(\mathbf{x}_t)$ , whereas  $\mathbf{P}_*$  is a fixed global optimal preconditioner. This also suggests that selecting a preconditioner that guarantees more local progress may lead to better performance, which we take advantage of to ensure a  $\gamma = 1/\sqrt{2d}$  competitive ratio.