SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To show that the last inequality is strict, assume that  $\langle \nabla h(\mathbf{q}),\mathbf{q} - 0\rangle -h(\mathbf{q}) = 0$  for the sake of contradiction. By Lemma C.1, the half-space  $\mathcal{H}\coloneqq \{\mathbf{p}\in \mathbb{R}_{\geq 0}^{d}:\langle [\nabla h(\mathbf{x})]_{+},\mathbf{p}\rangle >0\}$  contains only diagonals of invalid preconditioners, where  $[\nabla h(\mathbf{x})]_{+}\coloneqq \max \{\nabla h(\mathbf{x}),0\}$  entry wise. However,  $(^{1 / L})\mathbf{1}\in \mathcal{H}$  as  $[\nabla h(\mathbf{x})]_{+}\geq 0$  and should be invalid, which is a contradiction since  $f$  is  $L$ -smooth and  $^{1 / L}\mathbf{I}$  is valid. Therefore,  $\langle \nabla h(\mathbf{q}),\mathbf{q} - 0\rangle -h(\mathbf{q}) > 0$
Finally, we can write  $\mathbf{v}$  in terms of  $f$  and  $\mathbf{Q}$ . To do so, first define  $\mathbf{x}^{+} := \mathbf{x} - \mathbf{Q}\nabla f(\mathbf{x})$ , and the gradients of  $f$  at different points by  $\mathbf{g} := \nabla f(\mathbf{x})$  and  $\mathbf{g}^{+} := \nabla f(\mathbf{x}^{+})$ . Then, by the chain-rule,
$$
\nabla h (\mathbf {q}) = - \nabla f (\mathbf {x} - \mathbf {Q} \nabla f (\mathbf {x})) \odot \nabla f (\mathbf {x}) + \frac {1}{2} \nabla f (\mathbf {x}) \odot \nabla f (\mathbf {x}) = - \mathbf {g} ^ {+} \odot \mathbf {g} + \frac {1}{2} \mathbf {g} \odot \mathbf {g},
$$
$$
\begin{array}{l} \langle \nabla h (\mathbf {q}), \mathbf {q} \rangle - h (\mathbf {q}) = - \left\langle \mathbf {g} ^ {+}, \mathbf {Q g} \right\rangle + \frac {1}{2} \langle \mathbf {g}, \mathbf {Q g} \rangle - f (\mathbf {x} ^ {+}) + f (\mathbf {x}) + \frac {1}{2} \langle \mathbf {g}, \mathbf {Q g} \rangle \\ = f (\mathbf {x}) - \left\langle \mathbf {g} ^ {+}, \mathbf {Q g} \right\rangle - f (\mathbf {x} ^ {+}). \\ \end{array}
$$
Plugging these equations in the definition of  $\mathbf{v}$  yields
$$
\mathbf {v} = \frac {\nabla h (\mathbf {q})}{\langle \nabla h (\mathbf {q}) , \mathbf {q} \rangle - h (\mathbf {q})} = \frac {\left(\frac {1}{2} \mathbf {g} - \mathbf {g} ^ {+}\right) \odot \mathbf {g}}{f (\mathbf {x}) - \langle \mathbf {g} ^ {+} , \mathbf {Q g} \rangle - f (\mathbf {x} ^ {+})}.
$$
Remark on assumptions of Proposition 4.2. One may have noticed that we never use the assumption that  $\mathbf{Q}$  fails the Armijo condition (i.e., that  $h(\mathbf{q}) > 0$ ) in the proof of the proposition. In fact, the proposition holds for any  $\mathbf{q} \in \mathbb{R}_{\geq 0}^{d}$ . However, and crucially for our application, we have that  $\mathbf{q}$  is in the half-space  $\mathcal{H}_{>}(\mathbf{u})$  of invalid diagonals from Proposition 4.2. In multidimensional backtracking,  $\mathbf{q}$  is the diagonal of a preconditioner  $\mathrm{Diag}(\mathbf{q})$  that failed the Armijo condition  $h(\mathbf{q}) > 0$ . Since  $\mathbf{q}$  is close to the origin in multidimensional backtracking, we can ensure the half-space  $\mathcal{H}_{>}(\mathbf{u})$  contains a significant portion of our current set of candidate preconditioners, leading to significant shrinkage of the set of candidate preconditioners whenever CUT is invoked.
D Cutting-plane methods
D.1 Boxes
Given a box  $\mathcal{B}(\mathbf{b})$  for some  $\mathbf{b}\in \mathbb{R}_{\geq 0}^{d}$  and a vector  $\mathbf{u}\in \mathbb{R}_{\geq 0}^{d}$ , our cutting plane method needs to find a box  $\mathcal{B}(\mathbf{b}^{+})$  that contains  $\mathcal{B}(\mathbf{b})\cap \bar{\mathcal{H}}_{>}(\mathbf{u})$  which, hopefully, has smaller volume than  $\mathcal{B}(\mathbf{b})$ .
The next lemma gives a formula for the minimum volume box for any  $\mathbf{u}$ , which is used in the main text to define CUT in Equation (7). Moreover, we show that if the half-space  $\mathcal{H}_{>}(\mathbf{u})$  is close enough to the origin (since otherwise we might have  $\mathbf{b}^{+} = \mathbf{b}$ ), then we have a significant volume decrease.
Lemma D.1. Let  $\mathbf{b} \in \mathbb{R}_{\geq 0}^{d}$  and  $\mathbf{q} \in \mathcal{B}(\mathbf{b})$ . Let  $\mathbf{u} \in \mathbb{R}_{\geq 0}^{d}$ . Then the box  $\mathcal{B}(\mathbf{b}^{+})$  with minimum volume that contains  $\mathcal{B}(\mathbf{b}) \cap \mathcal{H}_{\leq}(\mathbf{u})$  is given by (using the convention that  $1 / \mathbf{u}[i] = +\infty$  if  $\mathbf{u}[i] = 0$ )
$$
\mathbf {b} ^ {+} [ i ] := \min  \left\{\mathbf {b} [ i ], ^ {1} / \mathbf {u} _ {[ i ]} \right\}, \quad \forall i \in \{1, \dots , d \}, \tag {17}
$$
Moreover, if  $(1/2d) \cdot \mathbf{b}$  is excluded by the half-space, that is,  $\mathbf{b} \in \mathcal{H}_{>}(\mathbf{u})$ , then  $\mathrm{Vol}(\mathcal{B}(\mathbf{b}^{+})) \leq (1/(d+1)) \mathrm{Vol}(\mathcal{B}(\mathbf{b}^{+}))$ .
Proof. Formula for  $\mathbf{b}^{+}$ . Finding the minimum volume box containing  $\mathcal{B}(\mathbf{b}) \cap \mathcal{H}_{\leq}(\mathbf{u})$
$$
\mathbf{b}^{+} = \operatorname *{arg  min}_{\mathbf{c}\in \mathbb{R}^{d}}\mathrm{Vol}(\mathcal{B}(\mathbf{c}))\quad \text{s.t.}\mathcal{B}(\mathbf{b})\cap \mathcal{H}_{\leq}(\mathbf{u})\subseteq \mathcal{B}(\mathbf{c}),
$$
is equivalent to finding the solution to the following optimization problem:
$$
\mathbf{b}^{+} = \operatorname *{arg  min}_{\mathbf{c}\in \mathbb{R}^{d}}\prod_{i}\mathbf{c}[i]\quad \text{s.t.}\max_{\mathbf{p}\in \mathcal{B}(\mathbf{c})\cap \mathcal{H}_{\leq}(\mathbf{u})}\mathbf{p}[i]\leq \mathbf{c}[i]\text{for each} i\in \{1,\ldots ,d\} .
$$
As the constraints separate over the coordinates, the minimization can be done for each coordinate separately. As the function is increasing in  $\mathbf{c}[i]$ , the minimum is achieved by making all the constraints tight, which gives the formula for  $\mathbf{b}^{+}$  in the statement of the lemma.
Volume decrease. Let us prove the second part of the statement. Thus, assume for the remainder of the proof that  $(1/2d) \cdot \mathbf{b} \in \mathcal{H}_{>}(\mathbf{u})$ . We first show that  $\mathrm{Vol}(\mathcal{B}(\mathbf{b}^{+})) \leq (1/(d + 1)) \mathrm{Vol}(\mathcal{B}(\mathbf{b}^{+}))$  if we assume that the update from  $\mathcal{B}(\mathbf{b})$  to  $\mathcal{B}(\mathbf{b}^{+})$  shrinks the box in only one coordinate, i.e.,
$$
\mathcal {I} := \{i \in [ d ]: \mathbf {b} [ i ] > 1 / \mathbf {u} _ {[ i ]} \} = \{i \in [ d ]: \mathbf {b} ^ {+} [ i ] \neq \mathbf {b} [ i ] \} \text {h a s e x a c t l y o n e e l e m e n t .} \tag {18}
$$
Assume the above holds and  $\mathcal{I} = \{j\}$ . Then, as  $(1/2d) \cdot \mathbf{b} \in \mathcal{H}_{>}(\mathbf{u})$  implies  $\langle \mathbf{u}, (1/2d)\mathbf{b} \rangle > 1$ ,
$$
1 <   \langle \mathbf {u}, (1 / 2 d) \mathbf {b} \rangle \leq \frac {1}{2 d} (\mathbf {u} [ j ] \mathbf {b} [ j ] + d - 1) \Rightarrow (d + 1) \frac {1}{\mathbf {u} [ j ]} \leq \mathbf {b} [ j ].
$$
This together with the fact that  $\mathbf{b}^{+}[i] = \mathbf{b}[i]$  for all  $i\neq j$  and  $\mathbf{b}^{+}[j] = 1 / \mathbf{u}[j]$  yields
$$
\operatorname {V o l} \left(\mathcal {B} \left(\mathbf {b} ^ {+}\right)\right) = \prod_ {i = 1} ^ {d} \mathbf {b} ^ {+} [ i ] = \frac {1}{\mathbf {u} [ j ]} \cdot \prod_ {i \neq j} \mathbf {b} [ i ] \leq \frac {1}{d + 1} \prod_ {i = 1} ^ {d} \mathbf {b} [ i ] = \frac {1}{d + 1} \operatorname {V o l} \left(\mathcal {B} (\mathbf {b})\right).
$$
To complete the proof, we only need to show we may assume (18) holds. Assume the opposite, that is, that there are two distinct coordinates that shrink from  $\mathbf{b}^{+}$  to  $\mathbf{b}$ . We will show that the volume shrinks more, meaning the above bound also applies. Formally, assume there are  $j, k \in \mathcal{I}$  that are distinct. For this part, it will be useful to denote by  $\mathbf{b}^{+}(\mathbf{u})$  the point defined in Equation (17) for a given vector  $\mathbf{u}$ . We will show we can construct  $\mathbf{u}' \in \mathbb{R}_{\geq 0}^{d}$  such that  $\mathrm{Vol}(\mathbf{b}^{+}(\mathbf{u})) \leq \mathrm{Vol}(\mathbf{b}^{+}(\mathbf{u}'))$  while maintaining the property  $(1/2d)\mathbf{b} \in \mathcal{H}_{>}(\mathbf{u}')$  and such that  $\mathbf{b}^{+}(\mathbf{u}')[i] \neq \mathbf{b}[i]$  for all  $i \in \mathcal{I} \setminus \{j\}$ , which makes (18) follow by induction. Indeed, define  $\mathbf{u}' \in \mathbb{R}_{\geq 0}^{d}$  by
$$
\mathbf {u} ^ {\prime} [ i ] := \mathbf {u} [ i ] \text {f o r} i \notin \{j, k \}, \quad \mathbf {u} ^ {\prime} [ j ] := \frac {1}{\mathbf {b} [ j ]}, \quad \text {a n d} \mathbf {u} ^ {\prime} [ k ] := \mathbf {u} [ k ] + \frac {\mathbf {b} [ j ]}{\mathbf {b} [ k ]} \left(\mathbf {u} [ j ] - \frac {1}{\mathbf {b} [ j ]}\right). \tag {19}
$$
First, note that  $(1 / 2d)\mathbf{b}\in \mathcal{H}_{>}(\mathbf{u}^{\prime})$  since
$$
\begin{array}{l} \langle \mathbf {u} ^ {\prime} - \mathbf {u}, \mathbf {b} \rangle = \mathbf {b} [ j ] (\mathbf {u} ^ {\prime} [ j ] - \mathbf {u} [ j ]) + (\mathbf {u} ^ {\prime} [ k ] - \mathbf {u} [ k ]) \mathbf {b} [ k ] \\ = \mathbf {b} [ j ] \left(\frac {1}{\mathbf {b} [ j ]} - \mathbf {u} [ j ]\right) + \left(\frac {\mathbf {b} [ j ]}{\mathbf {b} [ k ]} \left(\mathbf {u} [ j ] - \frac {1}{\mathbf {b} [ j ]}\right)\right) \mathbf {b} [ k ] = 0 \\ \end{array}
$$
and, thus,  $1 < \langle \mathbf{u}, (1/2d)\mathbf{b} \rangle = \langle \mathbf{u}', (1/2d)\mathbf{b} \rangle$ . Let us now show that  $\operatorname{Vol}(\mathcal{B}(\mathbf{b}^{+}(\mathbf{u}))) \leq \operatorname{Vol}(\mathcal{B}(\mathbf{b}^{+}(\mathbf{u}')))$ . Since  $\mathbf{b}^{+}(\mathbf{u})[i] = \mathbf{b}^{+}(\mathbf{u}')[i]$  for  $i \notin \{j, k\}$ , we have
$$
\begin{array}{l} \frac {\operatorname {V o l} \left(\mathcal {B} \left(\mathbf {b} ^ {+} (\mathbf {u})\right)\right)}{\operatorname {V o l} \left(\mathcal {B} \left(\mathbf {b} ^ {+} \left(\mathbf {u} ^ {\prime}\right)\right)\right)} = \frac {\mathbf {b} ^ {+} (\mathbf {u}) [ j ]}{\mathbf {b} ^ {+} \left(\mathbf {u} ^ {\prime}\right) [ j ]} \cdot \frac {\mathbf {b} ^ {+} (\mathbf {u}) [ k ]}{\mathbf {b} ^ {+} \left(\mathbf {u} ^ {\prime}\right) [ k ]} \\ = \frac {\operatorname* {m i n} (\mathbf {b} [ j ] , 1 / \mathbf {u} _ {[ j ]})}{\operatorname* {m i n} (\mathbf {b} [ j ] , 1 / \mathbf {u} ^ {\prime} _ {[ j ]})} \cdot \frac {\operatorname* {m i n} (\mathbf {b} [ k ] , 1 / \mathbf {u} _ {[ k ]})}{\operatorname* {m i n} (\mathbf {b} [ k ] , 1 / \mathbf {u} ^ {\prime} _ {[ k ]})} \\ = \frac {1 / \mathbf {u} [ j ]}{\mathbf {b} [ j ]} \cdot \frac {1 / \mathbf {u} [ k ]}{1 / \mathbf {u} ^ {\prime} [ k ]} \quad (\text {s i n c e} j, k \in \mathcal {I} \text {a n d b y (1 9)}) \\ = \frac {1}{\mathbf {b} [ j ] \mathbf {u} [ j ]} \cdot \frac {1}{\mathbf {u} [ k ]} \left(\mathbf {u} [ k ] + \frac {\mathbf {b} [ j ]}{\mathbf {b} [ k ]} \left(\mathbf {u} [ j ] - \frac {1}{\mathbf {b} [ j ]}\right)\right) \\ = \frac {1}{\mathbf {b} [ j ] \mathbf {u} [ j ]} \cdot \frac {1}{\mathbf {u} [ k ] \mathbf {b} [ k ]} (\mathbf {b} [ k ] \mathbf {u} [ k ] + \mathbf {b} [ j ] \mathbf {u} [ j ] - 1). \\ \end{array}
$$
To get that  $\mathrm{Vol}(\mathbf{b}^{+}(\mathbf{u}))\leq \mathrm{Vol}(\mathbf{b}^{+}(\mathbf{u}^{\prime}))$  , we can show that last line is bounded by  $<  1$  . Using the substitution  $\alpha \coloneqq \mathbf{b}[j]\mathbf{u}[j]$  and  $\beta \coloneqq \mathbf{b}[k]\mathbf{u}[k]$  , we want to show that
$$
\frac {\alpha + \beta - 1}{\alpha \beta} <   1 \iff \alpha \beta - \alpha - \beta + 1 > 0 \iff (\alpha - 1) (\beta - 1) > 0.
$$
This holds if  $\alpha > 1$  and  $\beta > 1$ , is implied by  $j, k \in \mathcal{I}$  since  $\alpha = \mathbf{b}[j]\mathbf{u}[j] > 1$  and  $\beta = \mathbf{b}[k]\mathbf{u}[k] > 1$ . A simple induction shows we may assume (18) holds. To see that  $(\alpha + \beta - 1) / \alpha \beta < 1$ , note that
Equipped with the above lemma, we are in position to prove Theorem 5.1.
Proof of Theorem 5.1. Property (a), holds by induction because, for any  $\mathbf{u}_t$  used in a call to CUT, we have  $\mathbf{P}^* \in \mathcal{H}_{\leq}(\mathbf{u}_t)$  since  $\mathbf{P}^*$  is valid and since by Proposition 4.2 the half-space  $\mathcal{H}_{\leq}(\mathbf{u}_t)$  contains only diagonals of invalid preconditioners. For (b), fix  $t \in \{1, \dots, T\}$  and recall that in this case we have  $S_t = \{\mathrm{Diag}(\mathbf{p}) : \mathbf{p} \in \mathcal{B}(\mathbf{b}_t)\}$  and  $\mathbf{P}_t = (1/2d) \cdot \mathrm{Diag}(\mathbf{b}_t)$ . The competitive ratio of  $1/2d$  follows since  $\mathrm{Diag}(\mathbf{b}_t)$  is the preconditioner that maximizes  $\|\nabla f(\mathbf{x}_t)\|_{\mathbf{P}}$  for  $\mathbf{P} \in \mathcal{B}(\mathbf{b}_t)$ . Finally, for (c) by Lemma D.1 we have that every call to CUT makes the volume of the set decrease by  $1/c := 1/(d+1)$ . Moreover, one can easily verify that  $\mathbf{b}_t[i] \geq \min\{1/L, \mathbf{b}_0[i]\}$  for all  $i \in \{1, \dots, d\}$  since  $\mathcal{B}((1/L) \mathbb{1})$  contains only diagonals of valid preconditioners. Therefore, for  $\mathbf{b}_{\min}[i] := \min\{1/L, \mathbf{b}_0[i]\}$ , the volume of  $\mathcal{B}(\mathbf{b}_t)$  cannot be smaller than  $\mathcal{B}(\mathbf{b}_{\min})$  for all iteration  $t$ . Therefore, the number of times CUT is invoked is no more than
$$
\log_ {c} \bigg (\frac {\operatorname {V o l} (\mathcal {B} (\mathbf {b} _ {0}))}{\operatorname {V o l} (\mathcal {B} (\mathbf {b} _ {\min }))} \bigg) = \log_ {c} \bigg (\prod_ {i = 1} ^ {d} \frac {\mathbf {b} _ {0} [ i ]}{\mathbf {b} _ {\min } [ i ]} \bigg) \leq \log_ {c} ((\| \mathbf {b} _ {0} \| _ {\infty} L) ^ {d}) = d \log_ {c} (\| \mathbf {b} _ {0} \| _ {\infty} L).
$$
D.2 Axis-aligned ellipsoids
We now analyze the cutting-plane method using axis-aligned ellipsoids. Interestingly, the results that we prove in this sections are connected to some of the results from Goemans et al. (2009) via polarity theory. We defer a discussion on this connection to the end of this section.
Different from the main body, it will be helpful for the analysis of the method and proofs of the results to not restrict ellipsoids to the non-negative orthant, as was done in the main text for ease of exposition. For any symmetric positive definite matrix  $\mathbf{A} \in \mathbb{R}^{d \times d}$ , define the ellipsoid given by  $\mathbf{A}$  by
$$
\mathcal {E} (\mathbf {A}) := \{\mathbf {x} \in \mathbb {R} ^ {d}: \langle \mathbf {x}, \mathbf {A x} \rangle \leq 1 \}.
$$
When  $\mathbf{A}$  is diagonal, we say that  $\mathcal{E}(\mathbf{A})$  is axis-aligned. Moreover, we may slightly overload our notation by defining  $\mathcal{E}(\mathbf{a})\coloneqq \mathcal{E}(\mathrm{Diag}(\mathbf{a}))$ .
General ellipsoids. Although we are ultimately interested in working solely with ellipsoids defined by diagonal matrices, we will start by looking at more general ellipsoids, and then exploit symmetry in our case to derive the result in Lemma 5.2. We start with an ellipsoid  $\mathcal{E}(\mathbf{A})$  where  $\mathbf{A}$  is a positive definite matrix. Then, given a vector  $\mathbf{u} \in \mathbb{R}^d$ , we are interested in finding an ellipsoid the intersection
Figure 14: Illustration of the ellipsoids in Theorem D.2 in the left. In the right an illustration of the symmetrized intersection of halfspaces  $\mathcal{S}(\mathbf{u})$  used in the proof of Lemma 5.2 together with the ellipsoid  $\mathcal{E}(\mathbf{u} \odot \mathbf{u})$  used in the convex combination in the lemma.
of  $\mathcal{E}(\mathbf{A})$  with the half-spaces defined by  $\mathbf{u}$  and  $-\mathbf{u}$  that contain the origin, that is, the set
$$
\mathcal {E} (\mathbf {A}) \cap \{\mathbf {x} \in \mathbb {R} ^ {d}: \langle \mathbf {x}, \mathbf {u} \rangle <   1 \} \cap \{\mathbf {x} \in \mathbb {R} ^ {d}: - \langle \mathbf {x}, \mathbf {u} \rangle <   1 \} = \mathcal {E} (\mathbf {A}) \cap \{\mathbf {x} \in \mathbb {R} ^ {d}: | \langle \mathbf {x}, \mathbf {u} \rangle | <   1 \}.
$$
The following theorem shows how to find an ellipsoid that contains the above intersection, and how to guarantee its volume is smaller than  $\mathcal{E}(\mathbf{A})$  if  $\mathbf{u}$  is large enough. Interestingly, note that
$$
\{\mathbf {x} \in \mathbb {R} ^ {d}: | \langle \mathbf {x}, \mathbf {u} \rangle | <   1 \} = \{\mathbf {x} \in \mathbb {R} ^ {d}: (\langle \mathbf {x}, \mathbf {u} \rangle) ^ {2} <   1 \} = \mathcal {E} (\mathbf {u u} ^ {\top}).
$$
The set  $\mathcal{E}(\mathbf{uu}^{\top})$  is a degenerate ellipsoid, in the sense that it is not a compact set, and any  $\mathbf{p}$  orthogonal to  $\mathbf{u}$  is contained in  $\mathcal{E}(\mathbf{uu}^{\top})$ . Still, the next theorem shows how to find a convex combination of  $\mathcal{E}(\mathbf{A})$  and  $\mathcal{E}(\mathbf{uu}^{\top})$ —which always contains  $\mathcal{E}(\mathbf{A}) \cap \mathcal{E}(\mathbf{uu}^{\top})$ —that is guaranteed to have volume smaller than  $\mathcal{E}(\mathbf{A})$  if  $\mathbf{u}$  is large enough. The following result can be seen as the polar result of Goemans et al. (2009, Lemma 2).
Theorem D.2. Let  $\mathbf{A} \in \mathbb{R}^{d \times d}$  be positive definite and let  $\mathbf{u} \in \mathbb{R}^d$ . Let  $\lambda \in (0,1)$  and define
$$
L (\mathbf {A}, \mathbf {u}) := \lambda \mathbf {A} + (1 - \lambda) \mathbf {u u} ^ {\mathsf {T}}.
$$
Then  $\mathcal{E}(\mathbf{A})\cap \mathcal{E}(\mathbf{u}\mathbf{u}^{\top})\subseteq \mathcal{E}(L(\mathbf{A},\mathbf{u}))$  and
$$
\mathrm {V o l} (\mathcal {E} (L (\mathbf {A}, \mathbf {u}))) = \sqrt {\frac {\lambda}{\lambda + (1 - \lambda) \cdot \ell} \cdot \frac {1}{\lambda^ {d}}} \cdot \mathrm {V o l} (\mathcal {E} (\mathbf {A}))
$$
In particular, if  $\ell \coloneqq \| \mathbf{u}\|_{\mathbf{A}^{-1}}^2 >d$  and
$$
\lambda = \frac {\ell}{d} \cdot \frac {d - 1}{\ell - 1}, \tag {20}
$$
then  $\lambda \in (0,1)$  and  $\mathrm{Vol}(\mathcal{E}(L(\mathbf{A},\mathbf{u}))) = \nu_d(\mathbf{u})\mathrm{Vol}(\mathcal{E}(\mathbf{A}))$  where
$$
\nu_ {d} (\mathbf {u}) = \sqrt {\frac {1}{\lambda^ {d}} \cdot \frac {d - 1}{\ell - 1}} = \left(\frac {d}{\ell}\right) ^ {d / 2} \left(\frac {\ell - 1}{d - 1}\right) ^ {(d - 1) / 2} \in (0, 1). \tag {21}
$$
Proof. First, note that for any  $\mathbf{p} \in \mathcal{E}(\mathbf{A}) \cap \mathcal{E}(\mathbf{uu}^{\mathsf{T}})$  and any  $\lambda \in (0,1)$  we have
$$
\langle \mathbf {p}, L (\mathbf {A}, \mathbf {u}) \mathbf {p} \rangle = \lambda \langle \mathbf {p}, \mathbf {A p} \rangle + (1 - \lambda) \langle \mathbf {p}, \mathbf {u} \rangle \leq \lambda + (1 - \lambda) = 1.
$$
Thus,  $\mathcal{E}(L(\mathbf{A},\mathbf{u}))\subseteq \mathcal{E}(\mathbf{A})\cap \mathcal{E}(\mathbf{uu}^{\mathrm{T}})$ . For the volume decrease, recall that for ellipsoids  $\mathcal{E}(\mathbf{A})$  we have  $\mathrm{Vol}(\mathcal{E}(\mathbf{A})) = V_d / \sqrt{\operatorname*{det}(\mathbf{A})}$  where  $V_{d}$  is the volume of the unit sphere in  $\mathbb{R}^d$ . By the matrix-determinant lemma, we have
$$
\det (L (\mathbf {A}, \mathbf {u})) = \left(1 + \frac {1 - \lambda}{\lambda} \cdot \left\langle \mathbf {u}, \mathbf {A} ^ {- 1} \mathbf {u} \right\rangle\right) \det (\lambda \mathbf {A}) = \left(1 + \frac {1 - \lambda}{\lambda} \cdot \ell\right) \lambda^ {d} \det (\mathbf {A}).
$$
$$
\mathrm {V o l} (\mathcal {E} (L (\mathbf {A}, \mathbf {u}))) = \sqrt {\frac {1}{(1 + \frac {1 - \lambda}{\lambda} \cdot \ell)} \cdot \frac {1}{\lambda^ {d}}} \cdot \mathrm {V o l} (\mathcal {E} (\mathbf {A})) = \sqrt {\frac {\lambda}{\lambda + (1 - \lambda) \cdot \ell}} \cdot \frac {1}{\lambda^ {d}} \cdot \mathrm {V o l} (\mathcal {E} (\mathbf {A})).
$$
Finally, for  $\lambda$  defined as in (20) we have
$$
\begin{array}{l} 1 + \frac {1 - \lambda}{\lambda} \cdot \ell = 1 + \left(1 - \frac {\ell (d - 1)}{d (\ell - 1)}\right) \frac {d (\ell - 1)}{\ell (d - 1)} \cdot \ell \quad = 1 + \left(\frac {d (\ell - 1)}{\ell (d - 1)} - 1\right) \cdot \ell , \\ = 1 + \left(\frac {d (\ell - 1) - \ell (d - 1)}{\ell (d - 1)}\right) \cdot \ell = 1 + \frac {\ell - d}{d - 1} = \frac {\ell - 1}{d - 1}, \\ \end{array}
$$
which yields the desired formula for  $\nu_{d}(\mathbf{u})$ .
On the norm of  $\mathbf{u}$ . The above theorem has a requirement on the norm of the vector  $\mathbf{u}$  that defines the half-space  $\mathcal{H}_{\leq}(\mathbf{u})$ . However, in our cutting plane method we obtain  $\mathbf{u}$  from Proposition 4.2 and Proposition 4.3, which do not have any guarantees on the norm of  $\mathbf{u}$  explicitly. Crucially, at any given iteration  $t$  of multidimensional backtracking with ellipsoids, we select a candidate preconditioner  $\mathbf{P}_t = \mathrm{Diag}(\mathbf{p}_t)$  such that  $\| \mathbf{p}_t\|_{\mathbf{A}} = 1 / \sqrt{2d}$ . Then, if it fails the Armijo condition in (4) and  $\mathbf{u}_t$  is as given by Proposition 4.2, then we have  $\mathbf{p}_t\in \mathcal{H}_>(\mathbf{u}_t)$ , that is, the separating hyperplane excludes  $\mathbf{p}_t$ . As we will show, this implies that  $\| \mathbf{u}\|_{\mathbf{A}^{-1}}$  is large.
Lemma D.3. Let  $\mathbf{A} \in \mathbb{R}^{d \times d}$  be positive definite and  $\mathbf{p} \in \mathbb{R}_{\geq 0}^d$  be such that  $\| \mathbf{p} \|_{\mathbf{A}} \leq \gamma$  for some  $\gamma > 0$ . Let  $\mathbf{u} \in \mathbb{R}_{\geq 0}^d$  be such that  $\mathbf{p} \in \mathcal{H}_+(\mathbf{u})$ . Then  $\| \mathbf{u} \|_{\mathbf{A}^{-1}} > 1 / \gamma$ .
Proof. For the sake of contradiction, assume  $\| \mathbf{u}\|_{\mathbf{A}^{-1}}\leq 1 / \gamma$ . Then  $\| \mathbf{u}\|_{\mathbf{A}^{-1}}\cdot \| \mathbf{p}\|_{\mathbf{A}}\leq 1$ . Thus, by the Cauchy-Schwartz inequality,
$$
\langle \mathbf {u}, \mathbf {p} \rangle = \left\langle \mathbf {A} ^ {- 1 / 2} \mathbf {u}, \mathbf {A} ^ {1 / 2} \mathbf {p} \right\rangle \leq \left\| \mathbf {A} ^ {- 1 / 2} \mathbf {u} \right\| \cdot \left\| \mathbf {A} ^ {1 / 2} \mathbf {p} \right\| = \| \mathbf {u} \| _ {\mathbf {A} ^ {- 1}} \cdot \| \mathbf {p} \| _ {\mathbf {A}} \leq 1.
$$
This is a contradiction since  $\mathbf{p} \in \mathcal{H}_{>}(\mathbf{u})$  and, therefore,  $\langle \mathbf{u}, \mathbf{p} \rangle > 1$ .
On the volume decrease. Although the formula  $\nu_{d}(\mathbf{u})$  in Equation (21) can be hard to interpret, we show a simple bound when  $\| \mathbf{u}\|_{\mathbf{A}^{-1}}^{2}\geq 2d$ .
Lemma D.4. Let  $\mathbf{A} \in \mathbb{R}^{d \times d}$  be a positive definite matrix and  $\mathbf{u} \in \mathbb{R}^d$  be such that  $\|\mathbf{u}\|_{A^{-1}}^2 > d$ . For  $c \coloneqq d / \ell \in (0, 1)$  we have  $\nu_d(\mathbf{u}) \leq \sqrt{c \cdot e^{1 - c}}$ , where  $\nu_d$  is defined as in (21). In particular, if  $\|\mathbf{u}\|_{A^{-1}}^2 > d$ , then  $\nu_d(\mathbf{u}) \leq \sqrt[4]{e} / \sqrt{2}$ .
Proof. Define  $\ell \coloneqq \| \mathbf{u}\|_{\mathbf{A}^{-1}}^2 >d$  and  $c\coloneqq {}^d /\ell \in (0,1)$  . Then,
$$
\begin{array}{l} \nu_ {d} (\mathbf {u}) ^ {2} = \left(\frac {d}{\ell}\right) ^ {d} \left(\frac {\ell - 1}{d - 1}\right) ^ {(d - 1)} = \frac {d}{\ell} \cdot \left(\frac {d}{\ell} \cdot \frac {\ell - 1}{d - 1}\right) ^ {(d - 1)} = c \cdot \left(c \cdot \frac {d / c - 1}{d - 1}\right) ^ {(d - 1)} \\ = c \cdot \left(\frac {d - c}{d - 1}\right) ^ {(d - 1)} = c \cdot \left(1 + \frac {1 - c}{d - 1}\right) ^ {(d - 1)} \leq c \cdot e ^ {1 - c}, \\ \end{array}
$$
where the last inequality follows since  $1 + x \leq e^{x}$  for all  $x \in \mathbb{R}$ . In particular, note that  $c \in (0,1) \mapsto c \cdot e^{1 - c}$  is increasing since the derivative of the mapping is positive on  $(0,1)$ . Thus, if  $\| \mathbf{u} \|_{\mathbf{A}^{-1}} \geq 2d$ , then  $c \leq \frac{1}{2}$  and  $c \cdot e^{1 - c} \leq (1/2) \cdot e^{1/2}$ .