SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \frac {\tilde {H} _ {i j}}{\sqrt {\tilde {H} _ {i i} \tilde {H} _ {j j}}} - \frac {H _ {i j}}{\sqrt {H _ {i i} H _ {j j}}} \\ = \frac {1}{\sqrt {H _ {i i} H _ {j j}}} (\tilde {H} _ {i j} \frac {\sqrt {H _ {i i} H _ {j j}}}{\sqrt {\tilde {H} _ {i i} \tilde {H} _ {j j}}} - H _ {i j}) \\ = \frac {1}{\sqrt {H _ {i i} H _ {j j}}} \left(g _ {i} g _ {j} \frac {\sqrt {H _ {i i} H _ {j j}}}{\sqrt {\tilde {H} _ {i i} \tilde {H} _ {j j}}} + H _ {i j} \left(\frac {\sqrt {H _ {i i} H _ {j j}}}{\sqrt {\tilde {H} _ {i i} \tilde {H} _ {j j}}} - 1\right)\right) \\ \end{array}
$$
The following Lemma bounds the change in the inverse of preconditioner  $Y^{-1}$ , when there is a rank one perturbation to  $H\succ 0$  in following LogDet problem (11):
$$
\begin{array}{l} Y = \operatorname *{arg  min}_{X\in S_{n}(\mathcal{G})^{+ + }} - \log \det \left(X\right) + \operatorname {Tr}(XH) \\ = \operatorname *{arg  min}_{X\in S_{n}(\mathcal{G})^{+ + }}\mathrm{D}_{\ell \mathrm{d}}(X,H) \\ \end{array}
$$
Lemma A.8 (Rank 1 perturbation of LogDet problem (11)). Let  $H, \tilde{H} \in S_n^{++}$ , such that  $\tilde{H} = H + gg^T / \lambda$ , where  $g \in \mathbb{R}^n$ . Also,  $\tilde{Y} = \arg \min_{X \in S_n(\mathcal{G})^{++}} \mathrm{D}_{\ell \mathrm{d}}(X, \tilde{H})$  and  $Y = \arg \min_{X \in S_n(\mathcal{G})^{++}} \mathrm{D}_{\ell \mathrm{d}}(X, H)$ , where  $\mathcal{G}$  is a chain graph, then
$$
\left| (\tilde {Y} ^ {- 1} - Y ^ {- 1}) _ {i i + k} \right| \leq G _ {\infty} ^ {2} \kappa (k \beta + k + 2) \beta^ {k - 1} / \lambda ,
$$
where  $i, i + k \leq n$ ,  $G_{\infty} = \| g \|_{\infty}$  and  $\max_{i,j} |H_{ij}| / \sqrt{H_{ii} H_{jj}} \leq \beta < 1$ . Let  $\kappa(\mathrm{diag}(H)) := \text{condition number of the diagonal part of } H$ , then  $\kappa := \max(\kappa(\mathrm{diag}(H)), \kappa(\mathrm{diag}(\tilde{H})))$ .
Proof. Using Lemma A.2 will give the following:
$$
\begin{array}{l} \left| \left(\tilde {Y} ^ {- 1} - Y ^ {- 1}\right) _ {i i + k} \right| \\ = \left| \frac {\tilde {H} _ {i i + 1} \dots \tilde {H} _ {i + k - 1 i + k}}{\tilde {H} _ {i + 1 i + 1} \dots \tilde {H} _ {i + k - 1 i + k - 1}} - \frac {H _ {i i + 1} \dots H _ {i + k - 1 i + k}}{H _ {i + 1 i + 1} \dots H _ {i + k - 1 i + k - 1}} \right| \\ = \left| \sqrt {\tilde {H} _ {i i}} \tilde {N} _ {i i + 1} \dots \tilde {N} _ {i + k - 1 i + k} \sqrt {\tilde {H} _ {i + k i + k}} \right. \\ - \sqrt {H _ {i i}} N _ {i i + 1} \dots N _ {i + k - 1 i + k} \sqrt {H _ {i + k i + k}} \\ = \sqrt {\tilde {H} _ {i i} \tilde {H} _ {i + k i + k}} \left| \tilde {N} _ {i i + 1} \dots \tilde {N} _ {i + k - 1 i + k} - N _ {i i + 1} \dots N _ {i + k - 1 i + k} \sqrt {H _ {i i} H _ {i + k i + k} / \tilde {H} _ {i i} \tilde {H} _ {i + k i + k}} \right| \\ \end{array}
$$
where  $N_{ij} = H_{ij} / \sqrt{H_{ii}H_{jj}} < 1$  (Since determinants of  $2\mathrm{x}2$  submatrices of  $\mathrm{H}$  are positive). Expanding  $\tilde{N}_{ii+1} = N_{ii+1} + \theta_{ii+1}$  (from Lemma A.7), subsequently  $\tilde{N}_{ii+2} = N_{ii+2} + \theta_{ii+2}$  and so on will give
$$
\begin{array}{l} \left| \tilde {N} _ {i i + 1} \dots \tilde {N} _ {i + k - 1 i + k} - N _ {i i + 1} \dots N _ {i + k - 1 i + k} \sqrt {\frac {H _ {i i} H _ {i + k i + k}}{\tilde {H} _ {i i} \tilde {H} _ {i + k i + k}}} \right| = \\ \left| \theta_ {i i + 1} \tilde {N} _ {i + 1 i + 2} \dots \tilde {N} _ {i + k - 1 i + k} + N _ {i i + 1} \left(\tilde {N} _ {i + 1 i + 2} \dots \tilde {N} _ {i + k - 1 i + k} - N _ {i + 1 i + 2} \dots N _ {i + k - 1 i + k} \sqrt {\frac {H _ {i i} H _ {i + k i + k}}{\tilde {H} _ {i i} \tilde {H} _ {i + k i + k}}}\right) \right| \\ \end{array}
$$
$$
\begin{array}{l} = \left| \theta_ {i i + 1} \tilde {N} _ {i + 1 i + 2} \dots \tilde {N} _ {i + k - 1 i + k} + N _ {i i + 1} \theta_ {i + 1 i + 2} \tilde {N} _ {i i + 3} \dots \tilde {N} _ {i + k - 1 i + k} + \dots + N _ {i i + 1} \dots N _ {i i + k - 1} \theta_ {i + k - 1 i + k} \right. \\ + N _ {i i + 1} \dots N _ {i i + k} \left(1 - \sqrt {\frac {H _ {i i} H _ {i + k i + k}}{\tilde {H} _ {i i} \tilde {H} _ {i + k i + k}}}\right) | \\ \leq \left(\sum_ {l = 0} ^ {k - 1} \left| \theta_ {i + l i + l + 1} \right|\right) \beta^ {k - 1} + \beta^ {k - 1} \left| 1 - \sqrt {\frac {H _ {i i} H _ {i + k i + k}}{\tilde {H} _ {i i} \tilde {H} _ {i + k i + k}}} \right|, \\ \Rightarrow \left| (\tilde {Y} ^ {- 1} - Y ^ {- 1}) _ {i i + k} \right| \leq \sqrt {\tilde {H} _ {i i} \tilde {H} _ {i + k i + k}} \cdot \left(\left(\sum_ {l = 0} ^ {k - 1} | \theta_ {i + l i + l + 1} |\right) \beta^ {k - 1} + \beta^ {k - 1} \left| 1 - \sqrt {\frac {H _ {i i} H _ {i + k i + k}}{\tilde {H} _ {i i} \tilde {H} _ {i + k i + k}}} \right|\right) \\ \end{array}
$$
where  $\max_{i,j}|N_{i,j}|$ ,  $\max_{i,j}|\tilde{N}_{i,j}|\leq \beta < 1$ . Expanding  $\theta_{i + li + l + 1}$  from Lemma A.7 in the term  $|\theta_{i + li + l + 1}|\sqrt{\tilde{H}_{ii}\tilde{H}_{i + ki + k}}$  will give:
$$
\begin{array}{l} \left| \theta_ {i + l i + l + 1} \right| \sqrt {\tilde {H} _ {i i} \tilde {H} _ {i + k i + k}} \\ = \left| \sqrt {\tilde {H} _ {i i} \tilde {H} _ {i + k i + k}} \frac {g _ {i + l} g _ {i + l + 1}}{\lambda \sqrt {\tilde {H} _ {i + l i + l} \tilde {H} _ {i + l + 1 i + l + 1}}} + \sqrt {\tilde {H} _ {i i} \tilde {H} _ {i + k i + k}} N _ {i + l i + l + 1} \left(\sqrt {\frac {H _ {i + l i + l} H _ {i + l + 1 i + l + 1}}{\tilde {H} _ {i + l i + l} \tilde {H} _ {i + l + 1 i + l + 1}}} - 1\right) \right| \\ \leq \left| \sqrt {\tilde {H} _ {i i} \tilde {H} _ {i + k i + k}} \frac {g _ {i + l} g _ {i + l + 1}}{\lambda \sqrt {\tilde {H} _ {i + l i + l} \tilde {H} _ {i + l + 1 i + l + 1}}} \right| + \left| \sqrt {\tilde {H} _ {i i} \tilde {H} _ {i + k i + k}} N _ {i + l i + l + 1} \left(1 - \sqrt {\frac {H _ {i + l i + l} H _ {i + l + 1 i + l + 1}}{\tilde {H} _ {i + l i + l} \tilde {H} _ {i + l + 1 i + l + 1}}}\right) \right| \\ \end{array}
$$
Since  $H_{i + li + l}H_{i + l + 1i + l + 1}\leq \tilde{H}_{i + li + l}\tilde{H}_{i + l + 1i + l + 1}$
$$
\begin{array}{l} 1 - \sqrt {\frac {H _ {i + l i + l} H _ {i + l + 1 i + l + 1}}{\tilde {H} _ {i + l i + l} \tilde {H} _ {i + l + 1 i + l + 1}}} \leq \max  \left(1 - \frac {H _ {i + l i + l}}{\tilde {H} _ {i + l i + l}}, 1 - \frac {H _ {i + l + 1 i + l + 1}}{\tilde {H} _ {i + l + 1 i + l + 1}}\right) \\ \leq \max  \left(\frac {g _ {i + l} ^ {2}}{\lambda \tilde {H} _ {i + l i + l}}, \frac {g _ {i + l + 1} ^ {2}}{\lambda \tilde {H} _ {i + l + 1 i + l + 1}}\right) \\ \end{array}
$$
Using the above,  $H_{i,i} / H_{j,j} \leq \kappa$ , and  $|g_i| \leq G_\infty, \forall i,j \in [n]$ , gives
$$
\begin{array}{l} \sqrt {\tilde {H} _ {i i} \tilde {H} _ {i + k i + k}} | \theta_ {i + l i + l + 1} | \leq G _ {\infty} ^ {2} \kappa / \lambda + \beta G _ {\infty} ^ {2} \kappa / \lambda \\ \leq G _ {\infty} ^ {2} \kappa (1 + \beta) / \lambda \\ \end{array}
$$
Thus the following part of  $\left|\left(\tilde{Y}^{-1} - Y^{-1}\right)_{ii + k}\right|$  can be upperbounded:
$$
\sqrt {\tilde {H} _ {i i} \tilde {H} _ {i + k i + k}} \left(\left(\sum_ {l = 0} ^ {k - 1} \left| \theta_ {i + l i + l + 1} \right|\right) \beta^ {k - 1}\right) \leq G _ {\infty} ^ {2} \kappa (1 + \beta) k \beta^ {k - 1} / \lambda
$$
Also,  $\sqrt{\tilde{H}_{ii}\tilde{H}_{i + ki + k}}\beta^{k - 1}\left|1 - \sqrt{\frac{H_{ii}H_{i + ki + k}}{\tilde{H}_{ii}\tilde{H}_{i + ki + k}}}\right|\leq \beta^{k - 1}\kappa G_{\infty}^2 /\lambda$  , so
$$
\left| \left(\tilde {Y} ^ {- 1} - Y ^ {- 1}\right) _ {i i + k} \right| \leq G _ {\infty} ^ {2} \kappa (k \beta + k + 2) \beta^ {k - 1} / \lambda
$$
Lemma A.9 ( $\mathcal{O}(\sqrt{T})$  upper bound of  $T_{2}$ ). Given that  $\kappa(\mathrm{diag}(H_t)) \leq \kappa$ ,  $\|w_t - w^*\|_2 \leq D_2$ ,  $\max_{i,j} |(H_t)_{ij}| / \sqrt{(H_t)_{ii}(H_t)_{jj}} \leq \beta < 1$ ,  $\forall t \in [T]$  in Algorithm 1, then  $T_{2}$  in Appendix A.2.1 can be bounded as follows:
$$
T _ {2} \leq \frac {2 0 4 8 \sqrt {T}}{\eta \hat {\epsilon} ^ {5}} (G _ {\infty} D _ {2} ^ {2})
$$
where  $\lambda_{t} = G_{\infty}\sqrt{t}$ , and  $\epsilon = \hat{\epsilon} G_{\infty}\sqrt{T}$  in Algorithm 1, and  $\hat{\epsilon} \leq 1$  is a constant.
Proof. Note that  $T_2 = \frac{1}{2\eta} \cdot \sum_{t=1}^{T-1} (w_{t+1} - w^*)^T (X_{t+1}^{-1} - X_t^{-1})(w_{t+1} - w^*) \leq \sum_{t=1}^{T-1} D_2^2 \left\| (X_{t+1}^{-1} - X_t^{-1}) \right\|_2 / (2\eta)$ . Using  $\| A \|_2 = \rho(A) \leq \| A \|_\infty$  for symmetric matrices  $A,$  we get
$$
\begin{array}{l} \left\| X _ {t + 1} ^ {- 1} - X _ {t} ^ {- 1} \right\| _ {2} \leq \left\| X _ {t + 1} ^ {- 1} - X _ {t} ^ {- 1} \right\| _ {\infty} \\ = \max  _ {i} \left(\sum_ {j} \left| \left(X _ {t + 1} ^ {- 1} - X _ {t} ^ {- 1}\right) _ {i j} \right|\right) \\ \leq 1 6 \frac {G _ {\infty} \kappa}{\sqrt {t} (1 - \beta) ^ {2}} \quad (\text {L e m m a A . 8}) \\ \leq 1 0 2 4 \cdot \frac {G _ {\infty} \kappa}{\sqrt {t} \hat {\epsilon} ^ {4}} \\ \end{array}
$$
Now using  $\kappa \leq 2 / \hat{\epsilon}$  (using Equation (29) and  $(H_{t})_{ii} > \hat{\epsilon})$  and summing up terms in  $T_{2}$  using the above will give the result.
Putting together  $T_{1}, T_{2}$  and  $T_{3}$  from Lemma A.5, Lemma A.9 and Lemma A.6 respectively, when  $\epsilon$ ,  $\lambda_{t}$  are defined as in Lemma A.9:
$$
T _ {1} \leq \frac {1 6 D _ {2} ^ {2} G _ {\infty} \sqrt {T}}{\epsilon^ {2} \eta},
$$
$$
T _ {2} \leq \frac {2 0 4 8 \sqrt {T}}{\eta \hat {\epsilon} ^ {5}} \left(G _ {\infty} D _ {2} ^ {2}\right) \tag {31}
$$
$$
T _ {3} \leq \frac {4 n G _ {\infty} \eta}{\hat {\epsilon} ^ {3}} \sqrt {T} \tag {32}
$$
Setting  $\eta = \frac{D_2}{\hat{\epsilon}\sqrt{n}}$
$$
R _ {T} \leq T _ {1} + T _ {2} + T _ {3} \leq O (\sqrt {n} G _ {\infty} D _ {2} \sqrt {T})
$$
A.2.5 Non-convex guarantees
Minimizing smooth non-convex functions  $f$  is a complex yet interesting problem. In Agarwal et al. [1], this problem is reduced to an online convex optimization, where a sequence of objectives  $f_{t}(w) = f(w) + c\| w - w_{t}\|_{2}^{2}$  are minimized. Using this approach Agarwal et al. [1] established convergence guarantees to reach a stationary point via regret minimization. Thus non-convex guarantees can be obtained from regret guarantees and is our main focus in the paper.
A.3 Numerical stability
In this section we conduct perturbation analysis to derive an end-to-end componentwise condition number (pg. 135, problem 7.11 in [27]) upper bound of the tridiagonal explicit solution in Theorem 3.1. In addition to this, we devise Algorithm 3 to reduce this condition number upper bound for the tridiagonal sparsity structure, and be robust to  $H_{t}$  which don't follow the non-degeneracy condition: any principle submatrix of  $H_{t}$  corresponding to a complete subgraph of  $\mathcal{G}$ .
Theorem A.10 (Condition number of tridiagonal LogDet subproblem (11)). Let  $H \in S_n^{++}$  be such that  $H_{ii} = 1$  for  $i \in [n]$ . Let  $\Delta H$  be a symmetric perturbation such that  $\Delta H_{ii} = 0$  for  $i \in [n]$ , and  $H + \Delta H \in S_n^{++}$ . Let  $P_{\mathcal{G}}(H)$  be the input to 11, where  $\mathcal{G}$  is a chain graph, then
$$
\kappa_ {\infty} ^ {\ell d} \leq \max  _ {i \in [ n - 1 ]} 2 / \left(1 - \beta_ {i} ^ {2}\right) = \hat {\kappa} _ {\infty} ^ {\ell d}, \tag {33}
$$
where,  $\beta_{i} = H_{ii + 1},\kappa_{\infty}^{\ell d}\coloneqq$  componentwise condition number of (11) for perturbation  $\Delta H$
The tridiagonal LogDet problem with inputs  $H$  as mentioned in Theorem A.10, has high condition number when  $1 - \beta_{i}^{2} = H_{ii} - H_{ii + 1}^{2} / H_{i + 1i + 1}$  are low and as a result the preconditioner  $X_{t}$  in
SONew (Algorithm 1) has high componentwise relative errors. We develop Algorithm 3 to be robust to degenerate inputs  $H$ , given that  $H_{ii} > 0$ . It finds a subgraph  $\tilde{\mathcal{G}}$  of  $\mathcal{G}$  for which non-degeneracy conditions in Theorem 3.2 is satisfied and (14) is well-defined. This is done by removing edges which causes inverse  $H_{I_jI_j}^{-1}$  to be singular or  $(H_{jj} - H_{I_jj}^T H_{I_jI_j}^{-1} H_{I_jj})$  to be low. In the following theorem we also show that the condition number upper bound in Theorem A.10 reduces in tridiagonal case. To test the robustness of this method we conducted an ablation study in Table 5, in an Autoencoder benchmark (from Section 5) in bfloat16 where we demonstrate noticeable improvement in performance when Algorithm 3 is used.
Theorem A.11 (Numerically stable algorithm). Algorithm 3 finds a subgraph  $\tilde{\mathcal{G}}$  of  $\mathcal{G}$ , such that explicit solution for  $\tilde{\mathcal{G}}$  in (14) is well-defined. Furthermore, when  $\mathcal{G}$  is a tridiagonal/chain graph, the component-wise condition number upper bound in (33) is reduced upon using Algorithm 3,  $\hat{\kappa}_{\ell d}^{\tilde{\mathcal{G}}} < \hat{\kappa}_{\ell d}^{\mathcal{G}}$ , where  $\hat{\kappa}_{\ell d}^{\tilde{\mathcal{G}}}$ ,  $\hat{\kappa}_{\ell d}^{\mathcal{G}}$  are defined as in Theorem A.10 for graphs  $\tilde{\mathcal{G}}$  and  $\mathcal{G}$  respectively.
The proofs for Theorems A.10 and A.11 are given in the following subsections.
Algorithm 3 Numerically stable banded LogDet solution
1: Input:  $\mathcal{G}-$  tridiagonal or banded graph,  $H-$  symmetric matrix in  $\mathbb{R}^{n\times n}$  with sparsity structure  $\mathcal{G}$  and  $H_{ii} > 0,\gamma-$  tolerance parameter for low schur complements.   
2: Output: Finds subgraph  $\tilde{\mathcal{G}}$  of  $\mathcal{G}$  without any degenerate cases from Lemma A.13 and finds preconditioner  $\hat{X}$  corresponding to the subgraph   
3: Let  $E_{i} = \{(i,j):(i,j)\in E_{\mathcal{G}}\}$  be edges from vertex i to its neighbours in graph  $\mathcal{G}$    
4: Let  $V_{i}^{+} = \{j:i <   j,(i,j)\in E_{\mathcal{G}}\}$  and  $V_{i}^{-} = \{j:i > j,(i,j)\in E_{\mathcal{G}}\}$  , denote positive and negative neighbourhood of vertex i.   
5: Let  $K = \left\{i:H_{ii} - H_{I_i i}^T H_{I_i I_i}^{-1}H_{I_i i}\right.$  is undefined or  $\leq \gamma \}$    
6: Consider a new subgraph  $\tilde{\mathcal{G}}$  with edges  $E_{\tilde{\mathcal{G}}} = E_{\mathcal{G}}\setminus (\bigcup_{i\in K}E_{i}\cup (V_{i}^{+}\times V_{i}^{-}))$    
7: return  $\hat{X}\coloneqq$  SPARSIFIED_INVERSE  $(\tilde{H}_t,\tilde{\mathcal{G}})$  , where  $\tilde{H}_t = P_{\tilde{\mathcal{G}}}(\bar{H}_t)$
A.3.1 Condition number analysis
Theorem A.12 (Full version of Theorem A.10). Let  $H \in S_n^{++}$  such that  $H_{ii} = 1$ , for  $i \in [n]$  and a symmetric perturbation  $\Delta H$  such that  $\Delta H_{ii} = 0$ , for  $i \in [n]$  and  $H + \Delta H \succ 0$ . Let  $\hat{X} = \arg \min_{X \in S_n(\mathcal{G})^{++}} \mathrm{D}_{\ell \mathrm{d}}(X, H^{-1})$  and  $\hat{X} + \Delta \hat{X} = \arg \min_{X \in S_n(\mathcal{G})^{++}} \mathrm{D}_{\ell \mathrm{d}}(X, (H + \Delta H)^{-1})$ , here  $\mathcal{G} := \text{chain/tridiagonal sparsity graph and } S_n(\mathcal{G})^{++}$  denotes positive definite matrices which follows the sparsity pattern  $\mathcal{G}$ .
$$
\begin{array}{l} \kappa_ {\ell d} = \lim  _ {\epsilon \rightarrow 0} \sup  \left\{\frac {\left| \Delta \hat {X} _ {i j} \right|}{\epsilon \left| \hat {X} _ {i j} \right|}: | \Delta H _ {k, l} | \leq | \epsilon H _ {k, l} |, (k, l) \in E _ {\mathcal {G}} \right\} \\ \leq \max _ {i \in [ n - 1 ]} 1 / (1 - \beta_ {i} ^ {2}) \\ \end{array}
$$
where,  $\kappa_{\ell d} :=$  condition number of the LogDet subproblem,  $\kappa_2(.) \coloneqq$  condition number of a matrix in  $\ell_2$  norm,  $\beta_i = H_{ii+1} / \sqrt{H_{ii}H_{i+1i+1}}$
Proof. Consider the offdiagonals for which  $(\hat{X} + \Delta \hat{X})_{ii+1} = -H_{ii+1} / (1 - H_{ii+1}^2) = f(H_{ii+1})$ , where  $f(x) = -x / (1 - x^2)$ . Let  $y = f(x)$ ,  $\hat{y} = f(x + \Delta x)$  and  $|\Delta x / x| \leq \epsilon$  then using Taylor series
$$
\begin{array}{l} \left| \frac {(\hat {y} - y)}{y} \right| = \left| \frac {x f ^ {\prime} (x)}{f (x)} \right| \left| \frac {\Delta x}{x} \right| + O ((\Delta x) ^ {2}) \\ \Longrightarrow \lim  _ {\epsilon \rightarrow 0} \left| \frac {(\hat {y} - y)}{\epsilon y} \right| \leq \frac {x f ^ {\prime} (x)}{f (x)} \\ \end{array}
$$
Using the above inequality, with  $x\coloneqq H_{ii + 1}$  and  $y\coloneqq \hat{X}_{ii + 1}$
$$
\begin{array}{l} \lim  _ {\epsilon \rightarrow 0} \left| \frac {\Delta \hat {X} _ {i i + 1}}{\epsilon \hat {X} _ {i i + 1}} \right| \leq \frac {1 + H _ {i i + 1} ^ {2}}{1 - H _ {i i + 1} ^ {2}} \tag {34} \\ \leq \frac {2}{1 - H _ {i i + 1} ^ {2}} \\ \end{array}
$$
Let  $g(x) = x^{2} / (1 - x^{2})$ , let  $y_{1} = g(w_{1}),y_{2} = g(x_{2}),\hat{y}_{1} = g(w_{1} + \Delta x),\hat{y}_{2} = g(x_{2} + \Delta x)$ . Using Taylor series
$$
\begin{array}{l} \left| \frac {(\hat {y} _ {1} - y _ {1})}{y _ {1}} \right| = \left| \frac {x _ {1} f ^ {\prime} (x _ {1})}{f (x _ {1})} \right| \left| \frac {\Delta x _ {1}}{x _ {1}} \right| + O \left(\left(\Delta x _ {1}\right) ^ {2}\right) \\ \left| \frac {(\hat {y} _ {2} - y _ {2})}{y _ {2}} \right| = \left| \frac {x _ {2} f ^ {\prime} (x _ {2})}{f (x _ {2})} \right| \left| \frac {\Delta x _ {2}}{x _ {2}} \right| + O ((\Delta x _ {2}) ^ {2}) \\ \Rightarrow \lim  _ {\epsilon \rightarrow 0} \frac {\Delta y _ {1} + \Delta y _ {2}}{\epsilon (1 + y _ {1} + y _ {2})} \leq \max  \left(\frac {2}{1 - x _ {1} ^ {2}}, \frac {2}{1 - x _ {2} ^ {2}}\right) \\ \end{array}
$$
Putting  $x_{1} \coloneqq H_{ii + 1}$ ,  $x_{2} \coloneqq H_{ii - 1}$  and analyzing  $y_{1} \coloneqq H_{ii + 1}^{2} / (1 - H_{ii + 1}^{2})$  and  $y_{2} \coloneqq H_{ii - 1}^{2} / (1 - H_{ii - 1}^{2})$  will result in the following
$$
\lim  _ {\epsilon \rightarrow 0} \left| \frac {\Delta \hat {X} _ {i i}}{\hat {X} _ {i i}} \right| \leq \max  \left(\frac {2}{1 - H _ {i i + 1} ^ {2}}, \frac {2}{1 - H _ {i i - 1} ^ {2}}\right) \tag {35}
$$
Since  $\hat{X}_{ii} = 1 + H_{ii+1}^2 / (1 - H_{ii+1}^2) + H_{ii-1}^2 / (1 - H_{ii-1}^2)$ . Putting together Equation (35) and Equation (34), the theorem is proved.
A.3.2 Degenerate  $H_{t}$
In SONew (1), the  $H_{t} = P_{\mathcal{G}}(\sum_{s=1}^{t} g_{s} g_{s}^{T} / \lambda_{t})$  generated in line 4 could be such that the matrix  $\sum_{s=1}^{t} g_{s} g_{s}^{T} / \lambda_{t}$  need not be positive definite and so the schur complements  $H_{ii} - H_{ii+1}^{2} / H_{i+1i+1}$  can be zero, giving an infinite condition number  $\kappa_{\infty}^{\ell d}$  by Theorem A.10. The following lemma describes such cases in detail for a more general banded sparsity structure case.
Lemma A.13 (Degenerate inputs to banded LogDet subproblem). Let  $H = P_{\mathcal{G}}(GG^T)$ , when  $\epsilon = 0$  in Algorithm 1, where  $G \in \mathbb{R}^{n \times T}$  and let  $g_{1:T}^{(i)}$  be  $i^{th}$  row of  $G$ , which is gradients of parameter  $i$  for  $T$  rounds, then  $H_{ij} = \left\langle g_{1:T}^{(i)}, g_{1:T}^{(j)} \right\rangle$ .
- Case 1: For tridiagonal sparsity structure  $\mathcal{G}$ : if  $g_{1:T}^{(j)} = g_{1:T}^{(j+1)}$ , then  $H_{jj} - H_{jj+1}/H_{j+1j+1} = 0$ .
- Case 2: For  $b > 1$  in (14): If  $\mathrm{rank}(H_{J_j J_j}) = \mathrm{rank}(H_{I_j I_j}) = b$ , then  $(H_{jj} - H_{I_j j}^T H_{I_j I_j}^{-1} H_{I_j j}) = 0$  and  $D_{jj} = \infty$ . If  $\mathrm{rank}(H_{I_j I_j}) < b$  then the inverse  $H_{I_j I_j}^{-1}$  doesn't exist and  $D_{jj}$  is not well-defined.
Proof. For  $b = 1$ , if  $g_{1:T}^{(j)} = g_{1:T}^{(j+1)}$ , then  $H_{jj+1} = H_{jj} = H_{j+1j+1} = \left\| g_{1:T}^{(j)} \right\|_2^2$ , thus  $H_{jj}-H_{jj+1}/H_{j+1j+1} = 0$ .
For  $b > 1$ , since  $H_{I_j I_j}$ , using Guttman rank additivity formula,  $\mathrm{rank}(H_{jj} - H_{jj + 1}^2 / H_{j + 1j + 1}) = \mathrm{rank}(H_{J_j J_j}) - \mathrm{rank}(H_{I_j I_j}) = 0$ , thus  $H_{jj} - H_{jj + 1}^2 / H_{j + 1j + 1} = 0$ .
Furthermore, if  $\operatorname{rank}(H) \leq b$ , then all  $b + 1 \times b + 1$  principal submatrices of  $H$  have rank  $b$ , thus  $\forall j, H_{J_j J_j}$  have a rank  $b$ , thus  $D_{jj}$  for all  $j$  are undefined.
If  $GG^{T} = \sum_{i=1}^{T} g_{i} g_{i}$  is a singular matrix, then solution to the LogDet problem might not be well-defined as shown in Lemma A.13. For instance, Case 1 can occur when preconditioning the input layer of an image-based DNN with flattened image inputs, where  $j^{th}$  and  $(j + 1)^{th}$  pixel can be highly correlated throughout the dataset. Case 2 can occur in the first  $b$  iterations in Algorithm 1 when the rank of submatrices  $\mathrm{rank}(H_{I_j I_j}) < b$  and  $\epsilon = 0$ .
Table 3: float32 experiments on Autoencoder benchmark using different band sizes. Band size 0 corresponds to diag-SONew and 1 corresponds to tridiag-SONew. We see the training loss getting better as we increase band size
Band size	0 (diag-SONew)	1 (tridiag-SONew)	4	10
Train CE loss	53.025	51.723	51.357	51.226
A.3.3 Numerically Stable SONew proof
Proof of Theorem A.11
Let  $I_{i} = \{j : i < j, (i,j) \in E_{\mathcal{G}}\}$  and  $I_{i}' = \{j : i < j, (i,j) \in E_{\tilde{\mathcal{G}}}\}$ . Let  $K = \{i : H_{ii} - H_{I_i}^T H_{I_i I_i}^{-1} H_{I_i i} \text{ is undefined or } 0, i \in [n]\}$  denote vertices which are getting removed by the algorithm, then for the new graph  $\tilde{\mathcal{G}}$ ,  $D_{ii} = 1 / H_{ii}, \forall i \in K$  since  $H_{ii} > 0$ .
Let  $\bar{K} = \left\{i:H_{ii} - H_{I_i^i}^T H_{I_i^i I_i}^{-1}H_{I_i i} > 0,i\in [n]\right\}$ . Let for some  $j\in \bar{K}$ , if
$$
l = \arg \min  \left\{i: j <   i, i \in K \cap I _ {j} \right\},
$$
denotes the nearest connected vertex higher than  $j$  for which  $D_{ll}$  is undefined or zero, then according to the definition  $E_{\tilde{\mathcal{G}}}$  in Algorithm 3,  $I_j' = \{j + 1, \dots, l - 1\} \subset I_j$ , since  $D_{jj}$  is well-defined,  $H_{I_jI_j}$  is invertible, which makes it a positive definite matrix (since  $H$  is PSD). Since  $H_{jj} - H_{I_jJ}^T H_{I_jI_j}^{-1} H_{I_jJ} > 0$ , using Guttman rank additivity formula  $H_{J_jJ_j} \succ 0$ , where  $J_j = I_j \cup j$ . Since  $H_{J_j'J_j'}$  is a submatrix of  $H_{J_jJ_j}$ , it is positive definite and hence its schur complement  $H_{jj} - H_{I_jJ}^T H_{I_j'I_j'}^{-1} H_{I_j'J} > 0$ . Thus for all  $j \in [n]$ , the corresponding  $D_{jj}$ 's are well-defined in the new graph  $\tilde{\mathcal{G}}$ .
Note that  $\kappa_{\ell d}^{\tilde{G}} = \max_{i\in [n - 1]}1 / (1 - \beta_i^2) < \max_{i\in \bar{K}}1 / (1 - \beta_i^2) = \kappa_{\ell d}^{\mathcal{G}}$ , for tridiagonal graph, where  $\beta_{i} = H_{ii + 1}$ , in the case where  $H_{ii} = 1$ . This is because the arg  $\max_{i\in [n - 1]}1 / (1 - \beta_i^2)\in K$ .
A.4 Additional Experiments, ablations, and details
A.4.1 Ablations
Effect of band size in banded-SONew Increasing band size in banded-SONew captures more correlation between parameters, hence should expectedly lead to better preconditioners. We confirm this through experiments on the Autoencoder benchmark where we take band size  $= 0$  (diag-SONew), 1 (tridiag-SONew), 4, and 10 in Table 3.
Effect of mini-batch size To find the effect of mini-batch size, in Table 4, We empirically compare SONew with state of the art first-order methods such as Adam and RMSProp, and second-order method Shampoo. We see that SONew performance doesn't deteriorate much when using smaller or larger batch size. First order methods on the other hand suffer significantly. We also notice that Shampoo doesn't perform better than SONew in these regimes.
Table 4: Comparison on Autoencoder with different batch-sizes
Baseline\Batch size	100	1000	5000	10000
RMSProp	55.61	53.33	58.69	64.91
Adam	55.67	54.39	58.93	65.37
Shampoo(20)	53.91	50.70	53.52	54.90
tds	53.84	51.72	54.24	55.87
bds-4	53.52	51.35	53.03	54.89
Effect of Numerical Stability Algorithm 3 On tridiag-SONew and banded-4-SONew, we observe that using Algorithm 3 improves training loss. We present in Table 5 results where we observed significant performance improvements.
Table 5: bffloat16 experiments on Autoencoder benchmark with and without Algorithm 3. We observe improvement in training loss when using Algorithm 3
Optimizer	Train CE loss - without Algorithm 3	Train CE loss - with Algorithm 3
tridiag-SONew	53.150	51.936
band-4-SONew	51.950	51.84