SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[37] L. Luo, C. Chen, Z. Zhang, W.-J. Li, and T. Zhang. Robust frequent directions with application in online learning. Journal of Machine Learning Research, 20(45):1-41, 2019. URL http://jmlr.org/papers/v20/17-773.html.
[38] J. Martens and R. Grosse. Optimizing neural networks with Kronecker-factored approximate curvature. In International conference on machine learning, pages 2408-2417. PMLR, 2015.
[39] S. Merity, C. Xiong, J. Bradbury, and R. Socher. Pointer sentinel mixture models, 2016.
[40] Y. Nesterov. A method for unconstrained convex minimization problem with the rate of convergence  $o(1 / k^2)$ . 1983.
[41] N. Qian. On the momentum term in gradient descent learning algorithms. *Neural Networks*, 12 (1):145–151, 1999. ISSN 0893-6080. doi: https://doi.org/10.1016/S0893-6080(98)00116-6. URL https://www.sciencedirect.com/science/article/pii/S0893608098001166.
[42] J. Schmidhuber. Deep learning in neural networks: An overview. *Neural Networks*, 61:85–117, jan 2015. doi: 10.1016/j.neunet.2014.09.003. URL https://doi.org/10.1016%2Fj.neunet.2014.09.003.
[43] S. Shalev-Shwartz et al. Online learning and online convex optimization. Foundations and Trends® in Machine Learning, 4(2):107-194, 2012.
[44] D. F. Shanno. Conditioning of quasi-Newton methods for function minimization. Mathematics of computation, 24(111):647-656, 1970.
[45] N. Shazeer and M. Stern. Adafactor: Adaptive learning rates with sublinear memory cost, 2018.
[46] H.-J. M. Shi, T.-H. Lee, S. Iwasaki, J. Gallego-Posada, Z. Li, K. Rangadurai, D. Mudigere, and M. Rabbat. A distributed data-parallel pytorch implementation of the distributed shampoo optimizer for training neural networks at-scale, 2023.
[47] D. R. So, W. Manke, H. Liu, Z. Dai, N. Shazeer, and Q. V. Le. Primer: Searching for efficient transformers for language modeling, 2022.
[48] T. Tieleman and G. Hinton. Lecture 6.5—rmsprop: Divide the gradient by a running average of its recent magnitude. coursera: Neural networks for machine learning. 2012.
[49] L. Vandenberghe, M. S. Andersen, et al. Chordal graphs and semidefinite optimization. Foundations and Trends® in Optimization, 1(4):241-433, 2015.
[50] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need, 2023.
[51] T. Wang, A. Roberts, D. Hesslow, T. L. Scao, H. W. Chung, I. Beltagy, J. Launay, and C. Raffel. What language model architecture and pretraining objective work best for zero-shot generalization?, 2022.
[52] L. Zhang, S. Shi, and B. Li. Eva: Practical second-order optimization with kronecker-vectorized approximation. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=_Mic8V96Voy.
[53] R. Zhang, S. Fattahi, and S. Sojoudi. Large-scale sparse inverse covariance estimation via thresholding and max-det matrix completion. In International Conference on Machine Learning, pages 5766-5775. PMLR, 2018.
[54] M. Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th international conference on machine learning (icml-03), pages 928–936, 2003.
A Supplementary material
A.1 Properties of LogDet subproblem 14
A.2 Regret bound analysis 15
A.2.1 Regret bound decomposition 15
A.2.2 Properties of tridiagonal preconditioner 16
A.2.3 Upperbounding Regret 17
A.2.4  $\mathcal{O}(\sqrt{T})$  Regret 20
A.2.5 Non-convex guarantees 23
A.3 Numerical stability 23
A.3.1 Condition number analysis 24
A.3.2 Degenerate  $H_{t}$  25
A.3.3 Numerically Stable SONew proof 26
A.4 Additional Experiments, ablations, and details 26
A.4.1 Ablations 26
A.4.2 Memory Requirements 27
A.4.3 Hyperparameter search space 27
A.4.4 Additional Experiments 27
A.4.5 Convex experiments 30
A.1 Properties of LogDet subproblem
Proof of Theorem 3.2
The optimality condition of (11) is  $P_{\mathcal{G}}(X^{-1}) = P_{\mathcal{G}}(H)$ ,  $X \in S_n^{++}(\mathcal{G})$ . Let  $Z = L^{-T}D^{-1}L^{-1}$ , then  $P_{\mathcal{G}}(Z) = H$
$$
Z L = L ^ {- T} D ^ {- 1} \Longrightarrow Z L e _ {j} = L ^ {- T} D ^ {- 1} e _ {j}
$$
Let  $J_{j} = I_{j} \cup j$ , where  $I_{j} = \{j + 1, \dots, j + b\}$  as defined in the theorem, then select  $J_{j}$  indices of vectors on both sides of the second equality above and selecting the  $J_{j}$  indices:
$$
\left[ \begin{array}{l l} Z _ {j j} & Z _ {j I _ {j}} \\ Z _ {I _ {j} j} & Z _ {J _ {j} J _ {j}} \end{array} \right] \left[ \begin{array}{l} 1 \\ L _ {I _ {j}} \end{array} \right] = \left[ \begin{array}{c} 1 / d _ {j j} \\ 0 \end{array} \right] \tag {15}
$$
Note that  $L^{-T}$  is an upper triangular matrix with ones in the diagonal hence  $J_{j}^{th}$  block of  $L^{-T}e_{j}$  will be  $[1,0,0,\ldots]$ . Also, since  $P_{\mathcal{G}}(Z) = H$
$$
\left[ \begin{array}{c c} Z _ {j j} & Z _ {j I _ {j}} \\ Z _ {I _ {j} j} & Z _ {J _ {j} J _ {j}} \end{array} \right] = \left[ \begin{array}{c c} H _ {j j} & H _ {j I _ {j}} \\ H _ {I _ {j} j} & H _ {J _ {j} J _ {j}} \end{array} \right]
$$
Substituting this in the linear equation 15
$$
\left[ \begin{array}{c c} H _ {j j} & H _ {j I _ {j}} \\ H _ {I _ {j} j} & H _ {J _ {j} J _ {j}} \end{array} \right] \left[ \begin{array}{c} 1 \\ L _ {I _ {j}} \end{array} \right] = \left[ \begin{array}{c} 1 / d _ {j j} \\ 0 \end{array} \right]
$$
$$
\left[ \begin{array}{c c} H _ {j j} & H _ {j I _ {j}} \\ H _ {I _ {j} j} & H _ {J _ {j} J _ {j}} \end{array} \right] \left[ \begin{array}{c} d _ {j j} \\ d _ {j j} \cdot L _ {I _ {j}} \end{array} \right] = \left[ \begin{array}{c} 1 \\ 0 \end{array} \right]
$$
$$
H _ {j j} d _ {j j} + d _ {j j} H _ {I _ {j} j} ^ {T} L _ {I _ {j} j} = 1
$$
$$
H _ {I _ {j j}} d _ {j j} + d _ {j j} H _ {I _ {j} I _ {j}} L _ {I _ {j j}} = 0
$$
The lemma follows from solving the above equations. Note that here we used that lower triangular halves of matrices  $L$  and  $H$  have the same sparsity patterns, which follows from the fact that banded
graph is a chordal graph with perfect elimination order  $\{1,2,\ldots ,n\}$ . Furthermore,  $X_{t}$  is positive definite, since as  $(H_{jj} - H_{I_jj}^T H_{I_jI_j}^{-1}H_{I_jj})$  is a schur complement of submatrix of  $H$  formed by  $J_{j} = I_{j}\cup \{j\}$ .
Proof of Theorem 3.1 The proof follows trivially from Theorem 3.1, when  $b$  is set to 1.
A.2 Regret bound analysis
Proof sketch of Theorem 3.3. We decompose the regret into  $R_{T} \leq T_{1} + T_{2} + T_{3}$  in Lemma A.1 and individually bound the terms. Term  $T_{2} = \frac{1}{2\eta} \cdot \sum_{t=1}^{T-1} (w_{t+1} - w^{*})^{T} (X_{t+1}^{-1} - X_{t}^{-1})(w_{t+1} - w^{*})$  depends on closeness of consecutive inverses of preconditioners,  $(X_{t+1}^{-1} - X_{t}^{-1})$ , to upperbound this we first give explicit expressions of  $X_{t}^{-1}$  for tridiagonal preconditioner in Lemma A.2 in Appendix A.2.2. This explicit expression is later used to bound each entry of  $(X_{t+1}^{-1} - X_{t}^{-1})$  with  $O(1/\sqrt{t})$  in Appendix A.2.4, this gives a  $O(\sqrt{T})$  upperbound on  $T_{2}$ . To show an upperbound on  $T_{3} = \sum_{t=1}^{T} \frac{\eta}{2} \cdot g_{t}^{T} X_{t} g_{t}$ , we individually bound  $g_{t}^{T} X_{t} g_{t}$  by using a Loewner order  $X_{t} \preceq \|X_{t}\|_{2} I_{n} \preceq \|X_{t}\|_{\infty} I_{n}$  and show that  $\|X_{t}\|_{\infty} = O(1/\sqrt{T})$  and consequently  $T_{3} = O(\sqrt{T})$ .
A.2.1 Regret bound decomposition
In this subsection we state Lemma A.1 which upper bound the regret  $R_{T}$  using three terms  $T_{1}, T_{2}, T_{3}$ .
Lemma A.1 ([26]). In the OCO problem setup, if a prediction  $w_{t} \in \mathbb{R}^{n}$  is made at round  $t$  and is updated as  $w_{t+1} := w_{t} - \eta X_{t} g_{t}$  using a preconditioner matrix  $X_{t} \in S_{n}^{++}$
$$
\begin{array}{l} R _ {T} \leq \frac {1}{2 \eta} \cdot \left(\left\| w _ {1} - w ^ {*} \right\| _ {X _ {1} ^ {- 1}} ^ {2} - \left\| w _ {T + 1} - w ^ {*} \right\| _ {X _ {T} ^ {- 1}}\right) (16) \\ + \frac {1}{2 \eta} \cdot \sum_ {t = 1} ^ {T - 1} \left(w _ {t + 1} - w ^ {*}\right) ^ {T} \left(X _ {t + 1} ^ {- 1} - X _ {t} ^ {- 1}\right) \left(w _ {t + 1} - w ^ {*}\right) (17) \\ + \sum_ {t = 1} ^ {T} \frac {\eta}{2} \cdot g _ {t} ^ {T} X _ {t} g _ {t} (18) \\ \end{array}
$$
$$
\begin{array}{l} \left\| w _ {t + 1} - w ^ {*} \right\| _ {X _ {t} ^ {- 1}} ^ {2} = \left\| w _ {t} - \eta X _ {t} g _ {t} - w ^ {*} \right\| _ {X _ {t} ^ {- 1}} ^ {2} \\ = \left\| w _ {t} - w ^ {*} \right\| _ {X _ {t} ^ {- 1}} ^ {2} + \eta^ {2} g _ {t} ^ {T} X _ {t} g _ {t} \\ - 2 \eta \left(w _ {t} - w ^ {*}\right) ^ {T} g _ {t} \\ \Longrightarrow 2 \eta \left(w _ {t} - w ^ {*}\right) ^ {T} g _ {t} = \left\| w _ {t} - w ^ {*} \right\| _ {X _ {t} ^ {- 1}} ^ {2} - \left\| w _ {t + 1} - w ^ {*} \right\| _ {X _ {t} ^ {- 1}} ^ {2} \\ + \eta^ {2} g _ {t} ^ {T} X _ {t} g _ {t} \\ \end{array}
$$
Using the convexity of  $f_{t}, f_{t}(w_{t}) - f_{t}(w^{*}) \leq (w_{t} - w^{*})^{T}g_{t}$ , where  $g_{t} = \Delta f_{t}(w_{t})$  and summing over  $t \in [T]$
$$
\begin{array}{l} R _ {T} \leq \sum_ {t = 1} ^ {T} \frac {1}{2 \eta} \cdot \left(\left\| w _ {t} - w ^ {*} \right\| _ {X _ {t} ^ {- 1}} ^ {2} - \left\| w _ {t + 1} - w ^ {*} \right\| _ {X _ {t} ^ {- 1}} ^ {2}\right) (19) \\ + \frac {\eta}{2} \cdot g _ {t} ^ {T} X _ {t} g _ {t} (20) \\ \end{array}
$$
The first summation can be decomposed as follows
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left(\| w _ {t} - w ^ {*} \| _ {X _ {t} ^ {- 1}} ^ {2} - \| w _ {t + 1} - w ^ {*} \| _ {X _ {t} ^ {- 1}} ^ {2}\right) \\ = \left(\| w _ {1} - w ^ {*} \| _ {X _ {1} ^ {- 1}} ^ {2} - \| w _ {T + 1} - w ^ {*} \| _ {X _ {T} ^ {- 1}} ^ {2}\right) \\ + \sum_ {t = 1} ^ {T - 1} \left(w _ {t + 1} - w ^ {*}\right) ^ {T} \left(X _ {t + 1} ^ {- 1} - X _ {t} ^ {- 1}\right) \left(w _ {t + 1} - w ^ {*}\right) \\ \end{array}
$$
Substituting the above identity in the Equation (19) proves the lemma.
Let  $R_{T} \leq T_{1} + T_{2} + T_{3}$ , where
$$
\cdot T _ {1} = \frac {1}{2 \eta} \cdot \left(\left\| w _ {1} - w ^ {*} \right\| _ {X _ {1} ^ {- 1}} ^ {2} - \left\| w _ {T + 1} - w ^ {*} \right\| _ {X _ {T} ^ {- 1}}\right)
$$
$$
\begin{array}{l} T _ {2} = \frac {1}{2 \eta} \cdot \sum_ {t = 1} ^ {T - 1} \left(w _ {t + 1} - w ^ {*}\right) ^ {T} \left(X _ {t + 1} ^ {- 1} - X _ {t} ^ {- 1}\right) \left(w _ {t + 1} - w ^ {*}\right) \tag {21} \\ \cdot T _ {3} = \sum_ {t = 1} ^ {T} \frac {\eta}{2} \cdot g _ {t} ^ {T} X _ {t} g _ {t} \\ \end{array}
$$
A.2.2 Properties of tridiagonal preconditioner
In this subsection, we derive properties of the tridigonal preconditioner obtained from solving the LogDet subproblem (11) with  $\mathcal{G}$  set to a chain graph over ordered set of vertices  $\{1,\dots ,n\}$ :
$$
\begin{array}{l} X _ {t} = \underset {X \in S _ {n} (\mathcal {G}) ^ {+ +}} {\arg \min } - \log \det  (X) + \operatorname {T r} (X H _ {t}) (22) \\ = \underset {X \in S _ {n} (\mathcal {G}) ^ {+ +}} {\arg \min } \mathrm {D} _ {\ell \mathrm {d}} \left(X, H _ {t} ^ {- 1}\right) (23) \\ \end{array}
$$
The second equality holds true only when  $H_{t}$  is positive definite. Although in Algorithm 1 we maintain a sparse  $H_{t} = H_{t - 1} + P_{\mathcal{G}}(g_{t}g_{t}^{T} / \lambda_{t})$ ,  $H_0 = \epsilon I_n$  which is further used in (22) to find the preconditioner  $X_{t}$ , our analysis assumes the full update  $H_{t} = H_{t - 1} + g_{t}g_{t}^{T} / \lambda_{t}$ ,  $H_0 = \epsilon I_n$  followed by preconditioner  $X_{t}$  computation using (23). Note that the preconditioners  $X_{t}$  generated both ways are the same, as shown in Section 3.2.
The following lemma shows that the inverse of tridiagonal preconditioners used in Algorithm 1, will restore  $H_{i,j}$ , when  $(i,j)$  fall in the tridiagonal graph, else, the expression is related to product of  $H_{i+k,i+k+1}$  corresponding to the edges in the path from node  $i$  to  $j$  in chain graph. This lemma will be used later in upper bounding  $T_2$ .
Lemma A.2 (Inverse of tridiagonal preconditioner). If  $\mathcal{G} =$  chain/tridiagonal graph and  $\hat{X} =$  arg  $\min_{X\in S_n(\mathcal{G})^{++}}\mathrm{D}_{\ell \mathrm{d}}(X,H^{-1})$  , then the inverse  $\hat{X}^{-1}$  has the following expression
$$
\left(\hat {X} ^ {- 1}\right) _ {i j} = \left\{ \begin{array}{l l} H _ {i j} & | i - j | \leq 1 \\ \frac {H _ {i i + 1} H _ {i + 1 i + 2} \dots H _ {j - 1 j}}{H _ {i + 1 i + 1} \dots H _ {j - 1 j - 1}} \end{array} \right. \tag {24}
$$
$$
\hat {X} ^ {- 1} \hat {X} ^ {(j)} = e _ {j}
$$
Where  $\hat{X}^{(j)}$  is the  $j^{th}$  column of  $\hat{X}$ . Let  $\hat{Y}$  denote the right hand side of Equation (24).
$$
\begin{array}{l} (\hat {Y} \hat {X}) _ {j j} = \hat {X} _ {j j} \hat {Y} _ {j j} + \hat {X} _ {j - 1 j} \hat {Y} _ {j - 1 j} + \hat {X} _ {j j + 1} \hat {Y} _ {j j + 1} \\ = \hat {X} _ {j j} H _ {j j} + \hat {X} _ {j - 1 j} H _ {j - 1 j} + \hat {X} _ {j j + 1} H _ {j j + 1} \\ = 1 \\ \end{array}
$$
The third equality is by using the following alternative form of Equation (12):
$$
\left(\hat {X} ^ {(1)}\right) _ {i, j} = \left\{ \begin{array}{l} 0, \text {i f} j - i > 1 \\ \frac {- H _ {i , i + 1}}{\left(H _ {i i} H _ {i + 1 , i + 1} - H _ {i + 1 , i + 1} ^ {2}\right)}, \text {i f} j = i + 1 \\ \frac {1}{H _ {i i}} \left(1 + \sum_ {j \in \operatorname {n e i g} _ {\mathcal {G}} (i)} \frac {H _ {i j} ^ {2}}{H _ {i i} H _ {j j} - H _ {i j} ^ {2}}\right), \text {i f} i = j \end{array} \right., \tag {25}
$$
where  $i < j$ . Similarly, the offdiagonals of  $\hat{Y}\hat{X}$  can be evaluated to be zero as follows.
$$
\begin{array}{l} (\hat {Y} \hat {X}) _ {i j} = \hat {Y} _ {i j} \hat {X _ {j}} j + \hat {Y} _ {i j - 1} \hat {X} _ {j - 1 j} + \hat {Y} _ {i j + 1} \hat {X} _ {j + 1 j} \\ = \hat {Y} _ {i j} \hat {X} _ {j j} + \hat {Y} _ {i j} \frac {H _ {j - 1 j - 1}}{H _ {j - 1 j}} + \hat {Y} _ {i j} \frac {H _ {j j + 1}}{H _ {j j}} \hat {X} _ {j + 1 j} \\ = 0 \\ \end{array}
$$
Lemma A.3. Let  $y \in \mathbb{R}^n$ ,
$\beta = \max_{t}\max_{i\in [n - 1]}\left|\left(H_{t}\right)_{ii + 1}\right| / \sqrt{\left(H_{t}\right)_{ii}\left(H_{t}\right)_{i + 1i + 1}} < 1$  then
$$
y ^ {T} X _ {t} ^ {- 1} y \leq \| y \| _ {2} ^ {2} \| \operatorname {d i a g} (H _ {t}) \| _ {2} \left(\frac {1 + \beta}{1 - \beta}\right),
$$
where  $X_{t}$  is defined as in Lemma A.2.
Proof. Let  $\tilde{X}_t^{-1} = \mathrm{diag}(H_t)^{-1/2}\hat{X}_t\mathrm{diag}(H_t)^{-1/2}$
$$
y ^ {T} X _ {t} ^ {- 1} y \leq \left\| \operatorname {d i a g} \left(H _ {t}\right) ^ {1 / 2} y \right\| _ {2} ^ {2} \left\| \tilde {X} _ {t} ^ {- 1} \right\| _ {2} \tag {26}
$$
Using the identity of spectral radius  $\rho(X) \leq \|X\|_{\infty}$  and since  $\tilde{X}$  is positive definite,  $\left\|\tilde{X}_t^{-1}\right\|_2 \leq \|\tilde{X}_t^{-1}\|_{\infty}$
$$
\begin{array}{l} \left\| \tilde {X} _ {t} ^ {- 1} \right\| _ {2} \leq \max  _ {i} \left\{\sum_ {j} \left| (\tilde {X} _ {t} ^ {- 1}) _ {i j} \right| \right\} \\ \leq 1 + 2 \left(\beta + \beta^ {2} + \dots\right) \\ \leq \frac {1 + \beta}{1 - \beta} \\ \end{array}
$$
The second inequality is using Lemma A.2. Substituting this in Equation (26) will give the lemma.
A.2.3 Upperbounding Regret
The following Lemma is used in upper bounding both  $T_{1}$  and  $T_{3}$ . In next subsection, we'll upper bound  $T_{2}$  as well.
Lemma A.4. Let  $\beta = \max_{t\in [T]}\max_{i\in [n - 1]}|(H_t)_{ii + 1}| / \sqrt{(H_t)_{ii}(H_t)_{i + 1i + 1}}$ , then
$$
1 / (1 - \beta) \leq 8 / \hat {\epsilon} ^ {2},
$$
where,  $\hat{\epsilon}$  is a constant in parameter  $\epsilon = \hat{\epsilon} G_{\infty}\sqrt{T}$  and consequently used in initializing  $H_0 = \epsilon I_n$  in line 1 in Algorithm 1,
$$
\begin{array}{l} 1 / (1 - \beta) = \max  _ {t} \max  _ {i \in [ n - 1 ]} \frac {1}{1 - \left| (\hat {H} _ {t}) _ {i i + 1} \right|} (27) \\ = \max  _ {t} \max  _ {i \in [ n - 1 ]} \frac {1 + \left| (\hat {H} _ {t}) _ {i i + 1} \right|}{1 - (\hat {H} _ {t}) _ {i i + 1} ^ {2}} \quad \text {(w h e r e} (\hat {H} _ {t}) _ {i i + 1} = \frac {(H _ {t}) _ {i i + 1}}{\sqrt {(H _ {t}) _ {i i} (H _ {t}) _ {i + 1 i + 1})}} \\ \leq \max  _ {t} \max  _ {i \in [ n - 1 ]} \frac {2 \left(H _ {t}\right) _ {i i} \left(H _ {t}\right) _ {i + 1 i + 1}}{\left(H _ {t}\right) _ {i i} \left(H _ {t}\right) _ {i + 1 i + 1} - \left(H _ {t}\right) _ {i i + 1} ^ {2}} \quad (\text {s i n c e} | (H _ {t}) _ {i i + 1} | \leq \sqrt {\left(H _ {t}\right) _ {i i} \left(H _ {t}\right) _ {i + 1 i + 1}}) \\ \leq \max  _ {t} \max  _ {i \in [ n - 1 ]} \frac {2 \left(H _ {t}\right) _ {i i} \left(H _ {t}\right) _ {i + 1 i + 1}}{\det  \left(\left[ \begin{array}{l l} \left(H _ {t}\right) _ {i i} & \left(H _ {t}\right) _ {i i + 1} \\ \left(H _ {t}\right) _ {i + 1 i} & \left(H _ {t}\right) _ {i + 1 i + 1} \end{array} \right]\right)} (28) \\ \end{array}
$$
Note that  $\begin{bmatrix} (H_t)_{ii} & (H_t)_{ii+1} \\ (H_t)_{i+1i} & (H_t)_{i+1i+1} \end{bmatrix} \succeq \epsilon \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}$  (using line 1 in Algorithm 1), thus  $\det \left( \begin{bmatrix} (H_t)_{ii} & (H_t)_{ii+1} \\ (H_t)_{i+1i} & (H_t)_{i+1i+1} \end{bmatrix} \right) \geq \det \left( \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} \right) = \epsilon^2$ . The numerator last inequality can be upperbounded by bounding  $(H_t)_{ii}$  individually as follows:
$$
\begin{array}{l} (H _ {t}) _ {i i} = \sum_ {s = 1} ^ {t} (g _ {s}) _ {i} ^ {2} / \lambda_ {s} \\ = \sum_ {s = 1} ^ {t} \left(g _ {s}\right) _ {i} ^ {2} / \lambda_ {s} \\ = \sum_ {s = 1} ^ {t} (g _ {s}) _ {i} ^ {2} / \left(G _ {\infty} \sqrt {s}\right) \\ \leq \sum_ {s = 1} ^ {t} G _ {\infty} ^ {2} / (G _ {\infty} \sqrt {s}) \\ \leq \sum_ {s = 1} ^ {t} \frac {G _ {\infty}}{\sqrt {s}} \\ \leq 2 G _ {\infty} \sqrt {t} \tag {29} \\ \end{array}
$$
Substituting the above in (28) gives
$$
\begin{array}{l} 1 / (1 - \beta) \leq \max _ {t} \frac {8 G _ {\infty} ^ {2} t}{\hat {\epsilon} ^ {2} G _ {\infty} ^ {2} T} \\ \leq \frac {8}{\bar {\epsilon} ^ {2}} \\ \end{array}
$$
Lemma A.5 (Upperbound of  $T_{1}$
$$
T _ {1} \leq \frac {1 6 D _ {2} ^ {2} G _ {\infty} \sqrt {T}}{\hat {\epsilon} ^ {2} \eta}, \tag {30}
$$
where  $D_{2} = \max_{t\in [T]}\| w_{t} - w^{*}\|_{2}$  and  $G_{\infty} = \max_{t}\| g_{t}\|_{\infty}$
Proof. Since  $X_{T}$  is positive definite
$$
\begin{array}{l} T _ {1} \leq \frac {\left\| w _ {1} - w ^ {*} \right\| _ {X _ {1} ^ {- 1}} ^ {2}}{2 \eta} \\ = \frac {\left(y ^ {(1)}\right) ^ {T} X _ {1} ^ {- 1} y ^ {(1)}}{2 \eta} \quad \text {(w h e r e} y ^ {(1)} = w _ {1} - w ^ {*}) \\ \leq \frac {\left\| y ^ {(1)} \right\| _ {2} ^ {2} \left\| \operatorname {d i a g} \left(H _ {1}\right) \right\| _ {2}}{2 \eta} \cdot \frac {1 + \beta}{1 - \beta} \quad (\text {L e m m a A . 3}) \\ \leq \frac {D _ {2} ^ {2} \left(G _ {\infty} ^ {2} / \lambda_ {1} + \epsilon\right)}{2 \eta} \cdot \frac {1 + \beta}{1 - \beta} \quad \text {(l i n e 4 i n A l g o r i t h m 1)} \\ \leq \frac {8 D _ {2} ^ {2} \left(G _ {\infty} ^ {2} / \lambda_ {1} + \epsilon\right)}{\hat {\epsilon} ^ {2} \eta} \quad (\text {L e m m a A . 4}) \\ \leq \frac {8 D _ {2} ^ {2} \left(G _ {\infty} + \hat {\epsilon} G _ {\infty} \sqrt {T}\right)}{\hat {\epsilon} ^ {2} \eta} \quad \left(\text {S i n c e} \lambda_ {t} = G _ {\infty} \sqrt {t} \text {a n d} \epsilon = \hat {\epsilon} G _ {\infty} \sqrt {T}\right) \\ \leq \frac {1 6 D _ {2} ^ {2} G _ {\infty} \sqrt {T}}{\hat {\epsilon} ^ {2} \eta} \quad (\hat {\epsilon} <   1) \\ \end{array}
$$
Lemma A.6  $(O(\sqrt{T})$  upperbound on  $T_{3}$
$$
T _ {3} = \sum_ {t = 1} ^ {T} \frac {\eta}{2} \cdot g _ {t} ^ {T} X _ {t} g _ {t} \leq \frac {4 n G _ {\infty} \eta}{\hat {\epsilon} ^ {3}} \sqrt {T}
$$
where,  $\| g_t\|_{\infty}\leq G_{\infty}$  and parameters  $\epsilon = \hat{\epsilon} G_{\infty}\sqrt{T}$ ,  $\lambda_{t} = G_{\infty}\sqrt{t}$  in Algorithm 1.
Proof. Using Theorem 3.1, nonzero entries of  $X_{t}$  can be written as follows:
$$
(X _ {t}) _ {i i} = \frac {1}{H _ {i i}} \left(1 + \sum_ {(i, j) \in E _ {\mathcal {G}}} \frac {H _ {i j} ^ {2}}{H _ {i i} H _ {j j} - H _ {i j} ^ {2}}\right)
$$
$$
(X _ {t}) _ {i i + 1} = - \frac {H _ {i i + 1}}{H _ {i i} H _ {i + 1 i + 1} - H _ {i i + 1} ^ {2}}
$$
where,  $E_{\mathcal{G}}$  denote the set of edges of the chain graph  $\mathcal{G}$  in Theorem 3.1. Also, for brevity, the subscript is dropped for  $H_{t}$ . Let  $\hat{X}_t = \sqrt{\operatorname{diag}(H)} X_t\sqrt{\operatorname{diag}(H)}$ , then  $\hat{X}_t$  can be written as
$$
(\hat {X} _ {t}) _ {i i} = \left(1 + \sum_ {(i, j) \in E _ {\mathcal {G}}} \frac {\hat {H} _ {i j} ^ {2}}{1 - \hat {H} _ {i j} ^ {2}}\right),
$$
$$
(\hat {X} _ {t}) _ {i i + 1} = - \frac {\hat {H} _ {i i + 1}}{1 - \hat {H} _ {i i + 1} ^ {2}},
$$
where,  $\hat{H}_{ij} = H_{ij} / \sqrt{H_{ii}H_{jj}}$ . Note that  $\hat{X}_t \preceq \| \hat{X}_t\|_2 I_n \preceq \| \hat{X}_t\|_\infty I_n$ , using  $\max \{|\lambda_1(\hat{X}_t)|, \ldots, |\lambda_n(\hat{X}_t)|\} \leq \| \hat{X}_t\|_\infty$  (property of spectral radius). So we upperbound  $\| \hat{X}_t\|_\infty = \max_{i \in [n]} \{|\hat{X}_t)_{11}| + |(\hat{X}_t)_{12}|, \ldots, |(\hat{X}_t)_{ii-1}| + |(\hat{X}_t)_{ii}| + |(\hat{X}_t)_{ii+1}|, \ldots, |(\hat{X}_t)_{nn}| + |(\hat{X}_t)_{nn-1}|\}$  next. Individual terms  $|(\hat{X}_t)_{ii-1}| + |(\hat{X}_t)_{ii}| + |(\hat{X}_t)_{ii+1}|$  can be written as follows:
$$
\begin{array}{l} \sum_ {(i, j) \in E _ {\mathcal {G}}} | (\hat {X} _ {t}) _ {i j} | = 1 + \sum_ {(i, j) \in E _ {\mathcal {G}}} \frac {\hat {H} _ {i j} ^ {2}}{1 - \hat {H} _ {i j} ^ {2}} + \frac {| \hat {H} _ {i j} |}{1 - \hat {H} _ {i j} ^ {2}} \\ = 1 + \sum_ {(i, j) \in E _ {\mathcal {G}}} \frac {| \hat {H} _ {i j} |}{1 - | \hat {H} _ {i j} |} \\ \leq 2 \max _ {i \in [ n - 1 ]} \frac {1}{1 - | \hat {H} _ {i i + 1} |} \\ \end{array}
$$
The last inequality is because  $|\hat{H}_{ij}| \leq 1$ . Thus,  $\| \hat{X}_t\|_\infty \leq 2\max_{i\in [n - 1]}\frac{1}{1 - |\hat{H}_{ii + 1}|}$ . Now
$$
\begin{array}{l} g _ {t} ^ {T} X _ {t} g _ {t} \leq g _ {t} ^ {T} \operatorname {d i a g} \left(H _ {t}\right) ^ {- 1 / 2} \hat {X} _ {t} \operatorname {d i a g} \left(H _ {t}\right) ^ {- 1 / 2} g _ {t} \\ \leq \| \hat {X} _ {t} \| _ {\infty} \| \operatorname {d i a g} \left(H _ {t}\right) ^ {- 1 / 2} g _ {t} \| _ {2} ^ {2} \quad \left(\left\| \hat {X} _ {t} \right\| _ {2} \leq \left\| \hat {X} _ {t} \right\| _ {\infty}\right) \\ \leq 2 \max _ {i \in [ n - 1 ]} \frac {1}{1 - | \hat {H} _ {i i + 1} |} g _ {t} ^ {T} \operatorname {d i a g} (H _ {t}) ^ {- 1} g _ {t}. \\ \end{array}
$$
Using  $\mathrm{diag}(H_t) \succeq \epsilon I_n$  (step 1 in Algorithm 1), where  $\epsilon = \hat{\epsilon} G_{\infty}\sqrt{T}$  as set in Lemma A.8, gives
$$
\begin{array}{l} g _ {t} ^ {T} X _ {t} g _ {t} \leq 2 \max _ {i \in [ n - 1 ]} \frac {1}{1 - | \hat {H} _ {i i + 1} |} \frac {\| g _ {t} \| _ {2} ^ {2}}{\epsilon G _ {\infty} \sqrt {T}} \\ \leq 2 \max  _ {i \in [ n - 1 ]} \frac {n G _ {\infty}}{\hat {\epsilon} (1 - | \hat {H} _ {i i + 1} |) \sqrt {T}} \\ \leq \frac {2 n G _ {\infty}}{\hat {\epsilon} (1 - \beta) \sqrt {T}} \quad \left(\text {w h e r e} \beta = \max  _ {t \in [ T ]} \max  _ {i \in [ n - 1 ]} \left| (\hat {H} _ {t}) _ {i i + 1} \right|\right) \\ \end{array}
$$
Summing up over  $t$  gives
$$
\begin{array}{l} \sum_ {t} \frac {\eta}{2} g _ {t} ^ {T} X _ {t} g _ {t} \leq \sum_ {t} \frac {1 6 n G _ {\infty} \eta}{\hat {\epsilon} ^ {3} \sqrt {T}} \quad (\text {U s i n g L e m m a A . 4}) \\ \leq \frac {1 6 n G _ {\infty} \eta}{\epsilon^ {3}} \sqrt {T} \\ \end{array}
$$
A.2.4  $\mathcal{O}(\sqrt{T})$  Regret
In this section we derive a regret upper bound with a  $\mathcal{O}(T^{1/2})$  growth. For this, we upper bound  $T_{2}$  as well in this section. In (21),  $T_{2} = \sum_{t=2}^{T} (w_{t} - w^{*})^{T} (X_{t}^{-1} - X_{t-1}^{-1})(w_{t} - w^{*})$  can be upper bounded to a  $\mathcal{O}(T^{1/2})$  by upper bounding entries of  $X_{t}^{-1} - X_{t-1}^{-1}$  individually. The following lemmas construct a telescoping argument to bound  $\left| (X_{t}^{-1} - X_{t-1}^{-1})_{i,j} \right|$ .
Lemma A.7. Let  $H, \tilde{H} \in S_n^{++}$ , such that  $\tilde{H} = H + g g^T / \lambda$ , where  $g \in \mathbb{R}^n$ , then
$$
\begin{array}{l} \frac {\tilde {H} _ {i j}}{\sqrt {\tilde {H} _ {i i} \tilde {H} _ {j j}}} - \frac {H _ {i j}}{\sqrt {H _ {i i} H _ {j j}}} \\ = \frac {g _ {i} g _ {j}}{\lambda \sqrt {\tilde {H} _ {i i} \tilde {H} _ {j j}}} + \frac {H _ {i j}}{\sqrt {H _ {i i} H _ {j j}}} \left(\sqrt {\frac {H _ {i i} H _ {j j}}{\tilde {H} _ {i i} \tilde {H} _ {j j}}} - 1\right) = \theta_ {i j} \\ \end{array}
$$