SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Suppose that  $R \geq 10$ ,  $M = \exp(\Theta(R^2 + \log n))$ , and  $l > 0$ .
- Assume that  $\| A \| \leq R$ . Here  $\| A \|$  denotes the spectral norm of matrix  $A$ .
- Suppose that  $b \geq \mathbf{0}_n$  and  $\|b\|_1 \leq 1$ . Here  $\mathbf{0}_n$  denotes a length- $n$  vector where all the entries are zeros. (Here  $b \geq \mathbf{0}_n$  denotes  $b_i \geq 0$  for all  $i \in [n]$ )
- Assume that  $w_{i}^{2} \geq 200 \cdot \exp(R^{2}) + l / \sigma_{\min}(A)^{2}$  for all  $i \in [n]$ . Here  $\sigma_{\min}(A)$  denotes the smallest singular value of matrix  $A$ .
- Let  $x_0$  denote an starting/initial point such that  $M\|x_0 - x^*\|_2 \leq 0.1l$ .
- We use to  $\epsilon \in (0, 0.1)$  represent our accuracy parameter.
- We use  $\delta \in (0, 0.1)$  to represent failure probability.
There is a randomized algorithm that
- runs  $\log (\| x_0 - x^*\| _2 / \epsilon)$  iterations
- spend
$$
O \left(\left(\operatorname {n n z} (A) + d ^ {\omega}\right) \cdot \operatorname {p o l y} \left(\log (n / \delta)\right) \right.
$$
time per iteration,
and finally outputs a vector  $\widetilde{x}\in \mathbb{R}^d$  such that
$$
\Pr \left[ \| \widetilde {x} - x ^ {*} \| _ {2} \leq \epsilon \right] \geq 1 - \delta .
$$
Proof. The proof framework follows from approximate Newton (second order method) literature [134, 135, 136, 137, 138, 139, 140, 141, 104, 105, 107, 110, 111, 142, 143, 144, 145].
Following from Lemma D.47, we know the Hessian of the loss function is positive definite.
Following from Lemma D.48, we know the Hessian of  $L$  is Lipschitz.
Following Section 9 in [105], by running Algorithm 4, we complete the proof.
We remark that  $\omega$  denotes the exponent of matrix multiplication (i.e.,  $n^{\omega}$  is the time of multiplying an  $n\times n$  matrix with another  $n\times n$  matrix). The most naive algorithm gives  $\omega = 3$ . Currently, the state-of-the-art algorithm gives  $\omega = 2.373$ .